# Towards Cross-Lingual Transfer Based on Self-Learning Conversational Agent Model

Olena Yurchenko, Olga Cherednichenko, Alina Trofimova-Herman, Yevhen Kupriianov

► To cite this version:

Olena Yurchenko, Olga Cherednichenko, Alina Trofimova-Herman, Yevhen Kupriianov. Towards Cross-Lingual Transfer Based on Self-Learning Conversational Agent Model. 6th International Conference on Computational Linguistics and Intelligent Systems (CoLInS 2023), Apr 2023, Kharkiv, Ukraine. pp.194-205 (Volume II). hal-04121498

HAL Id: hal-04121498

https://hal.science/hal-04121498

Submitted on 7 Jun 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Towards Cross-Lingual Transfer Based on Self-Learning Conversational Agent Model

Olena Yurchenko <fnref n="1" />, Olga Cherednichenko <fnref n="2" />, Alina Trofimova-Herman <fnref n="1" /> and Yevhen Kupriianov <fnref n="1" />

**Abstract**

In this work, the goal is to develop a model of conversation system that would be able to acquire knowledge on its own, learning in a dialogue with a person. The implementation of the project lies in an interdisciplinary area including didactics, linguistics and natural language processing, and, in our opinion, the results will have important theoretical and practical significance for each field of science. The idea of the project is creating the intelligence core for the Self-Learning Conversational Agent (SLCA) that acquires knowledge through reinforcement learning using natural didactic models in the dialogue with a person in a natural language. The solution of the set tasks lies at the intersection of artificial intelligence and machine learning methods with linguistic methods of discourse analysis and text synthesis by means of a language of meanings. As a case-study we offer to consider learning a foreign language learning due to the challenges of rapid mastering it and improving and maintaining skills in its usage which are still unsolved.

**Keywords** self-learning model, didactic models, natural language processing, computational semantics, semantic analysis, conversational agents

## 1. Introduction

Today in communication models in natural language far from all the available potential is being realized. Most software solutions use only a small part of the features of the natural language structure including semantics. They often ignore semantics and discourse in terms of creation of the communication process. In our opinion, the elaboration of linguistic support of computer systems using artificial intelligence, capable of building and understanding texts and speech in natural language, is an urgent task.

The goal of the project is to develop a learning model of the Self-Learning Conversational Agent (SLCA) based on an extended improved methodology for learning language and evaluating its capabilities.

A child first studies their native language in the process of natural language learning, i.e. in the dialogue with parents. Later they accumulate knowledge (meanings) and the ways to express it in the native language while communicating with teachers at school, friends, colleagues at work etc. After that, while studying a foreign language (or a specific domain language) they use the created knowledge base, giving only a different form of expression to the meanings existing in their head. That is why for creation of the basis for SLCA training we suggest to build such a model in which the meanings of

COLINS-2023: 7th International Conference on Computational Linguistics and Intelligent Systems, April 20–21, 2023, Kharkiv, Ukraine
EMAIL: Olena.Yurchenko@khpi.edu.ua (O. Yurchenko); olga.cherednichenko@eric.univ-lyon2.fr (O. Cherednichenko); alinatrofimovaherman@gmail.com (A. Trofimova-Herman); eugeniokuprianov@gmail.com (Ye. Kupriianov)
ORCID: 0000-0002-6074-0241 (O. Yurchenko); 0000-0002-9391-5220 (O. Cherednichenko); 0000-0002-0857-4514 (A. Trofimova-Herman); 0000-0002-0801-1789 (Ye. Kupriianov)

<figure><img src="image_2.png" /></figure>

<footnote n="1">¹ National Technical University “Kharkiv Polytechnic Institute”, Kyrpychova str. 2, Kharkiv, 61002, Ukraine</footnote>

<footnote n="2">² Univ Lyon, Univ_Lyon 2, UR ERIC, 5 avenue Mendès France, 69676 Bron Cedex, France</footnote>
situations would be preserved and their form of expression would be chosen according to the needs of
the user.

It is suggested that solving the problem of constructing a cross-lingual transfer model enables the SLCA intelligence core to use the general concepts (meanings) regardless of the language, culture, or sphere of human activity, and in the process of further dialogue with a person it will be possible to learn certain forms of meaning expression in professional, age, social and other jargons, corresponding to the situations.

The SLCA model must be bi-directional, i.e., it can both learn itself and subsequently train a human or another machine. It is supposed to improve considerably the process of gaining knowledge, as it is enough to confirm the correct use of an expression in a certain situation instead of repeating it regularly.

Thus, it becomes possible to apply the SLCA model for any language in any sphere of human activity: robotics, manufacturing, medicine, psychology, education, social networks, e-commerce etc.
In order to develop and practically implement such a system, it is necessary to solve the following tasks:

1. To simulate the process of understanding the meaning of a statement in the context of a natural language by recreating situations in a dialogue in which the meaning of the same concept can change or be ambiguous;

2. To construct an algorithm for teaching the correct choice of one or another form of expression in each dialogical situation;

3. To transform the ‘meanings’ of situations in natural human language into an artificial metalanguage and store them in the knowledge base as a graph model;

4. To teach the system a certain ‘natural language’ by the type of machine learning with reinforcement, which is the closest to natural didactic methods, in our opinion;

5. To teach the system not only to formulate statements in a dialogue on a certain topic for a certain field of knowledge in natural language, but also to replenish the memorized scenarios with new situational contexts.

This way, the solution of the set tasks will lie at the intersection of artificial intelligence and machine learning methods with linguistic methods of discourse analysis and text synthesis by means of a language of meanings represented in the form of a knowledge graph.

We suggest testing the proposed dialogue-based learning model as well as its practical application to be conducted in the domain of foreign language learning, where:

1. There is a lack of human and time resources for working out the multitude of possible contextual situations;

2. It is possible to anticipate the context of the dialogue and hence use transformer models of machine learning.

Therefore, a massive users’ interest and the challenges of rapid mastering a foreign language or improving it and maintaining skills in its usage are still unsolved.

The rest of the paper is organised in the following way. The next section contains a brief overview of approaches for dialogue system creation. The state-of-the-art findings are discussed in the ‘related works’ section. Then we describe the fundamentals of the methodology for achieving the objectives and underline the main ideas. In the fifth section, we present our vision together with developed framework, and discuss the results in the final section.

## 2. Background

The need for data research arises not only in business in order to increase its profits, but also in the society and on the part of individual citizens to understand and justify socially significant decision-making. The project BI4people (https://anr.fr/Project-ANR-19-CE23-0005) is created to help in solving those problems. The aim of BI4people is to bring the power of data analysis to the largest possible audience. However, the challenge is that the data analysis requires expert knowledge and available resources providing information in the special domain language. Collaborative analysis and decision-making give an opportunity to achieve the goals by communicating and learning over a stakeholder group. Social networks, quizzes, brainstorms are tools for collaborative BI. So the main idea is to collect people in one virtual or real space and encourage them leave their comments or opinions for general purpose. Moreover, reusing another collaborator’s results or comments makes general decision-making
too collaborative. In order to organize the collaborative session, we have to communicate in the same language in terms of understanding terminology and peculiarities of data analytics domain. It leads to the issue of virtual assistant creation. We think that Cross-Lingual Transfer Model is the best way to learn domain languages and can provide a real profit towards collaborative session handling.

It should be mentioned that today chatbots are already widely used in various human activities: sales, support and marketing, etc. It is an excellent tool for handling many typical user queries, which saves staff's time and significantly reduces the workload of human operators by automatic elimination of typical queries. However, there are many cases of negative experience of communication with a chatbot and the resultant reluctance to continue interaction. In general, a person's refusal to have a dialogue with a chatbot can be explained by the following:

* The chatbot's automatic imposition of response scripts and the inability to solve the problem 'in a human way';
* The rapid learning of 'bad words' rather than normal statements.

As a result, we need to have a tool for justification and verification of the expressions used by a chatbot. We can conclude that only communication with a domain expert (person to person) in each specific situation can provide reasoning for the application of certain forms of utterances in each context and while discussing lead to development and modification of the situations themselves as well as to creation of the necessary 'fine-tuning' of the chatbot.

Among the best-known examples of successful dialogue systems there is GPT-3 by OpenAI and BERT by Google. Both are built on the Large Language Model (LLM), a natural language processing model that uses large amounts of text data to learn to predict the next words and phrases in a given text. These models are usually trained on large data sets such as text corpora from various sources such as books, news articles, web pages and social media, which significantly slows down or even blocks the process of system learning for low-resource languages, for example.

In addition, although these systems are used for various tasks, such as text generation, machine translation, text classification and answering questions, such systems are not able to speculate or to reason by analogy, because they neither have the view of the world, nor maintain 'a conversation about nothing'. So, for instance, in response to the sentence 'I learnt in a different way...' we only get an apology from the system, not a continuation of the human dialogue.

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /><figcaption>**Figure 1:** An example of an unsuccessful human dialogue with ChatGPT-3</figcaption></figure>

Thus, the novelty of our system compared to the Large Language Model is the creation of a machine learning dialogue model that takes a human dialogue in its native natural language as its template and afterwards can be used as a simulator to teach a foreign natural language or its social or professional dialects to a human.

## 3. Related Works

Despite a large amount of research in the field of chatbot development, many questions have remained open so far. The analysis of publications on the topic [1-4] shows that current trends lie in the application of deep reinforcement learning. Compared to other methods, solutions based on reinforcement learning make it possible to increase the flexibility and adaptability of the dialogue [5, 6], eliminate unwanted or offensive language [1, 2], maintain engagement [7, 8] etc. As the data of the
majority of studies show, for example [6, 8, 9], quite good results are achieved, however, they all use
pre-prepared large corpora of dialogues. The need for a large text corpus for training is a strong
constraint for the implementation of models for low-resource languages or specific domains.

In order not to be limited by the resources of a natural language, we suggest a hybrid combination
of unsupervised cross-lingual representational learning [10, 11] and supervised learning model through
a human dialogue according to a language system model [12], in which, in addition to the structure and
the substance, a subject has been introduced as an obligatory element of the natural language system
capable to identify, interpret the structural-substantial properties and elements of the system, and
perform operations with them according to certain rules and algorithms. This interpretation reflects the
properties of natural language as the means of conveying information. Such properties include, first,
substantive properties of a language because the language embodies ‘materially’ in information-
communication systems. A human being controlling the learning process, an already trained computer
system, a neural network or anything like that can be a subject.

We also offer to refer to the linguistic model of ‘Meaning-Text’, which represents language as a
multilevel model of conversions of the meaning into the text and back, since this theory postulates a
multilevel model of language. That means the one in which the construction of the text, based on a
given meaning, does not occur directly, but through a series of transitions from one level of language
to another. Each level is characterized by a set of its own units and rules of representation, as well as a
set of rules for transition from this level of representation to the neighbouring ones.

Priority in our work is given to the transition from the meaning to the text, i.e. the generation of new utterances within the proposed semantic field, as we believe that a description of the text interpretation process can be derived from discourse analysis [13]. We base on the theoretical assumption that the number of existing semantic units (semes) is smaller than the number of words described by them). Similarly, the number of semantic relations between semes is smaller than the number of relations between meanings of words in a phrase. This enables us to use the model of the metalanguage of meaning to preserve it and generate new texts [14, 15, 16].

The process of a dialogue synthesis focuses on reconstructing the underlying linguistic picture of
the world, specific to each language, and pays special attention to the study of synonymic and
periphrastic means of language [17], as well as the interpretation of complex meanings through simpler
ones [18].

Based on the state-of-the-art analysis we hypothesize that the advantages of the proposed approach
will be:

* Faster process of learning, unlike, for example, corpus models that require time to create large corpora of texts.

* Easier ‘fine-tuning’ to a particular language or domain compared to global language models like GPT-3 or BERT, which will provide greater performance.

* A clearer representation of the same meaning, existing as concepts in many languages, generated in different forms of utterances, which is characteristic of the natural speech we aim to recreate.

* Reducing the number of examples which are needed to train the model by using transfer learning.

# 4. Methods and Materials

The peculiarity of the proposed teaching model is that it is self-studying in the process of a dialogue and is based on the experience of natural human learning of a foreign language (or a new professional domain). Let us consider the case study of learning a foreign language. We should emphasize that, for example, a 7-11 year old child, whose vocabulary has already been formed and is about 30 thousand lexical units, when learning a foreign language in a foreign language environment usually shows exponential growth during the first 6-10 months. Moreover, the smaller the child is, the easier the learning. Everyone starts speaking in the target language starting from 3d-6th month of being in a constant foreign dialogue, except adults who have already formed notions (concepts) and connections between them in another language. Therefore, the learning process of adults under the same conditions is much slower and depends more on the conscious desire and the time devoted to learning. However,
if an adult is sufficiently motivated and open to learning and they cannot be around the teacher all the time to have a continuous dialogue, then this person requires a special tool like virtual dialogue assistant. This way it becomes an immediate task for self-learning conversational systems.

**4.1. Didactic approaches to natural language teaching**

Taking human learning in a natural foreign language as an example, among the many methodological approaches, two main ones should be highlighted, on which the work of our conversational model will be based:

1. Communicative (speech-based) learning;

2. Learning based on speech models (situations).

In doing so, we must consider the peculiarities of the discourse in which the communication takes place, as well as the peculiarities of the language used.

By discourse we mean speech as well as other processes of speech activity, such as text or dialogue. On this basis, by 'meanings' we mean the actual realisation of the meanings of words and expressions in a speech context, fixed at the level of language.

Visually, this relationship can be represented as a three-dimensional space in which the word meanings, which are located in the plane XY and which convey the meaning of the lexeme only by getting into certain contexts and interacting with each other, are inextricably linked and interacting with each other Axis X represents the lexical meaning of the word, axis Y represents the grammatical meaning of the word, axis Z represents the syntactics of the word (lexeme meaning).

<figure><img src="image_5.png" /><figcaption>**Figure 2:** Representation of the meanings in a semantic core</figcaption></figure>

At the same time, it is obvious that the semes as the minimal elements of meanings can be embedded in language both in the lexicon and in other levels of language, such as grammar: morphology and syntax.

Thus, understanding language as a multilevel model implies that the generation of a speech process in a natural language based on a given meaning does not occur directly, but through a series of transitions from one formal level of representation to another: from the morphological (grammatical level), through the syntactic (text level) to the semantic (meaning level).

Each level is characterised by a set of its own units and rules (models) of representation, as well as a set of rules of transition from this level of representation to neighbouring ones, which are supposed to be represented as knowledge graphs [16].

Consequently, a self-studying conversational system can both store the analysed meanings derived from human dialogue and synthesise them itself.
## 4.2. Metalanguage as a formal model of natural language processing

As a semantic parser we propose to use a model of transforming the syntactic structure of a query into a semantic one by means of conceptual graphs, which will be stored in the knowledge base in the form of a formalized metalanguage of meanings. The conceptual graph is a way of semantic or, otherwise, conceptual representation of situations and knowledge in models of understanding of natural language. The nodes of the graph are those lexical units that express categories and concepts and relate to each other by formal and associative links. The orientation of the connections in the graph is always directed from the concepts of a higher level of generalization to the concepts that characterize them.

We define an elementary meaning as a pair of connected neighbouring nodes in a graph. Such links are not always named, sometimes they only capture the fact of some interaction between two words (student - institute, oak - tree). The graph consists of a set of connected elementary meanings, which enter allowed combinations, revealing lexically active and passive valences of this or that word. The connected part of the graph that connects the two nodes forms a subgraph.

The process of creating a semantic category begins with a so-called ‘key word’ being extracted from the core structure of the sentence and the semantic analyser is the final stage of the processing. Once the linguistic processor is completed, the analysed information is transferred to the indexing stage and the text synthesis algorithm is activated.

Models that mimic human speech activity include grammatical correctness models, reflecting the ability to distinguish right from wrong in the language, and functional models, revealing the ability to relate speech content (content plan) to its form (expression plan).

Let us consider an example of a functional model constructed by us based on the Meaning-Text theory (MTT), according to which language is represented as a means for its bearers to perform two intellectual operations:

* Expressing one's thoughts to the others, i.e. encoding the meanings (content) embedded in texts (text construction, synthesis);

* Understanding others' thoughts, i.e. performing the reverse operation of extracting meanings (content) from perceived texts (text understanding, analysis).

A dialogue model can be thought of as a logical device imitating the above operations in their simplest manifestations related exclusively to knowledge of language (vocabulary and grammar). Such a model of some language is a complexly organised set of rules whose purely mechanical application should ideally provide:

* A transition from a given utterance in a particular language to a formal description of its meaning, i.e. to its semantic representation or semantic record.

* Transition from the given content, i.e. semantic representation, to the appropriate utterance in each language.

Here is an example of the generation of several synonymous phrases, one of which is, for example: ‘Orwell is confident that his activism in politics makes his works the best’. The semantic structure that sets the overall meaning of the phrases is shown in Fig. 3.

The underlined word ‘confident’ means that this element of the semantic structure is communicatively dominant. The structure given can be realised in the following forms of phrases:

1. Orwell does not doubt the positive impact of his political activism on the quality of his writing.

2. Orwell has no doubt that the fact that he is politically active makes his works better.

3. Orwell has no doubt that his political activism has a positive effect on the quality of his writings.

4. Orwell is confident that his works are made better by his political activism.

5. That the quality of his works is improved because of his political activity is not in any doubt to
Orwell.

6. Orwell is convinced that his political activities have a positive influence on his work.

7. Orwell is convinced that the quality of his work is improved by his political activism.

8. Orwell is convinced that his work is made better by his political engagement.
<figure><img src="image_6.png" /><figcaption>**Figure 3:** Example of the semantic structure</figcaption></figure>

In our opinion, following the linguistic models of the MTT reveals most fully the phenomenon of language mastery, which implies the human ability:

1. To choose language units (not only lexical) expressing a given content.

2. To correctly combine language units with certain meanings.

3. To rephrase statements with retaining their content (semantic invariant) and thereby not only to answer the question put, but to maintain the conversation, generating new statements within the given semantic field.

Depending on which aspect of speech activity is modelled - listening or speaking - functional models are divided into analytic and synthetic ones. The complete analytic model of a particular language takes as 'input' a segment of a text (usually not less than an utterance) and gives as 'output' a semantic metalingual record (semantic representation) of it (i.e. its interpretation). The complete synthetic model of a language, being opposite to the complete analytical model, receives a semantic record (i.e. a representation of a fragment of meaning) as 'input' and as 'output' it gives a set of synonymous texts in this language representing the needed meaning. Analysis and synthesis models constitute an important component of the self-learning conversational system we are developing.

### 4.3. Methodology for achieving the objectives

From a linguistic point of view, the following methods are proposed to solve the tasks set:

1. Discourse analysis and synthesis will be carried out from a pragmatic point of view, using the logical-semantic approach, which implies obtaining additional information to model the dialogue by transforming its sentences (identifying corollaries, presuppositions, synonymous constructions, etc.).

2. The metalanguage of meanings is modelled using mathematical methods in the form of knowledge graphs (universal concepts and the relationships between them), using linguistic methods of simplification and paraphrasing.

3. Learning experiments are carried out using didactic methods of foreign language teaching.

Thus, the implementation of the project lies in the interdisciplinary field between didactics, linguistics, and machine processing of natural language and, in our opinion, the results will have important theoretical and practical significance for each direction.
## 5. Results

People realise that knowledge base can help in the decision-making process, avoid repeating mistakes and wasting time. When attempting to reuse knowledge, the lack of mechanisms to identify, find and recover reusable knowledge plays an important role. It can be concluded that the problem of reusage has three main causes. The first relates to the understanding of the context and the solution applied, the second relates to the content of the knowledge, and the last relates to the entire knowledge base system that is used. Dialogue does not exist outside a certain context. Understanding the context is based on the depth of knowledge one possesses. This allows them to correctly interpret the statements, follow the purpose of the conversation, maintain the topic, and make the right decisions.

Conversational agents and chatbots training faces significant challenges due to the complexity of modelling in-depth knowledge. Such knowledge is accumulated over the course of a person's life, and the form of its expression (speech, text) reflects their social status, education, age group, occupational specialisation to the same extent as the characteristics of the natural language they speak.

It is generally recognised that knowledge can be explicit (it can be documented) or tacit (it requires face-to-face interaction to transfer it) and should be considered in the creation process.

Knowledge graphs have been proved useful for building relationships between information sources by connecting objects of different types. This is especially useful for synthesising information using multimodal data, such as medical records consisting of X-rays and medical reports in the form of texts or tables. In artificial intelligence, a knowledge graph is usually represented by a set of triples such as subject-predicate-object or object-attribute-value. Many tools exist to obtain such triples and identify the entities that form them.

All these features must be considered for creating the intelligence core of the Self-Learning Conversational Agent. This leads to a conceptual model of dialogue formation based on the triangle of ‘meaning’ – ‘context’ – ‘expression’ (fig. 4).

<figure><img src="image_7.png" /><figcaption>**Figure 4:** Conceptual model of the dialogue formation based on the triangle of ‘meaning’–‘context’–‘expression’</figcaption></figure>

The natural language construction, ‘expression’, can be defined as the representation of meaning in a particular context. The context determines the subset of expressions that are used in each piece of speech. Meanings are formed under the pressure of context and linguistic constructions. All this determines the relation of textual meanings, contexts and speech expressions that are used in the dialogue. This is a limitation of both purely linguistic approaches and machine learning techniques in terms of conversational model creation.

The dialogue exists within a context and understanding that context is crucial for building knowledge from a conversation. To effectively gain knowledge from the context of a dialogue, the following strategies can be used. Firstly, to build knowledge from a dialogue, it is important to pay attention to the history of the conversation. It includes understanding what has been said before, what the current topic is, and how the conversation has progressed over time. Another important strategy for building knowledge from a dialogue is to identify and track key concepts and entities that are being discussed.
This includes people, places, events, and ideas that are relevant to the conversation. By tracking these entities, you can gain a deeper understanding of the context of the dialogue and how it relates to broader knowledge domains.

Overall, to build knowledge from the context of a dialogue, it is important to pay attention to the history of the conversation, identify and track key concepts and entities, use external knowledge sources, and consider the social context of the dialogue. There are several techniques that can be used to help the model reuse knowledge obtained in the dialogue, such as memory networks, knowledge bases, reinforcement learning, and hybrid approaches. A knowledge base can be used to store relevant information, such as facts or domain-specific knowledge.

Capturing and saving the meaning of context over a dialogue requires a combination of NLP techniques, including contextual embeddings, attention mechanisms, memory mechanisms, and discourse analysis. By incorporating these techniques into the model architecture and training process, the model can better capture the nuances of language and generate more relevant and coherent responses.

Here is a scheme to depict a general framework for a conversational agent (fig.5):

<figure><img src="image_8.png" /><figcaption>**Figure 5:** A general framework for a conversational agent</figcaption></figure>

Let us consider each step.

*   **User Input:** This is the user's input to the conversational agent, usually in the form of natural language text or speech.

*   **Natural Language Understanding:** This step involves processing the user input to extract meaning and relevant information. This typically involves techniques like named entity recognition, part-of-speech tagging and dependency parsing.

*   **Dialogue State Tracking:** The dialogue state tracking module uses the output of the natural language understanding module to update the current state of the conversation. This involves keeping track of relevant context and history of the conversation.

*   **Dialogue Policy:** The dialogue policy module decides what the conversational agent should say next, given the current state of the conversation. This typically involves using a machine learning algorithm, such as a reinforcement learning algorithm or a rule-based system, to choose the most appropriate response.

*   **Natural Language Generation:** This step involves generating a natural language response based on the output of the dialogue policy module. This typically involves techniques like text generation and text-to-speech conversion.

*   **Response:** This is the final output of the conversational agent, which is presented to the user as a natural language text or speech response.

This framework provides a basic overview of the main components of a typical conversational agent. It is possible to build a conversational agent based on a transformer model. In fact, transformers have been used successfully in many natural languages processing applications, including chatbots and conversational agents. Compared to seq2seq models, transformers have several advantages, including the ability to handle longer input sequences and a more efficient training process. However, it is important to note that the choice of model architecture depends on the specific requirements of the
application. While transformers may be better suited for some applications, seq2seq models can also be
effective, especially when paired with reinforcement learning techniques.

Transfer learning is a powerful technique for building conversational agents as it allows developers
to leverage pre-trained language models to improve the performance of their models with less training
data and computing resources. It is important to choose a pre-trained language model that is well-suited
to the task and has been trained on a large amount of high-quality data. The next step is fine-tuning the
language model. Once the language model has been fine-tuned, use it to train a dialogue generation
model, such as a seq2seq model or transformer-based model. To further improve the performance of a
conversational agent, reinforcement learning can be incorporated to optimize the dialogue generation
model over time. Reinforcement learning can be used to optimize the model's dialogue generation over
time, based on user's feedback.

By using transfer learning to fine-tune a pre-trained language model and incorporating reinforcement
learning to optimize the dialogue generation model, a powerful conversational agent can be created
with relatively few training data and computing resources. However, it is important to note that building
effective conversational agents is a complex and ongoing process that requires careful attention to users'
needs and their feedback.

6. Discussion

Capturing and saving the meaning of context over a dialogue is a complex task that requires a deep
understanding of natural language processing techniques. One way to capture the meaning of context
is to use contextual word embeddings, such as ELMo, BERT, or RoBERTa. By incorporating attention
mechanisms, the model can better capture the meaning of context and generate more relevant responses.
Memory mechanisms, such as memory networks or recurrent neural networks with external memory,
allow the model to store and retrieve relevant information from previous turns in the dialogue.

Discourse analysis techniques can be used to analyze the structure and flow of the dialogue and identify patterns and themes that are relevant to the context [21]. In the context of a conversational agent, discourse analysis can be used to analyse the structure and content of the conversation between the user and the agent. This can help to identify patterns in the conversation, understand how the conversation is progressing, and identify any issues or problems that may arise. Discourse analysis is an important tool for understanding how language is used in different contexts and can be a useful tool for building more effective and naturalistic conversational agents.

Training a conversational agent requires a large amount of training data, ideally in the form of
human-generated dialogues. The amount of training data required depends on several factors, including
the complexity of the application, the number of dialogues turns, and the diversity of the conversation
topics. Collecting large amounts of high-quality training data can be a time-consuming and expensive
process, especially for specialized domains. Another limitation of training data is the potential for bias.
Finally, it is worth noting that even with large amounts of training data, conversational agents may still
struggle to handle certain types of inputs or respond appropriately in certain situations. This is because
natural language is inherently ambiguous and context-dependent, and there may be many possible
interpretations of a given input or response.

7. Conclusion

The dialogue does not exist outside a certain context and understanding that context is crucial for
building knowledge from the dialogue. The dialogue can be used to build knowledge. The topic of the
dialogue is a critical aspect of the context. By analysing the language and structure of the conversation,
we can infer the topic being discussed and use that information to build knowledge about the subject
matter. Dialogue often involves the discussion of different entities, such as people, places or things. By
analysing the dialogue context, we can identify the relationships between these entities and use that
information to build knowledge about how they are connected. Understanding the intents and goals of
the speakers in the dialogue can provide important context for building knowledge. Dialogue often takes
place within a specific cultural and social context. By analysing the language and structure of the
conversation, we can identify these factors and use that information to build knowledge about the cultural and social context in which the dialogue is taking place.

Thus, by analysing the context of the dialogue, we can extract a wealth of information that can be used to build knowledge about the subject matter being discussed and the broader context in which the conversation is taking place. This information can be used to improve dialogue systems, develop better natural language processing algorithms, and provide more accurate and relevant information to the users.

## 8. Acknowledgements

The research study depicted in this paper is partially funded by the French National Research Agency (ANR), project ANR-19-CE23-0005 BI4people.

## 9. References

[1] Cai, S., Han, D., Li, D., Zheng, Z., & Crespi, N. (2022). An reinforcement learning-based speech censorship chatbot system. Journal of Supercomputing, 78(6), 8751-8773.

[2] Chai, Y., & Liu, G. (2018). Utterance censorship of online reinforcement learning chatbot. In Proceedings - International Conference on Tools with Artificial Intelligence, ICTAI (Vol. 2018-November, pp. 358-362). IEEE Computer Society.

[3] Rajamalli Keerthana, R., Fathima, G., & Florence, L. (2021). Evaluating the performance of various deep reinforcement learning algorithms for a conversational chatbot. In 2021 2nd International Conference for Emerging Technology, INCET 2021. Institute of Electrical and Electronics Engineers Inc.

[4] Kumar, R., & Ali, M. M. (2020). A Review on Chatbot Design and Implementation Techniques. International Research Journal of Engineering and Technology, 2791-2800.

[5] Yin, C., Zhang, R., Qi, J., Sun, Y., & Tan, T. (2018). Context-uncertainty-aware chatbot action selection via parameterized auxiliary reinforcement learning. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 10937 LNAI, pp. 500-512). Springer Verlag.

[6] Liu, J., Pan, F., & Luo, L. (2020). GoChat: Goal-oriented Chatbots with Hierarchical Reinforcement Learning. In SIGIR 2020 - Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 1793-1796). Association for Computing Machinery, Inc.

[7] Cuayáhuitl, H., Lee, D., Ryu, S., Cho, Y., Choi, S., Indurthi, S., Kim, J. (2019). Ensemble-based deep reinforcement learning for chatbots. Neurocomputing, 366, 118-130.

[8] Le, N., Siddique, A. B., Jamour, F., Oymak, S., & Hristidis, V. (2021). Generating Predictable and Adaptive Dialog Policies in Single- And Multi-domain Goal-oriented Dialog Systems. International Journal of Semantic Computing, 15(4), 419-439.

[9] Permatasari, D. A., & Maharani, D. A. (2021). Combination of Natural Language Understanding and Reinforcement Learning for Booking Bot. Journal of Electrical, Electronic, Information, and Communication Technology, 3(1), 12.

[10] Ruder I. Vulić S. and Søgaard A. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research 65 (2019), 569-631.

[11] Ruder S. Unsupervised Cross-lingual Representation Learning. URL: https://www.ruder.io/unsupervised-cross-lingual-learning/#unsupervised-deep-models.

[12] Shirokov, V. (2021) Language. Information. System. Kyiv: PH "Akademperiodyka", 164 p.

[13] Kadmon, N. (2001). Formal Pragmatics: Semantics, Pragmatics, Presupposition, and Focus. Oxford: Blackwell Publishers.

[14] Blackburn, P. & Bos, J. (2005). Representation and Inference for Natural Language: A First Course in Computational Semantics, Stanford, CA: CSLI Publications.
[15] Fox, C. (2010). Computational Semantics, In Alexander Clark, Chris Fox, and Shalom Lappin, editors. The Handbook of Computational Linguistics and Natural Language Processing. Malden, MA: Wiley-Blackwell, 394-428.

[16] Fähndrich, J., Trollmann, F. (2021). AI Reading, or Automatic Semantic Decomposition into Knowledge Graphs and Symbolic reasoning through Marker Passing. INFORMATIK 2021. Gesellschaft für Informatik, Bonn: 891-902.

[17] Milićević, J. (2007). La paraphrase. Modélisation de la paraphrase langagière. Bern: Peter Lang. 395 pp.

[18] Al-Thanyyan, S. S., & Azmi, A. M. (2021). Automated Text Simplification. ACM Computing Surveys.

[19] Y. Kuprianov, Semantic State Superpositions and Their Treatment in Virtual Lexicographic Laboratory for Spanish Language Dictionary, in: Proceedings of the 1st International Conference Computational Linguistics And Intelligent Systems, COLINS-2017, Kharkiv, Ukraine, 2017, pp. 37-46. URL: https://colins.in.ua/wp-content/uploads/2017/12/COLINS-2017_conference_proceedings.pdf.

[20] Dong, Xin and Gerard de Melo. "A Robust Self-Learning Framework for Cross-Lingual Text Classification." Conference on Empirical Methods in Natural Language Processing (2019).

[21] Heidi E. Hamilton, Deborah Tannen, Deborah Schiffrin. The Handbook of Discourse Analysis. John Wiley & Sons, Mar 30, 2015. Language Arts & Disciplines. 976 pages