## 2 Related Work
In recent studies, generating responses based on Seq2 Seq models has achieved significant improvements in various applications, from machine translation [19; 20; 21], text summarization [22; 23; 24], and chatbots [5; 6; 25]. Seq2 Seq-based models are widely used in response generation because they usually achieve a better performance than earlier models. However, they have several drawbacks to multi-turn conversations. They learn by maximizing the probability of generating a response based on the previous dialogue turn using maximum likelihood estimation (MLE). Moreover, MLE may have difficulty estimating the specific targets of chatbot systems. Furthermore, the training dataset includes numerous generic responses, which can cause Seq2 Seq models to generate common and boring responses like \"I don't know\" regardless of the input [8]. These types of responses can lead to the termination of the conversation or fall into an endless loop after three turns [26]. [PERSON] et al. recently introduced a hierarchical neural model that captures relationships between turns in a conversation [5; 11]. They improved their model by training it on a dataset of question-answer pairs and pre-trained embeddings. Additionally, [PERSON] et al. [27] proposed using maximum mutual information (MMI) instead of the MLE objective function for response generation tasks to increase response diversity. Although this improvement may generate more appropriate responses, it is still challenging to achieve the goal of a chatbot: simulating human-to-human interactions by providing informative responses to engage users. Another research direction considers dialogues as a Markov Decision Process (MDP) [13; 14; 15] and uses reinforcement learning techniques to address related issues. RL is a general-purpose framework for sequential decision making and is typically described as an agent that interacts with unknown environments. Many tasks such as generation, reasoning, information extraction, and dialogue can be formulated as sequential decision making. Therefore, in recent years, deep reinforcement learning has garnered a lot of attention in NLP [28]. In chatbot systems, RL-based models consider a conversation as a sequential decision process that operates over its state space, action set, and strategy to address the challenges of multi-turn dialogues in previous models. These models define a dialogue as either a Markov Decision Process (MDP) [13; 14] or a Partially Observable Markov Decision Process (POMDP) [29; 30; 31]. Using reinforcement learning (RL), they monitor the state transition process, take appropriate actions (utterances), and obtain information from the user [28]. The authors of [26] simulated a conversation between two virtual agents, evaluated action sequences using policy gradient methods, and presented rewards for three useful dialogue attributes: informativeness, coherence, and ease of answering. Another study suggested a method based on reinforcement learning to create a chatbot using a generation model that generates sequences for a task-oriented model [32]. The experiments showed that this method results in more natural conversations that more efficiently accomplish task objectives. According to a recent study [15], the authors developed a conversational system for learning policy by incorporating three reward functions. The first reward function evaluates the similarity between previous utterances and the topic presentation. The second one measures semantic coherence using mutual information between the generated response and previous utterances. The final reward function encourages the model to produce grammatically correct and fluent responses. [PERSON] et al. [33] proposed an actor-critical model to implement deep reinforcement learning (DRL). The model was trained in parallel using data collected from various dialogue tasks and tested on 18 tasks from PyDial [34]. The results showed that the model achieved robust learning efficiency.