## 1 Introduction
For a chatbot model to be effective, it must consider the logical relationship between turns in a conversation. Therefore, the chatbot must consider the surrounding contexts in the ongoing conversation when generating a response. For example, in Table 1, there is a dialogue between a boy and his mother. In this conversation, the answer in turn 7 is based on the information \"bus\" mentioned in the previous turn (turn 3). This example highlights the importance of a chatbot managing the flow of information and connecting utterances coherently and meaningfully throughout the entire conversation. This study aims to use contextual information to generate responses effectively. At position \(i\) in a multi-turn conversation, the task of the conversation modeling is to generate a response for the following position \(i+1\). In this proposed conversation model, we use two contexts: the left and right contexts of the current utterance. The left-context \begin{table}
\begin{tabular}{l l} \hline Turn 1 & Mom, how can we get to the supermarket? \\ Turn 2 & We can take a **bus**. \\ Turn 3 & Does this **bus** go there? \\ Turn 4 & I can’t see clearly. \\ Turn 5 & Let’s step in. \\ Turn 6 & No, it’s not right. \\ Turn 7 & Mom! What’s **bus** number? \\ \hline \end{tabular}
\end{table} 
Table 1: A conversation between two people in which the last turn requires information from earlier in the conversation. stores information from previous utterances, providing historical context for the current one. The right-context, on the other hand, indicates how the current utterance may affect future utterances. We use the left-context by integrating multiple preceding utterances from position \(i\) backward into the Seq2 Seq structure to gather contextual information and generate the corresponding response for position \(i+1\). The more challenging task is how to use the right-context. When the model generates a response at position \(i+1\), there is no utterance available at subsequent positions. So how can we exploit the right-context during the training process? To exploit the right contextual information, we consider the conversation as a Markov decision process like [13; 14; 15]. Then, we apply the reinforcement learning technique to manage it. Instead of stopping the generation of utterances at position \(i+1\), we utilize the result to generate utterances at subsequent positions such as \(i+2\), \(i+3\), and so on. The model is designed to simultaneously optimize the entire utterance sequence, including positions \(i+1\), \(i+2\), and beyond. This is achieved using a combination of a deep learning model and a reinforcement learning mechanism, known as the deep reinforcement learning model. Using this model, we have incorporated both the left-context and right-context of the current utterance into the model's training process. Unlike recent studies that often focus on designing the architecture of models for generating responses, our goal is to describe and understand contextual factors that can improve the model's performance in generating responses. In this study, we take a more general approach, using both the left-context and right-context of the current utterance when generating the subsequent response. Our model aims to generate responses in a multi-turn conversation and is a potent combination of Seq2 Seq architecture and reinforcement learning algorithm. Seq2 Seq is capable of leveraging historical information as a left-context. In contrast, the reinforcement learning algorithm leverages the future impact of the current response. This optimization process helps the model generate appropriate responses that align with the previous conversation and guide its future impact. The contributions of this paper are summarized as follows:  
* Using historical information in the conversation for the Seq2 Seq model can improve consistency in the conversation.