## 3 The Proposed Model
### Bi-Directional Context Using Deep Reinforcement Learning  
To solve the above problems, we propose formulating the conversation as a reinforcement learning problem and using long-term rewards to optimize response generation. Moreover, to gain insight from the long-context success of a conversation, the proposed model provides an utterance generation model conditioned on the impact of a generated response in an ongoing dialogue. We define the right-context as capturing the influence of utterances in the future. We first predict the best response corresponding to the historical context and then fine-tune the model with a desirable goal conditioned on the future context \(c_{R}\).  
That goal can be achieved with a Reinforcement Learning algorithm through a Markov Decision Process (MDP). MDP is a machine learning method to solve decision problems by interacting with the environment to reach desired goals [47], and is used to solve decision-making problems sequentially. It consists of an agent, including a learner, a decision-maker, and the environment, encompassing all external factors. MDP is a collection of states \(S\), a set of actions \(A\), a transition function \(P\), and a reward function \(R\). Given an MDP \((S,A,P,R)\), the model is trained to find a policy \(\pi\) that solves the problem. From an algorithmic perspective, the agent can learn a reward function \(R\), and the reward function \(R\).  
Figure 3: The architecture of decoder, in which each BERT block has cross-attention layers added between the multi-head attention and the feed-forward network.  
tive, a policy is a conditional probability distribution over the set of actions \(A\). During the interaction, the agent takes action \(a\) according to a policy \(\pi\). The environment updates to a new state based on the agent's action \(a\). More specifically, the agent interacts with the environment at each of a sequence of discrete time steps \(t=0,1,2,\ldots\). At the time \(t\), each pair of the current state \(s_{t}\in S\) and the action \(a_{t}\in A(s)\) creates a transition tuple \((s_{t},a_{t},r_{t+1},s_{t+1})\), with \(s_{t+1}\) being the next state, and the agent also receives a numerical reward, \(r_{t+1}\in R\). The environment and agent produce a sequence \(s_{0},a_{0},r_{1},s_{1},a_{1},r_{2},s_{2},a_{2},r_{3},\ldots\), where \(r_{t}\) and \(s_{t}\) are defined by discrete probability distributions conditioned on the previous state and action. At time \(t\), the transition probability to the next state \(s_{t+1}\), given particular values of the preceding state \(s_{t}\) and action \(a_{t}\), is defined by  
\[p(s_{t+1},r_{t+1}|s_{t},a_{t}) \tag{5}\]  
for all \(s_{t},s_{t+1}\in S\), \(r_{t}\in R\) and \(a_{t}\in A(s)\).  
In the dialogue task, given input \(s_{t}\), we need to find a response \(s_{t+1}\) that optimizes a measure of semantic relevance to \(s_{t}\). RL can be used to learn when a part of the conversation was generated. We can define a conversation as an MDP \((S,A,P,R)\) and solve it using RL. The corresponding MDP \((S,A,P,R)\) is defined as follows [28]: the set of states \(S\) is defined as the conversation history, \(A\) is a set of generated responses for the system to reply to the user, and \(P\) is the transition probability function trained using the BERT-based encoding \(p_{RL}(s_{t+1}|s_{t},c_{L})\), where \(c_{L}\) is the left-context at time \(t\). The reward function, denoted as \(R\), represents the forward-looking reward received for each chosen action and plays a crucial role in achieving a successful conversation. Operationalizing \(R\) can improve the models to achieve the desired goal.