## 3 The Proposed Model
### Left-Context with BERT-Based Model
#### 3.1.2. _Integrating Left-Context with BERT2 BERT_  
A human can control the informational flow in their conversation during a long-term conversation. However, most current models are incapable of handling the history of a conversation. That is the reason those models usually generate repetitive and generic responses. Our model solves this problem by using a left-context, denoted as \(c_{L}\), which serves as contextual information from the conversation history. This context is defined as a \(k\) consecutive sequence of previous utterances \(s_{t-k},\ldots,s_{t-2},s_{t-1}\) at the turn \(t\). Based  
Figure 1: The architecture of pre-trained BERT model. on the principle that more information is better, this left-context can provide additional information during the training process. Our proposed model for generating historical contextual responses is a transformer-based encoder-decoder model. It has been proven to improve performance on many tasks using the encoder-decoder model [41; 42]. However, such models require a massive dataset for pre-training before fine-tuning for a desired task. Recently, a study showed that skipping the cost of the pre-training process and using a pre-trained encoder allows the transformer-based encoder-decoder model to achieve competitive results in text generation tasks [43; 44; 45]. Inspired by their study, we use the BERT2 BERT architecture [43] and warm-start the encoder and decoder with the BERT-based checkpoint [46]. The proposed model takes an input message \(s_{t}=\{w_{1}^{t},w_{2}^{t},\dots,w_{|s_{t}|}^{t}\}\) and its left-context \(c_{L}\) as input. This sequence is then fed into an encoder, which is a stack of BERT-based encoder blocks. Each block consists of a self-attention layer and two feed-forward layers, as shown in Figure 2.  
The encoder maps the input sequence \([c_{L},s_{t}]\) to a contextualized encoded vector \(\bar{X}^{BERT}\), as follows:  
\[\bar{X}^{BERT}=f_{\theta_{enc}}([c_{L},s_{t}]) \tag{1}\]  
The architecture of the decoder block is similar to the encoder block (as shown in Figure 3). However, the decoder block is conditioned on the contextualized encoded vector \(\bar{X}^{BERT}\), and the model also includes a cross-attention layer in addition to the decoder. Each decoder block is more extensive than the encoder, consisting of a self-attention layer, two feed-forward layers, and a cross-attention layer that obtains contextual information from the vector \(\bar{X}^{BERT}\). In addition, a linear layer called the LM Head is included on top of the last decoder block, which maps the output vectors to the logit vectors \(L\). During the training process, the decoder maps the contextualized encoded \(\bar{X}^{BERT}\) and a target sequence \(s_{t+1}\) to logit vectors \(L\). The probability distribution of the target sequence \(s_{t+1}\) is factorized from conditional distributions of the next word using [PERSON]'s rule:  
\[p_{Bert2 Bert}(s_{t+1}|[c_{L},s_{t}])=p_{\theta_{dec}}(s_{t+1}|\bar{X}^{BERT})= \prod_{j=1}^{|s_{t+1}|}p_{\theta_{dec}}(w_{i}^{t+1}|w_{0i-1}^{t+1},\bar{X}^{ BERT}) \tag{2}\]  
Figure 2: The architecture of encoder. The encoder is initialized based on multiple BERT blocks, each of which is composed of a multi-head attention and feed-forward network. The logits define the distribution of the target sequence \(s_{t+1}\) conditioned on the input sequence \(s_{t}\) through a softmax operation. As a result, each generated token is defined by the softmax of the logit vector as follows:  
\[p_{\theta_{dec}}(w_{i}^{t+1}|w_{0;i-1}^{t+1},\hat{X}^{BERT})=Softmax(l_{i}) \tag{3}\]  
where \(l_{i}\) is the \(i^{th}\) token in logit vector \(L\). We define \(s_{t+1}=\{w_{1}^{t+1},w_{2}^{t+1},\ldots,w_{|s_{t+1}|}^{t+1}\}\) as the ground-truth output for a given input sequence \(s_{t}\).