## 3 Learning Response Weight  
The response weight aims at providing a decoder with the relatedness between a response and each token in a query. Table 1 shows that it enhances the quality of a response to identify key tokens in a query under the response context and reflect them into response generation. In this table, \(\mathbf{Q}\) and \(\mathbf{R}\) indicate a query and its response, respectively. The bold words in the queries are the key tokens that are related highly with a response, and the underlined words are the tokens highlighted by the attention of a transformer encoder-decoder. In the first example, the encoder-decoder attention focuses only on _'orange or gold'_, but the response contains the word _'choice'_ due to the word _'pick'_ in the query. All other examples also show a similar phenomenon. The expressions of _'try'_ and _'still waiting'_ appear at the responses because _'shot'_ and _'haven't received'_ in the queries are response-related. Therefore, it helps to generate diverse responses to identify such key tokens in a query. Figure 1 shows the overall architecture for computing the response weight. The response weight is obtained while representing a query \(q\) into a query vector \(\mathbf{q}^{\mathbf{a}}\) with a transformer encoder so that \(\mathbf{q}^{\mathbf{a}}\) can reflect a potential response of \(q\). The transformer encoder is trained with a bitext classification task whose goal is to predict whether a query \(q\) entails a response \(r\). In Figure 1, the bitext classification is solved by the _bitext classifier_ which is implemented as a bilinear function, that is, the bitext classifier \(f_{bt}\) determines the entailment between \(q\) and \(r\) by the following:  
\[f_{bt}(\mathbf{q}^{\mathbf{a}},\mathbf{r})=\mathbf{q}^{\mathbf{a}}W_{bt}\mathbf{r}, \tag{1}\] \begin{table}
\begin{tabular}{l} \hline \hline \(\mathbf{Q}\)**: Orange** or gold, **pick** one \\ \(\mathbf{R}\): Orange. Easy _choice_ \\ \hline \(\mathbf{Q}\): I’d love if you **gave it a shot**! \\ \(\mathbf{R}\): All right. I’ll give it a _try_. Do you have it somewhere I can download it? \\ \hline \(\mathbf{Q}\): I **ordered** march 11 th and I **haven’t received** anything yet. \\ \(\mathbf{R}\): Ordered same day, _still waiting_ on mine as well. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The examples which show that the response context affects response generation in the Reddit data set. where \(\mathbf{q^{a}}\) and \(\mathbf{r}\) are vector representations of \(q\) and \(r\), respectively, and \(W_{bt}\) is a trainable weight matrix. This classifier is trained to maximize the log-likelihood by optimizing \(W_{bt}\) and the parameters of the transformer encoder with the Adam optimizer [26]. Note that \(\mathbf{q^{a}}\) is encoded by the transformer encoder. Thus, the transformer encoder is trained to express \(\mathbf{q^{a}}\), similar to \(\mathbf{r}\) when \(q\) and \(r\) are a conversation pair from a real corpus. On the contrary, it encodes \(\mathbf{q^{a}}\) differently from \(\mathbf{r}\), when \(q\) and \(r\) are from negatively-sampled data. The representation of \(q\) into \(\mathbf{q^{a}}\) is done as follows. Denote \(q=\langle q_{1},q_{2},\dots,q_{n}\rangle\) as a query composed of \(n\) tokens. A special token \(q_{0}\) is added at the beginning of \(q\), and it plays a similar role to the [cls] token of BERT. Thus, the output of the transformer encoder becomes \(\mathbf{q}=\langle\mathbf{q}_{0},\mathbf{q}_{1},\mathbf{q}_{2}\dots,\mathbf{ q}_{n}\rangle\), where \(\mathbf{q}_{0}\) aggregates the sequence representation of the query. To obtain a richer representation, \(\mathbf{q}_{0}\) is linearly transformed to \(\mathbf{s}\) as follows:  
\[\mathbf{s}=W_{s}\mathbf{q}_{0}+b_{s}, \tag{2}\]  
where \(W_{s}\) and \(b_{s}\) are trainable parameters. Since \(\mathbf{s}\) is a summary of the query \(q\), the importance of each query token \(q_{i}\) with respect to \(\mathbf{s}\) is computed by \(e_{i}=\mathbf{s}\cdot\mathbf{q}_{i}\).