## 5 Experiments
### Response Weight  
Since the response weight aims at finding the importance of each query token, its performance is verified with a _keyword detection task_. The proposed response weight is compared with five variations of MAUI [31; 33], a strong keyword detection system trained with various features. MAUI-df is a default MAUI model trained with a decision tree and tf-idf, while MAUI-wv and MAUI-br use the structured skip-n-gram [34] and the Brown cluster feature [35] as well as the features of MAUI-df. MAUI-brwv uses all features stated above, and MAUI-out has the same structure as MAUI-brwv but is trained with the news articles used in the work of [PERSON] et al. [36]. Note that MAUI-out is the only variant that is not trained with the MAUI Twitter data.  
\begin{table}
\begin{tabular}{l c c c} \hline \hline  & **\# Train** & **\# Valid** & **\# Test** \\ \hline Dailydialog & 45,337 & 3851 & - \\ MAUI Twitter & - & - & 500 \\ Reddit & 1,352,961 & 40,000 & 40,000 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Simple statistics on the data sets for response weight and response generation.  
Table 3 summarizes the evaluation results. RW in this table is the proposed response weight. Since the proposed response weight is not a direct keyword detector, the words in a query are chosen as keywords when their \(a_{i}\) in Equation (4) is greater than 0.3. All performances are measured for four extracted keywords. While MAUI-brwv shows the highest performance on F1-score, the proposed model ranks third. Note that RW is trained with Dailydialog, not with MAUI twitter data. Thus, it is difficult to compare the performance of RW directly with those of MAUI variants. Nevertheless, it outperforms MAUI-df, MAUI-wv, and MAUI-out. In particular, the fact that RW achieves higher performance than MAUI-out, another model trained with non-MAUI twitter data, proves the effectiveness of RW to identify keywords from a query.  
Figure 3 depicts the change of the response weight **a** during its training when the query is \"_Orange or gold, pick one_\" and the response is \"_Orange. Easy choice._\" RW-1 and RW-10 in this figure are the response weights after one epoch and ten epochs, respectively, and Enc-Dec is the average attention of all heads in the encoder-decoder attention layer of a standard transformer. When the training of the proposed response weight is at one epoch, the meaningless or general tokens in the query such as '_or_' and ',' have high weight. This is because neither the query nor the response is reflected enough to the weight. On the other hand, '_orange_' and '_pick_' have high attention after 10 epochs, since they are deeply related to '_orange_' and '_choice_' in the response.  
The most important finding in this figure is that RW-10 shows a different weight distribution from Enc-Dec. Enc-Dec pays high attention to '_gold_', '_pick_', and '_one_'. Though the transformer reflects responses during its training, Enc-Dec is affected much more by the self-attention of the query. Thus, the word like '_one_' that is not strongly related with the response is also spotlighted. In addition, the attention difference in Enc-Dec is insignificant. To sum up all these results, better responses can be generated by enhancing the influence of the response through the response weight.  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model** & **P** & **R** & **F1** \\ \hline MAUI-df & 53.97 & 53.15 & 53.56 \\ MAUI-wv & 55.80 & 54.45 & 55.12 \\ MAUI-br & 71.95 & 75.01 & 73.45 \\ MAUI-brwv & **72.05** & **75.16** & **73.57** \\ MAUI-out & 55.54 & 48.74 & 51.92 \\ RW & 55.75 & 55.57 & 55.66 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Precision, recall and F1-score on MAUI Twitter data set. Scores in bold stand for the leadership among the models  
Figure 3: The distributions of the attentions for an example query and response. The distributions are obtained from the proposed response weight and the standard transformer.