## 4 Evaluation
### Experimental Results  
In our first experiment, we evaluated the effectiveness of the knowledge encoder at the architecture level. The results are summarized in Table 2.  
In Table 2, CE is a model that only uses QK-Attention vector, and CE + KE uses QC-QK-Attention. As shown in Table 2, CE+KE exhibit better performances for all measures except PPL. PPL is the average number of options when a language model is generating words. With a lower PPL, the model has fewer options for generating words, such as general words (\"I know\"). Hence, PPL is not important for chatbots that need to generate various responses. This means that QC-QK-Attention improves the quality of responses.  
In the second experiment, we compared the performance of the proposed model with those of state-of-the-art models. For a fair comparison, we downloaded the code provided by the authors of each model from GitHub ([[https://github.com/lizekang/ITDD](https://github.com/lizekang/ITDD)]([https://github.com/lizekang/ITDD](https://github.com/lizekang/ITDD))) [31]. Then, we trained and tested the models by using the refined Wizard-of-Wikipedia dataset. Table 3 compares the performance of these models.  
In Table 3, Dinan-End2 End and Dinan-TwoStage are the most well-known knowledge-grounded chatbot models proposed by [PERSON] et al. who deployed the Wizard-of-Wikipedia dataset [1]. Dinan-End2 End performs the end-to-end task, and Dinan-TwoStage separately performs knowledge prediction and response generation. ITDD is a state-of-the-art model based on Transformer, and it generates a response using a two-pass decoder. We replaced the document with the context. As shown in Table 3, the proposed model exhibits better performances for all measures except PPL. The compared models show lower PPL because they generate more general words than the proposed model does, as shown in Table 4. Table 4 shows an example of the response of each model. Dinan-End2 End, Dinan-TwoStage, and ITDD models generated answers such as \"I know\" or \"I'm not sure.\" These general responses help lower PPL. However, the proposed model generated a response that mentioned the names of the band members, reflecting the acquired knowledge.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**F1-Score**} & \multirow{2}{*}{**PPL**} & \multicolumn{4}{c}{**BOW Embedding**} \\ \cline{4-6}  & & & **A** & **G** & **E** \\ \hline CE & 0.210 & **75.71** & 0.259 & 0.356 & 0.162 \\ CE+KE & **0.267** & 81.40 & **0.320** & **0.411** & **0.210** \\ \hline CE: Context encoder; KE: Knowledge encoder; PPL: Perplexity; A: Average; E: Extrema; G: Greedy; BOW: Bag-of-words. & & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance comparison depending on changes in the architecture (The bolts indicate the highest scores in each evaluation measure).  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**F1-Score**} & \multirow{2}{*}{**PPL**} & \multicolumn{4}{c}{**BOW Embedding**} \\ \cline{4-6}  & & & **A** & **G** & **E** \\ \hline Dinan-End2 End [1] & 0.172 & 70.03 & 0.127 & 0.248 & 0.118 \\ Dinan-TwoStage [1] & 0.150 & 50.55 & 0.127 & 0.235 & 0.104 \\ ITDD [31] & 0.193 & **11.16** & 0.250 & 0.332 & 0.136 \\ Proposed Model & **0.267** & 81.40 & **0.320** & **0.411** & **0.210** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance comparison between the proposed and previous models. (The bolts indicate the highest scores in each evaluation measure).