# Generative Model Using Knowledge Graph for Document-Grounded Conversations
## 4 Evaluation  
### Datasets and Experimental Settings  
We evaluated the proposed model using the Wizard-of-Wikipedia (WoW) dataset [5] and the Doc2 dial dataset [14]. The WoW dataset is an open-domain dialogue dataset. It contains multi-turn conversations between two speakers, an apprentice, and a wizard. The relevant documents are not accessible to the apprentice, whereas the wizard can access knowledge from relevant documents during chatting. The apprentice plays the role of a curious learner about a chosen document (topic). The wizard plays the role of a knowledgeable expert, keeping the conversation engaging. The Doc2 dial dataset is a goal-oriented dialogue dataset. It contains multi-turn conversations grounded in relevant documents from four domains for social welfare: SSA, VA, DMV, and student aid. The conversations of Doc2 dial have two speakers; a user and an agent. The two speakers aim to generate responses and inquiries with document-based or dialogue-based contexts, maintaining a free-form conversation. As shown in Table 2, in the WoW dataset, we removed outliers with more than 10,000 tokens. The average length of the documents is 2644.5 tokens, and the maximum length is 9935 tokens. The documents of the WoW dataset refer to the top passages retrieved for the last two turns of dialogue (by the apprentice and wizard) and the present turn, containing the gold knowledge. In the Doc2 dial dataset, the average length of the documents is 964.7 tokens, and the maximum length is 8891 tokens. The documents of the Doc2 dial dataset represent the given documents for the conversation. These datasets are suitable for our experiments because most documents are longer than the maximum input length.  
We evaluated our model using the following evaluation measures: F1-score [5; 9; 15], BLEU [20; 21], and SacreBLEU [22]. We used the F1-score and BLEU for the WoW dataset. BLEU measures the number of generated responses that contain word n-gram overlaps with the gold responses, as shown in the following equation:  
\[BLEU=min\left(1,\ \frac{Length\ of\ a\ generated\ sentence}{Length\ of\ a\ gold\ sentence}\right)\left(\prod_{i=1}^{n}precision_{i}\right)^{\frac{1}{n}} \tag{5}\]  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & **Minimum** & **Maximum** & **Average** \\ \hline WoW & 592 & 9935 & 2644.5 \\ Doc2 Dial & 174 & 8891 & 964.7 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics of the document length for both the Wizard-of-Wikipedia dataset and the Doc2 dial dataset.  
where \(n\) denotes the maximum length of \(n\)-grams, which is commonly set to 4, and \(precision_{i}\) is the word \(i\)-gram precision (i.e., the number of correct word \(i\)-grams divided by the number of word \(i\)-grams in a generated sentence). F1-score is calculated from precision and recall, as shown in the following equation:  
\[\begin{split} precision&=\frac{W_{gold}\cap W_{ generated}}{|W_{generated}|}\\ recall&=\frac{W_{gold}\cap W_{generated}}{|W_{ gold}|}\\ \text{F1-score}&=\frac{2\times precision\times recall }{precision+recall}\end{split} \tag{6}\]  
where \(W_{gold}\) denotes a unigram word list of the gold response, \(W_{generated}\) denotes a unigram word list of the generated response, and \(|W|\) indicates the length of the word list. We used SacreBLEU for the Doc2 dial dataset because it was used to evaluate DialDoc Shared Task Subtask2.