# Knowledge Interpolated Conditional Variational Auto-Encoder for Knowledge Grounded Dialogues
## 6 Experiments and Analysis
### Experimental Settings
#### 6.1.3 Metrics  
We employ four types of automatic metrics to evaluate our proposed model and the baseline models: Per PerpLexity (PPL), three embedding metrics (_Embedding Average (AVE)_, _Embedding Extreme (EXT)_, and _Greedy Matching (GRY)_), _Distinct\({}_{1}\) (Dist\({}_{1}\))_ and _Distinct\({}_{2}\) (Dist\({}_{2}\))_, and our calculated Bilingual Evaluation Understudy metrics \(\Delta\)BLEU\({}_{k}\). **PPL:** It is the exponentiation of the word entropy, and describes how well the generative model predicts the expected responses. It does not directly capture coherence, as a low perplexity value indicates an accurate prediction, but can not guarantee a coherent text. Coherence depends on factors like transitions, readability, consistent topics, and logical structures. Diversity in text generation refers to variations and novelty, which perplexity does not capture. **AVE, EXT, and GRY:** Rather than using n-gram overlapping-based metrics like Bilingual Evaluation Understudy (BLEU) or Recall-Oriented Understudy for Gisting Evaluation(ROUGE), we report word embedding-based similarity metrics [52] to capture the semantic alignment between generated responses and ground truth. Specifically, we adopt three embedding metrics: _Embedding Average (AVE)_, _Embedding Extreme (EXT)_, and _Greedy Matching (GRY)_.  
AVE calculates the average similarity between consecutive sentence embeddings in generated text, indirectly enhancing coherence/diversity. It incorporates contextual information from previous words, aligning the generated text with the overall topic/theme and improving coherence/diversity. EXT involves selecting the most unique or extreme word embeddings to generate text that deviates from typical language patterns. It aims to enhance diversity but may result in inconsistencies and reduced coherence. A low embedding extreme score suggests repetition or redundant sentences, while a high score indicates incoherence. GRY is a method employed to identify the most pertinent sentences from a larger body of text. Its purpose is to ensure consistency by selecting text segments that are highly relevant to a given query. While emphasizing coherence, greedy matching tends to prioritize closely associated sentences, potentially leading to a dearth of diversity in the generated text. **(Dist\({}_{1}\)) and (Dist\({}_{2}\)) [53]:** We use (Dist\({}_{1}\)) and (Dist\({}_{2}\)) to evaluate the diversity of responses, which calculates the ratio of unique unigrams (or bigrams) to the total number of generated words. It determines the ratio of unique n-grams to the total number of n-grams produced. A higher distinct value suggests a greater range of content. However, in certain instances, a higher distinctness can result in reduced coherence as it may introduce unrelated ideas that disrupt the overall flow and cohesiveness of the text. \(\Delta\)**BLEU\({}_{k}\)**: We use \(\Delta\)BLEU\({}_{k}\) to measure the engagement of knowledge. To assess whether the knowledge is articulated diversely and engagingly, we define a unique automatic metric that measures the ratio of knowledge utilization in responses based on the word overlap between responses and provided knowledge. We initially compute BLEU\({}_{k}(g)\) as the average BLEU score, considering the generated response \(\hat{y}\) as the hypothesis and given knowledge as the reference. An exceptionally high BLEU\({}_{k}(\hat{y})\) signifies an excessive copy of external knowledge and makes the response less engaging. We further consider BLEU\({}_{k}(y)\) of the gold standard response created by human \(y\) as the ground truth and compute the average absolute difference between BLEU\({}_{k}(\hat{y})\) and BLEU\({}_{k}(y)\), namely \(\Delta\)BLEU\({}_{k}\), to measure the engagement of knowledge:  
\[\Delta\text{BLEU}_{k}=\frac{1}{N}\sum_{y,\hat{y}}|\text{BLEU}_{k}(y)-\text{ BLEU}_{k}(\hat{y})|, \tag{13}\]  
where \(N\) is the number of samples in the test set, \(y\) and \(\hat{y}\) are gold and generated responses from the same context and knowledge.