# TedBR: Target-Oriented Dialog with Bidirectional Reasoning on Knowledge Graph
## 4 Experiment
### Metrics  
#### 4.5.1 Automatic Evaluation  
To assess the models' proficiency in generating effective bridging responses, we commence by conducting an automatic evaluation employing widely used text generation metrics, as follows. (1) BLEU [49], commonly known as bilingual evaluation understudy, has emerged as a widely employed metric to assess the quality of machine-translated text. It proves \begin{table}
\begin{tabular}{l l l} \hline \hline
**Topics** & **Subtopics (The Most Common Are Listed)** & **Quantity** \\ \hline entertainment & cinema, museum, painting, concert & 631 \\ education & history, math, school, university & 589 \\ family & parent, friend, marriage, children & 1143 \\ workplace & job, stress, interview, employee & 1053 \\ health & sport, drinks, meal, vegetables & 893 \\ daily life & hobby, money, relax, food & 1454 \\ travel & transport, weather, outside, car & 692 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Number of topic statistics. applicable in evaluating translations in any language, given the presence of discernible word boundaries within the text. Typically yielding a score ranging from 0 to 100, BLEU quantifies the degree of similarity between a reference text and a hypothesis text. A higher BLEU score indicates superior-quality translations. \[ \begin{split} BP=\left\{\begin{array}{cc}1&\text{if }c>r\\ e^{1-\frac{r}{c}}&\text{if }c\leq r\end{array}\right.\\ \text{BLEU}=BP\cdot\exp\left(\sum_{n=1}^{N}w_{n}\log P_{n}\right)\end{split}  \tag{13}\]  
As shown in the above formula, \(BP\) is a factor that penalizes excessive abbreviation in translations. It ensures that concise translations are appropriately penalized. \(c\) represents the length of the candidate translation (the machine-generated translation). \(r\) is the effective reference length, which is the length of the reference translation closest to the candidate translation in terms of the number of words. \(\exp\) is the exponential function, which is used to weigh the precision of different N-grams. \(\sum_{n=1}^{N}\) signifies that the BLEU score considers the precision of 1-g, 2-g, up to N-gram. \(w_{n}\) corresponds to the weight coefficients assigned to different N-grams. \(P_{n}\) is the precision of the N-gram. \(N\) denotes the characteristic length of the N-gram under examination. The BLEU score considers precision up to N-gram. Furthermore, we use SacreBLEU [50], which provides the hasske-free computation of shareable, comparable, and reproducible BLEU scores. It reports the total BLEU score that accounts for the overlap across 1-4 N-grams instead of only 4-g.  
(2) ROUGE-L [51] relies on the longest common subsequence (LCS) approach, wherein a comparison is made between the model output and the reference. Specifically, it identifies the most extended sequence of words, which may not be consecutive but maintains the same order standard to both the model output and reference. A lengthier shared sequence signifies higher similarity between the two sequences. \[ \begin{split} R_{lcs}=\frac{LCS(X,Y)}{m}\\ P_{lcs}=\frac{LCS(X,Y)}{n}\\ F_{lcs}=\frac{\big{(}1+\beta^{2}\big{)}R_{lcs}P_{lcs}}{R_{lcs}+ \beta^{2}P_{lcs}}\end{split}  \tag{14}\]  
\(LCS(X,Y)\) is the length of the longest common subsequence of \(X\) and \(Y\); \(m\) and \(n\) represent the length of the reference text and generated text (usually the number of words contained); \(R_{lcs}\) and \(P_{lcs}\) represent recall and accuracy, respectively. \(F_{lcs}\) is the final calculated ROUGE-L.  
(3) The evaluation of textual diversity incorporates Distinct-1 and Distinct-2 [52]. Distinct-1 measures the occurrence of distinct words or phrases within the generated text, while Distinct-2 focuses on the frequency of unique adjacent word or phrase pairs, commonly known as bigrams. Distinctness is quantified on a numerical scale from 0 to 1. A higher value approaching 1 signifies an augmented abundance of varied and unique words or bigrams in the generated text.