# TedBR: Target-Oriented Dialog with Bidirectional Reasoning on Knowledge Graph
## 4 Experiment
### Ablation Studies  
We conducted ablation studies to analyze the relative contributions of the main components in TodBR. These studies involved removing the bidirectional reasoning module and investigating the optimal number of prompt words and graph nodes. Our findings indicate that the model's performance exhibited varying degrees of decrease when these components were ablated. Specifically, the absence of bidirectional reasoning, lack of prompt words, and unsuitable dialog subgraph size resulted in observable declines in the model's performance.  
**Does the bidirectional reasoning work?** The importance and indispensability of both directions, namely the source direction and target direction reasoning, are exemplified in the first two rows of Table 7, where \"w/o source\" and \"w/o target\" denote the removal of source direction and target direction reasoning, respectively, such that only reasoning from one direction is considered. The results from these experiments demonstrate that, regardless of the direction, incorporating reasoning in the model contributes positively to its overall performance. Additionally, the performance degradation resulting from removing reasoning from the source and target is relatively similar, further highlighting the equal significance of both directions in improving the model's final performance.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{2}{c}{**In-Domain**} & \multicolumn{2}{c}{**Out-of-Domain**} \\ \cline{2-5}  & **Coherence** & **Logic** & **Coherence** & **Logic** \\ \hline DialoGPT & 1.34 & 1.55 & 1.34 & 1.85 \\ Multi-Gen & 1.88 & 1.98 & 1.53 & 1.99 \\ DKRN & 2.03 & 2.22 & 1.58 & 2.21 \\ CODA & 2.01 & 2.45 & 1.61 & 2.34 \\
**TodBR** & **2.13** & **2.47** & **1.74** & **2.38** \\ _kappa_ & _0.48_ & _0.50_ & _0.45_ & _0.51_ \\ \hline \hline \end{tabular}
\end{table}
Table 6: Human evaluation.  
**How many prompt words are suitable?** The terms \"w. fewer prompt words\" and \"w. more prompt words\" refer to the maximum number of prompt word sets generated during the bidirectional reasoning and decoding processes. The analysis of the experimental results reveals that the performance is significantly compromised regardless of whether there are fewer or more prompt words. This can be attributed to the fact that fewer words decrease the likelihood of obtaining the correct answer, while more words introduce noise during the decoding process. However, the impact on performance is more pronounced when fewer cue words are utilized. Notably, the degradation in performance is more evident when employing more prompt word sets due to the response generator's filtering capabilities during decoding. Based on these findings, an average of three entity keywords per sentence is a more suitable size for achieving optimal performance.  
**How much graph information do we need?** While constructing a dialog subgraph, our approach involved the utilization of pertinent keywords extracted from the dialog utterance to search within the commonsense knowledge graph. The resulting neighboring nodes were subsequently added to our evolving dialog subgraph. Notably, the precise number of neighboring nodes incorporated was not predetermined, prompting an investigation into the optimal quantity. To explore this matter, we conducted a series of systematic experiments. As illustrated in Figure 4, augmenting the quantity of neighbor node information only sometimes yields improved performance. Excess information can introduce noise and adversely impact the model's performance. Nonetheless, it is noteworthy that the decline in performance as the number of nodes increases is not profoundly significant. Following thorough experimental validation, we determined that the most favorable trade-off between performance and efficiency is achieved by considering up to 100 neighbor nodes during graph reasoning.