# Multi-Hop Question Generation with Knowledge Graph-Enhanced Language Model
## 4 Results
### Case Study  
In order to better understand the generation of GPT-2, KGEL, and human annotations, a case study is conducted. The study presents four examples, two of which are selected from the positive generation and two from the negative generation. The negative generation produced by KGEL is demonstrated in Tables 4 and 5, whereas the positive generation is depicted in Tables 6 and 7. The case study provides insight into the strengths and weaknesses of the models and helps to evaluate the quality of the generated questions. We further evaluate the performance of KGEL through a human evaluation of 150 randomly selected test samples. The evaluation scores are given as integers on a scale of 1 to 5, with a higher score indicating better performance. Three raters were asked to evaluate the generated questions from GPT-2, KGEL, and ground truth based on three metrics: fluency, answerability, and completeness. Fluency assesses the logical and grammatical correctness of the generated questions. Answerability assesses the relevancy and answerability of the generated questions in relation to the context. Completeness assesses the ability of the generated questions to involve multi-hop reasoning across scattered evidence in the context. The results of the human evaluation are shown in Table 8. Our results show a significant improvement over the baseline model, as demonstrated by the following three observations. * The high fluency scores obtained by both KGEL and GPT-2 with a pre-trained language model highlight the ability of these models to generate grammatically correct and logical questions. However, the lower answerability and completeness scores of GPT-2 compared to human performance indicate its limitations in identifying key information from the context and determining \"what to ask\". * The improved completeness score of KGEL compared to GPT-2 highlights the effectiveness of our proposed reasoning module in recognizing multi-hop relationships and identifying relevant entities. This demonstrates the better capability of our model in generating questions with multi-hop relations. * The better performance of KGEL in terms of answerability, compared to GPT-2, is a result of the incorporation of entity graph in our reasoning module. However, further improvement is required in the ability of the model to identify the correct target aspect to ask, in order to enhance answerability. As shown in Table 4, it is shown that both GPT-2 and KGEL get the wrong attention for the entity to ask. One challenge is that the support facts do not have any clue about \"[PERSON]\". In Table 5, both GPT-2 and KGEL ask a wrong question, unable to decide what to ask. The generation of KGEL is slightly better because the generation is more readable. \begin{table}
\begin{tabular}{|p{142.3 pt} p{142.3 pt}|} \hline
**Support Fact A:** & Ulysses is a decommissioned robotic space probe whose primary mission was to orbit the Sun and study it at all latitudes. \\
**Support Fact B:** & His professional career has included work on the Voyager, Galileo, and Ulysses space missions, and is currently working as a consultant in aerospace power and propulsion systems. \\ \hline
**Answer:** & Sun \\ \hline
**GPT-2:** & What is the name of the support space of the space named after the physicist who has work on the voyager? \\
**KGEL:** & [PERSON] worked on the casting of the decommissioned spacecraft that are setting what landform? \\
**Ground Truth:** & [PERSON] was a part of the space missions that have a primary destination of what celestial body? \\ \hline \end{tabular}
\end{table} 
Table 4: Negative example 1. In Table 6, KGEL generates a variant of a golden question with a similar meaning as the human annotation, whereas the GPT-2 generates inexplicable words. The presented example serves as an exemplary solution to the primary challenge of accurately identifying critical evidence.