# Chinese Neural Question Generation: Augmenting Knowledge into Multiple Neural Encoders
## 4 Experiments
### Human Evaluation and Results  
Compared to machine evaluation of question generation, human evaluation provides more reliable assessment of the quality of questions but is more labor intensive. In this study, we evaluated s2s+Att, NQG++, NQG++MP, BERT-SQG, Multi-encQG and Multi-encQG++ models with human judges. Three Chinese linguistic majors randomly selected 800 passages from the test set in the dataset and labelled 800 questions generated by each model from the dataset. Each question was judged as acceptable or unacceptable following a typical judgment protocol used in Chinese question generation [7], which is based on the following major criteria: (1) grammaticality correctness, (2) semantic correctness and (3) correct match of the reference answer and question type. The inter-rater reliability of the question acceptance was measured by using Fleiss [47], which indicated moderate or higher agreements (scores ranged from 0.42 to 0.67) between human annotators for the set of questions generated by each model.  
Figure 3: The impact of the number of ranked triples encoded on the Multi-encQG++ model performance in WebQA dataset.  
Figure 4: Attention distribution when Multi-encQG generated Q6.  
Figure 5: Global attention distribution when NQGKG++ generated question Q6.  
In addition, if the question was labelled as unacceptable, the question type was given by the human annotators based on the question word used. Chinese question types can be difficult to determine by defining regular expression matching rules. Compared to the English question type, which has a unique interrogative pronoun (e.g., where, which, what) that appears at the beginning of the question, the same Chinese question type can be indicated by different interrogative pronouns possibly appearing in any place of a question [48]. For example, to express a where question in Chinese, a variety of interrogative pronouns can be used, including (which area), (where) and (which place), which can appear in the middle or at the end of a question. Thus, the human annotation of the question type provides a more reliable assessment than the system evaluation. In this study, we introduce another evaluation metric for Chinese question generation called the same question type ratio (SQTR), which measures the ratio of the generated questions that have the same question types as the reference question even if some question words may be different.  
Table 6 shows that the Multi-encQG++ model reaches the best acceptance among six question generation models and slightly outperformed mTS by 3% in the dataset mainly because the complexity of the QG process is reduced using the passage compression step and the answer-focused multi-encoder sequence-to-sequence neural model with double attention mechanism. Compared to the Single-encQG, the Multi-encQG++ with the double attention mechanism increased the acceptance score by 8% because the Multi-encQG++ model is more answer focused and more likely to use the right question words. In fact, Table 7 shows that the Multi-encQG++ model reached the best SQTR across four main question types, indicating this model can pay better attention to the answer and decode more accurately the question words. The baseline s2s+Att model reached the lowest acceptance since it lacks focus on the answer. In addition, it has been found that these models reached better recalls for what and who questions than where and which questions. This could be explained by the larger number of what and who questions being present in the training set.  
Figure 6: Final double attention distribution when NQGKG++ generated Q6.