## 4 Experiments
### Experimental Results  
Experiments are conducted on Datasets I-IV and the experimental results for the accuracies of subject recognition are shown in Table 2. For single-relation questions, BERT-CRF outperforms GP and EGP, and INNM further outperforms BERT-CRF. Among all of the INNM methods, INNM-II achieves the highest accuracy of 98.7% on Dataset I and 94.1% on the whole SQ (I + II), which outperforms BERT-CRF by a margin of 3.3%, while INNM-I achieves the highest accuracy of 27.4% on Dataset II. For complex questions, however, EGP outperforms BERT-CRF and GP and INNM-III achieves the highest accuracy of 92.2% on Dataset III, 6.4% on Dataset IV, and 71.0% on the whole WQSP (III + IV), which outperforms BERT-CRF by a margin of 11.0%. In addition, to abnormal questions in Dataset II, general flat NER models fail to recognize golden subjects because of the difference between the golden entity and subject, while our proposed models could recognize some of them correctly. To the normal questions in Dataset I, INNM-I fails to outperform BERT-CRF because EGP is not so effective in this case. As a result, it is a better choice to employ INNM-II where BERT-CRF (the more effective model) is employed first. For complex questions, even normal questions in Dataset III could contain several interference words or entities, so it is much more difficult to recognize subjects correctly. In this case, EGP outperforms BERT-CRF and our INNM-I further outperforms EGP. Further outperformance would be achieved by employing INNM-III, where EGP (the more effective model) is employed first. Unfortunately, all methods fail to achieve satisfactory performance in Dataset IV. In fact, for many of the questions in Dataset IV, the golden entity is not similar to the golden subject. For example, the golden entity \"jk\" (cannot even be encoded in BERT) in \"who was vice president when [PERSON] was president\" is not similar to the golden subject \"[PERSON] f.kennedy\". In this case, it is much easier to add such aliases to the KB than recognize them correctly by a model. After the subject is recognized, to single-relation questions, we can simply employ a BERT-based RE model (also achieved by bert4 keras) to predict the relation and retrieve the answer. To complex questions, relation predicting and subject recognition are usually considered as two individual tasks, so we only show experimental results for overall accuracies in SQ, which are shown in Table 3. BERT-CRF shows the best performance among traditional methods. Our models (INNM-I,II,III) further outperform it by a margin of 2.5%, 3.0%, and 2.7% respectively. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Method** & **Dataset I** & **Dataset II** & **SQ** & **Dataset III** & **Dataset IV** & **WQSP** \\ \hline BERT-CRF & 97.0 & 0 & 90.8 & 79.8 & 0 & 60.0 \\ GP & 95.5 & 0 & 89.4 & 80.3 & 0 & 60.4 \\ EGP & 95.6 & 0 & 89.5 & 81.8 & 0 & 61.5 \\ \hline BERT-CRF + AM & 98.2 & 24.8 & 93.5 & 89.7 & 6.4 & 69.1 \\ INNM-I & 98.0 & 27.4 & 93.5 & 91.2 & 6.4 & 70.2 \\ INNM-II & 98.7 & 26.1 & 94.1 & 89.8 & 6.2 & 69.1 \\ INNM-III & 98.6 & 22.2 & 93.8 & 92.2 & 6.4 & 71.0 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Experimental results for accuracies (%) of subject recognition. To further evaluate the robustness of our model, additional experiments are conducted on Datasets V and VI, and the experimental results for the accuracies of subject recognition are shown in Table 4. All INNM methods outperform BERT-CRF, and INNM-III shows the best performance among all methods listed, which outperforms BERT-CRF by an average margin of 10.4%. In addition, as EGP requires less training time than BERT-CRF in various NER tasks, our INNM-I also requires less training time (20% reduced) than BERT-CRF. To INNM-II and INNM-III, as both of BERT-CRF and EGP should be trained, more training time (80% increased) is required.