# Conversational AI over Military Scenarios Using Intent Detection and Response Generation
## 4 Experiments
### Discussion  
Here, we conduct an error analysis of the module performance and discuss the challenges of research and limitation. For the intent classification module, we found that the \"what\" category has the highest error rate (12%) because there are some queries with different interpretations, such as \"why\", \"how much\", \"how to do\", etc. For example, \"how many days can each cavalry company unit fight independently?\" \"How many exchange centers and medium-sized communication centers can a communication force establish?\" These false-negative examples of the \"what\" leads to lower recognition performance than other classes. It will be possible to further segment the user's intent in the future to guarantee that every feature of that intent is clearly defined. For the slot-filling module, the accuracy for five types of entities is identified by the military personal name as the highest (0.917), followed by time (0.901), location (0.823), weapon (0.788), and military event (0.776) as the lowest. There are two main reasons for the poor attribution of military events: (1) Event names are longer than other entities, making it more challenging to identify the limits of the entity. Still, the system identifies part of the names of military activities. (2) The event name contains time or location, which is misjudged as another entity. For the retrieval-based answering module, we use the learning-to-rank method to achieve results. From the learning curve point of view, with the increase in data, the efficiency of the system response improves (NDCG = 0.678). Candidate sentences are  
Figure 8: Users’ satisfaction for system response. Figure 7: Performance of the learning-to-rank model. added only when the intent of the sentence is the same as the intent of the user's query. Then, we adopt these entities in the sentence as sorting features, and finally, sort them based on the LambdaMART algorithm. As we analyze why the correct sentence is not ranked first, we discover that pronouns may represent entities in sentences or omit them; thus, some candidate sentences do not identify entities related to the query, resulting in a low ranking score. Based on the methods comparison in related literature, [PERSON] [46] compared CNN and SVM, two ML algorithms with good performance records in the current NLP literature. However, the CNN model is not necessarily better than the SVM model based on a detailed statistical analysis of the experimental results. Under these experimental conditions, the SVM model using the radial basis function kernel produced statistically better results. However, SVM has its limitations. SVM is not suitable for large datasets because the complexity of algorithm training depends on the size of the dataset [69]; SVM is not ideal for training imbalanced datasets, which causes the hyperplane to be biased towards the minority class [70]. In terms of performance, choosing an \"appropriate\" kernel function is crucial[PERSON] For example, using a linear kernel when the data are not linearly separable can lead to poor algorithm performance[PERSON]  
Two factors for the superior performance of the state-of-the-art are rich training datasets and high-speed hardware such as GPUs. We choose the CRF-based method, mainly considering the amount of data and training cost. Since obtaining a large amount of military training data in Chinese is a challenge, this study implements a dialogue AI system applied to military training scenarios using a limited military dialogue dataset. Using a CRF-based model is indeed a baseline approach[PERSON] Nonetheless, this is an initial and fruitful result for the agency. For future work, we consider applying transfer learning (meta-learning) to extend multiple military domains with small datasets to improve the scalability of dialogue systems.