# Conversational AI over Military Scenarios Using Intent Detection and Response Generation
## 4 Experiments
### Datasets and Measures  
A total of 1307 human-labeled sentences are included in the experimental dataset used for intent classification (who, where, when, and other). Table 7 shows the four types of intent quantitative sentences. As shown in Table 8, the experimental datasets were used to train the five NER models, including people, weapons, places, events, and time. The intent detection and slot-filling tasks in the NLU stage are evaluated using F1-score (Equation (9)) and accuracy (Equation (10)). In the following equations, true positive (TP) and true negative (TN) are the numbers of accurately predicted positives and negatives, respectively. Conversely, false positive (FP) and false negative (FN) are the numbers of wrongly predicted positives and negatives, respectively. Thus, Precision = \(\mid\)TP \(\mid\) / \(\mid\)TP \(\mid\) and Recall = \(\mid\)TP \(\mid\) / \(\mid\)TP \(\mid\) + \(\mid\)FN \(\mid\).  
\[\text{F1-score}=\frac{2\times\text{Precision}\times\text{Recall}}{\text{ Precision}+\text{Recall}} \tag{9}\]  
\[\text{Accuracy}=\frac{\mid\text{TP}+\text{TN}\mid}{\mid\text{TP}+\text{FP}+ \text{TN}+\text{FN}\mid} \tag{10}\]  
The LTR model was evaluated using the normalized discounted cumulative gain (NDCG), which gives the normalized relevance score of the files retrieved by the search  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**No.** & **Intent** & **Entity** & **Templates** \\ \hline \multirow{3}{*}{7} & Where & Unit & Where is the + [Location] + located? \\ \cline{2-4}  & Where & Unit & Where is the command of army located? \\ \hline \multirow{3}{*}{8} & Where & Weapon & Where did the + [Location] + discover the + [Weapon]? \\ \cline{2-4}  & Where & & Where did the brigade discover two enemy tanks? \\ \hline \multirow{3}{*}{9} & What & Event & What + is the basis for performing this + [Event]? \\ \cline{2-4}  & What & & What is the criteria for performing this ground mission? \\ \hline \multirow{3}{*}{10} & What & Weapon & What + is the range of this + [Weapon]? \\ \cline{2-4}  & What & & What is the range of this tank’s main gun? \\ \hline \hline \end{tabular}
\end{table}
Table 6: _Cont._  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Who** & **Where** & **When** & **What** \\ \hline \# of sentence & 335 & 320 & 329 & 323 \\ \hline \hline \end{tabular}
\end{table}
Table 7: The number of datasets used for intent classification.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Person** & **Weapon** & **Location** & **Event** & **Time** \\ \hline \# of sentence & 1005 & 1007 & 1006 & 1005 & 1000 \\ \hline \# of entity & 3693 & 2465 & 1843 & 2399 & 1820 \\ \hline \hline \end{tabular}
\end{table}
Table 8: The number of datasets used for NER training.  
engine at each rank position. Files closer to the top are given a higher weight (and therefore have a greater degree of influence on NDCG), as shown in Equation (11).  
\[NDCG_{p}=\frac{DCG_{p}}{IDCG_{p}} \tag{11}\]  
where _IDCG_ is ideal discounted cumulative gain, and _rel\({}_{p}\)_ represents the list of relevant documents in the corpus up to position \(p\).  
\[DCG_{p}=\sum_{i=1}^{p}\frac{rel_{i}}{log_{2}(i+1)} \tag{12}\]  
_DCG_ is based on the principle that highly relevant documents appearing lower in a search result list will be penalized by having their relevance grade reduced logarithmically proportional to their position in a search result list. Finally, the query generation module is evaluated by quantifying user satisfaction.