## 5 Experiments and Results  
To avoid subjectivity and the high human evaluation cost, the quality of the chatbot output is typically evaluated using machine translation evaluation metrics such as BLUE [36] or text summarization metrics such as ROUGE [37]. The BLEU (Bilingual Evaluation Understudy) score compares the chatbot-generated output text (so-called hypothesis) with a human-produced answer (so-called reference or gold-standard) text and indicates how many \(n\)-grams in the output text appear in the reference. The BLUE score can be any value within the interval \([0,1]\) and is mathematically defined in Equation (2). \[BLUE=BP\times\left(\prod_{n=1}^{N}\textit{precision}_{n}\right)^{1/N}, \tag{2}\]  
where \(N\) is the a maximum \(n\)-gram number \(n=[1,N]\) (\(N=4\) in our evaluations). \(BP\) (brevity penalty) and \(\textit{precision}_{n}\) are defined with Equations (3) and (4), respectively. \[BP=\min\left(1,\exp\left(1-\frac{\textit{ref}_{\textit{length}}}{\textit{ out}_{\textit{length}}}\right)\right), \tag{3}\]  
where \(\textit{ref}_{\textit{length}}\) is a reference length, and \(\textit{out}_{\textit{length}}\) is the chatbot output length. \[\textit{precision}_{n}=\frac{\sum_{n}\min\left(m_{\textit{out}}^{n},m_{ \textit{ref}}^{n}\right)}{\sum_{n^{\prime}}m_{\textit{out}}^{n^{\prime}}}, \tag{4}\]  
where \(m_{\textit{out}}^{n}\) is the number of \(n\)-grams in the chatbot output matching the reference, \(m_{\textit{ref}}^{n}\) is the number of \(n\)-grams in the reference, and \(\sum\limits_{n^{\prime}}m_{\textit{out}}^{n^{\prime}}\) is the total number of \(n\)-grams in the chatbot output. The BLUE scores were calculated using the python implemented _blue_score_ module from the _translate_ package in the _nltk_ platform [38]. The _blue_score_ parameters were initialized with the default values, except for the smoothing function, which was set to _method2_[39] (adding 1 to both numerator and denominator); \(n\)-gram weights were set to 0.25 and \(N=4\). Depending on the interval the BLUE score values are in, the chatbot result can be interpreted as useless (if BLUE \(<\) 10), hard to get the gist (10-19), the gist is clear, but has significant errors (20-29),average quality (30-40), high quality (40-50), very high quality (50-60), and better than human (>60). These values and interpretations were taken from the Google Cloud's AI and Machine Learning product description in Reference [40]. The ROUGE (Recall-Oriented Understudy for Gisting Evaluation) metric measures the similarity of the chatbot-generated output to the reference text. _ROUGE-N\({}_{precision}\)_, _ROUGE-N\({}_{recall}\)_, and _ROUGE-N\({}_{f-score}\)_ are mathematically defined by Equations (5)-(7), respectively. \[ROUGE-N_{precision}=\frac{m_{overlap}^{n}}{m_{out}^{n}}, \tag{5}\]  
where \(m_{out}^{n}\) is the number of \(n\)-grams in the chatbot output, and \(m_{overlap}^{n}\) is the number of overlapping \(n\)-grams in the chatbot output and the reference. \[ROUGE-N_{recall}=\frac{m_{overlap}^{n}}{m_{ref}^{n}}, \tag{6}\]  
where \(m_{ref}^{n}\) is the number of \(n\)-grams in the reference, and \(m_{overlap}^{n}\) is the number of overlapping \(n\)-grams in the chatbot output and the reference. \[ROUGE-N_{f-score}=\frac{2\times ROUGE-N_{precision}\times ROUGE-N_{recall}}{ ROUGE-N_{precision}+ROUGE-N_{recall}}. \tag{7}\]  
In addition to ROUGE-N, we also evaluated ROUGE-L, which measures the longest common subsequence (LCS) of tokens and even catches sentence-level word order. _ROUGE-L\({}_{precision}\)_, _ROUGE-L\({}_{recall}\)_, and _ROUGE-L\({}_{f-score}\)_ are mathematically defined by Equations (8)-(10), respectively. \[ROUGE-L_{precision}=\frac{LCS}{m_{out}}, \tag{8}\]  
where \(m_{out}\) is the number of tokens in the chatbot output. \[ROUGE-L_{recall}=\frac{LCS}{m_{ref}}, \tag{9}\]  
where \(m_{ref}\) is the number of tokens in the reference.