## 5 Experiments and Results
\[ROUGE-L_{f-score}=\frac{2\times ROUGE-L_{precision}\times ROUGE-L_{recall}}{ ROUGE-L_{precision}+ROUGE-L_{recall}}. \tag{10}\]  
Both ROUGE-N and ROUGE-L differ from the BLUE score because they do not have the brevity penalty (responsible for high-scoring outputs matching the reference text in length). ROUGE-N computes the \(n\)-gram match only for the chosen fixed \(n\)-gram size \(n\), whereas ROUGE-L searches for the longest overlapping sequence of tokens and does not require defining the \(n\)-grams at all. Both ROUGE-N and ROUGE-L can be any value within the interval [0, 1]. Despite this, there are neither defined thresholds nor their interpretations; higher ROUGE-N and ROUGE-L values denote the higher intelligence of the chatbot. Both metrics are important for our intra-comparison purposes. For ROUGE-N and ROUGE-L evaluation, we used Python library _rouge_[41], with \(N=2\) in all our evaluations. To train and test our seq2 seq models and to overcome the problem of a very small dataset (described in Section 3), we used five-fold cross validation (20% of the training data were used for validation). The obtained results were averaged and the confidence intervals (with a confidence level of 95% and alpha = 0.05) were calculated. \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multicolumn{1}{c}{**Experiments**} & \multicolumn{3}{c}{**ROUGE-2**} & \multicolumn{3}{c}{**ROUGE-L**} \\ \cline{2-7}  & **Precision** & **Recall** & **f-Score** & **Precision** & **Recall** & **f-Score** \\ \hline EN & 0.458 \(\pm\) 0.035 & 0.478 \(\pm\) 0.025 & 0.463 \(\pm\) 0.031 & 0.525 \(\pm\) 0.033 & 0.559 \(\pm\) 0.022 & 0.522 \(\pm\) 0.030 \\ \hline LT & 0.467 \(\pm\) 0.042 & 0.475 \(\pm\) 0.044 & 0.469 \(\pm\) 0.043 & 0.526 \(\pm\) 0.035 & 0.552 \(\pm\) 0.045 & 0.522 \(\pm\) 0.038 \\ \hline ft\_encoder\_EN & 0.307 \(\pm\) 0.020 & 0.323 \(\pm\) 0.025 & 0.310 \(\pm\) 0.022 & 0.385 \(\pm\) 0.018 & 0.423 \(\pm\) 0.021 & 0.382 \(\pm\) 0.017 \\ \hline ft\_encoder\_LT & 0.401 \(\pm\) 0.038 & 0.411 \(\pm\) 0.038 & 0.403 \(\pm\) 0.038 & 0.476 \(\pm\) 0.039 & 0.498 \(\pm\) 0.041 & 0.470 \(\pm\) 0.041 \\ \hline ft\_encoder\_decoder\_EN & 0.326 \(\pm\) 0.022 & 0.363 \(\pm\) 0.024 & 0.337 \(\pm\) 0.023 & 0.416 \(\pm\) 0.023 & 0.483 \(\pm\) 0.025 & 0.422 \(\pm\) 0.024 \\ \hline ft\_encoder\_decoder\_LT & 0.296 \(\pm\) 0.040 & 0.405 \(\pm\) 0.060 & 0.331 \(\pm\) 0.044 & 0.372 \(\pm\) 0.033 & 0.491 \(\pm\) 0.050 & 0.388 \(\pm\) 0.036 \\ \hline BERT\_encoder\_EN & 0.395 \(\pm\) 0.034 & 0.418 \(\pm\) 0.032 & 0.401 \(\pm\) 0.034 & 0.474 \(\pm\) 0.029 & 0.515 \(\pm\) 0.026 & 0.474 \(\pm\) 0.029 \\ \hline BERT\_encoder\_LT & 0.400 \(\pm\) 0.026 & 0.427 \(\pm\) 0.030 & 0.407 \(\pm\) 0.027 & 0.469 \(\pm\) 0.032 & 0.518 \(\pm\) 0.037 & 0.471 \(\pm\) 0.034 \\ \hline BERT\_encoder\_decoder\_EN & 0.402 \(\pm\) 0.019 & 0.426 \(\pm\) 0.020 & 0.409 \(\pm\) 0.019 & 0.473 \(\pm\) 0.023 & 0.517 \(\pm\) 0.023 & 0.477 \(\pm\) 0.023 \\ \hline BERT\_encoder\_decoder\_LT & 0.369 \(\pm\) 0.026 & 0.386 \(\pm\) 0.029 & 0.372 \(\pm\) 0.026 & 0.442 \(\pm\) 0.030 & 0.472 \(\pm\) 0.026 & 0.436 \(\pm\) 0.025 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Averaged Recall-Oriented Understudy for Gisting Evaluation (ROUGE)-2 and ROUGE-L precision/recall/f-score values with confidence intervals using the LSTM encoder and decoder model with different vectorization types on the dataset with removed punctuation. The best values in the column for EN and LT are in bold. BERT—Bidirectional Encoder Representations from Transformers; ft—FastText.