## 6 Results and Discussion
### Experiment (A)  
The goal of the first experiment is to highlight the different performances of NLU platforms. For this reason, we expected performances which were not uniform among different NLU platforms but, on the other hand, we supposed that the outcomes would be similar on the different randomly built datasets described in Section 5.1 (from here called D1, D2\(\ldots\)D10). Table 2 presents the performance for each service in terms of Error rate and F-score (We used the Python function f1_score belonging to the sklearn module. As the dataset was unbalanced, we set the parameter \"average\" equal to \"weighted\".) on the ten different datasets, while the last two columns show the average and standard deviation. The best result regarding F-score for each dataset is reported in bold. As we supposed, there are several differences among the selected platforms in terms of general performance. The average F-score is quite similar between Dialogflow and Watson, which is over 0.86; however, Watson has a greater standard deviation. Luis' results are slightly worse than the other two with a 0.82 F-score, while Wit.ai obtains the worst performance (slightly above 0.71). In our datasets, neither Luis nor Wit.ai managed to overcome Dialogflow or Watson in terms of F-Score. We can see that most platforms have low standard deviation (StD) and that their results are stable considering our ten different datasets. The number of examples is not the same for all the intentions in our evaluation. Consequently, the performance of the NLU services can be quite different among the intentions, as we can see in Figure 5 where we present separately the F-score outcome for every single intention. The Intention ID and number of examples used to train the services are shown in the legend of the figure. Looking at these diagrams, we notice how intention I9, which uses the highest number of elements, has one of the highest median and lowest dispersion among all services. I2, on the other hand, is the intention trained with the lowest  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c} \hline \hline  & & **D1** & **D2** & **D3** & **D4** & **D5** & **D6** & **D7** & **D8** & **D9** & **D10** & **AVG** & **StD** \\ \hline \multirow{3}{*}{Watson} & Err & 0.136 & 0.144 & 0.091 & 0.121 & 0.114 & 0.174 & 0.151 & 0.1364 & 0.167 & 0.144 & 0.138 & 0.023 \\  & FS & 0.857 & 0.851 & **0.907** & 0.878 & **0.887** & 0.822 & **0.845** & 0.863 & 0.832 & 0.857 & 0.860 & 0.02 \\ \hline \multirow{3}{*}{Dailoflow} & Err & 0.129 & 0.144 & 0.144 & 0.106 & 0.121 & 0.151 & 0.159 & 0.106 & 0.136 & 0.144 & 0.134 & 0.017 \\  & FS & **0.877** & **0.851** & 0.856 & **0.894** & 0.875 & **0.857** & 0.844 & **0.893** & **0.871** & **0.863** & **0.868** & **0.016** \\ \hline \multirow{3}{*}{Luis} & Err & 0.174 & 0.197 & 0.144 & 0.167 & 0.159 & 0.174 & 0.182 & 0.167 & 0.144 & 0.182 & 0.169 & 0.016 \\  & FS & 0.825 & 0.785 & 0.853 & 0.823 & 0.840 & 0.828 & 0.815 & 0.831 & 0.851 & 0.819 & 0.827 & 0.018 \\ \hline \multirow{number of examples and, for this reason, its performance considerably varies among the different NLU platforms. Nevertheless, in all of them the median is far below 0.8 and the boxes are very stretched. Dialogflow has better overall results and the boxes are shorter than other platforms, but it presents an anomaly: the intention I2 has a significantly worse median value compared to Watson and Luis and its dispersion index is also very high. This may be due to the fact that Dialogflow needs a slightly higher number of training instances to achieve good performance, compared to the others. The outcomes of Watson and Luis are pretty good, with the median value always over 0.7. Wit.ai achieves the worst performance among the platforms tested. In fact, two of the intentions tested have a median value under 0.6 while only the other two intentions manage to reach a median value over 0.8. Furthermore, most of the intentions tested have a very high dispersion index.