# Comparing the Quality and Speed of Sentence Classification with Modern Language Models
## 4 Results and Discussion  
### Preliminary Study  
Figure 2 shows that increasing the size of the sentence-level vector representation created by the RNN increased the TT. Simultaneously, for the short questions in the TREC6 data set, sentence embeddings as short as 16 elements enabled high performance. A higher dimensionality of sentence embeddings did not appear to improve the MCC score since no statistically significant differences were found; however, owing to a relatively low increase in the TT, we used a hidden size of 256 to ensure that we provided sufficient space for the information present in the sentences.  
Figure 3 illustrates the effect of the mini-batch size on the framework performance, given a time-consuming training regimen. For the given setup, there were no statistically significant differences in the MCC scores with small mini-batch sizes of 4 and 8. However, the small mini-batch sizes were significantly superior to the higher values of 32 and 64. Additionally, the TT with a mini-batch size of 8 was significantly shorter than that with a batch size of 4. Therefore, we used a mini-batch size of 8. The full results of the statistical analysis have been published, along with the MPD and python code used in our research [18].  
Figure 4 presents the effect of shuffling examples during training. With shuffling, the framework achieved higher MCC values, and the differences were statistically significant. Therefore, we decided to shuffle the training examples.  
Figure 2: Impact of the hidden size on the framework performance given a time-consuming training regimen (initial learning rate = 0.1, minimal learning rate = 0.0001, annealing factor = 0.5, patience = 20, mini-batch size = 32, shuffle = True) on the TREC6 data set. MCC: Matthews Correlation Coefficient.  
Table 3 depicts the influence of the training-rate-related parameters on the performance and TT of the adopted framework for the TREC6 set and the MPD. The performance discrepancies for the MPD and the TREC6 set measured in points of the mean MCC score were 0.05 and 0.036, respectively. When the mean TT was considered, the differences were up to 0.116 h for the MPD and 0.643 h for the TREC6 set. It can be observed that the relation of the TT achieved in various test runs was very similar in both data sets. This allowed us to hypothesize that the relation of the TT between test runs did not vary strongly with the choice of data set. However, this was not the case when the LMs quality was concerned. We believe that the proposed training regimes for the TREC6 data set, when applied to the MPD, caused each model to be as well trained, with only one exception of a very short training scheme p5 lr.2 mlr.008a.5. This was likely caused by the smaller size of the MPD, which allowed the LMs to fit the data in a shorter time.  
Based on this exploratory study, we decided to use a training regimen providing almost the best performance for both the TREC6 set and the MPD, namely p20 lr.1 mlr.00 za.5. For the TREC6 set, the selected training procedure allowed for a 0.933 mean MCC score at a decent TT cost of a median of 0.269 h per trained model. Therefore, the configuration of the adopted framework in the final LM comparison was as follows: patience = 20, initial learning rate = 0.1, minimal learning rate = 0.002, annealing factor = 0.5, mini-batch size = 8, hidden size = 256, with example shuffling during training.  
Figure 4: Effect of shuffling the training examples on the framework performance given a time-consuming training regimen (initial learning rate = 0.1, minimal learning rate = 0.0001, annealing factor = 0.5, patience = 20, hidden size = 256, batch size = 32) on the TREC6 data set.  
Figure 3: Impact of the mini-batch size on the framework performance given a time-consuming training regimen (initial learning rate = 0.1, minimal learning rate = 0.0001, annealing factor = 0.5, patience = 20, hidden size = 256, shuffle = True) on the TREC6 data set.