# Building a Production-Ready Multi-Label Classifier for Legal Documents with Digital-Twin-Distiller
## 4 Results
### Data Augmentation  
The dataset contained a significant proportion of categories with a very low abundance. A widely used approach for boosting the classifiers' performance is to use text augmentation, which means increasing the number of documents by generating \"synthetic\" instances [40; 54]. In this study, we have tried using Easy Data Augmentation (EDA) text augmentation techniques which includes Random Deletion, Random Insertion, Synonym Replacement, and Random Swap [40; 55]. As the relevant legal expressions usually occur in a specific order in the text, we wanted to preserve these, therefore we did not use the Random Swap technique during this study. EDA augmentation techniques have been successfully applied in sentiment-analysis of short Hungarian political texts to increase the efficiency of classification [36]. The methods are openly accessible as a Python package via pip ([[https://pypi.org/project/text-augmentation/](https://pypi.org/project/text-augmentation/)]([https://pypi.org/project/text-augmentation/](https://pypi.org/project/text-augmentation/)), accessed on 25 January 2021). From the original _\"Termination of employment \"_ dataset, different sizes of positive samples were selected from the original ca. 5000 documents to measure the performance of the augmentation. The initial numbers of these sets were: 10, 20, 50, 100, 250, and 500 documents. In this study, these numbers are referred as base numbers because these numbers served as a base for the augmentation. We have compared datasets where the ratio of positive and negative labels was 1:1 and 1:2.57 (the latter is the stratified ratio on the whole dataset). During the experiment, the effect of augmentation was investigated by  
Figure 4: Effect of positive document counts on classifier performanceaugmenting the original data to a step higher in the base number sequence (for example, producing 20 documents from 10 or 250 from 100). Each result was gained by repeating the calculations 10 times and averaging them. First, we determined how much performance is gained by using data augmentation compared to not using any. This was calculated by subtracting the \(F_{1}\) value measured on the original data from the \(F_{1}\) value measured on the augmented data. For instance, if we augmented 10 documents to 20 (10 original and 10 augmented samples), then the \(F_{1}\) score found with this dataset was compared with the \(F_{1}\) score from using just the original 10 documents. The difference between these two metrics is indicated in Figure 5 as \(\Delta A\). Next, we considered the difference between the augmented and the real, larger dataset. For this, we subtracted the \(F_{1}\) value measured on augmented data from the \(F_{1}\) value measured on original data. For instance, the \(F_{1}\) score calculated using 20 augmented documents was subtracted from the \(F_{1}\) score measured on 20 original documents. This difference is marked with \(\Delta B\). The augmentation methods have an \"Alpha\" parameter (\(\alpha\)), which refers to the ratio of the changed tokens (words) during the augmentation process [40; 54; 55]. For instance, if \(\alpha\) is set to 0.1, then 10% of the tokens will be modified during the augmentation (however, in case of Synonym Replacement, the number of tokens replaced may differ, as this method uses WordNet [56; 57] to identify synsets, and if the given token is absent, no change will be made). As Figure 5 shows, adding augmented data to the dataset improved the performance of the classification in almost every case. The classification performed well with the same size of original and augmented data, suggesting that with relatively small datasets (<100) the original data significantly outperforms the augmented data, but after 100 documents, the results seem to even out. Figure 6 shows a more detailed overall comparison of results achieved by the applied techniques and parameters. The results show that in the, augmented cases, 100 documents represent some kind of threshold.