# Building a Production-Ready Multi-Label Classifier for Legal Documents with Digital-Twin-Distiller
## 4 Results
### Validation on Benchmark Set  
A benchmark set was created by selecting 150 documents randomly from the held-out test set. Two experts annotated and cross-checked this dataset to ensure additional attention and reduce the probability of mistakes. After comparing the annotations with the machine-learning-based labeling results, mistakes in seven documents were identified. Two documents were mislabeled and were fixed based on the returned value of the classifier, one document received a label that was not present in the particular area of law, and in an additional four cases, a misspelling-type error occurred. Humans are typically prone to the two latter error types, while machines perform better. It shows the difficulty of this task that the human-level performance is lower than 100% after a cross-check, while the processing time of the total 170,000 documents would take about eight years of work with this methodology. The results of the benchmark are discussed in Table 4. It can be seen that about 57.3% of the randomly selected documents partly or wholly matched with labels given by the annotators. About 43% of the documents did not have a common label with the annotator-created labels. However, by examining the not matching documents, 22 out of the 64 documents have been given an other-typed label by the classifier, not providing the most specific label, but these are by no means wrongly labeled documents. As these legal categories are not totally disjunct sets (so many of them overlap), more than one solution can be acceptable, while the legal experts only gave the most specific labels to the documents. Another 14 of the 64 wrongly labeled documents had semantically similar labels to the ones given by the expert. For instance, \"Transport and freight contract\" and \"International transport contract\" or \"Matters concerning business organizations\" and \"Matters concerning companies\" are labels closely related to each other semantically. When a human legal expert annotates, they take into consideration not only the words in the written text, but also their legal background knowledge. Such categories proved to be hard for the classifiers to identify due to the selected vectorization form. Nevertheless, another 14 documents suffered from wrongly defined labels (defining a too broad category), and the classifier made mistakes in 14 documents. On a document level, 28 out of 150 documents were mislabeled, an 18.67% error rate. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Did Not Match** & **Partly Matched** & **Completely Matched** & **Sum** \\ \hline Count & 64 & 40 & 46 & 150 \\ \hline Percentage (\%) & 42.67\% & 26.67\% & 30.67\% & 100\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results on 150 documents tested manually on held-out test set. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**APD** & **NF** & **Positive Count** & **Negative Count** & **Average (\%)** & **Std (\%)** & **Min (\%)** & **Max (\%)** \\ \hline No & No & 32 & 10,012 & 54.15 & 24.82 & 0 & 100 \\ \hline No & Yes & 32 & 9072 & 73.95 & 24.14 & 0 & 100 \\ \hline Yes & No & 102 & 10,012 & 81.71 & 7.87 & 60.87 & 94.74 \\ \hline Yes & Yes & 102 & 9072 & 91.97 & 5.77 & 82.35 & 100 \\ \hline \Correlation between Categories  
During the validation of the cases, an interesting phenomenon was identified: how the classifier tended to make mistakes. This happened when a legal category did not occur by itself, but always in pair with other labels, for which _Contivance_ label is a good example. This originates from the meaning of convinance: someone cannot commit this crime without another crime being committed. However, if the category behaves like this and the joint categories are not uniformly distributed, e.g., in the majority of the cases _Convinance_ label is alongside _Fraud_, the classifier will likely be biased and misclassify cases that are only _Fraud_ cases to be _Convinance_ cases as well.