# Evaluating Human versus Machine Learning Performance in a LegalTech Problem
## 3 Results
### Inter-Annotator Agreement  
The aim of measuring the inter-annotator agreement is to assess an annotation process' reliability (IAA). During this evaluation, we did not use the reference set for the comparison. Figure 6: How many points the editors gained from one document in average based on the extracted information content compared to the machine learning algorithm. Figure 7: The performance of the best-performing legal editor and the group of legal editors who got unlabeled judgments over time. The three members of each group were considered as an annotator. The reliability of their annotations measured with Krippendorffs alpha (\(K_{\alpha}\)) [45; 46], which is a widely used statistical measure, it differs from most of the other IAA methods because it calculates disagreement between the different voters [47]. The \(K_{\alpha}\) statistical measure is selected because it can handle the missing data, various sample sizes, categories. The reliability measure is easy to interpret and does not depend on the number of categories [497 ], The simplest form of \(K_{\alpha}\) can be calculated by the following formula [45]:  
\[K_{\alpha}=1-\frac{D_{0}}{D_{e}}, \tag{1}\]  
where \(D_{0}\) stands for the observed disagreement among values, and \(D_{e}\) is for \"disagreement one would expect when the coding of units is attributable to chance rather than to the properties of these units\" [45]. The result of the \(K_{\alpha}\) calculation is a number between \(-1\) and \(1\), where \(1\) indicates perfect agreement, \(0\) indicates no agreement beyond chance and negative values indicate inverse agreement. \(K_{\alpha}\geq 0.8\) means usually the acceptance limit. Here the tentative results are also acceptable. The lowest limit for an acceptable agreement is when \(K_{\alpha}\geq 0.667\). This is the minimal requirement to consider an inter-agreement calculation reliable [45; 47]. The metric has an important component specific to the actual problem called the difference function, which is used to weight the numerator and denominator. The measuring agreement on set-valued items distance metric seemed to be the most appropriate in our case, due to a large number of possible annotation categories [50]. Figure 8 shows the calculated \(K_{\alpha}\) scores in percentage for each annotator group. There is a surprising result; the group of the laymen who used the computer assistance achieved the highest score in \(K_{\alpha}\). They achieved 55% reliability, which is significantly higher than the reliability of the professional editors. There are two reasons for this surprising result. Firstly, if we compare the result of the three groups who could not use the assistance of the machine learning methodology, we can see from (Figure 8) that these groups achieved similar reliability, independently from their experience. This reliability score was very low in these cases. The results suggest that the laymen group, who trusted the result of the computer annotation, achieved the highest reliability score in the survey. This indicates that the machine learning-based methodologies produce more consistent solutions for large databases than the humans. However, this highest reliability score is lower than the required minimum (\(K_{\alpha}\geq 66\%\)). The second reason is that the categorization group should be revised. The poor agreement between the human experts suggests that the labels are not straightforward and independent from each other, and they did not look for every possible combination due to the time limitations. Figure 8: [PERSON]’s Alpha scores in different annotator groups with (pre-labeled) and without machine suggested labels (unlabeled). The percentage agreement between the different groups was also calculated to examine further this reason for the poor reliability between the professional editors (Figure 9). This percentage agreement examines a more permissive measure than the \(K_{\alpha}\). It shows the ratio of documents, where all three members of the group gave at least one similar label to the given document.