# PrivacyGLUE: A Benchmark Dataset for General Language Understanding in Privacy Policies
## 6 Discussion
### Model-Pair Agreement Analysis  
PrivBERT, the top-performing model, differentiates itself from other models by its in-domain pretraining on the PrivaSeer corpus [27]. Therefore, we can infer that PrivBERT incorporated _knowledge_ of privacy policies through its pretraining and became specialized for fine-tuning tasks in the privacy language domain. We investigate this specialization using model-pair agreement analysis to detect examples where PrivBERT had a competitive advantage over other models. Consequently, we detect examples where PrivBERT was disadvantaged due to its in-domain pretraining.  
We compare \(10\times 10=100\) random seed combinations for all test-set pairs between PrivBERT and other models. Each prediction-pair can be classified into one of four mutually exclusive categories (B, P, O and N) shown below. Categories B and N represent examples that are either not challenging or too challenging for both PrivBERT and the other model respectively. Categories P and O are more interesting for us since they indicate examples where PrivBERT had a competitive advantage and disadvantage over the other model, respectively. Therefore, we focus on categories P and O in our analysis. We classify examples over all random seed combinations and take the majority occurrence for each category within its distribution.  
Both PrivBERT and the other model were correct, that is, (PrivBERT, Other Model)  
PrivBERT was correct and the other model was wrong, that is, (PrivBERT, Other Model)  
Other model was correct and PrivBERT was wrong, i.e., (\(\
eg\) PrivBERT, Other Model)  
Neither PrivBERT nor the other model was correct, that is, (\(\
eg\) PrivBERT, Other Model)  
Figure 4 shows a relative distribution of majority categories across model-pairs and PrivacyGLUE tasks. We observe that category P is always greater than category O, which correlates with PrivBERT outperforming all other models. We also observe that category P is often the greatest when compared against BERT, implying that PrivBERT has the most competitive advantage over BERT. Surprisingly, we also observe category O is often the greatest when compared against BERT, implying that BERT has the highest absolute advantage over PrivBERT. This is an insightful observation since we would have expected BERT to have the least competitive advantage given its lowest overall PrivacyGLUE performance.  
To investigate PrivBERT's competitive advantage and disadvantage against BERT, we extract several examples from categories P and O in the PrivacyQA task for brevity. Two interesting examples are listed in Table 4 and additional examples can be found in Table A4 in Appendix D. From Table 4, we speculate that PrivBERT specializes in example 1978 because it contains several privacy-specific terms such as \"third parties\" and \"explicit consent\". On the other hand, we speculate that BERT specializes in example 33237 since it contains more generic information regarding encryption and SSL, which also happens to be a topic in BERT's Wikipedia pretraining corpus as seen in Figure 1 and Table 2.  
Looking at further examples in Table A4, we can also observe that all sampled category P examples have the Relevant label, while many sampled category O examples have the Irrelevant label. On further analysis of the PrivacyQA test-set, we find that 71% of category P examples have the Relevant label and 61% of category O samples have the Irrelevant label. We can infer that PrivBERT specializes in the minority Relevant label while BERT specializes in the majority Irrelevant label as the former label could require more privacy knowledge than the latter.