# PrivacyGLUE: A Benchmark Dataset for General Language Understanding in Privacy Policies  
[PERSON]  
1 Information Systems Research Lab, HSLU--Lucerne University of Applied Sciences and Arts, Suurstoffi 1, CH-6343 Rottreuzu, Switzerland 1  
[PERSON]  
1 Information Systems Research Lab, HSLU--Lucerne University of Applied Sciences and Arts, Suurstoffi 1, CH-6343 Rottreuzu, Switzerland 1  
[PERSON]  
1 Information Systems Research Lab, HSLU--Lucerne University of Applied Sciences and Arts, Suurstoffi 1, CH-6343 Rottreuzu, Switzerland 1  
[PERSON]  
1 Information Systems Research Lab, HSLU--Lucerne University of Applied Sciences and Arts, Suurstoffi 1, CH-6343 Rottreuzu, Switzerland 1  
[PERSON]  
1 Information Systems Research Lab, HSLU--Lucerne University of Applied Sciences and Arts, Suurstoffi 1, CH-6343 Rottreuzu, Switzerland 1  
Footnote 1: [[https://www.mdpi.com/journal/applsci](https://www.mdpi.com/journal/applsci)]([https://www.mdpi.com/journal/applsci](https://www.mdpi.com/journal/applsci))  
Footnote 2: [[https://www.mdpi.com/journals/applsci](https://www.mdpi.com/journals/applsci)]([https://www.mdpi.com/journals/applsci](https://www.mdpi.com/journals/applsci))  
Footnote 3: [[https://www.mdpi.com/journals/applsci](https://www.mdpi.com/journals/applsci)]([https://www.mdpi.com/journals/applsci](https://www.mdpi.com/journals/applsci))  
Footnote 4: [[https://www.mdpi.com/journals/applsci](https://www.mdpi.com/journals/applsci)]([https://www.mdpi.com/journals/applsci](https://www.mdpi.com/journals/applsci))  
Footnote 5: [[https://www.mdpi.com/journals/applsci](https://www.mdpi.com/journals/applsci)]([https://www.mdpi.com/journals/applsci](https://www.mdpi.com/journals/applsci))  
Footnote 6: [[https://www.mdpi.com/journals/applsci](https://www.mdpi.com/journals/applsci)]([https://www.mdpi.com/journals/applsci](https://www.mdpi.com/journals/applsci))  
Footnote 7: [[https://www.mdpi.com](https://www.mdpi.com)]([https://www.mdpi.com](https://www.mdpi.com))  
In recent years, benchmarks have been gaining popularity in Machine Learning and Natural Language Processing (NLP) communities because of their ability to holistically evaluate model performance over a variety of representative tasks, thus allowing practitioners to compare and contrast different models on multiple tasks relevant for the specific application domain. General Language Understanding Evaluation (GLUE) [5] and SuperGLUE [6] are examples of popular NLP benchmarks which measure the natural language understanding capabilities of state-of-the-art (SOTA) models. NLP benchmarks are also developing rapidly in language domains, with LexGLUE [7] being an example of a recent benchmark hosting several difficult tasks in the legal language domain. Interestingly, we do not find similar NLP benchmarks in the privacy language domain for privacy policies. While this could be explained by privacy policies falling under the legal language domain due to their formal and jargon-heavy nature, we claim that privacy policies fall under a distinct language domain and cannot be subsumed under any other specialized NLP benchmark such as LexGLUE. To investigate this claim, we gather documents from Wikipedia [8], European Legislation (EURLEX) [9] and company privacy policies [10], with each corpus truncated to 2.5 M tokens. Next, we feed these documents into BERT and gather contextualized embeddings, which are then projected to a two-dimensional space using Uniform Manifold Approximation and Projection (UMAP) [11]. In Figure 1, we observe that the three domain corpora cluster independently, providing evidence that privacy policies lie in a distinct language domain from both legal and Wikipedia documents, and therefore require an independent NLP benchmark. With this motivation, we propose PrivacyGLUE as the first comprehensive benchmark for measuring general language understanding in the privacy language domain. Our main contributions are threefold:  
1.