Title: PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English
Benchmark: PLUE
Summary: The paper introduces the Privacy Policy Language Understanding Evaluation (PLUE) benchmark, a multi-task benchmark for evaluating the privacy policy language understanding across various tasks. They also collect a large corpus of privacy policies
to enable privacy policy domain-specific language model pre-training. They evaluate several
generic pre-trained language models and continue pre-training them on the collected corpus.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Privacy policy language understanding
Phenomenon defined by authors: Yes
Phenomenon definition: Privacy policies provide individuals with information about their rights and how their personal information is handled. The ability of LLMs to understand privacy policy is measured here.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: There are 6 different datasets, including multi-label classifification, NER, and QA tasks.
Task item definition: There are 6 different datasets, and each contain a paragraph/document about a privacy policy and relavant questions about that.
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size notes: Train:~38k, Validation:~7k, Tes:~10k
Sampling method: Unknown
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Law / 
New taxonomy: Specific Application / Law