Title: WaterBench: Towards Holistic Evaluation of Watermarks for Large Language Models
Benchmark: WaterBench
Summary: WaterBench is a benchmark for evaluating LLM watermarks across detection and generation quality. The paper also presents a hyper-parameter search method to control watermarking strength, and automatic evaluation using GPT4-Judge. The dataset is publicly available and human-validated. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: LLM watermarks
Phenomenon defined by authors: Yes
Phenomenon definition: A watermarked LLM generates texts "with a biased distribution of tokens, which distinguishes it from unwatermarked texts, ... , the goal of watermarking is to achieve high detection accuracy while maintaining the generation quality" (1517). 
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Task definition: WaterBench consists of 9 tasks with 5 unique task settings, spanning "a wide range of input and output length" (1520). The first setting is Short Input, Short Answer, and has two tasks to evaluate factual knowledge. The second setting is Short Input, Long Answer, with two Long-form QA tasks. The third category is Long Input, Short Answer, with reasoning and code completion tasks. The fourth setting is Long Input, Long Answer, with two summarization tasks. The last setting is open-ended generation, where the task is instruction-following. 
Task item definition: Each task is sourced from a different dataset and has its own features. 
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task source detail: Each task is sourced from a different dataset. 
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 2405
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Generation Metric and Generation Quality Drop are never explicitly defined in the paper. 
Metric definition detail: The paper defines watermarking strenth as the true positive rate to ensure all watermarks are of similar intensity during evaluation. WaterBench also uses GPT4-Judge, which "measures which model's output the GPT-4 system prefers when shown two responses for the same instruction" (1521). The paper reports the "True Positive Rate, True Negative Rate, Generation Metric, and Generation Quality Drop for all tasks" (1523). 100 responses are randomly sampled for human annotation as well. 
Metric aggregation: Simple Mean
Metric statistics: True Positive Rate, True Negative Rate, Generation Metric and Generation Quality Drop
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors highlight that evaluating watermark must "evaluate the generation and detection" methods, and ensure fair comparisons between watermarking methods (1517). Additionally, the authors highlight that the tasks must be diverse, go beyond "text completion," and measure "generation quality" and alignment with "human preferences" (1517-1518). 
Taxonomy: NLP / Detection
New taxonomy: General Application / LLM Detection