Title: CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation
Benchmark: CopyBench
Summary: COPYBENCH is introduced to evaluate both literal and non-literal reproduction of copyrighted content by language models (LMs), addressing a gap where previous research only considered literal similarities. Using copyrighted fiction books, COPYBENCH assesses literal and non-literal copying, finding that while literal copying is rare, non-literal copying, such as event and character copying, is more prevalent, especially in larger models. The benchmark reveals that training-time alignment can reduce literal copying but may increase non-literal copying, and current inference-time methods are more effective for literal copying than for non-literal copying, highlighting areas for improvement in copyright mitigation strategies.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Literal and non-literal copying in LM generations
Phenomenon defined by authors: Yes
Phenomenon definition: Literal copying assesses the extent to which
a model can reproduce copyright-protected content exactly as it appears in the source material, non-literal copying evaluates whether a
model generates outputs that, despite differing in
surface form (e.g., through paraphrasing), exhibit
a high degree of overlap in content.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: measure the degree of literal,
non-literal copying and fact recall on a list of
copyright-protected fiction book
Task definition detail: Text, copying, utility
Task source: Text snippets from books
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 4633
Dataset metadata available: No
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: copying types, utility
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / Copyright
New taxonomy: General Application / LLM Detection