Title: \inftyBench: Extending Long Context Evaluation Beyond 100K Tokens
Benchmark: InfiniteBench
Summary: The paper presents InfiniteBench, a new benchmark to evaluate LLMs’ ability to process, understand, and reason over ultra-long contexts over 100k tokens in length. InfiniteBench contains both real and synthetic tasks which present notable challenge to existing SOTA LLMs.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: long-context understanding and reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: "the ability to process long contexts is increasingly critical...  Textual documents, historical
dialogues, complex instructions, and cumbersome workflows, which constitute the data most directly processed in daily tasks, must be input to LLMs as long contexts for effective processing."
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Take a long input context and task instruction and/or question and provide an answer.
Task item definition: Each sample is represented as the input context, task/question, answer options (if applicable), and ground truth answer.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 3946
Dataset metadata available: Yes
Dataset metadata detail: avg input length, avg output length, annotation method
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragarph), Extended interaction (e.g. conversation, calling an API and processing the response)
Metric definition: LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: task type
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context