Title: IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning
Benchmark:  IL-TUR
Summary: This paper introduces IL-TUR, a benchmark designed to evaluate NLP models for legal text understanding and reasoning in the Indian legal context. The benchmark covers both monolingual (English, Hindi) and multilingual tasks across 9 Indian languages. It provides baseline models and a leaderboard with comparison in automating legal document processing.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Legal text understanding and reasoning in Indian and multilingual context
Phenomenon defined by authors: Yes
Phenomenon definition: The tasks should cater exclusively to the legal domain. Solving a task should require in-depth knowledge and understanding of the law and its associated areas. [...] Moreover, solving legal tasks should require knowledge about the law as well as commonsense knowledge and societal norms about the world.
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: There are 6 types of tasks: sequence classification, multi-class classification, Classification and extraction, classification, multi-label classification, retrieval, and generation
Task item definition: text (text, short text, paragraph, sentences) and the output label or text
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Crowd-sourced task examples (e.g. Prolific-created tasks), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: CJPE (Court Judgment Prediction with Explanation) or BAIL (Bail Prediction) are framed as supervised classification tasks using labeled court documents, which is a constructed task. 
Legal Named Entity Recognition (L-NER) and Rhetorical Role Prediction (RR), which model foundational capabilities for downstream legal NLP applications are representative tasks to model the legal system, the rest are partial real tasks as the use case is the real legal docs
Dataset size: ~15,100, but the information is provided for some tasks and datasets
Dataset size notes: some tasks/datasets have this info some not
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), BERTScore, GLEU
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric metascoring: majority@k (majority vote over k trials)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: "The tasks should cater exclusively to the legal domain. Solving a task should require in-depth knowledge and understanding of the law and its associated areas [...] as well as commonsense knowledge and societal norms."
Taxonomy: Law / 
New taxonomy: Specific Application / Law