Title: PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data
Benchmark: PARIKSHA
Summary: This paper presents Pariksha, a benchmark of culturally-nuanced prompts in 10 Indic languages created by native speakers. This benchmark is used to evaluate the agreement between human and LLM evaluators in two settings, pairwise comparisons and direct scoring.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Open-ended question answering in Indic languages
Phenomenon defined by authors: No
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Not defined
Task definition: Answer open-ended general and culturally-nuanced questions in 10 Indic language (Hindi, Tamil, Telugu, Malayalam, Kannada, Marathi, Odia, Bengali, Gujarati, and Punjabi).
Task item definition: A prompt.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 20 prompts * 10 languages = 200 prompts
Dataset metadata available: No
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: Both humans and the LLM evaluate the same prompt-response pairs using comparable guidelines. The evaluation includes pairwise comparisons scored using Elo ratings, and direct assessments based on linguistic acceptability, task quality, and hallucination metrics.
Metric aggregation: Simple Mean, Maximum Likelihood Estimation (MLE) Elo
Metric statistics: Simple mean (for direct assessment scores), Elo rating via Maximum Likelihood Estimation (MLE) (for pairwise comparisons), Fleiss’ Kappa and Percentage Agreement (for inter-annotator agreement and human-LLM agreement), and Kendall’s Tau (for human-LLM leaderboard rank correlation).
Metric subscores: Yes
Metric subscores detail: By language, evaluator (human or LLM), evaluation type (pairwise or direct assessment), model type (Indic, Proprietary and Open-Source base LLMs)
Metric metascoring: majority@k (majority vote over k trials), None
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: LLM as a Judge / 
Alternate taxonomy: Multilinguality
New taxonomy: General Application / LLM as a Judge