Title: MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
Benchmark: MLLM-as-a-Judge
Summary: This paper introduces the MLLM-as-a-Judge benchmark to assess the ability of MLLMs in assisting judges across diverse modalities, encompassing three distinct tasks: Scoring Evaluation, Pair Comparison, and Batch Ranking.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: MLLM-as-a-judge
Phenomenon definition: capability of MLLMs in tasks of Scoring Evaluation, Pair Comparison and Batch Ranking.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Take a single MLLM response, provide the score; Take two MLLM responses, compare which one is better; Take a batch of MLLM responses, provide a ranking
Task item definition: <two MLLM responses>, Judgement: B
Task source: Expert-crafted task examples (e.g. hand-written examples)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 17903
Dataset metadata available: Yes
Dataset metadata detail: input setting
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Correlation (Matthew's correlation, Pearson's r)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: we implement cross-validation between different annotators and conduct continuous monitoring to ensure they are maintaining objectivity and fairness.
Taxonomy: LLM as a Judge / 
New taxonomy: General Application / LLM as a Judge