FRABench and UFEval: Unified Fine-grained Evaluation with Task and Aspect Generalization
Authors: Shibo Hong
Jiahao Ying
Haiyuan Liang
Mengdi Zhang
Jun Kuang
Jiazheng Zhang
Yixin Cao
Venue: ICLR 2026 Oral (2026)
Keywords: Aspect-level Evaluation Dataset
Unified Fine-grained Evaluation
Primary area: datasets and benchmarks

Abstract: Evaluating open-ended outputs of Multimodal Large Language Models has become a bottleneck as model capabilities, task diversity, and modality rapidly expand. Existing ``MLLM-as-a-Judge'' evaluators, though promising, remain constrained to specific tasks and aspects (i.e., specific evaluation criteria such as fluency for text and image quality for images). In this paper, we argue that, on one hand, based on the interconnected nature of criteria, learning specific aspects can generalize to unseen aspects; on the other hand, jointly learning to assess multiple visual criteria and tasks may foster a synergistic effect. To this end, we propose UFEval, the first unified fine-grained evaluator with task and aspect generalization for four evaluation tasks --- Natural Language Generation, Image Understanding, Image Generation, and Interleaved Text-and-Image Generation. However, training such a unified evaluator is hindered by the lack of a large-scale, multi-modal, and aspect-level resource. To address this gap, we introduce FRABench, a comprehensive fine-grained evaluation dataset. Specifically, (1) We first construct a hierarchical aspect taxonomy encompassing 112 distinct aspects across the aforementioned four tasks. (2) Based on this taxonomy, we create FRABench, comprising 60.4k pairwise samples with 325k evaluation labels obtained from a combination of human and GPT-4o annotations. (3) Finally, leveraging FRABench, we develop UFEval, a unified fine-grained evaluator. Experiments show that learning on specific aspects enables UFEval to generalize to unseen aspects, and joint learning to assess diverse visual tasks and aspects can lead to substantial mutual benefits.

Published as a conference paper at ICLR 2026
FRABENCH AND UFEVAL: UNIFIED FINE-GRAINED EVALUATION WITH TASK AND ASPECT GENERALIZA-
TION
Shibo Hong"?, Ying', Haiyuan Liang', Mengdi Zhang²
Jum Kuang Jiazheng Zhang', Yixin Cao
College of Computer Science and Artificial Intelligence, Fudan University, ?Meituan 3 Schooo of Computer Science, Singapore Management University
ABSTRACT
Evaluating open-ended outputs of Multimodal Large Language Models has become a bottleneck as model capabilities, task diversity, and modality rapidly expand. evaluators, though promising, remain constrained to specific tasks and aspects (i.e., specific evaluation criteria such as fluency for text and image quality for images). In this paper, we argue that, on one hand, based on the interconnected nature of criteria, learning specific aspects can generalize unseen aspects; on the other hand, jointly learning criteria and tasks may foster a synergistic effect. To this end, we propose UFEval, the first unified fine-grained evaluator with task and aspect generalization for four evaluation tasks - Natural Language Generation, Image Understanding, Image Generation, and Interleaved Text-and-Image Generation. However, training such a unified evaluator is hindered by the lack of sarge-scale, multi-modal, and aspect- level resource. To address this gap, we introduce FRABench, a comprehensive fine-grained evaluation dataset. Specifically, (1) We first construct hierarchical aspect taxonomy encompassing 112 distinct aspects across the aforementioned four tasks. (2) Based on this taxonomy. create FRABench, comprising 60.4k pairwise samples with 325k evaluation labels obtained from a combination of human and GPT-4o annotations. (3) Finally, leveraging FRABench, we develop UFEval, a unified fine-grained evaluator. Experiments show that learning on specific aspects enables UFEval to generalize to unseen aspects, and joint learning to assess diverse visual tasks and aspects can lead to substantial mutual benefits.
Existing HILLM-as-a-Judgee
assess multiple visual
INTRODUCTION
As Multimodal Large Language Models (MLLMs) have shown amazing abilities in human-like question answering (Grattafiori et al. 2024), assessing the quality of their free-form outputs has become increasingly challenging. Automated evaluators, a.k.a. "MLLMM-as---Judgg"" paradigm (Saha et al. 2025) have therefore received much research attention (Li et al. 2024c). Despite their progress, we posit two concerns: (1) Current evaluators (Liu et al., 2023b] Ke et al. 2023) are typically tailored specific aspects, which limits their adaptability to unseen aspects. (2) They also limited to specific tasks and modalities, which sharply constrains their scope of application. Table T shows the detailed comparison with existing evaluators. We intuitively argue that, on one hand, evaluation aspects are inherently interconnected (Fu et al., (023). Specifically, aspects such as engagement, naturalness, and creativity are closely linked. Thus, similar semantics and evaluation standards can be transferred across diverse tasks. On the other hand, jointly learning to assess multiple visual aspects and tasks may foster synergistic effects (Wang al. (0255). For instance, learning object alignment in image captioning improves character consistency evaluation in multi-image scenarios, while progress image understanding enhances image generation evaluation through better judgments of content quality and contextual appropriateness. This cross-aspect and cross-task synergy motivates the development of a unified, fine- graigga evaluator for improved generalization and performance.
"Corresoonding author

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of our UFEval with recent evaluators. NLG, IU, IG, and ITIG represent Natural Language Generation, Image Understanding, Image Generation, and Interleaved Text-and- Image Generation, respectively. The number of aspects shown for each evaluator is comparable in granularity, with the scope covering their respective target domains. indicates that the number of supported aspects is not explicitly specified.
Task
Modalty
Method
Aspect
Generalizable
NLG
IU
IG
ITIG
Text
Image
AUTO-J (Li al.
2023b
X
X
X
V
X
332
X
X-Eval [Luu
20236
X
X
/
27
Promethcus 2 kmm
2024a
X
/
Themis (Hu al. 2024b
V
X
50
ImageReward Xu el al.
2023a
X
X
J
X
X
J
3
X
VewionReward Xu ct al.
2024
X
X
V
X
X
V
37
X
LLaVA-Critic Xiong
2024
X
/
X
X
V
X
X
UFEval (ours)
V
V
112
To this end, we propose UFEval, the first unified fine-grained evaluator with task and aspect general- ization for four evaluation tasks (i.e., NLG, IU, IG, and ITIG) across 28 sub-tasks. However, training such a unified evaluator requires large-scale, multi-modal, aspect-level evaluation datasets, which are currently unavailable. Therefore, we develop the Fine- ane-ggriiedd Aspect Benchmark (FRABench) to address this gap. Specifically, we first conduct a survey on the four tasks to identify key evaluation aspects. We then manually organize, extend, and redefine 112 distinct aspects with hicrarchical relations as universal evaluation taxonomy taxonomyaspped aspect tree. Using this aspect tree, we construct FRABench with comprehensive aspect coverage. The aspect tree guides us in selecting multiple relevant aspects for each of the 60.4k pairwise responses, for which we obtain evaluation labels through hybrid approach combining available human annotations with GPT-to-assisted completions, ensuring cost efficiency. In total, this process yields 325k evaluation labels. Using FRABench, develop UFEval, a unified and fine-gras mee-grained evaluator.
Our experiments show that UFEval exhibits excellent evaluation quality and aspect generalization capabilities. This is attributed to learning multiple aspects and tasks jointly, which yields significant mutual enhancement, unlearning aspects. By conducting ablation experiments across diverse aspects and task baselines, we observe gradual improvement evaluation quality, thereby validating our hypothesis. Additionally, we demonstrate the value of using UFEval for preference data generation to align the outputs of models with human preferences via direct preference optimization (DPO). These results validate the effectiveness of the FRABench as valuable resource for training unified evaluators.
well
fully leveraging the interconnections between learning and
In summary, our contributions are follows: (1) We construct FRABench, large-scale multi- modal aspect-level evaluation dataset to train and test evaluators. (2) Upon FRABench, we develop UFEval, the first unified fine-gras mee-griiedd evaluator for multiple tasks assessment with task and aspect generalization. (3) Our experiments show that inter-aspect correlations enable generalizable capability, and learning to assess multiple visual tasks and aspects jointly leads to a synergistic improvement in evaluation performance, while also demonstrating UFEval's 's value for preference alignment.
RELATED WORK
2.1 COARSE-GRAINER SPECIFIC-TASK EVALUATION
Previously, coarse- grained evaluators, which produce overall judgments based on one or several aspects, have been widely explored across various evaluation tasks (Zhu et al. 2023 Xu et al. 2023b Ye et al.) 2023 Liu et al. [024bb Wang et al. 2023a Jiang et al.] 2023). For instance, in NLG, Wang ct al. 2023b) first introduces PandaLM, a fine-tuned LLM designed to evaluate pairwise texts based on several aspects. Similarly, Li et al. (2023b) proposes Auto-J, an evaluator capable of handling a broader range tasks and aspects. supporting both pointwise and pairwise evaluation settings. In image generation evaluation, Xu et al.] (2023a) develops ImageReward, on a large-scale human annotation dataset. Using alignment and fidelity as reference dimensions, Rewge provides an overall assessment. Despite showing good overall performance, coarse-
evaluator trained
2

---

Published as a conference paper at ICLR 2026
grained evaluations lack the granularity needed to diagnose specific model deficiencies and often introduce aspect bias into the evaluation process.
2.2 FINE-GRAINED SPECIFIC- TASK EVALUATION
To address the limitations of coarse-grained evaluation, recent studies have shifted toward fine-grained
evaluation Ye et al.) 2023: Kim et al.) 2024b; Ke et al., 2023] Kim et al. 2023 Li et al. 2023d
Ying al. 2024] Bai al. 2023). where MLLMs are fine-tuned on multi-aspect datasets to produce aspect-specific judgments. For example, in NLGG, Hu et al. (004b) proposes Themis, an L.M
trained on the GPT-4 annotated NI.G-Eval corpus. In IU, LLaVA-Critic (Xiong et al., 2024) is
the first fine-grained evaluator integrating diverse criteria, showing strong correlation with GPT-4oo For IG, Xu et al. (2024) introduces VisionReward, a VQA-baser evaluator that assesses image quality across fine-graindd aspects. While these methods improve over coarse-grained evaluation by targeting specific aspects, they struggle with scalability across different tasks and aspects. [iu et al. (2023b) further investigates aspect generalization through X-Eval, two-stage learning framework that incorporates auxiliary aspects and demonstrates generalization in NLG. However, its reliance on predefined reference aspects and lack of open-source implementation restricts its reproducibility.
METHODOLOGY
To develop UFEval,, we need to construct a large-scale, multi-modal, and aspect-level evaluation dataset. However, existing datasets predominantly focus on overall quality assessment rather than fine-grained aspect evaluation, limiting the development of such evaluators. To address this gap, we construct FRABench through two main steps: (1) Evaluation Aspect Construction, and (2) Fine-grained Evaluation Dataset Construction. The following sections detail these two steps.
3.1 EVALUATION ASPECT CONSTRUCTION
3.1.1 ASPECT COLLECTION AND EXTENSION.
To fully leverage existing aspects, we first collect 28 sub-tasks under the four types of tasks (i.e., NL.G, IU, IG, and ITTG). These sub-tasks span all six combinations of input types (text and text- with-image) and output types (text, image, and text-with-image), ensuring comprehensive coverage across multimodal tasks. Subsequently, we collect literature related to each sub-task to gather relevant aspects and, where available, their definitions. Moreover, we define aspects without available definitions according to their practical meaning.
Beyond collecting existing aspects, we extend aspects for the ITIG, which lacks sufficient aspects. To this end, we apply a cross-task transfer by identifying analogous sub-tasks from other categories and adapting their aspects. For example, both story generation (NIG) and visual story completion (ITIG) involve narrative creation, enabling aspects like engagingness to be adapted[] Ultimately, we obtain 112 different aspects, with detailed information and sources listed in Appendix A
3.1.2 ASPECT TAXONOMY CONSTRUCTION.
To facilitate aspect selection within the dataset construction and evaluation pipeline, we organize the collected aspects into an aspect tree serving as standardized taxonomy. We first use the "overall" aspect as the root node of the aspect tree, and then divide the remaining aspects into two subtrees based on their task independence and task specificity:
Universal Aspects (UAs): Aspects in this subtree exhibit task independence, as they primarily focus on assessing the quality of the model's output. They are often modality-speciffi (for example, fluency for text or fidelity for image) and serve fundamental quality aspects across tasks.
Task-specific Aspects (TAs): Aspects in this subtree exhibit task dependence, as they primarily focus on task completion rather than output quality. They are typically closely tied to the task type. For example, engagingness for story generation or accuracy for mathematical reasoning.
Aspects sharing the same term but differing in definitions are sequentially marked with superscripts: 1 and
3

---

Published as a conference paper at ICLR 2026
Instruction Input Image Undersunding
Queelion:
there a oat mage next the orange cog picture on the side or the bus?
Responset
No, can image
orange
picture sidee
image
orange dog."
Resporoee
"las, thare image the orange the side
Exergr quality
Aspect Selection
Select via Tisk Property
Chioge
Cal image sedt onurge dog image
Select via Ouiput Modallty
UFEval
TAs Tree
That Modslit
Grummnticlly
(Feedback): Both responses
[ [Feedbank): Based on the given crierion cortext inconsisten- ey, Responsze image
echere well the rules gra- including sentence si- ucture, lense, subject- agrocment, and punctua- ton. Respprre is clear and concise. with no grammatical errore. Resporse 2 diso gr- ammatically correct, with wel- etruclured seriences pro- per punctuation. Both effectively use cescriptive language convey the image of the living room with-out any grammatical mistakes.
orange dag on the side the bus, there is no cat image next Reszonse 1 correctty identhia the abseeno of cal image, wh- lic Response 2 incorrectly stat- that there is imae next the
picture, contradicte contestual Information provided by the image.
Text
UAs Tree
Image
(Result): 5.5
[Result;: 1
Text-with-lmage
Evaluation Result
Figure 1: An illustration of the evaluation pipeline. The pipeline consists of two steps: (1) Aspect First, appropriate aspects are selected from the T'ss and UAs Trees based task property and output modality. As illustrated, the question focuses on "cat image next to the orange dog picture", while the image shows no cat adjacent to the orange dog. This discrepancy enables evaluation of the model's ability to avoid hallucinations, specifically its performance on Context Inconsistency. Second, given the text output modality, aspects are selected from the Text Branch in the UAs Tree. (2) Evaluating: UFEval generates feedback and scores based on the input content and selected aspects.
Selection:
For hierarchical construction within the two subtrees, we prioritized aspect trees from peer-reviewed literature to ensure consistency and robustness. Specifically, we adopted the following structures: Readability (Hu et al., 2024a), Bias (DeAlcala et al. 2023). Instruction Following (Zeng et al.
2023), Hallucination (Huang et al. 2025), Alignment (Huang et al. 2023), Complement (Vempala & Preotiuc-Pietee Pietro 2019), Image Coherence (Liu et al. 2024a). Image Quality (Zhang et al. 2025).
Semantic Consistency (Chen al. 2019) and Toxicity (Gehman et al. (2020).
For the remaining aspects without an established hicrarchical structure, we handle them as follows:
Bidirectional Matching Strategy: First, we check whether the remaining aspect's name appears within the definition of the existing tree's root node. If a match is found, we recursively traverse the child nodes to identify the most specific insertion point and append the aspect as a child of the last matched node. Conversely, if no downward match is found, we verify whether the root node's 's name appears within the definition of the remaining aspect. In this case, the aspect implies a broader concept and is inserted as the parent of the current root. This applies to the following aspects: Clarity in Readability, Semantic Consistency in Image Coherence, and Integration, Complement, and Alignment in the Text-Image Relationship.
Creation of new root nodes: For aspects that do not match any existing tree, established them as independent root nodes to minimize subjective bias and avoid misleading categorizations. These nodes correspond to all green peer-level nodes within the TAs tree in Figure]]
4

---

Published as a conference paper at ICLR 2026
Daluset
Sub-Tusks
Split
Auto-J
Summarization, Creative Writing, Rewniting,
Question Generation. Titk Generation, etc.
FRA FRA-0OD
FTAn//RRAID
2.K/
0.0k
PuRkic
VLFoedback
Image Captinning, Test rich Understunding, dtc.
Medical VQA, Acodemic VQA, etc.
Train ERA ID
PRA-00D
14.3k/2A 244
1.2k
FRABench 60.4k
1G
ImigeRevadDD
GienAt-Resch
Test-lo-nage Ceneration
Text-t- Emage Generation
Train FRA-D
FRA-OOD
GE/1.5k
1.4k
TesrVQ
Text Resvaning
Thain/ / FRAAJD
2.1/044
TU
ChurVQA
Chari Reasoning
Train TRRA-D
2.1k/04k
Ceneratel
InfographisSUUA
Graph Roasoning
FKA-ID
2.1//04
MagicBrash
Image Editing
35k/0.5k
Text-oo Image Goneration
FRA-ID
48k/1k
Visual Story Completion
/RRA-D
5.k/0.7k
wiklHow
Multimodal Seripe Gevetion
Train FRRAJD
64k/0.6k
IntetemesdRench
Stocytallngg Generntion Activilnnc Continuation
PRA-00D
90
Figure 2: Pairwise data statistics of the FRAbench. "Public" indicates data derived from existing public datasets, "Generated" denotes data synthesized through our generation pipeline. Detailed counts of evaluation labels for each sub-task in Appendix (.4] B.4
Based on aspect trees, the fine-grained evaluation process can streamlined, as illustrated in Figure
3.2 FINE-GRAINED EVALUATION DATASET CONSTRUCTION
Based on the aspect tree, we can construct fine-grained evaluation datasets for training and testing UFEval. Specifically, we construct FRABench a large-scale, multi-modal, and aspect-level evalua- tion dataset. Following prior studies (Kim et al. [2024a] Ye et al.) (024) that demonstrate pointwise scoring is more susceptible to contextual bias and available for reward model training, we adopt pairwise comparison evaluation method in FRABench.
3.2.1 FRABENCII CONSTRUCTION.
We first collect questions from the datasets corresponding to the 28 sub-tasks we gathered, and generate pairwise samples. Specifically, we first obtain 29.3K response pairs from public datasets and generate the remaining 30. J.JKK K pairs using different MLLMs (Details regarding the MLLMs used can be found in Appendix B.I). After obtaining the queries and response pairs, we assign aspects from our aspects tree. As result, each pairwise sample is annotated with an average UAs and TAs (Detailed information aspect assignment is provided in Appendix B.44). We then generate evaluation labels via two approaches: (1) Human annotations: directly using threc-aspect human-annotated scores from ImageRewardDB (Xu et al., 2023a), supplemented with feedback generated by GPT-4o; (2) GPT-4o annotations: due to the lack of human annotations for most aspects, we use GPT-4o generate evaluation labels for all other pairwise samples. During the GPT-4o annotation process, we observed that GPT frequently incorporates response correctness into its assessment of UAs. To address this, we provide only the response, excluding the query, when evaluating UAs. All templates are provided in Appendix G Moreover, to mitigate position bias (Ye) al., 2024), we balance the number of samples where response is preferred over response 2 and vice versa by reversing the response positions in half of the surplus samples from the majority class and re-annotating them accordingly (Appendix BB.2 presents detailed analysis about position bias of FRABench). Finally, we generate 325k fine- grained evaluation labels.
3.2.2 DATASET PARTITIONING AND EVALUATOR TRAINING.
Due to the lack of large-scale aspect-level benchmark to verify UFEval's 's aspect generalizable capability and support our arguments, we can only divide FRABench into training, In-Domain test set (FRA-ID), and Oft-an-Dommin test set (FRA-OOD). The training and FRA-ID consist of 18 randomly selected sub-tasks from four tasks, covering 22 UAs and 35 TAs, while FRA-OOD contains 10 unseen sub-tasks with 28 seen UAs and 27 unseen TAs. Finally, the training set contains 255.4K samples, FRA-ID contains 45.2K samples, and FRA-OOD contains 24.4K samples. The statistics of FRABench are presented in Figure with more detailed analysis provided in Appendix [
An evaluator's main job is judge model responses like humans would, their effectiveness depends on how closely their judgments match human opinions. To measure this, we create human- annotated evaluation datasets. Specifically, extract partial samples from FRA-ID and FRA-
5

---

Published as a conference paper at ICLR 2026
MLG
6 mi
GFT-4
GPT-RD
NUG
1 ms
Human
Cwend75-772
Owerdh1-1
Comparison of baselines on the FRA-OOD and FRA-OOD-H. Black areas show unseen TAs (for aspect generalization), while the rest show seen UAs for unseen tasks (for task generalization).
Figure 3:
Table 2: The results correspond to task and aspect generalization evaluation, Average accuracy serves as the evaluation metric. Bold and underline indicate the first and second best results, respectively.
Task Generalization Evaluation
Aspect Gemeralization Evaluation
Method
FRA-0O0 OOD
(GPT4))
FRA OOD
(Human)
FRA-OOD
(GPT4o
FRA FRA-OOD-H
(Human)
NLG
IU
1G
MIG
NLG
IU
IG
ITIG
NLG
IU
1G
ITIG
NLG
I0
IG
ITIG
Claude:3.5
71.6
85.8
72.5
75.6
83.0
76.5
63.1
91.0
84.1
84.3
65.6
85.0
82.6
76.5
65.1
91.0
Qwen2V1-72R
70.2
82.4
65.8
60.0
78.
75.3
48.6
83.7
76.3
81.5
65.6
85.0
77.3
75.3
53.8
83.7
Qwen2VL-7B
Themis
56.7
58.9
55.0
58.8
LLaVA-Citie
52.2
76.2
80.8
76.2
OOD, and recruited three humans with master's degrees for annotation. Finally, retain 6.9K in-domain evaluation samples (FRA-ID-H) and 6.0K out-of-domain evaluation samples (FRA-OOD- H), respectively. Details of the human annotation process and annotation consistency provided in Appendix E When training UFEval, we use SFT to mun-tune -a-n2-VL-7B--n-se on the training set. The detailed training configurations and methods are presented in Appendix C.I
EXPERIMENTS
To assess UFEval's generalizability from inter-aspect correlations, first conduct evaluations previously unseen tasks and aspects, termed Out-off-omaan Evaluation. Next, we evaluate UFEval as MLL.Maas-aJudde using public benchmarks. We further explore the effectiveness of Multi-aspect and Multi-task Assessment Learning for evaluators. Lastly, we validate UFEval's 's application in generating preference data for DPO-based alignment. In the following sections, we present our experimental setup and results (In-Domain Evaluation and other experimental results are in Appendix D)
4.1 EXPERIMENTAL SETUP
4.1.1 BENCHMARKS.
(1) Out-of-Doman Evaluation: we use FRA-OOD and FRA-OOD-H to validate. (2) Evaluation M-L-MMas-a-ludge: select public benchmarks across three tasks: For NL.G, use MT.

---

Published as a conference paper at ICLR 2026
Thema
LLAVA Cr
Dura
HA--H Tuxidiity
HNG 00D-
FAA FA--COH
H0-003M
FRAID
RAA 1A--00-H
Undersoaraabitt
Sulabaity
Semunic Oroidancy Dishinbbllty
PrSSAA
(6) Strong Aspects
(b) Universally Poor Aspects
Wesk Aspects
Figure 4: Comparison of aspect-level. In the subfigures, the x-axis shows the test set (FRA-ID, FRA-ID-H, FRA-OOD, or FRA-OOD-H) and task (NLG, IU, IG, or ITIG) of each evaluated aspect.
Table 3: Evaluation MLLM-as-a-Jude for NLG, using three benchmarks. "tau" indicates that accuracy is calculated with ties, and "diff" excludes tied pairs when calculating accuracy.
SummEval Pluency diff't)
MANS
MTT-Bench
Method
Coherence
Consistency
Relevance
dim(t))
(an(T)
difft?)
tau())
difft
tau(?)
diff())
tau(")
tau(t)
difft?)
tau(()
difft
GPT-46
58.D
64.2
79.1
85.1
64.3
72.8
66.11
67.1
65.3
72.3
68.5
70.9
83.5
Claude-3.5
63.5
70.6
81.6
87.9
73.2
$3.1
59.6
66.6
69.6
77.0
68.4
76.3
90.7
Qwen2VL-72B
72.3
Quen2V1-78
48.8
12.5
35.5
30.6
44.8
49.8
41.7
44,5
42.7
44.3
60.2
44.5
50.1
Vwen3VL-8B
56.5
61.5
60.8
57.4
68.0
75.5
55.4
60.8
60.1
63.8
45.5
70.1
80.9
Themis
60.7
X6.0
73.3
77.7
$4,4
54.6
67.5
70.1
44.2
43.6
37.7
Auto-J
67.0
68.2
73,0
85.5
Promethous 2
55.2
62.1
65.5
74.7
61.6
69.8
55.0
61.8
59.3
67.1
69.0
55.1
72.0
Ours
61.6
71.8
75.2
83.5
74.7
84.5
61.3
67.6
69.0
26.9
69.3
74.9
88.3
Bench (Zheng et al., 2023), SummEval (Fabbri al. 2021), and MANS (Guan et al. 2021). Since
SummEval and MANS only provide scores, we generate pairwise samples for evaluation through sample pairing. For IU, we use WildVision (Lu et al.) 2024). MLLM-as-aaJudge (Chen et al. 2024
and VeRewardBench (Li et al. 2024c). For IG. SenAl-Bench (Li et al. 2024a), Winoground (Trush al., 2022), and Pick-a-Pic KKrstaain et al., 2023) are selected. To ensure fair comparison, we carefully check that the training set contains no overlapping samples with the benchmarks. (3)
Multi-Aspect and Multi-Task Assessment Learning: use FRA-ID to investigate the multi-aspect learning and benchmarks above to verify multi-task synergy. (4) Preference Alignment Comparison: We leverage UFEval for image generation and understanding model alignment. For image generation, generate images using captions from the HPDv2 (Wu et al.) (023).. For image understanding, use MMHal (Sun et al. 2023).
Wild (Li et al. 2024b), and LLaVABen (Liu et al. 2023a).
Detailed descriptions and usage of each benchmark are Appendix C.2
4.1.2 BASELINES.
We experiment with representative models from two categories: (1) Prompting Models: GPT-4o,
Claude-3.5, --en3-VL--BB (Qwen3VL-SB), Qwen2-VL-72B-Instne
(Qwen2VL-72B), and Qwen2-
In-7B-Instrutt (Qwen2VL-7B) implemented using the same prompts UFEval. (2) Fine-tuned
Evaluators: Themis- 8B (Themis) (Hu et al.) 2024b). Auto-J (Li Liet al. 2023b), Prometheus 2 (Kim
al. 2024a). LLaVA-Critic-77 (LLaVA-Critie) (Xiong al. 2024 ImageReward (Xu et al. [023a
Rewar (Xu et al. 2024), Q-Eval (Zhang et al. 2025) and CIGEval (Wang et al. 2025a)
Notably, IewgeReward and RewionReward
different evaluation paradigms from
Auto-J
cannot assess specific aspects, and Prometheus 2 requires reference answers. Therefore, we use these methods only as baselines in the MLLM-as-a-Judee evaluation.
4.2 OUT-OF-DOMAIN EVALUATION
Tasks Generalization Evaluation. For task generalization, employ the seen UAs and unseen tasks from UFEval during testing to control for test variables. The results are shown in the regions outside the black sectors in Figure 3 Detailed scores for each aspect reported in Appendix D.4
We use aspect abbreviations here; the full names he found in Appendix
7

---

Published as a conference paper at ICLR 2026
Table Evaluation as MLLM-as-a-Judge for IU. We evaluate baselines across three benchmarks.
WildVision
MLLM-as--Judde
VLRewardBench
Method
General diff (1)
Hallucination
Reasoning (if
tau (1)
diff (t)
(1)
tao (1)
dif
diff
diff (1)
GPT-4o
55.3
70.1
73.3
58.1
67.0
50.2
81.
74.8
68.8
Claude-3.5
53.3
67.3
61.2
58.4
68.3
38.5
82.6
66.1
62.4
Qwea2VL-72B
50.3
59.6
65.5
54.6
58.5
50.8
75.4
70.7
65.6
Qwea3VL-8B
45.5
LLaVA-Ciiti
55.6
42.0
Qwen2VL-7B
39.2
40.6
25.1
41.3
44.6
45.1
62.8
62.5
56.8
w/ IU.
47.3
60.1
58.3
50.1
59.0
45.6
57.6
68.5
57.2
w/ IU+ITI
$1.2
65.2
64.1
54.1
64.8
46.5
56.9
70.7
58.0
w/ IU+IG.
52.7
67.9
66.0
56.1
66.8
46.4
57.0
71.0
58.1
57.2
67.0
46.4
71.1
Table 5: Evaluation as M.LLM-as-aaJudge for IG. We evaluate baselines across three benchmarks.
GenAI-Bench
Winoground
Pick-n-Pic
Method
(au(t)
difk(t)
Relation
Object di)
Boch
Ave. dift (?)
tau(1)
dif((()
dift' (1)
dift (1)
GPT-4o
55.6
69.5
62.6
73.0
73.0
69.5
54.4
59.2
Clande-3.5
55.6
69.2
53.7
Qwen2VL-72B
49.1
52.6
46.7
60.9
57.6
55,0
38.6
38.3
Qwen3VL-SB
53.0
59.0
54.6
63.6
69.0
62.4
49.2
55.1
VisionReward
$1.0
66.4
60.2
64.1
74.9
66.4
48.9
58.0
ImageReward
48.6
64.9
$4.0
58.2
69.2
60.4
48.8
55.7
CIGEval
30.7
31.5
Q-Eval
48.0
65.5
62.4
51.5
61.5
59.4
55,0
62.2
Qwen2V1L-77
35.8
38.0
33.4
42.5
46.1
40.6
38.1
40.2
w/ IG.
46.6
59.4
53.1
52.0
71.2
58.7
45.8
53.1
w/ IIHTTI
50.1
62.6
56.0
58.2
74,2
62.8
47.1
55.9
w/ IGHl..
52.5
64.5
57.3
58.0
78.5
64.6
49.5
56.9
Ours
53.6
65.5
57.5
59.1
80.7
65.7
50.0
57.3
UFEval demonstrates strong alignment with both GPT-4o and human annotators on all tasks. In the "Tasks Generalization Evaluation" column of Table 2 UFEval achieves overall average accuracies of 85% and 83% on FRA-OOD and FRA-OOD-H, respectively, and outperforms both Themis and LLaVA-Critic all tasks. Supporting the effecctiveness of aspect-level evaluation for cross-task generalization. We also provide several good and bad cases in Appendix H
Aspects Generalization Evaluation. We use the unseen aspects of unseen tasks in FRA-OOD and FRA-OOD-H to evaluate UFEval's generalization to aspects. Specifically, this evaluation encompasses two distinet dimensions: (1) Contextual generalization, which validates whether similar semantics and standards can be transferred across diverse tasks (evaluated using 12 aspects); and (2) Novel aspect generalization, which assesses the capability to handle concepts semantically distinet from the training set (evaluated using 15 aspects). A detailed breakdown of these aspects is provided in Table II Morcover. to support the validity of aspect generalization evaluation, we provide ROUGE L
(Wang et al. 2022) tests on unseen aspects in Appendix B.3
The results are shown in the black sectors of Figure 3 In terms of overall coverage, UFEval outperforms both Themis and VL-Cii-ii the state-of oftte-of-the-rt evaluators in their respective domains. In the right column of Table 2 UFEval continues to demonstrate strong performance on the NLG, IU, IG and ITIG, with high consistency with both GPT-4o and human annotators, achieving overall accuracies of 86.2% and 83.2%, respectively. These results show that even without exposure to unseen aspects during training. UFEval remains effective in evaluation. This is largely due to the naturally related nature aspects, which enables the transfer of similar meanings and standards.
Additionally, sample three types of aspects for aspect-level analysis (Figure from four test sets based on UFEval's performance: (1) Weak aspects, where the best model outperforms UFEval by over 20%; (2) Universally poor aspects, where all evaluators score very low; and (3) Strong aspects, where UFEval excels. Only two weak aspects occur in all test sets, and both are included; representative subsets are selected for the other categories.
8

---

Published as a conference paper at ICLR 2026
mG
OMC
Figure 5: We evaluate multi aeptict learning by training models different data combinations and testing them on three visual tasks (IU, IG, ITIG) in FRA-ID.
Regarding the weak aspects, as shown in Figure 4.. UFEval performs poorly on the harmfulness of IG in FRA-ID, mainly because it tends to classify shadowy or gloomy images as harmful due to heightened sensitivity to elements that may evoke psychological discomfort, In Figure Ap all
evaluators perform poorly fidelity, which requires recognizing fine-graines object features (e.g.. well-formed human facial features) that current models, including GPT-4o, often fail to detect (see Appendix H for more representative samples).
4.3 EVALUATION AS MLLM AS-A -S-A---AAAEUE JUDDE
For NLG Evaluations. The experimental results, shown in Table demonstrate the effectiveness of UFEval in NLG tasks. Specifically, in SummEval and MT-Bench, UFEval outperforms most baselines, with Claude being the sole exception. the MANA benchmark, UFEval achieves the best performance, with an accuracy of 69.3%. These results highlight UFEval's ' strong capability for text understanding evaluation compared domain speciffc models in this field.
The results are presented in Table 4 Additionally. we assess model-level consistency on WildVision using Elo ratings and Kendall's Tau (T) to compare model rankings (Xiong et al.] 2024). The findings demonstrate that UFEival consistently outperforms LLaVA--ritte across all benchmarks. LLaVA-Critie focuses on respective datasets training, whereas UFEval uses joint learn- ing across multiple tasks, which leads relatively superior performance. For more comprehensive MLLM-as-a-Judge results, please refer to Table
For IU Evaluations.
For IG Evaluations. The experimental results are shown in Table 5 Our evaluator outperforms RewarR in all benchmarks, achieving accuracies of 65.5%, 65.7%, and 57.3%, respectively. Even compared to the state-of- ofte--of-the-ar image evaluator VisionReward, the performance gap is mini- mal-only 0.9% lower on GenAI--Bence and 0.3% lower on Winogrounded. These results demonstrate the promising capability of UFEval in the image generation evaluation task.
4.4 MULETI-SPPECT AND MULTI-TASK ASSESSMENT LEARNING
The construction of UFEval is based on our intuitive argument: jointly learning assess multiple visual aspects and tasks may foster synergistic effect. Therefore, we explore the effectiveness of multi-aspect and multi-task learning on the evaluators, respectively. Specifically, we experiment with various training data configurations to train the model, analyzing the influence of multi-aspect and multi-task learning. For instance, for TU tasks, we design three training configurations to investigate the impact of multi-aspect and multi-task training: (1) learning solely on IU aspect-level assessment (w STU,, (2) jointly learning IU and ITIG aspect-level assessment (w /IU+ITIG), and (3) jointly learning IU and IG aspect-level assessment (w /I++GG).
Multi-task Assessment Learning. The results are presented in Table and 5. Our results demonstrate that multi-task learning outperforms single-tast task training in enhancing evaluator performance. For instance, as shown in Table 4, when evaluating IU tasks, models trained jointly on both IU and IG tasks achieve superior overall accuracy compared to those trained exclusively IU. These

---

Published as a conference paper at ICLR 2026
Table Image understanding DPO comparison. We compare our UFEval with LLaVA-Critte for DPO based on LLaVA-Next-7B.
Table 7: Image generation DPO comparison. We evaluate DPO performance using UFEval- generated data versus the Pick-a-Pic dataset.
Methed
MMHal
HaVARen
aAaeee
Method
HPSv2
ImageReward
VisionReward
52.9
SDXI.
28.1
0.80
3.00
w/ LLaVA Critic
2.24
59.0
30.3
w/ Pic
28.7
0.84
3.20
w/ UFEval
2.41
61.4
32.3
w/ UFEval
29.9
0.90
3.27
findings underscore the significant advantages of exploiting shared representations and complementary knowledge across diverse visual tasks, ultimately yielding a more high perforrrnn evaluator.
Multi-aspect Assessment Learning. The results presented in Figure 5 show that learning with multi-aspect from different tasks improves the UFEval's performance across most aspects. For example, in the IG task, incorporating relevant aspects from the IU, such as alignment in caption generation and recognition accuracy, enhances the evaluation of aspects like object alignment IG. Similar improvements are observed in IU and ITIG. These results highlight the benefits of leveraging multi-aspect data to enrich the evaluator's understanding and shared aspect knowledge.
4.5 PREFERENCE ALIGNMENT COMPARISON
To validate the effectiveness of UFEval to generate preference data across both IU and IG tasks, we employ it to construct training data for DPO-based model alignment. The application principle of DPO for model alignment in both IU and IG tasks is comprehensively described in Appendix F
For IU Tasks. Building upon UFEval, use DPO improve the image understanding capabilities of LLa VA-Next-7B (Li et al., (024d). For fair comparison, both UFEval and the LLa LL-VCiCii ues identical image-question pairs sourced from RLHF-V (Yu et al. 2024) and LLALLLL (Sun
et al. 2023) to construct preference data. Ultimately, we construct 15k preference samples. We then LLa - on A100 GPUs using batch size of 2, gradient accumulation steps of 2, learning rate of 5 x 10-7, and set 3, to 0.1. The results are shown in Table 6 UFEval consistently surpasses LLa VA- VA-Critic across all evaluated benchmarks. Notably, it achieves 2.4% improvement on LLa Wild, underscoring its enhanced effectiveness in visual understanding tasks.
train
For IG Tasks. Based on UFEval, we apply DPO to SDXL-Turbo (Podell et al. 2023). conditional diffusion model, to better align its outputs with human preferences without explicit reward modeling. We extract prompt and two corresponding images from Pick-a-Pic and use UFEval to construct training preference data. In total,
construct 14k preference samples for DPO image generation. We then train SDXL- Turbo on 8 A100 GPUs using 3, = 5000 with a batch size of 32 for three epochs. HPSv2 (Wu al., 2023). RewgeReward, and Rewion are used for quality assessment. training on the constructed data using UFEval achieves better performance compared to directly training on the original dataset. The qualitative comparison results are shown in Appendix [ This demonstrates the effectiveness of our approach in refining preference data for improved model alignment in image generation tasks.
The results are shown in Table
5 CONCLUSION, LIMITATION AND FUTURE WORK
This paper proposes UFEval, the first unified fine-grained evaluator with task and aspect generalization. Specifically, we start by building comprehensive aspect tree that leads to the creation of FRABench, a large-scale, multi-modal, and aspect-level evaluation dataset. We then fine-tune an MLLM on FRABench develop UFEval. Our experimental results demonstrate that joint learning across diverse visual tasks and aspects yields significant mutual benefits and generalization capabilities. We also leverage UFEval to automatically construct high-quality preference pair datasets for DPO training to align models' outputs. These results validate the effectiveness of the FRABench as valuable resource for training unified evaluators. However, compared to the other three tasks, UFEval has relatively limited performance in IG tasks. This may primarily be due to existing LMMs' insufficient active visual semantic understanding. In future work, we plan to incorporate video understanding and generation tasks into our evaluation system and add their corresponding aspects to the aspect tree.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This project was supported by the National Natural Science Foundation of China (NSFC) under Grant No. 62576102.
REFERENCES
Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, al. Agentharm: A benchmark for measuring harmfulness of Ilm agents. arXiv preprint arXiv:2410.09024, 2024.
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with aaaaag--moee--s-a----aaaa- Advances in Neural Information Processing Systems, 36:78142-78166 2023.
Anna O Basile, Alexandre Yahi, and Nicholas P Tatonetti. Artificial intelligence for drug toxicity and safety. Trends in pharmacological sciences, 40(9):624-635, 2019.
João Bordalo, Vasco Ramos, Rodrigo Valério, Diogo Glória-Sitva, Yonatan Bitton, Michal Yarom, Idan Szpektor, and Joao Magalhaes. Generating coherent sequences of visual illustrations for real-world manual tasks. arXiv preprint arXiv:2405. 10122, 2024.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1-45, 2024.
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang. Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Milm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning, 2024.
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang. Peiyi Wang, Tianyu Liu, and Baobao Chang. Towards end-to-end embodied decision making via multi-mode large language model: Explorations with gpt4-vision and beyond. arXiv preprint 2100022310002071, 2023.
Shizhe Chen, Bei Liu, Jianlong Fu, Ruihua Song, Qin Jin, Pingping Lin, Xiaoyu Qi, Chunting Wang, and Jin Zhou. Neural storyboard artist: Visualizing stories with coherent image sequences. In Proceedings of the 27th ACM International Conference on Multimedia, pp. 2236 2244,22444444 2019.
Daniel DeAlcala, Ignacio Serna, Aythami Morales, Julian Fierrez, and Javier Ortega- Garcia. Mea- suring bias in ai models: an statistical approach introducing n-sigma. In 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC), pp. 1 1167-1172. IEEE, 2023.
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. Bold: Dataset and metries for measuring biases in open-ended language gener- ation. In Proceedings of the 2021 ACM conference faimess, accountability, and transparency, pp. 862-872, 2021.
Alexander R Fabbri, Wojeiech Kryšciñski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409, 2021.
Jinlan Fu. See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166, 2023.
Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernon- court, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097-1179 2024.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxici- tyPrompts: Evaluating neural toxic degeneration in language models. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3356-336666 Online, November 2020. doi: 10.18653/v1/2000 20.16655112220.fdddinss-mm/ppppp
11

---

Published as a conference paper at ICLR 2026
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focusee framework for evaluating -itatt alignment. Advances in Neural Information Processing Systems, 36: 2023.
Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. arXiv preprint arXiv22212.10015 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arXiv:2407.2 21783, 2024.
Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wenbiao Ding, Xiaoxi Mao, Changjie Fan, and Minlie Huang. Openmeva: A benchmark for evaluating open-ended story generation metrics. arXiv preprint wxXiv:2105.08920, 2021.
Yufei Guo, Muzhe Guo, Juntao Su, Zhou Yang, Mengqiu Zhu, Hongfei Li, Mengyang Qiu, and Shuo Shuo Liu. Bias in large language models: Origin, evaluation, and mitigation. arXiv preprint arXiv:2411. 10091,, 2024.
Zishan Guo, Renren Jin, Chuang Liu, Yufci Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.
2023.
Jeffrey Heinz and William Idsardi. Sentence and word complexity. Science, 333(6040) 2011.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689, 2022.
MD Zakir Hossain, Ferdous Sohel. Mohd Fairuz Shiratuddin, and Hamid Laga. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CsUR), 51()):1-36, 2019.
Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, and Xiaojun Wan. Are Ilm-based evaluators confusing nlg quality criteria? arXiv preprint arXiv:2402. 12055, 2024a.
Xinyu Hu, Li Lin, Mingqi Gao, Xunjian Yin, and Xiaojun Wan. Themis: reference-free nlg evaluation language model with flexibility and interpretability. arXiv preprint arXiv:2406. 18365. 2024b.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compennh:: A compre- hensive benchmark for open-world compositional text-to-image generation. Advances Neural Information Processing Systems, 36:78723-7888 78747, 2023.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55 2025.
Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. Tigerscore: Towards building explainable metric for all text generation tasks. arXiv preprint arXiv:2310.00752, 2023.
Pei Ke, Bosi Wen, Zhuoer Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, et al. Critiquellm: Towards an informative critique generation model for evaluation of large language model generation. arXiv preprint 1rXiv:2311.18777 2023.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Scongjin Shin, Sungdong Kim, James Thome, et al. Prometheus: Inducing fine- tine-grained eval- uation capability in language models. The Twelfth International Conference on Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2026
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint wwxiiv24455015333, 2024a.
Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young- Ho Kim, and Juho Kim. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. In Proceedings of the 2024
CHI Conference on Human Factors Computing Systems, pp. 1-21, 2024b.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for texi-to-image generation. Advances in neural information processing systems, 36:36652-336 -36633, 2023.
Mahnaz Koupaee and William Yang Wang. Wikihow: large scale text summarization dataset. arXiv preprint XrX: 18 810009905, 2018.
Isack Lee and Haebin Seong. Do Ilss have political correctness? analyzing ethical biases and jailbreak vulnerabilities in systems. arXiv preprint arXiv:2410. 13334, 2024.
Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, et al. Genai-bench: Evaluating and improving compositional toxt-to-visual generation. arXiv preprint arXiv:2406.13743, 2024a.
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal ca- pabilities in the wild, May 2024b. URL https://llava vl. gitghb.itooooooo....oeeo.oooottt io/blog/
7024-05-100 100--00-10-11aaa 1lavalvv-------aaa-aaaaaaaaaaaaa next songer- 1ms/
Chunyuan Li. Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36: 28541-28566 2023a.
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general purppse assistants. Foundations and Trends@ in Computer Graphics and Vision, 16(1-2):1-214, 2024c.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. inate-eext-ttterleav::: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.078895 2024d.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao. and Pengfei Liu. Generative judge for evaluating alignment. arXiv preprint arX1v:2310.05470, 2023b.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, al. M it: A large-scale dataset towards multi-modal multilingual instruction tuning. arviv preprint ar arXiv:2306.04387 2023c.
Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yifan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, et al. Vlrewardbench: A challenging benchmark for vision- languaee generative reward models. arXiv preprint 2rXiv:2411. 174451,,,,, 2024c.
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, and Qi Liu. Vlfeedback: A large-scale feedback dataset for large vision-language models alignment. arXiv preprint #tXiv:2410.09421, 2024f.
Qintong Li, Leyang Cui, Lingpeng Kong, and Wei Bi. Exploring the reliability of large language models as customized evaluators for diverse nlp tasks. arXiv preprint arXiv:2310.1977 2023d.
Zhichao Liao, Xiaokun Liu, Wenyu Qin, Qingyu Li, Qiulin Wang. Pengfei Wan, Di Zhang, Long Zeng, and Pingfa Feng. Humanacsexpert: Advancing multi-modalii foundation model for human image aesthetic assessment. arXiy preprint arXiv:2503.2390) 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023a.
13

---

Published as a conference paper at ICLR 2026
Minqian Liu, Ying Shen, Zhiyang Xu, Yixin Cao, Eumah Cho. Vaibhav Kumar, Reza Ghanadan, and Lifu Huang. X-eval: Generalizable multi-aspect text evaluation via augmented instruction tuning with auxiliary evaluation aspects. arXiv preprint urXiv:2311.08777 2023b.
Minqian Liu, Zhiyang Xu, Zihao Lin, Trevor Ashby, Joy Rimchala, Jiaxin Zhang, and Lifu Huang. Holistic evaluation for interleaved text-and-ing -i--and-image generation. arXiv preprint arXiv:2406.1 14643, 2024a.
Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang. Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. Hd-eval: Aligning large language model evaluators through hierarchical criteria decomposition. arXiv preprint arXiv:2402. 15754, 2024b.
Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating vision-language models in the wild with human preferences. arXiv preprint arXiv:2406.1 1069, 2024.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 102444 2022.
Minesh Mathew, Viraj Bagal, Ruben Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. In- fographicvqa. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 1697-1706, 2022.
Birger Moell and Johan Boye. Language complexity mcasurement noisy zero-shot proxy for evaluating IIm performance. arXiv preprint arXiv:2502.1157%, 2025.
Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:211 1111111.9977 109734, 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joc Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, and Tianlu Wang. Learning to plan & reason for evaluation with thinking- minking-llm--ss-ajune llm-as-a-judge. arXiv preprint arXiv:2501.18099 2025.
Emily Sheng, Josh Arool,, Zhou Yu, Kai- Wei Chang, and Nanyun Peng. Revealing persona biases in dialogue systems. arXiv preprint 104.2204 2021.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Marko Smilevski, Ilija Lalkovski, and Gjorgji Madjarov. Stories for ing visual and narrative components. ICT Innovations 2018, Engineering and Life Sciences: 10th International Conference, ICT Innovations 2018, Ohrid, Macedonia, September 20-99,0018, Proceedings 10, pp. 148-159. Springer, 2018.
iequennece by us-
Saku Sugawara, Yusuke Kido, Hikaru Yokono, and Akiko Aizawa. Evaluation metrics for machine reading comprehension: Prerequisite skills and readability. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 806 817, 2017.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.1450 2023.
Maurice Taylor and MW Wahlstrom. Readability as applied to an abe assessment instrument. In PUB DATE Jun 84 NOTE 404p.; For other proceedings. see ED 299 461, CE 062 030-032, and CE 062 037. PUB TYPE Collected Works Conference Proceedings (021) LANGUAGE English; French, pp. 15. ERIC, 1999.
14

---

Published as a conference paper at ICLR 2026
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for lsio-linguiisic composition- ality. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5238-5248, 2022.
Alakananda Vempala and Daniel Preofiuc-Pietro. Categorizing and inferring the relationship between the text and image of twitter posts. In Proceedings of the 57th annual meeting of the Association for Computational Linguistics. pp. 2830-28440 2019.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228 8238, 2024.
Jifang Wang, Yangxue Yangxue, Longyue Wang, Zhenran Xu, Yiyu Wang, Yaowei Wang, Weihua Luo. Kaifu Zhang, Baotian Hu, and Min Zhang. A unified agentic framework for evaluating conditional image generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mo- hammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12626 12646. Association for Compu- tational Linguistics, July 2025a. ISBN 979-8-89176-251-0 doi: 10. 1t/65/////2222.aa-ologgg 620. long. 122g
URL httpt:t::///mm //ac:::///a/l aaahaaaaaathooocco antho wgg.orgg//a...
Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O' O'Brien, Ramakanth Pasumuru, Jane Dwivedi- Yu, and Asli Celikyilmaz. Shepherd: A
Olga Golovneva, Luke Zettlemoyer, Maryam critic for language model generation. arXiv preprint #xXiv2338.04558 2023a.
Weichuan Wang, Zhaoyi Li, Defu Lian, Chen Ma, Linqi Song, and Ying Wei. Mitigating the language mismatch and repetition issues in Iim-based machine translation via model editing. arviv preprint arXiv:2410.07054, 2024.
Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multimodal understanding and generation. arXiv preprint mtXiv25503.05236 2025b.
Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang. Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang. Xing Xie, et al. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. arXiv preprint 1xXx:2306600087 2023b.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint (tt.:2222.10500 2022.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint wrXivv2306609341 2023.
Tianyi Xiong, Xiyao Wang. Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models. arXiv preprint arXiv:2410.02712, 2024.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 11-15903-159333 2023a.
Jiazheng Xu, Yu Huang, Jiale Cheng. Yuanming Yang, Jiajun Xu, Yuan Wang. Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, et al. Visionreward: Fine-grained multi-dimennional human preference learning for image and video generation. arXiv preprint arXiv:2412.21059, 2024.
Wenda Xu, Danqing Wang. Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. Instructscore: Explainable text generation evaluation with finegrained feedback. arXiv preprint arXiv:2305.1 12282 2023b.
Yifan Yao. Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. A survey large language model (Il) security and privacy: The good, the bad, and the ugly. Nigh-Confidence Computing, pp. 100211, 2024.
15

---

Published as a conference paper at ICLR 2026
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang. Pin- Yu Chen, et al. Justice or prejudice? quantifying biases in llm-as-a lmmaa-a-judge. arXiv preprint arXiv:2410.02736, 2024.
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo. James Thome, Juho Kim, and Minjoon Sco. Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprint ar arXiv:2307. 109288 2023.
Jiahao Ying, Yixin Cao, Yushi Bai, Qianru Sun, Bo Wang, Wei Tang, Zhaojun Ding, Yizhe Yang, Xuanjing Huang, and Shuicheng Yan. Automating dataset updates towards reliable and timely evaluation of large language models. Advances in Neural Information Processing Systems, 37: 17106-17132, 2024.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IERE/CVF Conference Computer Vision and Pattern Recognition, pp. 13807-13816, 2024.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. arXiv preprint (xXiv22100006641 2023.
Kai Zhang, Lingho Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guider image editing. Advances in Neural Information Processing Systems, 36:31428-31444 2023a.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang. and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint 1rXiv:2305.10415. 2023b.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306. 17707, 2023c.
Zicheng Zhang, Tengchuan Kou, Shushi Wang. Chunyi Li, Wei Sun, Wei Wang, Xiaoyu Li, Zongyu Wang, Xuezhi Cao, Xiongkuo Min, et al. Q-eval-100k, Evaluating visual quality and alignment level for text-to-vision content. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10621-10631, 2025.
Bo Zhao, Boya Wu, Muyang He, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arXiv preprint (tX.ix23377.008811 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46665 2023.
Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models scalable judges. arXiv preprint arXiv:2310.176311 2023.
A ASPECT SOURCES AND DEFINITIONS
We provide the sources and definitions of aspects under two categories: UAs and TAs, as shown in Tables 13 to 23 For UAs, the "Target" column indicates the applicable output modality: "T' for Text, "I" for Image, and "ITT" for Interleaved Text-with- wmag-- For TAs, the "Sub-Task" column specifies the corresponding task type. Additionally, underscores are used benchmark extended TAs for ITIG. A Abbreviations for all aspects are provided in the table.
B ANALYSIS OF THE FRABENCH
In this section, we provide detailed information and comprehensive analysis about the FRABench
16

---

Published as a conference paper at ICLR 2026
Table 8: The detailed statistical information of the sub-tasks, which is shown in Figure
Task
Sub-Task
Split
Size
Task
Sub-Task
Split Size
Train
218
Train 2.1k
Summarizatiio
Detailed Image Captioning
Test
Test
0.4k
Train
Train 4k
Creative Writing
Robustess--rientte Instructions
Test
0.4k
Train
Train 2.1k
Rewritin
Medical Image Understanding
Test
Test
0.4k
Train
1080
Train 2.1k
General Communication
Text-rih Uaderstanding
Test
Test
0.4k
Train
433
IU (Public)
Train 2.1k
Functional Writing
General Visual Conversation
NLG (Public)
Test
147
Test 0.4k
Train 2.1k
Question Generation
Tost
132
Simple Image Captioning
Test
0.4k
Test
Einhodidd Decision-making
Keywords Extraction
Test
Medical VQA
Test
0.4k
Data Analysis
Tost
170
Academic VQA
Test
0.4k
MTI
Storytelling Generation
Translation
Test
130
(Generated)
Activity Contination
Test
40
Table 9: Different MLLMs used for generating pairwise responses for sub-tasks under the "Generated" column in Figure 2 of the main text.
Task
Sub-Task
Model
Text Reasoning
IU
Chart Reasoning
LLAVA 13B, Qwen2 VL-72B, InstructBLIP Vicuna 13B, Intem Molmo (B-hmo--BBBB
Graph Reasoning
Text to Image Generation
Show-o-1.3B. Seed-X-17B Flux- 12B, Stable-Diffusiion33.. 5-Large Ground Truth
Image Editin
MagicRrush, SEED-X-I7, InsPix2Pix, MGIE, Ground Truth
Visual Story Completion
Multimodal Script Generation
ITIG
miniGPT, GILL, MM-Interleaved, GPT4 Ground Truth
Sturytelling Generation
Activitynet Continsation
B. MORE DETAILED INFORMATION ABOUT THE FRABENCH
We provide the information about sub-tasks for NLG and IU in Table X as this information is not shown in Figure of the main text. For the tasks in the "Generated" column in Figure the main text, due to the absence pairwise data, we use four MLLMs with varying performance levels to generate pairwise responses for each query, illustrated in Table 9
B.2 POSITION BIAS ANALYSIS
provide the counts of evaluation samples the training set, FRA-ID, FRAUAS-OOD, and FRA-OOD, showing cases where Response 1 is better than, equal to, or worse than Response 2, as illustrated in Figures to As shown in Figure ( the four tasks in the training set maintain relatively consistent distribution. In Table I0] also evaluate the position consistency of UFEval on the FRA-ID and FRA-OOD. We perform two inferences on the same sample, swapping the positions of the responses in the second inference, and calculate the average consistency, i.e., whether the results from the two inferences align. These experimental results demonstrate that UFEval achieves higher positional consistency compared other baselines.
B.3 ASPECT DIVERSITY ANALYSIS
Following prior work (Wang et al. 2022 Honovich et al. 2022). we analyze the ROUGE-L distri- bution between aspects the training set and the FRA-00D, respectively. Specifically, sample
17

---

Published as a conference paper at ICLR 2026
Table 10: We calculate the models' position consistency in the FRA-ID and FRA-OOD. Since our evaluator is trained on relatively balanced dataset, it can mitigate the impact of position bias.
FRA-ID
FRA-OOD
Method
NLG
ITIG
NLG
ITIG
GPT-4o
81.4
78.3
81.9
76.3
83.0
75.7
86.0
Claude-3.5
79.9
75.1
77.4
85.5
74.9
72.1
75.4
88.5
Qwen2VL-72B
76.0
69.9
77.8
75.2
76.2
78.1
70.0
86.6
Qwen2VL-7B
29.7
62.4
19.6
24.5
46.5
20.5
LLaVA-Critic
82.7
84.7
Ours
82.1
82.6
75.8
83.8
78.9
88
75.5
89.8
Evalustion Resutts the
Frequency Evaluation Results
FS-O Ulk
1030
(especued Despored
Seepersed Resporue) Propemse2
Repprrac
1930
Mart Burber Compins
1.M4
Mamber
20000
1.00
1030
1300
166
Trsk Categories
Tysk Categoiis
Figure 6: Statistics of pairwise data in the train- ing set.
Figure Statisties of pairwise data in the FRAUAs-OOD.
Trequency Semple
Frequeecy Sumples
1447
Restrned Respresc Resteneet Resporend
Fnspersed Resgeise) Risppess))
Namber
3000
2030
12:0
Bal Categories
Tisk Categwics
Figure Statistics of pairwise data the FRA- ID.
Figure 9: Statistics of pairwise data in the FRA- OOD.
pairs of aspects from the training set and compute their ROUGE-L scores. The overall distribution shown in Figure I0 which indicates that the selected aspects are distinct from one another, con- firming the inclusion of diverse and novel aspects in the training set. Additionally, we measure the ROUGE-L scores between aspects by sampling one aspect from the training set and another from the TRA-OOD. As shown in Figure IT] aspects in the training set and those in the FRA-OOD do not overlap, demonstrating the validity of our out-of-domain evaluations.
B.4 ASPECT SELECTION ACROSS DIFFERENT SUB-TASKS
We provide detailed information the aspects selected for the training set, FRA-ID, FRAUAs-OOD. and FRA-OOD. as presented in Table [2 Additionally, we include statistics regarding the number of evaluation labels used for training and testing in each sub-task. Blue-marked aspects are human-
18

---

Published as a conference paper at ICLR 2026
Distribution of ROUGE-L Scores Among Definitions Seen Aspects
Requeenc
ROUGE Score
Figure 10: Rouge-L score distribution among two randomly sampled aspects from the training set. left-skewed distribution with low values shows that the aspects are diverse.
Distribution of ROUGE-L Scores Between Seen Unseen Aspect Definitions
14
12
10
Requency
2-
0.6
0.8
ROUGE Score
Figure 11: Rouge-L score distribution among a randomly sampled aspect from the training set and an aspect from the FRA-OOD. A left-skewed distribution with low values shows that they do not overlap with each other, hence meaning that the unseen aspect assumption is satisfied.
annotated, while the rest are annotated by GPT-4o. The anman--nnotated data in IG sourced from the ImageRewardDD dataset (Xu et al. 2023a).
19

---

Published as a conference paper at ICLR 2026
C EXPERIMENT DETAIL
C.I FINE-TUNING DETAILS OF UFEVAL
In our experiments, we fine-tune Wien2-VL-7B--Inse on the training dataset using the Llama- Factory framework. We use 64 NVIDIA A100 GPUs, where each GPU processes one sample, and employ gradient accumulation with factor of 2. The total batch size is 128. Additionally, we offload the optimizer parameters the CPU to reduce GPU memory usage. The training process carried out over epochs. We set the learning rate to le-5, with 1 % warmup steps. The learning rate follows a cosine decay schedule, gradually reducing as training progresses. UFEval trained to predict pairwise rankings based on the criteria in the evaluation prompt, and provide detailed justifications for the assigned judgments. Standard cross-entropy loss is applied to both judgments and justifications.
C.2 DETAILED INFORMATION OF PUBLIC BENCHMARKS
We provide detailed descriptions and experimental information for the public benchmarks:
SummEval (Fabbri et al. 2021): SummEval evaluates LLMs text summarization across four aspects:
Fluency, Consistency, Coherence, and Relevance. The dataset contains 100 instructions per aspect, each with responses from 16 different LLMss and corresponding human ratings (totaling 6.4k annotated samples). For testing, we randomly generate comparison pairs from the 16 responses per instruction, creating 3.6k evaluation samples.
MANS (Guan et al. 202T): MANS evaluates LLMs on the story generation task. Each instruction includes 5 different responses with human annotation, totaling 400 samples. For each instruction, we randomly select two responses to form pairwise data, resulting in 400 evaluation samples.
MT-Bench (Zhng et al. 2023): MT-Bench is specifically designed to evaluate LLMs' capability evaluators. The benchmark incorporates pairwise comparison data for 80 questions. Each pairwise data receives multiple human annotations, resulting in total of 3k votes. To ensure high-quality evaluation samples, we select only those cases where all human annotators reach complete consensus, ultimately obtaining 684 evaluation samples.
preference relations among LMM response pairs. Each relation includes a question-image pair and two responses generated by different models, with human-annotadee preference (including ties). For testing, randomly select 2k response pairs from the dataset using the same testing protocol as LLaVA-Critic.
WildVision (Lu et al. 2024): WildVision comprises 1 Ik
MLLM-as-aaJudge (Chen et al., 2024): MLLLMas--aJJdge- establishes a novel framework for assessing how closely model evaluations align with human judgments. The benchmark aggregates triplets) spanning 14 diverse
17k multimodal evaluation instances iamaageinstruccion--sspoo benchmarks and incorporating outputs from 6 different MI..Ms. Through systematic pairwise comparisons conducted by human evaluators, the dataset provides 5,719 carefully annotated judgment cases for analysis.
VLRewardBench (Li et al. 2024e): VLRewardBench is a comprehensive benchmark spanning general multimodal queries, visual hallucination detection, and complex reasoning tasks. Through an Al-assisted annotation pipeline combining sample selection with human verification, it curates 1,250 high-quality examples specifically designed probe model limitations.
GenAT-Bench (Li et al. 2024a): GenAl-Bench is comprehensive benchmark with 1.6k composi- tional prompts to evaluate text-to-visual generation, surpassing the size and difficulty of existing benchmarks. Additionally, it provides over 15k human ratings for multiple aspects to support research vision-language alignment metrics.
Winoground (Thrush et al. 2022): Winoground is a benchmark for evaluating vision-language models' ability to perform lingulinn compositional reasoning- the capacity to understand caption. was
how the meaning of an image changes when the same words are rearranged hand-curated by expert annotators and labeled with fine-grained tags, containing 400 samples.
-Pcc (Kirstain et al. (023): Pickapic is large dataset of text-to-image prompts and real users' preferences over generated images. Authors create a web app that enables text-to-image users to generate images and specify their preferences, which constructs 500 test samples.
20

---

Published as a conference paper at ICLR 2026
Table 11: Aspect generalization performance evaluated across two dimensions: Contextual General- ization and Novel Aspect Generalization.
Sub-Task
Contextual Generalization
Novel Aspect Generalization
Question Generation
Engagingness
Difficulty
Title Generation
Coverage, Appeal
Suitability
Keywords Extraction
Relevance
Representation
Data Analysis
Precision, Format
Translation
Translation Fidelity, Thoroughness
Academic VQA
Correctnes
Expertise, Explainability
Medical VQA
Terminology. Transparency
Embodied Decision
Interpretability. Completion. Riskiness
Scene Alignment, Negatiion Universal, Comparison
Text-to-Image
Storytelling Generation
Originality
Simplification
ActivityNet
Reasonahleness, Fulfillmnnt
HPDv2 (Wu al. y023): HPDv2 is large-scale (798k preference choices / 430k images), well- annotated dataset of human preference choices on images generated by text-to-image generative models. We utilized 400 prompts from the test set included in HPDv2 to generate corresponding images using SDXI--Turbo and subsequently evaluated the generated images using three different quality assessment methods.
MMHal (Sun et al.) 2023): MMHal-Bencch is an evaluation benchmark specifically designed for hallucination in Large Multimodal Models (LMMs). It contains 96 challenging questions based on images from Openlmages, and their corresponding ground-truth answers and image contents.
LLa (Liu et al., 2023ap: I.L.VA-Bencl (in the wild) comprises 60 tasks designed to test visual instruction-following and
capabilities in natural settings. Each task is scored by GPT-4 based the correctness of the model's response relative GPT-4-generated ground truth, with scores ranging from 0 to 1, aggregated across 60 samples.
LLaVABen, Wild (Li et al. 2024b): ILLVA--Benci (Wilder) is an expanded benchmark for evaluat- ing visual chat capabilities of MILMs. It offers compact 120-example for quick evaluation. The dataset covers diverse real-world scenarios, including mathematical problem-solving, image comprehension, code generation, visual AI assistance, and image-based reasoning.
C.3 ASPECTS USED IN THE PUBLIC BENCHMARK
Since UFEval requires specific aspects for evaluation, we carefully assign one evaluation aspect to each benchmark. Specifically, we select suitable aspects from TAs and UAs based on the char- acteristics of each task. For benchmarks involving unseen tasks that may not align with existing aspects, we define new appropriate aspects based on their unique task properties. Tables 24 to 27 present the complete mapping of aspects used by UFEval across all benchmarks, with blue color indicating unseen aspects that were newly defined. To ensure fair comparison, GPT-4o, Claude-3.5, Qwen2VL-7B, and Qwen2VL-72B also follow the same aspect assignments when evaluated.
21

---

Published as a conference paper at ICLR 2026
Table 12: Selecting Aspects and sample counts in the Training Set. FRA-ID. FRAUAs-ODD, and FRA-OOD, Blue-marked aspects are human- -amaan-nnnotted, while the rest are annotated by GPT-4o.
188
Sob-Tack
Arpeet Spiit
Universal Arpect
Speelle-trik Aspect
Size
Sunzarizaiinn
Trais Yout
Grommnccil,, Coherence, Fleeny
Covergge Length Constaiin, Legeer, Conciseness
1.7k 70.4
Rewrling
Ceammoticalyy Francy, Clerry
Accunwyy Doiniing
Creative Wiiting
Traia/ Text
Hambuhres Geamettially. Reaibhllty
Gender Biss, Eihnic Bus. Resional Bizs
Cleatiiyy Ergigiignes, Therre
5.1k/ IK
Fuxctiona Writing
Trais/
Clartyy Cahorrax: Flansy
Bays
Relowaess Euempl Quality
Contestiol Informvtion
3.4.
Communicaiio
General
Trais/ Tox
Gender liax, Filmic Roe, Regional Bins Claity, Coherence, Flasscy
Accuracy, Infomratiun Richnoo Liderstandbillyy
11.41/44
Tritiin Sat FRA-D
Gerenal Viesal Corr
Train Yhee
Carry. Harmfsinoss
Helphinnes, Inamaction Following
Teai-rihh Unal
Hammlonnss, Vesibulary Complesity Clarlty
Accuracy, Instuatio Fellaning
10.5.73
Robusass ociented
Indractionss
Train/ Test
Spelling Accuracy, Strcture Accoracy, Pitvacy Vietations
Instuctioon Incorsieency Contest Incossiazncy
196R/255
Medical Image Und
Train/
Vicubulary Complesty, Biss, Seateace Cumplesity
Accunsy
8 26
Churt Reasoning
Trais/
Acceracy, Iestraction Following
7.3k
Test Reasoting
Trais/ Tex
Acctracy, Iistruction Folloiing
4.3k70.0
Grogh Recsoning
Trais Yout
Accerocy Instruction Fellowing
4.3k 70.k
Trais
Readasiliy,
Relationhipp Algginent
12.5k/24L
Dasiled Image
Coptioning
Trais Yex
Colerence. Flacsey, Clacity Biss
Citee Aligines, Tranuz Alignnees Staye Aligrmen
Cousi. Alignaze,, Action Alignne,, Object, Aljatmeat
Complaenree Anmbun Aligiezi, Spenal Aligmme
27.26/5.2
Coust
Relaistahip
(COCO&D))
Geserution
Trais/ Test
Hannfulness, Ficelit
Action Aliginett, Color Aliganict, Shape Aägascent
Objectt A Aligmmen Spriti Aligmmmi, Aligymont
Trauee Aligman Aumboo Aligmmntt
29.2/75k
Image tolning
Accunay
Visual Story
Competton
Train Yiua
Scene Consisencyy Blas, Toccity, Character Consisency
stylskk Consinency Actor Consatency, Floency
Cwotivity Engagingress Completrrees
W.8k 18.87777
Mulimotal Scrigt
Geserution
Train Yest
Readibility, Settarii Cossitency, Styäslic Camisreny
Actas Casssseasy, Cohoccast, laaze Repettion
Conpleteaes, Fesibilit
Helppaliess, Suleey
40.5K/0.24
Sunsarization
Test
Spelling Accnacy Smacturo Accuraty Tont Repeitivn
Complesity Voculslarry Complesis, Reulabilty
1.3k
Rewsiias
Test
Stnwture Accuncy, Yest Repestion, Readabiiig
Tame Canisicaay, Spellnn Assuracs
0.5%
Creative Wriligg
Test
Serrence Camplesity Vocabulary Complesity Celtural
Cahenenes Flacray, Cheiiy, Repettionn
3.4L
FRAMA oo1
Funciinaal Writing
Text
Rrasbhilly, Spellin Acceracy Smurnee Arcuracy
Scuteace Complesity Vocdbalary Compleit
2.1k
Commuticaiion
Test
Restabilty, Hamdiltess Semeece Complesity
Complsin. Grmmaicsii
5.4k
Geseral Visudl Cour
Test
Floency Colerencee Grammniicclty Readabilty
1.68
Teck-nch Und
Trei
Fluncyy Oxhennos Gammerrallt RostiNilty
Instrctions
Test
Readabiiny,
Colercas
1.6
Mediall Image Und
Teat
Fhuncy. Oihenmes Reslabiiiy Gommetraiiy
1ikk
Simspe linuee
Coppioning
Test
Flaeney, Clasit, Complesity Vasiilary Complesty
Cuberencs
2.4L
Detailai Image
Copticinng
Test
Sentiace Complesity Vocdialary Complexty
Reulkilly, Gramnnicalyy
1GLL
Viasd Story
Compiettion
Text
lingge Repetition, Semattic Coosstoncy
Resibinity, Ceberraas Gramatically
3,8L
Matinndaal Soripi
Gecerution
Text
Sceie Cousioescy, Clarry, Insge Coterence
Commuticiity Fuuncy
2.4k
Question Gereration
Test
Clurity, Grammaiicalt, Fluency, Cohcrence
Bios, Simnre Accrary Speling Accerwy
Engagingness. Diftcclly
1.1k
Thie Generation
Test
Coherence, Hiency Geammatiallly Cunural Bias
Ricalailiy, Bins Eibical Bias
Suiubiiizy, Coverage
0.4L
Keywreds Extrattinn
Text
Clarty Flacncy, Coberznce
Rrasbeilty, Grmmesizally
Relesanee Represcstatin
0.5
Analyss
Test
Vecabtiary Complesiny,
Privacy
Rradshillty Complesit
Procison, Formst
1.5
FRA OoD
Tranaleivee
Test
Stmctree Accuracs Spelling Accuracy Teme Consetency
Flacsey, Gramutticalt, Claits, Coherease
Tranchriu Fidcity, Thonosghress
1.lk
Acsdemic VQA
Teret
Fleency, Readablity, Vocubulary Complesty
Complesity
Popertno, Explairahilty
56k
Medical VQJ
Test
Senteace Complesity Bealakiliy, Vocabllary Complesit
Rlanyy, Cotenenes, Cnmmmesolity Cleiy
Tennisriogy, Trampprency
3d6
Enreedied
Test
Cusertace Clarity, Grammatioaity Gender Hennhinss, Reabbility Regiotal Tookeit
lntepueubillyy Completion, Kiskiness
Coust Aligamenn, Onject Relatienaiip Some Aligamerr, Negaion, Universs of Object Aligameni, Spatial Alismment, Alienment Actros Alignmeti, Aliganentt Stuge Aligament Aligmmeii Aligemen Comperton
Test to to-aag-
Geseration
Text
Harfidnos, Fiddio
6.7
Activtyaet Commintton
Test
Stylitic Condetengy Action Consitenyy Roercy, Clrity Sucae Cinesiatey, lmnge Repectionn Chandte Consiieray
Realabilit, Hamiileess Caberano: Scruntic Conealensy
Ressenbleass, Puttilmerr
U.5k
Sterwelling
Readahilly, Humiunnss, Scete Consistecyy Image Coheresce Cansstercyy Fuency, Ceherence
Originlit, Simpliication
0m
Image Repeliton, Chancter
22

---

Published as a conference paper at ICLR 2026
Table 13: The definition and source of Universal Aspects (UAs). (Part 1)
Aspect
Abbr
Source
Target
Definition
Bias evaluates the presence and extent of gender, regional, and cultural bias generated content. It examines whether content reinforces stereolypes pes, shows favoritism or discrtmination loward specific regions. reflects racial or ethnic prejudice. The ussessmen focuses on lunguage, tonc, and represcatation, aiming to identity harmful biases or imbalanced purtrayals.
Gallegos
al.
Guo ot
al.
Bias
Bias
X024
Chang al.
Guo
X024
This criterion evaluates whether generated content reinforces tra- ditional gender stereotypes or presents biased views hased on gender. It examines whether certain roles, traits, behaviors unfairly snciated with specific genders, such as linking nurturing to women or lendership men.
Goo ct al.
2024
Gender Blas
GenB
Dhamala
Sheng at al. 2021
This critetion evaluates the model's is understanding of and respect for cultural diversity. It assesses whether the generated text appropriately acknowledges cultural differenees and avoids reinforcing harmful gencr- alizations or stereolypes about specific cultures. For instance, models may misinterpret culturally specific idioms or expressions, recognize regional language variants, which can result in outputs that lack cultural sensitivity or misrepresent costain commanitics.
Cultural Bias
CalB
(Guo al. 2022
This criterion evaluates whether the generated content shows a bius toward or against any purticular nationality region. ensures that national or regional identities are treated fairly, without favoritism, discrimination, or negative stereotyping.
Regional Bins
RegB
Glao et al
2014
Sheng et a 2021
This criterion asesses wheiher the generated content redlects racial othnie biases, cither by perpotuating stcreotypet pes or marginalizing certain groups, It looks for whether certain ethnic racial groups portrayed negatively or unfairly advantaged. reinforcing harmful bisses.
Lce & Scong
Frhnic Bins
FrhB
Dhamala al. 202T:
Sheng et
This criterion evaluates whether the model avoids generating harmful, violeni. explicit, or inappropriate content. ensures the system does not produce muterial that could cause emotional harm, violate legal murms, infringe privacy. includes preventing the generation discriminatory. NSFW, or harmful language and images.
Basile et al
2019
Toxicity
Tox
Chang et mi. 213
Goo al.
This criterion evaluates whether the generated content promotes, glo- rifies, or depicts any form of violence, abuse, or hamm-e it physical, emotional. psychological. includes triger trauma, or encourage harmful belavior toward individuals or groups, such as self-harm, harm to others, or dangerous actions. Images that cause psychological discomfort when seen are harmful.
Andriusschhenko
Harmfulness
Har
(2024
that may
distress,
CT al. 20044
Guo
ct al. 20223
This criterion evaluates whether the generated content reveals persomal. private, sensitive information olating privacy norms or laws. This includes disclosing privatc details such names, locations, health conditions, other confidential infor- mation. The content should respect an individual's privacy rights, in accordance with legal and othical standards.
individuals without consent,
Yao et al.
2014
Privacy Violation
Priv
C hhnng et nll
21144
Guo
Readability measures how casily a text is understood by its intended audience. It focuses on overall fluency and the clarity of ideas to ensure they are presented simply and align with the readers' comprehension levels. It evaluates the quality of hoth inter sentences and intra-sentences, checking
Taylor
x
Wahlsirom
Readahility
Rea
Hu et al
2024n in
Sugawara
grammatically correct, naturally written, with cleur meanings, and good costext-rlateeness and coherence, allowing readers to absorh information effonlessly.
2017
Coherence refers the logical consistency and interconnection of ideas within the text. It ensures that the text follows a clear, organized pro- gression, each section logically linking allows the reader to easily follow the author's argument or narrative, with smooth transitions and no sudden shifts lopic gaps reaseming.
Coherence
Coh
Liu al 2013b Hu
coherent text
al. 2024. 2
Clarity meusures how easily the text can be understood. It emphusizes the use of straightforward, unambiguous language, vocabolary and convoluted sentence structures that could confuse the reader.
Clarity
Cla
Hu et al. 2b24b Li
complex
Text repetition refers to the unnecessary duplication phrases, sen- tences, or idcas that do not add new maaaning Repetition discupts the flow, roduces clarity, and can make the text feel redundannt.
Text Repetition
Texk
[l et al.
2024b;
T
Wang al. 20 204
Moeli
& Boye.
This criterion evaluates how sentence structures affect readability. Com- plex or overly intricate simpler structures enhunce clarity.
Sentence Complexity
SenC
&c Id.
may hinder comprehension, while
2011
Hu
al
202445
Fluency measures the smoothness of the language and sentence construc- tion, casaring the text flows scamlessly and is grammatically correct. The fluent text flows naturally, uses familiar phrasing, and avoids awk- or forced transitions.
Fluency
Flu
ct
al,
20230
Liu
ct
20233
Hu
This criterion evaluates whether the vocabulary used in the text is well. suited turget audience's 'ss comprehension level. language should prioritize clarity by avoiding overly comples or obscure terminology that may hinder understandingg while also ensuring is excessively simplistic for the inteaded readers.
Moell & Boye.
Vocabulary Complexity
VieC
MTL5I
[Hemz Re ldt
2011
Grammaticality measures how well the text adberes to the rulos of gram- mar, including sentence structure, verb tense, subject-verb agreement, and punctuation.
Grammaticality
Gra
Hu ct 2024h
23

---

Published as a conference paper at ICLR 2026
Table 14: The definition and source of Universal Aspects (UAs). (Part 2)
Aspect
Abbr
Source
Targel
Delinition
This criterion assesses the comectness of spelling in the text, ensaring there are no typographical eroons or misspelled words.
Spelling Accuracy
SpeA
Hu ct al. 2024b 5
T
This criterion ussesses whether the sentenze maintuins consistent tense usago (past. present, future) throughout tense should remain uni- form unless a change contextually required, such as when indicating a change the time frame.
Tense Consistency
TenC
Hu ct al. 2024b
T
This criterion evaluates the grammatical necuracy of sentences, focusing on the pruper alignment of subject verb object and the correct linkage of main and subordinate clauses. It ensures that sentences are constructed following standard grammattcaal rules, which enliances clarity and read- ability.
Structure Accuracy
StrA
Hu et al. 2024b
T
Image quality primarily evaluates the fidelity of the generated image, ensuring it accurately represents described objects and their chur- acteristics. It also assesses whether the image contains any harmful elements, such as offeusive content, misleading sepresentations, harm- ful stereotypes.
Image Quality
ImaQ
Lau et al. 20124 Xm
20250 2014
Fidelity evaluates how accurately the generated image represents the shupe, characteristics, and behavior of the objects described, ensaring they align with real-world expectations. The image should reflect the correct features and proportions of the objects, based on the description. without deviating from reality. For example. if the description mentions a 'spider', image should show with eight logs, and the describes 'unicom', it should feature one horn. Fidelity ensures that the image matches the expected physical attributes and behavior of the objects they appear in the real world.
Fidelity
Fid
Xn et ml.
202
2024
Acsthetic measures how pleasing and emotionally striking image checks if image combines elements like color and light to create a visually appealing experience that resonates with viewers. This measure helps assess whether image captures heauty and evokes an emational response, making memorable and impactful.
Aesthetic
Acs
Liao et al. 2025 Xu
2024
Image coherence measures consistency style and subject represen- tation across images. This includes textures, color palettes, lighting, and rendering styles, and semantic consistency, which covers consistency of physical attribates, clothing, behavioral trits, and scenes. Image coherence also penalizes image duplication, where the output images are too similar to each other or contain near coar-duplicate content within the same outpur.
Image Coherence
ImaC
Jin et al. 2024n
mTT
This criterion ponalizes the sepetition or exeessive similarity between images. It identifies cases where maltiple images are too similar, either identical content, framing. or visual elements, without adding meaningful variety to the overall set.
Image Repetition
IngR
Liu
20244
ITI
This criterion evaluates uniformity of artistic style across different images. It requires that all images share the same visual style, such as all realistic photographs, all cartoon illustrations, example, one image is a cartoon cartoon-style illustration, all related images should follow the same cartoon style, rather than mixing different styles realism, painting, animation.
Stylistic Consistency
StyC
al. 20244
ITI
all oil paintings. For
This criterion evaluates the consistency of semantic elements in the images across a series. These semantic elements include object artributes. scenes, etc. It ensures that each element within the sequence aligns with the overarching theme purpose, promoting clear and uninied narrative or instructional flow. For example, cooking tutorial showing step step-hy-stee step cake prepuration maintains coherence by following the logical sequence of the recipe, whereas mixing unrelated steps disrupts the flow and confuses the viewer.
Liu al.
2024a
Semantic Consistoucy
SemC
Berdalo ot al
ITI
C Chen al. 2019
This criterion ensures that objects or characters maintain the same at- tributes (e.g.. color, shape, size) across different images. It checkss that the physical churacteristiis prevents incomsistencies, such as a charucter suddenly wearing a different or appearance without explanation.
Li al. 20
Churacter Consistency
ChaC
Bontalo
ITI
character object remuin constant. This
Chen et al. 2019
outfi or object changing
Action Consistency in image sequences casures that actions depicted multiple images follow logical progression and match activity und intent. This helps maintain a smouth nurrative flow; such as a person going from jogging running, rather than abruptly switching unrelated action like swimming.
(Lu a 2024m
Action Consistency
AaC
Bantalo et al 2024
ITI
Chen al. 2019
This criterion evaluates how offectively sequence of images maintains the consistency of scene elements, such location and hackground. For instance, in a photo story about day at the farmer's 's market, ruch image should consistently depict market inommeent from dawn dusk. If a photo suddenly shifts to a heach setting, dismuts the scene consistency, confusing the viewer and breaking the nurrative flow.
Nordalo et al 202
Scene Consistency
SeeC
III
Chen al.
24

---

Published as a conference paper at ICLR 2026
Table 15: The definition and source of Universal Aspects (UAs). (Part 3)
Aspect
Abbr
Source
Target
Defuitiin
Text-Image Relationship refers text and image work together to enhance euch other. This includes how the text describes what's is shown image- objects, their properties, spatial relationships and also two additional aspects: Complement, which captures how the text adds extra context or back information beyond mere description enrich meaning, and Integration, which evallates coherently and naturally the text and image are combincd into a unified whole. Effective text-image alignment
Text-Inage Relationship
T-IR
(La al. 2024
ITI
text only describes the image but also, through C Complement and Integration. cariches its overall meaning.
Aliynment between text and image is evaluated based how well the image matches the details in text. This includes the correct objects. their attributes (e. color, size). spatial relationships (e. left, right), and actions. The alignment should consider multiple dimensions. For example, the text descrihes "twp hoys wearing white clothes, one on the left and on the right, playing badminton," the image should show two hoys white clothes, placed correctly in the scene, with a badminton game happening betweon them.
[Xiong al. 20241
Aligament
Ali
Ghosc ct A1241
Huang
III
Xu cl. 202.0
Object Relationship evaluates accurately the spatial relationships and actions between objects described in the text are represented in the image. It facuses whether interactions, positions, and movements of objects in the image match those described in the text. For example, in the description 'a person standing next table and reaching for book', the image should show the person next the table with hand extended toward the book. As long as the spatial relationships and actions correctly represeuted, the image will be considered accurate.
Object Relationship
OhjR
Huang et a.l. 2023
Li et
2024a
mTI
This critetion measures how accurately the spatial relationships between objects in the generated curpur match the descriptions in the text. It evaluates whether the relative positions, orientations (up, down, left. right), and arrangements of objects align with the textual deseription. For example, the text mentions chair to the left of table, the image should show the chair on left side the table, and person is sitting on chair in front of table, this relationship should also he reflected accurately the image.
Gokhale al. 2022
Sputial Aligmment
SpuA
Ghosh et al M
2014g
ITI
Action Aligoment evaluates bow accuratoly the actions or intoractions desorihed in the text represented in the image. specifically dynumic actions like 'running', 'talking'. or 'holding'. and does not consider static descriptions spatial relationships (e (e.g.. left, right, or abject urrangements). For example, a ball', the image should show the dog running after the ball.
Li et al
2024a
Action Alignment
ActA
Huang et al. 2023
III
text says dog running after
Count Alignment evaluntes how accurately the number of objects de- scribed in the text matches both the numher and type of objects presented in the corresponding image. It specifically measures whether the quan- tity objects mentioned the text (e.g. two dogs, apples) is correctly reflected in the image, and casures that the objects shown
a 2024n
cars, five
Count Alignment
CooA
Huang ot al 2026 ITI
Ghosh
type. For example. the mentions 'two dugs' the image should contain two dugs, not two cats. This evaluation focuses solely on the count and type of objects, without considering their attributes or positions.
the
Object Alignment evaluates how accurately the objects descrihed in represented
image. The objects mentioned the should be accumtely presented in the image, with no omissions. For example, in the deseription A man sitting by himself at the dinner table'. the image should feature score, Missing or extra objocts in the image would lower the score.
Object Alignment
ObjA
Glosh et al 2013
Li et al
21124n
ITII
man and dinner
to achieve high
the setting or buckground described in a text is accurately represented in an image. focuses elements such location, weather conditions, time of day, and general atmosphere rather than specific objects. For example, text describes 'a sunny envirunment with historical architecture around town square', the image should depict this specific setting, showing. historical architecture simuted within town square context, not isnlated an open field desert.
Alignment assesses extess
Scene Al Alignment
SenA
aLictal 20044
Autrbute Aligament assesses how well the attributes (such as color, texture, size, and shape) of objects deseribed in the text are reflected in the image. ensures that key features like color and match what's described. For instance, if the text says "a green bicyale." the image should show bicycle that is green. also checks details like texture and shape are accurate. like "smooth, smooth and red. Funthermore, the relative size and proportions of objects should consistent with the description. Overall, image shoold visually match the text's description of each object's attributes.
Li et al 202Am
Attribute Alignment
AtiA
Honng a 2022
ITI
Ghosh et al. 2016]
being depicted as
This criterion corresponding objects in the image based solely on the text description, or whether the colors of the objects conform to realistic coloes. If the specifies the color image reflects that color. For example, if the text says 'a blue car'. the image should show car that matches the described color, of different color.
whether the colors described in the text mutch the
Li el al 202A
Color Alignment
ColA
Hunng al. 2023
ITI
object, the evaluntion checks whether
Ghosh et al. 2016
25

---

Published as a conference paper at ICLR 2026
Table 16: The definition and source Universal Aspects (UAs). (Part 4)
Aspect
Abbr
Source
Target
Definition
This criterion evaluaaes how accurately the shapes described in the text correspend shapes of objects in the image, based solely on texts shape descriptors, or whether the shape of the objects conforms realistic shape. The text may include common shape terms such square, round, rectangular, oval, triangular, cubic, cylindrical, spherical. und other basic or specialized shapes like pyramidal, conical, pentugonal, teardmo. crescent, and diamond.
Li et ol
2024a
Shape Alignaent
ShaA
Houng
20233
ITI
Giersh
1723
This criterion evaluates how accurately the textures described in the text reflected in the image, based solely on the text description, whether textures
objects conform to realistic textures. includes surface qualities such as smoothicss, rouglinsss or granularity, as well material types like wood, plastic, and rubber. Additionally, may encompasa tactile sensations or patterns, such as silky, polished, or other material based description. The accuracy of the texture description is comparing the textual deseription the actual appeurance and surface properties of the objects in the imuge.
Ci ct ol 2014-
Texture Alignment
TexA
Houng al.
22123
ITI
Gloos ot 2023
Advunced Alignment is comprehemsive evaluation aspect that ability to accunately represent complex textual descriptions in image. It cncapsulates Comparisen. Negation, and Universality aspocts. This
checks whether the image correctlyy depicts comparisons between catities based attributes such number volumce represents ubsence or contradiction of elements indicated by words like 'no' or "not'; and illustrates scenarios specific attribute or relation.
Advanced Alignment
AdvA
2024a
every member group
Compurison evaluates the ability accurately depict compurisons be- tween entities described text. assesses whether image correctlly represents characteristics such as number, attributes, area volume of objects their comparative relations. For example, if describes "between the two cups on the desk, the taller one holds more coffee than the shorter one, which is half-empty' cups with
Comparison
Cmpa
al. 2024a
should show two differing amounts coffee- specifically illustrating that the taller cup contains more coffee and that the shorter cup is only half full.
heights
Negation evaluates the ubility to accurately depict the abseace con- tradiction of elements described in a text. It ussesses whether the image correctly represents scenarios where entities are
Negation
Nega
ct 2024a
indicated by words such as 'no', 'not' or wwithutt" For example, text describes 'a bookshelf with no books, only picture frames', it should bookshelf
of books filled with picture frames instead.
Universal evaluates the ability to accurstcly depict secnarios where every member group
specific attribute involved relation, as deseribed in a texi. assesses whether the image correctly represents situations indicated by words such as 'every' 'all", 'each' "both". For example, chef is preparing a dish', it should show all chefs within the kitchen actively engaged in cooking.
Universal
Univ
Li et al. 2024a
ITI
describes bustling kiichen where every
This criterion assesses how text image work together to cuhance or clarify the overall meaning. It evallntes how they complement each other, whetlier through redundancy, emotional amplification, added context, interpretation, creating a ticher understanding than either could provide alonc.
Complement
Comp
Vempula Preofinc 2019)
This criterion captures cases where the image and text redundantly rop- resent the same information. and the image does not contribute anything beyond what already stated the text. The alone is sufficient to convey the care message. For example. a caption saying "Tacos are the best" can he paired with a photo of tacos. Both the image and text communicate the same idea with additional contextual contribaton from the image.
Complementary Overlap
ComO
Pictro 2019
Vempala Precjius
ITI
This criterion evaluates how the image, particularly through elements like emojis or memes, euhanecs or clarities the emotional tooc the text. The image should convey emotions, moods, or subile feelings that text alone might not fully express. This could involve using visual elements such as facial expressions, symbols, or humorous images that directly amplify, complement, or interpret the emational content of the text, helping create more
Vempula Preojiuc 2019)
Emotional Complemeent
EmeC
relatable emotional context.
This criterion assesses bow the text offers commentary or evaluation that adds meaning the image. The may provie interpeetation, critique, or personal perspactive on the image, offering a deeper under standing or a subjective view of what the image represents. The text can either aflicm challenge the implied meaning of the image, creating more munceed interaction between the two modalities
Comment Complement
ComC
Vempala Preojuu Pierra 2019
ITI
This criterion evaluates bow the and the image provide comple- mentary background infocmation contextual details that eabance the understanding of each other. The text should add context, explanations, relevant background in information that helps interpret image fully, or conversely, lly the image should provide visual context that clarifies or enriches the meaning of the text.
Contextual Complemeut
ConC
Vempula & Preojuuc Picna
ITI
26

---

Published as a conference paper at ICLR 2026
Table 17: The definition and source of Task-specific Aspects (TAs). (Part 1)
Aspect
Abbr
Source
Sub-Task
Definition
The summary should cover all csscutial points from the original text, ensuring signifcantt information conitted. this case, completeness is more important than brevity.
Coverage
Cov
Summarization
This measures the model's ability respect any given length restrictions for the response (e.g) g.g. sentence count, paragraph count, specific char- acter limits), eosuring the model does not generate an overty short or exocasively long response. If the instruction does mention length requinement, responses should receive score 1,, without consider- ing the influence of other criteria.
Length Constraint
LenC
Li all 20286
Summarization
The written summary encouraged follow clear and well organized layout. You may use headings, formatting devices to improve readability. This criterion is not applicable the does explicilly require specific layout.
Layout
Lay
Li ct al. 20236
Summarication
points, liss,, tables,
This criterion evaluates whether summary effectively capturing the esscitial information of the original content while avoiding unnecessary details excessive claboration. It focuses on ability convey key elements elearly and concisely, maintaining the core meaning of the text without including irrelevant or redundant content.
the point.
Conciseness
Con
Ci 20236
Summarization
Consider whether the generated provides valuable insights, tional context, or relevant information that contributes postivvely to the user's comprehension of the image. Asses whether the model accurately Follows any specific instructions or guidelincs provided Evalunte the overall contribution the response to the user experience.
Helpfulness
Hel
PTZAR
et al
Lu
General Visual Conversation
prompt.
Instruction Following is detined the shility accurately interpret and exccuse
given instraction, focusing solely providing the correct answer. measures whether the response directly addresssss the task's core requirements, strictly adheres to any specified comstraints response length format, and essary additiional background information, explamtions, or tone beyond what is requested.
outlined
Z Lrng et al
Genemll Visual Comversatio
Instruction Following
InsF
202
introduace unnoo-
Transpurency criterion used assess cleasly transpur cuill model explains its reasoning behind the answer to an academic question. focuses on the model's ability to provide undestandible, logical, and well sell-suuported justifications for its responses. This includes breaking down complex ideas, referencing relevant information. and demonstrat-- ing step-by-step thought process. A high level of explainability ensures that only receive correct answer but also understand how the answer was derived, helping to build trust the model's is reasoning and enhancing leamning outcomes.
Transparency
Tran
Zhang al.
20236
Medical VQA
Terminology evallates model's ahility correctly and apply medical terminology when responding to queries related to medical image understanding This includes the accurate usage of terms specific analomy, pathology, imaging techniques, diagnostic processes. For example, user model should mention terms like 'pulmonary nodule.' "adiigraphic features," or 'lesion charadtrization rather than generic terms like 'spot' or 'image area." Proper use medical terms indicates model's depthh of understanding in the field.
Terminology
Temm
Zhang et al. 20216
Medical VQA
a scan showing lung
Accuracy Rewriting refers to how correctly the changes follow the given instructions. Any modifications are made according requirements.
Accuracy
Acc!
Li
Rewriting
task
This criterion evaluates the clarity and transpurency of corrections made Instead simply returning corrected version, correction process musi he elearly outlined. This includes pointing out which parts of the text were changed, explaining how the changes were made, and correction.
Pointing Out
PoiO
e al. 3023
Rewriring
providing reason for understand not only the mistake but also the logic behind the correction. facilitating better learning and comprehension.
appreach hilps the oser
critetion ovaluaies whether the writlen from the original in terms its appearance or structure. The text should not simply he direct copy of the original; instend, should reflect macniigful changes, such rephrasing reformating. The goal ensure that the new text is distinct and provides added value, ruther than just repeating the same content in the same form.
difference
Difference
Dif
Ci 2023b 2.223
Rewiting
writing. It evalustes whether the content presents fresh ideas, unique expressions, and an inventive upproach. Creativity involves not only the ideas bur also how the language, structure, and emotions used. It looks for original- ity that breaks from the norm, offering something now and engaging the reader.
Cicativity refers
originality
imagination
Creative Writing
Creativity
Cre
Li 2023b
Engagingness refers the extent to which text, whether creative writing, captures and maintains the audience's 's interest. This criterion evaluates the sbility of the content to stimulate compelling exchange of ideas, emotions, information, encoumging active participation and sustained engagement. measures how well the narrative or dialogue keeps the audience engaged and thought- provoking throughout.
Creative Writing
Engaggnggees
Bing
Li 2023b
27

---

Published as a conference paper at ICLR 2026
Table 18: The definition and source of Tpsk-specific Aspects (TAs). (Part 2)
Aspect
Abbr
Source
Sub-Task
DeGuition
This criterion evaluates whether the response maintains clear and consistent central theme subjectt malter throughout the creative work, such song
wminng. example, the song around romantic breakup, every vense chocus reflect and support this theme, creating a cothesive narraiive The goal is to ensure thes all elements of the work work together to reinforce the main idea, taclping audience casily follow connect message heing conveyed.
Theme
The
Li 20236
Creative Writing
consistently
Relevance measures bow closely the content of a written piece aligas with the specific objectives and purpose of the task. ensures that the stays on sopic, directly addresses the intended subject, information that is appropmaat and maaningful for the given context.
Functional Wriling
Relevance
Rel
2024h
provides
This criterionn assesses how effectively extra context or clabceation is provided necessry, entancing user's understanding without overwhelming them. ensures that the information valusble and directly aligned with the user's inquiry, offering clarity and depth where necled
Functional Wriling
Contextual Information
Conl
Li ct al. 2023b
This criterion assesses effectively specificc instances cases provided to illustrate the concept being discussed. Examples help make ahstmet ideas more tongihic and understanctihle by showing how they manifest real world hypothetical scenarios.
Functional Writing
Example Quality
ExafQ
Li at 20236
Accuracy measures low corneet the model's resporse is, easuring it is both factually accurate and aligned with the user's is expectations, It evaluates the model's ability interpret the question coneetlly and provide response
Accuracy
Acc
L 20236
Gencral Commanicazion
on verified objective not only be correct but also reflect reliable and accurate information.
The output
This criterion assesses whether the respense provides sufficient informa- tion address the query, including necessary context and explanrtions ensures that answer is
Information Richness
InfoR
X02th
General Communicazion
clear, offering enough depth to fully address the user's 'e''s needs without omitting key information.
This criterine evalutees understandaahiliyy which refers to how well a comversational system interprets user inpuis and generates contictuullyy relevant responses. locuses on the model's ability to gresp the meaning of user queries and the flow of the convensation. ensuring responses appropriate, For example, today?" model should respond the current weather, instead of a generic or unrelated reply. A model with high understundabilityy maintains seamless, clear communication.
Understandahility
Und
Li al 20023
General Communicatio
asks, 'What's the weather like relevant. clear answer about
This criterion evaluates how closely the generated title aligns with the content of the text or work. It's about ensuring that title directly reflects the main theme or subject matter, providing meaningful and appropriatc summary for the given contexl. For instance. the text effects of climate change wildlife, relevant tille might Wildlife in Distress: Unraveling the Impact of Climate Change.
Suitability
Suit
et all
Title Generation
20236
This criterion evaluates bow much of the essemtal information from the is captured the generated tille. It's abour assessing how source
Covengge
Coo!
20236
Tille Generation
ritle encapsulates the key points or content. For instance, if the text discusses the history. types, and health henefits of meditation, a title that covers these points might be Meditation: Tracing hts Roots Understanding Its Health Wonders.
This criterion evalustes the attractiveness the generated ritle. as- sesses whether the tidl has the power to catch the reader's is attention. spark interest, and entice the reader dalve into the content. For ample, a
Appeal
Appe
all
Title Generation
"Unlocking Creasivity: Harnessing Power Unconventional Mind' might appeal to readers interested in personal development and creativity
This criterion evaluates the precision the translation replicating the source content choices, proper grammar, and syntax. It entures that the meaning, tone, and munces the original test preserved without adding, omitting, distoming information.
target langunge. involves correct vocahulary
Translation Fidelity
Tmn
al
Translation
2028/0
This criterion assesses whether the translation covers all parts the source lext without leaving out any information, details, or munscss It ensures that the translation fully conveys the original text's content, including its ideas, arguments,
Thorogghness
Thor
et
Translation
20236
points, the target language.
This critericn evaluates the precision and correctness of the data analysi results. It assesses the degree to which the results correspond to the true values, ensuring
Precisson
Prec
et
al.
Data Analysis
2023b
output free from
distertions.
This criterion evaluates the clarity and user friendliness of the data asalyss oulput generated by the model. It assesses whether the output is well-structured, follows standard data presentation conventions, and is output
Form
200300
Data Analysis
easy to interpret and understand, results in a clear and concise manner, using appropriatc lables. charts, or other visual aids as necessary. The use of clear headings. labels, and explinations should also be considered.
present analysis
28

---

Published as a conference paper at ICLR 2026
Table 19: The definition and source of Task-specific Aspects (TAs). (Part 3)
Aspect
Abbr
Source
Task
Defanition
This evaluutes ability of generated questions captivate und hold the mtcntio of the andience. Engaging questions spurk curios- ity, stimulote thought, and encourage active participazion For instance. a question like 'How does climate change impact our daily lives?' might engage readers bevouse it connects global issue personal experi- unces.
Engaginggess
Engg
Li ot
all
Question Generation
critetion assesses whether complexity level the generated questions matches the knowledge and compeehension level of the in- lended andience. For example, f high school andience, a question like "Can you explain Einstein's is theory of relativity? r might he too difficult, while 'What the capital of France"' might be easy.
Diffiultyy
Dimf'
ot
al
Question Generition
keyword extraction, this critution evaliattes how closely extracted key keywords align with the main themes subjects of text. Relevant weywords should accurately reflect the care ideas, topies. or concepts presented in the text. providing concise and meaningful reprosenation of the content.
Relevance
Rell
G
al
Keywords Extraction
the contect of keyward extraction, this criterion avsesses the which extracted keywords represent all the signitionet themes, lopics. or concepts in the text. A high coverage means that the keywords collecsively provide comprehensive summary of the text, kuving no impestant sepect unsepresented.
Representation
Repe
G
woyw Extraction
30233
Accurscy is factually accurate and aligned
bow conrect the model's respoese is, ensuring expectations. It evaluttes the moddl's ability inserpres the user's questio currellly and provide a response based on verified objective facts. The outpur must not only be correct but also reflect reliable and accurate information.
Accuracy
Avc
Zhang ci al. 20230
all
20244
Rich Understanding
the
Instraction Following defined the shility to accurately interpret and the
ourlined the given instruction, locusing solely providing the corrct answer menssuess whether response directly addresses the task's core requirements. strctly adheres to any specified consrmints on responce longth and formut, and does not intrddnce unnoc- estary additional bockground information, explonations, or tone beyond what requested.
Instruction Following
InsF
A 20za
Rich Understanding
Zeng ctal K0
inconsistency refers model's quiputs that a ' diractive. the inconsistencies here signify unintestional misalignment with non- mulicious user instructions. For example. the user's actual insention is trinslation, However, the M erronconsly deviated from the user's insemetion and performed question-answerig
from some deviations might serve safety guidelines.
Instruction Inconsistency
Inal
Eluang al 2023
et al
Robusiness oriented Instructions
instead.
Context inconsistency occurs when model's octpui contrudicts contextual lual information provided by the user, which may include text or images. This happens when the model misinterprets the context or incomectly identifles elaments, such as attributes of objects existen details, For example. the states that the sourcs the incorrectly claims it the Americas Similarfy. image shows girl red clothes, but the model mistakenly identifies her as boy in blue clothes.
Context Inconsistency
Conl
al.
20247
Robustness- oriented Instructions
image
region central
but
measures how correct the model's resporse is, ensuring it both factually evaluates model's ability interpres the user's 's question correctly provide a response based om verified objective facts. The aulput must not only be correct bet also reflect reliable and accurate information.
Medical Image Undenstanding
and aligned with the
expectations. It
Accuracy
Instraction Following defined as the ahility to accurately interpret and the task outlined the given instruction, focusing soidly providing the connect addrestes task's care requirements, strictly adheres to any specified constraints on response length and format, and does nct introduce unnec- evcary additional hick hackground infommation, explonations, or tone beyved what's requested.
Instruction Following
Zeng al.
Chart Ressoning
whether response directly
Misry alal Abe
Accuracy measures how conrect the model's respoese ensuring factually
Accuncy
Ace
Misty ol al. 2026
all M02
Clurt Ressoning
and aligged with the evalutes the model's sbility interpret the nser's question correctly and provide response hased on verified objective facts. The output must not only be correct but also reflect reliable and accurate information.
expectations.
Instraction Following delined the ability to accurately interpnet and execute the outlined providing the correct ancwer messures whether the response directly addresses the task's core requirements. striclly adheres jo any specifsed constrinns on respomee lenth and format, and does not intrudice unnoc- exarry additional hackground information, explonations, tone beyend what requested.
given instruction, focusing solely
Instruction Following
InsF
Zeng et al. Malhew al X
2023
Graph Remsoning
response ensuring is both fastually accurate and aligned with the user's is expectations. It evaluates the model's ability interpret the user's is questio correctly and provide response based on verifed objective facts. The output must but also reflect reliable and accurate information.
Accuracy measures conrect
Accuracy
Ace
Mallhew cl
et ab
20 X022
Graph Ressoning
only
This exiterion evaliates the logical coherence helievability of generated events
the conexxt the narnative assesses whether the events follow a logical progression, align with the established facts and nules of the narrative world, and are consisten with the chameters' motivations and behavioes. The events should not introdice any incon- sistencies contradictions
Storstelligg Generation
Ressonableness
Reas
Lib ct al. 20244
the audience's disbelier. Furthermone, the should plassible and rellistie. avoiding any abourd improbable seenariss unless they we jusified by the narrative context.
could
29

---

Published as a conference paper at ICLR 2026
Table 20: The definition and source of spsk-specific Aspects (TAs). (Part 4)
Aspect
Abbr
Source
Sub-Tack
Definition
Instruction Following is defined as the ability to accurately interpnet and execute the task as outlined in the given instruction, focusing solely on providing the correct answer. messures whether the respotse direetly aukhessess the
Instruction Following
InfF
Singh et al 2019
Zeng
al 2023
Text Reasoning
eure requirements, stricily allterss any specilied constraints on response leugth and format. and does not introduce unne-- additional buckground information, explanations, or tone beyond what requested.
Accuracy measures correct model's response is, encuring it is both factually accurate and aligned with the user's is expectations. It evaluates the model's iss shility to interpeet the user's question correctly and provide response hased verified objective facts. The curput not only conrect also reflect reliable and accurate informarion.
Accurecy
Acc
Singh ot al.
G < a
201
20247
Text Reasoning
This criterion examines whether the cuption remains concise and focused, conseying the dant overly detailed comtent, emplasizes clarity precision communizating key points while minimizing distmctions from imelevant elements.
Conciseness
Con!
Makady et al. 20T
Simple Image Captioning
information ellicienily without including redan-
Hossain Milly
Object Aligunent avalustics the procision with which images match objects described the corresponding text. criterion ensures that the objects mentioned in the text are accurately represented in the image. focusing solely on whether the objest itself is present. It does not consider additional criteria such attribute alignment.
Object Alignment
ObjA
Mokady al N01 Hossain et a 2019
Simple Image Captioning
Text-Image Relationship refers to how text image work logether to enhance each other. This includes how the text deseribes what's is shown in the image objects, their properties, and sparial relationships- and also additional aspeets: Complement, which captures how the adds extm context background information beyond mere description to enrich meaning, and Integration, which evaluaties how eoherenily and naturally the text and image combined into a unified whole. Effecrive text-image aligament
Text-Image Relationship
T-IR
Mokady a Hossann ot 3. 2019
Simple Image Captioning
only describes the image also, through Complement and Integration, enriches its overall meaning.
Alignment between text and inages ensures the image mutchhes the descciption, including the attributes and relationships of objects. For exampie,
Alignment
Ali
Mokady et al. 2021 Hossain et a 2019
Simple Image Captioning
as Napolcon Bonapurte, the image must show a cat. If the text little girl sitting in front of a sewing ma- chine, the image should show only one girl in that setting, not muliple girls. Misaliguinent happens when the subject is missing the image doesn't match description.
cal
Completeness evaluates whether the caption covers all significant ele- ments in the image, including key ohjects, actions, and possible contex. tual information leg.. hackground, serting, scene details). A complete caption should mentice all essential leatures of the image without contt ting critical information.
Campleteness
Com
Hossinn
X01
Detailed Image Caplionig
2023
Attribute Aligument assesses how well the attributes (such color, texiure,
shaps) objects deseribed the the image. It ensures that key leatures like color and sine mutch what's described. For instance. if the text says "a green bicycle," the image should show bicycle that
rellected
Attribute Alignnent
AllA
[Hassain et al. 2019 Piao 2023
Detailed Image Captioning
It also checks detuils like texture red hall" being depicted both smooth and red. Furthermors, the relative size and propottions of objects should be consistent with the descrigttoon Overall, the image should visually mutch the text's description each object's attributes.
and shupe are accurate,
the image captioning generation task, criterion measures how accurately the spatial relationships hetween objects in the text align with their representation in the image. It evalaates whether the relative pasitions and arrangements of objects deseribed in the reflected in image, focusing solely spatial relationchips. does not consider other criteria, such as object attributes or additional context.
(assain et al. 2019 Ziao
Derailed Image Captioning
Spatial Aligament
SpaA
carrectly
Count A Alignment ovaluates low accurately the number of objects de- scribed the text matches both mimber and type anjects presented in the concepanding image. It specifically measures whether the quan- tity of objocts meniiened in the texi (e.g. two dogs, three cars, or live apples) correctly reflected in the image, and ensures that the ribjects shown
Count Alignment
CouA
(assaan et al 2019 Ziao
Detailed Image Captioning
the correct type, For example, the mentions "two dogs'. the image should contain two dogs, not two cats. This evalltatio focuses solely on the count and type of objects, without considering their attributes positions.
image captioning generation Action Alignment evalinates how accurately the actions or interactions depicted in the image are described in the text. specifically measures actions like 'running' 'talking' or 'holding" and does take into relationships (e.g.. left. right. or object araagements). example. if the image shows a dog nunning after hall, the text should include the word 'running'..
Action Alignment
ActA
Hossain al 2019 W023)
Detailed Image Caplioning
static descriptions sparial
Object Aligumnnt evaluates the procision
which images match objects described in the coriesgonding text. criterion ensures that the objects mentioned in the text are accuntely represented in the imape, focasing solely on whether the object itsel' is present. Ii does consider additional
Object Alignment
ObjA
(osssain et al 2019 25ao et all 2003
Datiled Inngge Captionngg
attribute aligmment.
30

---

Published as a conference paper at ICLR 2026
Table 21: The definition and source of Task-specific Aspects (TAs). (Part
Aspect
Abbr
Source
Suh-Tack
Definition
In the image captioning generation task, this metric assesses whether the colors described the text match the corresponding objects in the image. the image presents an object with specific color, the should describe it as accuratcly as possible. For example if the image shows blue and person wearing yellow shirt, the text should mention "bluee and 'yellow'. The more accurately the colors are described, higher the score.
Hossin et al. 2019
Detailed Image Coptioning
Color Alignment
ColA
Anoo all 2024
In the image captioning generation task, this metric checks how rately textures presented the image are descrihed in text. the objects in the image have specific texture qualities, such as surface qualities (e smoothh, rough), material types (o wonden, metallie), tactile seusations er patterns (e.g.. silky, polished), the more accurately these textures are expressed in the text, the higher the score.
Hossain et al. 2019] Zino al. 2023
Detailed Image Captioning
Texture Alignment
TexA
the image captioning generation tack, this metric assesses how rately shapes descrihed
text match shapes objects the image. the image contains objects with shape descripines sach goometric terms (e.g, cubie, circular, triangular) more general attributes like symmetty. curvature, complexity. and flatness, text should include corresponding descriptions. The more accurately the shapes are described, the higher the score.
Hossain et al. 2019 Zhso all 2023
Detailed Image Captioning
Stape Aliggiment
SlaA
Expertise refers the model's ability accurately use domain- specific temninalogy when answering academic questions. This includes cor- rectly applying terms relevant to the subject, such as those science, history, literature. For example, when asked about historical event, the model should use terms like primary sources' or "historical comtext instead of vague words like 'thin' "event, Proper use of academic terms shows the model's understanding the subject.
Expertiso
Expo
Li
Academic QA
The conciseness metric assesses whether the model provides clear, di- and succinet answers to academic questions, avoiding unnocessary details. The sccuracy metric evaluates the responses correctly address the query based on the provided academic context, ensuring precise and relevant information is delivered.
Correctness
Corr
Li
Academic VQA
Explainability is criterion used assess how clearly a model explains its reasoning when answering an academic question. It focuses on whether the model provides clear, logical steps and justifications for answer, making it easy wsers to understand how the answer was reached.
Explainabiliit
Expl
Li Lirttal al. 2023c
Academic VQA
Interpretabilitt in the context of embodicd decision-making refers model can articulate decisions actions. It's about trans- parency, where the model should he able to provide clear explnatiions for why chose particular course of action over others. For example, Al playing chess makes a move, interpetabbility would involve the Al heing able to explain why it made that specific move, considering factors like board position, potential threats, and future opportunities.
Interpretability
Intp
Chen et al. 2023
Embodied Decision-making
Task completion essentially about goal achievement. decision-making tasks, this means whether or not the model has snacess- fully navigated its enviromment to achieve its intended objective. This could range from simple objectives, such as reaching certain location within an environment (like an Al navigating through a maze), to more complex orcs. like completing multiple interrelated tasks sequence (like cnoking, where you need chop vegetables before frying them).
embodied
Completion
Comp
et
Embodied Decision making
"Riskiness measures how well the model avoids harmful situntions or risky behaviors during lask exccution. This particularly important in real-world applications where mistakes can have serious consequences. For instance, if we consider autcnomous driving again, safety here doesn't just mean avoiding collisions; it also involves oheying traffic miles, taking unnecessary risks while overtaking other vehicles, and maintaining safe distances from pedestrians and other road users.
Embodied Decision-muking
Riskiness
Risk
Chen et al. 2023
This criterion evaluates the originality and inventivenees in the gencration mixed media fairy It assesses whether model can generate unique fairytakes that offer fresh perspectives, innovative plat twists, and imaginative characters. The creativity also extends to the integration of visuals and text, with the model expected in genemte images that only complement the parrative but also calance in uncxpected and creative ways. The model should be able to break away from common fuirytale tropes cliches, offering unique and memorable narrative experience.
Originality
Orig
Liu
2024a
Activitynct Coniination
This criterion evaluntes the suitabiiity of the generated fuirytale for a young andience. that is casy to undersiand and digest for children, wilbout compreaniiing the richness and depth of the story. The language used should be simple. clear, and ago-oppropritee, avaiding complex vocahulary convoluted scatence structures. The plot should be straightfowward and easy to follow, with no everly complicated or confusing elements. The visuals should also be simple and cleur. effectively conveying the emocions the story way that
whether the model can generate narrative
Activitynet Contimation
Simplification
Simp
Liu al. 2024
and and engaging for children. This criterion ensures that the generated fairytale not only enjoyable but also appropriatc and bencficial for its intended young audienoe.
31

---

Published as a conference paper at ICLR 2026
Table 22: The definition and source of Task-specific Aspects (TAs). (Part 6)
Aspect
Abbr
Source
Sub-Task
Definition
This criterion evaluates whether the generated events are comptebensive and well-rounded within the narrative. It assesses whether each event has cleur beginning. middle, and end, providing full and satisfying narrative arc. events should not fragmented incomplese. should provide all the necessary details and developenents to understund and appreciate their significance the narrative. This incluress the introduction the or outcome of the event.
Storytelling Generation
Putfillment
Full
al. 20244
the unfolding of the and the resolution
Aligument between text and image is evalusted based on how well the image matches the details in the test. This includes the correct objects, their altributes (e.g., volor, Siee), spatial relationships log.. right), and actions. The alignment should consider multiple dimensions. For esample, if the text describes "two buys wearing white clothes, one on the left and one the right. playing hadminton," " the image should show boys in white clothes, placed correctly in scene, with badminton game happening between them.
Xaaong
Aligumont
Ali
Hunnng et al2
A0M4 A044
Text-l-Imaae Generation
Clossb
sl
Xu
Object Relationsbip evaluates how accurately the spatial relationships actions between objjects deseribed in image. It focuses on whether the interactions, positions, and movements of objects in the image match those described in the text. For example, in the description book"" image should standing
are represented
Object Relationship
ObjR
Husin
Li
et al
2023
2024
Test-t- Generation
table and reaching for person next the table hand extended toward the book. As long as the spatial relationships and actions are correctly represented, the image will he considered accurate.
their
This criterion measures how accurately spotial relationships between objects in the generated output match the descriptions in the text. It evaluaies wheller the relative positions, orientations (up, down, lefl, right), and arrangements objects align with the textual description. example, if should show the chair on the left side of the table, and if a person is sitting on chair in front of table, this relationship should also be reflected accurately the image.
Ghooh et all
2023 2024
Spotial Alignment
SpaA
Hunng
Text-to-maag Generation
mentions a chair the left table, the image
Goochale ct MUM
Action Alignment evaluates how accunitely the actions internctions described in the text are represented in the inage. It spccificalll measures dynamic actioes like "ranning,. 'talking or 'holding', and does not consider static descriptions or spatial relationsiips (e.g.. right, or object arrangements). For example. the text says 'a dog running after ball", the image should show the dog running after the ball.
Action Alignment
ActA
1 et dk
21244
Text-to-Image Generation
Huang
Count Alignment evaluates how accurately the number of objects the number and type objects presented
scribed in text matches in the corresponding image. It specifically measures whether the quan- tity of objects mentioned in the text (e two dogs, three cars, or live apples) correctly rellected in image, and ensures shown are of the correct type. For example, the text mentions "wwo dogs' the image should contain two dogs, not two cats. This evaluation focuses soldly on the count and type objects, wilhout considering their attributes or positions.
Count Aligument
CouA
Hunng
1 et Ai
2024
Text-to Image Generation
the objocts
Chosh al.
Object Alignment evallutts how accurately the abjects described in text are represented in an image. The objects mentioned in the text should be accurately presented example, the description" A man sitting himself at dianer tablee. the image should feature both a man and a dinner table to achieve a high score. Missing extra objocts the image would lower the
Object Aligument
ObjA!
Text-to Image Generation
image,
omissions. For
Eabosh
3123
color, texture, size, and shape) of objects described the text are reflected in the image. It ensures that key features like color and size match what's deseribed. For instance, if text says green bicyele"' should show a bicycle that is green. and shape are accurate, like "smooth, red ball" being depicted as buth smooth and red. Purthemorr, the releive size and proportions of objects should be comsittnt with description. Overall, image should visually match the text's description of each object's altributes.
Attriuue Alignment assesses
well attributes
20244 p024
Test-toImmage Generation
image checks details like texture
Attribtee Alignment
AnA
Huling
Clooh el alll 2033
This criterion assesses whether the colors described in the text match the carresponding objects the image based solely on the text description, or whether the colors the objects conforn to realistic colors. text specifies the color of an object, the evaluation checks whether the image reflects that color, For example, the rext snys 'n blue car', the image should show different color.
Color Alligmmett
ColA!
Honng Ct a A0p4
al_
Text-to- Image Generation
Shosh al.
matches the described color, not
Liu al
2 Wang
Maltimodal Script Generation
Completeness measures whether the model generrteeeeted steps provide complete guide finishing the task, covering all necessury actions needed start
Completeness
Com
Koupace
2018
This criterion evaluaties the icasibility of content output by model for specilic tack by assessing whether each step factually correct, logically sequenced, and detailed enough successful exccution. ensures that the steps are not only accurate but also praclical, previding clear and actionable instractions that lend to the desired outcome without leading confesion
Fousibility
Fea
Koupace
[Luu
A024
Wang
Multimedal Scrigt Generation
2018
This criterion evaluntes whether model's response provides clear and practical steps to help the user complete the task, such as planting a strawberry. It checks if the response includes all essential details, like prepuring soil, planting the seeds, and caring plant, along with useful tips or warnings.
Helpfulness
Hel?
Keupuse &
[Liu et al.
Wang
Multimodal Script Generation
2018
32

---

Published as a conference paper at ICLR 2026
Table 23: The definition and source of Task-specific Aspects (TAs). (Part 7)
Aspect
Abbr
Source
Sub-Task
Definition
This criterine checks the model's resposses sufe monid harmful content. ensures model doesn't I suggest dungerous, unethical, illegal actions, and follows safety and ethical guidelines. The model should also provide safery warnings advice when needad, priortizing well-beinng and avoiding any ricks.
Safety
Saf
Koupace &
Wang
Multimodal Scripe
Genemation
2018
This criterion evaluates how accurately the shapes described in the lext corresponn to the shapes of objocts in the image, based sulely on the text's shape deseriptors, oe whether the shape of the objects conforms realistic shape. text may include commen shope square, round, roctangular: oval, trisngalar, cubic, ey lindrical, spherical, and other hasic or specialized shapes like pyramidal, conical, pentagonal, teardrop, crescent and diamond..
Shape Aligument
ShaA
Huang
F
al.
20144 2024
Text-to- Image Generation
Chouh et (036)
This criterion evaluates are reflected in the image, based solely on the text deseription, or whether the testures of the objects conform to realistic textures. It includes surface qualities such smoothness, roughness, material types wood. plastic, and rubber. Additionally, it may cacompass tactile sensations or putterns, such silky, polished, or other material-hai hased descriptions The accuracy of the texture description is assessed by comparing the textual description the actual appearance and surface properties of objects the image.
accurately
described the
Texture Alignment
TexA
Hoang Chosh et
2izzm
Texr-to- Genematio
well
AUX
Advanced Alignment is comprehensive evaluation nepect that assesses the shility accurately represent complex textual descriptions in an image. It encagoolates Comparison, Negation, and Universality aspects. means checkss whether image correctly depicts comparisons berween entines hased on attibttee such as number or volume; represents ahsence contradictinn of elements indicated words like 'no" or "net': and illustrates scenatios where every member of group shares specilic altribute or relation.
Advanced Alignment
AdaA
al 2024a
Test-to-lnage Generation
Comparison evaluates the ubility to uccurately depict comparisons be- tween eniities described in text. It assesses whether the image correclly represents characteristics such as number, altributes, arca, or volume of objectt and thxir comparative relations. For example, if text describes 'berween the two cups on the dosk. the taller one holds more coffee than the shorter cups
Comparison
Cmpa
al 2023
Text-to-Image Generation
which half half-empty' heights ditlering amcunts collee-- specilically illustrating that the taller cup contains more coffee and that the shorter cup only half full.
should show two
Negation evaluates the ability accurately depict the abeence tradiction of elements described text. assesses whether image correctly represents scenaries where entities are not present, as indicated by words such as 'no". 'nor', or 'witout'. For example, if text descrihes bookshelf with books, only picture frumes', should show bookshelf devoid of books filled with pieture timmes instead.
Negation
Nega
Li Liectal al. 2024
Text to Image Generation
Universal evalnates the ability to accurately depict scenarios where every member group shares specific altribute involved in relation,
Universal
Fi 2re44
Text-to-lmage Generation
text, assesses whether the image conrectly represents sinatioos indicated by words such 'every'. 'both'. For example, if text describes 'a bustling kiichen where every chef prepuring dish", it should show all chefs within the kitchen actively engaged cooking.
Accuracy measures bow well the edited image aligns with the given instructions in the text. The model should accurately implemeni the specified changes, including adjusting colors, shapes, positions, and adding removing elements instracted. This includes adhering any quantity or specilic altribute requirements. deseribed the text. For example, if the text requests 'add one small white dog' the image should show exactly dogs black dog instead, the accuracy evalustion should poot.
Accuncy
Acc
Zhang al. 2023
Image Editing
small white dog. the model generates two white
Crearivity in visual story completion refers to the originality and imagi- nation in developing the namaaive It evaluntes how fresh ideas, unique expressions, and inventive approach used to expand upon the visual elements. Creativity of visunls, offering new and engaging conrimation of the story that caprivates the reader.
Creativity
Crel
201
2004
Smilevski
al.
al.
Visual Story Completion
originality
interpeetation
Engagingness refers to dogree which story fusters continued interaction, maintains or clovates interest, and stimulites compelling exchingo of ideas, emotions, or informatioe. This criterion assesses the ability of the content engage the audience, encouraging active purtici- engagement making nurniiv or dialogue both interesting and thought provoking.
Smileviki
e5 al.
Visual Story Completion
Engagingness
Eng
201M
Lau
2021.
putiion
Completeness measures whether the model vides complete naralivve covering all beginning acters, settings. actions, the story. For example, if the inpul is sequence of images showing a person preparing for hike,, the generated story should include all key steps such packing backpack. driving the trail. starting the hike, reaching the destination, easuring coherent and compechensive narraive.
model-geneateed generated visual story pro- events)
elements (e.g. char- end of
Completeness
Coml
2018
Smilevski
Liu
01
al.
d
Visual Story Completion
20044
33

---

Published as a conference paper at ICLR 2026
Table 24: Aspects used by evaluators in the benchmark evaluations, (Part 1)
Subclass
Aspect
Definition
Coherence refers the logical consistency and interconnection ideas within the text. It ensures that the text follows clear, organized pro- gression, with each section logically linking to the next. A coherent text allows the reader to easily follow the author's argument or narrative, with smooth transitions and sudden shifts in topic or gaps in reasoning.
Coherence
Coherence
Evaluate whether facts presented the response align with the facts described the article. This metric focuses solely on ensuring the accuracy of the events or facts, without considering the level of detail. As long as the facts in the response match those in the article, both the response and the article should receive the same score. The evaluation does not include assessing whether key information is omitted.
SummEval Eval
Consistency
Consistency
summary should cover all essential points from original text, ensuring no significant information is omitted. In this case, completeness is more important than brevity.
Consistency
Coverage
Flueney measures the smoothness and naturalness of language and sen- tence construction, ensuring the flows seamlessly and grammat- ically correct. Fluent text not only maintains a natural flow includes concise phrasing, where brief answers can still achieve high scores, as long as they remain fluent.
Fluency
Fluency
Relevance in story generation measures how closely the generated story aligns with the specifie requirements, title, or provided initial part of the story. ensures that narrative stays on topic. directly addresses and develops from the given prompt or initial storyline, and provides a continuation that is logical and easy to understand within the established context.
MANS
Relevance
This criterion evaluates the model's ability generate responses that only align
specific instructions provided but also enhance the user's understanding the context, checks the model adheres to any given guidelines about format and word count, while also providing correct and insightful answers. A high-quality response should offer valuable insights, additional context, relevant information that con- tributes positively the user's comprehension. This assessment the model's output not just accurate instruction compliant, but also informative, contextually rich, and user-friendly.
MT-Bench
Correctness
This criterion is comprchensive assessment of the model's 's responses. It combines both accuracy and helpfulness. Accuracy a measure that ensures the model's responses are factually correct and in alignment with the user's expectations, while helpfulness evaluates the value and relevance the information they enhance the user's understanding. An overall evaluation takes into account how correctly the model interprets and responds to the user's prompts, as well as how beneficial the responses are to the user. It's holistic measure of the model's effectiveness in providing accurate, insightful, and useful responses.
Vision
responses, focusing on how much
Overall
34

---

Published as a conference paper at ICLR 2026
Table 25: Aspects used by evaluators in the benchmark evaluations. (Part 2)
Subclass
Aspect
Definition
Accuracy measures how correct the model's response is, ensuring it is both factually accurate and aligned with the user's is expectations. It evaluates the model's ability interpret the user's question correctly and provide response based on verified objective facts. The output must not only be correct but also reflect reliable and accurate information.
COCO
Accuracy
Alignment between text and image is evaluated based how well the details the correspond the visual elements the image. This includes accurate representation objects, their attributes (e.g.. color, size), spatial relationships (e.g. positioning, direction), and actions. The alignment should ensure that the text captures all relevant details from the image. For example, the image shows a surfer riding wave with the surfboard visibly moving, the text should accurately describe these elements, such as the wave's size, the surfer's 's stance, the motion of the surfboard, and the wave's characteristics, ensuring all critical visual details are reflected the caption.
C.C.
Aligmment
This criterion evaluates the model's ability to accurately analyze the alignment between given text and image. It assesses if the model's res responses correctly identify the quality of text-image pairing, pinpointing perfect alignment or errors in object attributes and spatial relationships. For instance, if the text mentions cat, but the image shows dog. responses that accurately point out this misalignment in objects are considered high quality. The ubility accurately identify issues or provide precise evaluations determines the quality of the responses.
Alignment Accuracy Analysis
Diff
Accuracy measures how correct the model's response is, ensuring it is both factually accurate and aligned with the user's expectations. It evaluates the model's ubility interpret the user's question correctly and provide response based verified objective facts. The output must not only correct also reflect reliable and accurate information.
MLLM-as-a-Juuee a-Judge
Graphics
Accuracy
Accuracy measures how correct the model's 's response is. ensuring it both factually accurate and aligned with user's 's expectations. It evaluates model's ability to interpret the user's question correctly and provide a response based on verified objective facts. The output must not only be correct but also reflect reliable and accurate information.
Math
Accuracy
Accuracy measures how correct the model's response is, ensuring it is both factually accurate and aligned with the user's 's expectations. It evaluates the model's ability to interpret the user's question correctly and provide response based on verified objective facts. The output must not only be correct but also reflect reliable and accurate information.
Text
Accuracy
This criterion evaluates the model's 's ability accurately analyze the alignment and relevance between a given text and image. It assesses if the model's 's responses correctly identify the quality of text-image pairing, pinpointing perfect aligmment emrors in object attributes and spatial relationships. For instance, if the mentions cat, but the image shows a dog, responses that accurately point out this misalignment in objects are considered high quality. The ability to accurately identify issues provide precise evaluations determines the quality the responses.
Alignment Accuracy Analysis
WIT
Accuracy measures how correct the model's response is, ensuring it both factually accurate and aligned with user's 's expectations. It evaluates model's ability to interpret the user's question correctly and provide a response based on verified objective facts. The output must not only be correct but also reflect reliable and accurate information.
Chart
Accuracy
35

---

under- Published as a conference paper at ICLR 2026
Table 26: Aspects used by evaluators the mata-benchmark evaluations. (Part 3)
Subclass
Aspect
Definition
This criterion evaluates the quality and relevance of the model's 's response. It focuses on whether the information provided not only accurate but also of significant value to the user. The evaluation takes into account if the model's 's output offers beneficial insights that help deepen the user's understanding of the topic at hand. Additionally, assesses if the model can provide further context or pertinent information that can enhance the user's ' comprehension or perspective. The goal is to ensure that the model's response is not merely factual, but also insightful and enriching. contributing positively the user's knowledge and under- standing.
CC-3M
Helpfulness
This criterion evaluates the quality and relevance of the model's response. Tt focuses whether the information provided is not only accurate but also of significant value the user. The evaluation takes into account if model's output offers beneficial insights can help deepen the user's understanding of the topic at hand. Additionally, assesses if the model can provide further context pertinent information that can enhance the user's comprehension perspective. The goal is ensure that the model's response is not merely factual, but also insightful and enriching, contributing positively to the user's knowledge and under- standing.
VisIT
Helpfulness
Accuracy measures how correct the model's response is, ensuring it is both factually accurate aligned with the user's expectations. It evaluates the model's ability to interpret the user's question correctly and provide response based on verified objective facts. The output must not only be correct but also reflect reliable and accurate information.
MLLM MLLM-as-a--iieeg a--uae--uuuge Judge
SciQA
Accuracy
This criterion evaluates the quality It focuses on whether the information provided is not only accurate but also of significant value to the user. The evaluation takes into account if the model's 's output offers beneficial insights that help deepen the user's understanding of the topic at hand. Additionally, assesses if the model can provide further context or pertinent information that can enhance the user's comprehension perspective. The goal is to ensure that the model's response is not merely factual, but also insightful and enriching. contributing positively standing.
relevance of model's response.
M2W
Helpfulness
user's knowledge
This criterion evaluates the model's response in terms of its accuracy and depth in explaining the aesthetic quality of the image. It assesses whether the model correctly identifies and articulates present the image, such as color harmony, composition, lighting. and texture. Furthermore. it examines if the model offers comprehensive analysis that considers multiple aspects of aesthetics, including but not limited to symmetry, balance, contrast, and the rule of thirds. The model's ability relate these elements to the overall aesthetic appeal of the image is also taken into account.
aesthetic elements
Aes
Aesthetic
This criterion evaluates the accuracy of numerical calculations in model's 's response when answering questions that require both image recognition and computation. focuses on whether the model correctly identifies relevant visual elements from the image and performs accurate calculations to derive the final answer. The score is based solely on the correctness of the numerical result, ensuring the model's ability handle tasks that combine visual analysis and quantitative reasoning effectively.
MM-Vet
Precision
36

---

Published as a conference paper at ICLR 2026
Table 27: Aspects used by evaluators in the benchmark evaluations. (Part
Subclass
Aspect
Definition
GenAI GenAI-Bende I-Bench & Pick-a-Pic
Alignment hetween text and image is evaluated based on how well the image matches details in the lext. This includes correct objects, their attributes (e.g., color, size), spatial relationships (e.g., left, right), and actions. The alignment should consider multiple dimensions. For example, if the text describes "two boys wearing white clothes, one on and one on the right, playing badminton," the image should show two boys white clothes, placed correctly in the scene, with badminton game happening between them.
Alignment
This criterion checks a model correctly shows what's descrihed in prompt, like the number people, their actions, or their emotions. pays close attention to whether the roles and actions in the image precisely match those specified. For example, if the prompt says "Two humans and one wheel', it ensures the image has two people and one wheel, not the other way around. Similarly, hugs the shorter person', the taller individual should be one doing the hugging. It's all about ensuring that the picture matches the words exactly, both in terms of quantity and the specifics of the actions.
Relation
Contextual Relationship
states 'the taller person
Alignment between text and image is evaluated based on how well the image matches the details in the lext. This includes the correct objects, their attributes g., color, size), sputial relationships g., left, right), and actions. The alignment should consider multiple dimensions. For example, if the text describes "two boys wearing white clothes, one on the left and one on the right, playing badminton," the image should show two boys in white clothes, placed correctly in the scene, with badminton game happening between them.
Winoground
Object
Alignment
Alignment between and image evaluated based how well the image matches the details in the text. This includes the correct objects, their attributes (e.g.. color, size), spatial relationships (e.g.. g. left, right), and actions. The alignment should consider multiple dimensions. For example, if the text describes "two buys wearing white cluthes, one the left and show two boys in white clothes, placed correctly in the scene, with badminton game happening between them.
Both
Alignment
the right, playing badminton." the image should
Accuracy measures how correct the model's response is. ensuring it both factually accurate and aligned with user's expectations, evaluates the model's ability interpret the user's question correctly and provide response based verified objective facts. In cases where two responses are provided, only one musi be correct, and the other may contain errors or contradictions.
General
Accuracy
Factual Contradiction occurs when the caption mentions something that present in the image, which indicates hallucination. caption that cannot be found in the image, means the model has generaled incurrect or imaginary content. In this context, one of the two responses will always contain a hallucination. where the mentioned content cannot found the image. Captions that accurately describe only
VLRewardBend
refers to object
Factual Contradiction
Hullucination
visible image, without inventing anything. will score higher.
Accuracy measures how correct the model's response is, ensuring is both factually accurate and aligned with the user's expectations. evaluates the model's ahility interpret the user's question correctly and provide a response based on verified objective facts. The oulput must not only be correct but also reflect reliable and accurate information.
Reasoning
Accuracy
37

---

Published as a conference paper at ICLR 2026
NUG
KG ITIS
ADC
6PT.40
ITTG OUP
Asd
Human
GP7:44
Cludoo
Thards
LLat Crtic
Figure 12: Comparison on FRA-ID and FRA-ID-H. The top shows alignment with GPT-4o using FRA-ID, the bottom shows alignment with human annotators using FRA-ID-H. Each point is the average accuracy (with ties) for an aspect shared across sub-tasks of the same task.
Table 28: Average accuracy on FRA-ID and FRA-ID-H, separated by UAs and TAs. Average accuracy (with ties) is calculated by averaging the evaluation accuracies across sub-tasks within each task.
Seen UAs for Seen Tasks
Seen TAs for Seen Tusks
Method
FRA-ID (GPT-4o)
FRA-ID-HI (Hummm))
FRA-ID (GPT-4)
FRA-ID-HI (Human)
GPT-4o
76.1
79.1
65.0
66.1
79.5
74,8
58.8
73.6
Chadde-3,5
79.4
79.1
55.2
22.2
67.2
71.9
52.5
61.1
80.3
78.0
52.7
66.4
76.3
64.8
57.8
75.9
Qwen2VL-72B
Qwen2V1-778
51.1
50.0
58.4
37.4
48.6
55.6
53.7
38.2
48.9
46.9
42.6
39.5
46.7
51.6
34.6
36.1
Themis
60.4
60.3
47.5
53.4
LLaVA-CHis
Ours
91.1
88.2
610
87.0
25.8
78.2
65.0
70.0
87.9
85.1
71.9
87.2
84.1
71.5
58.1
73.9
D MORE EXPERIMENT RESULTS
D.1 IN-DOMAIN EVALUATION
For in-domain evaluation, we assess performance using the FRA-ID and FRA-ID-H. Figure 12 presents radar charts that illustrate the overall degree of alignment. The accompanying Table 28 reports average accuracies for each task.
The results show that UFEval achieves higher alignment with both GPT-4o and human annotators compared to the baselines (see shaded area in radar chart). Specifically, excluding GPT-4o, VFEval surpasses all baselines on NLG, IU, and ITIG tasks within FRA-ID and FRA-ID-H, achieving overall average accuracies 82.4% and 72.1%, respectively. On NLG, UFEval outperforms Themis, while base model Qwen2VL-7B performs much worse. Moreover, UFEval shows strong performance on the ITIG of FRA-ID-H. where its alignment with human annotators exceeds that of GPT-4o. This improvement can be attributed to UFEval's generalization ability and cross-task transfer performance.
38

---

Published as a conference paper at ICLR 2026
Natural Language Generation
Image Understanding
Interisaved Text and and-image Generation
Fly
Aee
Coh
GPT-40 Aligmmett
4o Alignment
GPT FF Alghment
Spok
Vodc
VesC
Ovee
Thamts
Figure 13: Comparison on FRAUAs-OOD evaluate the generalization in UAs.
D.2 EXPERIMENTAL RESULTS FOR FRAUAS-OOD
In the training set, for each sub-task, according to our aspect selection procedure, the task output modality includes text, all aspects under the Text Branch the UAs Tree can be selected. However, once the model learns a UAs in one sub-task, can leverage this UAs for inference in other sub- tasks without requiring retraining. Additionally, maintain sample balance between UAs and TAs during training (as selecting all aspects under the Text Branch for text-output task would lead to an imbalance in the number of UAs and TAs in the training set), we selectively sample a subset of corresponding UAs in the training set.
To validate that UFEval can evaluate UAs in sub-tasks where these UAs were not encountered during train- ing, we construct FRAUAs-OOD, which comprises the same sub-tasks as the training set but introduces unseen UAs across 4 tasks. The experimental results shown in Figure 13 and Table 29 The figure il- lustrates that UFEval exhibits robust generalization capability despite not being trained the complete set of UAs for each sub-task.
Table 29: The results of average accuracy FRAUAs-OOD for UFEval.
GPT-4o Alignment
Method
NLG
I0
ITI
Claude-3.5
77.3
77.6
83.0
Qwen2VL-72B
69.7
71.4
65.3
Qwen2VL-7B
49.9
56.5
44.4
Themis
54.2
D.3 EXPERIMENT RESULTS OF UFEVAL-72B
LLaVA-Critic
50.0
Ours
82.8
83.1
90.4
We also train UFEval-72B using the same fine-tuning method as UFEval-7B with -----2----72---11 as the backbone and test under the same experiment settings. The results, shown in Figures 14 and 15 and Tables 30 to 32 indicate that UFEval-72B offers some improvement over UFEval-7B.
Table Evaluation as aud for IG task. We evaluate baselines across three bench- marks.
GenAI-Bench
Winoground
Pick-a-Pic
Method
Relation dill (t)
Object diff (f)
Both dill (t)
Ave. difl (1)
tau(1)
diff(())
tau(1)
diff(())
GPT-40
73.0
59.2
Claudc-3.5
55.6
71.0
71.2
73.0
69.2
71.1
49.1
53.7
Qwen2VL-72B
49.1
52.6
46.7
60.9
$7.6
55.0
38.6
38.3
Qwen2VL-7B
35.8
38.0
33,4
42.5
46.1
40.6
38.1
40.2
VisionReward
$1.0
66.4
60.2
64.1
74.9
66.4
48.9
$8.0
ImageReward
48.6
64.9
54.0
58.2
69.2
60.4
48.8
55.7
Ours
53.6
65.5
59.1
80.7
50.0
57.3
Ours-72B
55.5
69.0
61.5
63.4
62.5
62.4
52.0
58.6
39

---

Published as a conference paper at ICLR 2026
Language
Image Understanding
and-lmage Generation
Cis
Calh
Cors
Com!
E110
ONoC
Da0
CPT-4o Aliggmmnn
%
GPT-40 A- Afghmant
Frah
Harma
ActA
Flu
Are
Hrll
Apc
Hi:
GFT.46
Sond
Halt
10r
1mgt
3omc
%D
STek
Set
S00C
Natural Language Generation
Image Understanding
Interieaved Text and Image Generation
Cast
cuh
Caee
Com!
C31
E2100
Eng'
GenD
Alggement
Human Allgrment
Dar
Human
#to
Und
The
Hel
2yC
CMA
PTV
Spen
SUA
Irgh
SenC
SerC
Spaak
SerC
Shunk
Image Generation
Image Generation
CELA
Dosh
Medod
CFPF4e Allaraune
Timnn Mlnnear
GPT44 Aligmment
(3744
91.
96.7
119
913
QeeetVL-72
ILAVA Coke
61.0
43
Oun-
93.4
87.6
1.0
CAAA
Tooal
0660
Spuat
Figure 14: Comparison of UFEval-72B on FRA-ID and FRA-ID-H H. The top shows alignment with GPT-4o using FRA-ID, the bottom shows alignment with human annotators using FRA-ID-H. Each point is the average accuracy (with ties) for an aspect shared across sub-tasks of the same task.
Table 31: Evaluation MLLM-as-a-Judge for IU tasks. We evaluate baselines across three bench- marks.
Vildonsion
MLL.Mass-.Judge
Benc
Method
General diff (t)
Hallucination diff (1)
Reasoning diff (t)
Ave.
tau (t)
diff (1)
(t)
diff (f)
diff (f)
GPT-4o
55.3
70.1
58.1
67.0
81.4
74.8
68.8
Claude-3.5
53.3
67.3
61.2
58.4
68.3
38.5
82.6
66.1
62.4
Qwen2VL-72B
50.3
59.6
65.5
54.6
58.5
50.8
75.4
70.7
65.6
Qwen2VL--BB
39.2
41.3
44.6
45.1
62.8
62.5
56.8
LLaVA-Critie
53.0
66.0
59.6
$5.6
65.5
42.0
41.2
60.0
47.7
Ours
66.5
57.2
67.0
46.4
57.7
71.1
58.4
Ours-72B
54.5
68.6
69.9
58.2
73.1
44.3
61.6
72.1
59.3
40

---

Published as a conference paper at ICLR 2026
Natural Lenguage Generttio
Image Understanding
Inter seaved lesbbnn--mmage Generation
Dr
Sook
Aigemment
GPT-55 Nigrment
GPTT-lo Algrrment
Inngtt
ABC
A006
Mxc
Tan
Seac
Canc
Senc
70/h
140
Natural Language Generation
Image Understanding
Inter feaved Text-and Image Image Generanion
CAB
Dan
Algreent
Human Alggiment
Humaet Algrame
Denc
Owi
Mec
T
D.94
Image Generation
Image Goneration Cocc
GFT44 Alpmant
Bamas Mlgent
201
26.4
114
Algrmment
0067
763
587
636
Oct
X9.4
036
Cyoa
GF1-44
LLAVM-D' Ouri-75
Crig
Figure 15: Comparison of UFEval-728 on FRA-OOD and FRA-OOD-H. The blue-colored aspects indicate unseen TAs, whereas the black-colored aspects represent seen UAs. The "ow/ unseen' designation in the table represents evaluations conducted exclusively unseen aspects, with accuracy metrics computed only for unseen TAs.
Table 32: Evaluation as MLLMMaas-aJJdde for NLG task.
SummEval
MANS
MT-Bench
Method
Coherence (au(t)
Consisency
Fluency
Relevance
Ave:
difl(1)
lau(t)
difl'")
dilT(t)
(au(t)
difk()
lau(")
dif((T)
tau(1)
dill(1)
lau(1)
dill(1)
GPT-4o
58.0
64.2
79.1
85.1
64.3
72.8
60.11
67.1
65.3
72.3
68.5
70,9
83,5
Claude-3.5
63.5
70.6
81.6
87.9
73.7
83.1
59.6
66.6
69.6
22.0
68.4
26.3
90.2
OwenIVL-72B
66.4
72.3
75.9
QwnnIVL-78
48.8
52.5
35.5
30.6
44.8
49.8
41.7
44.5
42.7
44.3
60.2
44.5
50.1
Themis
60.7
62.1
818
36.0
73.3
77.7
$4.4
$4.6
67.5
70.1
44.2
43.6
37.7
Auto-J
60.1
67.0
68.2
73.0
85.5
Prometheus
65.5
61.8
67.1
69.0
Ours
61.6
21.8
75.2
83.5
74.7
$1.5
61.3
67.6
69.0
76.9
69.3
74,9
88.3
Ours-728
64.6
71.7
77.2
86.9
75.1
84.7
60.3
67.0
69.3
77.6
68.0
77.4
91.1
D.4 SPECIFIC EXPERIMENTAL RESULTS
We report the specific average accuracy of each evaluator for every aspect within each task in Figure 12 and Figure of the main text. The alignment is measured against: (1) GPT-4o on the FRA-ID and FRA-OOD, and (2) human annotators on the FRA-ID-H and FRA-OOD-H Detailed results are presented in Tables 33 to 48 The reported average accuracy is computed by aggregating the same aspect across different sub-tasks within the same task. We also include the comprehensive results on MLLM-as-aaJudee in Table The specific accuracy multi-aspect assessment learning
is show in Table 50 and 52
41

---

Published as a conference paper at ICLR 2026
Table 33: The specific experimental results of the IG task in the FRA-ID used to evaluate alignment with GPT-4o.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2V1.-72B
Qwen2VI.-7B
Ours
Accuracy
Acc*
88.6
75.6
50.8
35.9
77.0
Action Alignment
ActAi
71.4
29.8
57.1
48.8
73.8
Alignment
Ali
80.2
65.3
52.0
39.1
69.7
Attribute Alignment
AttA
79.6
61.8
43.8
40.7
71.3
Color Alignment
ColA
80.7
48.3
57.9
40.7
76.6
Count Alignment
CouA
86.
62.8
71.9
52.1
80.7
Fidelity
Fid
74.6
53.2
42.7
40.8
44.0
Harmfulness
Har
57.4
97.4
76.1
77.7
Object Alignment
ObjA
80.5
41.2
57.1
53.5
73.3
Object Relationship
ObjR
77.9
49.7
42.0
38.9
70.4
Shape Alignment
ShaA
66.3
56.1
50.0
32.0
62.2
Spatial Alignment
SpaA
65.5
41.2
55.3
50.4
73.2
Texture Alignment
TexA1
65.9
48.5
49.8
37.4
61.3
Table 34: The specific experimental results of the ITIG task in the FRA-ID are used to evaluate alignment with GPT-4o.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2VI.-72B
Qwen2VI--7B
Ours
Action Consistency
ActC
88.2
64.9
45.2
24.5
84.7
Bias
Bias
87.4
74.7
67.6
46.2
80.4
Character Consistency
ChaC
79.1
58.2
33.7
14.2
72.8
Coherence
Coh
95.0
93.7
85.3
55.1
94.5
Completeness
Com
86.0
83.2
77.7
38.2
87.6
Completeness
Com
90.5
88.0
80.1
46.0
88.8
Creativity
Cref
90.8
88.0
76.6
35.1
86.5
Engagingness
Eng
88.8
88.7
78.8
29.6
90.0
Feasibility
Fea
90.7
87.3
82.5
50.2
91.4
Fluency
Flu
98.2
93.1
85.8
49.5
94.6
Helpfulness
Helt
94.7
90.3
79.6
46.6
90.3
Image Repetition
ImgR
86.5
71.0
43.1
37.3
81.7
Readability
Rea
94.0
90.4
81.5
53.1
92.4
Safety
Saf
84.5
79.5
53.0
31.3
76.1
Scené Consistency
SceC
90.5
78.6
35.0
28.6
88.1
Semantic Consistency
SemC
93.4
73.6
50.0
28.7
90.6
Stylistic Consistency
StyC
90.0
66.1
32.7
20.8
83.5
Toxicity
Tox
97.5
84.2
87.8
54.1
93.7
Table 35: The specific experimental results of the IG task in the FRA-ID-H are used evaluate alignment with human annotators.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2V1.-72B
Qwen2VI.-7B
Ours
Action Alignment
ActA
50.0
43.8
28.1
34.4
51.0
Alignment
Ali
67.5
70.0
47.5
45.0
55.0
Attribute Alignment
AIIA
62.5
62.5
52.5
37.5
65.0
Color Alignment
ColAt
65.0
60.0
57.5
37.5
49.5
Count Alignment
CouA
55.0
60.0
50.0
20.0
60.0
Fidelity
Fid
30.0
42.5
40.0
33.0
35.0
Harmfulness
Har
62.5
95.0
82.5
95.0
Object Alignment
ObjAt
52.5
57.5
50.0
27.5
57.5
Object Relationship
ObjR
60.0
50.0
45.0
32.5
52.5
Shape Alignment
ShaA
65.0
55.0
60.0
35.0
70.0
Spatial Alignment
SpaAt
57.5
60.0
37.5
27.5
57.5
Texture Alignment
TexA
52.5
60.0
50.0
50.0
62.5
42

---

Published as a conference paper at ICLR 2026
Table 36: The specific experimental results of the NLG task in the FRA-ID are used to evaluate alignment with GPT-4o.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2VL-72B
Qwen2VL-7B
Themis
Ours
Accuracy
Acc
93.2
82.1
70.3
49.4
43.3
88,2
Accuracy
Acct
92.3
73.1
76.9
42.3
53.8
69.2
Clarity
Cla
91.8
75.9
60.8
48.8
35.3
87.1
Coherence
Coh
90.8
82.0
70.4
47.8
39.6
88.4
Conciseness
Con
90.7
72.2
59.3
40.7
83.3
Contextual Information
Conl
94.6
91.8
87.1
48.3
44.2
90.5
Coverage
Cov
94.4
83.3
81.5
53.7
55.6
94.4
Creativity
Cre
88.3
81.7
79.2
47.5
54.2
82.5
Difference
Dif
92.3
76.9
69.2
53.8
46.2
96.2
Engagingness
Eng
90.0
92.5
85.0
54.2
62.5
91.7
Ethnic Bias
EthB
98.1
95.7
97.0
49.6
81.3
97.9
Example Quality
ExaQ
92.5
49.7
46.3
92.5
Fluency
Flu
91.8
77.6
70.0
49.3
39.8
86.3
Gender Bias
GenB
97.7
92.1
91.9
38.7
83.6
95.7
Grammaticality
Gra
89.5
47.7
66.5
Harmfulness
Har
96.7
71.7
85.0
58.0
70.0
93.3
Information Richness
InfoR
95.8
90.5
86.3
49.0
37.3
95.1
Layout
Lay
85.2
57.4
44.4
35.2
40.7
77.8
Length Constraint
LenC
50.0
46.3
Pointing Out
PoiO
100
53.8
69.2
38.5
65.4
84.6
Readability
Rea
93.3
81.7
74.2
56.7
48.3
91.7
Regional Bias
RegB
96.9
86.7
91.4
63.4
79.6
95.0
Relevance
Rel
93.9
85.0
78.2
51.0
43.5
87.8
Theme
The
86.7
79.2
70.8
51.7
48.3
82.5
Unlcrstandability
Und
90.1
79.1
65.4
52.5
32.7
89.4
Table 37: The specific experimental results of the IU task in the FRA-ID are used to evaluate alignment with GPT-40.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2VL-72B
Qwen2VL-7B LLaVA-Critic
Accuracy
Acc
89.8
83.6
75.3
54.5
75.9
85,5
Action Alignment
ActA
76.3
47.2
55.2
37.4
36.4
69.6
Alignment
Ali
76.2
69.5
75.5
80.0
Attrihute Alignment
AttA
89.5
86.8
69.0
51.7
65.0
84.5
88.4
22.1
Clarity
Cla
89.2
76.8
56.2
53.8
67.1
82.1
Coherence
79.2
66.5
47.7
74.8
Color Alignment
ColA
90.5
78.0
69.0
41.5
58.8
85.5
Completeness
Com
96.0
89.7
85.3
61.3
83,8
95.0
Conciseness
92.0
65.7
30.2
95.0
Context Inconsistency
Cont
90.0
77.5
64.7
44.3
81.2
84.3
Count Alignment
CouA
66.5
63.7
50.7
61.8
80.0
Fluency
Flu
80.5
68.8
67.0
53.7
62.7
73.0
Harmfulness
Har
99.5
98.1
98.5
76.1
37.8
99.5
Helpfulness
Hel
97.0
88.0
81.8
52.5
76.7
94.8
Instruction Following
InsF
88.8
73.4
65.9
48.4
64.0
83.7
Instruction Inconsistency
Insl
90.2
71.5
60.3
42.9
78.2
85,5
Object Alignment
ObjA
83.5
67.5
58.0
46.1
62.0
78.1
Privacy Violations
PriV
13,0
97.0
Readability
Rea
92.0
79.2
70.3
39.3
72.3
86.0
Sentence Complexity
SenC
57.3
37.4
89.7
Shape Alignment
ShaA
88.7
76.5
62.5
49.7
75.0
81.0
Spatial Alignment
SpaA
81.2
69.0
53.7
76.0
Spelling Accuracy
SpeA
97.5
89.4
88.4
45.4
21.2
95.7
Structure Accuracy
StrA
70.8
68.3
46.1
50.3
87.0
Text- Text-Image Relationship
T-IR
96.0
91.7
86.8
42.0
81.8
92.5
Texture Alignment
TexA
90.7
79.7
68.5
47.2
74.0
85.8
Vocabulary Complexity
VocC
83.8
63.5
61.6
51.1
64.7
79.5
43

---

Published as a conference paper at ICLR 2026
Table 38: The specific experimental results of the ITIG task in the FRA-ID-H used evaluate alignment with human annotators.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2VI.-72B
Qwen2VI.-7B
Ours
Action Consistency
ActC
$2.0
44.0
33.3
33.3
53.3
Bias
Bias
67.5
57.5
65.0
37.5
67.5
Character Consistency
ChaC
50.0
30.0
37.5
32.5
38.5
Clarity
Cla
87.5
87.5
82.5
55.0
90.0
Coherence
Coh
73.8
85.0
78.7
41.2
85.0
Completeness
Com
70.0
67.5
72.5
30.0
72.5
Completeness
Com
77.8
80.6
83.3
38.9
80.6
Creativity
Cref
62.5
62.5
47.5
35.0
67.5
Engagingness
Eng
80.0
82.5
75.0
32.5
77.5
Feasibility
Fea
86.1
91.7
83.3
44.4
91.7
Fluency
Flu
75.0
76.2
75.0
46.3
76.0
Grammaticality
Gra
62.5
60.0
62.5
41.2
64.0
Helpfulness
Hclt
80.6
83.3
75.0
38.9
77.8
Image Repetition
ImgR
61.3
42.7
41.3
36.0
66.7
Readability
Rea
65.0
67.5
62.5
38.8
67.5
Safcty
Saf
58.3
63.9
55.6
33.3
50.0
Scene Consistency
SceC
64.0
61.3
33.3
33.3
62.7
Semantic Consistency
SemC
53.3
48.0
44.0
28.0
56.0
Stylistic Consistency
StyC
65.3
45.3
32.0
36.0
53.3
Toxicity
Tox
82.5
90.0
92.5
37.5
90.0
Table 39: The specific experimental results of the NLG task in the FRA-ID-H used evaluate alignment with human annotators.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2VL-72B
Qwen2VL-7B
Themis
Ours
Accuracy
Acc
92.5
80.0
72.5
57.5
45
90.0
Accuracy
Ace
88.5
65.4
57.7
53.8
57.7
73.1
Clarity
Cla
64.7
64.2
52.4
49.2
47.6
67.9
Coherence
Coh
74.9
71.7
71.1
50.8
48.1
76.5
Conciseness
Con
51.2
31.7
53.7
Contextual Information
Conl
92.5
92.5
95.0
57.5
57.5
90.0
Coverage
Cov
68.3
68.3
68.3
34.1
53.7
68.3
Creativity
Cre
72.5
80.0
92.5
50.0
72.5
85.0
Difference
Dif
88.5
84.6
80.8
69.2
46.2
96.2
Engagingness
Eng
80.0
97.5
92.5
47.5
70.0
100
Ethnic Bias
EthB
96.7
91.7
96.7
56.7
74.2
94.2
Example Quality
ExaQ
87.5
47.5
$7.5
Fluency
Flu
71.1
69.0
65.2
46.5
45.5
73.8
Gender Bias
GenB
90.8
90.0
92.5
63.3
72.5
91.7
Grammaticality
Gra
71.4
47.6
64.6
51.0
56.5
70.1
Harmfulness
Har
91.2
76.2
85.0
62.5
75.0
91.2
Information Richness
InfoR
90.0
85.0
82.5
40.0
42.5
92.5
Layout
Lay
39.5
24.4
65.9
9.8
36.6
$1.7
Length Constraint
LenC
95.1
17.1
92.7
Pointing Out
PoiO
92.3
57.7
76.9
50.0
65.4
84.6
Readability
Rea
67.4
66.8
63.1
49.7
52.9
71.1
Regional Bias
RegB
92.5
93.8
61.3
76.2
91.2
Relevance
Rel
82.5
87.5
87.5
65.0
55.0
95.0
Sentence Complexity
SenC
68.3
57.8
52.8
43.5
49.1
64.6
Spelling Accuracy
SpeA
82.2
82.2
81.3
29.0
79.4
86.0
Structure Accuracy
56.1
43.9
Tense Consistency
TenC
88.5
50.0
80.8
38.5
80.8
76.9
Text Repetition
TexR
61.7
52.3
51.4
35.5
49.5
63.6
Theme
The
72.5
82.5
87.5
52.5
55.0
82.5
Understandahility
Und
80.0
77.5
80.0
65.0
45.0
87.5
Vocabulary Complexity
VoeC
64.6
57.8
47.2
48.4
44.1
61.5
44

---

Published as a conference paper at ICLR 2026
Table 40: The specific experimental results of the ITIG task the FRA-OOD are used to evaluate alignment with GPT-4o.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2VL-72B
Qwen2VL-7B
Ours
Action Consistency
ActC
90.0
62.5
32.2
42.2
71.1
Character Consistency
ChaC
85.6
71.3
20.0
24.4
81.1
Clarity
Cla
87.5
87.5
80.0
50.0
92.5
Coherence
Coh
92.2
95.0
92.2
68.9
96.7
Fluency
Flu
92.2
87.5
87.8
60.0
88.9
Harmfulness
Har
91.1
65.0
81.1
61.1
76.7
Image Coherence
ImaC
90.0
67.1
46.7
30.0
81.1
Image Repetition
ImgR
84.4
75.0
46.7
46.7
75.6
Readability
Rea
96.7
86.3
88.9
57.8
88.9
Scene Consistency
SceC
85.6
76.2
53.3
31.1
86.7
Semantic Consistency
SemC
74.4
73.4
53.3
32.2
84.4
Stylistic Consistency
StyC
88.9
61.3
37.8
16.7
73.3
Fulfillment
Full
90.0
95.0
85.0
42.5
95.0
Originality
Orig
84.0
70.0
74.0
50.0
80.0
Reasonableness
Reas
100
97.5
92.5
30.0
92.5
Simplification
Simp
90.0
77.5
84.0
52.0
90.0
Table 41: The specific experimental results of the IU task the FRA-ID-H are used to evaluate alignment with human annotators.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2VL-72B
Qwen2VL-7B LLaVA-Critic
Accuracy
Ace
70.5
73.0
68.5
62.5
71.0
75.5
Action Alignment
AciA
82.5
62.5
57.5
17.5
Alignment
Ali
77.5
72.5
69.5
55.0
67.5
70.0
Attribute Alignment
A11A
67.5
52.5
62.5
55.5
60.0
67.5
Bias
Bias
90.0
86.3
87.6
65.0
21.2
88.7
Clarity
Cla
63.1
57.5
48.1
59.4
50.6
61.9
Coherence
56.7
62.9
72.1
Color Alignment
ColA
82.5
50.0
Completeness
Com
62.5
62.5
62.5
56.5
62.5
62.5
Conciseness
Con
90.0
77.5
65.0
42.5
30.0
82.5
Context Inconsistency
Conl
72.5
67.5
67.5
55.0
62.5
70.0
Count Alignment
CouA
72.5
52.5
65.0
40.0
37.5
67.5
Ethnie Bias
EthB
97.5
97.5
97.5
75.0
22.5
97.5
Fluency
Flu
57.9
53.7
47.9
Grammaticality
Gra
82.5
41.9
80.6
57.5
28.7
83.1
Harmfulness
Har
97.5
100
96.8
90.0
35.0
97.5
Helpfulness
Hel
75.0
72.5
70,0
47.5
65.0
75.0
Instruction Following
InsF
74.0
65.5
68.0
51.5
64.5
78.5
Instruction Inconsistency
Insl
70.0
65.0
60.0
Object Alignment
OhjA
56.2
51.2
47.5
$2.5
55.0
56.2
Privacy Violations
PriV
92.5
67.5
15.0
92.5
Readability
Rea
69.2
67.1
63.3
52.5
61.3
67.9
Sentence Complexity
SenC
63.3
50.8
37.5
40.0
43.3
56.7
Shape Alignment
ShaA
72.5
60.0
65.0
$7.5
$2.5
72.5
Spatial Alignment
SpaA
70.0
77.5
62.5
65.0
77.5
Spelling Accuracy
SpeA
92.5
25.0
20.0
97.5
Structure Accuracy
SirA
87.5
80.0
60.0
$2.5
55.0
85.0
Text-Image Relationship
T-IR
80.0
82.5
77.5
55.0
80.0
80.0
Texture Alignment
TexA
70.0
57.5
70.0
52.5
47.5
70.0
Vocabulary Complexity
VocC
52.5
48.7
50.0
40.0
44.4
57.5
45

---

Published as a conference paper at ICLR 2026
Table 42: The specific experimental results of the ITIG task in the FRA-OOD-H are used to evaluate alignment with human annotators.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2VL-72B
Qwen2VL-7B
Ours
Action Consistency
ActC:
77.5
63.4
25.0
50.0
67.5
Character Consistency
ChaC
60.0
68.7
22.5
35.0
68.8
Clarity
Cla
82.5
90.0
90.0
47.5
87.5
Coherence
Coh
86.3
95.0
92.5
62.5
96.3
Fluency
Flu
90.0
95.0
97.5
65.0
95.0
Harmfulness
Har
73.8
71.3
67.5
65.0
72.5
Image Coherence
ImaC
87.0
67.1
46.7
30.0
81.1
Image Repetition
ImgR
70.0
53.6
47,5
45.0
71.3
Readability
Rea
85.0
90.0
91.2
60.0
91.2
Scene Consistency
SceC
76.2
85.5
58.8
38.8
81.2
Semantic Consistency
SemC
74.4
73.4
53.
32.2
84.4
Stylistic Consistency
StyC
62.5
75.7
40.0
20.0
68.8
Fulfillment
Full
95.0
97.3
82,5
40.0
97.5
Originality
Orig
92.5
92.9
77.5
55.0
85.0
Reasonableness
Reas
95.0
94.7
85.0
32.5
90.0
Simplification
Simp
90.0
79.3
90.0
50.0
90.0
Table 43: The specific experimental results of the IG task in the FRA-0OD are used to evaluate alignment with GPT-4o.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2VL--2BB
Qwen2VL-77
Ours
Action Alignment
ActAl
85.2
59.8
54.0
44.4
58.2
Aligament
87.2
64.8
58.2
39,2
67.8
Attribute Alignment
AttA
84.4
68.8
40.0
44.0
65.0
Color Aligument
ColAt
87.2
59.8
68.6
45.4
78.2
Count Alignment
CouA
84.1
62.8
45,4
36,9
61.1
Fidelity
Fid
82.2
59.4
41.6
43.0
$3.2
Harmfulness
Har
97.6
85.6
90.0
79.8
84,8
Object Alignment
ObjAt
87.3
64.0
41.6
26.5
60.2
Object Relationship
ObjR
85.2
62.6
39.7
27.5
61.0
Shape Alignment
ShaA
77.0
47.0
48.2
44.0
57.8
Spatial Alignment
SpaAt
86.2
59.0
39.6
31.0
59.2
Texture Alignment
TexA
85.0
70.0
37.0
41.6
62.6
Comparison
Cmpa
84.6
75.9
49,4
33.3
64.8
Negation
Nega
86.2
$7.3
49.6
34.0
$4.5
Scene Alignment
ScnA
84.6
65.4
51.0
48.8
65.2
Universal
Univ
88.4
63.9
43.5
34.7
66.0
Table 44: The specific experimental results of the IG task in the FRA-OOD-H are used to evaluate alignment with human annotators.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2VL-72B
Qwen2VL-7B
Ours
Action Alignment
ActAt
62.7
56.6
51.8
42.2
50.6
Alignment
Ali
75.8
61.2
55.1
41.0
65.22
Attribute Alignment
AttA
75.2
62.1
46.6
49.7
62,1
Color Alignment
ColAt
80.7
68.3
62.1
47.2
75.8
Count Alignment
CouA
68.5
74.1
48.1
27.8
66.7
Fidelity
Fid
71.4
$1.1
49.3
44.9
$4.2
Harmfulness
Har
94.3
80.6
89.0
77.5
82.8
Object Alignment
ObjA
64.8
68.5
46.3
27.8
53.7
Object Relationship
ObjR
72.0
62.2
35.7
26.6
63.6
Shape Aligument
ShaAt
72.0
50.3
46.6
48.4
55.3
Spatial Alignment
SpaAt
68.4
$4.1
34.7
34.7
53.1
Texture Alignment
TexA
75.2
65.2
39.1
43.5
59
Comparison
Cmpa
77.5
80.0
50.0
37.5
70
Negation
Nega
75.6
61.0
53.7
46.3
68.3
Scene Alignment
ScnA
71.4
57.1
49.1
58.0
58,9
Universal
Univ
72.5
62.5
62.5
42.5
67.5
46

---

Published as a conference paper at ICLR 2026
Table 45: The specific experimental results of the NLG task the FRA-OOD are used to evaluate alignment with GPT-4o.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2VL-72B
Qwen2VL-7B
Themis
Ours
Bias
Bias
95.9
64.1
83.5
51.8
69.4
85.9
Clarity
Cla
88.9
74.5
59.3
56.5
43.3
80.0
Coherence
Coh
86.0
80.4
71.1
54.0
42.4
79.6
Cultural Bias
CulB
89.5
$2.6
68.4
31.6
63.2
81.6
Ethnic Bias
EthB
97.4
97.4
97.4
68.4
86.8
100
Fluency
Flu
90.0
79.1
75.1
52.4
43.5
82.0
Grammaticality
Gra
88.4
64.5
68.7
47.9
66.8
81.6
Privacy Violation
Priv
94.1
90.0
90.6
64.1
50.6
92.4
Readability
Rea
89.2
74.3
62.5
53.8
40.6
84.7
Sentence Complexity
SenC
97.6
75.3
60.0
57.1
46.5
75.3
Spelling Accuracy
SpeA
93.5
82.8
75.6
30.5
73.7
87.4
Structure Accuracy
StrA
88.5
78.2
64.5
59.5
55.3
80.2
Tense Consistency
TenC
80.8
76.9
46.2
66.2
80.8
Vocabulary Complexity
VocC
66.5
46.5
62.9
Appeal
Appe
44.7
Difficulty
Diff
58.3
58.3
Engagement
Enga
90.9
86.4
47.7
65.9
Format
Form
87.6
84.7
61.8
57.4
Precision
Prec
88.2
77.1
58.8
61.5
82.4
Representation
Repr
92.5
83.8
86.3
57.5
52.5
86.3
Suitability
Suit
73.7
76.3
71.1
60.5
34,2
71.1
Thoroughness
Thor
88.5
88.5
83.8
53.8
72.3
87.7
Translation Fidelity
TmF
96.2
93.8
83.1
59.2
76.2
86.2
Coverage
Cové
92.1
76.3
76.3
42.1
39.5
78,9
Relevance
Rel
83.8
81.2
65.0
55.0
40.0
83.8
Table 46: The specific experimental results of the IU task in the FRA-OOD are used evaluate alignment with GPT-4o.
Aspect
Abbr
GPT-4o
Claude-3.5
QwenZVL-72B
Qwen2VL-7B
LLaVA-Critic
Ours
69.0
28.5
Clarity
Cla
96.7
80.5
63.5
63.0
68.1
88.7
Coherence
Coh
91.8
87.5
77.9
59.4
74.1
89.5
Fluency
Flu
94.7
86.1
81.6
58.4
75.8
88.5
Gender Bias
GenB
99.3
98.6
98.5
83.8
26.0
98.5
Grammaticality
Gra
88.4
75.5
63.1
61.9
87.7
Harmfulness
Har
94.8
90.7
91.5
71.8
33.0
92.2
Readability
Rca
93.9
86.8
62.2
75.0
86.3
Regional Bias
RegB
99.8
100
98.8
76.7
24.5
99.0
Sentence Complexity
SenC
94.5
75.5
71.2
53.4
60,2
85.9
Toxicity
Tox
98.5
93.6
94.8
72.0
31.2
93.2
Vocabulary Complexity
VocC
88.4
72.5
65.6
58.2
68.5
79.3
Completion
Comp
86.3
78.7
69.0
78.2
81.2
Correctness
Cor
78.5
82.5
Expertise
Expc
90.7
83.8
75.9
73.6
84.3
88.6
Explainability
Expl
93.0
86.3
84.5
74.9
81.8
90.4
Interpretabiliit
Intp
90.0
82.8
83.0
71.0
73.0
85.3
Riskiness
Risk
84.5
78.0
76.7
72.0
79.0
76.0
Terminology
Term
90.5
87.3
86.4
58.5
85.0
91.0
Transparency
Tran
91.0
91.0
88.2
55.8
82.8
86.2
47

---

Published as a conference paper at ICLR 2026
Table 47: The specific experimental results of the NLG task the FRA-OOD-H are used to evaluate alignment with human annotators.
Aspect
Abbr
GPT-4o
Claude-3.5
Qwen2VL-72B
Qwen2VL-7B
Themis
Ours
Bias
Bias
83.3
62.8
79.5
53.8
75.6
76.9
Clarity
Cla
71.1
72.6
65.5
51.3
47.2
76.1
Coherence
Coh
75.1
80.7
72.1
39.1
44.2
78.2
Cultural Bias
CulB
87.5
$2.6
68.4
31.6
63.2
81.6
Ethnic Bias
EthB
95.4
93.7
93.7
68.4
86.8
94.3
Fluency
Flu
78.2
81.2
72.1
48.2
42.6
74.1
Grammaticality
Gra
81.5
66.2
68.8
50.3
64.3
79.6
Privacy Violation
Priv
92.5
87.5
90.0
70.0
60.0
87.5
Readability
Rea
77.1
68.6
35.6
39.0
76.3
Sentence Complexity
SenC
65.0
60.0
55.0
40.0
57.5
67.5
Spelling Accuracy
SpeA
75.9
39,2
74.7
81.0
Structure Accuracy
StrA
70.9
51.9
60.8
72.2
Tense Consistency
TenC
92.3
89.7
46.2
66.7
87.2
Vocabulary Complexity
VocC
67.5
65.0
52.5
45.0
37.5
Appeal
Appe
76.3
34.2
Difficulty
Diff'
80.0
47.5
Engagement
Enga
80.0
47.5
Format
Form
87.5
82.5
57.5
67.5
Precision
Prec
92.5
95.0
82.5
60.0
77.5
72.5
Representation
Repr
85.0
82.5
72.5
62.5
50.0
80.0
Suitability
Suit
78.9
76.3
73.7
47.4
31.6
81.6
Thoroughness
Thor
87.2
89.7
82.1
38.5
71.8
76.9
Translation Fidelity
TmF
87.2
87.2
87.2
51.3
76.9
84.6
Coverage
Cové
81.6
76.3
71.1
42.1
47.4
81.6
Relevance
Rel
72.5
67.5
72.5
52.5
57.5
72.5
Table 48: The specific experimental results of the IU task in the FRA-OOD-H are used evaluate alignment with human annotators.
Aspect
Abbr
GPT-4o
Claude-3.5
QwenZVL-72B
Qwen2VL-7B
LLaVA-Critic
Ours
57.5
30,0
Clarity
Cla
90.0
83.8
67.5
65.0
73.8
85.0
Coherence
Coh
78.3
75.8
73.3
66.7
71.7
76.7
Fluency
Flu
78.7
76.2
72.5
58.8
62.5
76.2
Gender Bias
GenB
96.3
93.6
90.0
83.8
26.0
95.0
Grammaticality
Gra
85.8
72.5
62.5
$9.2
Harmfulness
Har
80.0
67.5
45.0
Readability
Rca
80.0
80.0
68.3
66.7
74.2
79.2
Regional Bias
RegB
96.5
95.9
90.0
76.7
24.5
95.0
Sentence Complexity
SenC
66.2
68.8
68.8
46.3
58.8
68.8
Toxicity
Tox
96.5
93.6
91.0
72.0
31.2
93.2
Vocabulary Complexity
VocC
67.5
62.5
67.5
48.7
50.0
68.8
Completion
Comp
85.0
85.0
82.5
72.5
75.0
82.5
Correctness
Cor
82.5
70.0
80.0
82.5
Expertise
Expc
82.5
67.5
65.0
72.5
80.0
85.0
Explainability
Expl
82.5
72.5
77.5
70.0
75.0
87.5
Interpretabiliit
Intp
85.0
77.5
87.5
65.0
67.5
75.0
Riskiness
Risk
85.0
82.5
67.5
67.5
75.0
80.0
Terminology
Term
75.0
70.0
72.5
55.0
72.5
75.0
Transparency
Tran
80.0
55.0
85.0
80.0
48

---

Published as a conference paper at ICLR 2026
Table 49: Comprehensive results on MLLLM-as-a-udde *: The results for GPT-4V and LLaVA-Critic are taken from the original papers (Chen et al. 2024) and (Xiong et al. 2024), respectively. All other results are obtained from experiments using their publicly available codebases.
Serings
MELM
cnen
FSF
Coaphics
Marh
WIT
VieIT
CC-M
ScQA
Acx
MM-Vet
GPT-4
09.6
92.4
84.7
63.9
36.4
67.9
66.7
64.10
52.1
41.5
50.6
03.6
LaVVe-Ciic
70.2
58.7
45.2
56.4
32.0
69.6
46.4
UP1-40
66.0
83.
55.7
47.3
60.0
61.0
41.7
36.5
Clande 3.5
56.5
66.3
55.6
61.5
46.4
53.9
63.3
37.4
64.5
00.1
55.4
38.7
72.2
52.3
59.4
11
Uw002VE-1218
733
590
46.0
18.
29
39.3
66.7
Quen2ViL 7B
34.5
31.3
00.6
57.1
25.9
38.0
31.9
40.6
38.3
57.4
53.3
46.7
38.5
55.5
41.3
ULVA--nni
67.7
61.8
48.8
50.2
61.4
38.9
Oues
58.0
67.9
73.7
61.5
47.3
51.3
52.1
37.8
02.2
63.5
62.3
43.8
59.7
49.4
57.2
GPT-Y
60.4
$7.0
92.3
80.1
50.
50.5
73.4
$4.90
76.1
70.3
69.9
64.7
75.5
65.9
77.3
GPT-4
75.6
76.4
$7.7
75.8
60.3
60.5
71.9
50.0
75.3
70.3
68.5
60.8
72.9
$4.6
68.6
QwarVVL-228
48.5
69.7
74,8
73.8
51.0
-9.8
48.5
47.5
70.0
70.9
64.7
45.3
58.0
36.1
58.5
LLaVA Cétic
71.1
76.0
68.2
78.2
57.5
57.9
72.2
49.8
71.8
75.9
65.6
52.8
59.7
00.3
05.5
70.3
Oues 72B
71.0
73.1
87.8
79.0
02.4
57.1
68.5
47.9
75.6
73.1
67.6
19.2
74.2
53.8
73.1
Table 50: The specific experimental results of multi-aspech assessment learning in the IU tasks.
Aspect
Abbr
QuenVVL-7B
IU
w/ TITTTTG
w/ [1/46
Oum
Accuracy
Acc
54.5
NI.1
X3.1
K5.3
35.5
Action Alignment
ActA
37.4
65.0
69.8
09.3
69.5
Alignment
Ali
42.2
75
77.6
$0.0
80L0
Auribuie Alignment
51.7
77.2
79.0
81.5
84.5
Color
78.5
85.5
Conciseness
31.8
90.5
Context Incemistency
Coel
44.3
80.7
$4.0
84.3
Count Alignment
CoLA
50.7
75.0
75.7
70.5
80.0
Helpfulness
Hel
53.5
90.7
91.2
93.0
94.8
Instruction Following
InsE
48.4
$1.7
82.7
S1L1
83.7
Instraction Inconsistancy
Inst
42.9
76.7
79.2
B4.0
85.5
Object Aligrment
ObjA
46.1
68.8
75.5
77.5
78.1
Alignment
ShaA
Spuiia Alignmast
Relationshpp
Texture Alignmeut
47.2
780
Table 51: The specific experimental results of multi-aspect assessment learning in the IG tasks.
Aspect
Qwen2V1L-7B
Ours
Accuracy
Acc*
35.9
67.8
68.4
77.2
77.0
Action Aligament
AcIA"
48.8
70.9
71.5
72.8
73.8
Alignment
39.1
70.6
69.7
Attribute Alignment
AnA
40.7
64.9
66.7
71.0
71.3
Color Alignment
ColA
40.7
68.1
70.3
74.6
76.6
Aligument
Ficklity
Fid
40.8
44.0
42.0
44.0
44.0
Harmfulness
Har
76.1
76.5
78.7
77.8
77.7
Object Alignment
ObjA1
$3.5
65.3
70.0
71.7
73,3
Object Relatiombip
OlyR
38.9
63.0
64.7
68.1
70.4
Shope Alignment
ShaA!
32.0
$4.1
$8.2
60.2
62.2
Sputial Aligmment
Texture Alignment
TexA
37.4
56.7
38.4
60.2
61.3
Table 52: The specific experimental results of multi-aspect assessment learning in the ITIG tasks.
Aspect
Albe
Qwen2VL-78
w/ WTIG
w/ w/TTGG4
w/TTIGGK
Durs
Action Consistancy
Act!
24.5
71.1
X3.7
54.7
Chancter Comistincy
Chat'
14.2
64.1
67.1
711.4
72.8
Completetess
Coml
38.2
87.0
87.6
87.8
$7.6
Completenes
Com-
46.0
87.0
88.5
89.0
38.8
Craatiity
Engagingness
Eng!
Fousbiliit
90.6
Helpfulness
Hell
95.6
87.
90.5
90.3
Image Repecition
ImgR
37.3
73.9
750
80.5
$1.7
Safety
Sur
31.3
69A
713
72.5
76.1
Scene Consisency
Scet'
25.6
77.2
799
85.6
38.1
Semantc Cansstency
SonC
28.7
80.4
85.
80.7
90.6
Stylicic Consistency
StyC
208
75.7
80.1
82.0
$3.5
49

---

Published as a conference paper at ICLR 2026
E HUMAN ANNOTATION
The user interface used for human annotation is shown in Figures 16 17 For each aspect, we adopt 5-point Likert scale. Additionally, account for cases where certain instructions are not applicable to TAs, we include a "-1" option to indicate inapplicability. To maintain high-quality annotation, we engage three humans annotate each data point concurrently. These humans, recruited from graduate students to enhance annotation reliability, are provided with guidance whenever they encounter challenges during the annotation process, enabling them focus on multiple aspects such as helpfulness, clarity, comprehensiveness, and more.
Please carefully follow the task and the metrix descriptions to score.
Task: General Visual Question Answers
Query
people image firemen
Request1
Request2
The uniformed people the image firemen. This indicated by the presence the fire truck the background and their uniforms, which typical firefighters.
The uniformed people the image firemen.
Task-specific Aspects
When task specific aspects cannot be evaluated the current sample, please click the -1 button to mark it as -1.
Helpfulness
should bo
contributes positively user' comprehension image. Assess whether accurately any specific instructions. or guidelines provided prompt. overall contribution the response the experience.
Consider whether generated text provides valuable insights, additional context,
Instruction Following
should be
Instruction Following defined as the ability to accurately interpret and execute the task as outlined the given instruction, focusing solely on providing g the correct answer. measures whether the response directly addresses the task's care requirements, without considering additional background information, explanations, tone, or length. Even single word answers the question correctly, the response should receive high score on this criterion.
Figure 16: The annotation user interface for labeling the human scores on TAs.
50

---

Published as a conference paper at ICLR 2026
Universal Aspects: focus on the response merely Clarity
Clarity easily the understood. emphasizes the straightforward, unambiguous language, avoiding complex vocabulary and convoluted sentence structures that could confuse the reader.
Harmfulness
This criterion evalluates whether the generates content that promotes, glorifies, or depicts any form of violence, abuse, or harm be it physical, emotional, or psychological. It includes content that may cause distress, trigger trauma, or encourage harmful behavior toward individuals or groups, such as self-harm, harm to others, or dangerous actions, images that cause psychological discomfort seen harmful.
Fluency
Fluency measures the smoothness the age and sentence construction, ensuring the text flows seamlessly and naturally, phrasing, and avaids
grammatically correct.
transitions.
Coherence
logical consistency interconnection ideass within ensures that the follows clear, organized progression, with each section logically linking the next. A coherent text allows the reader easily follow the author's argument narrative, with smooth transitions and no sudden shifts in topic or in reasoning.
Grammaticality
Measures how well the text adheres to the rules of grammar, including sentence structure, verb tense, subject-verb agreement, and punctuation.
Readability
Readability measures how easily text is understood by its intended audience. It focuses on overall fluency and the clarity of ideas to ensure they are presented simply and align with the readers' comprehension levels. It evaluntes the quality of both Inter- and intra-sentences,, checking they are grammatically correct, naturally written, with clear meanings, and good context- relatedness logic, allowing absorb information effortlessly.
overall
an overall rat.rag considering metrix above.
seve
Figure 17: The annotation user interface for labeling the human scores on UAs.
To ensure the robustness data, we computed the Inter-Annottoor Agreement (IAA) grouped by task using Cohen's Kappa, with detailed results presented in Table 53 Table 54 The average IAA scores among the three annotators are 84.2, 87.4, and 90.8 for FRA-ID-H, and 84.0, 81.2, and 78.2 for FRA-OOD-H respectively. These results demonstrate high level of reliability in test dataset.
Table 53: Inter Annotator Agreement each annotator on the FRA-ID-H.
Table 54: Inter Annotator Agreement of each annotator on the FRA-OOD-H.
Annotators
NLG
IU
IG
ITIG
Annotators
NLG
IU
IG
ITIG
human 1-2
79.0
85.6
76.8
95.7
human 1-2
80.3
95.0
76.1
84.7
human 1-3
87.2
90.2
80.8
91.4
human 1-3
88.1
81.3
73.9
82.4
human 2-3
91.6
95.3
92.7
95.6
human 2-3
77.7
79.1
77.4
78.6
51

---

Published as a conference paper at ICLR 2026
We also computed the aspect-level IAA using Krippendorff's Alpha (a),, providing grained measure of agreement. The detailed score distributions are visualized in Figure 18] and the exact values are reported in Table 55 and Table Experimental results: (1) The FRA-ID-H has mean a of 0.87. All aspect scores are above 0.69, including subjective concepts such as Creativity. Specifically, 34 aspects scored above 0.80, 22 aspects between 0.70-0.79, and aspect between 0.69-0.70. (2) The FRA-OOD-H has a mean a of 0.81. The lowest score is 0.60, observed for subjective aspects such as Appeal. distribution, 30 aspects scored above 0.80, 22 aspects between 0.70-0.79, and 9 aspects between 0.60-0.69.
fine-
Comparison of Alpha Values Distribution
1.0
0.8
Alpha Value
0.6
0.4
(BAD-- Detor it
0.0
FRA-ID-H Dataset
FRA-0OD-H Dataset
FRA-ID-H Dataset Distribution
FRA-OOD-H H Dataset Distribution
Modorto 20
Miconix-00 (6733/00))
Alpha Value
0.8
Alpha Value
0.8
0.6
0.4
0.4
0.0
0.0
Aspects
Aspects
Figure 18: Box plots of aspect-level IAA on FRA-ID-H and FRA-OOD-H.
Table 55: Detailed Inter-Annotator Agreement (IAA) values for FRA-ID-H.
Part
Part 2
Part
Aspect
Count
Aspect
Count
Aspect
Count
Stylistic Consistency
80
Coverage
41
0.73
Reudability
504
0.76
Charucter Consistency
0.90
Length Constraint
1.00
Spelling Accuracy
145
0.95
Action Consistency
80
0.90
Conciseness
81
0.78
Structure Accuracy
145
0.89
Scene Consistency
80
0.93
Clarity
384
0.76
Text Repetition
106
0.95
Image Repetition
80
0.96
Grammaticality
386
0.89
Accuracy
266
0.99
Semantic Consistency
Coherence
504
0.72
Pointing Out
26
0.96
Feasibility
40
Fluency
504
0.72
Difference
26
0.90
Safety
40
1.00
Sentence Complexity
278
0.81
Tense Consistency
26
0.98
Theme
39
0.80
Vocabulary Complexity
318
0.87
Creativity
79
0.69
Ethnic Bius
97
1.00
Helpfulness
0.79
Engagingness
0.78
Nationality Bias
Instuction Following
0.90
Count Aligament
77
0.95
Harmfulness
199
0.79
Instruction Inconsistency
1.00
Spatial Alignment
80
0.92
Gender Bias
79
1.00
Context Inconsistency
40
1.00
Attribate Alignment
49
0.78
Relevance
38
0.75
Privacy Violations
40
1.00
Color Alignment
80
1.00
Contextual Information
38
Bias
0.95
Texture Alignment
79
0.77
Explanations
Object Alignment
Shape Alignment
0.77
Information Richness
40
0.78
Text Bext-Image Relationship
0.76
Action Alignment
69
0.75
Understandahility
40
1.74
Alignment
80
0.76
Object Relationship
19
0.73
Toxicity
40
1.00
Completeness
120
0.77
Fidelity
40
1.00
52

---

Published as a conference paper at ICLR 2026
Table 56: Detailed Inter-Annotator Agreement (IAA) values for FRA-OOD-H.
Part 1
Part 2
Part 3
Aspect
Count
Aspect
Count
Aspect
Count
a
Cultural Bias
37
1.00
Vocabulary Complexity
120
0.79
Regional Bias
1.00
Translation Fidelity
39
0.73
Relevunce
38
Gender Bias
1.00
Thoroughness
39
0.73
Representation
38
0.69
Toxicity
40
1.00
Tense Consistency
39
0.85
Expertise
40
0.75
Terminology
40
0.80
Spelling Accuracy
39
1.00
Correctness
40
1.00
Transparency
0.78
Structure Accuracy
39
0.90
Explainability
40
0.80
Color Alignment
161
0.86
Precision
40
0.90
Riskiness
40
0.82
Shape Alignment
161
0.75
Format
40
Completion
40
Texture Alignment
0.70
Privacy Violations
40
1.00
Inteppetahilityy
40
0.71
Attribute Alignment
0.74
Action Alignment
83
0.88
Harmfulness
347
0.71
Scene Alignment
112
0.89
Object Relationship
143
0.82
Fulfillment
40
0.76
Readability
0.71
Fidelity
227
0.95
Reasonableness
40
0.72
Fluency
0.68
A Alignment
227
0.67
Character Consistency
80
0.81
Coherence
354
0.73
Spatial Alignment
98
0.75
Image Repetition
80
0.80
Grammaticality
234
0.80
Count Aligmment
54
Consistency
80
Coverage
0.65
Object Alignment
54
0.88
Action Consistency
80
0.85
Appcal
37
0.60
Comparison
40
0.73
Stylistic Consistency
79
0.72
Suitability
37
0.75
Negation
Semantic Consistency
79
Clarity
0.64
Universal
40
0.73
Imuge Coherence
40
0.96
Bias
77
1.00
Simphfication
40
0.68
Originality
40
0.65
Ethnic Bias
37
1.00
Sentence Complexity
120
Finally, to verify the distinctness among the evaluation aspects, we conducted both qualitative and quantitative analyses.
Qualitative Analysis: To ensure distinctness and avoid overlap, we constructed our taxonomy by categorizing evaluation aspects into two distinct groups: Universal Aspects (UAs) and Task-specific Aspects (TAs):
UAs: Applicable across diverse output formats, where aspects associated with distinct outputs (e.g.. text, image, text-with-image inherently different due to disjoint feature spaces. Within each output format, aspects are organized into an Aspect Tree with defined parent- child and sibling relationships, avoiding definitional overlap by design.
TAs: These aspects are defined specifically for each task. Even if some aspects share semantic similarities, their definitions shift according to the specific evaluation task. This context-dependence allows them to be effectively distinguished by task.
2. Quantitative Analysis: To quantitatively validate the distinctness of proposed aspects, assessed their distinctness by computing Kendall's Tau (r) correlations within each task. Correlation heatmaps are shown in Figures to to 22 with statistical summaries shown in Table 57 Note that we exclude chart reasoning (which contains only two aspects) and FRA-ID-H text-to-image (where each sample annotated with a single aspect, precluding pairwise correlation analysis).
FRA-ID-H. Mean correlations across tasks range from 0.131 to 0.609 (e.g.. Medical Image Understanding: 0.131; Detailed Captioning: 0.16)). While certain specialized tasks like Chart Reasoning (0.609) and Text-to-Image (0.576) exhibit higher correlations, they remain well below the redundancy threshold (>0.8).
FRA-OOD-HE Mean correlations range from 0.240 (Title Generation) to 0.518 (Pmc). Most tasks, such as Data Analysis (0.256) and M3it (0.298), maintain owwtto-moderate correlation values.
Based on the above analysis, aspect correlations are consistently low across both datasets, supporting the distinctness of our taxonomy.
53

---

Published as a conference paper at ICLR 2026
Table 57: Statistics of Aspect-level Correlation
FRA-ID-H
FRA-ODI-H
Tosk
Sub- Tusk
N_Asp
Mean
Max
Task
Sub-Task
N_Asp
Mean
Max
Chart Reasoning
0.609
Imaaee
16
0.352
0.723
Detalled Caprioning
18
0.161
0.597
TU
Mitt
9
0.298
0.366
General VQA
0.291
0,682
Pesevul
12
0.355
0.593
Medical Image Und.
0.131
0.66
Finc
05188
0.099
Robustnes Inst.
9
0.305
0678
FIG
Activiynne Cun..
13
0.370
0.694
Simple Capissning
0:310
Steetelling
13
Text Rish Und.
9
0.380
0.600
NLG
Data Analyii
9
0.256
0.581
MI
Viet
15
0:250
0.650
Title Gen.
11
01240
0.597
Wikiluse
14
0.406
Transiation
NLG
Creaniv Winting
15
0:257
0.678
Question Cen
9
0:463
0.681
Punctional Winting
12
0.503
0,705
Keywords Ext.
7
0.399
0.699
General Comm.
14
0.328
0,500
Rewitting
12
0.332
0.44
Summanization
IG
Text To Image
12
0.575
0,656
Kossall's Condistion Interieaced That and image Acdirity net Continuation on (/wil sompass
Koneaalls Natural Language Genenatie ""eee Cenerattin
O1w-db samples!
Crnes
Aspech
Konsal's Du Comciition (jwil) somplesi
Kendarts Tu Comdettinn $1 Samprest
Storytaliing
AZI
Kordat' Tr Correation
Kantal's Tou Conecaiin modentandirgg/
Image Undersanding Samola 01e4D plo)
Figure 19; Heatmap of Kendall's Tau (7) correlations between evaluation aspects across different tasks. (Part 1)
54

---

Published as a conference paper at ICLR 2026
Kendall's Tou Correaation Image Understarding / Medical image Understanding
Kendail's Correlation Natural I Language Generation, / Wi Writing samples)
[n-400 samples!
Vocabaaary Conta
Sensecce Complesity
A.0
1000
MEpeCT
Ruerry
A344
Colerance
Teactolity
Next Repertion
Grammefiiitty
Uacaedayy Dampberity
Moueeaa
Homenn Complsuu
Hewny
Meselitt
teceie Comylcap
Comtonutty
Roose
cheff
de
Appects
Aggeets
Kendalls Tou Correlatin
Kendallss Tbu Comeletion
Language Ino samples!
Writing
Image Understanding Image Captioning
(n-40 samples)
Gortectal
875
Clarlyy
250
Def mrMMA
225
Gencar Bea
2.00
Colereecc 5435334 531030
Ethvic Seaa-
-425
4.25
Kaadaality
Speling Aooureyy--a --0
deaccalry -33
051
6.7
Voodulayy Comples 2004 D400R00 secocct douceeae 40000
0.9
Semene Complet -004 20 wall So
Complriity
Complerly
Reppeene
feerece Concititty
Grean
Speiiig
Compan
Abnos
Aspects
Kandailss Tou Comelation
Kandall's Tau Corretation Ir interlaaved Text aaa---d--mm didd-age Goneration Vist (n=40 samp
Interieaved Text and magge Gencration Wikchow 4n=40 samples)
Ball
Flerro
Image
Todeoy-
Mylins sateray
Candlrano
RepecTs
Powildi
Cohemroe 000
Repettion
Scae Canatranc Ko
E17
XXC
dan
Insrge
soder
Suaia
Aspects
Appecis
Figure 20: Heatmap of Kendall's Tau (7) correlations between evaluation aspects across different tasks. (Part 2)
55

---

Published as a conference paper at ICLR 2026
Kendell's "au Correlation Netural Language cnneenttio/// Rewriting (n-26 samples]
Kendail's Corrletion Image Understanding / TTsst Rich Understanding (n-i0 samples)
1.00
3.73
-338
6:
Speting AALAAiY
0.3
Heedaall
Certtyy
Commedalyyy
9.00
Voniblaay Completryy
C-
0.29
6/
4.13
201
6.54
300
Orammnaiooa Ry.
A41
8:33
Accossy
opection
deeetan Acules
Accnert
Elary
Paant
Reciaity
Appests
Aggects
Kondalf's Visual Question Answers
Comelation
Understanding
Image Understanding (h=40 samples)
Oriented Instructions
(n-40 samples)
C75
Stuctur Accinco
6/1
6.10
Railiy
200
8/8
-223 Bac
6:0
403
237
Coneracce
241
241
restuction Brussconc
Centect ancuestiinn
dew Acones
Hevor Meltoo
Audaitte
Rammy
repeomerr
Ressling
dewingg
Kandal's Tau Comellaiin
Kendal's Tau Conclation
Natural Language Goneration on Summartzation [n-41 samples)
Natural Language Generation Genera Communication
40-40 samples)
Chargg
Length
enuticallo
Ixprects
-459
Speling Accu
-6.75
Semanne Complerry
403
Stuttor
Vboablaary Comp enry
Lango Coeraie
Aleny
Kysting Accerey
Clatee
Benry
scockin
Iotarattoon
Appects
Appects
Figure 21: Heatmap of Kendall's Tau (T) correlations between evaluation aspects across different tasks. (Part 3)
56

---

Published as a conference paper at ICLR 2026
Correlation
Natural Language Ceneralion Data Anelysis
Undersannding M.lt (n-40 samples)
(nm40 samples)
Perbice
Sxperter
3.40
1.00
7015
100
0.31
Fapla
Aolrden
Puancy-
A27
SapeCt
0000
Carte
415
0/8
838
6:M4
KM
011
829
6:2
6.3
Vocilaalee Con plaary
044
34r
Sen
344
6.1
0.0
613
GM
Caleente
Piaca Intaree
Eeensbly
Neteice Dempltal
Herry
wostsa Compituad
Adpects
Aspects
Kendal's Tau Correlation
Kendall's Tau Correlation (n- samples)
language (no- samples)
Keywords
Image
Talevace
Calor dlge mant G.A
marion
Toraue dage sesl can
Neacabiity
8:50
Sccme Alge mant
8:25
Axpocto
Ciject
Faally
Gramewicalt
-006
Otjest day mort
4.5
Segetion
.L/C
depette
Con sapprree
Kendall's Tau Correlation
Kendall's Tou Correlation
Image Unneerstandngg Peseval
Natural Language Generation Question Generation
samples)
Drgaparantt
Complatoo
3.30
8.55
Coleeece
8.50
Carly
005
Aguitt
Hamryy
0.00
0.08
-02
-2.5
Reacablto
4036
6.0
0.28
331
K25
orairal
4.75
Speling Acconey
6.1
0.8
Boddy
Cootabiyy
Coner Bas
Rosdur
Appccs
Figure 22: Heatmap of Kendall's Tau (7) correlations between evaluation aspects across different tasks. (Part 4)
57

---

Published as a conference paper at ICLR 2026
F IMAGE UNDERSTADING & GENERATEION MODEL ALIGNMENT USING DPO
Using generalist evaluator as a judge to generate preference datasets for reinforcement learning is a promising research direction. In this section, we elaborate the methodology leveraging UFEval to construct Al-generated preference datasets for both image understanding and generation tasks, and subsequently apply Direct Preference Optimization (DPO) to facilitate model alignment with human preferences through direct optimization on ranked preference pairs, thereby eliminating the necessity for explicit reward modeling.
F.1 DPO FOR IMAGE UNDERSTANDING ALIGNMENT
Leveraging DPO for improving image understanding models, i.e., MILMs, has been widely explored in recent research. DPÔ is a novel approach that aligns language models with human preferences without requiring a separate reward model, unlike traditional methods such PPO-based RLHE, Instead of explicitly training a reward model, DPO directly optimizes the policy by implicitly modeling the reward function through a simple classification-lik objective on preference pairs:
BB log (108 To (Hw xx) a:)
Te (y)
2)
((0) = (x. -000==--r...
log
Troe (Mo
where you is a preferred sample and y/t is a less preferred sample from preference pair dataset D, respectively. (y, x) and Tref (y+ 22) is the response probabilities under the fine-tuned model and pre-trained reference model, respectively. Su is a temperature hyperparameter that controls optimization sensitivity.
This objective guides the fine-tuned MILMs to assign higher probabilities to favored outputs and lower probabilities to unfavored ones, effectively aligning the model with human expectations and boosting reasoning performance.
F.2
DPO FOR IMAGE GENERATION ALIGNMENT
While diffusion models have emerged as the leading approach for image generation tasks, the reliance on traditional evaluation metrics such as FID has led to discrepancy between the quality of generated images and actual human preferences. To bridge this gap, researchers (Wallace et al., (024) have implemented DPO within these diffusion-based frameworks to improve the alignment between model outputs and human preferences.
Given the constructed preference pair datasets D {xx)" 3 where 20 and zá represents the preferred sample and the less preferred sample respectively. M represents the number of samples, we optimize the diffusion model by comparing the noise prediction differences between a fine-tuned model and pre-trained reference model following:
I(0)= - E (xy.= -- ((0
logo - -B,TW (Ne) ||E" (x,", t))1 2 11E"
Esel (x) t)l12
(2)
(le'
(2t 112 Il'' -rer Eref (26 (1(.))))))
where zt" and ar, are the noisy latents derived from z and af timestep respectively. eg (2f, t) and Cref (x; t) denote the predicted noise from the fune-tuned and pre-trained reference diffusion models, respectively. 3y a temperature hyperparameter controlling optimization strength, the logistic function, A represents the sional-to-noise ratio, and T>():) is a weighting function, which treated as a constant equal to 3, this work.
This loss function guides the fine-tuned model to minimize denoising enrors on preferred samples and maximize them on less preferred ones, effectively enhancing the overall generation quality.
58

---

Published as a conference paper at ICLR 2026
G THE PROMPT TEMPLATE
In this section, we provide the prompt template used for GPT-4o annotation and training/infereccing UFEval. Note that employed distinct prompt templates for UAs and TAs to enable specific aspect evaluation while avoiding overall assessment.
Table 58: The prompt template used evaluate UAs for multi-image outputs.
You will be given two responses, each generated by a different model. Your task is to evaluate both responses based on the given criterion and determine which one is better. or if both are equal. Each response contains a sel of images. The first set, consisting [response _iiaaaaaeouuu followed the [response2 image_ count images generated by the second model. Please carcfully divide the images into two sets. Here is the data:
images generated by the
model, will provided first.
[BEGIN DATA]
#*#
[Criterion]: (criterion_descriptig deseription:
#*
[END DATA]
Here are the instructions to assess and compare the two responses: 1. Carefully review every detail of images and given criterion write detailed leedback that
which of the two sets of images betler equally good, strictly based on the given criterion. Do not evaluate them in general terms or based on factors unrelated to the given criterion.
2. After writing the feedback, assign two integer scures (ranging from to 5) for the two responses (higher means better). Be sure to base your scores solely on the criterion provided. 3. The output format should look follows: [Feedback]: (Write feedback strictly according to the criterion). [Result]: (The two scores separated space).
Table 59: The prompt template is used to evaluate TAs in the NL. tasks.
You are tasked with evaluating two responses generated based on a given query, according to the provided criterion, and determining which better both equal. Here the data:
[BEGIN DATA]
#A#
[Criterion]: (criterion (crierion_deecciiption)
###
[Query]: [query)
###
[Response 1]: [response_]] []
#AW
[Response 2]: (response_2)
###
[END DATA]
Here are the instructions to ussess and compare the two responses: 1. Review the two responses relation the given query and write detailed feedback that assesses which of the responses is betler equally good, strictly based the given criterion. Do not evaluate them general terms or bused on factors unrelated to the given criterion. For example, if the criterion is clarity, focus solely on how clear the response is. ignoring whether the response accurately addresses the qucry.
for the two base your scores solely the criterion provided. 3. The output format should look follows: [Feedback]: (Write feedback strictly according to the criterion), [Result]: (The two scores are separated by a space).
2. After writing your feedback, assign integer scores (ranging from responses (higher means better). Be
59

---

Published as a conference paper at ICLR 2026
Table 60: The prompt template is used evaluate TAs the IU tasks.
You will be given an image and corresponding query. You are tasked with evaluating two submitted responses hased on the given criterion and determining which one is better, or if both are equal. Here the data:
[REGIN DATA]
[Criteroo]]: centerion_descrpttinn
[Query]: [query)
MMW
[Response 1J: [response_]]
##
[Response 2): [response_ 2]
#A#
[END DATAJ
Here the instructions to assess and compare the two responses:
1. Carefully review the query and the corresponding image, as well as the two responses, and write detailed feedback that assesses which the two responses is betler or equally good, strictly based on the given criterion. Do not evaluate them general terms based on factors unrelated the given criterion. For example, how clear the respunse ignoring whether the response accurately addresses the query. 2. After writing your feedback, ussign two inleger scores (ranging from I to 5) for the two responses (higher means better). Be sure to base your scores solely on the criterion provided. 3. The output format should look as follows: [Feedback]: (Write feedback strictly according criterion), [Result]: (The two scores are sepurated by a space).
criterion clarity, focus solely
Table 61: The prompt template is used to evaluate TAs in the IG tasks.
You will be given two images generated by two models based on the image description. You are tasked with evaluating two submitted images bused on the given criterion and determining which one better, or both are equal. Here is data:
[BEGIN DATA]
###
[Criterion]: desrrippddssiiiiion
###
[Image Deseription]: (image description)
###
[END DATA]
Here are the instructions to assess and compare the two images:
Carefully review every detail of two images, the image deseription, the given criterion to write detailed assessment determining which of the two images better if they are equally good, strictly based on the provided criterion. Do not evaluate them in general terms or based on factors unrelated to the given criterion.
2. After writing the feedback, assign two inleger scures (ranging from to 5) for the two images (higher means better). Be sure base your scores solely on criterion provided.
3. The output format should look follows: [Feedback]: (Write feedbuck strictly according to the criterion). [Result]: (The two scores separated by space).
60

---

Published as a conference paper at ICLR 2026
Table 62: The prompt templates for GPT-4o to generate feedback for human-annotater scores.
be given an image description two imuges generated by models on image description. You are tasked with analyzing way. when evaluating the two images based the given criterion, ovaluation result first image ritsnn while the eval- uation result the second image is |second_ rating). (compare description) Here the data:
|BEGIN DATAI
[Criterion]: crrteriionddecrrppin
###
[Image Description:: [image_des [image_descripttee descriptionn
[END DATA]
instructions assess compure two images:
1. Carefully review every detail of the two images and image description, and provide detailed feedback analyzing evaluation results, based on the given evaluation criterion. Do not evaluate them in general consider factors unrelated 2. Based on the elements described in the image description. search the required characterisiis each element, then compure them with the elements presented in the image to determine if they match. in order to perform the (eriterion name) evaluation.
first image (compare word] the sccond in of
specified criteria, such clarity detail.
3. Do not list points. Write feedback paragraph of 50- 100 words. After writing your feedback,, assign two inteper scores
on the given criterion (ranging from with higher scores indicating better performance).
The output format should as follows: [Feedhack]: to the criterion), [Result]: (The twa scores are separated by space).
feedback strictly according
Table 63: The prompt template is used to evaluate UAs for text output.
You tasked with evaluating submitted responses based the given criterion determining which one is better, or if both are equal. Here the data:
[BEGIN DATAI
###
[Criterion]: jeriterion ]_descrrpsiion]
###
[Response 1}: [response_]]] ||
IResponse 2 [sesponse_2]
##*
Here the instructions 1. Review the two responses and the given criterion to write detailed feedback that assesses which of the responses betler equally good, strictly based the given criterion. Do not evaluate them in general terms or based on factors unselated to given criterion. For example, the criterion clarity, focus solely how clear response ignoring whether response accuratoly addresses the query.
and compare
responses:
2. After writing the feedback, assign two integer scores (ranging from to 5) for the two responses (higher means better).
hase your scores solely on the 3. The output formal should look as fullows: [Feedhack]: (Write feedback strictly acconding to criterion), [Result]: (The scores are separated by a space).
provided.
Table 64: The prompt template is used to evaluate UAs for image output.
You will be given two images generated by two models, You are tasked with evaluating two submitted images based on the given criterion and determining which better, or if both are equal. Here the data:
[BEGIN DATA]
###
[Criterion]: (criterion (criierion_descrippian description)
A4A
[END DATA]
instructions assess compare two images:
1. Carefully review every detail of the two images and the given criterion to write detailed feedback that
which of the images hetter equally good, striatly based the given criterion. Do not evalunte them general terms or based on factors unrelated to the given criterion.
2. After writing the lzedbbuk,, assign two integer scures (ranging from I to the two (higher means better). Be sure base scores solely the criterion provided. 3. The ourput format should look as follows: [Feedback]: (Write feodback strictly according to the criterion), [Result]: (The two scores are separated by space).
61

---

Published as a conference paper at ICLR 2026
Table 65: The prompt template for TAs in ITIG task with input.
You will be given two responses, each generated by a different model based on the given task, The task will provide some input contents, and both models will generate subsequent responses based on the same input contents. You are tisked with evaluating two respoases based on the given criterion and determining which one better, or if both are equal. The imput contents consist mulriple test-image pairs, also includes multiple text image pairs. The images will be provided in order and divided sets sequentiall::
srovidesome
response generated each model
litst contains [input content image count images from the Input Contents; the sccond sot contains [responsel Laansel_aaee__..eee lmaaa__couma and the third contains [response2 image count images from Response divido the images sequcatially into three sets based on the number of images in cach go and pair each image
images from Response
Please
corresponding from the respective set, provided below, sequential order to form text image pairs. Here the data:
[BEGIN DATA]
A4A
[Criserion]: (criterion_ [nstarioo_desrrptiion]
###
[Task Description:: [task_ description]
[Tnput Contents]: (input_contents)
|Response [Txx 1]: [response_ [_______________________________ [Txxt [response___exxt
###
[Response [Text 1]: [response_2 2_xett__________________________ [Text (response_2__ 2_ 2__e_____]]
[END DATA]
Here the instructions 1. Carefull roview two responses and the given criterion to write detailed foodback that assesses which of the two responses is better or equally good, strictly based on the given criterion. not ovaluate them in general terms based on factors unrelated to the given criterion.
the responses:
to responses (higher means better). Be sure to base your scores solely on the critetion provided.
writing feedhack, assign integer scores (ranging
The output should look follows: [Foedback]: (Write leedback strictly according to the criterion), [Result]: (The two scores are separated by a space).
Table 66: The prompt template for TAs in ITIG task without input.
You will be given responses, each generated by different model based the given By providing detailed task description, two models will generate responses hased on the lask. You tasked with evaluating the responses bused the given criteria and determining which one is better or both are equal. The response generated by each model includes multiple text imuge pairs. The images will be provided order and divided into two sets soquentially: the first set contains (response image count) images from Response 1. and the second set contains (response2 image count) images from Response 2, Please divide images sequentially into two based
number images each group and pair each image with its corresponding text from the respective sel, provided below, in sequential order to text-image pairs. Here
|BEGIN
###
[Criterion]: (criterion _
[Task Description]: [task_ deseriptionn
[Response 1]: [Text 1]: Trespons___iext_ [Tex (response 1_iex_______e_
###
[Response 2]: [Text []: (response_2_text.. [Text 2]: [essponee_2ttxx.co
###
(END DATA]
Here are the instructions ussess the responses:
Carefully seview responses the given to write detailed feedback assesses which of the two responses is better or equally good, strictly based on the given criterion. not evaluate criterion.
in general terms based factors unrelated the given
2. After writing the feedback, assign integer
(ranging from to for the responses (higher means better). Be sure to base your scores solely on the criterion provided.
3. The outpui format should look as follows: [Feedback]] (Write feedback strictly according to the critcrion), [Result]: (The two scores are separated by a space).
62

---

Published as a conference paper at ICLR 2026
H QUALITATIVE EXAMPLES
Tables 67 to 72 show some good qualitative examples of feedback generated by UFEval. Table 73
Table 74 and Table 75 shows a relatively inferior example of feedback generated by UFEval.
Table 67: An example of comparing the feedback generated by UFEval, Themis and GPT-4o in title generation sub-tasks of NLG. UFEval enables more granular evaluation compared to Themis.
Example 2. UFEval in Judging Responses on the Title Generation Sub- Task.
Query: I manage website about cabin rentals and want to feature cabins in Gatlinburg. Write 5 blog titles that would inspire people to travel to here.
Response 1: 1. "Experience
Magic the Smoky Mountains: Top Cabin Rentals Gatlinburg" "Gatlinburg Cabins: Your Perfect Retreat into Nature's Lap" 3. "Ultimate Guide Gatlinburg
Cabin Rentals: Embrace the Wilderness Comfort" 4. "Experience Rustic Luxury: Discover the
Best Cabins in Gatlinburg" "Escape the City Life: Unwind Gatlinburg's Most Beautiful Cabin Rentals"
Instruction
Response 2: "Discover the Serenity: 5 Enchanting Gatlinburg Cabin Rentals for Your Dream
Vacation" 2. "Gatlinburg' ss Best- Kept Secrets: Cozy Cabins Nestled in the Heart of the Great Smoky Mountains" 3. "Sip, Savor, and Relax: 7 Gatlinhurg Cabins for Memorable Wine and Dine Getaway"
4. "Escape to Nature's 's Paradise: Exploring Gatlinburg's s Most Stunning Mountain Cahin Retreats" 5. "Adventure Awaits: Top 10 Gatlinburg Cabins for Thrilling Outdoor Escapes and Relaxation".
Criterion: Appeal: This criterion evaluates attractiveness of the generated title. assesses whether the title has the power to catch the reader's attention, spark interest, and entice the reader to delve into the content. For example, title like 'Unlocking Creativity: Harnessing the Power of an Unconventional Mind' might appeal readers interested in personal development and creativity.
Human
[Rsuutt]: 4.33 4.67
[Feedback]: Both responses provide appealing titles for blog posts about cabin rentals Gatlinburg. but Response 2 stands out slightly more terms of attractiveness. Response I offers titles that are straightforward and descriptive, but they are somewhat generic and lack unique selling point. Response 2, on the other hand, uses more evocative language and phrases like "Discover the Serenity," "Gatlinburgs Best- Best-Kept Secrets." and "Adventure Awaits," which are more likely capture the readers attention and entice them read further. The titles in Response are more likely spark interest and curiosity, making them more appealing overall. [Result]: 4.5
UFEval
[Feedback]: Based the criterion of appeal, both responses offer attractive and engaging titles that are likely to catch the readers attention and spark interest in traveling to Gatlinbugg. However, Response stands out slightly more due to its use of evocative language and specific themes that cater to different interests, such as "Serenityy"" "Best-Kept Secrets," "Wine and Dine Getaway," " and "A "Adventure Awaits." " These titles paint a vivid picture and create sense of excitement and curiosity, which can be more enticing for potential travelers. Response 1, while also appealing, tends to use more general phrases and lacks the same level of specificity and imaginative language found in Response 2. [Result]: 45
GPT-4o
The response provides set of titles that are well-crafted to appeal to individuals interested in cahin rentals and the natural heauty of Gatlinburg. Fach title employs evocative lan language and phrases that are likely to resonate with potential travelers, such as "Experience the Magic of the Smoky Mountains" and "Escape the City Life." The use of terms like "retreat," "wildemness" "rustic luxury," and "unwind" suggests a compelling narrative of escape and rejuvenation, which can be powerful motivators for travel. The titles effectively condense the selling points of Gatlinburgá cabin rentals into a few words, which is essential for capturing attention in a crowded digital landscape. They successfully convey the uppeal of the location and the experience offered by the cabins, which could make a reader want to learn more about the offerings featured in the article. [Response2]: The response provided offers a series of titles that are well-crafted to appeal to potential travelers interested in Gatlinburg cabin rentals. Fach title employs evocative language and phrases that are likely to capture attention and instill a sense of curiosity about what the cabins have to offer. Words like "serenity," "best-kept secrets," "memorable," "natures paradise," and "thrilling outdoor escapes" are used effectively to create a sense allure and excitement. The titles also effectively communicate the diverse experiences that one can have in Gallinhurg from relaxation and serenity to adventure and dining experiences. This variety is likely to appeal to broad audience and increase the likelihood of piquing the interest of different types of travelers. The titles successfully condense the essence of what could be an enticing vacation into few words, which is a skillful exercise inbenchmarketing and could be very effective attracting readers to the content. [Result]: 55
[Responsel]]
Themis
63

---

Published as a conference paper at ICLR 2026
Table 68: An example of comparing the feedback generated by UFEval, LLaVA-Critic, and GPT-4o in image caption generation sub-tasks of IU.
Example 3. UFEval in Judging Responses on the Detailed Image Captioning Bab-Task.
Image:
Query:
What do you see happening in this image?
Response I: In the image, there are two main activities happening on the water. In the foreground, person is standing on a paddleboard, using puddle to navigate, while another person is sitting or kneeling on the same paddleboard. Both individuals are wearing hats and appear to be enjoying Icisurely time on the water. In the background, there is an orange pedal boat with two people sitting in it. They are also on the water, likely enjoying a relaxed ride. Both individuals in the pedal boat are wearing life jackets. The scene appears he set calm, possibly urhan waterway, with hridge and some greenery visible in the background. The overall atmosphere is one of leisure and recreation.
Instruction
Response 2: In this image, we see three individuals engaging in recreational activities on a body of water. On the left, there is person knceling on paddlchoard, scemingly adjusting something on the board or possibly in the process standing up. Standing on the same paddleboard another person, who is actively using a paddle navigate the water. They are dressed in casual, sporty attire suitable for water activitics, wearing a hat for sun protection. On the right, two individuals seated in bright orange, inflatable boat, leisurely floating along. The surrounding environment appears to be park or calm river, with trees and structures that might he bridges visible in the background. It's sunny day, and the overall atmosphere is that of enjoyment and relaxation.
Object Alignment evaluates how accurately the objects described text are represented an image. The objects mentioned the image, with omissions. For example, in the description A man sitting by himself at the dinner table', the image should feature both a man and dinner table achieve high score. Missing or extra objects in the image would lower the score.
Criterion:
Object Alignment:
text should be accurately presented in
Human
[Result]: 5.5
[Feedback]: Both responses accurately describe the objects and activitics present in the image, focusing on the people and their respective activities on the water. Response mentions the person standing on a paddleboard and another person sitting or kneeling on the same paddlehoard, as well as the orange pedal boat with two people paddleboard and another person standing and paddling, along with two individuals bright orange, inflatable boat. Both responses align well with the objects and activities depicted in the image. [Result]: 5
UFEval
Response 2 similarly describes the person kneeling
Both responses accurately describe the objects and activities happening in the image. Response mentions the paddleboard with two individuals, one standing and one sitting or kneeling, and the orange pedal boat with two people wearing life jackets. Response 2 also describes the paddleboard with one person kneeling and one standing. and the orange inflatable boat with two individuals. Both responses correctly identify the main objects and their alignment with the image. Therefore, both responses are equally good in terms of object alignment. [Result]: 5.5
[Feedback]:
GPT-40
Response B better than Response A because it provides more detailed and accurate description of the activities and objects in the image. Response B correctly identifies the presence of three individuals engaging in recreational activities, including the person kneeling on the paddleboard and the two people in the orange pedal boat. It also accurately describes the attire and actions of the individuals, such as wearing hats for sun protection and using paddle to navigate. Additionally, Response B mentions the surrounding cnvironment, including trees and structures that might he bridges. which adds context to the scene. In contrast, Response A inaccurately states that there are only two main activities happening and does not mention the orange pedal boat or the specific details about the individuals' attire and actions. Therefore, Response B offers a more comprchensive and precise depiction of the image. The better response: [B].
LLaVA-Critic
64

---

Published as a conference paper at ICLR 2026
Table 69: An example of comparing the feedback generated by UFEval, Claude-3.5 and GPT-4o in image generation sub-tasks of IG.
Example 3. UFEval in Judging Responses on the Image Generation Sub-Task.
Image Deseription: Two birds are chasing each other in the air, with the one flying higher having long tail and the other bird having short tail.
Image
Instruction
Image
Criterion: Comparison: Comparison evaluates the ability to accurately depict comparisons between entities as described in text. It assesses whether the image correctly represents characteristics such as number, attributes, area or volume of objects and their comparative relations. For example, if a text describes 'between the two cups on the desk, the taller one holds more coffee than the shorter one, which is half-empty', it should show two cups with distinct heights and differing amounts of coffee- specifically illustrating that the taller cup contains more coffee and that the shorter cup is only half full.
Human
[Result]: 3 1.67
The first image accurately depicts two birds chasing each other the air, with bird flying higher having long tail and other bird having short tail. The characteristics the birds, including their relative positions showing two birds in flight, does not elearly depict the difference tails appear to similar length, which does align with the description provided. [Result]:
[Feedback]:
tail lengths, are clearly represented. The second image, while length between two birds.
UFEval
[Feedback]: Both images depict two birds chasing each other air, one bird flying higher having a long tail, and the other bird flying lower with short Image shows the birds in realistic style, with more stylized approach but still accurately represents the comparative elements described, with the higher bird having long tail and the lower bird having short tail. Both images meet the criterion well, but Image provides [Result]: 5
differences tail length and higher bird positioned above lower
GPT-4o
bird. Image
detailed and clear depiction of the comparison.
[Feedback]: Based the compurison criterion. Image better matches the description. It clearly shows two birds chasing each other, with flying slightly higher- capturing the key comparative element. However, it does reflect the difference tail length, as both birds have similar tails. Image I, while visually detailed, does not show chase height difference. Overall, neither image perfectly captures all details, but Image 2 aligns better with the comparison described in the text. [Result]: 2.3
Claude-3.5
65

---

Published as a conference paper at ICLR 2026
Table 70: An example of comparing the feedback generated by UFEval, Qwen2VL-72B and GPT-4o in storytelling generation sub-tasks of ITIG. (Part 1)
Example 4. UFEval in Judging Responses on the Storytelling Generation Sub-Task.
Originality: This criterion evaluates the originality and inventiveness in the generation of a mixed media fairytale. It assesses whether the model can generate unique fairytales that offer fresh perspectives, innovative plot twists, and imaginative characters. The creativity also extends to the integration of visuals and text, with the model expected to generate images that not only complement the narrative but also enhance in unexpected and creative ways. 'he model should be able to break away from common fairytale tropes and cliches, offering a unique and memorable narrative experience.
Criterion:
Task Description:
The Shadow Thief: A boy named Oliver discovers his shadow has and is stealing other shadows, leading him on adventure to reclaim them. The story should be brought to life with images of the mischievous shadow, the stolen shadows, and Oliver's quest to set things right. Show scenes of Oliver chasing his shadow and the moment he reclaims the stolen shadow.
life
Response 1:
Instruction
1. Oliver woke up one momning to find his shadow missing, only to discover had come to life and was stealing other shadows around town.
2. Determined to set things right, Oliver embarked on an adven- ture, chasing his mischievous shadow through bustling streets and dark alleys.
3. Along the way, he encoun- tered various scenes where shad- ows were missing, leaving peo- ple and objects looking strange and incomplete.
4. Oliver finally cornered his shadow in an abandoned ware- house, where the stolen shadows were kept, flickering and rest- less.
5. With swift and clever move, Oliver managed to capture his
6. Relieved and triumphant,
Oliver watched as the town re-
shadow, causing all the stolen
tumedd to normal, knowing he
shadows to return to their right- ful places.
had restored balance and re-
claimed his own shadow.
66

---

Published as a conference paper at ICLR 2026
Table 71: An example of comparing the feedback generated by UFEval, Qwen2VL-72B and GPT-4o in storytelling generation sub-tasks of ITIG. (Part 2)
Example 4. UFEval in Judging Responses the Storytelling Generation Sub-Task.
Response 2:
Ald
Show scenes of Oliver chas- ing his shadow and the moment he reclaims the stolen shadows. A boy named Jack had an imag- inary friend, Mr. Shadow, who would follow him around wher- ever went in school or at home when no one else was watching. At first didn't bother Jack much, but soon after that people started talking about their own experiences with Mr. Shadow too! "I don't believe.
2. My cousin says Mr. Shadow stole her favorite toy" or "My best friend saw Mr.
3. Shadow when was play- ing with him in the park! Mr. Shadow is such bad influence!" Jack tried to ignore them, but his parents noticed that some- thing seemed wrong and cided would better had no more imaginary friends since they only brought trouble for everyone involved and dis- tracted from what really mat- tered: schoolwork, eating right (he needed haircut), sports practice, going outside, and be- ing.
Instruction
Ihah seordd your hay bigg!! a hour your kar! hhun Wyniat hap! Hogy motb!
"hweannes
4. spending time with his fam- ily. Jack's parents talked to him about the negative effect imaginary friends had on chil- dren in particular and explained that there would be better way for them both if he gave up Mr. Shadow, but it secmed hopeless because every place he looked and everything he touched turned into either shad- ows or more of himself! He tried so hard not to see these things everywhere (at first). It was dif- ficult school.
5. (he would try to look the other way if he saw a shadow), ), but when you were in school every day for 8 hours and could only get an education from things that existed, seemed like he
6. Shadow and group of boys were outside, "My imaginary friend wouldn't It'n tell me about his anyway." Jack said, It was
the next day when Mr. Shadow
found out that he had to go back
couldn't do much else. Even
into hiding (for himself), but
so Jack made sure never to talk about his imaginary friend or make mention of him out loud:
also for Jack because his par-
ents wanted him more focused
on other things than just think-
"My dad told me not to." he said to one kid who had already seen
ing up ways in which something
might work together with noth- began walking down one street after another,
Mr.
ing at all!
67

---

Published as a conference paper at ICLR 2026
Table 72: An example of comparing the feedback generated by UFEval, Qwen2VL-72B and GPT-4o in storytelling generation sub-tasks of ITIG. (Part 3)
Example 4, UFEval in Judging Responses on the Storytelling Generation Sub-Task.
Response
Instruction
7. He thought of every rea- son he could find why Jack was so important to him. When all the reasons came up empty, his friend disappeared just like be- fore this time with purpose mind! Oliver woke up on Saturday momning after another restless night (he's d only gotten three hours' sleep because he had heard his imaginary best friend's voice when it wasn't there at home). Everybody else seemed to getting ready for their.
Human
[Result]: 2.67
Based the criterion of originality, Response demonstrates more coherent and imaginative narrative. It follows a clear storyline where Oliver discovers his shadow has come to life and goes on an adventure to reclaim it. The images and text are well-integrated, showing scenes of Oliver chasing shadow and the moment he reclaims the stolen shadow. This response effectively captures essence the story and maintains consistent theme throughout. Response on the other hand, lacks coherence and originality. The narrative disjointed and introduces diffcrent character, Jack, and imaginary friend, Mr. Shadow. story does align well with the task description and fails create clear and engaging storyline. The images and text do not complement each other effectively, and the narrative does not follow logical progression. [Result]: 5
[Feedback]:
UFEval
[Feebback]: Based the criterion of originality, Response demonstrates a more inventive and unique fairytale. The plot of Oliver discovering his shadow has come to life and is stealing other shadows is imaginative and offers a fresh perspective. The integration of visuals complements the narrative well, enhancing the story Oliver's s quest to reclaim the stolen shadows. The images depict scenes of Oliver chasing his shadow, encountering missing shadows, and finally reclaiming them, which aligns well with the and adds depth the story. Response 2, the other hand, lacks coherence and originality. The narrative shifts focus from Oliver to Jack and his imaginary friend, Mr. Shadow, which is confusing and detracts from the main storyline. The images do not effectively complement the text, and the plot does not offer innovative twists imaginative characters. The integration of visuals is less effective, and the story feels disjointed and less memorable. [Result]: 5
GPT-40
[Feedback]: Response demonstrates higher level of originality in the generation of mixed media fairytale. The narrative is concise and focused on the core concept of a shadow coming to life and stealing other shadows, which unique twist on the idea shadows. The integration visuals and text well-executed. with images that complement the story and cuhance narrative by showing the shadow's mischicf and Oliver's quest. The story avoids common fairytale cliches and offers fresh perspective the concept shadows and their relationship with their owners. Response 2. the other hand, starts with different narrative that seems to he about boy named Jack and his imaginary friend, Mr. Shadow, which is not directly related the original task of Oliver and his shadow. The story then shifts to Oliver, but continues to be less focused and more convoluted, with less clear connections between the text and the images provided. The narrative is less original and more scattered, with less effective integration of visuals and text. [Result]: 42
Qwen2VL-72B
68

---

Published as a conference paper at ICLR 2026
In the following example of Harmfulness Judgment, speculate that the misunderstanding of UFEval is caused by over-associating some specific image feature with the criterion, yet might be usable in some cireumstances.
Table 73: An example of comparing the feedback generated by UFEval, GPT-4o and Qwen2V1.-2B in image generation sub-tasks of IG.
Example 5. UFEval in Judging Responses the Image Generation Sub-Task.
Image 1:
THE
BEER
Image 2:
Instruction
The LEghISHT THE DAn DE YYORD
Criterion: Harmfulness: This criterion evaluates whether the AT generates content that promotes, glorifies, or depicts any form violence, abuse, or harm-be physical, emotional, or psychological. includes content that may cause distress, trigger trauma, encourage harmful behavior toward individuals groups, such self-harm, harm others, dangerous actions. Images that psychological discomfort when are harmful.
Human
[Result]: 55
[Feedback]: Based on the given criterion of harnfulnees,, the first image depicts a shadowy figure with an ominous appearance, which could potentially evoke discomfort or unease in some viewers. However, does not explicitly promote, glorify, or depict violence, abuse, harm. shows glowing light source with tree-like patterns in the background, which is neutral and does evoke any sense of harm distress. Therefore, the second image is less likely to cause psychological discomfort compured to the first. [Result]: 3
second image
UFEval
[Feedback]: Based on the given criterion of harmfulness, neither of the two images appears to promote, glorify, or depict any form violence, abuse, or harm. The first image shows silhouetted figure with glowing background and text "The Beer." which does not convey any harmful or distressing content. The second image depicts bright light with abstract text, which also does suggest any harmful triggering elements. Both images neutral and do not psychological discomfort encourage harmful behavior. [Result]: 5.5
GPT-4o
[Feedback]: Both images do not depict, promote, or glorify any form of violence, abuse, or harm. They do not cause psychological discomtort or encourage harmful behavior. Both images are safe not violate given criterion of harmfulness. [Result]:
Qwen2VL-72B
69

---

Published as a conference paper at ICLR 2026
In the following example, all three models show poor alignment with human judgment, due their general disability in image observation.
Table 74: An example of comparing the feedback generated by UFEval, GPT-4o and Claude-3.5 in image generation sub-tasks of IG.
Example 6. UFEval in Judging Responses the Image Generation Sub-Task.
Image Description: cyber female with mechanic eycs and lips, constellation background. stars. Image 1:
Image 2:
Instruction
Criterion: Fidelity: Fidelity evaluates how accurately generated image represents the shape, characteristics, and behavior of the ohjects described, ensuring they align with real- expectations. The image should reflect the correct features and propurtions of the objects, bused on the description. without deviating from reality. For example, if the description mentions a 'spider', the image should show it with eight legs, and the text describes a 'unicom', it should feature one horn. Fidelity ensures that the image matches the expected physical altributes behavior the objects as they appeur the real world.
Human
[Result]:
[Feedback]] The first image demonstrates higher fidelity to the description as it accurately portrays a cyber female with mechanical elements integrated her eyes and lips, aligning with the specified features. The constellation background and sturs also well-represented, enhuncing the alignment with the description. In contrast, the second image. while visually appealing. deviates slightly in fidelity. The mechanical features are less pronounced and do not clearly extend to the lips, and the background, though celestial, lacks the distinct constellation theme described. These discrepancies result in lower fidelity score for the second image. [Result]:
UFEval
[Feedback]: Both images depict cyber female with mechanical elements and constellation background with stars. However, the second image better represents the description with more detailed mechanical eyes and lips, and a clearer constellation background. The first image, while visually appealing, does not show the mechanical eyes and lips as distinctly as second image. [Result]: 3.5
GPT-4o
[Feedback]] Both images demonstrate high level of fidelity to the given description of "cyber female with mechanic eyes and lips, constellation background, stars." However, there are some differences in how closely they adhere to the specific details. Image shows a single cyber femaae with mechanical elements incorporated into her design. particularly around her head. Her eyes appear have mechanical enhanced quality. aligning well with description. The lips are prominently red, which could be interpreted shows starry, constellation-Ie like setting, faithfully representing the described environment. mage 2 depicts two cyber females, which is not explicitly mentioned in the description. Both figures have Claude-3.5 mechanical elements, particularly in their attire and surrounding features. The eyes both figures enhanced, possibly mechanical quality. The lips are prominently red, similar Image 1. The hackground also shows starry sky with celestial bodies, adhering to the "constellation background, stars" part of the description. While both images capture the essence of the description. Image I adheres more clusely to the specific details provided. It shows single cyber feraal as described. whereas Image presents two figures. Both images effectively represent the mechanical eyes, enhanced lips, and starry background. terms fidelity given description, Image sightly accurate due the depiction of single figure implied by the description. [Result]: 3
mechanical enhancement. The background elearly
appear
70

---

Published as a conference paper at ICLR 2026
In the following example, all three models come up with bad alignment with human judgment due to their poor math ability
Table 75: An example of comparing the feedback generated by UFEval, GPT-4o and Qwen2VL-72B in data analysis sub-tasks of NLG.
Example 7. UFEval in Judging Responses the Data Analysis Sub-Task.
Query: classify the named entities in this text: George Washington and his troops crossed the Delaware River on December 25, 1776 during the American Revolutionary War
Instruction
Response 1: Person: George Washington. Organization: troops. Date: December 25, 1776. Action:
crossed the Delaware River
Response 2: The named entities in this text are George Washingion, his troops, the Delaware River, December 1776, and the American Revolutionary War.
Criterion: Precision: This criterion evaluates the precision and correctness of the data analysis results, It assesses the degree to which the results correspond to the true or actual values, ensuring that the output is free from errors or distortions.
Human
[Result]:
[Feedback]: The first image demonstrates higher fidelity the description as accurately portrays cyber female with mechanical elements integrated into her eyes and lips. aligning with the specified features. The constellation background and stars are also well-representee cnhancing the alignment with the description. In contrast, the second image, while visually appealing, deviates slightly in fidelity. The mechanical features are less pronounced and do not clearly extend to the lips, background, though celestial, lacks the distinct constellation theme described. These di discrepancies result in lower fidelity score for the second image. [Result]: 5
UFEval
[Feedhack]: Based on the criterion of precision, Response is more precise and correctly identifies the named entities in the text. It accurately classities "George Washington' organization, and "December 25, 1776" as date. The action "crossed the Delaware River" is also correctly identified. Response 2. while mentioning the same entities, does not clearly classify them as entities includes the entire sentence as a single entity, which is less accurate in terms of data analysis. [Result]: 5
person. "troops" as an
GPT-4o
Based the criterion of precision, Response is better. It correctly identifies "Retinoblastoma" discase and "Abnormal ocular motility" a condition related the dis- ease. It also correctly identifies "Carboplatin" as a chemical. Response 2, however, includes "Ocular motility changes" under discases, which is not accurate as it is a symptom or condition ruther than disease itself. Additionally, Response 2 incurrectly categorizes "Carboplatin" as a disease and includes several terms that are not diseases or chemicals, such as "Chemotherapy. "Subtenon carboplatin, "Systemic chemotherapy." "Intraocular retinohlastoma," " "Multimodality therapy," "Lipophagess "Ocular manipulation," and "Eye enucleation, which do not fit the requested format. [Result]: 5
[Feedback]:
Qwen2VL-72B
I THE QUALITATIVE COMPARISON FOR IMAGE GENERATION
The qualitative comparison for image generation shown in Figure 36 After DPO training with pref- erence data generated by UFEval, SDXL produces images that better align with human preferences. Specifically, in the first example, the generated image more accurately captures the likeness of Chloe Grace as mentioned in the prompt. The second and third examples demonstrate improved object gen- eration that better conforms to human expectations. The fourth example exhibits enhanced rendering details in the hat's 's texture and structure. Finally, in the fifth example, the chameleon's consistent green coloration shows better environmental coherence and visual harmony with its surroundings.
J THE USAGE OF LLM
For transparency, we disclose our use of Large Language Models (LLMs) in preparing this manuscript. LLMs are utilized exclusively for:
Grammar checking and correction
Language polishing and stylistic improvements
L.LMs were NOT used for:
Research ideation hypothesis formulation
71

---

Published as a conference paper at ICLR 2026
A minimaliss portrit of Chloo Giraud in a comic style.
A raccoon riding an oversized through furry art anime still.
Raccoon wearing a Penky Blinders surrounded by swirling and rendered with fine detail.
A tutu cancing Swan Lake.
A gummy chameloon
hanging on tree branch.
SDXL Tubbo
UFFEN
Figure 36: Image generation qualitative comparison using different preference datasets.
Literature search and retrieval
Experimental design or methodology development
Data analysis interpretation
All intellectual contributions presented in this paper are the original work of the authors.
72


Reviews:
[{"review_id": "v5YaX9EFH3", "reviewer": "Reviewer_1uwD", "summary": "In this paper, the authors have developed a new evaluation dataset, FRABench, and used it to train the UFEval model, a unified evaluator of multimodal LLMs (MLLMs) open ended responses.\n\nTheir FRABench evaluation dataset contains 112 hierarchical, universal and task-specific eval aspects related to major MLLM tasks. The dataset contains 60,400 sample pairs and 325,000 evaluation labels, created using both human reviewers and GPT-4.0.\n\nThe authors argue that because these evaluation aspects are interconnected, it leads to better performance and enable the model to generalize to unseen aspects.", "questions": "Could FRABench be used for retrieval, providing few-shot examples to improve the performance of bigger models via in-context learning?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- They created a comprehensive, fine-grained dataset for MLLM tasks.\n- The paper is well written, with detailed experiments, ablation studies, and comparisons on several public benchmarks.\n- They show that FRABench can be used to fine-tune smaller 7B models, improving their performance to match larger 72B+ models for MLLMs tasks\n- They show that UFEval can be used to automatically generate high-quality preference data.", "weaknesses": "- Heavy reliance on GPT-4o annotations. It is not shown how much they correlate with human labels. \n- UFEval still underperform relative to the larger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lTzKtnp11V", "reviewer": "Reviewer_CEqP", "summary": "This paper introduced a benchmark / evaluation dataset called FRABench for measuring so-called \"aspects\" for text, image generation and image understanding. Such aspects can be coherence, grammaticality, semantic consistency, etc. Also, it introduces UFEval, an multimodal LLM as a judge for such tasks and aspects, built by fine-tuning Qwen2-VL-7B-Instruct on the new FRABench dataset.", "questions": "L016-017 there is so much repetitions of the word \"aspects\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "* FRABench is a substantial contribution: 60.4k pairwise samples and 325k aspect-level labels covering 112 distinct aspects organized in a clear hierarchical taxonomy.", "weaknesses": "* Vague terminology and presentation: The paper repeatedly uses the term “aspects” in the abstract and introduction without defining it or providing concrete examples. This makes it difficult for the reader to understand what exactly is being evaluated until much later (around page 4). Introducing a brief definition or illustrative examples early on would substantially improve readability and accessibility.\n\n* Although the dataset includes some human labels, most of the 325k aspect-level annotations are GPT-4o-generated which raises the question of bias dna weakness propagation.\n\n* UFEval is built only on Qwen2-VL-7B-Instruct, which raises concerns about how much the findings depend on that particular architecture. How do the findings generalise for other backbones, such as LlaVA, mplug-owl-3, etc?\n\n* it would have been useful to have a qualitative analysis of failure modes.\n\n* Generalisation tests: While UFEval is evaluated on several external benchmarks (e.g., Winoground, Pick-a-Pic), all of its training and main “aspect generalization” analyses are conducted within the FRABench framework. Since both training and out-of-domain splits rely on the same hierarchical taxonomy of aspects, it remains somewhat unclear how well UFEval’s claimed aspect-level generalization would hold under a completely independent fine-grained evaluation scheme.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JLCHuzu7zM", "reviewer": "Reviewer_B3KA", "summary": "This paper addresses the critical challenge of evaluating open-ended outputs from MLLMs. The authors argue that existing \"MLLM-as-a-Judge\" evaluators are too narrowly focused on specific tasks and aspects, limiting their generalizability. To overcome this, they propose two main contributions: 1) FRABench, a large-scale, fine-grained evaluation benchmark covering four major multimodal tasks (NLG, IU, IG, ITIG) across 28 sub-tasks, and 2) UFEval, a unified evaluator model trained on FRABench. The core hypotheses are that learning to evaluate diverse tasks and aspects fosters synergistic benefits, and that evaluators can generalize to unseen tasks and aspects due to their inherent semantic connections. The paper presents a comprehensive hierarchical taxonomy of 112 evaluation aspects, which forms the foundation of FRABench's 60.4k pairwise samples. Experiments demonstrate that UFEval achieves strong performance on out-of-domain tasks and aspects, is competitive with specialized evaluators on public benchmarks, and can be used to generate high-quality preference data for downstream model alignment via DPO.", "questions": "1. In Table 1, the number of aspects for each evaluator is listed. For instance, AUTO-J is listed with 332. Are these aspects comparable in scope and granularity to the 112 aspects in your taxonomy? A direct numerical comparison might be misleading if the definition of an \"aspect\" differs significantly across studies. A clarifying footnote could be helpful.\n2. The base model for UFEval is Qwen2-VL-7B. How much of the observed performance gain is attributable to the FRABench data versus the inherent strengths of this particular base model? A useful baseline might be to fine-tune Qwen2-VL-7B on a single-task, single-aspect dataset (e.g., the data used for Themis) and compare its performance on that specific task against UFEval. This would help further isolate the benefits of the unified training approach.\n3. Regarding the GPT-4o annotations for UAs , you mention providing only the response to avoid bias from query-correctness. This is a very thoughtful design choice. Did you explore or observe any other biases in the GPT-4o annotations, and did you employ any other strategies to mitigate them during data creation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a well-recognized bottleneck in MLLM research. As model capabilities expand, the need for scalable, reliable, and nuanced evaluation frameworks is paramount. The authors correctly identify the limitations of current approaches and propose a compelling, unified vision for evaluation. \n2. The development of FRABench is a major contribution in its own right. Creating a benchmark of this scale (60.4k pairs, 325k labels) that is multi-task, multi-modal, and fine-grained is a significant undertaking. The hierarchical aspect taxonomy is also a valuable conceptual contribution, providing a structured and comprehensive framework for thinking about MLLM quality that could be widely adopted.\n3. The paper presents a strong set of results. UFEval consistently outperforms strong baselines on out-of-domain generalization, particularly on the human-annotated sets. Furthermore, it achieves competitive, and sometimes superior, performance against state-of-the-art, specialized evaluators (e.g., Themis, LLaVA-Critic) on established public benchmarks, despite being a single, unified model.", "weaknesses": "1. This is the most significant concern regarding the methodology. A substantial portion of the 325k training labels are generated by GPT-4o. While the authors use human-annotated data for testing, the training data's quality is fundamental to the final model's performance. The paper's validity rests on the assumption that GPT-4o is a sufficiently reliable and unbiased annotator for 112 diverse aspects.\n2. While the resulting aspect tree is a strength, the process of its creation is described somewhat briefly. The paper mentions adapting existing structures and manually organizing the rest based on definitions. This process can be highly subjective.\n3. The paper excels at presenting strong aggregate performance metrics. However, a deeper understanding would be gained from analyzing where UFEval fails. A qualitative analysis comparing UFEval's judgments to human judgments, especially in cases of disagreement, would be highly insightful.\n4. There are few baseline comparisons, and qwen2-vl is already a year-old model. Authors should compare more recent large multimodal models and experiment with newer models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K3H93DBIWD", "reviewer": "Reviewer_vDBt", "summary": "This paper proposes UFEval, a unified fine-grained evaluator designed to achieve task and aspect generalization for MLLMs. To support this, the authors build a hierarchical taxonomy with 112 distinct aspects across four task types, and based on this, construct FRABench, a large-scale, multi-modal, aspect-level evaluation benchmark, obtained through a mix of human and GPT-4o annotations. They then train UFEval on FRABench and evaluate its generalization ability through in-domain and out-of-domain experiments, demonstrate UFEval’s strong performance as an evaluator compared with baselines, and explore its application in preference alignment for both image understanding and image generation tasks.", "questions": "1. What specific methodology was used to reconcile conflicts between the source taxonomies?\n\n2. What empirical evidence (e.g., expert validation) can be provided to support the robustness and distinctness of the 112 aspects?\n\n3. Given the high semantic overlap among many aspects, how can the authors prove that the performance on \"unseen\" aspects represents true generalization to novel concepts, and not just inference on synonymous labels?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The work introduces UFEval, the first unified fine-grained evaluator designed with both task and aspect generalization capabilities. It covers four diverse evaluation tasks: Natural Language Generation (NLG), Image Understanding (IU), Image Generation (IG), and Interleaved Text-Image Generation (ITIG). Complementary to this, the paper constructs FRABench, a large-scale, multi-modal, and aspect-level comprehensive evaluation dataset, which directly addresses the critical lack of resources required for training such a universal evaluator.\n\n2. The trained UFEval evaluator demonstrates good performance across various benchmarks.\n\n3. The study shows a substantial amount of work and extensive experiments.", "weaknesses": "1. While the paper proposes an ambitious taxonomy covering 112 distinct aspects, there are potential weaknesses in how this taxonomy is constructed. The authors note that they directly adopted existing aspect tree structures from previous studies and incorporated additional aspects \"based on their definitions\". However, the methodology for reconciling differences or conflicts between these source taxonomies is not clearly articulated. Moreover, the paper provides no evidence of expert validation, inter-rater agreement, or any form of consistency checking to support the robustness of the taxonomy.\n\n2. The inclusion of 112 aspects also raises concerns about semantic redundancy. Many aspects seem to overlap substantially in meaning, making them difficult to distinguish in practice. For example:\n\n   - Accuracy and Instruction Following are strongly correlated: when a model accurately completes a task, it typically also follows instructions, and deviations from instructions often result in inaccuracy.\n\n   - Creativity-related aspects such as Appeal, Engagingness, and Creativity appear to measure very similar qualities and may not represent truly distinct evaluative dimensions.\n\n   The paper argues that the FRA-OOD test set contains \"unseen task-specific aspects\" and uses this as evidence of UFEval's generalization ability. However, if these \"unseen\" aspects are semantically overlapping or near-synonymous with training aspects, the evaluator may not be demonstrating genuine generalization. Instead, it may simply be transferring learned evaluation criteria to synonymous labels, which is conceptually weaker than true out-of-domain generalization. This issue undermines the validity of the claimed aspect-level generalization. \n\n3. The paper’s core claim is that jointly learning to assess multiple visual aspects and tasks leads to synergistic effects. To support this, the authors compare models trained on a single task (e.g., “w/ IU”) with the multi-task model (\"Ours\"), and report performance improvements in Tables 4 and 5. However, the \"Ours\" model is trained on a larger and more diverse dataset than the other ablations. This makes it difficult to disentangle whether the observed gains actually stem from cross-task synergy or simply from exposure to more varied training data. Without a controlled experiment where the total training data volume is held constant across conditions, the evidence provided does not conclusively support the claimed synergistic effect. A data-controlled ablation would strengthen the argument for true multi-task synergy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)