Can Vision–Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective.
Authors: Ruichuan An
Shizhao Sun
Danqing Huang
Mingxi Cheng
Yan Gao
Ji Li
YU QIAO
Jiang Bian
Venue: ICLR 2026 Poster (2026)
Keywords: Vision Language Models; Design aesthetics
Primary area: datasets and benchmarks

Abstract: Assessing the aesthetic quality of graphic design is central to visual communication, yet remains underexplored in vision–language models (VLMs). We investigate whether VLMs can evaluate design aesthetics in ways comparable to humans. Prior work faces three key limitations: benchmarks restricted to narrow principles and coarse evaluation protocols, a lack of systematic VLM comparisons, and limited training data for model improvement. In this work, we introduce AesEval-Bench, a comprehensive benchmark spanning four dimensions, twelve indicators, and three fully quantifiable tasks: aesthetic judgment, region selection, and precise localization. Then, we systematically evaluate proprietary, open-source, and reasoning-augmented VLMs, revealing clear performance gaps against the nuanced demands of aesthetic assessment. Moreover, we construct a training dataset to fine-tune VLMs for this domain, leveraging human-guided VLM labeling to produce task labels at scale and indicator-grounded reasoning to tie abstract indicators to concrete design regions.Together, our work establishes the first systematic framework for aesthetic quality assessment in graphic design.

Published as a conference paper at ICLR 2026
CAN VISION-LANGUAGI MODELS ASSESS GRAPHIC DESIGN AESTHETICS? A BENCHMARK, EVALUATION, AND DATASET PERSPECTIVE
Arctanx An Shizhao Sun Danqing Huang
Mingxi Cheng"
Yan Gao Li 3 Yu Qiao* Jiang Bian
Peking University "Microsoft Research Asia
Miicooooft *Central South University
ABSTRACT
Assessing the aesthetic quality of graphic design is central to visual communica- tion, yet remains underexplored in vision-language models (VLMs). We investi- gate whether VLMs can evaluate design aesthetics in ways comparable to humans. Prior work faces three key limitations: benchmarks restricted to narrow principles and coarse evaluation protocols, a lack of systematic VLM comparisons, and lim- ited training data for model improvement. In this work, we introduce AesEval- Bench, a comprehensive benchmark spanning four dimensions, twelve indicators, and three fully quantifiable tasks: aesthetic judgment, region selection, and pre- cise localization. Then, we systematically evaluate proprictary. open-source, and reasoning -uggnnntee VLMs, revealing clear performance gaps against the nu- anced demands of aesthetic assessment. Moreover, we construct a training dataset to fine-tune VLMs for this domain, leveraging human connan-guiedd VLM labeling to produce task labels at scale and indicator-groundt reasoning to tie abstract indi- cators to concrete design regions. Together, our work establishes the first system- atic framework for aesthetic quality assessment in graphic design. Our code and dataset will be released at: http:://ghhub.ccmm...ttttaaaaaas Bench
INTRODUCTION
The rapid development of vision-language models (VLMs) (Yang et al. 2025
Li et al. 2024a Hurst
jet al. 2024) has opened new opportunities for understanding and acting upon multimodal informa- tion. While they have already achieved remarkable progress in traditional vision tasks such image Li al. 2025b) and visual question answering (An et al., 2024 Zhang al. 2024 Lin et al. 2025). VLMS are increasingly expected to contribute new applications.
captioning (Lin et al. 2024b
Among these, graphic Basign-white integrates textual and visual elements to convey information across advertising, branding, and digital media- represnntssss promising direction given its broad societal and practical impact. Crucially, the success of graphic design critically depends on its aes- thetic quality, shaped by principles such as balance, contrast, and hierarchy.
In this work, we aim to explore a central question: can VLMs understand and evaluate aesthetic quality of graphic design manner compurable to humans? This question is of significant im- portance for at least three reasons. First, it can assist human designers by identifying where design falls short and explaining why, thereby enabling more effective improvement. Second, for generative AI systems, it provides the basis for automatic feedback loops that can guide iterative refinement without extensive human intervention. Third, it suggests an opportunity to extend VLMs beyond factual recognition toward aesthetic evaluation.
Despite its importance, research along this direction remains limited (see Table Tp. First, benchmarks inadequate. Those (Zhou et al. 2024| Huang et al. 2024) developed for natural photos often ig- nore design-specific factors such as typography, while early ones for graphic design typically cover only narrow subset of design principles (Lin et al. 2023 2024a Jiang & Chen 2025). Moreover,
Work was done during an internship at Microsoft Research Asia I Projecct Leader

---

Published as a conference paper at ICLR 2026
existing evaluation protocols are limited. Scoring- Scoring-based methods fail to indicate where poor aesthet- ics occur (Haraguchi et al. (2024). while description- based ones provide qualitative feedback that is difficult to quantity (Jung al. 2025). Second, comparisons between VLMs are missing. There has been no systematic evaluation across different VLMs, whether open-source or closed-source, in the context of design aesthetics. Third, training datasets are lacking, leaving the question of how to further improve VLM performance in this domain underexplored.
As first step toward addressing these limitations, we introduce AesEval-Bench, new benchmark for evaluating the aesthetic quality of graphic designs (see Figure D.. Drawing on prior literature (Li & Chen 2009] |Wangwiwattana & Meeyen 2024), we identify four critical dimensions typography, layout, color, and graphics that together comprehensively capture design aesthetics. These dimen- sions reflect the major factors that humans consistently emphasize when assessing visual appeal. To provide finer granularity, we further define twelve indicators that specify concrete aspects within each dimension, such as hierarchy and legibility under typography. For each indicators, we then de- sign three challenging tasks: 1) aesthetic judgment, which asks models to decide whether design is aesthetically pleasing (yes/no), providing a straightforward measure of overall perception; 2) re- gion selection, which requires models to choose from candidate regions where unpleasing elements appear, testing their ability to pinpoint problematic areas beyond a global judgment; 3) precise local- ization, which challenges models predict the exact bounding box (bbox) coordinates of unpleasing areas, offering the most detailed diagnosis and reflecting a deeper understanding of aesthetics. Un- like prior benchmarks O. AesEval-Bende not only covers broad range of aesthetic factors through its dimensions and indicators, but also defines evaluation tasks that are fully quantifiable via choice or bbox prediction formats, enabling systematic and reproducible assessment of design aesthetics.
fiimensions-tyyograaplll
With AesEval-Bench, we systematically evaluate VLMs on their ability to assess the aesthetic qual- ity of graphic designs (see Table T]. For each design, models perform three tasks across twelve indicators. We use human annotation as ground truth, and measure performance by accuracy (for aesthetic judgment and region selection) and bbox IoU (for precise localization). Overall, our results reveal clear gaps between current lfte-of-thhearr VI.Ms and the nuanced demands of aesthetic qual- ity assessment. Specifically, proprictary VLMs (c.g.. GPT series (Achiam et al. (223g) outperform open-source ones (c.g.. Qwen-VL (Wang et al. 2024), Intern- VL (Zhu et al. 2025), VL (Liu) ee al. )023)). Among open-source models, larger variants (32B, 72B) generally achieve better per- formance than smaller ones (7B). Surprisingly, reasoning-aummented VLMs (e.g., GPT-ol (Jaech) ct al., 2024). GPT-o3, Gemini-2.5-Prr (Comanici et al., (025)) offer no clear advantage over their non-reasoning counterparts. Together, these findings expose the limitations of existing VLMs and underscore the need for domain-specific training tailored to aesthetic quality assessment.
Building these findings, move beyond evaluation and turn to training VLMS for aesthetic quality assessment. To this end, we construct a training dataset consisting of three components: the task (what the model is asked do), the task label (the expected answer), and the reasoning path (the explanation leading to the answer) (see Figure 2) We treat the reasoning path as essential, since generic reasoning has shown little benefit. However, constructing such data poses two key challenges: producing task labels at scale is costly, and generating reasoning paths that genuinely improve performance requires new approaches. To address these, we introduce two solutions. First, human-guidde VIM labeling, where a small set of human annotations serve as in-context exam- plcs to instruct powerful VLMs in producing task labels. This approach maintains alignment with human understanding while reducing manual annotation costs. Second, indicator- indicator-rouunded reason- ing, where abstract indicators (e.g., hierarchy or layering) are explicitly tied to concrete regions in the design. Each reasoning path consists of bounding-box coordinates linked to the indicator and textual explanations for their relevance, providing fine-grained and interpretable supervision. We fine-tune VLMs with the task as input and both the reasoning path and the task label supervision, and evaluate on BosEvvl--Bench The results show consistent performance gains across all tasks and indicators (e.g.. 5.97%. 2.70% and 17.17%), demonstrating that human -ugaddd VLM labeling yields reliable labels and indicator indicator-grounded reasoning supplies effective supervision.
To sum up, our contributions are as follows:
We introduce AesEval -erchh, a comprehensive benchmark for assessing aesthetic quality of graphic designs spanning four dimensions, twelve indicators and three quantifiable tasks. We systematically evaluate proprietary, open-souree, and -uggeette VLMs, re- vealing clear performance gaps in aesthetic quality assessment.
2

---

Published as a conference paper at ICLR 2026
Table I: A comparison of AesEval-Bench with existing benchmarks for both image aesthetics and design aesthetics. We highlight key differences in their scale, task formats, source data, covered design dimensions, and the inclusion of reasoning paths.
Benchmark
Abata
Tank Formot
Saurcc
Source Typa
Dimeasion
Training Set
Reasening Path
Opan-suuree
Graphics
Color
Isage Anntiiiis Boarkksork
Huase al 2ood
Andliench
10k
Pra--orm
Photographie
lingle
Inage--nnly
Zhou 2014
Fras-fom
Phoingg 1piie
lingae
Imuge-unly
2025
Sana
Phonographic
Image-unly
Denign Anniaater Boaehonark
ResizaBesch
Choise
Grophic Design
Image+han
Lin 1 2004s
-1.
Choice
Grppli Design
Imsge+hos
Hanguche clal 2004
2L
Staring
Grgphic Design
Imager Ban
8
Bung
Desctiption
Choice
U3 Dexign
Image only
4
X
Jusg 2025
regresiion
1 Design
Image only
AnEnil (Oun)
4%
Bbox Regresioon
Chiiae
Grgghic Design
/
We construct a training dataset to fine-tune VLMs for this domain. Our approach intro- duces human-guided VLM labeling to produce task labels at scale and indicator-grounde reasoning to tie abstract indicators to concrete design regions. Experiments on AesEval- Bench show that this dataset consistently improves performance across all tasks.
2 RELATED WORKS
Aesthetic Quality Assessment. Aesthetic quality assessment DDng et al. 2017) aims to automat- ically evaluate visual appeal, serving as computational proxy for human judgment. Within this area, two major lines of research have emerged (see Table D. Image aesthetics assessment (Huang al. 2024, Zhou al. 2024 Jiang & Chen 2025) focuses on photographic images, where quality determined by factors such as color harmony, lighting, and subject placement. Design aesthetics Haraguchi et 2024 [Jng al.) 2025) targets graphic designs
assessment (Lin et al. 2023 such as posters, advertisements, or user interfaces, which depend on design-related factors including typography, hierarchy, and alignment. Our work falls within design aesthetics assessment.
2024a |
Despite its importance, design aesthetics assessment remains underexplored. Existing benchmarks capture only a narrow subset of design dimensions. For instance, (Lin et al. 2024a) omits graphics- related factors, while (Haraguchi
al. 2024) ignores both fonts and graphics. Furthermore, their task formulations lack rigor. Some adopt free-form question answering, which is difficult to quan- tify (Lin et al.] 2023), while others provide only holistic scores without identifying problematic regions, limiting interpretability and actionability (Jung et al., (025). Our work introduces a bench- mark that comprehensively covers design-related aesthetic factors across font, layout, graphics and color, defining well-structured and quantifiable tasks using choice and bbox prediction formats.
Vision-Language Models. Vision-Language Models (VLMs) (Wang al. 2024 Comanici al., 2025 Li et al. 2024a| Zhang et al. (022cc) have achieved remarkable performance on tasks such as image captioning (Luo et al. 2025a Li et al. 2024b; Zhang et al.) (025d) and visual question answering (An et al., 2024 Lin et al. 2024b] Luo et al., 2025b). Yet, their ability to assess the aesthetic quality of graphic designs remains largely unexplored. Prior work typically evaluates only one or two VLMSs (e.g.. (Haraguchi et al. (024) studies GPT). We provide systematic comparison across a broad set of VL.Ms, including proprietary, open-source and augmmette models.
Recently, increasing attention has been devoted to the reasoning capabilities of VLMs (Li et al.) 2025a Zhang et al. 2025b), For example, (Sarch et al. (025) employs tree-based search to improve reasoning chains, (Li et al. (025a) visualizes reasoning trajectories for transparency, and (Sun et al.) 2024 Shao et al. 2024: Cao et al.] 2025 Wu et al. 2025) explores grounded visual reasoning by jointly generating bounding boxes and textual explanations. In our work, observe that generic reasoning in current VLMs provides limited benefit for assessing design aesthetics. To address this, construct training dataset with reasoning paths that explicitly link abstract design indicators
3

---

Published as a conference paper at ICLR 2026
Grophtcc
Konp
(8)
Aacharic Judguat
[
lesteey
Arower: yos/
Paycelogr
Apped
Hermony
Controct
Region Selectie
Quaction: Sindiattnd sose prow afthensz
Arceival
COLD Prste
Layost
AAgroeen
Layeriin
White spece
Folonce
Predis Locallaation
Arowert ABCC
Hut eashis clonaiin kanster aich hegf soule
Engiiiiity
Herachy
the pervic poph iftowe soovary pers gloss onarr Toond
25
20% %
SELITI
Figure I: Overview of AesEval-Bench. (A) The four dimensions and twelve indicators considered in the benchmark. Numbers inside the circles indicate how many designs are labeled as flawed for each indicator. (B) Example designs illustrating the indicators, with regions exhibiting aesthetic issues highlighted by red boxes. Detailed textual explanations of all indicators are provided in the Appendix. (C) The three tasks, along with example questions and their expected answers.
to concrete regions of the design. Unlike grounded visual reasoning, which localizes semantically salient entities (e.g., "dog" or "chair"), our regions are indicator-centric, capturing higher-level concepts such hierarchy, alignment, and spacing that directly embody design principles.
BENCHMARK CONSTRUCTION
3.1 OVERVIEW
AesEvel-Bench formulates design aesthetics assessment question question-answering task. The input contains a task description and design image, optionally accompanied by metadata such as layout, font, or color information in JSON format. The output is the answer corresponding to the task.
To capture different aspects of design aesthetics assessment, we introduce three task types (Fig- ure [C): 1) aesthetic judgment asks whether design is aesthetically pleasing (yes/no), providing a measure overall perception. 2) region selection requires choosing from candidate regions where aesthetic issues appear, testing the ability to localize problematic areas beyond global judgment. 3) precise localization requires predicting the exact bounding box coordinates of problematic regions, offering fine-graine grained diagnosis. Each task is accompanied by the explanation of indicator the key factor humans consistently emphasize when evaluating visual appeal (e.g., hierarchy, layering, contrast). We consider twelve indicators (figure[(B)) grouped into four dimensions (Figure T A)).
For design images, we sample 1200 designs from the test split of Crello dataset (Yamaguchi 2021) which contains professional graphic designs with both the design image and its metadata. The expected answers differ across tasks. For aesthetic judgment, the answer is yes or no. For region selection, it is the index of one region among four candidates. For precise localization, it is the bounding box coordinates of the identified region or None if the design has no aesthetic issues.
Overall, AesEval-Bench comprises 4500 base question question-answer answer pairs (three tasks across 1500 de- signs), each further instantiated across twelve indicators to enable fine-grained evaluation.
3.2 CURATION PIPELINE
Establishing Dimensions and Indicators. Aesthetic quality in graphic design is inherently mul- tidimensional. To define rigorous benchmark, we first conducted a comprehensive literature view of classical and contemporary design principles (McCormack & Lomas 2020 Lou et al.]
2022
Lu et al. 2020). We then consulted professional designers to refine this taxonomy, ensur-
4

---

Published as a conference paper at ICLR 2026
Human Aligned Permutated
Ornemiasi
(122)
Human- Guided VLM Labeling
leght
Qunytesi
Indiided
Erry dinggn have
Faaring
Task Specific Instructions
hesthetic Grounding Reasoring Path
Design Element
Coreectesa
Glass tinage int antawrheki
CRAFT I
and Can hes
Indicator- Grounded Reasoning
aesracthv darmertand abcotts
permasse
Pear Olbes
shrigh raga Sarlan Miieee
ANSANC:
Correctees
General Repsoning
bead pon betoee dimmaaa
Geneone
the copermmtto slamett and <ieaad
11S, planter peaeher ouerral leigg makigg
Her'' bucted
Fain with bcmedii 2001 herigeon dhstoaate can rppient to tnt dbd rncouato
Anshetic
ing liei but.........................bb
(B)
Figure 2: (A) Illustration of two key steps in training data construction. Human-guided VLM label- ing enables scalable determination of whether designs exhibit aesthetic issues. indicator-groundee reasoning generates reasoning paths that explicitly link abstract indicators to concrete design regions (represented bbox coordinates). (B) Example highlighting the difference between non-reasoning models, generic reasoning models, and our andicatoo-groundee reasoning model.
ing alignment with both theoretical foundations and practical expertise. This process yielded four core dimensions- layoutt, font, graphics, and color (Figure [TA)) each further specified by con- crete indicators consistently emphasized in human aesthetic judgment. In total, we distilled twelve indicators that together capture the essential factors of design aesthetics (Figure
Constructing Potentially Flawed Designs. As introduced in Section 3.1 the design images in AesEval-Bench are sourced from the Crello dataset, which contains professional-quaile graphic de- signs. To effectively evaluate design aesthetics, the benchmark must include not only well-designed but also less appealing examples. We therefore repurpose Crello by introducing controlled perturba- tions, such as repositioning elements, altering font choices, or adjusting colors. These perturbations may either degrade the visual quality or leave it largely intact. For example, slightly enlarging heading might preserve hierarchy, whereas shifting it left could disrupt balance. Each base design undergoes one to three random perturbations, generating spectrum of variations that range from aesthetically unchanged to noticeably flawed, while still appearing realistic. Since Crello Yam- aguchi (202T) provides element-level metadata in JSON format along with separate design layers, these perturbations can be applied directly at the ISON level and rendered into new design images by recombining the modified metadata with the corresponding layers.
We engage human annotators to verify whether the per- turbed designs truly exhibit aesthetic issues. Before annotation, all annotators receive tutorial that includes examples of both well-designed and flawed cases, along with detailed explanations of the underlying reasons. During the review, each annotator is shown a design image together with a description of the focal indicator and asked to determine whether the design contains the corre- sponding flaw (yes no). For each design, we derive the final label by applying majority voting across multiple annotators to ensure consensus.
-luman-in-the-Lupp Aesthetic Review.
Generating Question-Answer Pairs. With metadata in JSON format, records of applied perturba- tions and human annotations, we can systematically construct answers corresponding to each task. For aesthetic judgment, the rule straightforward: if human annotators label a design as good, the ground-truth answer is no (i.e., no aesthetic issue); otherwise, it is yes. For region selection, if a design is labeled as good, the four answer choices consist of three randomly sampled bboxes from the metadata and a None option, with the ground-truth answer being None. If the design is labeled as flawed, the four choices include the bbox of the perturbed element, two randomly sampled bboxes from the metadata. and None, with the ground-truth answer set to the bbox of the perturbed element. For precise localization, a design is labeled as good, the ground- truth answer is None; otherwise, it corresponds to the exact bbox of the perturbed element.
5

---

Published as a conference paper at ICLR 2026
3.3 EVALUATION PROTOCOLS
For aesthetic judgment and region selection, both formulated as choice problems, we adopt accuracy as the metric, measuring the exact match between model predictions and the ground truth. For precise localization, the task combines two components: choice problem (predicting None when no aesthetic issuc exists) and a bounding box regression problem (predicting the exact bbox when an issue is present). Accordingly, intersection over union which quantifies the overlap between the predicted and ground-truth bboxes- for cases where bbox is required.
use accuracy for cases where the ground truth is None, and
4 TRAINING DATA CONSTRUCTION
Evaluation popular VLMs reveals clear gap between the capabiliteis of current state-of-the- arts VLMs and the nuanced requirements of aesthetic quality assessment. Moreover, reasoning- augmented VLMs show no clear performance gains (see Section 5.1).
To this end, we construct a training dataset, named AisEval-Train to fine-tune VLMs for this do- main. First, we adopt the same procedure benchmark construction construct potentially flawed designs (see Section 3.2). Next, since relying solely on human annotation to determine whether perturbed designs exhibit aesthetic issues is neither scalable nor cost-effective for training at large scale, we introduce human-guided VLM labeling. Then, we follow the benchmark construction to generate question-answer ansser pairs (see Section 3.2 Finally, we introduce indicator- groundde rea- soning to generate domain-spect reasoning paths aimed at improving task performance. In the following, we describe in detail the two steps that differ from benchmark construction.
Human-Guided VLM Labeling. We leverage small set of human annotations as demonstrations, together with the bbox coordinates of perturbed regions, as input to strong VL.Ms. The model instructed generate binary label indicating whether the perturbed design exhibits aesthetic issue (see Figure 2 A)). By incorporating human annotations, we preserve alignment with human judgment while substantially reducing manual annotation costs. Moreover, providing the perturba- tion region as a prior, which unavailable real-world scenarios, simplifies the labeling process and improves reliability. With these sources of guidance, while the generated labels may not be perfectly accurate, they yield a training set of sufficient quality to enhance fine-tuning performance.
Indicator-Groundee Reasoning. As illustrated in Figure B), generic reasoning often explains or analyzes a given indicator and task without grounding the discussion in relevant regions of the design. To address this limitation, we propose explicitly linking abstract indicators to concrete regions within the design. Specifically, we include both the bounding box (bbox) coordinates of relevant regions and textual explanations of their relevance to the indicator in the reasoning path.
To obtain such reasoning paths, we instruct powerful VLMs (e.g., GPT in our experiments) by providing them with the bbox coordinates of the target regions and the corresponding design layers (Figure A)). The model is required to output the provided coordinates alongside an explanation of how the region relates to the indicator, thereby ensuring that the reasoning path consistently contains the desired information. We further adopt task-specific strategics to determine the regions of interest. For aesthetic judgment, we directly use the bbox of the perturbed regions. For region selection, include both the perturbed and penturebd regions strengthen the model's ability to discriminate among candidate regions. For precise localization, we not only highlight the bbox of perturbed regions but also emphasize their relationship the overall design, enabling the model to better localize problematic regions within the global design context.
EXPERIMENT
5.1 BENCHMARKING VLMS ON AESEVAL-BENCH
Setups. We conduct a comprehensive evaluation of 10 VLMs spanning diverse model families and parameter scales. For non-reasoning models, consider open-source representatives such as LLaVA (Liu et al. 2023), Qwen2.5-VE VL (Bai et al. 2025). and Intern- VL3 (Zhu et al., 2025), as well closed-source GPT models (Jaech al. 2024 For reasoning-augmentedd models, we evaluate

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation on aesthetic judgment task. Overall acc is the average value of all indicators. The best and second- best results are highlighted in bold and underlined. respectively.
Model
Overall Ace
Layout
Color
Graphics
balance
laycring
whitespace
diggnsent
hamony
contrast
uppeal
paycholoy
lepibility
bicrarcly
quality
relevunce
Non-remsoning Models
M1VVA-158 138
0.5636
0.6759
0.4411
0.2401
0.2615
0.7660
0.6733
VB-78
0.6390
0.9076
0.8413
0.8225
0.9136
0.4009
0.3355
0.4430
0.3100
0.8185
0.3970
Ow.n-VL--32B
0.6458
0.5689
0.6862
0.6941
0.6209
0.5776
0.4948
0.0023
0.7618
0.7708
0.9001
VL.72B
0.6724
0.6752
0.7804
0.2620
0.2447
0.8739
0.7952
0.7729
0.5734
Issem VL3 5B
0.6331
0.6552
0.7577
0,7751
0,3487
0.3329
0.3082
0.5150
0.9486
0.6851
0.871
0.9991
Inem VL. 14B
0,6883
0.7406
0.3826
0,7563
0,7501
0.7706
0.85%c
0.5276
0.1912
0.8389
0.334
0.8273
0.7729
GPT-4o
0.2031
0.7583
0.6739
0.8891
0.0688
0.8129
0.8190
0.8257
0.3290
0.2506
0.7564
0.7857
0.3150
GPT-5
0,7252
0,3778
0.7632
0,7275
0,6110
0,6953
0.8775
0.4000
0.5257
0. 07442
0.7419
0,9023
0.8551
Ressoning-angaaane Modells
GPT el
0,5705
0.3384
0.7531
0.5522
0,3798
0.7149
0,5439
0,7427
0.6700
0.7049
0.7206
0,7318
0.8030
GPT 05
0.7100
0.3450
0.7597
0,558
0.3964
0,7715
0.7005
0.7993
0.6316
0.7615
0.7332
0,8084
0.7566
Gemmii-23 2.5 Pro
0633
0,3355
0.6924
0,5936
0,5089
0,7335
0.6495
0.6776
0.5604
0.588
0.699)
0,5217
0.6923
Expert Models for Imsage Aesthetics Averement
ArsExpen-7B
0.4056
0.5025
0.4142
0.3317
CLA636
0.3017
0.4147
0.3253
0.2318
0.267U
0.5073
0.6366
0.4904
UNIAN LLaVA
0.2900
0.233
0.2120
0.2471
0.2207
0.2735
0.3315
0.5041
0.3073
0.2893
0.5266
0.2410
0.2879
Table 3: Evaluation on region selection task. Overall acc is the average value of all indicators. The best and second-best results are highlighted in bold and underlined, respectively.
Model
Acc
Laynnl
Calor
Fool homrndy
Graphics
balance
layrring
whitoquon aligimett
harmary
centroed
appeal
peycholoy
lepiiliit
quality
relessnoo
Nun-reasoning Medels
LLaVA 13B
0,6005
0.5823
0.5713
0.5612
0.5918
0.6171
0.5156
0.6319
0,6856
061300
0,706
0.6329
0,5745
Owen: VL-IB (Base)
0,5795
0,5122
0.55/0
0.5148
0.3645
0,5822
0,545
0.5412
0.634
0.994
0.6379
0.6258
0,6190
Owen VL-320
0.6311
0,5931
0.5997
0.5012
0.5252
0.3618
0.008
0.616
0.5755
078635
0.62%
0.2600
0,7531
Owen-Nl V1-728
0.6626
0.5105
0.583
0.4547
0.538
0.5977
0.7225
0.6495
0.7940
0.728
0.793
07360
0.8015
Inem VL3 RB
0,5799
0,5242
0,4948
0.5685
0.5363
0,5527
0,5558
0.5903
0.642
06583
0.157
0.631
0.6415
Intem VL.3 14B
0:6378
0,5872
0.5204
0.5945
0.5745
0,6252
0,5897
0.6419
0,7035
0.7344
0.687
0.6109
0
4PT-4
0,0245
0.4714
0.4894
0 015007
0.6011
0,7606
0.8591
0.7165
0.815
0.633
0.9000
0.644
0.6855
GPT:S
0,6980
0.6484
0.5929
0.6630
0.6506
0.7229
0.7214
0.7510
0.5847
08035
0.7565
0.6953
0,7070
Reasoning ouginented Models
GPC GPT
0:3477
0.6319
0.5746
0.697
0.617%
0.6323
0,7092
0,5308
0.6263
0.5225
0,8272
0.5456
0.7113
0,7981
0.656
0.774
0.601
0,7466
Gemini-2.
06100
0.6810
Expert Model for Image Amchrtics Amseament
AodExperi- To
6.2833
0,2954
0.2250
0.3174
0,2631 0,4075
0,3436
0,2646
0.3176
0.3176
n3176
0,2588
0.2765
0.3682
UNIAMULAVA
0:2418
0,1619
0,2915
0.1552
0.1700
0,1516
0. 1760
0,2479
02777
0.3796
0.2661
0. 0,1988
GPT-ol, GPT-o3, and Gemini-2.5. P.o (Comanici et al. 2025). In addition, we include expert models specifically designed for image aesthetic assessment, namely AesExpert (Huang et al. (024) and UNIAA-LLAVA (Zhou et al. 2024). All models are evaluated under the same input setting, which consists of a question (see Figure a design image, and metadata in JSON format.
Results. The performance of VLMs evaluated following the protocols introduced in Section 3.3 Specifically, in addition to reporting scores for each individual indicator, we also provide an overall score computed as the average across all indicators.
Aesthetic Judgment. Table 2 presents the results. First, among non-reasoning models, GPT-5 achieves the highest performance, with an overall accuracy of 0.7252. This suggests that even the strongest VLMs still struggle with design aesthetics assessment. Second, reasoning-augmentee models do not outperform their non-reasoning counterparts (e.g., GPT-ol Jaech al. (2024) and GPT-o3 vs. GPT-4o Hurst et al, (2024) and GPT-5), indicating that generic reasoning provides little benefit in this domain. Third, expert models designed for image aesthetics assessment per- form worse overall, highlighting substantial gap between design aesthetics and image aesthetics. Finally, model performance varies across indicators. For instance, the Qwen-VI. series tends to perform better on legibility but worse on psychology compared to other VLMs.
Region Selection. Table repoorts the results. First, VLM performance on this task is generally worse than on aesthetic judgment, likely because it requires not only assessing whether design is pleasing but also identifying where flaws occur. Second, consistent with aesthetic judgment, GPT-5 achieves the best performance, while reasoning- augmented models show clear advantage. Finally, across model families, larger models (e.g.. 72B) typically outperform smaller ones (c.g.. 7B). We find some model has the phenomenon of overfitring, so we adopt weighted sum when calculating final score.
reasoning-augmmented
7

---

Published as a conference paper at ICLR 2026
Table 4: Evaluation on precise localization task for the choice component where the model should predict None if no aesthetic issues are present. Overall score is the average accuracy of all indicators. The best and second- best results are highlighted in bold and underlined, respectively.
Model
Overall Score
Layoat
Color
Font
Graphics
balance
layesing
whitespace
alignment
harmony
contrait
appeal
psycholoy
ligibliity
bienarchy
quality
relevance
Nen-reasoning Models
LLaVA 13B
0.6455
0.4523
0.51.5
0.5099
0,8714
0.3356
0.3723
0.2898
0.2474
0.7301
0.2455
0.5611
0.4573
Qwen-VLL VL-VB
0,5192
0.1044
0.5839
0.8546
05347
0.4375
0.5625
0.5655
0.5339
0,4528
0.334
05360
0:5415
GPT-4o
0.5690
0.5417
0.2081
0.5956
1679
0.8635
0.6372
0.7200
0.8713
0.4594
0.164
07762
0.5618
GPTS
0,0090
0.696
0.8143
0.910
06247
0.0057
0,6170
0.5643
0.5989
0.5464
0.614
0/612
0.5894
Reasoning auggnented Models
0,5022
0,4933
0.5884
0.5474
0.5190
GPT
0,5800
0.670
0.7854
0,6985
0.7195
0.3952
Gemmi-22.-Pm
0.5306
Expert Models for Imsage Aesibetics
7
0,317
0.3229
0,3146
0.3756
0.3172
0.4276
0.3562
0.3146
0,3267
0.3314
0.3803
0.3025
0.2804
Table 5: Evaluation on precise localization task for the bbox prediction component where the model should output coordinates of the aesthetic issues. Overall score is the average loU of all indicators. The best and second-best results are highlighted in bold and underlined, respectively.
Modd
Overall
Layout
Graphics
lngering
alignment
appeal
prycholoy
legibllit
quality
relevance
Non Non-rcasoning Models
LL-VA-13B
0,0559
0,0653
0.0080
0,0302
0,0172
0.1004
0.0792
0.0427
0.0303
0.0188
0,0102
0.1510
0.1360
Qwn-Vil-77 (Raee)
00514
0.0877
0 111569
0.001
0,0299
0.100
6.006
0.0099
0.2955
0/002
0/452
010994
0.006
GPT-4o GPT5
0.7113
0.2974
0.1899
0.2552
0.0833
0.1359
0.0664
02185
0.3144
0,2200
0.1270
0.1866
0.1545
0,2077
0.1613
0.1529
0.1525
0,1529
0.3385
0.1755
0.3835
0.3667
0,2338
Reasoning
Models
01
0.1286
0.0937
0.0767
0,1412
0.1226
0.1719
0.0204
0.1546
0.1236
0.1440
0.3441
0,1912
0.1617
03
0.1418
0.3619
0.1915
0,0552
0,3075
0.0703
0.0516
0.0760
0,1479
0.177
0.2796
0.1861
0.0968
Grmim-2.5-PP
B0977
0.0818
0 016200
0. 0.1052
0,7100
(.779)
(.400)
02257
00963
00005
0.0865
0 1014
Expert Models for Imsage Aesthetics Assesoment
AesExpect
0,0440
0,0203
010093
0,1084
010063
0.0049
00115
0.000
0.000
0.0067
Precise Localization, As described in Section 3.3 this task consists of two components, each eval- uated separately: a choice problem, where the model predicts None if no aesthetic issue exists (Table D and bbox prediction problem, where the model outputs the exact bbox of the aesthetic issue (T. We exclude some VLMs (e.g., Intern-VL. series and small Qwen-VI VL. models) be- cause they failed to produce meaningful bbox predictions. For the choice problem, VLMs achieve reasonable performance, with the best model reaching an overall score of 0.6090. For the bbox prediction problem, even the best-performing model, GPT-5, scores below 0.20, highlighting the substantial difficulty of precisely localizing aesthetic issues.
Discussions on Input Components. When benchmarking VLMs, the input consists of three compo- nents: (1) the question, which includes detailed explanation of the target indicator; (2) the design image; and (3) metadata in JSON format, containing layout, color, and font information. We analyze the contribution of each component to model performance using GPT-4o as a representative exam- ple. [ the results, where Full Model denotes the setting that uses all three compo- nents; Withoul Images removes the design image; Without Explanation omits the detailed indicator description; and Without Metainfo excludes the metadata. Our findings reveal three key insights. First, across all tasks, the design image is indispensable-itt removal results in the largest perfor- mance drop. Second, indicator explanations have limited influence for more intuitive indicators (e.g.. balance), but they play a crucial role for subjective indicators (e.g.. relevance or psychology), where clearer definitions are necessary. Finally, metadata has the greatest effect on precise localiza- tion, where its absence causes a larger decline in performance compared to aesthetic judgment or region selection. We hypothesize that this is because metadata provides explicit layout information, which aids bbox prediction in localization tasks.
5.2 FINE-TUNING VLMS WITH
Setups. We construct the training set following the pipeline described in Section 4 resulting in 30k question-answer answer pairs. In our experiments, we use Owen2.5-VVL-7B-Inttee representative model and adopt full- parammerr finetuning on the constructed dataset. The learning rate is set to 1e-6, with cosine scheduler and 3% warmup ratio. For computational efficiency,
(Bai et al. 2025) as
8

---

Published as a conference paper at ICLR 2026
PINA- Farter
Villo lagan
Vilue Sypection
Maead
Heasky Xxp
Lepodry
Eopheby
Appll
Appil
Appil
Figure 3: Results for model variants using different input components.
Table 6: Results and ablation study of fine-tuning VLMs using our constructed training set.
Notting
Sheral
Ovarall Loai
Choraal Rank
Cake appeal
Fist
Graplaio
*abaspace
poycokay
laghlity
Aestheti Jnllgwert (Acuraryy
QN-a-VL-1B (Bac)
0.8272
BA306
8.855
0.54:3
0.5211
05:35
6
01555
0.0430
0.3903
02133
0,3970
0.739
BAS77
0.8657
Poctie Sanylies
0,507
0.300)
5 1999
3335
11483
0.394
03:-0
0.955
0.356
0 3355
0.3366
5.804
Gere-Vi-ne (Bae))
10
31211
10%
015877
036380
0.636
(1511
0.080
D-9015
Raseeiau Padh
EL3732
5.5779
0.3322
0.5411
0.3122
0.041
BA2KS
0.583
0.315
33463
LUSSA
Procbc Locallstiis (94))
Osea VL.75 (koc)
0.100
0.0059
0.035
0.206
3,002
3.864
17.175
IL2515
3.193
0.333
58587
0.224
0.2848
0.155
03607
02313
31151
Pholive Sunagles
09611
0.199
8.1566
0.207
0.16.
0.15 0.185
0.155
0.159
0.3655
0.1745
0.183
0.361
0.2338
training is performed with 6 mixed precision and FlashAttention-2 (Dao et al. 2022). The vision encoder is kept frozen, while the language model parameters are tuned.
Main Results. Table/6 the results, where Qwen- VL-7B (Base) denotes the base model with- out finetuning, and #AesEval-Train refers to the model finetuned on our constructed training set. First, across all three tasks, finetuning with AesEval-Train yields substantial performance improve- ments. Moreover, on aesthetic judgment, the finetuned model surpasses even the largest Qwen-VI. variant (72B parameters), and on precise localization, it outperforms GPT-5 despite the latter having far more parameters. These results demonstrate that our proposed pipeline effectively constructs training data that significantly enhances model performance.
Ablation Studies. We investigate the impact of different data recipes on model performance. Ta- ble reports the results, where Reeasoning Path denotes training with plain question question-answer answer pairs without the proposed undicator-grounder reasoning, and -Positive Samples denotes training only on flawed designs. We observe that -Reasoning Path still improves performance across all three tasks, suggesting that incorporating domain-specific knowledge of design aesthetics is beneficial. However, its performance remains notably lower than that of the full variant with reasoning paths -rAesEval-Train), underscoring the effectiveness Positive Samples performs worse than both +AesEval-Train and -Reasoning Path, highlighting the importance of maintaining label balance in the training set.
indicator-grounded reasoning. In addition,
6 CONCLUSION
In this work, we introduce AesEval-Bench for design aesthetics assessment, which spans four di- mensions, twelve indicators and three quantifiable tasks. Based on it, we systematically evaluate proprietary, open-source and reasoning-augmente ics assessment. Furthermore, we construct a training dataset to fine-tune VLMs for this domain. Experiments show that this dataset significantly improves model performance across all tasks.
VLMs, revealing clear gaps in design aesthet-
First, as Crello serves as the source dataset, the benchmark does not cover all types of graphic design, such as infographies mobile UIs. Second, fully disentangled taxonomy for indicators is not yet available. Should a more rigorously disentangled taxonomy be proposed in the future, it would be valuable to adopt and update our indicators accordingly. Third, highly
Limitations.

---

Published as a conference paper at ICLR 2026
subjective aspects of design, such creativity, are not included. Finally, leveraging reinforcement learning to further enhance reasoning capabilities is left for future exploration.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Ruichuan An, Sihan Yang, Ming Lu, Renrui Zhang, Kai Zeng, Yulin Luo, Jiajun Cao, Hao Liang, Ying Chen, Qi She, et al. Mc-llava: Multi-concept personalized vision-language model. arXiv preprint arXiv:2411.11700 2024.
Ruichuan An, Sihan Yang, Ziyu Guo, Wei Dai, Zijun Shen, Haodong Li, Renrui Zhang, Xinyu Wei, Guopeng Li, Wenshan Wu, al. Genius: Generative fluid intelligence evaluation suite. arXiv preprint arXiv:2602. / 144, 2026.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
Meng Cao, Haoze Zhao, Can Zhang, Xiaojun Chang, Ian Reid, and Xiaodan Liang. Ground- rl: Incentivizing grounded visual reasoning via reinforcement learning. arXiv preprint arXiv:2505.20272, 2025.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein. Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626. 2025.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances neural information processing systems, 35:16344-16359, 2022.
Yubin Deng, Chen Change Loy, and Xiaoou Tang. Image aesthetic assessment: An experimental survey. IEEE Signal Processing Magazine, 34(4))80-106, 2017.
Ziyu Guo, Xinyan Chen, Renrui Zhang, Ruichuan An, Yu Qi, Dongzhi Jiang, Xiangtai Li, Manyuan Zhang, Hongsheng Li, and Pheng-Ann Heng. Are video models ready as zero-shot reasoners? an empirical study with the mme-cof benchmark. arXiv preprint irXiv:2510.26800 2025.
Daichi Haraguchi, Naoto Inoue, Wataru Shimoda, Hayato Mitani, Seiichi Uchida, and Kota Yam- aguchi. Can gpts evaluate graphic design based on design principles? In SIGGRAPH Asia 2024 Technical Communications, pp. 1-4, 2024.
Yipo Huang, Quan Yuan, Xiangfei Sheng, Zhichao Yang, Haoning Wu, Pengfei Chen, Yuzhe Yang, Leida Li, and Weisi Lin. Aeshench: An expert benchmark for multimodal large language models on image aesthetics perception. arXiv preprint 2XXi:2401.08276, 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint arXiv:2410.21226, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Fl-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16770, 2024.
Ruixiang Jiang and Changwen Chen. Multimodal Ilms can reason about aesthetics in zero-shot. arXiv preprint XrXiv:2501.09012 2025.
Sam Jung, Agustin Garcinuno, and Spencer Mateega. Ui-bench: A benchmark for evaluating design capabilities of ai text-to-app tools. arXiv preprint arXiv:2508.2044 20410, 2025.
10

---

Published as a conference paper at ICLR 2026
Hyolim Kang, Yunsu Park, Youngbeom Yoo, Yeeun Choi, and Seon Joo Kim. Open-ended hierarchi- cal streaming video understanding with vision language models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15-2255, 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Chengzu Li, Wenshan Wu, Huanyu Zhang. Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulié, and Furu Wei. Imagine while reasoning in space: Multimodal visualization-of-tuie of-tuughhtt arXiv preprint 2025a.
Congcong Li and Tsuhan Chen. Aesthetic visual quality assessment of paintings. IEEE Journal of selected topics in Signal Processing, 3(2):236 2522, 2009.
Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, et al. A survey on benchmarks of multimodal large language models. arXiv preprini OrXiv:2408.08633 2024b.
Junxian Li, Di Zhang, Xunzhi Wang, Zeying Hao, Jingdi Lei, Qian Tan, Cai Zhou, Wei Liu, Yaotian Yang. Xinrui Xiong, et al. Chemvim: Exploring the power of multimodal large language models in chemistry area. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39,
pp. 415-423, 2025b.
Jieru Lin, Danqing Huang, Tiejun Zhao, Dechen Zhan, and Chin- Lin. Designprobe: A graphic design benchmark for multimodal large language models. arXiv preprint arXiv:2404. 14801, 2024a.
Kevin Lin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, and Lijuan Wang. Designbench: Exploring and benchmarking dall-e 3 for imagining visual design. arXiv preprint arXiv:2310.15144 2023.
Weifeng Lin, Xinyu Wei, Ruichuan An. Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shang- hang Zhang, and Hongsheng Li. draw-and-understtand: Leveraging visual prompts enable mlims to comprehend what you want. arXiv preprint arXiv:2403.20271, 2024b.
Weifeng Lin, Xinyu Wei, Ruichuan An, Tianhe Ren, Tingwei Chen, Renrui Zhang. Ziyu Guo, Wen- tao Zhang, Lei Zhang, and Hongsheng Li. Perceive anything: Recognize, explain, caption, and segment anything in images and videos. arXiv preprint arXiv:2506.05302, 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36:34892 34916, 2023.
Yun Lou, Weiyue Gao, Pei Chen, Xuanhui Liu, Changyuan Yang, and Lingyun Sun. Designeva: A design-supported tool with multi- facti-faceted perceptual evaluation. In International Conference on Human- Computtr Interaction, pp. 508-519. Springer, 2022.
Yue Lu, Chao Guo, Yilun Lin, Fan Zhuo and FY Wang. Computational aesthetics of fine art paintings: The state of the art and outlook. Acta Automatica Sinica, 46(11))2239-22) 2020.
Meng Luo, Hao Fei, Bobo Li, Shengqiong Wu, Qian Liu, Soujanya Poria, Erik Cambria, Mong- Li Lee, and Wynne Hsu. Panosent: panoptic sextuple extraction benchmark for multimodal conversational aspect-based sentiment analysis. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 7667-7676, 2024.
Meng Luo, Shengqiong Wu, Liqiang Jing, Tianjie Ju, Li Zheng, Jinxiang Lai, Tianlong Wu, Xinya Du, Jian Li, Siyuan Yan, Jiebo Luo, William Yang Wang. Hao Fei, Mong-Li Lee, and Wynne Hsu. Dr.: A hierarchical fine- fininerined spatial-temporal grounding, 2025a. URL pattps://arxiaa wwtp:/wwwww.../////////////
cerepptiontemporallee comporal-cognitoon framework to diagnose video hallucination by
org abs
11866
Meng Luo, Shengqiong Wu, Liqiang Jing, Tianjie Ju, Li Zheng, Jinxiang Lai, Tianlong Wu, Xinya Du, Jian Li, Siyuan Yan, et al. Dr. A hierarchical prrceptionttemporal-cooooo frame- work to diagnose video hallucination by fine-grained spatial-temporal grounding. arXiv preprint arXiv:2509. 1866, 2025b.
11

---

Published as a conference paper at ICLR 2026
Jon McCormack and Andy Lomas. Understanding aesthetic evaluation using deep learning. In International conference on computational intelligence music, sound, art and design (part of EvoStar), pp. 118-133. Springer, 2020.
Joonhyung Park, Peng Tang, Sagnik Das, Srikar Appalaraju, Kunwar Yashraj Singh, R Manmatha, and Shabnam Ghadar. R-vim: Region-aware vision language model for precise gui grounding. In Findings of the Association for Computational Linguistics: ACL. 2025, pp. 9669-96669 6669-9685, 2025.
Gabriel Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael J Tarr, Aviral Kumar, and Katerina Fragkiadaki. Grounded reinforcement learning for visual reasoning. arXiv preprint arXiv:2505.23668, 2025.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hong- sheng Li. Visual cot: Advancing multi-modal language models with comprehensive dataset and benchmark for chain-of of-houghht reasoning. Advances in Neural Information Processing Systems, 37:8612-8642, 2024.
Kunal Singh, Shreyas Singh, and Mukund Khanna. Trishul: Towards region identification and screen hierarchy understanding for large vlm based gui agents. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 170-179, 2025.
Guangyan Sun, Mingyu Jin, Zhenting Wang, Cheng-Long Wang, Siqi Ma, Qifan Wang. Tong Geng, Ying Nian Wu, Yongfeng Zhang, and Dongfang Liu. Visual agents as fast and slow thinkers. arXiv preprint arXiv:2408.06:
2024.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
Chatchai Wangwiwattana and Areerat Meeyen. Design through ai eyes: Automating aesthetic as- sessments for learning graphic design concepts. In International Conference on diseppiiiyyy Trends Artificial Intelligence, pp. 336 336-347, 347. Springer, 2024.
Qiong Wu, Xiangcong Yang. Yiyi Zhou, Chenxin Fang, Baiyang Song, Xiaoshuai Sun, and Ron- grong Ji. Grounded chain-of chain-of-thought thoughtouht for multimodal large language models. arXiv preprint arXiv:2503. 12799, 2025.
Kota Yamaguchi. Canvasvae: Learning to generate vector graphic documents. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5481-5489 2021.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 25X5::5505.09388, 2025.
Di Zhang, Jingdi Lei, Junxian Li, Xunzhi Wang. Yujie Liu, Zonglin Yang. Jiatong Li, Weida Wang, Suorong Yang, Jianbo Wu, et al. Critic-v: Vlm critics help catch vlm errors in multimodal reason- ing. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9050-9061, 2025a.
Huanyu Zhang, Chengzu Li, Wenshan Wu, Shaoguang Mao, Yifan Zhang, Haochen Tian, Ivan Vulić, Zhang Zhang, Liang Wang, Tieniu Tan, et al. Scaling and beyond: Advancing spatial reasoning in milms requires new recipes. arXiv preprint arXiv:2504 15037, 2025b.
Huanyu Zhang, Wenshan Wu, Chengzu Li, Ning Shang, Yan Xia, Yangyu Huang, Yifan Zhang, Li Dong, Zhang Zhang, Liang Wang, et al. Latent sketchpad: Sketching visual thoughts to elicit multimodal reasoning in milms. arXiv preprint arXiv:2510.24514 245144, 2025c.
Qintong Zhang, Bin Wang, Victor Shea-Jay Huang, Junyuan Zhang, Zhengren Wang, Hao Liang, Conghui He, and Wentao Zhang. Document parsing unveiled: Techniques, challenges, and prospects for structured information extraction. arXiv preprint 169., 2024.
12

---

Published as a conference paper at ICLR 2026
YiFan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang. Qingsong Wen, Zhang Zhang. et al. Mme-realworld: Could your multimodal 1lm challenge high-resolution real-world scenarios that are difficult for humans? In The Thirteenth International Conference on Learning Representations, 2025d.
Ming Zhong, Yuanlei Wang, Liuzhou Zhang, Arctanx An, Renrui Zhang, Hao Liang, Ming Lu, Ying Shen, and Wentao Zhang. Veu-bridge: Hierarchical visual connotation understanding via semantic bridging. arXiv preprint arXiv:2511.8121, 2025.
Zhaokun Zhou, Qiulin Wang, Bin Lin, Yiwei Su, Rui Chen, Xin Tao, Amin Zheng, Li Yuan, Pengfei Wan, and Di Zhang. Uniaa: A unified multi-modal image aesthetic assessment baseline and benchmark. arXiv preprint arXhh:2404.096. 2024.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
13

---

Published as a conference paper at ICLR 2026
Dimension-Indicators Prompts:
Craphic-Qualin Resolution a measure image detail. Appropriate resolution ensures the best picture quality and readability. Does element have quality issue? Craphic-Relevanee Relevance refers to the direct connection between a graphic element and the meaning conveys. Does element have relevance issue?
Horroyy: Color Harmony refers to the overall coordination, pleasure and beauty of the entire color when there are two or more colors in an image. Color-Contrast: Color contrast refers to the contrasts, oppositions, and differences existing among various colors.
Color-Appreal: Color appeal refers the fact that selection and combination of colors can attract the attention of the audience.
Color-Psy color-Psychology Color psychology refers to the idea that color can trigger subjective psychological experiences and influence emotions, feclings, and behaviors.
Layout-Balance Balance is the distribution of visual weight design. can be sym- metrical (with equal weights on both sides) or asymmetrical (with unequal weights but still achieving visual balance).
Eeyout-Laycring Use size, color, contrast, and other visual cues to establish hierarchical structure of design elements to guide the audience's s eyes.
Layout- Whitespace: White space refers to the blank area around the elements in design. Effective utilization of white space can create balance, visual hierarchy, and clarity. Layout-Al fiuttAligmmntt: Alignment refers to the arrangement of design elements relative each other specific axis grid. Proper alignment creates organization, making the design easier to understand navigate.
of order and
The presentation of the font has hierarchical structure, users can scan the text obtain key information.
Cont-Hierarchy:
cont-Legibillit: Legibility refers the recognition of individual characters and the rela- tionships between them when they are arranged side by side.
Table 7: Showcase of descriptions of each indicator.
Task Reasoning Instructions:
Aesthetic Judgment Given the preview image and element image element bounding box bbox, please reason why the element is not aesthetic in the aspect of criteria. Your reason should only contain the analysis about criteria. think and reasoning. Once the element mentioned, you should add its bounding box after the element(only bounding hox).
Region Selection: Given a preview image, an image of an unsightly element and its bounding box bbox[0], and an image of an attractive element and its bounding box bbox[1], analyze why the first element is unsightly in terms of criteria based on the relationship between the elements. Your reasoning should only include an analysis of criteria. Please think and reason. Once an element is mentioned, you should add its bounding box after the element (bounding box only).
Precise Localization: Given preview image, element image, and the element's 's bound- ing box bhox, analyze why element is unsightly when viewed from the perspective of criteria, based on their relationship. Your reasoning should only include analysis of criteria. Think and reason. Once you mention the element, you should follow it with its bounding box (only the bounding box).
Table 8: Showcase of task reasoning instructions.
14

---

Published as a conference paper at ICLR 2026
A STATEMENT OF LLM USAGE
Large language models (LLMs) were consulted for technical guidance during implementation and debugging; following the collaborative drafting of the manuscript, we further employed LLMs to refine the prose and enhance the overall exposition.
B PROMPTS AND INSTRUCTIONS
Prompts establish a clear set of evaluation criteria. These are organized First, dimensions: Graphics, Color, Layout, and Font, each containing specific indicators into four core
like Font-Legibility. As shown in Table 7 every indicator is defined and paired with guiding question standardize the analysis.
Second, Task Reasoning Instructions (Table provide operational guidance for creating the rea- soning paths. They direct the analysis to focus element's intrinsic flaws, its relationship with other elements, or its immediate context, while critically mandating the inclusion of the element's bounding box (bbox) ground the reasoning in precise spatial evidence.
C ADDITIONAL RELATED WORK
Evaluation for VLMs. Recent years have witnessed a surge in benchmarks Li et al. (2024b) de- signed to evaluate Vision-Language Models (VLMs), ranging from general-purpose assessments of perception Luo et al. (2024) and reasoning Guo et al.
(2025); Zhang et al. (2025a). which encom- pass region-level Lin et (2024b) 2025); Park al. (2025) and hierarchical understanding Kang et al. 2205); Singh et al. (2025) Zhong et al. (2025).
) to more specialized evaluations in domains like chemistry Li et al. (2025)) or image generation
al. (2026). While existing benchmarks effectively evaluate general capability, they often overlook the nuanced, subjective dimensions of visual understanding. To address this, our work introduces a specialized benchmark focused on the aesthetic dimension, assessing how VLMs interpret artistic quality and visual appeal.
D DATA SOURCE SHOWCASE
To address the concern regarding the diversity of the Crello dataset, we provide visual sampling of its typical data in Fig. 4 As illustrated, the dataset covers comprehensive range design categories and aesthetics, including textual handwritten, photography- -driven and diverse art styles designs. This diversity ensures that our benchmark evaluates models on a realistic distribution of graphic design tasks, preventing bias toward any single visual style.
E FLAW INJECTION PIPELINE
F REAL WORLD DESIGN DATA SHOWCASE
To validate our model's generalization capabilities beyond the synthetic distributions of Crello, we compiled a distinct shown in Fig. 5) these samples were sourced directly from intermediate design drafts and annotated by professional designers, capturing the nuanced and often complex nature of authentic.
(OOD) test set consisting of real-world flawed designs. As
Mapping Real-World Flaws to Synthetic Perturbations. To further validate the design of our data construction pipeline, analyzed the collected real-world flawed designs (Figure 5) and mapped their defects to the operations in Algorithm 1, As observed in the "Startup Job Fair" poster, the text blends into the dark background, rendering illegible. This specific emror is simulated in
15

---

Published as a conference paper at ICLR 2026
Charlly
CHOOSE
TECHNOL BOY
(d)
Dott Do
STYLLDAA
THEY ARE NOT JUST FOR
Public Curator
C ficate
Vance Cyndico
1UG
(g)
(c)
(b)
(e)
Figure 4: Diverse design samples sourced from the Crello dataset. The collection demonstrates a wide spectrum of visual styles and structural layouts, including (a) minimalist typography-cente
fashion editorials featuring real human subjects, (c) vintage illustrations, (d) photorealistic tech mockups. (c) geometric abstract art, (f)
designs (e.g.. "Corporate Charity"), (b) photography-din
textured artistic typography, and (g) cyberpunk-thenne certificates. This visual evidence refutes the concern of stylistic homogeneity, confirming the dataset's 's robust coverage across design domains.
pipeline by Color Perturbation module. Similarly, the "End Violence" poster exhibits severe layer- ing issues where graphical elements obstruct critical text. This type of flaw is effectively reproduced by Layout Perturbation module, which applies random coordinate shifts (Position Shift) to el- ements, creating unintended overlaps and layout collisions. This structural alignment confirms that synthetic pipeline generates meaningful negatives align with the real world flaw designs.
Note on Dataset Diversity and Benchmark Showcase. It is important to note that for the Bench- mark Showcase (Sec. D. we intentionally selected examples with isolated and obvious flaws. This selection strategy was adopted strictly for pedagogical purposes-to provide clear, unambiguous vi- sualizations of each specific aesthetic indicator (e.g., illustrating exactly what "Balance" violation looks like in isolation). Readers should be aware that the full AesEval- Bench and - -rain datasets are significantly more diverse and challenging. They encompass a wide spectrum of diffi- culty, ranging from the clear, single-flaw examples shown in the showcase to complex, multi-flaw designs (similar to the real-world examples in Fig. 5 where multiple indicators (e.g., Alignment, I. Legibility, and Color Harmony) may be compromised simultaneously.
G TASK-SPECIFIC PROMPT SHOWCASES.
To ensure a rigorous evaluation of paasoing--augmente models (e.g., GPT-ol, GPT-o3), also formulated a specific set of Optimized Prompts designed to elicit their chain-of-thought capabilities. As detailed in Tab 9 distinct from the direct queries used for standard VLMs ("Original Prompt"), these optimized prompts explicitly instruct the model to engage in step-by-step analytical pro- cess. However, even task-specyf designed prompts could not improve the model's aesthetic understanding capability, revealing the limitations of general reasoning in this task.
H STATISTICS TESTS
We conducted rigorous hypothesis testing ensure the reliability of our results reported in Table 6.
16

---

Published as a conference paper at ICLR 2026
Algorithm Data Construction Pipeline via Synthetic Perturbation
Require: Original Design D, Metadata M (in JSON format), Perturbation Library P, Number of perturbed elements тt.
Ensure: Perturbed Design D', Updated Metadata M'.
1: Initialization: M' + M
2: Element Selection: Randomly select subset of elements E {E1, 62 ................................ end from M'.
3: for all 6 ε E do
4: Perturbation Selection; Randomly sample operation op E P applicable the type of ez-
5: Parameter Sampling: Sample perturbation intensity ó target attributes.
6: if op is Layoul Perturbation then
7: es.pos t e:.pos + L (1posp 6oos) {Shift position]
8: else if op is Font Perturbation then
9:
if op is Size Change then
10:
si. &- fj.size ((idzizee Ósize)
11:
else if op is Font Swap then
12:
e;.font Sample
(ei.font)
13:
end if
14:
else if op is Color Perturbation then
15:
op is Low Contrast then
ey.color SampleNear( [M. Mbbacggoondd color, else if op is High Contrast Clashing then eg.color + Invert Moveen(M.dominnttcom coominant_colorf Noise 19: end if
17:
else if op is Graphic//mage Perturbation then
21: if op is Replacement then
22:
e + Sample (ImageLibrary)
23:
else if op is Resolution Reduction then
24:
eu.quality <- Downsampleee,.ssre factor)
25: else if op is Blur then
26:
e,eeffect +- GaussianBlur(e,ssce
27: end if
28: end if
Update: Update element e; within metadata M'.
30: end for
31: Rendering: D' Render(M") (Re-render design using updated JSON]
32: return D'.
Aesthetic Judgment Region Selection: Since these are classification tasks, we applied McNemar's Test to analyze the discordance between models. The results show signifi- cant differences, with p-values of p = 0.004 and < 0.001, respectively, confirming the effectiveness of our fine-tuning pipeline.
Precise Localization: For loU scores, we performed a Paired T-Test. Our fine-tuned model achieves significant lead over the strongest baseline (GPT-5), with a t-statistic of 4.82 and p-value of 0.01.
I BENCHMARK SHOWCASE
In this section, we provide visual examples to better illustrate the evaluation criteria for the various aesthetic dimensions within the Beseval-Benchmark. As shown in Table [0] Table [4] Table I7] Table [5] Table [3] Table 12 and Table [6] each showcase presents a side-by-side comparison of designs that exemplify positive and negative attributes for specific criterion. These examples serve to clarify the standards used for judging aspects such as layout alignment, color harmony, graphic quality, and font legibility, offering a tangible guide to our benchmark' methodology.
17

---

Published as a conference paper at ICLR 2026
Invitation to event
ADS Onty
X STARTUP JOB FAIR
plas
World AIDS Day
(d)
TA-K ABOUT AJDS
Decembertt
600
END Violence Againet
(a)
SCHOOL MAGAZINE
PANOPAL'S LEORESS
Are You
ACAGEMIC ACHEVEMENTS
Young and Passionate?
SPONTS NIHLUGHTS
Dec
ARTS
connection cecere audeeiiin peppeetin
DAY FOR THE EUIH OAAAIIONIO NIVI EORENCEAACAA SETVEEEE
ALIVA SPITLIGHT
(c)
YOUR LOGO
(b)
(d)
PIRISID INVEALYY
Figure 5: Representative samples from real-world flawed cases collected by professional designers. Unlike the synthetically perturbed benchmark, these designs curated by professional designers to represent authentic aesthetic defects encountered in real-world workflows. The samples exhibit flaws such (a) visual clutter and inconsistent orientation (e.g., "World AIDS Day"), (b) spatial imbalance and disconnected elements (c.g.. "Startup Job Fair"), (c) grid-based alignment errors (c.g., "School Magazine"), and (d) typographic obstruction (e.g.. "End Violence"). This dataset serves as rigorous aut-Of-Distribution benchmark to evaluate model generalization beyond synthetic patterns.
Task
Original Prompt
Optimized Prompt
Aesthetic Judgment
SIndicatorsS + Answer it with one word 'yes' or 'no'.
Analyze the design based on SIndicatorsS. Evaluate the visual elements step-by-ste to determine if they meet the standard. Then answer with one word 'yes' or 'no'.
Region Selection
SIndicatorsS + Please only provide the index of Not aesthetic element given bbox choices. A. [] B.[ C. D.[]
Examine the candidate regions (A, B, D) regarding $IndicatorsS. Reason through the visual details to identify which specific region exhibits the flaw. Then output the index of that element.
Precise Localization
SIndicatorsS + Please only provide bounding box of the Not aesthetic element... If there is no any problems, please return 'None..
Analyze the entire design to pinpoint any element that violates $IndicatorsS. If a flaw found, step-by-step determine its exact spatial coordinates. Output the bounding box in format, return 'None..
Table 9: Comparison of Original Prompts Optimized Prompts for Reasoning Models.
18

---

Published as a conference paper at ICLR 2026
Benchmark Showcuse
Layout-Aliuumet Seaphic-Qualiin
MARSALA HAIR STUDIO
MARSALA HAIR STUDIO
Explanation: Clear background, center-aligned text.
The background is blurry and the words in the middle are not aligned.
Table 10: Examples of Layout-Alignmen and Sraphic-Quallty in LesSEval-Benchardd
Benchmark Showcase
FoghtLegbillty
2
Buy paicprice
malpprice
Explanation:
The yellow font in the lower left corner is clearly visible.
The yellow text the lower left corner becomes blurred and invisible.
Table 11: Examples of Font- Font-Legbility in AesEval- AesEval-Benchmark. Benchharr..
Benchmark Showcase
PROTEST FRIN CORPMAVIRUS?
PROTECT ME FROM
BORONAITRSS?
COVIDII
Explanation:
The background is woman wearing a The background a beautiful landscape photo, which mask, which relevant.
does not fit the theme.
Table 12: Examples of Graphic-Relevance in SosEval-Benchmak..
19

---

Published as a conference paper at ICLR 2026
Benchmark Showcase
b Layout- Whitespace, Layout-Layenn
OEIREOO CILLS
HAPPY CRI HODOX CERISINAS.
Explanation:
The text is in the blank space, with no blank space overlap.
The text has been moved to the animal's 's feet, resulting in white space below and stacked elements.
Table 13: Examples of Layoutt-Witesppe Whitespace and Layout-Layering in
Benchmark Showcase
Layout- Balance
CHRISTMAS
Explanation: The overall layout of the picture is bal- The background of the picture is moved upwards, and anced and coordinated.
the balance of the whole picture is broken.
Table 14: Examples of Layout- Balannc in
Benchmark Showcase
> Hiierr--y
MPP
Valontiee's & Day!
BUL BEAT & 3EF
Explanation: All the fonts in the picture consis- The font in the picture is disturbed and looks like it is tent, which gives sense of hierarchy. not on the same level the previous font. There no sense of hicrarchy.
Table 15: Examples of Layout- in AesEvva-Bennhmarkk
20

---

Published as a conference paper at ICLR 2026
Benchmark Showcase
Gruphic-uualit, Color-Contrast
HERRY CHELSTMAS
MEREY CHRISTMAS DOE
Explanation: Clear background, black and white col- ors have good contrast.
The background blurred and color changes from white to black, with no contrast to the backgriond.
Table 16: Examples of Sraphic-Qualige and Color-Contrast in Aes Benchharrk
Benchmark Showcase
b Color-Harmonny Color-Appealing, Cyych-osyy
AUTHENTIC
AITHENTE
Barber
Barber
Explanation:
The whole picture has harmonious and bcautiful colors.
The middle element turns green, which makes the whole picture look disharmonious and unappealing. Green is strange and cause bad psychological effects.
Table 17: Examples of Color-Harmony, Color-Appealing and Color-Psychology AesEval- Benchmark.
21


Reviews:
[{"review_id": "kEM3Zm1EWS", "reviewer": "Reviewer_8akP", "summary": "This paper introduces AesEval-Bench, a comprehensive benchmark for assessing graphic design aesthetics using vision–language models (VLMs). It covers four dimensions (typography, layout, color, graphics), twelve indicators, and three quantifiable tasks (aesthetic judgment, region selection, precise localization). The authors systematically evaluate a range of VLMs and propose a novel dataset construction pipeline using human-guided VLM labeling and indicator-grounded reasoning, showing that fine-tuning with this data improves performance.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Comprehensive Benchmark: AesEval-Bench advances the field by covering a wide range of design principles and providing quantifiable tasks.\nSystematic Model Evaluation: The paper compares proprietary, open-source, and reasoning-augmented VLMs, highlighting current limitations.\nInnovative Dataset Creation: The human-guided VLM labeling and indicator-grounded reasoning approach is scalable and interpretable.\nClear Experimental Protocols: Evaluation metrics and ablation studies are well described.\nPerformance Gains: Fine-tuning with the new dataset yields substantial improvements, even for smaller models.\nActionable Insights: The analysis of input components (image, explanation, metadata) is useful for future research.", "weaknesses": "Binary Annotation Limitation: The evaluation dimensions are annotated in a binary fashion (yes/no), reducing the complexity of the task to simple classification. This raises the question of whether large VLMs/LLMs are necessary, as simpler models or rule-based systems might suffice.\nOverkill of VLM/LLM for Binary Tasks: The paper does not convincingly justify the use of large VLMs/LLMs for benchmarking binary data. There is little discussion on the trade-off between model complexity and task requirements, and no baseline comparison with lightweight models.\nDataset Creation vs. Model Utility: While the dataset creation pipeline is interesting, its practical value is unclear. The fine-tuning experiments are performed only on open-source LLMs, which are shown to be suboptimal in the benchmarking. The real utility of the dataset would be demonstrated by showing improvements in strong proprietary models (e.g., GPT-5) when fine-tuned with this data.\nUnclear Value for State-of-the-Art Models: The benchmarking shows that models like GPT-5 already outperform open-source models out-of-the-box. However, the paper does not test whether a competitive variant of GPT-5 (via in-context learning may be) with the new dataset leads to meaningful gains. Without this, it’s unclear if the dataset provides real value beyond what top models can already achieve.\nMissing Baseline Comparisons: There is no comparison with traditional computer vision models or simple classifiers trained on the same binary labels, making it difficult to assess whether the proposed approach is truly necessary or effective.\nIndicator Entanglement: The twelve indicators are not fully disentangled, which could affect the interpretability and granularity of the results.\nSubjectivity: Highly subjective aspects of design, such as creativity, are not addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jsgFUzalfv", "reviewer": "Reviewer_vL31", "summary": "This submission presents AesEval-Bench (a graphic design aesthetic assessment benchmark with 4 dimensions, 12 indicators, 3 tasks) and AesEval-Train (a training dataset via human-guided VLM labeling and indicator-grounded reasoning), alongside evaluations of 10 VLMs and fine-tuning tests. However, fundamental flaws in benchmark validity, insufficient novelty, unrigorous analysis, and unsupported claims invalidate its contributions, justifying rejection.", "questions": "1. How do you verify that synthetic perturbations align with real-world aesthetic flaws? Please provide data on human experts' recognition of the intended flaws in perturbed designs.\n2. What task-specific prompting strategies were used for reasoning-augmented VLMs, and how were they optimized for design aesthetics?\n3. How do you rule out overfitting to synthetic data for the fine-tuned model's performance? Please provide cross-validation on real-world flawed designs.\n4. How do you ensure indicator independence? Please provide correlation analysis of model performance across indicators.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Addresses the underexplored gap of graphic design aesthetics assessment, distinct from general image aesthetics benchmarks, with potential utility for designers and generative AI systems.\n2. AesEval-Bench's structure (4 dimensions, 12 indicators, 3 tasks) aligns with established design principles, covering both global (aesthetic judgment) and local (region selection, precise localization) aesthetic evaluations.\n3. The AesEval-Train pipeline offers practical solutions to scale domain-specific annotation, particularly via human-guided VLM labeling and indicator-grounded reasoning.", "weaknesses": "1. Benchmark Data Lacks Authenticity: Flawed examples rely on synthetic perturbations (e.g., element repositioning, font alteration) of professional Crello designs. These artificial changes do not replicate real-world aesthetic defects, reducing the benchmark to testing sensitivity to synthetic manipulations rather than authentic aesthetic judgment.\n2. Insufficient Novelty: Core dimensions/indicators derive from existing design literature without novel taxonomical insights; tasks are standard in visual assessment; indicator-grounded reasoning is an incremental adaptation of existing grounded visual reasoning, not a transformative innovation.\n3. Unrigorous Evaluation: Claims about reasoning-augmented VLMs' poor performance lack controlled experiments (e.g., ignoring prompt/architecture differences); key design-focused baselines are omitted; performance metrics lack statistical significance tests, making model comparisons unreliable.\n4. Unsupported Fine-Tuning Claims: Ablation studies fail to rule out trivial factors (e.g., context length) for performance gains; the fine-tuned model's extremely low precise localization performance undermines claims of \"effective supervision\".\n5. Critical Limitations Unaddressed: Omits key design types (e.g., infographics) and subjective aesthetic aspects (e.g., creativity); no inter-annotator agreement data for human labels, compromising ground truth reliability; indicators are not disentangled, causing ambiguous evaluation signals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b659fWeCJA", "reviewer": "Reviewer_bVZv", "summary": "This paper introduces AesEval-Bench, a new benchmark and dataset for evaluating how well vision–language models (VLMs) assess the aesthetic quality of graphic designs. It systematically compares different VLMs and provides fine-tuning data, showing that while current models lag behind human-level aesthetic judgment, human-guided labeling and indicator-grounded reasoning significantly improve performance.", "questions": "1. Could you provide details on the training data?\n2. Could you provide justifications of using Crello dataset as data source?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Introduces a comprehensive benchmark (AesEval-Bench) for aesthetic evaluation.\n2. Provides systematic comparison and fine-tuning data for VLMs.\n3. Shows that human-guided labeling and reasoning improve aesthetic assessment.", "weaknesses": "1. Important details are missing. In section 4 TRAINING DATA CONSTRUCTION, if I did not miss anything in Sec.4, authors do not introduce most important details on training data, such as data scale, data source. Without those essential information, we cannot accurately evaluate the dataset and the following datasets. If it is sampled from the same source as the benchmark data, it may have have overfitting issues.\n\n2. Another big concern is, the source data is very limited, sampled from Crello dataset (Yamaguchi, 2021). Single source itself may not be a serious issue because the source dataset could be diverse. But authors fail to provide detailed analysis of the Crello dataset. I am not sure if it can be the qualitied source data. From examples showed in the submission, I feel like the design works are a bit similar to each other in terms of style. I am concerned about it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yvORJv6g7R", "reviewer": "Reviewer_GAuh", "summary": "This paper proposes and constructs AesEval-Bench, a benchmark for systematically evaluating the ability of vision-language models (VLMs) in graphic design aesthetic assessment. The contributions include: formalizing aesthetic evaluation as a multi-granularity question-answering task involving global judgment, region selection, and precise localization; building a high-quality, multi-dimensional benchmark based on professional design datasets; revealing the limitations of current VLMs through comprehensive experiments; and proposing a data construction method that combines human-guided annotation with indicator-grounded reasoning to generate effective training data for model improvement.", "questions": "1. The authors are advised to improve the table formatting by clearly highlighting the best and second-best results, and to include a comparison of VLMs fine-tuned on AesEval-Train with other baseline models, in order to validate the effectiveness of the proposed training approach.\n2. The authors should add an analysis of inter-annotator agreement (e.g., Cohen’s Kappa or ICC) and provide annotations regarding the annotators’ professional backgrounds, number, and selection criteria, to enhance the credibility and interpretability of the benchmark.\n3. It is recommended that the authors clarify the method for generating \"problem regions,\" including the strategy for candidate region segmentation (e.g., based on design elements or sliding windows) and the detailed process of introducing aesthetic flaws, to improve transparency and reproducibility.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper demonstrates significant novelty in the application of vision-language models (VLMs) by being the first to systematically explore their capability in assessing graphic design aesthetics. Unlike prior work focused on image captioning or general visual question answering, this study introduces AesEval-Bench—a multi-dimensional, multi-task benchmark that categorizes aesthetic evaluation into three granular levels: global judgment, region selection, and precise localization. It covers twelve specific design indicators across four core dimensions (layout, color, font, and graphics), grounded in professional design principles. This structured and fine-grained approach bridges a critical gap in AI support for creative domains, offering both theoretical insights and practical utility for automated design critique systems.", "weaknesses": "1. Lack of Highlighting Best/Worst Performances and Missing Comparison with Fine-tuned Models\nThe result tables (e.g., Table 2–5) present raw scores without highlighting the best and second-best performances (e.g., via bold or underline), nor do they indicate the worst or second-worst results. This makes it difficult for readers to quickly identify performance leaders and laggards. More critically, the tables only include existing general-purpose VLMs, but do not report results from the fine-tuned VLMs trained on the proposed AesEval-Train dataset. This omission is a major flaw, as it fails to validate whether the proposed training method actually improves model performance, undermining the paper’s claimed contribution.\n2. Insufficient Discussion on Human Annotation Reliability and Rater Background\nAlthough human annotations are central to constructing the benchmark and training data, the paper does not report inter-annotator agreement metrics (e.g., Cohen’s Kappa, Fleiss’ Kappa, or ICC). It also lacks details on the annotators’ qualifications (e.g., professional designers), the number of raters, or the selection criteria. Given the subjective nature of aesthetic judgment, the absence of such analysis raises concerns about the reliability of the ground-truth labels and the validity of the reported model performance.\n3. Ambiguity in the Definition of \"Problem Regions\" and Lack of Clarity in Problem Injection\nThe paper does not clearly specify how the candidate regions for the region selection and bbox prediction tasks are generated. Are they derived from structured design elements (e.g., text boxes, image frames), or produced via sliding windows, superpixels, or other segmentation methods? Furthermore, while the paper mentions that aesthetic flaws are introduced through human intervention to create negative samples, it fails to describe the concrete procedures for modifying the original designs. It also lacks details on who performed these modifications and according to what guidelines.\n4. Limited Evaluation of Model Generalization\nAll experiments are conducted on the Crello dataset, with no testing on out-of-domain design data. As a result, the reported performance may not generalize to other design contexts or cultural aesthetics, limiting the practical applicability and robustness of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)