GIR-Bench: Versatile Benchmark for Generating Images with Reasoning
Authors: Hongxiang Li
Yaowei Li
Bin Lin
Yuwei Niu
Yuhang Yang
Xiaoshuang Huang
Jiayin Cai
Xiaolong Jiang
Yao Hu
Long Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Evaluation
Unified Multimodal Model
Visual Generation
Primary area: datasets and benchmarks

Abstract: Unified multimodal models integrate the reasoning capacity of large language models with both image understanding and generation, showing great promise for advanced multimodal intelligence. However, the community still lacks a rigorous reasoning-centric benchmark to systematically evaluate the alignment between understanding and generation, and their generalization potential in complex visual tasks.
To this end, we introduce \textbf{GIR-Bench}, a comprehensive benchmark that evaluates unified models across three complementary perspectives. 
Firstly, we explore whether models can consistently leverage the same knowledge for both understanding and generation (GIR-Bench-Uni). 
Secondly, we investigate whether models can perform reasoning-centric text-to-image generation that requires applying logical constraints and implicit knowledge to generate faithful visual content (GIR-Bench-T2I).
Thirdly, we evaluate whether models can handle multi-step reasoning in editing (GIR-Bench-Edit). 
For each subset, we carefully design different task-specific evaluation pipelines tailored for each task. This enables fine-grained and interpretable evaluation while mitigating biases from the prevalent MLLM-as-a-Judge paradigm.
Extensive ablations over various unified models and generation-only systems have shown that: Although unified models are more capable of reasoning-driven visual tasks, they still exhibit a persistent gap between understanding and generation. The data and code for GIR-Bench are available at 
\url{https://anonymous.4open.science/r/GIR-Bench-7E40}.

Published as a conference paper at ICLR 2026
GIR-BENCH: VERSATILE BENCHMARK FOR GENER- ATING IMAGES WITH REASONING
Hongxiang Li Yaowei Li Bin Lin', Yuwei Niu?, Yuhang Yang", Xiaoshuang Huang', Jiayin Cai', Xiaolong Jiang', Yao Hu Long Chen It
The Hong Kong University of Science and Technology Peking University 3 University of Science and Technology of China Xiaohonnshu Inc.
Each chicken has two legs, and each rabbit has four legs. There are five animals total, and the number of legs visible on the ground fourteen. Show all animals. How many chickens and how many rabbits there?
Chickens Rabbits
What is the name of this landmark?
Gemini-
55-Flash
-Image
hi BAGEL
Merlion
(b) Reasoning to-Generation
User
photo of the half-liu -lon half-fish statue spouting water at the the Singapore River.
User
Solve the Sudoku puzzle. Fill in all the empty cells with the correct numbers.
6PT-5
grid:
4 2
4123
solution:
214 2
3214
hd
1342
4
2431
4
2
3
2
1
4
GPT-
4
1
2
3
Image-1
2
3
2
4
1
4
2
4
1
2
3
4
1
(a)
(c) Beasoning-to-Editiin
Figure 1: Illustration examples of GIR- GIR-Bench, which highlight misalignments between the reason- ing and generation capabilities of state-of-ther the unified multimodal models.
ABSTRACT
Unified multimodal models integrate the reasoning capacity of large language models with both image understanding and generation, showing great promise for advanced multimodal intelligence. However, the community still lacks rig- orous reasoning-cemt centric benchmark to systematically evaluate the alignment be- tween understanding and generation, and their generalization potential in complex visual tasks. To this end, we introduce GIR-Bench, a comprehensive benchmark that evaluates unified models across three complementary perspectives. Firstly, we investigate understanding genergtionettiinn consistency (GIR- Bench-UGC), asking whether models consistently leverage the same knowledge in both under- standing and generation tasks. Secondly, we investigate whether models can perform reasoning-centric text-to-image generation that requires applying logi-
Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
cal constraints and implicit knowledge to generate faithful visual content (GIR- Bench-T2I). Thirdly, we evaluate whether models can handle multi-step reason- ing in editing ent task-specific evaluation pipelines tailored for each task. This enables fine- grained and interpretable evaluation while mitigating biases from the prevalent ILLL--a----udge. paradigm. Extensive ablations over various unified models and generation-only systems have shown that: Although unified models are more capable of reasoning- driven visual tasks, they still exhibit persistent gap between understanding and generation. The data and code for BeR-Bench are available https: //g thub. com HHH//KUT---T LondsT--LogGruupp Ceieooopp//RRRR Bench..
For each subset, we carefully design differ-
INTRODUCTION
Image generation and editing techniques (Stability AI, 2024; Labs, 2024; Batifol et al., 2025; Liu al., 2025) have evolved rapidly, demonstrating strong capabilities in producing high-quality vi- sual content aligned with explicit prompts. However, existing models still struggle with complex visual generation tasks that require multi-step reasoning. This limitation has triggered the research focus toward unified multimodal models, where single model supports both image generation and understanding. By leveraging the intrinsic knowledge and reasoning abilities of multimodal large language models (MLLM), such unified approaches enable richer expressiveness and more control- lable image generation. Recent breakthroughs, such as Gemini-2.5-Flash Flass-Image (DeepMind, 2025) and GPT-Image (OpenAI, 2025b), further highlight the transformative potential of this paradigm, showing that unified models can fundamentally reshape real-world applications, empowering users to accomplish complex visual tasks through natural language interaction. Compared to generation- only models, unified models (Deng et al., 2025; Chen et al., 2025a; Xie et al., 2025b) promise sub- stantial gains. With these advances comes critical research question: how do comprehensively evaluate the extent to which enhanced multimodal understanding improves generation capabilities?
Earlier image generation benchmarks (Ghosh et al., 2023; Huang et al., 2023; Hu et al., 2024) mainly focused on object attributes and compositional evaluation, but remained limited to shallow mappings between text and visual content. Recent works (Zhao et al., 2025; Niu al., 2025; Sun et al., 2025) have attempted to consider the reasoning capability. However, existing benchmarks remain limited in both their evaluation dimensions and protocols, making them inadequate for capturing the full capabilities of unified models.
For evaluation dimensions, they cannot quantify the alignment between reasoning and generation within unified models. As shown in Figure I, we observe consistent misalignments in transferring knowledge to generation, reasoning to generation, and reasoning editing. For instance, a unified model may correctly recognize a real-world entity (e.g., the Merlion in Figure 1), but still fail to generate it with explicit description. Revealing and quantifying the gap is crucial, as it not only uncovers the inherent limitations of current unified models but also verifies whether they can deploy their knowledge and reasoning abilities equally across understanding and generation tasks. For evaluation protocols, prevalent benchmarks always propose some challenging tasks such as idiom interpretation to investigate unified models, but they cannot decompose these designed reasoning- centriccceenriccccci tasks into more evaluable and interpretable forms. Instead, they rely heavily on the MLLM-as-a-Judge paradigm, where MLLMs serve directly as evaluators and evaluation scores are obtained through visual question answering. Such dependence inevitably couples the evaluation result with the biases and limitations of multimodal models themselves.
To this end, we introduce GIR-Bench, a benchmark designed to systematically explore the capabil- ity boundaries of unified models in reasoning- -riien image generation and editing. The benchmark consists of three components: 1) GIR-Bench-UGC: To assess understanding- -generrtnnnnnnnnnn consis- tency (UGC), we evaluate whether models can reliably leverage the same knowledge for both rec- ognizing and generating real-world entities. Specifically, we compile 300 entities from zoology, botany, and geography, design implicit reasoning-oriente prompts, and pair them with curated ref- erence images (Figure I(a)). This design allows us to systematically measure the consistency gap between understanding and generation. 2) CIR-Bench-T2I: We then explore reasoning-centir centric text- to-image generation (c.f., Figure 1(b)). requires models not only to retrieve relevant knowledge but also apply precise logical reasoning to faithfully satisfy specified constraints. We design 300
2

---

Published as a conference paper at ICLR 2026
Numerical Reasoning
Spatial Layout
Text Rendering
World Knowledgee
Prompt: A photo of duuks and dege, There ore legs
Prompt: A picture showing o
Prompt: photo on org
Prompt photo of The gold leat covered bou der precorioualy balanced diff edge in Myanmar topped by
bicycle, deg, can, and cat. the
1988 Meke bilbbardd featuring action
call-ta- bald
onmil Displty the onniis cierly.
lic, onimate the left and
vehicles the right.
letters...
NIKE
S GPT-Teage-t Detection: Duck X Deg X. buck Dog
HDreae- II-Ful
Qeen-Teage
hui RAGEL
Real World
Detection: [vehicles, crimuls] Target: [animals, vehicles]
Detection: NIKE 1988
Sinilurity: Low
Target
Cotegery: Kyoletbyo Pageda Myyonmrr
Visual Puzzle
Visual Logic
Reasoning Perception
Proupt Resrrange shuffled joHa puzzle
Prompt Selve the given Susiku puzzie shown in the mage Fill empty the correct sccarding
Prompti Plxzae sesment the negion from the inage described as (the people most Ske jadges)...
image restore orggnol [My)
Input Image
Exput Trage
Teput Image
4
3
4
5
4
2
4
3
2
2
5
Groun Trunh
Flysh-
Groad Touth
Ini BAGEL
Groand Truth
GFT- GFT-Trage-3
Figure 2: Examples of leading models on the GIR-Bench. Designed complex and various tasks pose challenges to current models.
carefully crafted prompts spanning three dimensions: numerical reasoning, spatial layout, and text rendering. 3) GJIR-eech-Eddit, We further evaluate whether models can perform global planning and reasoning-driven local modifications (c.f., Figure Icc)). We construct 370 cases spanning vi- sual puzzles, visual logic, and reasoning perception. Each case consists of an input image and its associated ground-truth image, i.e., reducing evaluation bias.
In terms of evaluation, unlike previous works that heavily rely on the MLLM-as-a- Judge paradigm, we design series of task-specific evaluation pipelines tailored for each task. Our results demon- strate that these specialized evaluations not only provide fine-grained and interpretable assessments but also effectively mitigate the inherent biases of large multimodal models. Within GIR-Bench, we systematically evaluate 21 state-of-the-ae models. Our challenging benchmark reveals the lim- itations of leading models in performing generation tasks that require complex visual reasoning, as shown in Figure 2. Massive results show that integrating understanding with generation enables models to perform more complex visual tasks. However, for unified models, a significant gap mains between understanding and generation. Thus, effective integration of the two is crucial for unlocking the potential of unified multimodal models. In summary, our contributions are threefold:
We propose GIR-Bench, comprehensive reasoning- -entric benchmark that evaluates unified multimodal models across three perspectives.
We design a diverse suite of tasks with task-specific evaluation pipelines that provide fine-grained and interpretable metrics, moving beyond the limitations of the MLLM-as-a-Judge paradigm. Through extensive experiments, we reveal both the performance gap between unified and generation-only models and the internal gap between understanding and generation.
2 GIR-BENCH
In this section, we introduce the main components of GIR-Bench. Section 2.1 details the data sources, dataset construction, prompt suites and evaluation pipeline of CIR-Bench-UGG. In Sec- tion 2.2, we present the task dimensions and evaluation suite of GIR- HIR--Bench-22) Section 2.3 elab- orates on the evaluation dimensions, image sources and evaluation metrics of CIRR-Bncch-Edit. Fi- nally, the experiments and the insights derived from GIR-Bench will be discussed in Section 3.
3

---

Published as a conference paper at ICLR 2026
Image
Human Verification
Prompt: A photo of the
giant steel winged sculpture stonding beside the Al road near Gateshead.
Image Generation
Prompt Generation
Unified Moooeee
Category: Angel the North
Real-world Image Set
Question: of the following options
cor cornesponds to the landmork shown the <image>
Ophions:
Web Collection
(A) Colosveum Itely
(B) Sydney Opera Houtle Ausiralli
Understanding
(C) Vanus Willendrf Austria
Manual Filtering
(0) Angel the North United Kingdom
2 SPT-5: (D) Iel BAGEL: (D)
Figure 3: Illustration of GIR-Bench-UGG For each real-world entity, an implicit prompt drives text-
to-image generation, while the corresponding real image is used for image understanding evaluation.
2.1 GIR-BENCH-UGE
We collected 300 real-world entities from the Internet and open-source datasets, covering domains such as zoology, botany, and geography. Although previous works have explored the dimension of world knowledge, they typically remain limited to textual interpretation or text-image alignment without a deeper investigation. To evaluate reasoning ability, we utilize GPT-4o to generate implicit descriptions of each entity category, incorporating representative features such as visual appearance, historical context, and distinctive attributes. These prompts are then manually verified and refined to ensure that each uniquely corresponds to the real entity. Meanwhile, quality images to serve as reference exemplars for each entity. With entity categories, prompts, and reference image sets in place, we construct paired evaluation sets for both image understanding and image generation as shown in Figure 3. We compute the average DINOv3 (Siméoni et al., 2025) feature similarity between the generated image and the reference image sets as the evaluation metric. cases.
curate set of high-
For image understanding, reference images are used to formulate visual question-answenee This design enables to systematically investigate whether unified models can leverage the same underlying knowledge and reasoning for both understanding and generating the same entity.
2.2 GIR-BENCHH
EIR-Bench-T22 consists of three text-to-image generation tasks that require deep reasoning. We design 300 carefully crafted prompts and construct evaluation pipelines by object detection and text recognition models. Although such problems are trivial for multimodal models, existing image generation models often fail to reason correctly and generate the expected images.
Our design of BIR-Bench is guided by three concrete principles intended to keep the benchmark objective, interpretable, and reproducible:
Objectivity over Subjectivity: While many advanced reasoning scenarios (e.g., causal reasoning open-ended commonsense) typically rely on the "MLLLM-as-a-Judg"" paradigm, this approach often introduces bias. We prioritized tasks with deterministic solutions (e.g., the unique solution in Sudoku, the original image in Jigsaw Puzzles, or exact answers in arithmetic). This ensures that our evaluation results are reproducible and indisputable.
Availability of Ground Truth: We exclusively selected tasks where ground truth can be pro- grammatically generated or strictly verified. For instance, Spatial Layout is verified via bounding box coordinates, and Text Rendering is checked via OCR. This quantifiable nature is a prerequisite for building a rigorous benchmark.
Focus on Implicit Reasoning & Planning: We explicitly excluded tasks solvable by simple seyyoord-to-maage mappings. The selected tasks (e.g.. Visual Puzzle and Numer i cal Reasoni in) compel the model to perform implicit reasoning global planning before gener- ating visual content that satisfies the constraints. This is key to measuring whether a model truly comprehends the logical constraints within a prompt.
Numerical Reasoning. We design prompts that explicitly state mathematical constraints and record the corresponding ground truth, e.g., the chicken-rabbit problem. while models must reason over these constraints infer the correct objects and their quantities. For example, given the prompt
4

---

Published as a conference paper at ICLR 2026
"A photo of ducks and dogs. There are total of 10 legs visible, and the total number of animals is 4. Display all the animals clearly.", the expected output is 3 ducks and 1 dog. For evaluation, we apply object detection to extract the categories and counts of objects in the generated image and compare them against the ground truth. Notably, a case is counted as correct only when all object counts extracted from the generated image match the expected numbers. This strict criterion is necessary since the quantities of objects specified in the prompts are logically interdependent. Partial correctness indicates a broken reasoning chain, whereas full correctness ensures that the model has followed the intended reasoning process.
Spatial Layout. We design prompts that specify how objects should be arranged according to constraints (e.g. categorical rules or ordered attributes) and record the corresponding ground truth layouts. For example, the prompt "A picture showing bicycle, a dog, a car, and cat. Arrange the ttms in horizontal line, with animals on the left and vehicles on the right." requires that the bounding boxes of animals appear to the left of those of vehicles. We evaluate results by extract- ing bounding boxes from generated images via object detection and verifying whether the spatial relations conform to the specified constraints.
Text Rendering. While existing models already demonstrate strong text rendering capabilities, their ability to reason over textual knowledge remains underexplored. To this end, we collect 60 short quotes slogans and design implicit descriptions that correspond to them. For example, the target text "Just do if" corresponds to the prompt "A photo of an original 1988 Nike billboard featur- ing the brand's iconic three-word call-to-action slogan in bold capital letters." For evaluation, first extract text from generated images. Unlike traditional settings, our prompts are deliberately im- plicit, which often leads models to generate the target text along with additional irrelevant content. Common metrics (e.g., word accuracy and edit distance) are not suitable, since they wrongly penal- ize extra words. Our goal is instead to assess whether the model successfully generates the ground truth text while allowing the presence of additional content. To this end, we propose the word-level continuous substring score the primary evaluation metric. It is defined swe(g. p) [Wasn(g.p)) where (g) denotes the set of words in the ground truth g. and Wmatch (g. p) counts the number of ground truth words that are fully covered by continuous character spans in the predicted text p.
(W(g)]
2.3
GIR- EDIT
HiR-Bench-Edit evaluates the image editing capabilities of models along three novel dimensions. Unlike previous works, each editing case in our benchmark includes both an input image and a corresponding ground truth image, thereby mitigating bias in evaluation.
Visual Puzzle. We filter real-world images collected in Section 2.1 to retain near-square, high- resolution photos (minimum side length 1024 px and aspect ratio < 1.2), and resize each to square. The processed images are partitioned into grids, and the tiles are randomly permuted such that at least half of the tile positions are altered. Given the shuffled image and the accompanying instruction, the model is required to reconstruct the original image, thereby evaluating its ability to integrate semantic understanding with spatial reasoning in order to restore both the global struc- ture and the local coherence of natural images. For evaluation, we compute the Fréchet Inception Distance (FID) (Heusel al., 2017) between generated images and ground truth. To facilitate com- parison with other metrics, we further normalize FID to the range [0, 1]. where larger values indicate better reconstruction quality.
Visual Logic. We construct a high-quality dataset of Sudoku puzzles using a systematic generation pipeline. For ground truth solutions, we employ a penstraint-rropagation algorithm that maintains candidate sets for rows and columns, filling the grid iteratively with randomized choices while en- suring the validity of standard Sudoku constraints. For input puzzles, we adopt a deductive removal strategy, ensuring that each removed digit preserves the uniqueness of the solution. The puzzles and their corresponding solutions are then rendered into images. For evaluation, we apply text detection to extract the digits and their positions from generated images, and compute accuracy by comparing predictions against the ground truth.
Reasoning Perception. We select high-quality images from the LISA (Lai et al., 2024) dataset and filter those with small aspect ratios and resize them to a square. Editing instructions are constructed from their implicit textual descriptions. The task requires models edit the target regions into
5

---

Published as a conference paper at ICLR 2026
Table 1: Results of various multimodal understanding and generation models on GIR-Bench-UGC.
Image Generation
Image Understanding
Type
Model
7nology Botany Geography Overall Zoology
Botany Geography Overall
Owen2.5VL-7B
0.943
0.990
1.000
0.978
Qwen2.5-VL.322B
0.951
0.990
1,000
0.976
Und
GPT-5
0,983
1.000
1.000
0.994
Ge5ini-2.5-Flash
0.991
1.000
1.000
0.997
SD-3.5-Large
0.263
0.163
0.437
0.288
Gen
HiDrram-Il-Full
0.298
0.218
0.617
0.378
FLUX.I-schnell
0.239
0.197
0.440
0.292
Show-o2-7B
0,200
0.128
0.265
0.198
0.894
0.910
1.000
0.935
Janus-Pro-7b
0.201
0.111
0.321
0.211
0.813
0.810
1.000
0.874
BI.P33-NFXT--FT--B
0.260
0.169
0.359
0.263
0.951
0.970
1,000
0.974
Ovis-U1-3B
0.225
0.130
0.377
0.244
0,878
0.850
1,000
0.909
OmniGen2
0.214
0.218
0.451
0.294
0.932
0.940
0.984
0.952
OnPPic--Metquerr-999
0.269
0.195
0.440
0.301
Unified
UniWorld-V1
0.236
0.220
0.451
0.302
BAGEL-7B
0.242
0.200
0.445
0.295
0.911
0.900
1.000
0.937
BAGEL-7B w/ CoT
0.256
0.243
0.525
0.341
0.935
0.970
1.000
0.968
Qwen-Image
0.293
0.319
0.677
0.429
3cmmni-22.5-Flashe
0.449
0.559
0.772
0.593
GPT-Image-1
0.568
0.700
0.800
0.689
green while keeping the background unchanged. Conceptually, this corresponds to segmentation, but since directly predicting binary masks is out of scope for editing models, we use this proxy for- mulation. Specifically, we instruct the model to segment the target region and render it in solid, fully opaque green while leaving the rest of the image unchanged. For evaluation, we extract the edited regions from the model outputs and convert them into binary masks. The procedure combines color- threshold detection centered on the target green, enhanced, with channel-ratio based seed detection obtained by comparing the input and generated images. The resulting binary mask then compared with the ground-trut -ruth mask using Intersection-over Uvin- (IoU) as the evaluation metric.
EXPERIMENTAL RESULTS AND INSIGHTS
3.1 EXPERIMENT SETTINGS
Evaluated Models. We evaluate 21 representative models: Multimodal understanding models:
Qwen2.5-VL-70 (Bai et al., 2025), Qwen2.5-VL-322 (Bai et al., 2025), GPT-5 (OpenAl, 2025a),
Gemini-2.5 1omini-2.5-Flash (Comanici et al., 2025); Image generation models: SD-3.5-Large (Stability AI,
2024), HiDream-Il-Full (Cai ct al., 2025), I-schell (Labs, 2024); Image editing models:
FLUX.I-Komter (Batifol al., 2025), ICEdit (Zhang al., 2025), Step1X-Edis (Liu et al., 2025); Unified multinodal models: Uniworld (Lin et al., 2025), UniPic2 (Wei et al., 2025), BAGEL (Deng al., 2025), Ovis-U1 (Wang al., 2025a), OmniGen2 (Wu et al., 2025b), Show-o2 (Xie et al., 2025b), Janus-Pro (Chen et al., 2025b), BLIP3o (Chen et al., 2025a), Qwen-Image (Wu et al., 2025a), GPT-Image-1 (OpenAI, 2025b), Gemini-2.55 1.5-Flash-Image (DeepMind, 2025).
Implementation Details. For object detection, we employ the grounding capability of IntervV33.5 38B (Wang et al., 2025b) detect object categories and bounding boxes from generated images. For text detection, we use PPOCR v5 (Cui et al., 2025) to detect and recognize textual content, retaining only segments with confidence scores greater than 0.5.
3.2 EVALUATION ON REAL-WORK UNDERSTANDING AND GENERATION
Table I reports results on GIR-Bench-UGG, which jointly evaluates real-world entity understanding and generation. The left panel shows understanding accuracy, while the right panel presents gen- eration performance under implicit prompts. Overall, unified models outperform generation-onle systems reasoning-ces centrre generation tasks, indicating that joint training across understanding and generation yields tangible benefits. GPT-Image- achieves the best performance and signifi- cantly outperforms other models, providing strong upper bound. Unexpectedly, open-source uni-

---

Published as a conference paper at ICLR 2026
0.60
Calegory Input Prompt Input
Prompt: A photo of the sculpture cast Arturo Modica in 1989 impromptu gift symbolizing market resilience ofter the 1987 crash, now standing beside New York' primary stock exchonge.
0.55
0.50
Category: Charging Bull United States
0:45
30:40
0.35
0.30
0.25
Owen- Image
BAGEL w/ Cat
UniWorid--1
Und MQ
BAGEL
OmniGen12
BUP3O-NEXT.SP
Owis-U12
Janus- Pro
ARTIO MAR ormn
1987
owen-image (Prompt Input)
Qwen Qwen-image (Category Input)
World Image
Figure 4: Performance decline from category inputs to implicit prompts.
Figure 5: Qualitative cases in HIRR-Benc--GGC showing both direct category inputs and implicit prompt inputs.
fied models do not show a clear advantage in generation compared to strong generation-only models. To further probe this phenomenon, we conducted two complementary analyses:
Knowledge and Reasoning Capacity. We further probed whether generation failures stem from missing world knowledge from difficulty in reasoning. To this end, we evaluated image under- standing and reported the results in Table 1. Unified models exhibit consistently strong understand- ing: BLIP3o attains the highest overall score of 0.974, while the lowest, Janus-Pro, still achieves 0.874. Such uniformly high scores suggest that entity recognition and basic reasoning are not the main bottlenecks. Moreover, with respect to implementation, it is worth noting that WniWorld and UniPic2 do not release official image understanding scripts. Both are developed upon frozen Qwen2.5-VL-7B, and thus their understanding capability can be reasonably approximated by the performance of -wvn2.55VL-7B.
Bridging Reasoning and Generation. To disentangle the effect of reasoning, we compare two types of inputs: (i) category inpur, which directly specifies the target entity using the template "a photo of (category)" and (ii) the implicit prompt input, which requires reasoning to infer the entity. Figure presents the per-model performance under both conditions. Across all models, scores with prompt input exhibit marked reduction compared to category input, often by a significant margin. This pattern indicates that while models capable of rendering entities when explicitly named, their performance deteriorates when the entity must be reasoned. The limitation therefore lies not in the ability to render the object itself, but in transferring reasoned constraints into the generative process. We further illustrate this gap through qualitative case studies in Section 4.
3.3 EVALUATION ON REASONING-CENTRI TEXT-TO-IMAGE GENERATION
We report results on GIR-Bench T2I in Table Unified models outperform generation-only sys- tems on reasoning-centric generation tasks, indicating that joint training across understanding and generation yields tangible benefits. Nevertheless, even the strongest proprietary models (i.e., GPT- Image-1 and Semini-2.5-Flash--magge are far from perfect, highlighting the limitations of current unified models in visual generation with reasoning.
Specifically, for numerical reasoning, the highest score just 0.362 with Saaiiii2.5-Flash-mmaa underscoring that models are largely unable to generate correct quantities when reasoning over im- plicit prompts. Among open-source unified models, BAGEL w/ CoT exhibits substantial gain over its base counterpart (0.249 vs. 0.057), suggesting that explicit chain-of offuthu helps transfer arith- metic constraints into the generative process. spatial Layout, proprietary models again dominate, with 2emmini2.5--Flash- Image at 0.759. Within open-source unified models, BAGEL w/ CoT (0.448) outperforms Qwen-Image (0.368) and Ovis-U1 (0.356), indicating that explicit planning improves global arrangement beyond local object placement. For text rendering, under implicit descriptions, the performance gap between proprietary and open-source models is the most pronounced: GPT- Image-1 (0.813) and Gemini-2.5. Flas--aash-lmmnnm Image (0.806) substantially outperform all others. While many models can render text reliably with explicit input text, they struggle to produce correct out- puts when the target text must first be reasoned and then generated. Notably, BAGEL w/Co' does
7

---

Published as a conference paper at ICLR 2026
Table 2: Results of various multimodal models GIR-Bench-T22 and CiR-Bench-Fddin
CIR-Bench-T21
GIR-Bench-Edii
Type
Model
Numerical Spatial Reasoning Layout Rendering
Text
Visual Visual Reasoning Overall Puzzle Logic Perception
Overall
SD-3.5-Large HiDream-Il-Foll
0.107
0.069
0.227
0.134
Gen
0.062
0.230
0.180
0.157
FLUX.I-schnell
0.045
0.195
0.238
0.159
-euXte-Kontett-de
0.045
0.000
0.271
0.105
Edit
ICEdit
0.023
0.030
0.233
0.095
StepIX-Edit
0.037
0.041
0.136
0.071
Show-o2-7B
0.023
0.035
0.010
0.023
Janus-Pro-7b
0.062
0.046
0.006
0.038
BLIP3o-NEXT-SFT-BB
0.040
0.159
UniWorld-V1
0.062
0.207
0.146
0.138
0.038
0.013
0.111
0.054
UmiPic2-Metaquerr--P9
0.107
0.184
0.126
0.139
0.107
0.029
0,261
0.132
Ovis-U1-3B
0.051
0.379
0.082
0.171
0.197
0.033
0.049
0.093
Unified
OmniGen2
0.056
0.207
0.167
0.143
0.030
0.051
0.137
0.073
Qwen-Image
0.153
0.368
0,150
0.224
Qwen-Image-Fie
0,229
0.012
0.234
0.158
BAGEL-7B
0.056
0.287
0.163
0.169
0.131
0.058
0,104
0.098
BAGEL-7B w/ COT
0.249
0,460
0.120
0.276
0.221
0.050
0.149
0.140
femmnn-2.5-Flash-Bee
0.362
0,782
0.806
0.650
0.382
0.249
0.399
0.343
GPT-Image-1
0.328
0.724
0.813
0.622
0.406
0.201
0.445
0,351
Prompt
SPT- Immaee
Semin-22
Bagel w//oT
photo of weather- snimed Texion floy upp loyod rollynn hanored 1836 fartreess The byed next English.
CONE AND TAKE
emien D nrtelda
Remember
A photo lophops, phonos, and chains rumber of laptops
loptop:
chairs than the number phenes Altogether, there ore four meme Despla all the ebjects.
phone:
photo
Caperbagen horbour imsp indd Hons Christion Andersens fory tole.
Figure 6: Illustrative examples from BUR-Bench-UGG and GIR-Bench-T22.
not improve over BAGEL 10. vs. 0.163), suggesting that current reasoning traces are not yet ef- fectively grounded in the generation process, thereby reflecting the broader gap between reasoning and generation. We further provide qualitative analysis in Section 4.
3.4 EVALUATION ON REASONING-CENTRI IMAGE EDITING
Table (right) presents results on GIR-Bench-Edii, which covers three editing tasks: visual puz- zle, visual logic, and reasoning perception. In contrast to the trends observed Section 3.2 and Section 3.3, the performance gaps across model categories noticeably narrower, with all models demonstrating constrained abilities. Even the strongest proprietary models (i.e., Gemini-2.5- 5-aash- Image and GPT- GPT-Image- 1) often fail, highlighting limitations in fine-grained local control and pixel- level information maintenance for image reconstruction and region editing. While explicit reasoning mechanisms occasionally improve results, the overall effectiveness remains limited.
8

---

Published as a conference paper at ICLR 2026
Prompt
Erpat Image
Image-1
Gerrr Emage
Bagel W/CoT
Gast-inage
Groand
pizzie thon is Sudkiu the
4
3
1
2
3
4
S
1 4
3
empty
image
4
3
2
2
1
2
occensing Sudelu munee...
2
2
4
1
3
4
1
9
8
9
3
2
3
2
4
1
3
4
3
6
5
4
2
Hlose dogmen the nagin (ssmenhing modee the coffee more dvect)...
ines
Resmaagg shaffied yelture the crigtal photo (oouter frandres
Figure 7: Representative outputs from GIR-Bench-Eade
<thinko
Prompt: photo of Martin Luther King Jr. addressing the crowd of 1963 March Washington, with his manuscript showing the celebrated four-word refrain. The disployed text in English. Target: have dream
The model should generate en image featuring Martin Luther King delivering his II Have Dream' speech, with the manuscript cleanly visible dream' prominently disployed. Here's the finished detailed prompt: Martin Luther King Jr. stonding at a podium, addressing large, diverse crowd at the 1963 March Washington, with his manuscript hands, monuscript prominently displays celebrated four word refrain, "I have dream," bold English letters....
the
Bagel w/CoT Reasoning
clearly visible
Extended Prompt
from CoT Bagel CoT
Bagel w/o CoT
Bagel w/CoT
Madl
LBvE A CREAM
Figure 8: Misalignment between reasoning and generation.
4 QUALITATIVE ANALYSIS
Qualitative Results. Figure 6 and Figure 7 illustrate representative outputs across image generation and editing. For image generation, current models satisfy only part the coupled constraints and struggle to faithfully generate correct visual content through reasoning. For image editing, visual puzzles yield globally coherent yet unfaithful reconstructions, visual logic suffers from digit errors despite clean grids, and reasoning perception reveals limitations in region preservation and editing.
Misalignment between Reasoning and Generation. Figure presents a case study with Qwen- Image. comparing outputs under category input and prompt input. While the model produces faith- ful generations when the category is explicitly provided, it fails to reason correctly and generate the correct entity with the implicit prompt. To further investigate this misalignment, we analyze BAGEL w/CoT by comparing its reasoning process with the generated images. As shown in Fig- ure 8, BAGEL correctly infers the ground-truth text (i.e. "I Have Dream") within its reasoning process and even expands it into an explicit prompt. However, when asked to generate directly from the original implicit description, fails to render the target text. By extracting the expanded prompt from the reasoning process and re-feeding it into the model, the target text is successfully generated. This observation suggests that reasoning the target text from an implicit prompt is not the primary difficulty, and the real challenge lies in reliably transferring the inferred content into the generation.

---

Published as a conference paper at ICLR 2026
Qwen-Image
World Image
Gemini-2.5. 1.5--aash--eaee -magg
Prompt: photo of the Bahd't Worship belhi shoped resemble blooming white flower. Category: Lotus Temple India
Prompt: A photo pens 6 pencils. Four pens and all pencils were taken Show what remains.
Target: pen x
UMNgos-Judge
LMM--as-Juee Judge
Ours
Ground
Ground Truth
Figure 9: Comparison with MLLM-as -asd-------Judee.
Qualitative Comparison with MI.LM-as-a-Judge. Figure 9 contrasts MLLMM-as-a-Jude with our explicit, task-specific metrics. MLLM-as-a-Judge sometimes assigns high scores based on superfi- cial resemblance or biased reasoning, whereas our metrics correctly penalize mismatches with the ground truth. For example, the Lotus Temple case, judge-based scoring mislabels the Qwen- Image output as correct despite clear structural inaccuracies. In the pens-and-pencie case, it fails reward Gemini-2. 2.5-aii-2a--a Flash-Image for producing the exact target. These cases highlight that our metrics yield more consistent and interpretable evaluations.
Misalignment Understanding- Generaaiinnnnnn Based on our quantitative ablations and qualitative case studies presented, we identify three primary factors driving the misalignment between the un- derstanding and generation capabilities of unified models:
Asymmetry of Reasoning Capabilities: Our experiments reveal fundamental asymmetry: models often possess the knowledge but fail to "activate" it during the generation process. As illustrated in our case studies and experiments (in Figure 4 and Figure 5), the model success- fully generates the correct entity when explicitly named (e.g., "Charging Bull United States") but fails when the same entity must be reasoned from an implicit prompt. This suggests that the rea- soning capability is localized within the LLM component. While the model correctly solves the intermediate logic, the image generator- which is optimized for visual fidelity rather than logical adherence- fails to receive or adhere to this reasoned state, leading to generation failures.
Information Bottleneck in Heterogeneous Architectures: Many unified models evaluated in MRR-Bench employ heterogeneous architectures that couple a capable LLM with a separate gen- eration head via lightweight interface. This design creates significant information bottleneck. While the LLM performs deep reasoning, the interface often compresses this rich semantic state into limited number of condition tokens. Consequently, fine-grained logical constraints are often diluted or lost before reaching the pixel generation stage.
Lack of Interleaved Data: Current multimodal pre-training relies heavily on static image-text pairs These datasets map final text description directly to an image but fail to capture the process of generation. Existing models lack exposure to interleaved "reasoning-trace data (e.g., [Reason iooningg ing Step] - [Inte e entee i Vi susll State] - [Re i nned Image]) ])- Without such data, the model struggles to learn how to decompose complex abstract instruction into sequential plan for visual execution.
5 CONCLUSION
In this work, we presented GIR- BiR-Bench, a reasoning-centir benchmark that systematically evaluates unified multimodal models across understanding, generation, and editing. By grounding evaluation in explicit, task-specific metrics rather than the MILLM-as-Judee paradigm, GIR-Bench exposes fine-grained weaknesses that are otherwise hidden behind holistic scores. Extensive experiments and analyses show that while unified models consistently surpass generation-only models reasoning- intensive tasks, they still struggle with reliably transferring reasoning into faithful visual outputs. These findings point to the importance of advancing unified models capable of seamlessly integrat- ing reasoning and generation.
ACKNOWLEDGEMENT
This work was supported by the Hong Kong SAR RGC General Research Fund (16219025), Na- tional Natural Science Foundation of China Young Scholar Fund Category B (62522216), National Natural Science Foundation of China Young Scholar Fund Category (62402408), and Hong Kong SAR RGC Early Career Scheme (26208924).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 139.223330....223, 2025.
Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dock- horn, Jack English, Zion English, Patrick Esser, Sumith Kulal, et al. Flux. kontext: Flow match- ing for in-context image generation and editing in latent space. arXiv e-prints, pp. arXiv-2506, 2025.
Qi Cai, Jingwen Chen, Yang Chen, Yehao Li, Fuchen Long, Yingwei Pan, Zhaofan Qiu, Yiheng Zhang, Fengbin Gao, Peihan Xu, et al. Hidream-il: A high-efficient image generative foundation model with sparse diffusion transformer. arXiv preprint arXiv:2505.22200 2025.
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi
Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecttre,, training and dataset. arXiv preprint arXiv:2505.09500 2025a.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, arXiv preprint arXiv:2501. 17855001171.,.1 2025b.
G. Comanici et al. Gemini 2.5: Pushing the frontier with advanced reasoning. arXiv preprint, 2025. URL https: : /arxivv.vvv/../////a org/ abs/ 06261.
Cheng Cui, Ting Sun, Manhui Lin, Tingquan Gao, Yubo Zhang, Jiaxuan Liu, Xueqing Wang, Zelun Zhang, Changda Zhou, Hongen Liu, Yue Zhang, Wenyu Lv, Kui Huang, Yichao Zhang, Jing Zhang, Jun Zhang, Yi Liu, Dianhai Yu, and Yanjun Ma. Paddleocr 3.0 technical report, 2025. URL wwtph:tt/////..t. /ttp///axvvvv/////////.. org/ /btpa/bb////255555 05595.
Google DeepMind, 2025.
URL https://developeee www.:peeee....ggooo...o...ooooo. com/en/ introducing gemini 2 wwwsh-laassgseeee
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Wei- hao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. /4683, 2025.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 52132- 52152, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment, 2024.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723-787777, 2023.
Black Forest Labs. Flux. https: https:///iithine a fore-oforrtttteee -aaesstlaab//ee labs/flux, 2024.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9579-9589, June 2024.
Ouxiang Li, Yuan Wang, Xinting Hu, Huijuan Huang. Rui Chen, Jiarong Ou, Xin Tao, Pengfei Wan, Xiaojuan Qi, and Fuli Feng. Easier painting than thinking: Can text-to-image models set the stage, but not direct the play? arXiv preprint 1rXiv:2509.035/6, 2025.
11

---

Published as a conference paper at ICLR 2026
Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, et al. Uniworld: High-resolution semantic encoders for unified visual understanding and generation. arXiv preprint arXiv:2506.03147. 2025.
Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, Guopeng Li. Yuang Peng. Quan Sun, Jingwei Wu, Yan Cai, Zheng Ge, Ranchen Ming, Lei Xia, Xianfang Zeng, Yibo Zhu, Binxing Jiao, Xiangyu Zhang, Gang Yu, and Daxin Jiang. Steplx-edin x-eit: practical framework for general image editing. arXiv preprint arXiv:2504. 1at.:22044..771,, 2025.
Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai yu, Liang Zhao, Yisong Wang, Jiaying Liu, and Chong Ruan. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal under- standing and generation, 2024.
Yuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Kunpeng Ning, Chaoran Feng. Bin Zhu, and Li Yuan. Wise: A world knowledge-infone inffommeed semantic eval- uation for text-to-image generation. arXiv preprint arXiv:2503.07266 2025.
OpenAl. Gpt-5 system card. Technical report, OpenAI, 2025a. URL com/gpt-55t 5- 1e------ystt-mmmeem car.d
openai...
OpenAl, 2025b. URL httttt/s///ee /tppttpp:///ppeiiiiie wwtp//oow////////.o//oot.ct.wttt http:////ooo////////aooooaoaaooo
Xichen Pan, Satya Narayan Shukla. Aashu Singh. Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu. Ji Hou, and Saining Xie. Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.0062 06256, 2025.
Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang. Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. DINOv3, 2025. URL https: Stcttts:/aaxiv..o/ abs/2508. 10104.
AI 2024.
Kaiyue Sun, Rongyao Fang, Chengqi Duan, Xian Liu, and Xihui Liu. T2i-reasonbench: Bench- marking reasoning-informed informed text-to-image generation. arXiv preprint arXiv:2508.17477 2025.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.
arXiv preprint
0988:8, 2024.
Guo-Hua Wang, Shanshan Zhao, Xinjie Zhang, Liangfu Cao, Pengxin Zhan, Lunhao Duan, Shiyin Lu, Minghao Fu, Jianshan Zhao, Yang Li, and Qing-Guo Chen. Ovis-ul technical report. arXiv preprint arXiv:2506.23000 2025a.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internv13.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508. 18265, 2025b.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409, 18869, 2024.
Hongyang Wei, Baixin Xu, Hongbo Liu, Cyrus Wu, Jie Liu, Yi Peng, Peiyu Wang, Zexiang Liu, Jingwen He, Yidan Xietian, Chuanxin Tang, Zidong Wang, Yichen Wei, Liang Hu, Boyi Jiang, William Li, Ying He, Yang Liu, Xuchen Song, Eric Li. and Yahui Zhou. Skywork unipic 2.0: Building kontext model with online rl for unified multimodal model. 2025. URL https:// arxiv.c oerivioor////oo
orgg abs 9.09... 04548.
12

---

Published as a conference paper at ICLR 2026
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng. Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng. Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun Wen, Wensen Feng, Xiaoxiao Xu, Yi Wang, Yichang Zhang. Yonggiang Zhu, Yujia Wu, Yuxuan Cai, and Zenan Liu. Qwen-image technical report, 2025a. URL https: /ttppps///axii...// /ttp//xxivvv..///.//.aa org. /bs/ 2508. 02324.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Viyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 13848, 2024a.
Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang. Wanli Li, Xiyan Jiang. Yexin Liu. Junjie Zhou, Ze Liu, Ziyi Xia, Chaofan Li, Haoge Deng. Jiahao Wang, Kun Luo, Bo Zhang. Defu Lian, Xinlong Wang, Zhongyuan Wang, Tiejun Huang, and Zheng Liu. Omnigen2: Exploration to advanced multimodal generation. arXiv preprint arXiv:2506. 18871, 2025b.
Junfeng Wu, Yi Jiang, Chuofan Ma. Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable multi-modal generators. arXiv preprint arXiv:2412.04332 2024b.
Xie, Trevor Darrell, Luke Zettlemoyer, and XuDong Wang. Reconstruction alignment improves unified multimodal models. arXiv preprint arXiv:2509.07295 2025a.
Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models. arXiv preprint arXiv:2506. 15564, 2025b.
Yang Ye, Xianyi He, Zongjian Li, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, and Li Yuan. Imgedit: A unified image editing dataset and benchmark. arXiv preprint arXiv:2505.202: 2025.
Zechuan Zhang, Ji Xie, Yu Lu, Zongxin Yang, and Yi Yang. In-context edit: Enabling instructional image editing with in-context generation in large scale diffusion transformer. arXiv preprint arXiv:2504.20690 2025.
Xiangyu Zhao, Peiyuan Zhang, Kexian Tang, Hao Li, Zicheng Zhang, Guangtao Zhai Junchi Yan, Hua Yang. Xue Yang, and Haodong Duan. Envisioning beyond the pixels: Benchmark- ing reasoning-informed visual editing. arXiv preprint arXiv:2504.02826, 2025.
Xianwei Zhuang, Yuxin Xie, Yufan Deng, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou. Vargpt: Unified understanding and generation in visual autoregressive multimodal large language model, 2025, URL https: http///axvvve/.//////eee/eaae org abs 2001.1111110
ICLR2026
A ETHICS STATEMENT
This work introduces GIR-Bench. a benchmark designed to evaluate unified multimodal models on reasoning-drived image generation and editing tasks. Our study does not involve human subjects, personally identifiable information, or sensitive private data. All images used the benchmark either collected from publicly available open-source datasets generated through automated pipelines, and we have carefully filtered the data to avoid inappropriate or harmful content.
We acknowledge that generative models, when misused, could produce misleading or harmful out- is constructed solely for research purposes, with a focus
puts. To mitigate such risks, on advancing transparent and interpretable evaluation of multimodal reasoning. We explicitly avoid releasing prompts or data that could be exploited for malicious generation, and we emphasize that benchmark should not be deployed in downstream applications without appropriate safeguards.
13

---

Published as a conference paper at ICLR 2026
Pronpt
GPT-Imege-1
Rogel w/GoT
Quean-inage
Target
photo of the lotus- bud shaped
telecammurico
lower rising
tallest
photo the Mgeer floodploi considered the worlds largest earthen buildig.
A striking bloom with a weid pink oncerenze shezed fiottened Centrol cal noiforests phyte. Knean
A striking frower with Feathery, plime-like infersscencess ective tropical subtropical regions Historically gardens for ornamentel appeal...
muscular harding with lang, and distinctive double
dewclove ite hind legs.
Historically ly used Fronce
guard move flocks. fovourite
Nepoleon
A sturdly dog with lion-
like mone and doep set eyent known for anownn
origine for
fovourtte
Chinese emperors....
Figure 10: Examples from GIR-Bench-UGG
B RELATED WORK
B. UNIFIED MULTIMODAL MODELS
Research in multimodal generation has shifted from modality-specific architectures to unified mod- els that handle both understanding and generation across modalities (Lin et al., 2025; Wu et al., 2024b;a; Ma et al., 2024; Wei et al., 2025; Wu et al., 2025b; Chen et al., 2025b; Wang et al., 2025a; Zhuang et al., 2025; Xie et al., 2025a). Chameleon (Team, 2024) pioneered an early-fusion, token- based transformer that generate and interpret text and images interchangeably. matching or surpassing proprietary systems. Emu3 (Wang et al., 2024) shows that pure autoregressive model- ing can achieve general multimodal intelligence without relying diffusion. MetaQueries (Pan et al., 2025) introduces learnable query tokens to interface frozen multimodal LLM with a dif- fusion decoder, enabling knowledge-augmentedd image generation without degrading understanding performance. Some works attempt to integrate diffusion into unified frameworks, BLIP3-o (Chen al., 2025a) uses a diffusion transformer to generate semantically rich CLIP image features and employs sequential pretraining (understanding followed by generation). BAGEL (Deng et al., 2025) a decoder-only foundation model pretrained on trillions of interleaved multimodal tokens, exhibit- ing emergent capabilities in complex multimodal reasoning such free-form image manipulation. Proprietary systems e.g., GPT-Image-1 (OpenAI, 2025b) and Gemini 2.5 Flash Image (DeepMind, 2025), further showcase the powerful capabilities of unified multimodal models and have given rise to a wide range of real-world applications. These advancements underscore the need for compre- hensive and reasoring-centric benchmarks.
14

---

Published as a conference paper at ICLR 2026
Frompt
GPT-Image-l
Genri-2 5ia-------iaa-nnggee
Bogel w/CoT
Quen-image
Target
photo of heys, lampo, and blankets. nuncer kays sounl taice sumber lampa, equal number of lamos Altegetherr there are five items, Diáploy the desects.
key:
lomp: 1
blankett 2
phote of pictes, and bottles. The namber of cups is tne more than slares. the number of
plate: bottle:
number
totdl there ore four itens.
Show ofl the objects.
photo morble tablet The Copitoline Mussums bearing Consert suconett thyes word report of victory Latin disployed Engiish
VENI VIDI VICI
CARS CITIEN
YARCHTOTT
come, now. conquared
ECAN
R DEIIITTY ARM EXTIN
photo Beriomin Fronklins Poor Richard Almonack opened directly profin, the sentence underlined
Hichard
money
The displyed hext is Englih.
Figure 11: Examples from GIR-Bench-
B.2 BENCHMARKS FOR UNIFIED MULTIMODAL MODELS
Existing benchmarks (Huang al., 2023: Niu al., 2025; Zhao et al., 2025: Li et al., 2025) for unified multimodal models cover various aspects but still fall short of probing reasoning. GenEval (Ghosh al., 2023) uses object detection check whether generated images match explicit constraints on object co-occurrence, position, count and colour, limiting its scope to compositional alignment. ImgEdit (Ye et al., 2025) introduces large-scale, high-quality dataset of 1.2 million edit pairs covering both single-turn and multi-turn image editing tasks, and proposes ImgEdit-Bent a benchmark evaluating instruction adherence, editing quality and detail preservation; however, its focus remains surface-level editing fidelity rather than deeper reasoning. WISE (Niu et al., 2025) provides 1,000 prompts spanning cultural commonsense, spatio-temporal reasoning and natural sci- ence, and measures knowledge-ingg -imaee alignment with its WiScore metric, but its evaluation depends on the inherent knowledge of multimodal I.LMs, making it prone to bias. "e2l-ReasoBBench (Sun al., 2025) introduces 800 prompts across idiom interpretation, textual design, entity and scientific reasoning, but its two-stage evaluation ILLM-generated questions and LMM-based scoring) cannot disentangle understanding from generation. RISEBench (Zhao et al., 2025) extends evaluation to visual editing tasks involving temporal, causal, spatial and logical reasoning and scores instruction adherence, appearance consistency and visual plausibility using human and multimodal judges, yet it again depends on large models as judges rather than explicit, interpretable metrics. Current bench- marks emphasize explicit alignment and surface-level fidelity, lean heavily on multimodal models as judges, and lack framework to decouple comprehension from generation.
C IMAGE EDITING INSTRUCTIONS
To ensure consistency and clarity in evaluating editing capabilities, design explicit instructions for the three tasks in EIR-Bench-Edii. Each instruction specifies the editing objective, the con- straints to be preserved, and the expected output format. Placeholders such (caption) and {category) are replaced with task-specific descriptions or categories for each case.
Reasoning Perception:
Please segment the region from the image described (capt ion). Out- put the image with a solid, fully opaque green (#00FF00) segmentation mask overlay, and leave the rest of the image unchanged.
15

---

Published as a conference paper at ICLR 2026
Froept
Irput mmag
Goman 2. Plach Umage
Begel witdi
Geen Eai
Ground Teath
Soke the phes bistkk Fill emply calls with cornect rembere eccording rales....
puatie shown the image with the correct nembers
aepment the rag ot (DT one the greand (Wich pag etil be ailee)
from
from the imoge daery bad fochurer of the ranfencn envrsmment develipeeet doerss then....
Rooriage the cheffted restare enginal phanto
Reatrange the shaffted ramtar the onginal phono
Figure 12: Examples from LIR--Bench-Edit.
Visual Logic: Solve the given Sudoku puzzle shown in the image. Fill in all the empty cells with the correct numbers according to Sudoku rules. Keep the existing numbers and the grid lines unchanged. Write the solution numbers in consistent font and size, aligned within each cell, that the final image shows the completed Sudoku solution.
Visual Puzzle: Rearrange the shuffled jigsaw puzzle image restore the original photo of {caption].
D MORE QUALITATIVE RESULTS
In this section, we provide representative qualitative results from the three components of GIR- Bench. Figure 10 shows examples from EIR-BBnch-UUGC Figure 11 from GIR-Bench- and Figure 12 from GIR-Bor HiRR-Bench-Edig These cases illustrate typical challenges faced by current models and highlight the aessoning-generaate misalignment observed across different tasks.
E ROBUSTNESS AND VALIDATION OF AUTOMATED EVALUATION METRICS
To ensure the reliability of GIR-Bench, we conducted a comprehensive analysis of the robustness of our automated evaluation metrics compared to human judgment and MLLM-based judges. As illustrated Figure 13, our automated metrics demonstrate a high Pearson correlation of p Rs 0.96 approaches exhibit lower consis-
with human annotation globally. In contrast, MLLM-as-a-JMe tency and a notable tendency towards score inflation, particularly in lower performare regimes (visible points floating significantly above the diagonal in the lower-left quadrant). This quanti- tative evidence supports our choice of using deterministic, verifiable metrics over stochastic MLLM judgments.
16

---

Published as a conference paper at ICLR 2026
Ours: pos::p=0.964
Gemini p=0.795
Qwen3VL-328: p=0.720
Metric Score
0.6
X
10ck
100
X
0.4
Ours
Gemini 2.5
Qwen3VL-328
0.0
0.2
0.6 Human Annotation
0.4
0.8
1.0
Figure 13: Correlation Analysis of Evaluation Protocols. We aggregated evaluation scores across all tasks in GIR-Bench to compare our metrics against MLLL--as-a-Judee methods. Our taec-ipe metrics (Green) show strong linear correlation (p 0.96) with human judgment, demonstrating high robustness. contrast, MLLM judges (Red/Blue) exhibit higher variance and inflation bias.
We specifically investigated edge cases where automated tools might misjudge correct outputs. Our observations indicate:
Mismatches infrequent and typically occur only when objects rendered in highly abstract, stylized forms, are heavily occluded, or when text is severely distorted beyond standard legibility. In such cases, human annotators often also find the content ambiguous. Ranking Stability: Crucially, these edge cases are uniformly distributed and do not dispropor- tionately affect specific models. Consequently, the relative ranking models remains consistent. The significant performance gap is robust to these minor metric fluctuations.
Rare False Negatives:
F EXTENDED DISCUSSION: THE GAP BETWEEN UNDERSTANDING AND GENERATION
In this section, we provide in-depth analysis of the performance gap observed between the un- derstanding and generation capabilities of unified multimodal models. Based on our quantitative ablations in Figure 4) and qualitative studies in Figure 5), identify the root causes of this misalignment and propose potential research directions to bridge this gap.
F.1 ROOT CAUSES OF THE PERFORMANCE GAP
We identify three primary factors driving the misalignment where models fail to transfer their rea- soning capabilities into faithful visual generation:
Asymmetry of Reasoning Capabilities: Our experiments reveal a fundamental asym- metry: models often possess the requisite knowledge but fail to "activate" it during the generation process. As illustrated in our case study on the "Charging Bull" (as shown in Figure 5), the model successfully generates the correct entity when explicitly named (e.g.,
17

---

Published as a conference paper at ICLR 2026
"Charging Bull United States") but fails when the same entity must be reasoned from implicit prompt. This suggests that the reasoning capability is primarily localized within the Large Language Model (LLM) component. While the model correctly solves the in- termediate logic (e.g., identifying the entity), the image cenerator-wiie is optimized for visual fidelity rather than logical adherence-fait to receive adhere this reasoned state, leading to generation failures.
Information Bottleneck in Heterogeneous Architectures: Many unified models evalu- ated in GIR-Bench (e.g.. BAGEL (Deng et al., 2025), Qwen-Image (Wu et al., 2025a)) employ heterogeneous architectures that couple highly capable autoregressive LLM with a separate generation head via a lightweight interface. This design creates a significant information bottleneck. While the LLM performs deep reasoning. the interface often com- presses this rich semantic state into a limited number condition tokens or a concise embedding. Consequently, fine-grained logical constraints (e.g., exact counts in Numerical Reasoning tasks) are often diluted or lost before reaching the pixel generation stage.
Lack of Process-Oriented Training Data: Current multimodal pre-training relies heavily on static image-text pairs. These datasets map final text description directly an image but fail to capture the process of generation. Existing models lack exposure to interleaved greasoning-trace data (e.g., [Reasoning Step) > [Ineermediate Visual State] - [Ref ined Imagel). Without such data, the model struggles to learn how to decompose complex abstract instruction into a sequential execution plan for visual generation.
F.2 FUTURE DIRECTIONS
To narrow the gap between understanding and generation, we propose three concrete research direc- tions supported by our empirical findings:
Explicit Chain-of-Thoughh (CoT): Our experimental results with BAGEL CoT (Ta- ble 2) demonstrate that externalizing the reasoning process significantly boosts perfor- mance. For instance, accuracy on the Numerical Reasoning task improves from 0.057 0.249. This confirms that forcing the model output explicit textual plan acts as a crucial bridge, converting implicit logic into explicit constraints that the generator can better attend to and execute.
Native Unified Architectures: We argue that future work should towards natively unified paradigms where text and image tokens are treated equally within single back- bone. This would effectively climinate the heterogencous interface bottleneck, allowing the visual generation process to attend directly to the full, uncompressed reasoning states of the MILM.
We advocate for the construction reasoning-trace datasets. Future benchmarks and training sets should provide not only the final ground-truth image but also the logical execution trace. This would enable models to be supervised directly on the reasoning-te togeerrationn mapping, rather than relying solely on end-to-end alignment.
Dataset Evolution:
G CLARIFICATION FOR THE TEXT RENDERING METRIC (Swe)
We illustrate the robustness of our proposed metric ($we) compared to standard accuracy (Sace) using the Ground Truth: "Make It Happen"
Table 3: Comparison of metric behaviors under different generation scenarios.
Ground Truth:
"Make It Happen"'
Case
Generated Text (OCR Output)
Swc (Ours)
Saoe
(Perfect)
"Make It Happen"
1.00
1.00
B (Extra Words)
"Poster says xxxxx Make It Happen"
1.00
0.00
C (Merged)
"MakeItHappen"
1.00
0.86
(Missing)
"Make It"
0.67
0.50
18

---

Published as a conference paper at ICLR 2026
The cases in Table 3 demonstrate the specific advantages of Swei
Case B (Robustness to Context): Unified models often generate descriptive text or arti- facts (e.g., "Poster says xxxxx") alongside the target. This effectively handles the "chatty" nature of these models. Sace fails due to the length penalty imposed by the extra characters, whereas Sie correctly credits the presence of the valid content.
Case C (Robustness to Layout): Tight visual layouts artistic fonts often lead to merged OCR outputs. Swe successfully identifies the legible words within the merged string, while Saec penalizes the missing whitespace.
Case D (Penalty for Missing): Both metrics correctly penalize missing content. It is important to note that &we acts as a striet filter: partial matches (e.g. "Make It Hap") are not credited, ensuring that the metric does not reward incomplete generations.
19


Reviews:
[{"review_id": "2F02cvSK9q", "reviewer": "Reviewer_9Tai", "summary": "The paper introduces GIR-Bench, a novel benchmark designed to evaluate the reasoning capabilities of unified multimodal models in image generation and editing tasks. It provides a suite of challenging tasks that require models to combine visual understanding with logical reasoning, spanning three components: aligning understanding vs generation for the same concept, reasoning-centric text-to-image generation, and reasoning-driven image editing. The authors also develop specialized evaluation metrics for each task to obtain fine-grained, interpretable assessments instead of relying on subjective large-model judgments. Experiments on 21 state-of-the-art models demonstrate the benchmark’s effectiveness, revealing a significant gap between what current models can understand and what they can faithfully generate under complex reasoning constraints.", "questions": "1. Could the authors clarify how the ground-truth outputs for the complex editing tasks (e.g., solving a Sudoku puzzle or rearranging a jigsaw image) were obtained? Understanding whether these reference outputs were generated by an algorithm, manually created, or produced by another model would help in assessing the objectivity and fairness of the evaluation for those tasks.\n2. How robust are the automated evaluation metrics (such as object detection for counting and OCR for text) to variations in the model outputs? It would be useful to know if the authors observed any cases where these tools misjudged a correct output, and what steps were taken to mitigate potential evaluation errors in such situations.\n3. Why were the particular reasoning tasks and domains chosen for inclusion in GIR-Bench, and do the authors believe these cover the most critical reasoning skills needed for image generation? A discussion of the task selection rationale — and whether there are plans to extend the benchmark with other reasoning scenarios (e.g., causal reasoning or commonsense reasoning in images) — would help clarify the benchmark’s comprehensiveness and how it might evolve.\n4. The results indicate a significant gap between models’ understanding capabilities and their generation performance under reasoning constraints; have the authors investigated the root causes of this gap or potential ways to reduce it? Including more analysis or hypotheses (for example, whether the limitation stems from model architecture, training data, or reasoning strategy) and suggesting directions for improving reasoning in image generation would strengthen the paper’s conclusions and usefulness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper fills an important void by focusing on reasoning-driven image generation, an aspect where prior image generation benchmarks were limited or shallow. The benchmark is thoroughly validated by testing it on 21 state-of-the-art models, including both unified multimodal models and traditional image generation models.\n2. A notable strength is the introduction of task-specific evaluation pipelines, replacing the common practice of using a multimodal LLM as a subjective judge.", "weaknesses": "1. Some tasks in the benchmark, though creative, are somewhat niche or artificial (e.g., asking an image generation model to solve a Sudoku puzzle or reassemble a jigsaw). These scenarios are not typical use cases for image generation models; thus, poor performance might reflect the models’ lack of exposure to such tasks rather than a general reasoning failure, raising questions about the real-world relevance of these evaluations.\n2. The paper could offer more in-depth analysis of the observed performance gaps and the reasoning task scope. It identifies a gap between understanding and generation, but does not deeply explore why certain models fail (e.g., is it due to knowledge retrieval limits, reasoning logic, or image-generation fidelity?) and stops short of suggesting how future models might overcome these shortcomings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7u3y1qdtfL", "reviewer": "Reviewer_R2PX", "summary": "This paper introduces GIR-Bench, a novel and comprehensive benchmark designed to systematically evaluate the reasoning-driven image generation and editing capabilities of unified multimodal models. The authors argue that existing benchmarks fall short in rigorously assessing the alignment between reasoning and generation, often relying on biased MLLM-as-a-Judge paradigms. GIR-Bench addresses this by evaluating models across three complementary perspectives:\n1. GIR-Bench-Uni (Knowledge-to-Generation): Assesses whether models can consistently leverage the same knowledge for both understanding and generating real-world entities from implicit descriptions.\n2. GIR-Bench-T2I (Reasoning-to-Generation): Investigates reasoning-centric text-to-image generation, focusing on numerical reasoning, spatial layout, and text rendering tasks that require logical constraints and implicit knowledge.\n3. GIR-Bench-Edit (Reasoning-to-Editing): Evaluates multi-step reasoning in image editing through visual puzzles, visual logic (Sudoku), and reasoning perception (segmentation based on implicit descriptions).\n\nA key methodological contribution is the design of task-specific evaluation pipelines for each subset, moving beyond the MLLM-as-a-Judge paradigm to provide fine-grained and interpretable assessments. The benchmark was used to evaluate 21 state-of-the-art models, revealing a persistent gap between understanding and generation capabilities in unified models, despite their overall superiority to generation-only systems in reasoning-driven visual tasks. The data and code are made publicly available.", "questions": "1. Could the authors elaborate further on their hypotheses regarding the underlying causes of the reasoning-generation misalignment? For instance, do they believe it's more of a representational issue (how reasoning is encoded), an architectural bottleneck (how reasoning modules interact with generation modules), or a training data challenge (lack of sufficiently diverse and complex reasoning-generation pairs)?\n2. Given that the evaluation relies on external object detection and text recognition models, did the authors perform any analysis or sanity checks on the performance of these tools when applied to the generated images, which might differ significantly from their typical training data? Are there cases where the evaluators themselves might misinterpret generated content, leading to skewed scores?\n3. For the text rendering task, the \"word-level continuous substring score\" is a custom metric. Could a few more illustrative examples be provided, perhaps in the appendix, to clearly demonstrate its calculation, especially for edge cases where the generated text might have extra words, missing words, or partial matches?\n4. While Table 1 and 2 present results, a more explicit \"Ablation Study\" section could be beneficial. For example, the comparison of BAGEL with and without CoT is a good start. Are there other controlled experiments that could shed light on specific design choices or components of unified models?\n5. Are there plans to extend GIR-Bench to include more complex forms of reasoning (e.g., counterfactual, moral, scientific hypothesis generation) or to incorporate other modalities beyond visual generation (e.g., video, 3D)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies and rigorously tackles the crucial problem of evaluating the alignment between reasoning and generation in unified multimodal models, a gap not adequately addressed by previous benchmarks. GIR-Bench covers a broad spectrum of reasoning capabilities, from knowledge recall and simple arithmetic to complex spatial arrangement, text rendering from implicit cues, and multi-step image editing.\n2. The shift from subjective MLLM-as-a-Judge evaluations to concrete, task-specific metrics (e.g., object detection for counts/spatial layout, OCR for text, FID for puzzles, IoU for segmentation) is a major strength, leading to more reliable and interpretable results. The evaluation of 21 diverse state-of-the-art models provides a comprehensive overview of the current landscape and validates the benchmark's effectiveness in revealing model limitations.\n3. The paper clearly demonstrates the \"understanding-generation gap\" and the difficulty models face in translating inferred knowledge/reasoning into faithful visual outputs, even when the reasoning process itself is correct (as shown with BAGEL w/ CoT analysis).", "weaknesses": "1. While the paper effectively identifies the reasoning-generation gap, it offers less in terms of a deeper mechanistic analysis of why this gap persists. Is it primarily an architectural limitation, a data scaling issue, a fundamental challenge in aligning discrete symbolic reasoning with continuous pixel generation, or an optimization problem? A more in-depth discussion or hypothesis generation regarding the root causes could provide even more actionable insights for model developers.\n2. The evaluation pipelines rely on external object detection (InternVL3.5-38B) and text recognition (PPOCR v5) models. While these are strong models, their performance and potential failure modes on diverse, potentially synthetic or \"hallucinated\" images generated by the tested models are not thoroughly discussed. The robustness of these evaluators to the specific characteristics of generated content could impact the final scores.\n3. The top-performing models (GPT-Image-1, Gemini-2.5-Flash-Image) are proprietary. While this is an accurate reflection of the current SOTA, it means the most successful approaches are not open for inspection or direct improvement by the research community, which limits the actionable insights for open-source development.\n4. While comprehensive within its defined scope, the benchmark primarily focuses on specific types of reasoning (numerical, spatial, textual, knowledge recall). Other forms of reasoning, such as causal reasoning, temporal reasoning, or abstract concept manipulation, are not explicitly covered. This is more of a future expansion opportunity than a weakness of the current work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oZvubmsvbl", "reviewer": "Reviewer_RiLt", "summary": "GIR-Bench is a reasoning-centric benchmark for unified multimodal models. It tests (1) whether models use the same knowledge consistently across understanding vs. generation, (2) whether they can do reasoning-heavy text-to-image generation, and (3) whether they can edit images via multi-step reasoning. Findings: unified models beat pure generators on reasoning tasks, but there’s a persistent gap between reasoning/understanding and actually rendering the correct visual outputs, especially under implicit prompts.", "questions": "1. The new swc metric allows extra content without penalty, which makes sense for implicit prompts. However, how does it handle variations like typos, capitalization differences, or partial matches (e.g., \"Just do\" instead of \"Just do it\")? Have the authors tested its robustness against human annotations?\n2. The authors argue that the proposed pipelines avoid biases of the MLLM-as-a-Judge paradigm, but have the authors compared metrics (e.g., swc for text rendering, IoU for reasoning perception) against MLLM-based scores on a subsample? If so, what was the agreement level, and in cases of discrepancy, which better aligned with human preferences?\n3. Are the selected domains (zoology, botany, geography) representative enough to evaluate general reasoning transfer across modalities? The chosen domains seem to be biased towards factual knowledge about nature instead of reasoning chains that apply across general contexts.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* It introduces three innovative and well-differentiated tracks whose tasks exhibit minimal overlap in the model capabilities they test, ensuring a comprehensive evaluation of reasoning, generation, and editing skills. \n* All selected tasks are designed to be assessed through rule-based evaluation, and the entire framework follows interpretable, deterministic criteria, which greatly enhance the reliability and reproducibility\nof the results.\n* The benchmark covers a wide spectrum of models — both open-source and closed-source — encompassing image generation, image editing, and unified vision-language models, thereby providing a broad and fair comparison across model types.", "weaknesses": "* Each subset contains only about 300 entities, which is limited to yield statistically solid or generalizable results. While the authors claim that “overall, unified models outperform generation-only systems on reasoning-centric generation tasks, indicating that joint training across understanding and generation yields tangible benefits,” the paper does not provide detailed data analysis to support this conclusion. \n* The experimental results are relatively sparse, and the accompanying analyses lack the depth and precision needed to validate the authors' claims firmly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CbaXWYaPDU", "reviewer": "Reviewer_6bCn", "summary": "The paper introduces GIR-Bench, a comprehensive benchmark designed to evaluate reasoning-driven image generation and editing in unified multimodal models—systems that combine visual understanding, reasoning, and generation in a single architecture.\n\nInstead of relying on the subjective and biased “MLLM-as-a-Judge” paradigm, the authors design objective, fine-grained metrics (e.g., object counting, IoU, FID, word-level substring matching) tailored to each task.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper demonstrates high originality not by inventing a new model, but by redefining how we evaluate unified multimodal systems. It identifies a critical yet overlooked gap: the misalignment between reasoning (understanding) and visual generation/editing. While prior benchmarks focus on surface-level text-image alignment or isolated reasoning tasks, GIR-Bench is the first to systematically probe reasoning-to-generation fidelity across three carefully designed axes—knowledge consistency, constrained generation, and reasoning-driven editing. The formulation of tasks like implicit-entity generation, logic-constrained layout, and puzzle-based editing represents a creative synthesis of symbolic reasoning, visual semantics, and generative modeling.", "weaknesses": "GIR-Bench focuses on deductive and constraint-based reasoning (e.g., arithmetic, spatial ordering, Sudoku), but omits other critical reasoning modalities such as causal, temporal, counterfactual, or commonsense physical reasoning. For instance, generating an image of “a glass falling off a table” requires modeling gravity and motion—reasoning not captured by current tasks.\n\nThe word-level continuous substring score (swc) allows extra text but does not penalize hallucinated or semantically incorrect text that happens to contain the target substring (e.g., generating “Just do it now!” for “Just do it” is acceptable, but “Just do it wrong” would also score fully). More critically, the metric ignores position, font, size, or contextual integration—all crucial for faithful rendering.\n\nGIR-Bench-Edit uses synthetic or highly constrained tasks (jigsaw, Sudoku, green-mask segmentation). These do not reflect practical editing needs like object insertion/removal, attribute modification (“make the car red”), or style transfer guided by reasoning (“render this building in Art Deco style based on its era”).\n\nWhile the paper notes the “reasoning-generation gap,” it does not systematically analyze where the disconnect occurs: Is it in the multimodal alignment layer? The diffusion/autoregressive decoder? The instruction-following interface? Without architectural or ablation insights, the benchmark remains diagnostic but not prescriptive.\n\nAll metrics are automated. While this avoids MLLM-as-a-Judge bias, it misses perceptual or contextual fidelity that humans would notice (e.g., a correctly counted but anatomically distorted animal).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)