A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation
Authors: Yukang Feng
Jianwen Sun
Chuanhao Li
Zizhen Li
Jiaxin Ai
Fanrui Zhang
Sizhuo Zhou
Yifan Chang
Shenglin Zhang
Yu Dai
Kaipeng Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: high-quality dataset
multimodal dataset
interleaved image-text synergy
interleaved evaluation
Primary area: datasets and benchmarks

Abstract: Recent advancements in Large Multimodal Models (LMMs) have significantly improved multimodal understanding and generation.
However, these models still struggle to generate tightly interleaved image-text outputs, primarily due to the limited scale, quality and instructional richness of current training datasets.
To address this, we introduce \textbf{InterSyn}, a dataset that features:
(1) large scale, comprising 1.8M multimodal samples;
(2) high quality, supported by our proposed \textbf{Self-Evaluation with Iterative Refinement (SEIR)} method for rigorous automated quality refinement;
(3) rich instructional diversity, ensured through diverse well-designed question templates, based on human preferences and covering a 3500-topic hierarchy.
These characteristics make InterSyn particularly well-suited for training LMMs in interactive image–text generation capabilities.
To evaluate the capabilities, we propose \textbf{SynJudge}, a reliable automatic evaluator that aligns closely with human judge and outputs four interpretable scores: Text Content Completeness (TCC), Image Content Completeness (ICC), Image Quality (IQ), and Image–Text Synergy (ITS).
These scores are complementary, covering both content and quality as well as cross-modal interaction, thereby forming a comprehensive evaluation framework.
Experimental results on InterSyn subsets of up to 200K samples show that 25K–50K already yield substantial improvements, while scaling to 100K/200K brings further gains in TCC, ICC, and especially ITS, highlighting InterSyn’s:
(1) scalability, as performance consistently improves with more data;
(2) efficiency, as significant gains are achievable even with smaller subsets, making it accessible to researchers with varying computational resources.

Published as a conference paper at ICLR 2026
A HIGH QUALITY DATASET AND RELIABLE EVALUA- TION FOR INTERLEAVED IMAGE-TEXT GENERATION
Yukang Feng" 12.3
F Jannen 1..3. Chuanhao Li', Zizhen Li Jiaxin Ai?.,
Fanrui Zhang'. Sizhuo Zhou?: Yifan Chang", Shenglin Zhang', Yu Dai Kaipeng Zhang2.3t Nankai University ?Shanghai Innovation Institute Shanda Al Research *Shanghai Al Laboratory yukangfeng@mail nankai edu. kaipeng. whangessandee com
ABSTRACT
Recent advancements in Large Multimodal Models (LMMss) have significantly improved multimodal understanding and generation. However, these models still struggle to generate tightly interleaved image-text outputs, primarily due to the limited scale, quality, and instructional richness of current training datasets. To address this, we introduce InterSyn, a dataset that features: (1) large scale, com- prising 1.8M multimodal samples; (2) high quality, supported by our proposed Evlluaatlttion with Iterative Refinement (SEIR) method for rigorous auto- mated quality refinement; (3) rich instructional diversity, ensured through diverse well-designed question templates, based on human preferences and covering a 3500-topic hierarchy. These characteristics make InterSyn particularly well-suited for training LMMs in interactive image-text generation capabilities. To evaluate the capabilities, we propose SynJudge, a reliable automatic evaluator that aligns closely with human judge and outputs four interpretable scores: Text Content Completeness (TCC), Image Content Completeness (ICC), Image Quality (IQ). and Image-Text Synergy (ITS). These scores are complementary, covering both content and quality as well as cross-modal interaction, thereby forming a com- prehensive evaluation framework. Experimental results InterSyn subsets of up to 200K samples show that 25K-50K already yield substantial improvements, while scaling to 100K/200K brings further gains in TCC, ICC, and especially ITS, highlighting InterSyn's s: (1) scalability, as performance consistently improves with more data; (2) efficiency, significant gains achievable even with smaller sub- sets, making it accessible to researchers with varying computational resources.
INTRODUCTION
Multimodal understanding and generation are critical capabilities toward artificial general intelli- gence. In the past two years, multimodal large language models (MLLMs) (Liu al. 2023 Chen et al.) [222c;; Wang et al.) (2244)) have shown remarkable performance in multimoaal understand- ing and even surpassed humans in some areas, while we have also seen many impressive advances in high quality image generation (Esser et al.) 2024b; Betker et al. (2023). However, these models are often limited to generating either text or image outputs in isolation, while real-world scenarios typically require tightly interleaved multimodal outputs.
Recently, pioneer unified LMMs, such as Janus-Pro (Chen et al. 2025b), have shown great potential. However, they struggle to generate cnttruction-ollowiinn interleaved image-text outputs, manifesting issues such as semantic drift, low image-text synergy, and poor image quality.
The main challenges lie in the limited scale, quality, and instructional richness of existing datasets. Even with existing datasets (Zhu et al., 2023 Laurençon et al. 2023 Chen et al.] 22024a] b) Xu et al.)
2024), these challenges remain due to their critical limitations: (1) Limited scale: Focus narrow tasks and typically contain no more than tens of thousands of samples, limiting their applicability to broader real-world scenarios; (2) Unstable quality: Built web-crawled sources (Yang al.
1* Equal contributions.
24 Corresponding author.

---

Published as a conference paper at ICLR 2026
Ces imapre thelives soue on manseer?
namart
The daring Fasthall la rosl imporantt mudeer heal ug ane ba dll martos play kg unick Ereyy Jynus kimepphrrs.
tase tihee and hight eyen thoe lov ruch edjos oot swpello happy.
nacarss The nolly piouest Pease adtioondly kos that oar Inon 1r
Neow wan giv nos liter you please drewe whatl
Glaripandes realio on
Gunel la
awens coteri Franch
parts vardou aboot moaoooo so%s
going
ra meding quii.
then tedding le poweeree puper
by clsariaaioon pardel have
ge pors.
Foe
dil over line.
recaloo on the seaads Do shre saca wee gaing on ooald you thene scame
Veagen aoutors have your fraaly plocure docter word
Iny eayes, dedarrs are
When' vensleeinnnnnn
will costs of
apdl sumorren
earts ehing rays
apprnpiian
Prey valuyy diegrom parlinms
wobder d'hes, Alt,, be cautiaan
is bringg hope and light agened thees
crving spors song seaide
wwen bether you cout drom
rorandl sal
the srone brige the ear/
where vdga carling
aurface
feep your
aroli
y cleek sgo dlon It' sundaa begin la sst My taree palnts of Harm gras Hakking
rileart pickeee
you
marl vater beboo stenme tarty.
Figure 1: InterSyn: Topic hierarchy and interleaved question answering samples
202
Laurençon et al., 2023 or reused corpora (Zhou et al. 2018 Zhu et al. 2023) with incon-
sistent quality and lack standardized quality control mechanisms; (3) Low interaction complexity: Rely on static documents single-turn prompts, thus failing to capture the contextual richness and interleaved structure of authentic human conversations.
To address the above issues, we introduce InterSyn-the first fully automated, high quality, large scale dataset for instruction-following multi-turn question answering with interleaved image-tex text responses. With 1.8M single-tume and 50,000 multi-turn dialogues across 8 domains and 3,500 top- ics (as shown in Figure D. InterSyn provides extensive coverage for diverse real-world scenarios. InterSyn's 's quality is further enhanced by our Self-Evaluation with Iterative Refinement (SEIR) method, which embeds self-checking and feedback loops into each generation step, enhancing se- mantic completeness and cross-modal synergy. To ensure rich instructional diversity, we extract human-like query styles through question templates that capture varied linguistic structures, sup- ported by a rich topic hierarchy for diverse, instruction-drre -rrivn dialogues.
To evaluate interleaved image- text generation. several related benchmarks (An et al. 2024 Zhou al. 2024, Chen al., [025a]] Xia al., 2024 Liu et al., 2024) have been proposed, but they still suffer from the following limitations: (1) Limited domain and scale: small, task-specific sets can- not cover realistic multi-turn dialogue needs; (2) Costly manual evaluation: accurate assessment still hinges on human, whose expense and delay hinder large scale, rapid benchmarking: (3) Weak alignment with human preference: current automatic metrics significantly diverge from human judgments on fine-gra fnninee multimodal reasoning; (4) Narrow evaluation scope: emphasise surface correctness while overlooking synergy and overall answer quality.
Therefore, propose SynJudge, reliable and comprehensive judge model for evaluating in- terleaved image-text generation with high alignment to human judgment. SynJudge provides in- terpretable, quantitative feedback across four key dimensions: text content completeness (TCC),
2

---

Published as a conference paper at ICLR 2026
image content completeness (ICC), image quality (IQ), and image-text synergy (ITS). Unlike tra- ditional image-text text consistency metrics, the ITS metric focuses on rewarding tight, complementary alignment between the textual and visual modalities while penalizing redundancy.
To validate our contributions, we conduct a series of experiments. First, the effectiveness of our SEIR method is demonstrated, showing substantial quality improvements over non-refined base- line. Additionally, comparisons with several existing models show that SEIR consistently outper- forms them across all evaluation metrics. Next, we validate SynJudge, which exhibits the strongest alignment with human judgment, showing a significantly smaller deviation compared to zero-shot MLLM evaluators. Finally, we validate the utility of InterSyn by fine-tuning models on randomly sampled subsets, up to 200K examples. Results reveal significant performance gains with as few as 25K/50K samples, underscoring the dataset's high data density and efficiency. These results confirm that our methods and dataset contribute to improved multimodal model performance.
Our main contributions are summarized as follows: (1) We present InterSyn, large scale dataset of 1.8M high-fidelity samples, distinguished by its instructionally rich and complex dialogues that span over 3,500 topics: (2) We propose SEIR, a method that ensures high quality data generation across refinement steps with minimal manual effort; (3) We introduce SynJudge. a multi-dimenniinnll evaluation model for scoring interleaved outputs, enabling fine-grained assessment and effective feedback for model improvement; (4) We conduct comprehensive experiments demonstrating that InterSyn substantially enhances LMM performance in instruction alignment, image-text synergy, and multi-turn reasoning, contributing the advancement of unified multimodal systems.
2 RELATED WORK
Models for Interleaved Image-Text Generation. Recent advances in MLLMs, such as Flamingo (Alayrac et al. 2022). InternVL. (Chen et al. 2024d). and VW (Wang et al. 2024a),
have substantially improved multimodal understanding. Meanwhile, diffusion models (Ramesh ett al. 2022, Betker et al. 2023 Esser et al. (2024) achieve strong visual generation performance.
To unify understanding and generation, models such as MiniGPT-5 (Zheng al., (023) and Show- o (Xie et al. 2024) combine autoregressive text generation with diffusion-basee image synthesis.
More recent efforts (Team 2024] Wu al.] [024; Wang al. [024b] Chern al., 2024) adopt uni- fied autoregressive frameworks for interleaved generation. However, lacking targeted, high quality training data, these models are not explicitly optimized for foolluction-followingg and often struggle to maintain coherence and cross-modal consistency- gap InterSyn is specifically designed to fill.
Datasets for Interleaved Image-Text Generation. High quality interleaved image-text data cru- cial for training multimodal models. Existing large scale datasets like MMC4 (Zhu et al. 2023), OBELICS (Laurençon et al. 2023), and CoMM (Chen et al. 2024b) are primarily document-level corpora constructed from web sources, but often suffer from noise, weak alignment and low inter- action intensity. Several benchmarks, such as OpenLEAF (An et al. 2024). InterleavedBench (Liu) et al. 2024), and ONG (Zhou al.) 20244, focus on specific tasks. LeafInstruct (Xu et al.; 2024 constructs an interleaved image-text dataset by filtering samples from existing corpora (Zhu al. 2023 Huang et al. 2016 Zhou et al. 2018). However, both benchmarks and datasets remain limited in scale and instructional diversity. To this end, we introduce InterSyn, a large scale, high quality dataset with diverse, multi-turn dialogues and automated refinement.
Evaluation for Interleaved Image-Text Outputs Early multimodal evaluation metrics indepen- dently assessed text quality (Papineni et al. 2002 Lin 2004) and image quality (Heusel al. 2017
Salimans et al.) 2016). Subsequent metrics (Hessel et al. 2021 Li et al. 2023; Lin et al. 2024
Chen et al. [2023, Lu et al., (004) targeted image-text consistency, yet still inadequately eval waled the quality of interleaved outputs. More recent efforts, including InterleavedEval (Xu et al. 2024) and CoMM (Chen et al., 2024b), leveraged MLLMS for holistic assessment, but often exhibit mis- alignment with human judgment. OpenING (Zhou et al.] 2024) proposed IntJudge for pairwise comparisons, but it lacks fine- -re-graided, quantitative scoring for individual responses, limiting its ap- plicability for model training and refinement. In contrast, the proposed SynJudge provides a more comprehensive evaluation by assessing both content completeness and modality synergy, aligning more closely with human judgment.
3

---

Published as a conference paper at ICLR 2026
Preparatory
Filtaring benchmark
Quastion femplate
Topic extraction
Topic
Final answer and tamporary caption
25 particpoats propose 1,000 questons.
Filter questons and build opan comaln GA
Transform ligh-qulltty
questonn into
Sort the topics and construct a baskc topii hicrardhy
Al synthesie and manual optmization to expand the toplc hierarcht
bandhmark
standardized templates.
Self-Evaluation Neratiive Refinement (SEIR)
LLM:
MLLM:
GM: FLUX
Owen
GPT
SeepSoast
TLLAWA
InterVVL
LLIVA
Question Refinement
Image Refinement
Templates
Topics
Go
Natorial Contost
Answer Refinement
Generator
Evaluator
Refiner
K
30
qusston::
ComArnstton Sateyy
Evaluator
Evaluator
Contert
Contect
quauton: a
ql
Refiner
Refiner
Caption
G-
Generator
Generator
Final question:q q
Temp: caption:
2,7
image
Figure 2: Overview of the InterSyn Dataset Construction Framework. The top panel illus- trates the dataset preparatory work, covering question collection, filtering, template standardization, and topic expansion. The bottom panel illustrates the Self-Evaluation with Iterative Refinement (SEIR) method, which employs Question Refinement (QR): An initial question qo is refined into the final question g based the topic and question template. (2) Answer Refinement (AR): Using q. the final answer a and a tem- porary caption y are iteratively refined. (3) Image Refinement (IR): Initialize co with 7. Refine the caption and image until the final image / is produced. The right-side legend details the inputs and historical context (H((t-)) used at each stage.
Evaluate-Refin loop across three cascaded stages. (1)
3 INTERSYN DATASET AND SYNJUDGE EVALUATOR
3.1 OVERVIEW
In this section, present comprehensive framework for both the construction and evaluation of interleaved image-text generation. First, we describe the InterSyn dataset construction pipeline, de- tailing the preparatory work and our proposed SEIR method (illustrated in Figure 2p. Subsequently, we introduce SynJudge, specialized evaluator designed to assess the quality of interleaved outputs, along with the specific metries and benchmarks established to validate the model's performance.
3.2 DATASET PREPARATORY WORK
InterSyn's 's preparatory work involves five major stages:
Question Collection. We recruited 25 participants, each providing 40 questions drawn from natural conversational scenarios, resulting in a total of 1,000 questions.
Question Filtering and Benchmark. We combined LLM--bsed filtering and expert review to select high quality questions. Redundant, ambiguous, uncommon, and overly subjective samples were removed based predefined criteria. In total, 500 questions were reserved basic benchmark.
Question Template Extraction. Based on the selected high quality questions, we constructed a set of generalized question templates that capture the conversational query style (i.e., the common lin-
4

---

Published as a conference paper at ICLR 2026
guistic forms people use to pose requests), independent of any specific knowledge content, thereby enabling scalable question generation. See the appendix $ for details.
Basic Topic Hierarchy. We performed Al-assisted topic extraction from the filtered questions and manually organized the results to build a basic topic hierarchy, ensuring clear logical dependencies and coherent topie relations.
Topic Hierarchy Expansion. We further refined and expanded the basic topic hierarchy to improve both coverage and gramularity. Combining Al-assisted topic suggestions with expert curation, constructed a well-structured hierarchy that supports diverse and scalable data generation.
3.3
SEIR METHOD
The Sell-Evaluation with Iterative Refinement (SEIR) method enhances data quality via a Generate- Evaluate- Refine loop. For each conversation turn i, we define a generic refinement operator + that transforms an initial content (e.g., question or answer) 2o into a final content x through K iterations. The loop terminates when the evaluator returns suggestions or when maximum iteration depth is reached. The update rule at iteration k is:
Fk - -r.ffinn
where 8y = -Me
Here, Meeat acts judge, analyzing the current content 2 against context C (e.g., the target topic, preceding question, or dialogue history) to provide specific suggestions sg. MMrefine then produces improved version Tg. We apply this operator sequentially across three stages for the current turn t:
(t) from template and Step 1: Question Refinement (QR). We first generate an initial question 90 topic. To ensure clarity and depth, we apply & obtain the final question (t))
( 2( (1) C {z, ((1)))))
The evaluator critiques the initial question du based on the topic and conversation history Ht-1 continuing the refinement until the question meets quality standards or the iteration limit is reached.
(t)
Step 2: Answer Refinement (AR). Using the final question (t)), we generate an initial answer- caption pair (ag (()). The refinement loop ensures the text is comprehensive and the caption is relevant, yielding the final answer (!)) and a temporary caption (())
(t) (a (0)) =((())) .70) 10)) = (t) H ((1-1)).
(3)
Step Image Refinement Finally, we generate and refine the image. We utilize the temporary caption (1) from AR as the initial caption cp In each iteration k; we generate an image ((!) which is then critiqued by a VLM (V) against the textual context (t()), (t)) and dialogue history (- 1),
The VLM's feedback s(k) guides the generation of refined caption c41
1 G((())
(k) = = (1) (t) H
C+11
(6) Mrefine (ck (1))..
(4)
This cycle repeats until satisfaction or the iteration limit K is met, yielding the final image (t).
3.4 INTERSyN COMPOSITION
InterSyn contains approximately 1.8 million single-turn samples and 50,000 multi-turn dialogues. Data quality is ensured through the SEIR method, which iteratively refines samples to improve an- accuracy, conversational coherence, and image-text synergy. InterSyn offers rare combination of diversity and quality, providing a robust foundation for training unified multimodal models with strong cestructionfoolooinn and contextual reasoning capabilities.
5

---

Published as a conference paper at ICLR 2026
3.5 SYNJUDGE: A RELIABLE EVALUATOR FOR INTERLEAVED OUTPUTS
To reliably evaluate the complex, instruction-following capabilities of interleaved image-text gen- erators, we propose SynJudge, a reliable and comprehensive judge model that demonstrates high alignment with human judgment.
3.5.1 1 EVALUATION DIMENSIONS
SynJudge is designed to provide interpretable, quantitative feedback by assessing responses across four key, complementary dimensions (See the appendix for details.):
Text Content Completeness (TCC): Assesses whether the generated text completely and accu- rately answers the user's question, covering all sub-tasks and constraints.
Image Content Completeness (ICC): Assesses whether the generated images correctly depict the key objects, scenes, or concepts requested in the prompt.
Image Quality (IQ): Evaluates the visual fidelity, aesthetics, and overall quality of the generated images, penalizing artifacts, distortions, or low resolution.
Measures the cross-modal relationship. Unlike simple consistency metries, ITS specifically rewards tight, complementary alignment where the text and images work together to form cohesive answer. It penalizes redundancy (e.g., text merely describing the image) and irrelevance.
Image-Text Synergy (ITS):
3.5.2 TRAINING AND MODEL SELECTION
To ensure robust evaluation, first constructed high-quality, human-annotatee dataset. The pro- cess began by collecting a diverse set of interleaved responses from various generators (detailed in $4.1.2). These responses were then rigorously scored by a panel of expert annotators across the four dimensions (TCC, ICC, IQ. ITS). To ensure high reliability, each sample was scored by multiple experts. Any scoring inconsistencies were resolved through a mandatory discussion pro- tocol until unified, final score was reached for each sample. This effort yielded total of 48 ,0000 annotated pairs, which were split into 38,400-sample training set and a held-out 9,600-sample validation set. Using this data, we fine-tuned strong MLLM backbones, including Qwen VL and In- tern VL. As demonstrated in our experiments (see $4.4), the model trained from Qwen VL achieved the strongest alignment with human judgment, exhibiting the lowest RMSE and highest agreement (A Consequently, we designate this SowenL--tratee model as our final SynJudge.
3.5.3 EVALUATION BENCHMARK FOR GENERATORS
To rigorously assess the quality of interleaved image-text outputs, established a fixed Evaluation Benchmark. This benchmark comprises a curated series of 4,000 questions (500 human-authored from $3.2 and 3,500 SEIR-generated) spanning the full topic hierarchy. By requiring different mod- els to generate responses to this identical set of questions, we ensure a fair, standardized, and com- prehensive comparison of their followii and multimodal generation capabilities.
EXPERIMENT
4.1 EXPERIMENTAL SETUP
4.1.1 EVALUATION DIMENSIONS FOR INTERLEAVED OUTPUTS
We adopt the four evaluation dimensions defined in $3.5.I TCC, ICC. IQ, and ITS. Briefly, TCC and ICC assess content correctness in text and image respectively; IQ focuses on visual fidelity; and ITS specifically measures the cross-modal synergy and alignment.
To guantitatively assess generators' performance, we adopt the mean score (average score across every dimension) and variance score (stability across questions). See the appendix $1r details.

---

Published as a conference paper at ICLR 2026
4.1.2 EVALUATED GENERATORS
We evaluate 13 multimodal generators G capable of producing interleaved image-text outputs and we categorize them into two groups based on whether they natively support interleaved generation:
(1) Non-Interleaved Generators: These generators produce text and images sequentially through
modular pipelines, which include: Emu3 (Wang et al., 2024b), Janus-Pro (Chen et al. 2025b),
VII.A-U (Wu et al. 2024), Show-o (Xie et 2024), Show-o- Tww--o-Turooo (Xu et al. 2025). Liquid (Wu
et al. 2025). D-DIT (Li et al.
2025 GPT-4o AT + DALL-E3 (Betker et al. 2023 and
Gemini2.5 (Comanici et al., 2025) FLUX (Black Forest Labs 2024).
(2) Interleaved Generators: These generators generate interleaved image-text outputs within unified process, including VARGPT (Zhuang et al. 2025a), VARGPT-vI.1 (Zhuang al. 2025b),
Anole (Cher et al., 2024) and BAGEL (Deng et al. 2025).
4.2
EFFICIENCY OF INTERSYN
4.2.1
DATA EFFICIENCY AND SCALABILITY
We fine-tune four generators randomly sampled InterSyn subsets of sizes 25k/50k/100k/2000 and evaluate with SynJudge. Table [ shows consistent improvements as data increases. Notably, even 25k/50k samples already yield clear gains across all dimensions, and further scaling to 200k contin- ues to improve TCC, ICC, and especially ITS, which highlights the InterSyn dataset's effectiveness in enhancing both semantic alignment and answer completeness.
Table 1: Fine-tuning results on varying subset sizes of InterSyn. Performance consistently improves as training data scales from 25K to 200K samples, demonstrating the dataset's effectiveness and scalability. Notably, just 50K samples yield substantial gains across all models, with continued im- provement in content and synergy metries (TCC, ICC, ITS) at larger scales. All scores are SynJudge means.
Anole
VILA-U
VARGPT-111
BAGEL
Subset
TCC
ICC
IQ
11'
TCC
1CC
IQ
11S
TCC
ICC
IQ
ITS
TCC
ICC
1Q
ITS
2.46
3.37
2.19
3.11
3.89
4.23
2.87
25k
3.35
3.25
3.01
2.40
2.95
3.78
3.38
2.90
3.51
2.45
2.90
2.55
3.45
4.02
4.21
3.28
3.19
3.69
4.19
100k
3.51
3.41
3.13
2.87
3.33
3.88
3.31
3.28
3.73
3.22
3.66
3.20
3.87
4.09
4.31
3.78
200k
3.52
3.08
3.11
3.49
4.01
3.40
3.47
3.39
3.72
3.53
4.13
4.18
4.25
4.02
4.2.2
RETENTION OF GENERAL UNDERSTANDING
Table 2: Understanding performance after 50k InterSyn fine-tuning. Values in paren-
Crucially, the substantial gains in interleaved genera- tion capabilities (demonstrated in $4.2.I) do not come
theses denote the change (A) from the base.
the cost of understanding performance. As shown in Table 2] the models' performance on stan- dard understanding benchmarks (Fu al. 2023] Liu et al. (2025) Yue et al., [2024: [. et al. 2024) remains robust after fine-tuning.
Model
MME-P
MLIBench
MMMU
SEEDBeench
VILA-U
334 (+8)
57.1 (40.8)
VARGPT
1465 (23)
66.8
57.2
63.6
VARGPT
1658 266
(1.6
(6.3
BAGEL
1646
4.2.3 EFFECTIVENESS OF MULTI-TURN DATA
To verify the effectiveness of our multi-turn dialogue data, we designed experiment to assess its impact on models' conversational capabilities. We fine-tuned two models capable of multi-turn generation, Anole and VARGPT-vI.1. on different compositions of single-turn and multi-turn data, while keeping the total training size fixed at 50k samples. The goal is to demonstrate that training with multi-turn data enhances model's 's ability to maintain context and quality across extended conversation. The performance, evaluated by TCC, ICC, IQ. and ITS, is presented in Table 3
The results clearly demonstrate the value of multi-turn training data. First, the inclusion of multi- turn data does not compromise first-turn performance, which remains high across all trained settings. This is expected, the SEIR pipeline ensures comparable data quality regardless of dialogue type.

---

Published as a conference paper at ICLR 2026
Table Effectiveness of multi-turn data on conversational performance across across different dia- logue test turns. Models are trained on different proportions of single-turn and multi-turn data.
Model
Setting
Training Data
Turn I
Turn 2
Turn
Single
Multi
TCC
ICC
IQ
ITS
TCC
ICC
IQ
ITS
TCC
ICC
IQ
ITS
Raseline
3.09
2,92
2.80
2.75
Anole
Trained
Trained
50k
3.47
3,48
3.28
3.13
3.10
2.74
3.25
2.85
3.00
2.70
2.40
2.85
2.60
2.45
2.05
Trained
0
50k
3,52
3.24
3.10
2.94
3.40
3.05
2.90
2.55
3.27
2.85
2.70
2.25
Baseline
3.26
1.01
1.23
0.68
3.10
0.95
1.18
0.72
2.90
0.97
0.90
0.65
VARGPT-v1.1
Trained
50k
3.68
3.12
3.67
3.00
3.40
2.90
3.45
3.05
2.60
2.65
Trained
25k
25k
3.65
3.18
3.66
3.18
3.45
2.95
3.40
2.90
3.21
2.78
3.24
2.66
Trained
3.10
Second, and more importantly, multi-turn training significantly reduces performance degradation in later turns. Models trained only on single-tur data show a steeper drop in quality as the conversa- tion progresses. In contrast, training with multi-turn dialogues mitigates this degradation, particu- larly for ITS, by explicitly teaching the model to maintain context across extended conversational dependencies. This confirms the effectiveness of our multi-tur turn dataset in fostering more coherent and consistent multi-turn generation.
4.3 EFFECTIVENESS OF SEIR
To validate the effectiveness the SEIR, we conduct experiments on both the iterative refinement process and the final output quality comparison with other generators.
4.3.1 VALIDATION OF ITERATION REFINEMENT
In this section, all evaluations in this section are conducted by human judge to establish ground truth. For question quality, Figure 3 shows that QR improves quality over the first three iterations but plateaus thereafter. Based on this, we set the QR to 3, achieving 99.5% of peak quality while reducing computational cost by 40%. For answer quality, we evaluate across different iterations of AR and IR, using a set of 7,000 questions generated through three rounds of QR. As shown in Table 4 the results confirm that AR primarily improves content completeness, while IR enhances multimodal synergy, demonstrating the effectiveness of the SEIR method. Experimental results show that when both AR and are set to 4 or 5, the improvements become marginal. Based on this, fix the number of AR and IR iterations to 3 in dataset construction settings.
Table 4: Impact of answer refinement (AR) and im- age refinement (IR) answer quality. The table ports human evaluation mean scores across four mensions (TCC, ICC, IQ. ITS). AR improves TCC, ICC, and ITS, while IR further enhances ICC and ITS, confirming the effectiveness of itcrative refine- ment.
$.M
AR
IR
TCC
ICC
IQ
ITS
Avsngee Quendion Qualty Szoms
0
0
3.85
4.01
4.42
3.79
1
4.19
4.21
4.38
4.07
Tandion Tun
2
2
4.34
4.35
4.41
4.38
3
0
4.42
4.11
4.37
4.04
Figure Impact of question refinement (QR) on question quality. This plot shows the quality scores across different QR itera- tions. Quality improves significantly over the first three iterations but plateaus thereafter.
3
4.42
4.33
4.43
4.35
4.42
4.43
4.39
4.46
4.42
4.47
4.52
4
4.44
4.50
4.45
4.53
5
5
4.45
4.49
4.43
4.53
8

---

Published as a conference paper at ICLR 2026
4.3.2 COMPARISON OF SEIR OUTPUTS WITH OTHER GENERATORS
We evaluate the SEIR pipeline against 13 baseline generators using the fixed Evaluation Bench- mark defined in |3.5.3 In this section, we employ both human judges and SynJudge to assess the generator outputs across the four evaluation dimensions.
As shown in T both human judges and SynJudge confirm that Syn) achieve the highest mean scores across all dimensions. They outperform the strongest baseline, GPT-4o+DALL-E, by margin of 0.34-0.66 with the largest gap observed in ITS. Furthermore, InterSyn exhibits very low variance (below 0.61), reflecting its consistent output quality and the robustness of our automated method. Crucially, the significant performance gap between InterSyn and the best SOTA generators reveals that even top models still struggle with image- text alignment and complementarity, indicating substantial room for future improvement.
samples (Inter-
Table 5: Generator performance evaluated by human judge and SynJudge. Each entry is reported as mean (variance), where the value outside the parentheses denotes the mean score and the value inside the parentheses denotes the variance.
Human
SynJudge
Generator
TCC
ICC
IQ
ITS
TCC
ICC
1Q
ITS
Anole
3.06 (1.47)
2.95 (2.05)
2.89 (1.87)
2.25 (2:47)
3.09 (1.47)
3.01 (2.11)
2.92 (1.93)
2.26 (2.55)
DDIT
0.38 (1.17)
3.51 (0.83)
3.29 (0.85)
0.37 (1.21)
0.28 (0.86)
3.67 (0.75)
3.34 (0.92)
0.26 (0.87)
Emu3
3.38 (0.86)
3.86 (0.86)
3.87 (0.59)
3,26 (1.31)
3.37 (0.74)
3.92 (0.66)
3.85 (0.80)
3.37 (1.16)
VILA-U
2.45 (2.58)
3.62 (0.91)
3.35 (0.97)
2.31 (3.01)
2.46 (2.73)
3.72 (0.92)
3.37 (1.12)
2.19 (2.96)
Liquid
2.88 (0:76)
3.82 (0.69)
3.67 (0.70)
(1.42)
2.87 (0.71)
3.86 (0.75)
3.76 (0.85)
3.06 (1.36)
Janus-Pre
2.93 (0.95)
3.25 (1.21)
3.16 (0.99)
2.55 (1.75)
2.96 (0.96)
3.30 (1.09)
3.11 (1.11)
2.62 (1.87)
Show-o
3.32 (1.31)
3.65 (0.94)
3.52 (0.89)
(1.94)
3.49 (1.12)
3.79 (0.82)
3.57 (0.96)
3.30 (1.70)
Show-o-Tuth
3.48 (1.12)
(0.88)
(0.94)
(1.60)
3.50 (1.06)
3.89 (0.86)
3.64 (1.01)
3.45 (1.44)
VARGPT
2.60 (0.66)
0.94 (2.37)
0.94 (2.32)
0.55 (1.99)
2.55 (0.30)
0.94 (2.39)
0.87 (2.17)
0.67 (2.16)
VARGPI-v1.1
3.13(0.83)
0.89(1.98)
1.16(2111)
0,72(1.83)
3,26(0.76)
1.01(.12)
1.23(1.95)
0.68(1.96)
BAGEL
3.92(0.81) 2.97(0.83)
(.111 2.81(1.21) 4.18(0.75)
4.23(0.66) 3.89).772)
2.87(1.33)
Gemini-Flux
3.94 (0.94)
4.06 (0.58)
4.43 (0.47)
3.81 (0.90)
3.97 (0.57)
4.12 (0.71)
4.48 (0.64)
3.84 (L.II)
GPT-4o+DALL-E
4.05 (0.37)
4.08 (0.48)
4.41 (0.57)
3.94 (0.64)
3.99 (0.65)
4.10 (0.81)
4.45 (0,58)
3.87 (1.16)
SEIR
4.41 (0.55) 4.46 (0.55)
4.47 (0.53)
4.51 (0.57)
4.39 (0.61)
4.49 (0.63)
4.44 (0.45)
4.53 (0.51)
4.4
RELIABILITY OF SYNJUDGE
4.4.1
EVALUATION SETUP FOR JUDGES
To identify a reliable automatic evaluator that aligns closely with human scoring, conduct comparative experiment. The protocol is designed to rigorously measure each candidate judge's deviation from a human-annotated ground truth.
We evaluate a total of five model-based judges against our human ground truth. The candidates are: (1) Human Judge (Ground Truth): A panel of ten experts whose scores serve as the gold standard. A cross-review protocol was used to ensure scoring reliability and mitigate individual bias. (2) Zero-Shot MLLM Judges: Three off-the-shelf MLLMs used for automated assessment: GPT-4o (OpenAl 2024), Qwen (Bai et al. 2025), and Intern VL2.5 (Chen et al.] (024c). (3) SynJudge Candidates (Finetuned): We fine-tuned two strong MLLM backbone candidates, and Intern L--traind,, to create our proposed evaluator.
The evaluation is conducted on test set of 9,600 human-annotated question-answer pairs. To
provide a comprehensive assessment of judge performance, we use two complementary metrics: (1)
Root Mean Squared Error (RMSE): Measures the magnitude of the deviation from human scores. (2)
Human Agreement (A @ I): Measures the percentage of scores that are within 1-point tolerance of the human rating, reflecting practical reliability. A lower RMSE and a higher A @ indicate stronger alignment with human judgment. The SynJudge candidates were trained of 38,400 human-ann annotated pairs. Detailed definitions of the metrics and training hyperparameters of SynJudge are provided in the appendix (D2. D.4..
separate training set

---

Published as a conference paper at ICLR 2026
4.4.2 COMPARISON RESULTS
O report the performance of each judge using both RMSE and A @@ metrics. The results show a clear trend: finetuned judges decisively outperform zero-shot models on both metrics.
Figure Hand Table
The QwenVL-trainee judge demonstrates the strongest alignment with human preferences. achiev- ing both the lowest average RMSE and the highest average A@ of 95.4%. This high agreement rate signifies that over 95% of scores are within one point of human judgment, confirming its high re- liability. In contrast, zero-shot judges like GPT-4o lag significantly, with A @ scores around 86.5%. Based on this superior performance across complementary metrics, we select Qwen ---traned as final SynJudge.
Table 6: Judge performance comparison. We report average RMSE (lower is better) and Human Agreement (A @ 1, higher better) against human scores. The best result in each row highlighted in bold. Q..tannd demonstrates the strongest alignment.
Gunli
Dimeusioo
GPT-1o
QwenVL
IntervVL.
QuenVL..rainel
IniemVVLtrainedd
TCC (RMSE)
1.01
0.81
0.96
0.54
0.55
ICC (RMSF)
1.02
1,09
0.90
0.72
0.70
0.72
Figure 4: Visualization of RMSE for different judges.
ITS (RMSE)
1.18
1.20
L.03
0.67
0.72
0.875
0.945
CONCLUSION
In this paper, we present InterSyn, a large scale, high quality multimodal dataset designed for instruction-following and interleaved image-text generation. Constructed via the fully automated SEIR method, InterSyn combines scale, diversity, and fidelity, supporting multi-turn dialogues where each response is refined to achieve not only semantic completeness but also tight image-text synergy-ensuring that visual and textual modalities complement each other to convey meaning col- laboratively. To complement InterSyn, we introduce SynJudge, a dinteiidimniio automatic eval- uator specifically designed to assess interleaved outputs across four key dimensions, including ded- icated metric for image-text synergy. Unlike traditional metrics focused on surface-level alignment or consistency, SynJudge emphasizes the semantic interplay between images and text, rewarding complementary relationships while penalizing redundancy or disjointness. Extensive experiments validate the effectiveness of both InterSyn and SEIR. Models fine-tuned on InterSyn consistently outperform strong baselines, showing notable improvements in instruction alignment, multimodal reasoning, and especially the ability to produce coherent, synergistic interleaved content. We be- lieve this work lays a solid foundation for future research in scalable multimodal data generation, robust synergy-centrin evaluation, and the development of general-purpose multimodal intelligence systems that understand and communicate across modalities in a truly integrated manner.
Reproducibility Statement. To facilitate the verification of our findings and support future re- search, we are committed to making our work fully reproducible. The complete codebase, including scripts for the SEIR data generation pipeline, SynJudge training, and all evaluation protocols, will be made publicly available on GitHub upon publication. We will also release the full InterSyn dataset (1.8M samples), all benchmark sets used for our main experiments, and the trained weights of our final SynJudge evaluator. The core of our methodology relies on publicly available models, and all prompts, model configurations, and hyperparameters are extensively detailed in the appendix to ensure that our results can be precisely replicated.
REFERENCES
Jean- Man-BBatistt Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In Proceedings of the Advances in Neural Information Processing Systems, 2022.
10

---

Published as a conference paper at ICLR 2026
Jie An, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Lijuan Wang, and Jiebo
Luo. Openleaf: A novel benchmark for open-domain interleaved image-text generation. In Pro- ceedings of the 32nd ACM International Conference Multimedia, MM '24, pp. 11 11137-11145, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706868. doi: 0011445/3664647.36655511 Utt.https: //ooi. org/
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song. Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13922. 2025.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dallee-3. pdf, 2(3):8, 2023.
Black Forest Labs. Flux. cessed: 2024-11-05.
forest ptooest-labbee
2024. Ac-
Dongping Chen, Ruoxi Chen, Shu Pu, Zhaoyi Liu, Yanru Wu, Caixi Chen, Benlin Liu, Yue Huang, Yao Wan, Pan Zhou, and Ranjay Krishna. Interleaved scene graphs for interleaved text-and-image generation assessment, 2025a. htttt:t/p//xxxi>>> org/ /abs//411.... 17188
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Confer- ence on Computer Vision, pp. 370-387, Springer, 2024a.
Wei Chen, Lin Li, Yongqi Yang, Bin Wen, Fan Yang, Tingting Gao, Yu Wu, and Long Chen. CoMM: A coherent interleaved image-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2406. 10462, 2024b.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint MXiv:2501.178// 2025b.
Yixiong Chen, Li Liu, and Chris Ding. X-iqe: explainable image quality evaluation for text-to- image generation with visual large language models, 2023. URL
abs/2305. 10843
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024c.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguisic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24199 2024d.
Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large mul- timodal models for interleaved image-text generation. arXiv preprint arXiv:2407.06/3 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, and so on. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL httttttt///xxiii wtwp://..rgaaab////a/eeeee httpp//g/aa/2/50..0.0 06261
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Wei- hao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. /4683, 2025.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024a.
11

---

Published as a conference paper at ICLR 2026
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference Machine Learning, 2024b.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Marie-Framcine Moens, Xuan- jing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, pp. 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: https/////22211..lpmaaaaaa emmmpp-- main 595
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Ting-Hau Hao Huang, Francis Ferraro, Nasrin Mostafazadch, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In Proceedings of the 2016 conference of the North American chapter of the association for compu- tational linguistics: Human language technologies, pp. 1233-1239 2016.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander Rush. Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36:71 1111666377700 2023.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Sced-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13299-13308, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19730-19742. PMLR, 2023.
Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang.
Dual diffusion for unified image generation and understanding, 2025. URL/https://axxt
org/ 2501. 00289
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oul, pp. 74-81, 2004.
Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Revisiting the role of language priors in vasion-language models, 2024. URL htthtttss///xixi org. abs/
2306. 01879
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023,
Minqian Liu, Zhiyang Xu, Zihao Lin, Trevor Ashby, Joy Rimchala, Jiaxin Zhang, and Lifu Huang- Holistic evaluation for interleaved text-and-image generation. arXiv preprint arXiv:2406. 14643, 2024.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? In European Conference Computer Vision, pp. 216-233. Springer, 2025.
Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. Limscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2026
OpenAL. Hello GPT-4o. http:///wwa./oooaaaaaa.//a////a/ 2024 Accessed: 2024-05-26.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/125, 1(2):3, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 29, Cur- ran Associates, Inc., 2016. URL TRLLhtttss///ppceeeeeet ngs neur ips cctppperrfelesss files/
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.008/18 2024.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing languae model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191,, 2024a.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li. Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. (0t::2409.888000 18869, 2024b.
Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable and unified multi-modal generators, 2025.
URL https: /prsiv//axxv..o org/ abs/ aww./41212 2412. 04332
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, al. Vila-u: unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.044 04429, 2024.
Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang, Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li, et al. Mmie: Massive multimodal interleaved compre- hension benchmark for large vision-language models. arXiv preprini arXiv:2410. 10139, 2024.
Jinheng Xic, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang. Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint ar arXiv:2408. 12528, 2024.
Chenkai Xu, Xu Wang. Zhenyi Liao, Yishun Li, Tianqi Hou, and Zhijie Deng. Show-o turbo: Towards accelerated unified multimodal understanding and generation, 2025. URL https:
//arxiv.org/ant abs////b///// 2502. 05415
Zhiyang Xu, Minqian Liu, Ying Shen, Joy Rimehala, Jiaxin Zhang, Qifan Wang, Yu Cheng, and Lifu Huang. Lateralization LoRA: Interleaved instruction tuning with modality-specialized adap- tations. arXiv preprint arXiv:2407.03660 2024.
Yue Yang, Artemis Panagopoulou, Qing Lyu, Li Zhang, Mark Yatskar, and Chris Callison- Burcch. Visual goal-step inference using wikiHow. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2167-2179, Online and Punta Cana. Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/w1/2021 1/t/3///1/22..mmmlppp main. 165. URL https::tt///.. //aci antho lotoooloyyy.. ogy. wwwporg//2222. 2202 .mmn p- ma n. 65)
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
13

---

Published as a conference paper at ICLR 2026
Kaizhi Zheng, Xuehai He, and Xin Eric Wang. MiniGPT-5: Interleaved vision-and -ad---ndd-lllngg and-language gener- ation via generative vokens. arXiv preprint arXiv:2310.02239, 2023.
Luowei Zhou, Chenliang Xu, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos. In Sheila A. McIlraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Secome AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-8). New Orleans, Louisiana, USA, February 2-7, 2018, URL https://dol
pp. 7590-7598. AAAI Press, 2018. doi: 10. N0.169/AAAALV3311.10 org/10 16096669331... v v3211. 12342
Pengfei Zhou, Xiaopeng Peng, Jiajun Song, Chuanhao Li, Zhaopan Xu, Yue Yang. Ziyao Guo, Hao Zhang, Yuqi Lin, Yefei He, Lirui Zhao, Shuo Liu, Tianhua Li, Yuxuan Xie, Xiaojun Chang, Yu Qiao, Wenqi Shao, and Kaipeng Zhang. Gate opening: comprehensive benchmark for judging open-ended interleaved image-text generation, 2024.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Young- jae Yu, Ludwig Schmidt, William Yang Wang. and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. In Thirty-seventh Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2023. URL https:
//openreview.o wttp://wwwwweeeto..u.e..a. (ttorrmsssttttttossssss......
Xianwei Zhuang. Yuxin Xie, Yufan Deng, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou. Vargpt: Unified understanding and generation a visual autoregressive multimodal large language model, 2025a. httpt/tttsss/rtett
org.vv.rrraaaee abs///b///// 2501 12327
Xianwei Zhuang. Yuxin Xie, Yufan Deng, Dongchao Yang, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou. Vargpt-v1.1: Improve visual autoregressive large unified model via it- crative instruction tuning and reinforcement learning, 2025b. URL https://arxiv..
abs foss/504. 02949
14

---

Published as a conference paper at ICLR 2026
A STATEMENT ON LLMS USAGE
The authors used large language models (LLMs) during the writing process solely for language refinement and editing. It should be explicitly stated that LLMs were employed in any core aspects of the research, including the formulation of research ideas, the design of methodologies. the execution of experiments, or the development of conclusions. All scholarly contributions were made independently by the authors.
B APPENDIX OVERVIEW
The all supplementary document is organized as follows:
Comparison of datasets and Samples of InterSyn are shown in $C
Supplementary analysis of experimental Data are shown S
Question templates and topic hierarchy shown in $
The evaluation dimensions for question and answer shown in
Prompts used in this work are shown in G
The benchmark samples are shown in {H
Human annotation platform are shown in
Limitations of this study are shown
C COMPARISON OF DATASETS AND INTERSYN SAMPLES
C.I COMPARISON OF DATASETS
The table provides comprehensive comparison of InterSyn with representative multimodal datasets and benchmarks. Existing datasets such as MMC4 and OBELICS primarily rely on large scale web- crawled corpora, often lacking instruction-followt following capabilities and multi-turn structures. Other resources like CoMM and SPaVV improve data cleanliness but remain limited to single-turn interactions without tight semantic supervision.
Recent efforts including LeafInstruct introduce instruction-followin supervision but still oper- in single-turn formats. Meanwhile, benchmark-orrentee resources-suce as OpenLEAA ISG- BENCH, MMIE, InterleavedBench, and OpenING-focur focus on evaluating generation quality but are constrained by small scale and limited turn complexity.
In contrast, InterSyn is the first to offer a large scale, multi-turn, fnstruction-folloiing dataset specif- ically designed for interleaved image-text generation. Built with the SEIR method, InterSyn not only ensures high quality iisual-textual synergy but also scales to 1.8 million samples samples-odeer orders of magni- tude larger than existing benchmarks. Its emphasis on dialogue coherence, iterative refinement, and synergistic multimodal responses fills critical gap in current resources, laying the groundwork for developing and evaluating truly unified multimodal generation models.
Table Comparison of Multimodal Datasets and Benchmarks. Abbreviations: Cat. Category; Inst. instruction-followingg MT. = multi-turn; DS. dataser; BM. = benchmark; Gen. genera- tion.
Name
Cat.
Source
Method
Size
Inst, MT.
InterSyn
DS.
Collected questions
SEIR
1.8M
MMC4
DS.
Common Crawl
CLIP-based
101.2M Doc.
X
filtering
15

---

Published as a conference paper at ICLR 2026
Name
Cat.
Source
Method
Size
Inst.
MT.
OBELICS
DS.
Common Crawl
Multi-granulartty filtering
141M pages
X
CoMM
DS.
WikiHow, Sto-
Multi-perspective filtering
227K Doc.
X
X
ryBird, eHow,
ShareGPT4V
DS.
GPT-4V cap-
Share-Captioner
1.2M pairs
x
tions
LeafInstruct
DS.
MMC4, VIST, YouCook2,
Text & image quality filtering
38,272
x
OpenLEAF
BM.
User Queries
GPT-4 Gen. and human review
660
X
ISG-BENCH
BM.
VIST, CoMM, manual Gen.
Model Gen. & 1,150 human review
x
MMIE
BM.
WikiHow, VIST, Math-
Sampling & re- construction
20,103
x
Vista, etc.
InterleavedBench BM.
VIST, How, etc.
Wiki-
GPT-4o Gen.
815
x
human review
OpenING
BM.
YouTube, Google, etc.
Manual pipeline
5,400
x
C.2 SINGLE-TURE SAMPLES
Samples of data are shown in Figure
C.3 MULTI-TURN SAMPLES
Samples of data are shown in Figure
D SUPPLEMENTARY ANALYSIS OF EXPERIMENTAL
D.1 SYMBOLS AND NOTATIONS FOR SEIR METHOD
We summarize the key symbols used in the SEIR method below:
T: Set of question templates.
Z: Set of topics.
T' E N+: Number of conversation turns.
K E N Number of refinement iterations at each stage.
Htt 1) History of the conversation up to turn t represented (( ((0)))))))))))) 11)
(
Final question generated at conversation turn
fg
(t),
Question after k refinement iterations at conversation turn L.
a (0), Final text answer generated at conversation turn
(1), Text answer after i refinement iterations at conversation turn t.
(1); Temporary caption associated with the text answer at conversation turn f.
()) Temporary caption after k refinement iterations.
k
16

---

Published as a conference paper at ICLR 2026
What fun sand game do kids enjoy of the beach? Any super describe
Could you describe the peaceful scene of canyon under the stars for I'd love clearly mind.
of?
Building sundcasties ties beloved activity for kids, involving creative construction with molds or simple tools to form castles, complete with moots, towers, and decorations like shells and flags. Safety tips include using sun protection and keeping hydrotedd. Variations include themed castles or competitive castle- sarle-building games.
tranquil conyon starlit sky, where the scent of pine. fills the air, the rustling of leaves whispers the
and the cool night wraps around you, creating an immersive, serene experience.
Con you descnbbbe what looks like when the shorp outiines of snow covered peaks stond out against crystal-clast night sky? sounds breathtakingl Could you also show scene?
What does dear look like when it's covened light dusting snow?
image
Imagine standing beneath a stor- studded, erystol-clear right sky. where the topped mountains sharply against deep blue heavens. The gentle moonlight bothes the pooks in silvery sheen, their crisp outlines starkly defined, and
light dusting of snow delioste, loyer that highlights its for texture, creating glistening effact that emphasizes its body
stark ontiers, as it's softly glowing against the winter backdrrop.
clear feels almost tangible, enveloping you in serene colm.
Could you describe the scene of stone house nestled omong the mountains ond surrounded by bright red leaves? And please image well
Could you describe for me scene whene cherry blossom
petals are floating on tranqui pond with their reflections the water? Also, please show it with image.
charming nestles cozily among Towering mountains, embraced by sea of vivid, bright red leoves, crafting breathtakingly becutiful Scene.
On the glassy surface of the pond, delicate cherry blossom petais drift gently, their soft mirrored perfectly creating dreamy. peaceful sight.
Figure 5: Examples of single-turn conversation
( Final image caption conversation turn
Image caption after k refinement iterations.
10); Final generated image at conversation turn t.
(0 :Generated image after l: refinement iterations.
My Language model used for text generation and refinement.
My: Vision-language model used for image caption evaluation and refinement.
MG: Text-to-image generation model.
P-(+): Prompt function for generating model response.
ps(:): Prompt function for generating refinement suggestions.
p+(-): Prompt function for applying refinements.
D.2 EVALUATION METRICS FOR JUDGES AND GENERATORS
To facilitate quantitative evaluation of judges and generators, design of metrics.
17

---

Published as a conference paper at ICLR 2026
Hove you soan any awesome photos of ancient murals old, weathered rock pillors? Where find same?'
What does the sky look like at Northem Lights Observation Station, and con describe and drow what amazing might like?
oncient murals found across globe, Ajanto Caves in India, featuring Buddhist derties, to the Magse Caves in China, depicting and France, showcasing prehistoric
panoramic night sky, featuring shimmering, dynomic green and purple aurores that ripple and flow across the heavens, against copped countless stors, creating surreal and enchantting atmosphere.
historical
Chouvet-
animal drowings.
What tools techniques were used create ancient rock pillar paintings, and are there any records or examples showing the process?
What equipment do you necommend taking great photos of the Northern Lights at the observation station?
Ancient pillor pointings pigments hemotite, ochre, and charcaal, mixed with binders such as water animal fats. Tools ronged from brushes mode animal hair plant fibers fingers
To copture the Northem Lights, DSLR mirroriess between 800 and 3200, on aperture f/2 8 or wider, and shutter speed of 15 30 Equip yourself with wide-ongle lens, sturdy tripod, nemote shutter release, extra batteries. Dress wannly and bring hand wormers for your batteness.
ISO
Examples
the Cave Altamira Spain and Bhimbetka in India show varied techniques, reflecctigg diverse cultural
environmental contexts.
How have natural forces impocted the preservation of oncient murals on weathered rock pillars? Can you give exomple mural that nature significantly changed?
What wildlife can you spot near the Northern Lights Observation Station, and how would they appear under the curora?
Natural forces including erosion, rainfall, temperature varictions, acid rain, and biological growth like lichen and dograded murais. At Ajonto Caves, Indio, murols show exteensive fading. cracking, and discoloration, with some sectionns nearly vanished under
Near the Norther Lights Observation Station, you might see reindeer grazing. arctic foxes exploning, and perched, The aurora costs mystical glow. enhancing visibility and adding a shimmer to their and fur
and lichen, highlighting severe impact environmental factors.
Figure 6: Examples of multi-turn conversation
Mean Calculation Let Ii,d denote the score given by a judge to the i-th sample generated by generator, under evaluation dimension d, and let N be the total number of samples. Then, for each (judge, generator) pair, the mean score is computed as:
Sa
Dz,d
(5)
The mean score Sa reflects the average performance of a generator, as evaluated by a specific judge under dimension d.
Variance Calculation
To estimate the variability of the scores, we compute the variance:
(xi,d Sa)²
N
The variance σn captures the consistency of the generator's performance across different questions. A higher variance indicates greater inconsistency in quality.
18

---

Published as a conference paper at ICLR 2026
Root Mean Squared Error (RMSE) To measure the agreement between model judge M and human judge H, we compute the RMSE between their respective scores for each sample:
1 N
RMSE,
Fid
Here, "idd and add denote the scores assigned by the model-based and human judge respectively. RMSE quantifies the deviation between model-based judge's scores and those of human judge in dimension Lower RMSE values indicate higher alignment with human preferences, and thus higher reliability of the model-based judge.
Human Agreement within Tolerance (A ()) While RMSE measures the magnitude of error, can be sensitive to outliers. To provide complementary view of judge reliability, we also introduce the Human Agreement within Tolerance (A () metric. This metric calculates the percentage of evaluations where the judge's score falls within a specified tolerance margin, т, of the human score. Given the subjective nature of the scoring task, we set a tolerance of point. The metric, A@I, is calculated as:
A@I
N Σ I I(
< 1)
where T(-) is the indicator function, which is 1 if the condition is true and 0 otherwise. A higher A @@] score indicates that the judge's scores are more frequently in close agreement with human evaluators, reflecting greater practical reliability.
This evaluation framework provides comprehensive analysis of the performance of all the genera- tors across multiple dimensions, ensuring objective comparison from both human and model-base perspectives.
D.3 MODELS USED IN THE SEIR METHOD
To construct the dataset, we adopt Qwen2.5-32B Swen2.5-32B-Instee -nnstuuttt as the language model (M2), In- ternVL2 5-26B as the vision-language model (My), ) and FLUX. I-dev the text-to-image gen- eration model (Mc). These open-source models are used as the default configuration for our SEIR framework. Importantly, the framework is modular by design- each component be substituted with other models, offering flexibility for different deployment environments or research needs.
To assess the generality and robustness of SEIR, we systematically evaluated its performance across a range of model configurations. Specifically, we experimented with different combinations of open- source LMs (InterLLM. DeepSeek-R1, Qwen) and VLMs (Intern QwenVL), while keeping the generative model fixed as Flux. In addition, to benchmark against righ-performing closed-source alternatives, we included a configuration that uses GPT-4o as both the LM and VLM, and DALL-E3 as the generative model.
The results, presented in Table demonstrate several key trends:
SEIR consistently improves data quality. Across all configurations, we observe notice- able improvements in all four evaluation Simensions-TTCS ICC, IQ and -TS-after ap- plying SEIR. The gains are particularly significant in the TCC, ICC and ITS dimensions, reflecting SEIR's ability to enhance the semantic alignment and cooperative informative- ness of multimodal outputs.
Initial quality varies across model combinations. Among the open-source configura- tions, those involving Qwen (e.g.. Qwen+InternVL or QwenVL) generally exhibit stronger performance in the No_SEIR stage. In contrast, IntemLLM+QennV and DeepSeek-R1 Intern VL show relatively weaker initial consistency, suggesting differences in language- vision alignment quality across model families.
19

---

Published as a conference paper at ICLR 2026
SEIR is especially effective for lower-performing combinations. The relative improve- ments are more pronounced for model combinations with lower baseline performance. For example, InternLM + QwenVL improves by 0.60 in TCC and 0.70 in ITS, and DeepSeek- R1+InternVL. shows notable gains across all metrics despite its initially modest perfor- mance.
SEIR narrows the gap between open- and closed-source models. While the GPT- 4o+DALL-E3 configuration achieves the highest initial quality across all metrics, the appli- cation of SEIR allows open-source configurations reach comparable performance levels. For instance, Qwen+InternVL VL + Flux achieves 4.42 (TCC) and 4.51 (ITS) after SEIR, which closely rivals the 4.44 and 4.54 obtained by GPT-40+DALL-E3.
Text- Image Synergy (TTS) shows the largest variance. This dimension benefits the most from SEIR optimization, particularly in cases where image-fext redundancy or disconnec- tion was prevalent before refinement. The improvement indicates SEIR's effectiveness in jointly adjusting both modalities to produce more complementary multimodal answers.
Overall, these results confirm that SEIR is robust and generalizable enhancement framework. It consistently improves dataset quality across a wide range of model backbones, and significantly reduces the reliance expensive closed-soure, models. Consequently, adopt the open-source setup of Qwen+ +L+Flux as our default configuration, balancing quality, flexibility, and cost- effectiveness. During the dataset generation and experimental testing process, this work consumed approximately 90,000 H100 hours.
Table 8: Comparison of dataset quality before and after SEIR optimization using different model configurations.
method
LM
VLM
GM
TCC
ICC
1Q
ITS
No_SEIR
InternLM
Intern
Flux
3.72
3.98
4.30
3.70
SEIR
InternLM
Intern VL
Flux
4.34
4.35
4.38
4.42
SoRR
InternLM
Qwen VL
Flux
3.66
3.89
4.28
3.68
SEIR
InternI.
VI
Flux
4.26
4.32
4.35
4.38
No SEIR
DeepSeek-R
Intern VL
Flux
3.65
3.82
4.37
3.66
SEIR
DeepSeck-RI
Intern VL
Flux
4.20
4.28
4.43
4.19
No_R
Qwen
QwenVL
Flux
3.80
3.93
4.35
3.75
SEIR
Qwen
QW
Flux
4.37
4.40
4.36
4.52
No SEIR
Qwen
Intern VL
Flux
3.85
4.01
4.42
3.79
SEIR
Qwen
InternVL
Flux
4.42
4.47
4.44
4.51
No_SEIR
GPT-4o
GPT-4o
DALL-E3
4.05
4.08
4.41
3.94
SEIR
GPT-4o
GPT-4o
DALL-E3
4.44
4.46
4.43
4.54
D.4 HYPERPARAMETERS USED FOR TRAINING JUDGE MODEL
We fine-tuned two large multimodal models, Intern VL2.5-8B and DwwnVVL2.5-77 as judge models evaluate the interleaved image-text content completeness. We followed common practices for large scale model fine-tuning, applying weight decay regularization, learning rate warmup, and gra- dient clipping to ensure training stability. All experiments were conducted using prexdsore training on distributed GPU clusters.
For the Qwen judge model, we adopted multi-GPU training setup using devices with total training batch size of 8, obtained by setting per-device batch size of 1 and a gradient accumulation step of The model was trained using the AdamW optimizer with 3, = 0.9, 83 = 0.999, and decay strategy with a warmup ratio 10%. fixed random seed of 42 was used for reproducibility. Evaluation was conducted using batch size of 8 per device, resulting in total evaluation batch size of 32.
x 10- The initial learning rate set to 1 x 10 and scheduled using cosine
For the Intern VL2.5-8B referee model, adopted multi-GPU training setup, using 4 devices with a total training batch size of 8. This was achieved by setting the batch size per device to and the gradient accumulation steps to 2. The model trained using the Adam W optimizer, with
20

---

Published as a conference paper at ICLR 2026
10-a, gradient clipping threshold of 1.0, and weight decay of 0.05. The initial learning rate was set to 4 x 10 and a cosine annealing strategy was employed for adjustment, with warm-up ratio of 3%. For reproducibility, fixed random seed of 42 was used. During evaluation, the batch size per device was 8, resulting in a total evaluation batch size of 32.
81 0.9, B2 0.999,
D.5 FURTHER VERIFICATION OF SEIR EFFECTIVENESS
To investigate the impact of the SEIR method, we conduct a fine-grained ablation by varying AR and IR iterations. Results in Sable that both AR and IR contribute positively performance. Specifically, increasing AR iterations mainly improves TCC, ICC, and ITS, while IR iterations fur- ther enhance ICC and ITS. These results confirm that AR enriches textual content and coherence, while IR reinforces visual relevance and multimodal synergy.
Table 9: Ablation study: impact of AR and IR iterations Anole and VILA-U. "null" indicates baseline performance without SEIR- based training.
(a) Performance of Anole.
(b) Performance of VILA-U.
AR
IR
TCC
ICC
IQ
ITS
AR
IR
TCC
ICC
1Q
ITS
null
null
3.09
3.01
2.92
2.26
null
null
2.46
3.72
3.37
2.19
0
3.33
3.17
2.92
2.77
3.07
3.83
3.37
3.13
1
3.37
3.21
3.07
2.71
3.34
2
2
3.41
3.27
3.03
2.79
2
3.17
3.79
3.39
3.27
3
0
3.47
3.20
3.03
2.85
3
0
3.17
3.81
3.36
3.27
3
1
3.51
3.25
3.11
2.93
3
3.17
3.81
3.4
3.3
3
2
3.49
3.3
3.08
2.91
3.21
3.85
3.37
3.29
3
3
3.52
3.32
2.94
3.19
3.83
3.39
3.33
D.6 SYNJUDGE TRAIN/TEST DATA
Our full annotated data contains 48,000 samples. These 48,000 questions were generated by the SEIR method through three iterations, with the question template and topic hierarchy ensuring suf- ficient diversity across topics and conversational query style.
To obtain broad distribution of multimodal crestion-answer (QA) outputs, these 48,000 ques- tions were randomly assigned to different generators, which then produced responses. This strat- egy ensures that the resulting QA pairs cover wide quality spectrum. After generation, all ques- tion-answer pairs were annotated by trained human annotators for TCC, ICC, IQ, and ITS over a two-week period.
The annotated dataset was split into training set (80%) and a test set (20%). The training set was used fine-tune candidate judge models (e.g., Qwen VoweenV__trained, Intern VL__trineed) while the test set was reserved for evaluation. For the evaluation, we used RMSE as the metric to measure how closely the scores from different judges align with ground-truth human annotations: a lower RMSE indicates better judge. This evaluation process is what leads to the selection of our final model, SynJudge.
D.7 DETAILED ANALYSIS OF THE JUDGE'S SCORING
Judge Deviation Analysis. To better understand each judge's scoring behavior, we report the distribution of absolute score differences in Tables 10-14 Each table shows, for a given judge, the proportion of samples where the model's score differs from the human reference by to 5. These detailed distributions provide fine-grained view of the judges' alignment with human evaluators.
Judge Agreement Analysis. Table provides detailed breakdown of the Human Agreement within Tolerance (A@1) for each judge across all evaluated generators and dimensions. As for- mally defined in Appendix $D.2 this metric reflects the proportion of scores where the absolute difference between the model and human judge is no more than one point (r 1), which we con- sider an acceptable margin for subjective tasks.
21

---

Published as a conference paper at ICLR 2026
The results reinforce our findings from the main paper. The finetumed judges consistently achieve higher agreement rates. QwenVL.trrined achieves the highest average A@ of 95.4%. In- VL trained also shows strong performance 94.5% significantly better than the non-finetuned baselines. In contrast, zero-shot models like GPT-4o and Qwen exhibit noticeably lower agreement, around 86.5% and 87.5% respectively, indicating that they less reliable for fine- grained evaluation without specialized tuning. These findings further justify our selection of Qwen trained the backbone for SynJudge.
22

---

Published as a conference paper at ICLR 2026
Table 10: Gap proportion between GPT-4o and human scoring.
Score
Model
Dimension
3
5
TCC
0,463
0.343
0.159
0,031
0.004
0.0
ICC
0.525
0.325
0.106
0.037
0.007
0.0
Anole
IQ
0.52
0,309
0.128
0.036
0.007
0.0
ITS
0.536
0.289
0.111
0.056
0.007
0.001
TCC 0.654
0.318
0.025
0.003
0.0
ICC
0.686
0.276
0.031
0.003
0.002
0.002
GPT-40+DALL-E
IQ 0,636
0.295
0.065
0,002
0.0
0.002
ITS
0.635
0.303
0.039
0.019
0.002
0.002
TCC 0.938
0.024
0.017
0,012
0.009
0.0
ICC 0.467
0.379
0.111
0.037
0.006
0.0
DDIT
IQ 0.434
0.404
0.135
0.023
0.004
0.0
ITS
0.843
0.043
0.023
0.022
0.027
0.042
TCC
0.5
0.398
0.085
0.016
0.001
0.0
ICC
0.535
0.333
0.082
0.037
0.012
0.001
Emu3
IQ 0.546
0.367
0.074
0.012
0.001
0.0
ITS
0.454
0.358
0.123
0.052
0.013
0.0
TCC 0.589
0.329
0.066
0.013
0.002
0.001
ICC
0.686
0.251
0.037
0.017
0.009
0.0
SEIR
TQ 0.665
0.281
0.051
0.001
0.001
0.001
0.577
0.3
0.063
0.037
0.016
0.007
TCC 0.636
0.311
0.037
0.011
0.003
0.002
ICC
0.705
0.237
0.036
0.015
0.007
0.0
Gemini+Flux
IQ 0.695
0.256
0.047
0.002
0.0
0.0
ITS
0.657
0.267
0.045
0.023
0.006
0.002
TCC
0.31
0.419
0.234
0.036
0.001
0.0
ICC
0.412
0.395
0.124
0.06
0.009
0.0
Janus-Pro
IQ 0.428
0.369
0.164
0.034
0.005
0.0
ITS
0.369
0.407
0.131
0.078
0.014
0.001
TCC 0.465
0.389
0.12
0.024
0.002
0.0
ICC
0.513
0.368
0.073
0.039
0.006
0.001
Liquid
IQ 0.478
0.399
0.094
0.025
0.002
0.002
ITs
0.357
0.414
0.14
0.071
0.016
0.002
TCC 0.525
0.343
0.104
0.026
0.002
0.0
ICC
0.475
0.367
0.101
0.043
0.013
0.001
Show-o
IQ 0.445
0.382
0.139
0.029
0.005
0.0
ITS
0.471
0.331
0.114
0.06
0.019
0.005
TCC 0.432
0.34
0.17
0.056
0.002
0.0
ICC
0.53
0.332
0.095
0.032
0.011
0.0
Show-o- Turbo
IQ 0.433
0.383
0.16
0.019
0.005
0.0
ITS
0.457
0.36
0.122
0.044
0.013
0.004
TCC
0.351
0.356
0.235
0,051
0.007
0.0
ICC
0.861
0.064
0.037
0.03
0.005
0.003
VARGPT
IQ 0.875
0.082
0.034
0.003
0.003
0.003
0.864
0.071
0.032
0.024
0.008
0.001
TCC
0.617
0.249
0.105
0.024
0.004
0.001
ICC
0.428
0.405
0.117
0.046
0.004
0.0
VILA-U
IQ 0.403
0.37
0.172
0.047
0.007
0.001
ITS
0.562
0.254
0.1
0.038
0.026
0.02
23

---

Published as a conference paper at ICLR 2026
Table 11: Gap proportion between Intern VL and human scoring.
Score
Model
Dimension
3
5
TCC
0.523
0.294
0.147
0.034
0.001
0.001
ICC
0.574
0.286
0.106
0.03
0.003
0.001
Anole
IQ
0.504
0.272
0.176
0.042
0.005
0.001
ITS
0.592
0.26
0.116
0.022
0.01
0.0
TCC
0.658
0.326
0.015
0.001
0.0
ICC
0.709
0.248
0.037
0.003
0.001
0.002
GPT-40+DALL-E
IQ 0,647
0.269
0,075
0.006
0.0
0.003
ITS
0.653
0.281
0.041
0.021
0.0
0.004
TCC
0.942
0,022
0.02
0.005
0.01
0.001
ICC 0.574
0.327
0.081
0.015
0.003
0.0
DDIT
IQ 0.485
0.347
0.136
0.029
0.003
0.0
ITS
0.927
0.025
0.026
0.01
0.01
0.002
TCC
0.586
0.329
0.075
0.008
0.002
0.0
ICC
0.629
0.27
0.072
0.022
0.006
0.001
Emu3
IQ 0.531
0.328
0.12
0.021
0.0
0.0
ITS
0.496
0.351
0.105
0.044
0.004
0.0
TCC 0.589
0.352
0.048
0.01
0.001
0.0
ICC
0.708
0.239
0.039
0,012
0.001
0.001
SEIR
TQ 0.692
0.246
0.057
0.003
0.001
0.001
0.611
0.271
0.075
0.031
0.009
0.003
TCC 0.629
0.332
0.032
0.004
0.002
0.001
ICC
0.705
0.241
0.045
0.005
0.004
0.0
Gemini+Flux
IQ 0.694
0.248
0.056
0.002
0.0
0.0
ITS
0.633
0.279
0.066
0.018
0.002
0.002
TCC 0.37
0.348
0.235
0.046
0.001
0.0
ICC
0.52
0.327
0.124
0.026
0.003
0.0
Janus-Pro
IQ 0.378
0.31
0.238
0.066
0.008
0.0
ITS
0.471
0.313
0.161
0.049
0.006
0.0
TCC 0.438
0,389
0.139
0.03
0.003
0.001
ICC
0.552
0.339
0.076
0.03
0.003
0.0
Liquid
IQ 0.42
0.401
0.137
0.033
0.006
0.003
ITS
0.433
0.343
0.149
0.059
0.013
0.003
TCC 0.592
0.314
0.074
0.02
0.0
0.0
ICC 0.556
0.319
0.1
0.02
0.004
0.001
Show-o
IQ 0.473
0.308
0.181
0.032
0.006
0.0
ITS
0.49
0.319
0.129
0.05
0.008
0.004
TCC 0.528
0.333
0.118
0.02
0.001
0.0
ICC
0.581
0.316
0.082
0.017
0.004
0.0
Show-o- Turbo
IQ 0.447
0.324
0.199
0.023
0.006
0.001
ITS
0.492
0.347
0.113
0.038
0.007
0.003
TCC
0.271
0.368
0.287
0.062
0.009
0.003
ICC
0.857
0.069
0.057
0.011
0.003
0.003
VARGPT
IQ 0.857
0.083
0.043
0.01 1
0.003
0.003
0.872
0.051
0.055
0.014
0.007
0.001
TCC 0.675
0.238
0.069
0.015
0.002
0.001
ICC
0.553
0.316
0.099
0.028
0.004
0.0
VILA-U
IQ 0.408
0.303
0.205
0.076
0.005
0.003
ITS
0.684
0.195
0.087
0.023
0.009
0.002
24

---

Published as a conference paper at ICLR 2026
Table 12: Gap proportion between Qwen VL and human scoring.
Score
Model
Dimension
3
5
TCC
0.783
0.189
0.027
0.001
0.0
0.0
ICC
0.558
0.258
0.092
0.06
0.022
0.01
Anole
IQ
0.577
0.294
0.102
0.022
0.005
0.0
ITS
0.592
0.189
0.121
0.061
0.032
0.005
TCC
0.679
0.299
0.022
0.0
0.0
0.0
ICC
0.716
0.234
0.041
0.006
0.001
0.002
GPT-40+DALL-E
IQ 0.579
0.325
0,081
0.012
0,003
0.0
ITS
0.571
0.296
0.094
0.029
0.004
0.006
TCC 0.947
0,021
0.013
0.008
0.01
0.001
ICC 0.512
0.339
0.107
0.035
0.007
0.0
DDIT
IQ 0.484
0.317
0.17
0.029
0.0
0.0
ITS
0.939
0.021
0.017
0.008
0.013
0.002
TCC
0.628
0.316
0.05
0.006
0.0
0.0
ICC
0.561
0,312
0.039
0.041
0.031
0.016
Emu3
IQ 0.58
0,345
0.073
0.002
0.0
0.0
ITS
0.543
0.277
0.118
0.043
0.015
0.004
TCC 0.574
0.339
0.076
0.009
0.002
0.0
ICC
0.67
0.256
0.051
0.013
0.007
0.003
SEIR
TQ 0.667
0.264
0.063
0.003
0.001
0.002
0.566
0,283
0.082
0.042
0.017
0.01
TCC 0.607
0.354
0.031
0.004
0.003
0.001
ICC
0.548
0.358
0.04
0.027
0.011
0.016
Gemini+Flux
IQ 0.677
0.254
0.064
0.003
0.002
0.0
ITS
0.613
0.325
0.042
0.012
0.004
0.004
TCC
0,433
0.343
0.196
0.027
0.001
0.0
ICC 0.454
0.29
0.166
0.054
0.025
0.011
Janus-Pro
IQ 0.429
0.35
0.161
0.053
0.006
0.001
ITS
0.502
0.276
0.132
0.059
0.028
0.003
TCC 0.621
0.298
0.064
0.015
0.002
0.0
ICC
0.571
0.311
0.089
0.024
0.004
0.001
Liquid
IQ 0.656
0.283
0.044
0.013
0.004
0.0
ITS
0.478
0.239
0.115
0.067
0.09
0.011
TCC 0.542
0.35
0.088
0.016
0.004
0.0
ICC
0.536
0.316
0.105
0.036
0.004
0.003
Show-o
IQ 0.457
0.364
0.143
0.029
0.006
0.001
ITS
0.571
0.26
0.094
0.051
0.021
0.003
TCC 0.436
0.336
0.187
0.038
0.003
0.0
ICC
0.538
0.311
0.114
0.029
0.007
0.001
Show-o- Turbo
IQ 0.554
0.343
0.078
0.024
0.001
0.0
ITS
0.469
0.306
0.137
0.045
0.032
0.011
TCC
0.707
0.241
0.044
0.006
0,002
0.0
ICC
0.858
0.069
0.046
0.015
0.009
0.003
VARGPT
IQ 0.852
0.073
0.052
0.015
0.005
0.003
0.832
0.054
0.057
0.036
0.018
0.003
TCC 0.823
0.151
0.017
0.005
0.003
0.001
ICC
0.443
0.377
0.149
0.031
0.0
VILA-U
IQ 0.447
0.356
0.15
0.041
0.006
0.0
ITS
0.678
0.165
0.104
0.034
0.013
0.006
25

---

Published as a conference paper at ICLR 2026
Table 13: Gap proportion between Intern VL trained and human scoring.
Score
Model
Dimension
3
5
TCC
0.871
0.106
0.023
0.0
0.0
0.0
ICC
0.738
0.193
0.05
0.012
0.007
0.0
Anole
IQ
0.703
0.206
0.068
0.019
0.003
0.001
ITS
0.773
0.161
0.052
0.013
0.001
0.0
TCC
0.801
0.191
0.008
0.0
0.0
ICC
0.767
0.211
0.019
0.001
0.002
0.0
GPT-40+DALL-E
IQ 0.777
0.205
0,014
0.001
0,002
0.001
ITS
0.778
0.201
0.013
0.004
0.003
0.001
TCC 0.967
0.013
0,012
0.007
0.001
0.0
ICC 0.77
0.186
0.034
0.006
0.004
0.0
DDIT
IQ
0.72
0.215
0.045
0.019
0.001
0.0
ITS
0.962
0.015
0.014
0.007
0.001
0.001
TCC 0.775
0.2
0.021
0.004
0.0
0.0
ICC
0.747
0.196
0.037
0.016
0.004
0.0
Emu3
IQ 0.748
0.2
0.038
0.009
0.001
0.004
ITS
0.741
0.185
0.039
0.031
0.004
0.0
TCC
0.726
0.25
0.017
0.006
0.001
0.0
ICC
0.816
0.159
0.022
0.002
0.0
0.001
SEIR
TQ 0.737
0.241
0.008
0.002
0.002
0.01
0.732
0.233
0.028
0.005
0.001
0.001
TCC 0.786
0.195
0.012
0.004
0.003
ICC
0.728
0.243
0.02
0.007
0.002
0.0
Gemini+Flux
IQ 0.746
0.236
0,016
0.002
0.0
0.0
ITS
0.806
0.142
0.028
0.02
0.001
0.003
TCC 0.855
0.133
0.012
0.0
0.0
0.0
ICC
0.725
0.209
0.057
0.008
0.001
0.0
Janus-Pro
IQ 0.699
0.226
0.052
0.019
0.004
0.0
ITS
0.679
0.219
0.081
0.018
0.001
0.002
TCC 0.846
0.138
0.012
0.004
0.0
ICC
0.796
0.169
0.021
0.012
0.001
0.001
Liquid
IQ 0.716
0.224
0.052
0.006
0.001
0.001
ITS
0.71
0.205
0.066
0.017
0.002
0.0
TCC 0.788
0.176
0.023
0.01
0.003
0.0
ICC 0.743
0.205
0.028
0.019
0.004
0.001
Show-o
IQ 0.718
0.215
0.059
0.006
0.001
0.001
ITS
0.676
0.214
0.082
0.021
0.004
0.003
TCC 0.718
0.213
0.054
0.013
0.002
0.0
ICC
0.684
0.243
0.052
0.015
0.006
0.0
Show-o- Turbo
IQ 0.713
0.242
0.024
0.017
0.004
0.0
ITS
0.715
0.214
0.06
0.01
0.001
0.0
TCC
0.797
0.15
0.047
0.006
0.0
0.0
ICC
0.913
0.04
0.032
0.008
0.004
0.003
VARGPT
IQ 0.922
0.033
0.03
0.01
0.001
0.003
0.917
0.027
0.038
0.016
0.001
0.001
TCC 0.906
0.078
0.012
0.002
0.002
0.0
ICC
0.719
0.208
0.062
0.01
0.001
0.0
VILA-U
IQ 0.702
0.216
0.059
0.022
0.001
0.0
ITS
0.827
0.122
0.044
0.005
0.001
0.001
26

---

Published as a conference paper at ICLR 2026
Table 14: Gap proportion between QwenVL trained and human scoring.
Score
Model
Dimension
3
5
TCC
0.857
0.121
0.022
0.0
0.0
0.0
ICC
0.713
0.225
0.052
0.006
0.004
0.0
Anole
IQ
0.701
0.229
0.05
0.016
0.004
0.0
ITS
0.748
0.189
0.048
0.013
0.001
0.001
TCC
0.783
0.207
0.0
0.0
ICC
0.765
0.209
0.023
0.001
0.001
0.001
GPT-40+DALL-E
IQ 0.757
0.227
0,015
0.0
0,001
0.0
ITS
0.773
0.187
0.029
0.009
0.002
0.0
TCC 0.974
0.015
0.007
0,003
0,001
0.0
ICC
0.732
0.223
0.039
0.005
0.001
0.0
DDIT
IQ 0.733
0.227
0.033
0.004
0.003
0.0
ITS
0.969
0.013
0.013
0.004
0.001
0.0
TCC
0.757
0.22
0.021
0.002
0.0
0.0
ICC
0.729
0.233
0.019
0.019
0.0
0.0
Emu3
IQ 0.749
0.224
0.021
0.005
0.001
0.0
ITS
0.781
0.17
0.04
0.007
0.001
0.001
TCC 0,699
0,279
0.018
0.003
0.001
0.0
ICC
0.659
0.309
0.022
0.01
0.0
0.0
SEIR
TQ 0.704
0.281
0.014
0.001
0.0
0.0
0.707
0.268
0.02
0.004
0.001
0.0
TCC 0.735
0.236
0.022
0.004
0.003
0.0
ICC
0.708
0.262
0.023
0.004
0.003
0.0
Gemini+Flux
IQ 0.712
0.263
0.024
0.001
0.0
0.0
ITS
0.703
0.271
0.015
0.01
0.001
0.0
TCC 0.841
0.147
0.011
0.001
0.0
0.0
ICC
0.672
0.262
0.054
0.012
0.0
0.0
Janus-Pro
IQ 0.667
0.252
0.072
0.007
0.001
0.001
ITS
0.224
0.064
0.01
0.001
0.001
TCC 0.833
0.152
0.011
0.004
0.0
ICC
0.704
0.245
0.03
0.018
0.002
0.001
Liquid
IQ 0.699
0.244
0.044
0.01
0.003
0.0
ITS
0.661
0.269
0.06
0.009
0.001
0.0
TCC 0.774
0.195
0.021
0.01
0.0
0.0
ICC
0.688
0.247
0.043
0.022
0.0
0.0
Show-o
IQ 0.722
0.248
0.023
0.004
0.002
0.001
ITS
0.693
0.244
0.051
0.01
0.001
0.001
TCC 0.69
0.246
0.047
0.016
0.001
0.0
ICC
0.672
0.265
0.039
0.023
0.001
0.0
Show-o- Turbo
IQ 0.624
0.299
0.068
0.008
0.001
0.0
ITS
0.747
0.197
0.045
0.009
0.001
0.001
TCC
0.82
0.159
0.017
0.004
0.0
0.0
ICC
0.913
0.047
0.031
0.009
0.0
0.0
VARGPT
IQ 0.922
0.05
0.027
0.001
0.0
0.0
0.925
0.038
0.019
0.016
0.001
0.001
TCC
0.89
0.097
0.012
0.001
0.0
0.0
ICC
0.639
0.277
0.058
0.025
0.001
0.0
VILA-U
IQ
0.646
0.272
0.072
0.01
0.0
0.0
ITS
0.8
0.127
0.06
0.011
0.002
0.0
27

---

Published as a conference paper at ICLR 2026
Table 15: Evaluation accuracy A@)
comparison across judges
Model
Dim.
GPT-4o
Intern VL
InternVL.ttraned
QwenVZ VL
Qwen Vintriind
TCC
0.805
0.816
0.977
0.972
0.977
ICC
0.848
0.859
0.930
0.816
0.937
Anole
IQ
0.828
0.776
0.908
0.870
0.929
ITS
0.823
0.851
0.933
0.779
0.936
TCC
0.971
0,983
0.991
0.978
0.990
ICC
0.961
0.956
0.977
0,949
0.974
GPT-4++DALI-E
IQ
0.931
0.916
0.981
0.903
0.983
ITS
0.938
0,933
0.978
0.866
0.959
TCC
0.960
0.963
0.979
0.967
0.989
ICC
0,846
0,901
0.955
0.850
0.954
DDIT
IQ
0.837
0.831
0.934
0.801
0.958
ITS
0.885
0.951
0.976
0.958
0.982
TCC
0.896
0.915
0.974
0.944
0.975
ICC
0.867
0.899
0,942
0.871
0.961
Emu3
IQ
0.912
0.858
0.946
0.924
0,972
ITS
0.811
0.846
0.925
0.819
0,950
TCC
0.917
0.940
0.975
0.912
0.978
ICC
0.936
0.946
0.973
0.925
0.967
SEIR
IQ
0.945
0.937
0.976
0.930
0.983
ITS
0.876
0.881
0.964
0.848
0.974
TCC
0.946
0.960
0.980
0.961
0.970
ICC
0.941
0.945
0,970
0.905
0.969
Gemini+Flus
IQ
0.950
0.941
0.981
0.930
0.975
ITS
0.924
0.911
0.946
0.936
0.973
TCC
0.727
0.716
0.987
0.775
0.987
ICC
0,806
0.846
0.932
0.743
0.933
Janus-Pro
IQ
0.796
0.687
0.924
0.778
0.918
ITS
0.775
0.783
0.897
0.777
0.924
TCC
0.853
0.827
0.983
0.918
0.984
ICC
0.880
0.890
0,964
0.881
0.948
Liquid
IQ
0.877
0.820
0.939
0.939
0,942
ITS
0.770
0.775
0.914
0.716
0.929
TCC
0.867
0.905
0.963
0.891
0,969
ICC
0.842
0.874
0.947
0.852
0.935
Show-o
IQ
0.827
0.781
0.932
0.820
0.969
ITS
0.801
0.808
0.889
0.829
0.936
TCC
0.771
0.861
0.929
0.772
0.935
ICC
0.861
0.895
0.926
0.848
0.936
Show-o- Turbo
IQ
0.815
0.770
0.954
0.897
0,922
ITS
0.816
0.838
0.928
0.775
0.943
ТСC
0.706
0.638
0.946
0.948
0.978
ICC
0,924
0,924
0.953
0.926
0.959
VARGPT
IQ
0.956
0.939
0.953
0.924
0.971
ITS
0.934
0.922
0.944
0.884
0.962
TCC
0.865
0.912
0.982
0.973
0.986
ICC
0.832
0.868
0.926
0.818
0.915
VILA-U
IQ
0.772
0.711
0.916
0.802
0.917
ITS
0.815
0.879
0.948
0.842
0.926
Average
A@
0.865
0.866
0.945
0.875
0.954
28

---

Published as a conference paper at ICLR 2026
Mean value
Varience value
TCC
ICC
2
IQ
ITC
Anole
DDIT
Emu3
VILA-U
Liquid
Janus-Pro
Show-o
Show-a Turba
VARGPT
Gemini+Flux
GPT +DALL-E
AURORA
Figure 7: A visualization of mean and variance of different generators
D.8 DETAILED ANALYSIS OF THE CAPABILITIES OF DIFFERENT GENERATORS
Mean and Variance Analysis. Figure presents the mean and variance of evaluation scores across TCC, ICC, IQ. and ITS for different generators. Several important observations emerge from the results. First, DDIT exhibits the lowest mean scores in both TCC (0.38) and ITS (0.37), indicat- ing poor content coverage and weak image-text synergy. Second, VARGPT shows mean scores below 1 across ICC, IQ, and ITS, suggesting significant deficiencies in visual generation capabil- ities and multimodal alignment. Third, VILA-U demonstrates the highest variance in TCC and ITS among all models, implying that its performance is highly unstable across different questions. In contrast, Gemini+Flux and GPT-40+DALL-E achieve mean scores above 4.0 across all evalua- tion dimensions, reflecting generally strong performance. However, their relatively high variance ITS reveals that they still struggle with maintaining image-text consistency and complementarity across samples. Most notably, SEIR method consistently outperforms all other generators across all four dimensions, achieving the highest mean scores while maintaining the lowest variance. This indicates not only superior quality but also high stability and robustness in both textual and visual generation. These findings collectively highlight the importance of both quality and consistency for robust multimodal generation, and demonstrate the synergy of SEIR in constructing high quality, stable datasets.
E QUESTION TEMPLATES AND TOPIC HIERARCHY
In our framework, question templates are designed to model the style of human queries rather than to encode domain knowledge. Specifically, these templates capture the recurrent syntactic and pragmatic structures through which users naturally formulate requests. For example, variations such as "Please recommend some equipment needed for hiking.". ". "Tell me some equipment needed for hiking," "Could you tell me what equipment needed for hiking?" all convey the same derlying intent but differ in their linguistic form. By systematically enumerating such structures, we obtain broad coverage of syntactic patterns for conversational queries (e.g., "can you ... "please ..."), "), while leaving the semantic content to be drawn from a large topic hierarchy. This separation ensures that the linguistic diversity of user queries can be effectively modeled: the question tem- plate specifies how a request is asked, whereas the the 3500-topic hierarchy defines what the request about. The combination of these two components enables our dataset achieve high diversity while faithfully reflecting real-world user interaction styles.
E.1 QUESTION TEMPLATES
Do you know +++? Can you draw picture of it for me?
Do you know what 1ooks 1ike? Please draw image of for me.
29

---

Published as a conference paper at ICLR 2026
I' very interested in Please help describe it and draw portrait of it.
can you imagine what *** looks like? Please draw portrait of it for me. Have you paid attention to ***? Can you tell me something about it?
Besides, can you depict it in painting? What do you think like?
looks like? Can you draw what this place might look
What coes *** look like? Can you draw a sketch of it for ne?
Do you know about ***? Please draw an image of them for me. heard that draw an image it for me.
is very attractive. Can you introduce it to me? Then
need painting of *** now. please help me describe it and draw it. I'm paying lot of attention to *** now, Do you know it? By the way, help me craw a picture to introduce it.
Can you draw a picture of ***? Besides, knowledge about ***?
science Have you over seen the scene of ***? what does ..* usually look like? Please draw a picture for me.
you give
Will there be in the of **? can you draw picture of this seene for me?
What is the most wonderful *** you have ever seen? Please draw scene pi picture of *** for me.
What king of wonders can seen in ***? I' really curious about the scene. Can you show it to me?
Can you imagine ***, with and **, creating a moment of ** and **? Help me draw them.
can you describe *** for me? It would be even better if there is painting.
Please introduce and draw about it. Hey, can you tell me *** draw picture?
E.2 TOPIC HIERARCHY
To provide clearer picture of the dataset's composition, the table 16 below shows the distribu- tion across our 8 primary domains, including the number of categories and fine-grained topics within each. As you can see, the data is well-distributed across diverse topics like Natural Scenery (19.88%), Cultural Scenery (15.87%). and Animals (14.77%), ensuring broad and deep coverage.
Table 16: Data distribution of InterSyn across primary domains. The table details the percentage domain distribution (Domain Dist.), distribution per Category (Dist. per Cat.), number of categories (# Categories), and number of topics for each domain (# Topics), demonstrating broad and deep coverage.
Domain
Domain Dist. (%)
Dist. per Cat. (%)
Categories # Topics
Animals
14.77
1.64
9
517
Plants
10.13
1.27
8
355
Natural Scenery
19.88
2.48
8
695
Cultural Scenery
15.87
1.98
8
555
Objects
10.65
1.33
8
373
Activities
11.20
1.40
8
392
Food
6.71
0.84
8
235
Culture
10.80
1.35
8
378
"Animals";
"Terrestrial Animals"; ["Giant panda", "Snow leopard", "Black bear", "Red panda", "Tibetan antelope", "Argali".... "Marine Animals": ["Blue whale", "Killer whale", "Great white shark", Humpback whale", "polphin", "Octopus"...
30

---

Published as a conference paper at ICLR 2026
"Extinct Animals": ['Dinosaur", "Dodo", "Noolly manmoth", "Saber-toothee tiger", "Pterosaur", "stegosaurus".... "Domesticated Animals": ["Pet cat", "Pet dog", "Pet bird", "Mouse", Ornamental fish", "Cow", "Sheop", "Pig".... ),
"Plants":
"Edible Plants": ("Rice", "Wheat", "Corn", "Sorghum", "Cat", "Buckwheat", "Quinoa", "Millet", "Barley"....l
"Medicinal Plants": I"Ginseng", "Reltberry", "Coptis chinensis", Notoginseng", "Astragalus membranaceus", "Angelica sinensis" "Ornamental Plants": ["Rose", "Tulip", "chrysanthemun", "peony", "Chinese peony", "Lily", "Narcissus", "Hyacinth", "iris".... },
"Natural Scenery":
"Mountains Forests": ["The flag cloud of Mount Everest in the Himalayas "Alpine meadows and wildflowers in the Alps"...],
"Mater Weather"; ("Hawaiian volcanic lava flowing into the sea", "The bluc-comed church in Santorini, Ereece", "A dugout canoe in the lagoon of Tahiti"...
"Deserts & Velcanoes": ["The sharp ridges on the backlit side of sand dunes", "The winding silhouette of a camel caravan"...] "Seasons & landforms": ("Red-crowned cranes dancing in the winter snow in Hokkaice, Japan", "The tulip maze in Keukenhot Gardens in the Netherlands in spring"....
,,
"Cultural Scenery":
"Cities & villages"; ["The mirror-like water surface of the terraced fields in Yuanyang, Yunnan", "The lavender fields in Provence, France "...],
"Religion Religious Sites": ["Devout believers praying front of the Western Wall in Jerusalem", "The play of light and shadow under the done of st...l,
"Heritages Wonders"; ["The giant paintings of the Nazca Lines in Peru seen from above", "The Treasury the end of the Siq in Petra, Jordan", "The Pyrami the Sun and Moon Teotihuacan,...
},
"Objects":
"Household Daily Items": ["Bench", "Chair", "Sofa", "Coffee table", Booksheli", "Kardrobe", "Desk", "Dressing table", "Bed", "Dining tabie", "Dining chair"...ll,
"Military Security"; ["Pistol", "Rifle", "Submachine gun", "Machine gun "Artillery", "Missile", "Tank", "Armored vehicle", "Fighter jet"...l. "Tools Equipment"; ["Fire hydrant", "Wrench", "Screw", "Hammer", Shovel", "Screvdriver", "Tape measure", "Electric drill", "Pliers", "Saw ", "File", "Soldering iron"...],
"Energy Industry": ("Generator", "Solar panel", "Wind turbine", Rydraulic generator", "Battery", "Inverter", "Transformer", "Charging pile", "Oil drum", "Gas cylinder"...ll "Transportation & Communication": ["Bicycle", "Car", "Kotorcycle", Airplane", "Bus", "Train", "Truck", "Ship", "Traftic light", "TricycLe", "Electric scceter"....
},
"Activities";
"Daily Life & Occupations": ("Doctor", "Firefighter", "Farmer", "Teacher "Lawyer", "Craftsman", "Researcher", "Photographer", "Singer", "Dancer "Painter", "Journalist"....
"Emotional &Social Interactions"; ["Hug", "Kiss", "Meet", "Talk", " Lecture", "Study", "Shake hands", "Confort", "Colebrate", "Take group photo", "Quarrci", "Share", "Nave"...].
"Sports & Laber": ["Run", "Play basketball", "Play football", "Play volleyball", "Play badminton", "Play tennis", "Play table tennis", "Jump ", "Ride a bike", "Box", "Wrestle"...
1,
"Food":
31

---

Published as a conference paper at ICLR 2026
"Regional Cuisines": ('Mapo Tofu", "Ramen", "Braised Pork Belly in Soy Sauce", "Scrambled Eggs with Tomatces", "Shredded Pork with Green Peppers ", "Braised Beef with Potatoes"...."
"Baked Goods & Desserts"; ["Caramel Pudding", "Macaron", "Donut", "Cake", "Yogurt", "French Croissant", "Italian Tiramisu", "German Black Forest Cake", *Japanese Wagashi "ichii... "l,
"Processed Foods": ("Snacks", "Canned Food", "Frozen Food", "Biscuits", Chocolate Biscuits", "Ice Cream", "Popcorn", "Potato Chips", "Canned Fish "Frozen Dumplings"
"Beverages"; l"Red Wine", "Chinese Paijiu", "Beer", "Coke", "Cuice", "Tea ", "Milk", "Soda water", "French Champagne", " "italian Hapresso", Japanese Sake", "Korean Nakgeolli"....""
"Pet Food": ["Dog Food", "Cat Foed", "Chew Sticks", "Bones", "Pet Canned Food", "Freeze-Dried Chicken Pieces", "Salmon-Flavored Cat Treats"..."
),
"Culture";
"Material Culture": ("Hanfu (Han Chinese Clothing)", "Qipao (Cheongsam) ", "Kimono", "Indian Sari", "Western Suit", "Wedding Dress", "Tangzhuang ( Tang-stylc Costume)", "Mongolian Rebe"... "Spiritual Culture"; ["The Dragen Totem in Ancient China", "The Phoenix Totem", "The Eagle Totem of the Native Americans", "The Wolf Totem", "The Rainbos Serpent Totem of the Australian Aborigines".... "Behavioral Culture"; |"Traditienal Chinese Wedding", "Restern Church Wedding", "Coming-of-Kge Ceremony", "Crowning Ceremony", "Sacrificial Ceremony", "Japanese Tea Ceremeny Etiquette", "The Hengi (Nose Rubbing) of the Macri People", "The Namaste of India", "The Apprenticeship Ceremony in Thailand", "The Torch Festival Ceremony of the Yi Ethnic Group".... ..
F EVALUATION DIMENSIONS FOR DATASET QUALITY
F.1 QUESTION EVALUATION DIMENSIONS
Reasonableness of Expression: The question statement is smooth, without any grammatical errors, and the words are used accurately and appropriately. For example, "Please introduce the Great Wall to me and also give the Great Wall, and give me a picture" has problem of confused expression. Such questions will affect the model's understanding of the intention and make it difficult to give accurate answer.
picture of the Great Wall" is reasonable expression; while "Tell about
Clarity of Requirements: Clearly indicate that the model is required to provide both text and image responses simultaneously. For instance, "Introduce the appearance characteristies of Notre- Dame de Paris and provide a high-definition frontal picture", which clearly puts forward the dual requirements of text description and image acquisition; if the question is just "What does Notre-Damt de Paris look like", without clearly stating the image requirement, it does not meet the requirements and cannot effectively guide the model to give a comprehensive response.
Focus of the Theme: The question revolves around a single and clear theme and will not jump between multiple unrelated themes. For example, "Introduce the geographical features of Mount Fuji and attach distant view of Mount Fuji",, with the theme focused on Mount Fuji; while "Tell me about Mount Fuji and then talk about the Eiffel Tower, and give two corresponding pictures", which involves two different themes, may lead to unclear logical answers from the model and is not conducive to the standardized construction of the dataset.
Feasibility and Clarity: Based on common sense judgment, the content involved in the question something that the model has the ability to answer through language and images, and there is no way of multiple interpretations, and the model can accurately grasp the questioner's intention. For example, "Describe the living habits of giant pandas and give a picture of a panda eating bamboo", the model can answer based on its existing knowledge reserve and image generation ability, and the intention is clear; however, "Tell me what it's like for a person to take a bath in volcanic magma and give picture", such questions seriously deviate from reality and lack scientific basis. The model can
32

---

Published as a conference paper at ICLR 2026
neither answer based existing knowledge, nor is there real- world reference for image generation, which will lead to absurd and meaningless generation results and greatly reduce the reliability and practicality of the dataset.
Appropriateness of Length: The length of the question is moderate, which not only contains enough key information guide the model to generate high quality answers but also is not too long and complicated for the model to grasp the key points. Generally speaking, short and concise questions are helpful for the model to quickly understand the intention, such as "Introduce the For- bidden City and give panoramic picture"; but being too short may lack sufficient information, such as "Forbidden City, picture": and overly long and cumbersome questions. like "Please introduce in detail the process of the changes of the Forbidden City since its construction in the Ming Dy- nasty through various dynasties, including the evolution of architectural styles, the transformation of functional uses and other aspects, and provide a high-definition panoramic picture that can com- prehensively display the current overall layout of the Forbidden City. At the same time, ensure that the picture contains the main palaces, courtyards, city walls and other iconic elements of the Forbid- den City", may cause confusion when the model processes it. The ideal length can be determined according to practical experience and testing. Usually, about 15 50 words is more appropriate, which can convey the requirements completely and also facilitate the model to process efficiently.
F.2 INTERLEAVED IMAGE-TEXT ANSWER EVALUATION DIMENSIONS
Text Content Completeness (0-5 points): This dimension only focuses on the correspondence between the text response and the question, whether the content precisely matches the user's needs, and whether the information is complete and error-free. It does not consider the output of any other dimensions and evaluation criteria.
0 points: No text appears: point: The text answer has nothing to do with the question; it is completely wrong, completely divorced from the question, and there is no positive response to the text requirement; there is less content but there
truncations and random spitting characters. points: The text answer can only cover a small part of the elements required in the question, and there is large amount of unreasonable content; there is very obvious phenomenon of text truncation that seriously affects the original information; the content is very long or very short, which seriously affects the reading. points: The answer can correspond to key elements, there a small amount of unreasonable content, and there may be omissions of key information; the content is too long or too short, but the information basically corresponds. 4 points: The required elements of the question are basically all corresponding, there is no unreasonable content, there is a omission of key information, or the answer is awkward; the content slightly longer or shorter, but the answer is very correct. 5 points: The content of the answer exactly corresponds to the question, there is no unreasonable content, and the answer is smooth and fluent, with full content.
Image Content Completeness (0-5 points): This dimension only focuses on the correspondence between the image content and the question (considering the content of the picture, the degree to which the image content answers the question). Whether the key parts are retained, and whether there is an obvious lack of objects.
O points: No image appears; 1 point: The content of the image is completely wrong, and no key elements are depicted at all; the image has no connection to the problem, even if the image itself of good quality. 2 points: About half of the key elements required for the problem are missing, and there are a large number of unreasonable elements; the elements in the figure may have some connection to the problem, but it is almost impossible identify what they are. points: Only a small number of key elements required for the problem are missing in the figure, most of the elements can be fully identified, and there are only a few unreasonable content. 4 points: Basically lack the elements required for the problem, and there may be minor flaws in some details. 5 points: All the elements required for the question are completely corresponding, the main body is intact, and the picture content answers the question very well.
Image Quality (0-5 points): This dimension only focuses on the performance of the basic genera- tion technology of the image (do not consider the content of the picture). Whether it clear, whether there are blurred, noisy or out-of-focus areas, truncations or damages (that is, the judgment of image aesthetics and subjective quality).
0 points: No picture; 1 point:
The image is very ugly, and it is almost impossible to identify the image content. 2 points: The image looks ugly, the overall image is blurred but can be barely recog-
33

---

Published as a conference paper at ICLR 2026
nized; 3 points: The image is medium in appearance, and the main elements can be distinguished, but other elements are blurred. points: The image looks good, the picture is relatively clear, and there is no visible blurring phenomenon; 5 points: The image looks good, the details are sharp without blur, and the image quality is very high.
This dimension evaluates the degree of alignment and comple- mentarity between the textual and visual components of a response. It focuses not only on how well the entities scenes described in the text are accurately and completely depicted in the image, but also on whether the text and image together form coherent and mutually supportive answer to the question.
Teat Synergy (0-5 points):
O points: The image and text are completely unrelated. Additionally, if either the image or the text missing (i.e., "null"), the response is assigned points. point: The image and text are minimally related, with only few elements weakly corresponding. The response lacks coherence and fails to effectively address the question. points: Around half of the key elements described in the text are reflected in the image, but significant mismatches remain. The overall synergy poor. 3 points: Most elements between the text and image are consistent, but a few important mismatches or omis- sions in key entities scenes reduce the completeness of the response. 4 points: Nearly all elements between the text and image are consistent, with only minor mismatches in non-critical details. The response answers the question well, but there may be redundancy between the modalities, lim- iting their complementarity. 5 points: The text and image are perfectly aligned, with all described elements accurately and fully presented. The two modalities work together in a complementary way to form complete and informative response without unnecessary duplication.
G ALL PROMPTS USED IN THIS WORK
G.1 PROMPTS USED IN SEIR METHOD
Only a simple example of single-round dialogue generation prompt provided here. The most detailed prompts are given in detail in the open-source code. Detailed prompts can be found in our code.
Here is the prompt for the question generation:
am bui building question-answer dataset. The topic of this dataset item is ((topic)). Your task is to generate a question based on this topic.
The length of the question should not exceed 50 words. Here is the question template: \n/ques_temp).
The new question you generate can refer to the sentence pattern of the question template.
The questicn must meet the follewing detailed requirements:
1. IIcorporatee Image Request Naturally*:: Clearly express the need to generate a picture, but use varied and creative expressions make the request feel natural and human-like. Avoid repetitive phrases like 'naybe generate picture.' Instead, use diverse sentence structures to request the image.
2. **Varied Sentence Structuresw:: Diversify hox questions phrased.
Use different ways of asking, such as open-ended questions, hypothetical
scenarios, or requests for examples.
3. -*Conciseness and clarity*:: Ensure the question is still concise and immediately understandable but withcut sounding repetitive formulaic. Avoid redundant language.
4. TTpoiic Relevance**: Keep the question focused on the given topic (1
topic)), ensuring it remains engaging and meaningful. Avoid weak connections to the topic.
5. **Approachable Tone**: Use conversational, approachable tone that mimics real human interactions. Keep friendly and engaging, avoiding overly formal or robotic expressicns.
6. +exical Simplicity with Creativity**; Use everyday vecabulary with occasional creative language that fits the topic. knsure accessibilits for broad audience while maintaining interest.
34

---

Published as a conference paper at ICLR 2026
7. **Question Value and Inspiraticner] Make the question thought- provoking or creative, capable of inspiring meaningful answers. Avoid overly simple or overly complex questions.
8. ++Image Context*+; Clearly specify what kind of picture is expected, but do creatively.
Output only the generated questien directly. Do not include explanations, instructions, or any extra text.
Here is the prompt get the the suggestions for the question:
am currently construct ing a question-answer dataset. The first step is to imitate human needs and tone based on certain topic and ask a question.
This questien needs to include the requirement for generating textual content and a picture.
The topic is: ((topic)). The following is a question generated based on this Anlid_nnool_giii You need to analyze the quality of this question from human perspective such whether the question is teo wordy?
Is the question sentence pattern net commonly used in human daily communications How well does the question fit the topic?
Does the tone of the question sound human? Are there any uncommon expressions in the sentence?
Is meaningless question? Does the question contain request for generating an image? Is the generated question easy answer? so on. You need to help me provide revisicn suggestions. It would be best if the suggestiens are concise and brief, and not too long.
If you think the original questien is not good in other aspects, you need to help me give modification suggestions.
Only output the modification suggestions in the end, and there is no need to output the modified results.
Your output should conform to this format format) you think the original questien is good enough, you don't need to give improvement suggestions. You only need to output None. Therefore, your final output is either None or the modification suggestiens.
Here the prompt for the question modification:
I am currently constructing a question-answer dataset. \n The following is the original question generated by LLK: Anlooloodggi AnAn However, believe the quality of this question can improved, as doesn't t sound like something people would naturally ask daily communi Catuuniiaiiin.. I have provided some modification suggestions: (mod suggestion). An Please revise the question based these suggestions and the given topic making it sound more natural and human-like." Finally, output only the modificd question without any additional text.
Here the prompt get the answer of the question:
Currently, I'm constructing a question-answa dataset. This is the current question: \n(final_q/\n
Since this question usually contains requirement for textual answer and image generation., but you don't need to generate the actual image. Instead, you should generate an answer and a description of the image according to the question.
To ensure high Image-Text Synergy (ITS), write the answer line so it gives the write the caption line so adds complementary details that the text omits; the two lines must stay tightly aligned, avoid duplication, and together convey more than either cculd alone.
explanation while referencing key visual elements, and
Therefore, your response should include an answer to the question: answer allowed
; and description of the image: caption. And you
35

---

Published as a conference paper at ICLR 2026
output responses like 'I can't generate images.' You need to pretend that you can.
The image description must not exceed 65 words. This last point is very important! You just need to output in two 1ines and there should be no other content. The output content: start the first 1ine with 'answer:', representing the answer; start the second line with 'caption:', representing the caption.
Your answer should be related to the previous content and must not be repetitive.
Here the prompt for the suggestions for answer modification:
Currently, I'm constructio a question-answer dataset.
Here is the question: \n(final_g) An. The question usually includes the requirement for textual answer and image generation.
Then, here is the answer to this question: loloo///ooodaliliilo
The answer is divided into two parts, including the textual answer to the question and image descriptien.
Do you think the combination this answer fully meet the requirements of the question? Are the image description and the answer content consistent and not redundant??
image description can
How is the correlation among the question, the answer and the image descriptien? Does conform the habits of human answering questions? If you were a knowledgeable human expert, how do you think you would answer this question? Would the answer seem too wordy?
Would the overlap between the answer and the image description be too high? Can the image description well summarize picture? you were a nitpicking critic,
you think there are areas for improvement in this question, the answer and the image description?
Would the image description be tec short and not rich enough in content?
Are there any discriminatory elements in the answer and the image description? And on. You can give modification suggestiens based on the above aspects. Or if you think the answer is unreasonable in other aspects, you also need to give your medification suggestiens.
in addition the modi addification suggesti ons need to be divided into two parts: the answer and the image description. And the content needs be concise and condensed, not overly leng.
or if you think the answer and the image description are already perfect, you don't need to put forward improvement suggestions, and just output None.
Therefore, your final output either None the modification suggestions.
Only output the modification suggestions in the end, and there is no need to output the modified results. Your Your output should confern. to this format {json_format]
None.
Here is the prompt for the answer modification:
You are tasked with improving the cutput model output based the suggestion teedback.
Here is the context and what you need to do step by step: \oln \nin Model Output to Modify (old_ac): \n(old_ac) This is the current answer generated by the model. The answer is divided into two parts:\n
'answer': This is the text answer to the question. An ''captionn: This is the image description associated with the answer. An)
Modification Suggestion (mod lro__ac_sugee suuggessggtt
immod_ac_suggestion An This is the suggestion for improving the model's output, including corrections or enhancements to both the 'answer' and 'caption' parts. Anla Your task is to:
According to the provided mod_ec_sugesstion, update the 'answer' and caption" sections ald_ac. In
36

---

Your output should conforn. to this format Published as a conference paper at ICLR 2026
Ensure that the updated 'caption' does not exceed 65 words. An Follow the specified format strictly. An'n
Important: You just need to output in two lines and there should be no other content.
The output content: start the first line with 'answer:', representing the answer; start the second line with 'caption'', representing the caption.
Here is the prompt for the suggestions for caption modification:
Currently,
constructinn ng question-answes dataset.
Here is the question: \n(final_qi This question usually contains a request for generating textual content and a picture.
Then, this is the original answer final_a: {final_a) and the image description old_e: (old[c) generated according to this question, You now need to evaluate the quality of the image description and the image based on the question and the answer. Does the image match the image description?
When proposing revisions, follew the Image-Text Synergy (ITS) principle: suggest changes that make the picture (and its caption) cemplement rather than repeat the fixed textual answer, depict the visual elements the answer references, reduce redundancy irrelevant details, and keep full factual consistency could alone.
that imagettext together convey more than e' ther
How is the degree of correlation between the image description and the content of the answer to the question? Can the image description well summarize the content of the picture?
Are there any unreasonable objects or behaviors in the image? Is the image description clear and not wordy? And so on. You can give modification suggesticns regarding the image description based on the above aspects. Suggestions in other aspects not mentioned above are alse highly encouraged to be put forward.
The revision suggestions you previde need to concise and condensed, and shouldn't be too long.
If you think the image descripticn and the image for this question and answer are already perfect, then you don't need to put forward any suggestions and just output None.
Therefore, your final output only or the modi fication suggesti
Only output the modification suggestions in the end, and there is no need to output the modified iison_formatt None.
Here is the prompt for the caption modification:
Currently, I'm constructing question- answer dataset. The question usually includes Then, this is the image descripticn of the answer: \n(old_o)\n Then I think the quality of the image description is not very high. have previded some modi fication suggestions here: \n(mod___suggestio
requirement for textual answer image generation.
Please regenerate the image description according to these suggestions. The length of the picture description should not exceed 65 words. In the end, you only need to output the medified image description.
G.2 INTERLEAVED IMAGE-TEXT ANSWER EVALUATION PROMPT USED MLLM
Here is the prompt for evaluating the interleaved image-text answer:
You are an experienced, fair and impartial judge. Next, will provide you with a conversation where human interacts with different GPTs on daily topics. In this scenario, the human will pose a text question, and the GPT's response is based on this question. This response usually includes a piece of and image information, but there may be
37

---

Published as a conference paper at ICLR 2026
exceptions where there is only text only image information. Now you need to reasonably rate the respense given by the GPT. <chatbegin> represents the start of the Q&A data, and <chatend> represents the end of the Q&A data. The rating of the response is divided into the following four dimensicns, and you should rate the response fairly and impartially according the criteria of each dimension.
Here are the four dimensions for evaluating the response:
<Interleaved Image-Text Answer Evaluation Dimensions> ***
The content of your output rating must strictly conform to the following
format:
[Text Content Completeness: +; Image Content Completeness: Image
Quality: *; Image-Text Synergy:
your score for different dimensicns, only as a score in (0, 1, 2, 3, 4, 5). You need to strictly conduct the grading.
Here is the data you need to evaluate, and you need to evaluate the quality of the Answer from the above four dimensions (both text and Image may be "null", and the fact that one of them is "null" will not affect the rating of other dimensions.)
H BENCHMARK SAMPLES
The partly benchmark examples obtained after modification based on the questions raised by the participants follows:
Are there fireflies in the forest on a summer night? can you draw a picture of this scene for me?
What could wonderful concert scene be like? Please draw scene of for What kind wonders can be seen the forest on a cold winter night? m very curicus about what that scene would be like. Can you show it to me
Who is the king of the African savanna? Can you draw picture to depict it?
Can you imagine a serene ocean scene with a setting sun and some seagulls creating calm and relaxing mement? Please draw it for me.
Can you describe forest for me? It would be even better if there is painting.
Hey, can you tell me a really terrifying legend and draw vivid picture of it?
Can you describe the scenes in a futuristic music video? If possible, can you quickly draw sketch? I'm really cager to your ideas! Hey! What the daily life of people the military usually like? Also, can you show me what soldier in military uniform looks like? Describe the scene of huge lightning bolt during a storm. Draw picture of this scene.
Can you imagine how the concept of the Tree Life is presented in different religions? Perhaps a painting showing its symbolism would be helpful.
Can you describe what the snowy scene in blizzard is like? want to see such landscape.
Introćuce delicious snack. Describe its appearance, ingredients, what makes it so appealing, Also, draw what it looks like.
Describe the traditional decorations of the Lantern Festival and show me picture of lively lantern display.
Can you describe clear bluc sky? Then draw an image of it.
with vivid orange-red leaves under
What are some interesting behavicrs of cats? Can you show me & picture of a cat marking its territory?
Can you quickly draw picture of the Christ the Redeemer statue in Rio de Janei and share some interesting facts about it?
38

---

Published as a conference paper at ICLR 2026
Eey! Can you describe spring garden scene? really want to hear enouga details, and you need to draw it according the description! Do you know what happoned 100 years ago? Please draw history-related picture! Thank you!
need picture of the age of the dinosaurs now. Do you know about past history? Please draw picture for can you describe what basketball court like? Draw a basketball moment for me.
What is the Lantern Festival like? Can you show some pictures of traditional lanterns?
what kind casual outfit do you think is suitable wearing on relaxed Saturday afternoon? can you draw what it looks like?
Can you describe and perhaps draw picture showing person practicing yoga in tranquil park at sunrise? need landscape picture of the countryside. Please describe it and draw an image for me.
can you draw a picture of an airplane for mo? Also, give no somo popular science knowledge about it.
1 HUMAN ANNOTATION PLATFORM
We develop human annotation platform to evaluate the quality of interleaved image-text responses. Annotators assess each response across four predefined dimensions, focusing on the content and coherence between visual and textual elements. To ensure annotation reliability, rros-vaaliatiin is conducted on high-rated samples. An overview the annotation interface is shown in Figure S
O
antoomeed
1.9
Figure 8: Human annotation platform to evaluate the quality of interleaved image-text responses.
LIMITATIONS OF THIS STUDY
While our work introduces InterSyn, the first large scale, ciirnction-followinn dataset for multi- turn, interleaved image-text dialogues, and proposes SynJudge, a comprehensive automatic evalu- ator emphasizing image-text synergy, several limitations remain that suggest directions for future improvement.
First, although our SEIR framework substantially enhances output quality through multi-stage re- finement, the visual fidelity of generated images is inherently constrained by the upper bounds of current text-to-image models. This may limit the expressiveness and precision of visual responses, particularly for fine-grained specialized topics.
Second, our current dataset is restricted to one image per dialogue turn, which simplifies the mod- eling process but diverges from real-world scenarios where understanding generating multiple images simultancously is often necessary-e.2 comparative reasoning, procedural steps, or spa- tial reasoning tasks. While have experimentally validated the feasibility of multi-image dialogue
39

---

Published as a conference paper at ICLR 2026
generation using alternative synthesis pipelines, such functionality is not yet reflected in the released dataset.
Third, the SynJudge evaluator is currently designed to assess single-image responses. meaning it does not fully capture the additional complexity and multimodal dependencies introduced by multi- image contexts. Extending SynJudge to support multi-image evaluation is a promising future direc- tion.
Finally, although InterSyn spans diverse domains and fine- tie-ggained topics, future work could enhance its coverage of highly structured tasks or multi-modal reasoning chains that involve deeper world knowledge long-term dialogue coherence.
These limitations highlight important opportunities for scaling interleaved image-text datasets and improving evaluators toward more generalizable, high-fidelity multimodal generation systems.
40


Reviews:
[{"review_id": "wLy6Hc5NzV", "reviewer": "Reviewer_FgAL", "summary": "This paper introduces InterSyn, a large-scale, high-quality dataset (1.8M samples, including 50K multi-turn dialogues) designed for instruction-following interleaved image-text generation. The dataset is constructed via a novel Self-Evaluation with Iterative Refinement (SEIR) pipeline, which iteratively improves question, answer, and image quality with automatic feedback loops. To evaluate multimodal generation, the authors also propose SynJudge, an automatic evaluator providing four interpretable metrics: Text Content Completeness (TCC), Image Content Completeness (ICC), Image Quality (IQ), and Image–Text Synergy (ITS). SynJudge demonstrates high correlation (95% agreement within 1 point) with human judgment. Experiments show that InterSyn substantially improves multimodal models’ interleaved generation capabilities. Fine-tuning on as few as 25K samples yields noticeable gains, and scaling to 200K produces further improvements in all metrics, especially ITS. SEIR-generated samples outperform all baselines—including GPT-4o+DALL-E and Gemini+Flux—on both human and automatic evaluation.", "questions": "No other questions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and very easy to follow.\n2. This paper has significant practical contribution. The introduction of InterSyn addresses a key bottleneck in multimodal AI: the lack of large-scale, high-quality, and instructionally rich datasets for interleaved image–text generation.\n3. The proposed data generation pipeline SEIR and the evaluation criteria SynJudge is reliable and useful for future work.\n4. This paper conduct strong experimental validation which proves the efficiency of this dataset.", "weaknesses": "I think this paper is very good and I don't find some obvious weakness from my perspective. But as a dataset work, this paper has limited discussion on ethical considerations. Given the scale of synthetic multimodal data and some data are collected from commercial software, the paper lacks discussion on potential biases, copyright issues, or misuse of synthetic images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xh7FzsMYCK", "reviewer": "Reviewer_miq9", "summary": "This paper introduces InterSyn, a large-scale, high-quality multimodal dataset designed to advance the training of multimodal LLMs for interleaved image-text generation. The primary contribution of this work is the creation of a comprehensive dataset comprising 1.8 million multimodal samples, which are refined using the proposed Self-Evaluation with Iterative Refinement (SEIR) method, thereby ensuring high data quality. This addresses major challenges in current datasets, including limited scale, poor quality control, and low interaction complexity. In addition, the paper proposes SynJudge, a robust evaluation framework that quantitatively assesses image-text synergy, content completeness, and quality across both modalities. Through extensive experiments, the authors demonstrate that InterSyn significantly improves the performance of multimodal LLMs in tasks involving instruction-following and complex, multi-turn dialogues. The results show that even with smaller subsets of the dataset, substantial performance gains can be achieved, underscoring both the scalability and efficiency of the dataset.", "questions": "1. While SynJudge is shown to be reliable, could there be any challenges or biases in how it aligns with human judgment?\n2. Could you provide more details on how the SEIR method performs with different refinement depths？\n3. The paper mentions that the dataset includes multi-turn dialogue data. Compared to single-turn dialogues, how can the effectiveness of multi-turn dialogues be demonstrated?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Interleaved multimodal generation is a promising capability for LLMs, and a few pioneer models can achieve it. An open-source, high-quality, large-scale, and comprehensive dataset and benchmark are still urgent. It is a significant contribution to the community.\n2. The proposed InterSyn is large-scale and high-quality, with 1.8M samples spanning 3.5K topics, and includes 50k multi-turn dialogues.\n3. The proposed SEIR method is well-designed and can ensure high quality of the dataset. The proposed Image-Text Synergy (ITS) score and SynJudge judge model are reliable. These modules have all been verified through human evaluation.\n4. Extensive fine-tuning experiments demonstrate the effectiveness of the dataset across different dimensions. Models fine-tuned on InterSyn show consistent performance gains.", "weaknesses": "1. The SEIR method enhances image generation, but the current capabilities of text-to-image models still limit the visual fidelity. This is not an issue with the method itself, but rather with the models’ ability to generate highly precise and expressive images.\n2. In the synthesis of multi-turn dialogues, there may be some redundancy between dialogue turns. It’s unclear whether the authors specifically controlled for this aspect of quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y2MGxKrxjM", "reviewer": "Reviewer_SzP4", "summary": "This paper introduces InterSyn, a large-scale interleaved multimodal dataset (1.8M samples) that addresses LMMs' limitations in generating tightly interleaved image-text outputs. Using the proposed Self-Evaluation with Iterative Refinement (SEIR) method, the authors ensure high-quality data with rich instructional diversity across a 3500-topic hierarchy. The paper also contributes SynJudge, an evaluation framework that assesses generation capabilities through four metrics: Text Content Completeness, Image Content Completeness, Image Quality, and Image-Text Synergy. Experiments demonstrate that even smaller subsets (25K-50K) of InterSyn yield substantial improvements in model performance, while scaling to larger subsets (100K-200K) brings additional gains, particularly in image-text synergy.", "questions": "1. What is the performance of state-of-the-art models, such as GPT-5 (text) + GPT-Image-1 (image), or Gemini 2.5 Pro (text) + Nano Banana (image)?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The proposed InterSyn dataset has a large scale and is curated with a carefully designed SEIR pipeline.\n2. The experiments cover a wide range of generation models and judges.", "weaknesses": "1. Dataset Quality Limitations: The quality of the dataset is significantly constrained by the FLUX image generation model. A more robust data curation approach would involve collecting and filtering high-quality images from the Internet based on the generated questions and answers, rather than relying solely on synthetic image generation.\n\n2. Clarity and Organization Issues:\n  - The authors use about 1.5 pages to describe the SEIR with 13 formulations, despite it being a relatively straightforward process that could have been clearly illustrated in Figure 2. Unfortunately, Figure 2 is exceedingly difficult to interpret, featuring unexplained notations, complex organization, and lacking essential text labels.\n  - Despite the excessive length of section 3.3, critical information is absent. The authors present SynJudge as a key contribution, yet fail to clearly define it. The term first reappears after Section 2 in Line 309 (\"We fine-tune four generators... And evaluate with SynJudge\"). It remains unclear whether SynJudge refers specifically to the Qwen-trained judge or also encompasses the four metrics (TCC, ICC, IQ, and ITS). Furthermore, details regarding these four metrics are omitted from the main text, compromising the paper's self-consistency. Essential benchmark information, such as sample size, is also entirely missing.\n\n3. Problematic Experimental Results and Conclusions:\n  - The finetuned models are evaluated exclusively on the proposed dataset (with minimal results in Table 2). It is unsurprising at all that finetuning on a small subset of this dataset can lead to performance improvement. A critical question remains: Can fine-tuning on InterSyn generalize to other related or out-of-distribution benchmarks?\n  - Table 2: The authors claim that \"the substantial gains in interleaved generation capabilities do not come at the cost of core understanding performance,\" yet the evidence suggests otherwise. Performance degradation on MMBench and MMMU (-1.5 to -2.5) appears evident.\n  - Table 3: The assertion that multi-turn data improves performance is not well-supported by the results. For \"single25k + multi25k,\" both models' scores are lower than or comparable to those with \"single50k.\" Moreover, \"multi50k\" scores are even lower than \"single25k + multi25k\", contradicting the authors' claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8IacP7dcCb", "reviewer": "Reviewer_pmWK", "summary": "This paper proposes InterSyn, a large-scale dataset (1.8M single-turn, 50k multi-turn samples) and a Self-Evaluation with Iterative Refinement (SEIR) pipeline for generating high-quality interleaved image–text dialogues. It also introduces SynJudge, an automatic evaluator providing four interpretable scores:\nText Content Completeness (TCC)\nImage Content Completeness (ICC)\nImage Quality (IQ)\nImage–Text Synergy (ITS)\nExperiments show that models fine-tuned on subsets of InterSyn (25k–200k) exhibit consistent gains in instruction-following and synergy metrics without degrading general multimodal understanding benchmarks. SynJudge is validated against human ratings, showing strong correlation and reliability.", "questions": "1.How is convergence or stability of the SEIR refinement process defined and verified? Does SEIR always improve data quality, or could iterative self-consistency cause overfitting or redundancy?\n2.What evidence supports the “Markovian” assumption in SEIR—have you analyzed long-range contextual dependencies? How is topic consistency or contextual inheritance quantitatively measured?\n3.How do you ensure that SynJudge’s fine-tuning on human data does not cause data leakage? Given the small human set, how well does SynJudge generalize to unseen domains?\n4.Are confidence intervals or inter-annotator agreement (e.g., Krippendorff’s α) reported for human validation?\n5.What human-in-the-loop or bias-filtering mechanisms are used to control cultural or demographic bias in InterSyn?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "①Timely contribution addressing a real bottleneck. The lack of high-quality, instruction-following, interleaved image–text data is a genuine gap. InterSyn fills this niche with scale and structure.\n②Systematic dataset generation method. SEIR introduces an iterative, self-corrective data generation framework (Question → Answer → Image refinement), demonstrating consistent quality improvements.\n③Comprehensive evaluation framework. SynJudge formalizes multi-dimensional metrics beyond text-only or image-only correctness, introducing “synergy” as a critical dimension.\n④Strong empirical coverage. Experiments span multiple model types (non-interleaved and interleaved generators), multiple data scales, and both human and automatic evaluations.\n⑤High reproducibility commitment. The paper provides reproducibility details and plans to release data and evaluation tools, which is rare and commendable.", "weaknesses": "(A) Theoretical clarity and rigor of SEIR\nThe SEIR process (Question Refinement → Answer Refinement → Image Refinement) is well-described but lacks formal definition of convergence or stability. It’s unclear whether the refinement iterations always improve data quality or sometimes overfit to self-consistency artifacts. The “Markovian” assumption is stated but not justified—this ignores potential long-range dependencies across turns. The method claims to “ensure topic consistency and contextual inheritance,” yet no quantitative metrics for coherence or redundancy are provided.\n\n(B) SynJudge validation may not prove true generalization\nThe human-alignment test (RMSE) is based on fine-tuning with human-annotated data, which might lead to data leakage between evaluator and benchmark distributions. The human set (9.6k test / 38.4k train) is relatively small compared to 1.8M total data, raising questions about generalization across unseen content domains. “95.4% ” sounds impressive, but no confidence intervals or inter-annotator agreement scores (e.g., Krippendorff’s alpha) are reported.\n\n(C) Data authenticity and bias control\nInterSyn heavily relies on AI-generated data with human-like templates. However, no explicit human-in-the-loop verification or bias filtering is described beyond early-stage question curation. SEIR may propagate LLM biases (e.g., Western imagery, gender stereotypes, or cultural homogeneity) through iterative reinforcement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)