Why Settle for One? Text-to-ImageSet Generation and Evaluation

## Abstract

Abstract Despite remarkable progress in Text-to-Image models, many real-world applications require generating coherent image sets with diverse consistency requirements. Existing consistent methods often focus on a specific domain with specific aspects of consistency, which significantly constrains their generalizability to broader applications. In this paper, we propose a more challenging problem, Text-to-ImageSet (T2IS) generation, which aims to generate sets of images that meet various consistency requirements based on user instructions. To systematically study this problem, we first introduce T2IS-Bench with 596 diverse instructions across 26 subcategories, providing comprehensive coverage for T2IS generation. Building on this, we propose T2IS-Eval , an evaluation framework that transforms user instructions into multifaceted assessment criteria and employs effective evaluators to adaptively assess consistency fulfillment between criteria and generated sets.
Subsequently, we propose AutoT2IS , a training-free framework that maximally leverages pretrained Diffusion Transformers’ in-context capabilities to harmonize visual elements to satisfy both image-level prompt alignment and set-level visual consistency.
Extensive experiments on T2IS-Bench reveal that diverse consistency challenges all existing methods, while our AutoT2IS significantly outperforms current generalized and even specialized approaches. Our method also demonstrates the ability to enable numerous underexplored real-world applications, confirming its substantial practical value. Visit our project in https://chengyou-jia.github.io/T2IS-Home .

## 1 Introduction

Recent advancements in Text-to-Image (T2I) models Esser et al. (2021); Rombach et al. (2022); Saharia et al. (2022); Betker et al. (2023); Baldridge et al. (2024); Labs (2024); OpenAI (2025) have substantially enhanced the ability to generate visually compelling and semantically faithful images.
However, as illustrated in Figure [1](https://arxiv.org/html/2506.23275v2#S0.F1), most practical scenarios, *e.g.*, product design, process illustrations, or character creation, often require not a single image, but a coherent image set. Such image set must maintain set-level visual consistency with varying degrees of identity preservation Huang et al. (2024a), uniform style Hertz et al. (2024), and logical coherence Song et al. (2025).
This novel requirement poses huge challenges for current T2I models, which are primarily focused on image-level prompt alignment rather than set-level visual consistency.

Previous attempts addressing this coherence have largely concentrated on specific facets of consistency Li et al. (2024a); Ye et al. (2023); Song et al. (2025) or specific combinations of consistency types Mao et al. (2024); Chen et al. (2024). For instance, consistent character generation approaches Tewel et al. (2024); Liu et al. (2025); Li et al. (2024a) aim to preserve character identity across various contexts, while methods such as StoryDiffusion Zhou et al. (2024b) are tailored for generating stylistically coherent comic sequences with localized identity preservation.
Despite promising progress within specific domains, they heavily rely on specialized data, techniques, and model architectures, suffering from additional resource and model design costs.
More importantly, their focus on such specific aspects of visual coherence significantly limits their generalizability to a wider variety of applications.
To this end, we propose a more challenging problem, Text-to-ImageSet (T2IS) generation, which aims to receive diverse user instructions and generate image sets that satisfy multifaceted consistency requirements. We address this challenge through two key contributions:

(1) T2IS Benchmark: We first present T2IS-Bench, the pioneering benchmark crafted specifically for T2IS generation. Derived from a comprehensive collection of real-world requirements, T2IS-Bench encompasses 596 representative tasks distributed across 26 detailed subcategories.
Figure [1](https://arxiv.org/html/2506.23275v2#S0.F1) showcases illustrative examples of diverse tasks. Based on this, we introduce T2IS-Eval, an evaluation framework that automatically transforms user instructions into multifaceted assessment criteria across three aspects of consistency: identity, style, and logic. Our framework then leverages the strong multi-image recognition capabilities of large-scale models Bai et al. (2025) to serve as effective consistency evaluators and obtain logits of "Yes-or-No" answers for each criterion to assess the consistency fulfillment. More importantly, unlike previous methods that focus on a single aspect of consistency Hessel et al. (2021); Fu et al. (2023), T2IS-Eval leverages transformed criteria to enable adaptive and interpretable consistency evaluation.

(2) T2IS Generation: We propose AutoT2IS, a training-free framework that maximally leverages the remarkable in-context generation capabilities of pretrained Diffusion Transformers (DiTs) Labs (2024); Huang et al. (2024c). Specifically, AutoT2IS first employs a structured recaptioning approach to systematically parse user instructions into informative prompts for both individual images and the global image set. Subsequently, we introduce a novel set-aware generation with the divide-and-conquer strategy: first binding individual images’ prompts to their respective independent latents during early denoising stages to establish unique content characteristics, then integrating these latents through the multi-modal attention mechanism with a global consistency prompt. This enables each visual latent to simultaneously attend to its individual prompt, the global consistency prompt, and other images’ visual latents. This way dynamically harmonizes visual elements to satisfy both image-level prompt alignment and set-level visual consistency, effectively bridging the gap between isolated image generation and set-level consistency without requiring specialized fine-tuning.

We conduct comprehensive experiments on the proposed T2IS-Bench to evaluate various approaches, including unified models (*e.g.*, Show-o Xie et al. (2024), Janus-Pro Chen et al. (2025)), compositional frameworks (*e.g.*, Gemini+Flux Labs (2024)), and agentic frameworks (*e.g.*, ISG-Agent Chen et al. (2024), ChatDiT Huang et al. (2024d)). Our evaluations indicate that while current methods have achieved good performance on prompt alignment, there remains a significant gap in achieving set-level consistency, highlighting the value of this underexplored research direction. Furthermore, both quantitative and qualitative results demonstrate AutoT2IS substantially outperforms existing methods, achieving improvements across all aspects of consistency. AutoT2IS also achieves competitive or even superior performance in specific domains compared to specialized methods.

Interestingly, during the development of this work, advanced commercial image generation models like Gemini 2.0 Google Cloud (2025) and GPT-4o OpenAI (2025) were released.
This positions T2IS-Bench as the first platform to systematically evaluate and analyze their T2IS capabilities.
Our assessments reveal that these models excel in consistency but often sacrifice image quality and prompt alignment.
To this end, as shown in the bottom of Figure [1](https://arxiv.org/html/2506.23275v2#S0.F1), AutoT2IS can be seamlessly integrated with models like GPT-4o to further boost performance. By offering a transparent benchmark alongside a robust baseline, our work lays a strong foundation for the advancement of T2IS research.

## 2 Related Work

Text-to-Image Generation. With the advancement of diffusion models Ho et al. (2020); Song et al. (2020), Text-to-Image (T2I) models Ramesh et al. (2021; 2022); Esser et al. (2021); Rombach et al. (2022); Saharia et al. (2022); Betker et al. (2023); Podell et al. (2023); Esser et al. (2024); Baldridge et al. (2024); Labs (2024); OpenAI (2025); Jia et al. (2025) have demonstrated exceptional capabilities in generating high-quality images that accurately align with textual descriptions. Early approaches, such as DALL-E 2 Ramesh et al. (2022) and Stable Diffusion Podell et al. (2023), employed Latent Diffusion Models Rombach et al. (2022) (LDMs) with U-Net backbones Ronneberger et al. (2015) for efficient denoising in compressed latent space. Recent developments have advanced the field through the integration of transformer architectures into diffusion models, known as Diffusion Transformers (DiTs) Peebles & Xie (2023); Esser et al. (2024); Labs (2024). These models leverage global attention mechanisms to capture complex dependencies in data, resulting in improved scalability and image fidelity. However, existing methods primarily focus on generating individual images, with limited exploration into the generation of coherent image sets. In this paper, we investigate and harness the in-context generation capabilities of DiTs to maximize their potential for addressing T2IS tasks.

Consistent Text-to-Image Generation. Existing methods related to T2IS generation span various domains, primarily
including consistent character generation Li et al. (2024a); Tewel et al. (2024); Liu et al. (2025) and
storytelling Mao et al. (2024); Zhou et al. (2024b). In character generation, training-based approaches such as
PhotoMaker Li et al. (2024a) aim to preserve identity via parameter-efficient fine-tuning, while training-free methods
like One-Prompt-One-Story Liu et al. (2025) exploit the contextual consistency of language models without
additional training. In contrast to character generation, storytelling further demands style coherence across images. Make-a-Story Rahman et al. (2023) incorporates a visual memory module to capture contextual cues, and
StoryDiffusion Zhou et al. (2024b) introduces Consistent Self-Attention to enhance both identity and style
consistency.
However, when tackling new tasks like process generation Song et al. (2025) that require strong logical consistency for generating drawing processes, previous methods often fall short and need customized task-specific fine-tuning Song et al. (2025). This suggests that consistencies in existing methods are generally domain-specific and lack generalizability. In this work, we propose T2IS generation to address a broad range of consistency requirements.

## 3 Methodology

Our goal is to develop T2IS methods that aim to generate coherent image sets from diverse user
instructions. To achieve this objective, we first provide a comprehensive T2IS benchmark with an evaluation framework for diverse tasks in Sec. [3.1](https://arxiv.org/html/2506.23275v2#S3.SS1). In Sec. [4](https://arxiv.org/html/2506.23275v2#S3.F4), we introduce our generation method, AutoT2IS, which consists of Structured Recaption and Set-Aware Generation to generate image sets that satisfy both image-level prompt alignment and set-level visual consistency.

### 3.1 T2IS Benchmark

T2IS-Bench. We first construct a comprehensive benchmark that reflects real-world demands on diverse consistency, as illustrated in Figure [2](https://arxiv.org/html/2506.23275v2#S3.F2). The construction follows a multi-stage process:

First, we conduct an extensive analysis of real-world T2IS applications and categorize these into five major groups, covering diverse consistency requirements. For instance, Character Generation primarily focuses on identity preservation, while Process Generation emphasizes logical consistency. It is noteworthy that falling within the same groups does not mean the requriements of consistency are exactly the same. Rather, categories within each group vary in the extent to which they integrate and balance additional dimensions of consistency.

Second, we collect diverse data from two complementary sources. We incorporate established benchmarks including Consistency+ Liu et al. (2025), ISG-Bench Chen et al. (2024), IDEA-Bench Liang et al. (2024) and others Song et al. (2025); Huang et al. (2024e; c). Additionally, we curate examples from real-world platforms, such as e-commerce websites (*e.g.*, Amazon, Taobao), image generation communities (*e.g.*, Civitai), social media platforms (*e.g.*, RedNote), and other sources. From academic benchmarks, we select representative examples based on their diversity, while from platforms, we prioritize examples with user engagement and popularity.

Third, we standardize all these examples by employing MLLMs such as GPT-4o to transform these into conversational user instructions. We further augment the dataset through in-context learning techniques Dong et al. (2022) to ensure balanced samples across all subcategories. All samples undergo rigorous verification by PhD-level domain experts, with duplicate or highly similar instances eliminated through a combination of automated similarity detection using BertScore Zhang* et al. (2020) and manual review.

Through the above process, we compile T2IS-Bench, containing 596 high-quality user instructions distributed across 26 distinct subcategories. Our benchmark offers the most comprehensive coverage to date, establishing a solid foundation for advancing T2IS. Detailed information regarding category definitions, representative examples, and dataset distribution statistics is provided in Appendix [A](https://arxiv.org/html/2506.23275v2#A1).

Figure: Figure 2: Illustration of the proposed T2IS-Bench. Upper Left: The collection process. Lower Left: Task examples displays. Right: Categories with their corresponding quantities.
Refer to caption: x2.png

T2IS-Eval. Although previous methods Hu et al. (2023); Wiles et al. (2024); Hosseini et al. (2025) have adopted a question-answering paradigm to perform fine-grained dynamic evaluation, their criteria definition and execution are not suitable for T2IS tasks. Therefore, we propose T2IS-Eval, a comprehensive evaluation framework that assesses image sets across three critical consistency dimensions: identity, style, and logic. For each dimension, we employ LLMs to generate targeted evaluation questions as assessment criteria that reflect the consistency requirements corresponding to the user instructions. As illustrated in Figure [3](https://arxiv.org/html/2506.23275v2#S3.F3), the criteria across different dimensions include:

- •
Identity criteria measure the consistency of identity preservation across images. This includes whether specific entities (*e.g.*, characters, objects) maintain consistent appearance features (*e.g.*, size, color, shape), and whether their emotional expressions or key identifying elements persist.
- •
Style criteria assess the style uniform across images, including consistency in illustration style (*e.g.*, watercolor, cartoon, 3D rendering), harmony in color palette usage, and others.
- •
Logic criteria evaluate whether the image set maintains reasonable causal relationships and narrative coherence. This includes environmental consistency across scenes, proper depiction of cause-and-effect relationships, and logical alignment between actions and their consequences.

Then, we systematically assess how well image sets satisfy these criteria. Inspired by VQAscore Lin et al. (2024), we discovered that large-scale MLLM Bai et al. (2025) with strong multi-image recognition capabilities can serve as excellent consistency judges. Based on this insight, we formulate our evaluation criteria as questions that are sequentially applied to image pairs, obtaining logit scores for "Yes-or-No" answers that quantify the degree of consistency fulfillment. In Appendix [B](https://arxiv.org/html/2506.23275v2#A2), we demonstrate the effectiveness of this approach and its advantages over direct scoring by MLLMs. Overall, unlike previous methods that focus on a single consistency dimension Fu et al. (2023), our approach leverages transformed criteria to enables adaptive consistency evaluation with fine-grained measurements across multiple dimensions.

In addition to consistency evaluation, we also incorporate established methods to comprehensively assess generation quality. Specifically, we employ MPS Zhang et al. (2024) to evaluate the aesthetic quality of generated images, and VQAscore Lin et al. (2024) to measure the alignment between each image and its corresponding instruction. Notably, all three evaluation components (aesthetics, prompt alignment, and visual consistency) leverage vision-language models and employ a unified scoring mechanism based on next-token prediction logits or text-image matching logits. Consequently, these quantitative metrics are normalized to a consistent range $[0,1]$, providing better interpretability and holistic assessment.

Figure: Figure 3: Overview of our T2IS-Eval. For each dimension, LLMs are prompted to generate 2-4 criteria; we only display a subset. Highlighted portions show the core elements for evaluation.
Refer to caption: x3.png

### 3.2 T2IS Generation

In this section, we introduce AutoT2IS, a training-free framework for T2IS generation. We aim to maximally leverage the remarkable in-context generation capabilities Huang et al. (2024c) of pretrained Diffusion Transformers (DiTs Peebles & Xie (2023); Labs (2024)) to handle diverse T2IS tasks. We delineate two key steps, as depicted in Figure [4](https://arxiv.org/html/2506.23275v2#S3.F4). Specifically, AutoT2IS first utilizes a structured recaption to parse user instructions into informative prompts for both individual images and the complete image set, thereby capturing richer semantic information.
Subsequently, we introduce set-aware generation that effectively applies the recaptioned individual prompts and consistency requirements to the entire image set, transforming semantic contents into visually consistent outputs.

Structured Recaption. Let $y$ denote the given user instruction, which contains requirements for individual images and overall consistency. We first employ LLMs to identify key content for each image and consistency requirements for the entire set. The recognized elements are represented in a structured format $S=\{E,C\}$, where $E=\{e_{1},e_{2},...,e_{n}\}$ represents the content for $n$ individual images, and $C=\{c_{1},c_{2},...,c_{m}\}$ denotes the $m$ consistency requirments extracted from $y$.

Based on the structured information $S$ and original user instruction $y$, we generate enhanced prompts for each image and global consistency requirements:

$$ $\displaystyle p_{i}=f_{\text{recap}}(e_{i},y)\;\forall i\in\{1,2,.,n\},\quad g=f_{\text{consist}}(C,y)$ (1) $$

where $p_{i}$ represents the detailed prompt for the $i$-th image with denser fine-grained details to improve fidelity, and $g$ represents the global consistency description that ensures the model adheres to overarching principles across all images. Functions $f_{\text{recap}}$ and $f_{\text{consist}}$ are implemented using LLMs to expand the structured information into comprehensive textual descriptions.

Set-Aware Generation. While DiT models exhibit powerful in-context capabilities, existing approaches Huang et al. (2024c; b); Tan et al. (2024) often require fine-tuning to activate these abilities, which compromises the model’s original generalization performance. As shown in Figure [4](https://arxiv.org/html/2506.23275v2#S3.F4), we introduce set-aware generation that enables consistency requirements to be expressed through textual prompts and multi-modal attention in a training-free way.

We empirically find that these few binding steps are sufficient to establish the unique content of each image, allowing subsequent steps to focus on resolving consistency issues across the image set. Inspired by this, we design a divide-and-conquer strategy to separate DiT’s native denoising capabilities into two distinct phases. In the divide step, which is applied during the early stages of the denoising process with a total timestep $t$, we generate independent noise latents $x_{t}^{i}$ for each image $i\in\{1,2,...,n\}$ and denoise them separately with their corresponding prompts $p_{i}$. This binding process is executed only during the initial $r$ steps of the denoising process.

In the conquer phase, we concatenate all independently denoised latents $x_{t-r}^{i}$ into a grid layout $X_{t-r}=[x_{t-r}^{1},x_{t-r}^{2},...,x_{t-r}^{n}]$. This concatenation enables previously isolated regions to collaborate through DiT’s multi-modal attention mechanism Peebles & Xie (2023); Esser et al. (2024). Formally, for each denoising step $t^{\prime}\in\{t-r,t-r-1,...,0\}$, the attention computation can be expressed as:

$$ $\text{Attention}(Q,K,V,M)=\text{softmax}\left(\frac{QK^{T}+M}{\sqrt{d_{k}}}\right)V$ (2) $$

where $Q=W_{Q}X_{t^{\prime}}$ represents queries derived from the concatenated latent $X_{t^{\prime}}$, $K=[W_{K}^{\text{text}}p;W_{K}^{\text{text}}g;W_{K}^{\text{image}}X_{t^{\prime}}]$ and $V=[W_{V}^{\text{text}}p;W_{V}^{\text{text}}g;W_{V}^{\text{image}}X_{t^{\prime}}]$ index keys and values from both the prompt embeddings $P=[p_{1},p_{2},...,p_{n},g]$ and the latent representation itself. The mask matrix $M\in\mathbb{R}^{N\times(N_{p}+N_{g}+N)}$ is defined as:

$$ $M_{i,j}=\begin{cases}0,&\text{if }i\in x^{k}_{t^{\prime}}\text{ and }j\in\{p_{k},g,X_{t^{\prime}}\}\\ -\infty,&\text{otherwise}\end{cases},\quad\text{where }k\in\{1,2,\dots,n\}.$ (3) $$

This mask ensures that each visual latent from a specific region $x^{k}_{t^{\prime}}$ to simultaneously attend to three critical elements, as illustrated in Figure [4](https://arxiv.org/html/2506.23275v2#S3.F4):

- •
Its corresponding individual prompt $p_{i}$ for precise semantic content preservation;
- •
The global consistency prompt $g$ for harmonious coherence across the image set;
- •
Visual latents in other images for enhanced cross-image alignment and contextual integration.

Through this enhanced multi-modal attention mechanism, the model dynamically harmonizes visual elements to satisfy both local alignment and global consistency requirements, effectively bridging the gap between isolated image generation and set-level consistency without specialized fine-tuning.

Figure: Figure 4: The pipeline of our AutoT2IS, consists of structured recaption and set-aware generation.
Refer to caption: x4.png

## 4 Experiments

Implementation Details. We utilize DeepSeek-R1 DeepSeek-AI (2025) as the foundational LLM for criteria generation and structured recaptioning. Qwen2.5-VL-7B Bai et al. (2025) serves as our consistency evaluator. For Set-Aware generation, we employ FLUX.1-dev Labs (2024) as the base model, executing a total of 20 denoising steps, i.e., 2 steps for the divide phase and the remaining for the conquer phase. For the concatenated latent, we apply positional encoding at new positions to preserve the original multi-modal attention structure.

Due to space limitations, we present the most significant experimental results in the following, including quantitative comparison of both generalized and specialized methods with ours on T2IS-Bench (Sec. [4.1](https://arxiv.org/html/2506.23275v2#S4.SS1)), qualitative visual comparisons to demonstrate the broad applications of our method (Sec. [4.2](https://arxiv.org/html/2506.23275v2#S4.SS2)), and ablation studies on core components (Sec. [4.3](https://arxiv.org/html/2506.23275v2#S4.SS3)).
Please refer to the appendix for more comprehensive information. We provide detailed experimental settings in App.[C](https://arxiv.org/html/2506.23275v2#A3), reliability analysis of T2IS-Eval in App.[D](https://arxiv.org/html/2506.23275v2#A4), failure cases in App.[F](https://arxiv.org/html/2506.23275v2#A6), limitations and future work in App.[E](https://arxiv.org/html/2506.23275v2#A5).

**Table 1: Comparisons with various generalized generation models on T2IS-Bench. The upper shows open-source models, while the lower presents commercial models. The best and second-best results among the open-source models are highlighted in bold and underlined, respectively.**
| Model | Aesthetics | ˜ Prompt Alignment | Visual Consistency | Avg. |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Entity | Attribute | Relation | Identity | Style | Logic |  |  |  |
| Anole Chern et al. (2024) | 0.170 | 0.534 | 0.611 | 0.570 | 0.115 | 0.148 | 0.090 | 0.264 |
| Show-o Xie et al. (2024) | 0.206 | 0.780 | 0.785 | 0.776 | 0.233 | 0.287 | 0.285 | 0.409 |
| Janus-Pro Chen et al. (2025) | 0.333 | 0.787 | 0.785 | 0.781 | 0.224 | 0.272 | 0.300 | 0.435 |
| Gemini & SD3 Esser et al. (2024) | 0.500 | 0.796 | 0.794 | 0.789 | 0.287 | 0.244 | 0.320 | 0.480 |
| Gemini & Pixart Chen et al. (2023) | 0.447 | 0.743 | 0.765 | 0.747 | 0.206 | 0.279 | 0.268 | 0.440 |
| Gemini & Hunyuan Li et al. (2024b) | 0.410 | 0.758 | 0.774 | 0.765 | 0.197 | 0.276 | 0.271 | 0.436 |
| Gemini & Flux-1 Labs (2024) | 0.533 | 0.791 | 0.790 | 0.786 | 0.249 | 0.302 | 0.328 | 0.490 |
| ChatDit Huang et al. (2024d) | 0.414 | 0.717 | 0.726 | 0.726 | 0.296 | 0.326 | 0.310 | 0.455 |
| ISG-Agent Chen et al. (2024) | 0.256 | 0.667 | 0.703 | 0.682 | 0.146 | 0.178 | 0.163 | 0.338 |
| AutoT2IS (ours) | 0.520 | 0.729 | 0.756 | 0.743 | 0.359 | 0.414 | 0.356 | 0.515 |
| Gemini 2.0 Flash Google Cloud (2025) | 0.430 | 0.738 | 0.747 | 0.743 | 0.428 | 0.383 | 0.392 | 0.509 |
| GPT-4o OpenAI (2025) | 0.445 | 0.663 | 0.683 | 0.693 | 0.400 | 0.463 | 0.383 | 0.501 |
| AutoT2IS + GPT-4o | 0.567 | 0.754 | 0.761 | 0.763 | 0.441 | 0.520 | 0.416 | 0.571 |

### 4.1 Benchmarking T2IS: Quantitative Comparison of Generalized and Specialized Methods

Experiment Setups. We evaluate diverse frameworks for T2IS generation capabilities: (1)  unified models like Show-o; (2)  compositional models that use Gemini Team et al. (2023) to generate sub-captions with state-of-the-art T2I models; and (3)  agentic models such as ChatDit that incorporate multi-step planning and generation processes. We also evaluate commercial models including Gemini 2.0 Flash and GPT-4o. Detailed setups for implementing these models are provided in Appendix [J](https://arxiv.org/html/2506.23275v2#A10).

Visual Consistency Challenges All Generalized Methods. As illustrated in Table [1](https://arxiv.org/html/2506.23275v2#S4.T1), most models perform well on prompt alignment, with average scores all exceeding 0.5, demonstrating the significant progress T2I models have made in alignment capabilities. However, all models exhibit significant deficiencies in visual consistency. All open-source methods score below 0.35 across all consistency dimensions, highlighting the value of exploring this research direction. Comparatively, our method achieves the best consistency capabilities among open frameworks, showing notable improvements over previous methods and delivering the best overall performance.

Commercial Models Still Need Improvement. As shown in the bottom of Table [1](https://arxiv.org/html/2506.23275v2#S4.T1), we evaluate state-of-the-art commercial models Google Cloud (2025); OpenAI (2025). These models, benefiting from large-scale pretraining, demonstrate superior instruction understanding and better consistency than open-source alternatives. However, our results reveal that they still struggle with T2IS generation tasks. During T2IS generation, their aesthetic quality and prompt alignment significantly deteriorate, even performing worse than many open-source models. This shortcoming can be mitigated by incorporating Auto-T2IS’s structured recaption to enrich semantic information, which achieves significant improvements in quality, alignment, and consistency. These findings underscore the necessity for improved T2IS task instruction comprehension in commercial models.

AutoT2IS Matches/Exceeds Specialized Methods. Table [2](https://arxiv.org/html/2506.23275v2#S4.T2) provides comparisons between our unified generation approach and specialized methods. These specialized methods are typically tailored and optimized for specific domains, allowing them to perform exceptionally well in those areas but limiting their adaptability across a wide range of T2IS tasks. The results in their respective domains on T2IS show that our method achieves competitive or even superior performance in areas where specialized methods have been pre-trained. This suggests that AutoT2IS effectively explores broader consistency principles while avoiding overfitting, thereby maintaining the model’s inherent generative capabilities. These findings highlight the value of unified methods for diverse T2IS tasks.

**Table 2: Comparisons of specialized methods and our unified AutoT2IS across different domains.**
| Model | Aesthetics | ˜ Prompt Alignment | Visual Consistency | Avg. |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Entity | Attribute | Relation | Identity | Style | Logic |  |  |  |
| Character Generation: Multi-view, Multi-scenario, Portrait Design |  |  |  |  |  |  |  |  |
| PhotoMaker Li et al. (2024a) | 0.413 | 0.793 | 0.799 | 0.762 | 0.482 | 0.458 | 0.450 | 0.552 |
| OnePrompt Liu et al. (2025) | 0.600 | 0.805 | 0.807 | 0.766 | 0.520 | 0.512 | 0.470 | 0.590 |
| X-Flux AI (2024) | 0.428 | 0.834 | 0.804 | 0.780 | 0.531 | 0.529 | 0.521 | 0.553 |
| AutoT2IS (ours) | 0.557 | 0.798 | 0.788 | 0.764 | 0.609 | 0.520 | 0.509 | 0.619 |
| Style Design Generation: Creative Style, Font, IP product design |  |  |  |  |  |  |  |  |
| IPAdapter-Flux Team (2024) | 0.514 | 0.856 | 0.833 | 0.846 | 0.386 | 0.455 | 0.517 | 0.582 |
| AutoT2IS (ours) | 0.475 | 0.789 | 0.791 | 0.792 | 0.339 | 0.391 | 0.478 | 0.533 |
| Story Generation: Movie Shot, Comic Story, Children Book |  |  |  |  |  |  |  |  |
| ICLora-Story Huang et al. (2024c) | 0.408 | 0.711 | 0.739 | 0.702 | 0.078 | 0.190 | 0.119 | 0.361 |
| StoryDiffusion Zhou et al. (2024b) | 0.581 | 0.684 | 0.728 | 0.670 | 0.182 | 0.207 | 0.141 | 0.413 |
| Story-Adapter Mao et al. (2024) | 0.578 | 0.749 | 0.766 | 0.734 | 0.181 | 0.328 | 0.226 | 0.463 |
| AutoT2IS (ours) | 0.534 | 0.629 | 0.698 | 0.660 | 0.262 | 0.405 | 0.237 | 0.456 |
| Process Generation: Growth, Draw, Building |  |  |  |  |  |  |  |  |
| MakeAnything Song et al. (2025) | 0.358 | 0.636 | 0.665 | 0.652 | 0.329 | 0.340 | 0.222 | 0.408 |
| AutoT2IS (ours) | 0.578 | 0.727 | 0.754 | 0.739 | 0.314 | 0.346 | 0.274 | 0.493 |

### 4.2 Visualizing T2IS: Qualitative Comparison and Diverse Application Showcase

AutoT2IS Excels at Common T2IS Tasks. Figure [5](https://arxiv.org/html/2506.23275v2#S4.F5) presents a qualitative comparison between our AutoT2IS and existing methods across several common T2IS tasks. As demonstrated, generalized methods like Show-o and Flux exhibit significant limitations in identity preservation, style uniform, and logical coherence. In contrast, AutoT2IS produces more visually consistent image sets. When evaluated against specialized methods in multi-scenario character generation, our approach maintains exceptional identity consistency while ensuring precise prompt alignment, achieving results comparable to the SOTA method 1Prompt1Story Liu et al. (2025). Notably, when it comes to challenging logical consistency generation tasks, existing methods like MakeAnything Song et al. (2025), despite being specifically trained for these scenarios, often compromise their fundamental generation capabilities, resulting in diminished image quality. Conversely, our training-free method preserves the model’s original generation capabilities and simultaneously maximizing its inherent consistency potential.

Figure: Figure 5: Qualitative comparison between our AutoT2IS and existing methods on common T2IS tasks. Black labels indicate generalized methods, while red labels represent specialized methods.
Refer to caption: x5.png

AutoT2IS Enables More Real-world Applications. Beyond common T2IS tasks, our AutoT2IS enables to address previously underexplored applications with significant real-world value, as illustrated in Figure [6](https://arxiv.org/html/2506.23275v2#S4.F6). These applications have been largely overlooked due to the lack of a unified approach to varying degrees of different consistency. For instance, IP Product Design (Task 2) requires coherence between IP concept identity and product style. Similarly, Growth Process Generation and Physical Law Illustration (Tasks 4 and 6) demand comprehensive consistency across identity, style, and logical dimensions. The diversity of the accomplished tasks demonstrates that our Auto-T2IS effectively addresses diverse consistency challenges within a unified model, highlighting its potential for broad applicability across various T2IS practical scenarios.

Figure: Figure 6: AutoT2IS enables novel T2IS applications with significant real-world value.
Refer to caption: x6.png

### 4.3 Ablation Studies

To better understand the contribution of each stage in AutoT2IS, we conduct ablation studies. Table [3](https://arxiv.org/html/2506.23275v2#S4.T3) presents the quantitative results across different evaluation dimensions.

**Table 3: Ablation studies on key stages.**
| Variants | Aesthetics | Alignment | Consistency |
| --- | --- | --- | --- |
| Full | 0.520 | 0.742 | 0.376 |
| w/o SR | 0.512 | 0.695 | 0.241 |
| w/o SG | 0.326 | 0.611 | 0.318 |

Importance of Structured Recaption (SR). The structured recaption plays a critical role by reformulating user inputs into more structured and detailed descriptions that better guide the image generation process. When we replace the structured recaption with a baseline approach that simply uses an LLM to generate sub-captions, we observe a significant drop across all evaluation metrics. These results highlight the importance of semantic completeness and relational coherence for effective T2IS generation.

Importance of Set-Aware Generation (SG). The set-aware generation is responsible for creating a consistent set across multiple images, while ensuring the quality and alignment of each sub-image. We replace this stage with a baseline approach that directly concatenates prompts to guide the model in generating multi-grid images Huang et al. (2024c; d). As shown in results, despite achieving reasonable consistency, the quality of individual images and text-image alignment significantly decreased. This demonstrates the importance of the divide-and-conquer approach in Set-Aware Generation, where the model first understands the content of individual image before rendering them with visual consistency. We include additional ablation with visualizations in Appendix [I](https://arxiv.org/html/2506.23275v2#A9) for more intuitive understanding.

## 5 Conclusion

In this paper, we advance the field of Text-to-ImageSet (T2IS) by introducing innovative frameworks for both T2IS generation and evaluation. We present T2IS-Bench, a comprehensive benchmark featuring 596 representative tasks across 26 detailed subcategories. Alongside this, we introduce T2IS-Eval, an evaluation framework that enables diverse consistency assessment. Furthermore, we introduce AutoT2IS, a training-free framework that fully exploits the exceptional in-context generation capabilities of DiT models, effectively harmonizing visual elements to meet both image-level prompt alignment and set-level visual consistency. By providing a transparent benchmark and a robust baseline, our work establishes a solid foundation for the future of T2IS research.

## References

- AI (2024)
XLabs AI.
x-flux: Flux model fine-tuning scripts.
[https://github.com/XLabs-AI/x-flux](https://github.com/XLabs-AI/x-flux), 2024.
Apache License 2.0.
- An et al. (2023)
Jie An, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Lijuan Wang, and Jiebo Luo.
Openleaf: Open-domain interleaved image-text generation and evaluation.
*arXiv preprint arXiv:2310.07749*, 2023.
- Bai et al. (2025)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin.
Qwen2.5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Baldridge et al. (2024)
Jason Baldridge, Kevin Shih, Yael Li, Zongyi Wang, Shweta Prabhudesai, Sashi Cheluvaraju, Xin Li, Lucy Chai, Miao Ding, Bryan Catanzaro, et al.
Imagen 2: Tuning text-to-image diffusion models for photorealism and generalization.
*arXiv preprint arXiv:2401.18680*, 2024.
- Betker et al. (2023)
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al.
Improving image generation with better captions.
*Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf*, 2(3):8, 2023.
- Chen et al. (2024)
Dongping Chen, Ruoxi Chen, Shu Pu, Zhaoyi Liu, Yanru Wu, Caixi Chen, Benlin Liu, Yue Huang, Yao Wan, Pan Zhou, and Ranjay Krishna.
Interleaved scene graph for interleaved text-and-image generation assessment.
*arXiv preprint arXiv:2411.17188*, 2024.
- Chen et al. (2023)
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li.
Pixart-$\alpha$: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023.
- Chen et al. (2025)
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan.
Janus-pro: Unified multimodal understanding and generation with data and model scaling.
*arXiv preprint arXiv:2501.17811*, 2025.
- Chern et al. (2024)
Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu.
Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation.
*arXiv preprint arXiv:2407.06135*, 2024.
- DeepSeek-AI (2025)
DeepSeek-AI.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- Dong et al. (2022)
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al.
A survey on in-context learning.
*arXiv preprint arXiv:2301.00234*, 2022.
- Esser et al. (2021)
Patrick Esser, Robin Rombach, and Björn Ommer.
Taming transformers for high-resolution image synthesis.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 12873–12883, 2021.
- Esser et al. (2024)
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al.
Scaling rectified flow transformers for high-resolution image synthesis.
In *Forty-first International Conference on Machine Learning*, 2024.
- Fu et al. (2023)
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola.
Dreamsim:learning new dimensions of human visual similarity using synthetic data.
In *Advances in Neural Information Processing Systems*, volume 36, pp. 50742–50768, 2023.
- Google Cloud (2025)
Google Cloud.
Generate images with gemini | generative ai on vertex ai, 2025.
URL [https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/image-generation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/image-generation).
Accessed: 2025-05-09.
- Hertz et al. (2024)
Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or.
Style aligned image generation via shared attention.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 4775–4785, 2024.
- Hessel et al. (2021)
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.
Clipscore: A reference-free evaluation metric for image captioning.
In *Empirical Methods in Natural Language Processing (EMNLP)*, 2021.
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
*Advances in neural information processing systems*, 33:6840–6851, 2020.
- Hosseini et al. (2025)
Seyed Mohammad Hadi Hosseini, Amir Mohammad Izadi, Ali Abdollahi, Armin Saghafian, and Mahdieh Soleymani Baghshah.
T2i-fineeval: Fine-grained compositional metric for text-to-image evaluation.
*arXiv preprint arXiv:2503.11481*, 2025.
- Hu et al. (2023)
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith.
Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 20406–20417, 2023.
- Huang et al. (2024a)
Jiehui Huang, Xiao Dong, Wenhui Song, Hanhui Li, Jun Zhou, Yuhao Cheng, Shutao Liao, Long Chen, Yiqiang Yan, Shengcai Liao, et al.
Consistentid: Portrait generation with multimodal fine-grained identity preserving.
*arXiv preprint arXiv:2404.16771*, 2024a.
- Huang et al. (2024b)
Lianghua Huang, Wei Wang, Zhi-Fan Wu, Huanzhang Dou, Yupeng Shi, Yutong Feng, Chen Liang, Yu Liu, and Jingren Zhou.
Group diffusion transformers are unsupervised multitask learners.
2024b.
- Huang et al. (2024c)
Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Yutong Feng, Yu Liu, and Jingren Zhou.
In-context lora for diffusion transformers.
*arXiv preprint arXiv:2410.23775*, 2024c.
- Huang et al. (2024d)
Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Chen Liang, Tong Shen, Han Zhang, Huanzhang Dou, Yu Liu, and Jingren Zhou.
Chatdit: A training-free baseline for task-agnostic free-form chatting with diffusion transformers.
2024d.
- Huang et al. (2024e)
Zehuan Huang, Yuan-Chen Guo, Haoran Wang, Ran Yi, Lizhuang Ma, Yan-Pei Cao, and Lu Sheng.
Mv-adapter: Multi-view consistent image generation made easy.
*arXiv preprint arXiv:2412.03632*, 2024e.
- Jia et al. (2025)
Chengyou Jia, Changliang Xia, Zhuohang Dang, Weijia Wu, Hangwei Qian, and Minnan Luo.
Chatgen: Automatic text-to-image generation from freestyle chatting.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 13284–13293, 2025.
- Labs (2024)
Black Forest Labs.
Flux: Inference repository.
[https://github.com/black-forest-labs/flux](https://github.com/black-forest-labs/flux), 2024.
Accessed: 2024-10-25.
- Li et al. (2024a)
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan.
Photomaker: Customizing realistic human photos via stacked id embedding.
In *IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024a.
- Li et al. (2024b)
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, Dayou Chen, Jiajun He, Jiahao Li, Wenyue Li, Chen Zhang, Rongwei Quan, Jianxiang Lu, Jiabin Huang, Xiaoyan Yuan, Xiaoxiao Zheng, Yixuan Li, Jihong Zhang, Chao Zhang, Meng Chen, Jie Liu, Zheng Fang, Weiyan Wang, Jinbao Xue, Yangyu Tao, Jianchen Zhu, Kai Liu, Sihuan Lin, Yifu Sun, Yun Li, Dongdong Wang, Mingtao Chen, Zhichao Hu, Xiao Xiao, Yan Chen, Yuhong Liu, Wei Liu, Di Wang, Yong Yang, Jie Jiang, and Qinglin Lu.
Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding, 2024b.
- Liang et al. (2024)
Chen Liang, Lianghua Huang, Jingwu Fang, Huanzhang Dou, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Junge Zhang, Xin Zhao, and Yu Liu.
Idea-bench: How far are generative models from professional designing?
*arXiv preprint arXiv:2412.11767*, 2024.
- Lin et al. (2024)
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan.
Evaluating text-to-visual generation with image-to-text generation.
In *European Conference on Computer Vision*, pp. 366–384. Springer, 2024.
- Liu et al. (2025)
Tao Liu, Kai Wang, Senmao Li, Joost van de Weijer, Fahad Shahbaz Khan, Shiqi Yang, Yaxing Wang, Jian Yang, and Ming-Ming Cheng.
One-prompt-one-story: Free-lunch consistent text-to-image generation using a single prompt.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=cD1kl2QKv1](https://openreview.net/forum?id=cD1kl2QKv1).
- Mao et al. (2024)
Jiawei Mao, Xiaoke Huang, Yunfei Xie, Yuanqi Chang, Mude Hui, Bingjie Xu, and Yuyin Zhou.
Story-Adapter: A Training-free Iterative Framework for Long Story Visualization, 2024.
- OpenAI (2025)
OpenAI.
Introducing 4o image generation.
[https://openai.com/index/introducing-4o-image-generation/](https://openai.com/index/introducing-4o-image-generation/), 2025.
Accessed: 2025-03-25.
- Peebles & Xie (2023)
William Peebles and Saining Xie.
Scalable diffusion models with transformers.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 4195–4205, 2023.
- Podell et al. (2023)
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.
Sdxl: Improving latent diffusion models for high-resolution image synthesis.
*arXiv preprint arXiv:2307.01952*, 2023.
- Rahman et al. (2023)
Tanzila Rahman, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Shweta Mahajan, and Leonid Sigal.
Make-a-story: Visual memory conditioned consistent story generation.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 2493–2502, 2023.
- Ramesh et al. (2021)
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.
Zero-shot text-to-image generation.
In *International Conference on Machine Learning*, pp. 8821–8831. PMLR, 2021.
- Ramesh et al. (2022)
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
Hierarchical text-conditional image generation with clip latents.
*arXiv preprint arXiv:2204.06125*, 2022.
- Rombach et al. (2022)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 10684–10695, 2022.
- Ronneberger et al. (2015)
Olaf Ronneberger, Philipp Fischer, and Thomas Brox.
U-net: Convolutional networks for biomedical image segmentation.
In *Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18*, pp. 234–241. Springer, 2015.
- Saharia et al. (2022)
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Tim Salimans, Jonathan Ho, David J Fleet, Phillip Isola, et al.
Photorealistic text-to-image diffusion models with deep language understanding.
*arXiv preprint arXiv:2205.11487*, 2022.
- Song et al. (2020)
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models.
*arXiv preprint arXiv:2010.02502*, 2020.
- Song et al. (2025)
Yiren Song, Cheng Liu, and Mike Zheng Shou.
Makeanything: Harnessing diffusion transformers for multi-domain procedural sequence generation.
*arXiv preprint arXiv:2502.01572*, 2025.
- Tan et al. (2024)
Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang.
Ominicontrol: Minimal and universal control for diffusion transformer.
*arXiv preprint arXiv:2411.15098*, 2024.
- Team et al. (2023)
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al.
Gemini: a family of highly capable multimodal models.
*arXiv preprint arXiv:2312.11805*, 2023.
- Team (2024)
InstantX Team.
Instantx flux.1-dev ip-adapter page, 2024.
- Tewel et al. (2024)
Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon.
Training-free consistent text-to-image generation.
*ACM Transactions on Graphics (TOG)*, 43(4):52, 2024.
doi: 10.1145/3658157.
URL [https://doi.org/10.1145/3658157](https://doi.org/10.1145/3658157).
- Wiles et al. (2024)
Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kajić, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Pinelopi Papalampidi, Ira Ktena, Chris Knutsen, et al.
Revisiting text-to-image evaluation with gecko: On metrics, prompts, and human ratings.
*arXiv preprint arXiv:2404.16820*, 2024.
- Xie et al. (2024)
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou.
Show-o: One single transformer to unify multimodal understanding and generation.
*arXiv preprint arXiv:2408.12528*, 2024.
- Ye et al. (2023)
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.
Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.
2023.
- Zhang & Agrawala (2025)
Lvmin Zhang and Maneesh Agrawala.
Packing input frame contexts in next-frame prediction models for video generation.
*Arxiv*, 2025.
- Zhang et al. (2024)
Sixian Zhang, Bohan Wang, Junqiang Wu, Yan Li, Tingting Gao, Di Zhang, and Zhongyuan Wang.
Learning multi-dimensional human preference for text-to-image generation.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 8018–8027, 2024.
- Zhang* et al. (2020)
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi.
Bertscore: Evaluating text generation with bert.
In *International Conference on Learning Representations*, 2020.
URL [https://openreview.net/forum?id=SkeHuCVFDr](https://openreview.net/forum?id=SkeHuCVFDr).
- Zhou et al. (2024a)
Pengfei Zhou, Xiaopeng Peng, Jiajun Song, Chuanhao Li, Zhaopan Xu, Yue Yang, Ziyao Guo, Hao Zhang, Yuqi Lin, Yefei He, Lirui Zhao, Shuo Liu, Tianhua Li, Yuxuan Xie, Xiaojun Chang, Yu Qiao, Wenqi Shao, and Kaipeng Zhang.
Opening: A comprehensive benchmark for judging open-ended interleaved image-text generation.
*arXiv preprint arXiv:2411.18499*, 2024a.
- Zhou et al. (2024b)
Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou.
Storydiffusion: Consistent self-attention for long-range image and video generation.
*Advances in Neural Information Processing Systems*, 37:110315–110340, 2024b.

## Appendix A More details of T2IS-Bench

We first present the categorization of each group. Then, we provide comprehensive statistical information about the entire dataset.

### A.1 Group Definitions

We construct our benchmark by carefully curating tasks from five groups that comprehensively evaluate different aspects of visual consistency:

Character Generation. This group focuses on maintaining consistent character identity across diverse contexts. Models must generate characters with consistent physical attributes (facial features, body proportions, clothing), and distinguishing characteristics even when shown in different poses, viewing angles, expressions, and environments.

Design Style Generation. This group evaluates the ability to maintain a cohesive visual style and aesthetic across different design elements and contexts. Models must focus on creating a unified visual language that ties different design pieces together, *e.g.*, posters, product packaging, brand materials, or decor elements, while ensuring the style remains distinctive and purposeful.

Story Generation. This group requires generating image sets that convey clear story progression while maintaining consistent characters and visual style. Tasks span movie scenes, comic panels, children’s books, and news illustrations.

Process Generation. This group emphasizes temporal and causal consistency in depicting step-by-step procedures. Models must generate logically ordered sequences illustrating processes like growth, construction, cooking, or physical phenomena. Each sequence should maintain consistent visual style and recognizable elements across steps.

Instruction Generation. This group evaluates logical consistency in educational and instructional content. Tasks involve creating image sets that effectively communicate procedures, historical information, product usage, or travel guidance. Models must ensure consistent visual presentation, clear identification of elements, and logical sequencing.

In the Appendix [K](https://arxiv.org/html/2506.23275v2#A11), we provide detailed descriptions and examples for each category in each group.

Figure: Figure 7: Distribution of T2IS-Bench in terms of image set size and instruction length. The heatmap shows the density of tasks, with darker blue indicating higher concentration.
Refer to caption: figure/appendix/benchmark_distribution.png

### A.2 Dataset Statistics

Our T2IS-Bench dataset contains a diverse collection with varying image set sizes and instruction lengths. As shown in Figure [7](https://arxiv.org/html/2506.23275v2#A1.F7), the distribution of tasks exhibits several notable characteristics:

Image Set Size Distribution: While tasks range from 2-8 images per set, we focus primarily on sets requiring 3-5 images, with a mean of 4.24 images. This range provides sufficient complexity for evaluating consistency while remaining computationally tractable. Tasks with 2 images are too simple to meaningfully assess consistency, while those above 5 images become overly complex, so we concentrate the distribution around 3-5 images per set with a peak at 4 images.

Instruction Length Distribution: The instructions range from 20 to 175 words, with a mean length of 99.85 words. Shorter instructions (20-50 words) provide flexibility in image generation by specifying only key requirements, while longer instructions (150-175 words) offer more precise control over visual elements and relationships. To balance flexibility and control, most tasks have instructions between 50-150 words, allowing sufficient detail to guide generation while avoiding over-specification.

This distribution ensures our benchmark provides a comprehensive evaluation of models’ capabilities in handling both simple and complex T2IS generation tasks.

### A.3 Dataset Curation Process

Our dataset underwent a meticulous curation process to ensure high-quality and diverse instructions, comprising three main phases:

1. Collection Guideline: We first designed a standardized collection guideline to ensure instruction quality and diversity. It includes:

- •
Clear definitions of task types and target outcomes.
- •
Instruction templates and examples covering various identity, style, and logic dimensions.
- •
A checklist to ensure that each instruction requires non-trivial generation beyond simple captioning.

Each instruction and its category label were double-reviewed by PhD-level experts, with conflicts adjudicated by an additional expert.

2. Sampling Procedure: To ensure broad coverage across 26 generation categories, we applied stratified sampling:

- •
Starting from a large pool of open-ended instructions, we selected over 650 candidates.
- •
We ensured varying complexity (simple, moderate, abstract) across and within categories.
- •
Categories demanding high-level logical reasoning (e.g., story or process) were intentionally oversampled for robustness.

3. Human Feedback and Refinement: Finally, a Refinement Phase was conducted where experts reviewed model outputs for each instruction, evaluating:

- •
Clarity and executability.
- •
Factual correctness.
- •
Avoidance of overly long or NSFW content.

Low-quality instructions were revised or discarded. Feedback from early T2IS-Bench users further guided refinements. The final dataset includes 596 rigorously curated instructions.

## Appendix B More details of T2IS-Eval and Comparison with Existing Benchmarks

### B.1 More details of T2IS-Eval

Visual Consistency Evaluation. We utilize DeepSeek-R1 DeepSeek-AI (2025) as the foundational LLM for criteria generation, and Qwen2.5-VL-7B Bai et al. (2025) serves as our consistency evaluator. Specifically, we first resize all images in the image set to 512$\times$512 pixels. Then, we pair the images sequentially and feed each pair to Qwen2.5-VL-7B with generated criteria. For each pair, we compute the logits for “Yes” and “No” responses, apply softmax to obtain probabilities, and use the probability of “Yes” as the consistency score. The final score for each dimension is calculated by averaging the scores across all pairs within that dimension.

Compared to simple answer tokens, answer likelihoods provide more fine-grained measurements while avoiding potential uncertainty and hallucination issues that may arise from directly asking for numerical scores Lin et al. (2024). In Appendix [D](https://arxiv.org/html/2506.23275v2#A4), we further validate through experiments the effectiveness of using Qwen2.5-VL-7B as a consistency evaluator and demonstrate the advantages of this approach.

Prompt Alignment Evaluation. For evaluating prompt alignment capabilities, we adopt VQAScore Lin et al. (2024), a well-established method for assessing image-text alignment. Specifically, we employ DeepSeek-R1 DeepSeek-AI (2025) to generate alignment criteria for each sub-image from three perspectives: Entity, Attribute, and Relation. We then evaluate each sub-image against these criteria and compute the average score. This approach effectively assesses whether T2IS models can maintain strong image-level prompt alignment capabilities when generating multiple coherent images.

Aesthetic Quality Evaluation. We further employ MPS Zhang et al. (2024) to evaluate the overall aesthetic quality of generated images. MPS introduces a preference condition module built upon the CLIP model to learn diverse aesthetic preferences, including composition, light contrast, color matching, clarity, tone, style, depth of field, atmosphere, and artistry of human figures. The final aesthetic score for an image set is computed by averaging the MPS scores of all individual images within the set.

Holistic Average. All evaluation metrics are normalized to the range of [0,1], enabling holistic assessment. Considering the significant progress in Aesthetic Quality and Prompt Alignment, we assign different weights when computing the holistic assessment score: 0.2 for Aesthetic Quality, 0.3 for Prompt Alignment, and 0.5 for Visual Consistency. This weighting scheme reflects both the relative importance and the current challenges in T2IS generation.

### B.2 Comparison with Existing Benchmarks

As shown in Table [4](https://arxiv.org/html/2506.23275v2#A2.T4), compared to previous benchmarks like OpenLeaf, OpenING, and IDEA-Bench, T2IS-Bench with T2IS-Eval offers several key advantages:

Comprehensive Coverage: T2IS-Bench provides extensive coverage across 26 diverse categories while maintaining a reasonable benchmark size of 596 tasks. In contrast, existing benchmarks have limited domain coverage, ranging from just 2 domains in Consistory+ to 10 domains in IDEA-Bench.

Multi-level Evaluation: Our benchmark enables detailed evaluation at both the individual image level prompt alignment and set-level consistency. While some benchmarks like OpenLeaf and OpenING support sub-image evaluation, and others like IDEA-Bench assess consistency, T2IS-Bench uniquely combines both aspects for comprehensive assessment.

Adaptive Assessment: T2IS-Bench provides adaptive evaluation criteria tailored to each instruction’s specific requirements. For example, character generation tasks are evaluated based on identity consistency, while process generation tasks focus on logical coherence of steps. This instruction-specific evaluation approach contrasts with existing benchmarks that use fixed evaluation criteria regardless of task type, enabling more nuanced and relevant assessment of model capabilities.

**Table 4: Comparison with existing consistent generation benchmarks.**
|  | Instruction<br>(# Numbers) | Category<br>(# Numbers) | Category<br>Coverage | Sub-image<br>Evaluation? | Consistency<br>Evaluation? | Adaptive<br>Evaluation? |
| --- | --- | --- | --- | --- | --- | --- |
| Consistory+ Liu et al. (2025) | 200 | 2 |  | ✓ | ✓ | ✗ |
| Leaf An et al. (2023) | 30 | 4 |  | ✗ | ✗ | ✗ |
| ING Zhou et al. (2024a) | 56 | 5 |  | ✗ | ✗ | ✗ |
| ISG Chen et al. (2024) | 1,150 | 8 |  | ✓ | ✗ | ✗ |
| IDEA Liang et al. (2024) | 100 | 10 |  | ✗ | ✓ | ✗ |
| GDT Huang et al. (2024b) | 200 | 13 |  | ✓ | ✓ | ✗ |
| T2IS-Bench | 596 | 26 |  | ✓ | ✓ | ✓ |

## Appendix C Detailed Experimental Settings

We employ FLUX.1-dev Labs (2024) as our base model. We execute a total of 20 denoising steps with a guidance scale of 3.5. Each invidual image has a final resolution of 512$\times$512 pixels, with initial latent dimensions of 32$\times$32. For image sets of size 2, 3, and 5, we arrange them in a 1$\times$n grid layout during the conquer phase. For sets of size 4, we utilize a 2$\times$2 grid arrangement. For sets larger than 5 images, we adopt a sliding window approach with a window size of 4 and stride of 2. Notably, our experiments reveal that the 2$\times$2 grid configuration achieves superior results, likely due to the closer proximity between regions. In contrast, the 1$\times$5 arrangement tends to introduce consistency mismatches between the first and last images. We provide additional visual demonstrations in Appendix [F](https://arxiv.org/html/2506.23275v2#A6).

## Appendix D Reliability Analysis of T2IS-Eval

In this section, we conduct experiments to validate the reliability of our T2IS-Eval framework by comparing with two types of human evaluations. Our analysis focuses on assessing how well Qwen2.5-VL’s judgments align with human assessments of visual consistency between image pairs.

#### Human Binary Assessment.

We first collected binary judgments (consistent/inconsistent) from human annotators on 200 diverse image pairs. For each pair, three annotators were asked to make a yes/no decision on whether the images were visually consistent. The final human binary label was determined by majority voting.

#### Human Fine-grained Assessment.

We also gathered fine-grained consistency scores from human raters on a 0-5 scale, where 0 indicates complete inconsistency and 5 indicates perfect consistency. This allowed capturing more nuanced degrees of visual consistency that may be missed by binary judgments. Each image pair was rated by three annotators and scores were averaged.

#### Alignment between Model and Human Judgments.

As shown in Table [5](https://arxiv.org/html/2506.23275v2#A4.T5), Qwen2.5-VL’s binary decisions achieved a Pearson Similarity of 0.78, demonstrating strong alignment with human judgments in distinguishing consistent from inconsistent image pairs. For fine-grained assessment, our model outputs likelihood scores across different consistency levels, offering more nuanced measurements while avoiding potential hallucination issues that could arise from directly generating numerical scores. To evaluate the reliability of this fine-grained assessment, we scaled the model’s likelihood outputs from [0,1] to [0,5] to match the human rating scale. This scaled likelihood distributions achieved a Pearson Similarity of 0.61 with human fine-grained ratings, validating that T2IS-Eval can effectively capture subtle variations in visual consistency that align well with human preferences.

**Table 5: Comparison between human and model evaluations on visual consistency assessment. We report the Pearson Similarity between human annotators and model predictions.**
| Method | Human Binary | Human Fine-grained |
| --- | --- | --- |
| Qwen2.5-VL Binary | 0.78 | - |
| Qwen2.5-VL Fine-grained | - | 0.61 |

## Appendix E Limitations and Future Work

While our proposed approach demonstrates promising results for generating consistent image sets, there are several limitations and opportunities for future work.

#### Baseline Nature of Current Approach.

Our current divide-and-conquer strategy, while effective, represents a baseline approach to the problem of image set generation. More sophisticated methods for ensuring consistency and managing the generation process could potentially yield better results.

#### Resolution and Memory Constraints.

The current implementation is optimized for generating images at 512$\times$512 resolution. When scaling to higher resolutions such as 1024$\times$1024, the memory requirements increase substantially, particularly during the conquer phase where multiple images need to be processed simultaneously. This limitation becomes especially pronounced when generating larger image sets, necessitating further research into memory-efficient techniques.

#### Consistency Challenges.

Our experiments reveal several key challenges in T2IS generation, including maintaining visual consistency across spatially distant positions, handling complex instructions requiring strong reasoning, and coordinating detailed sub-stories across multiple frames. We provide detailed analysis and examples of these challenges in Appendix [F](https://arxiv.org/html/2506.23275v2#A6).

#### Future Work.

Future research could focus on:

- •
Developing memory-efficient architectures for high-resolution generation through progressive growing or sparse computation methods Zhang & Agrawala (2025) to address the resolution and memory constraints.
- •
Exploring large-scale pretraining strategies Google Cloud (2025); OpenAI (2025) and instruction understanding to enhance the model’s capability in handling instructions requiring strong reasoning and domain knowledge.
- •
Improving long-range consistency through advanced attention mechanisms and hierarchical modeling to better maintain visual coherence across spatially distant positions.
- •
Developing better prompt alignment strategies to effectively coordinate detailed sub-stories and ensure narrative coherence across multiple frames.

## Appendix F Failure Cases Analysis

We analyze three main types of failure cases observed in our experiments:

#### Consistency Degradation in Grid Layout.

In the 1*5 grid layout, we observe consistency degradation across the image sequence. Figure [9](https://arxiv.org/html/2506.23275v2#A9.F9) shows this limitation where the logo’s appearance noticeably shifts between the first and last images in the sequence, highlighting the challenge of maintaining visual consistency across spatially distant positions.

#### Complex Instruction Understanding.

Figure [9](https://arxiv.org/html/2506.23275v2#A9.F9) demonstrates the model’s limitations in precisely following complex instructions that require strong reasoning and domain knowledge. When given instructions involving physical laws while maintaining specific constraints across the sequence, the model struggles to accurately execute all requirements simultaneously.

#### Story Prompt Alignment in Detailed Story Generation.

As shown in Figure [10](https://arxiv.org/html/2506.23275v2#A9.F10), while our method maintains overall style and character consistency, coordinating detailed sub-stories across multiple frames remains challenging. The model struggles to perfectly align individual sub-stories and local details when generating rich narratives that must both work independently and contribute to a cohesive sequence. This highlights the need for better prompt alignment strategies in complex storytelling.

These limitations point to important areas for future research, including better handling of long-range consistency and improved instruction understanding for complex scenarios requiring domain knowledge and precise control.

## Appendix G Analysis of Challenge Categories

we include below a summary table showing the performance across five high-level categories. As shown, tasks that require strong logical consistency, such as Story and Process generation, pose significant challenges to current models in terms of maintaining logical coherence across the image set. Moreover, Story Generation often involves multiple distinct identities, making identity consistency particularly difficult for existing models. These findings help reveal the specific weaknesses of current approaches and can guide future research.

**Table 6: Performance Summary Across High-Level Categories**
| Category | Dimension | Show-o | Gemini & Flux-1 | Gemini & SD3 | AutoT2IS |
| --- | --- | --- | --- | --- | --- |
| Character Generation | Style | 0.438 | 0.439 | 0.409 | 0.520 |
| Character Generation | Identity | 0.441 | 0.454 | 0.435 | 0.609 |
| Character Generation | Logic | 0.495 | 0.516 | 0.515 | 0.510 |
| DesignStyle Generation | Style | 0.353 | 0.343 | 0.344 | 0.391 |
| DesignStyle Generation | Identity | 0.285 | 0.285 | 0.302 | 0.339 |
| DesignStyle Generation | Logic | 0.405 | 0.471 | 0.452 | 0.478 |
| Story Generation | Style | 0.196 | 0.235 | 0.187 | 0.405 |
| Story Generation | Identity | 0.120 | 0.145 | 0.115 | 0.262 |
| Story Generation | Logic | 0.153 | 0.182 | 0.179 | 0.237 |
| Process Generation | Style | 0.195 | 0.201 | 0.198 | 0.346 |
| Process Generation | Identity | 0.139 | 0.143 | 0.142 | 0.314 |
| Process Generation | Logic | 0.185 | 0.222 | 0.212 | 0.274 |
| Instruction Generation | Style | 0.265 | 0.310 | 0.311 | 0.437 |
| Instruction Generation | Identity | 0.190 | 0.231 | 0.238 | 0.315 |
| Instruction Generation | Logic | 0.197 | 0.256 | 0.250 | 0.315 |

## Appendix H More Results for Reliability Analysis

### H.1 Statistical Significance Analysis

We conducted additional experiments and reported the mean and standard deviation across three runs with different random seeds to analyze the effect of random variation. (Due to computational constraints, we focused on methods with manageable inference costs.)

As shown in Table [7](https://arxiv.org/html/2506.23275v2#A8.T7), the mean scores are highly consistent with the single-run results originally reported, and the standard deviations are very small, indicating that the observed improvements are statistically significant. Importantly, the conclusions drawn from the multi-run experiments remain identical to those from the single-run results. Moreover, given the large number of instructions (596 in total), the aggregate scores are inherently robust to small fluctuations caused by random seeds.

**Table 7: Mean and Standard Deviation of Evaluation Metrics Across Three Runs.**
| Method | Identity | Style | Logic |
| --- | --- | --- | --- |
| Show-o | 0.239$\pm$0.005 | 0.293$\pm$0.006 | 0.292$\pm$0.005 |
| Janus-Pro | 0.240$\pm$0.021 | 0.287$\pm$0.020 | 0.320$\pm$0.029 |
| Gemini & Flux-1 | 0.253$\pm$0.004 | 0.307$\pm$0.005 | 0.333$\pm$0.004 |
| Gemini & SD3 | 0.260$\pm$0.014 | 0.303$\pm$0.014 | 0.329$\pm$0.008 |
| AutoT2IS (ours) | 0.353$\pm$0.008 | 0.412$\pm$0.006 | 0.353$\pm$0.003 |

**Table 8: P-value Matrices for Paired T-tests Across Evaluation Dimensions.**
| Method | Show-o | Janus-Pro | Gemini & Flux-1 | Gemini & SD3 | AutoT2IS (ours) |
| --- | --- | --- | --- | --- | --- |
| Show-o | — | 0.5968 | 0.0230 | 0.1785 | 0.0025 |
| Janus-Pro | 0.5968 | — | 0.2510 | 0.3058 | 0.0134 |
| Gemini & Flux-1 | 0.0230 | 0.2510 | — | 0.5456 | 0.0018 |
| Gemini & SD3 | 0.1785 | 0.3058 | 0.5456 | — | 0.0085 |
| AutoT2IS (ours) | 0.0025 | 0.0134 | 0.0018 | 0.0085 | — |

In addition to reporting mean ± standard deviation with different random seeds, we have now conducted paired t-tests between some methods over the 596-instruction evaluation set in Table [8](https://arxiv.org/html/2506.23275v2#A8.T8). These results clearly indicate that AutoT2IS (ours) achieves statistically significant improvements (p < 0.05, often p < 0.01) over baselines.

### H.2 LLM Performance and Reliability

Consistency Evaluator. In T2IS-Eval, the LLM is utilized solely for generating the initial question criteria. These criteria undergo a manual review process to filter out ambiguities and hallucinations. Once finalized, they are consistently applied across all methods being compared, ensuring fairness and reliability. Our manual verification process indicates that over 90% of the generated criteria are reliable. The remaining cases are either regenerated or manually rewritten to ensure quality.

Structured Recaption. For AutoT2IS, we selected DeepSeek-R1 after evaluating several alternatives, as shown in Table [9](https://arxiv.org/html/2506.23275v2#A8.T9). DeepSeek-R1 demonstrated superior performance at a significantly lower cost. This module employs an LLM as a proxy for structured prompt generation, which does not affect our overall contribution. It could be substituted with human-written captions, albeit at a higher cost. With advancements in LLM technology, its effectiveness is expected to improve further.

**Table 9: Performance of Different Structured Recaption Variants.**
| Variants | Aesthetics | Alignment | Consistency |
| --- | --- | --- | --- |
| DeepSeek-R1 | 0.520 | 0.742 | 0.376 |
| GPT-4o | 0.514 | 0.735 | 0.360 |
| Gemini 2.0 | 0.515 | 0.724 | 0.362 |
| DeepSeek-V3 | 0.508 | 0.718 | 0.325 |
| GPT-4o-mini | 0.502 | 0.721 | 0.320 |

## Appendix I Ablation Studies with Visualizations

Figure [11](https://arxiv.org/html/2506.23275v2#A9.F11) illustrates the impact of our key components through visual examples. Without Structured Recaption (SR), the model lacks comprehensive global semantic constraints, which significantly impairs its ability to maintain consistency across the image set. While individual images may appear well-formed in isolation, they exhibit noticeable discrepancies in visual elements and artistic style when viewed as a sequence. This demonstrates that SR plays a crucial role in enforcing coherent visual and semantic relationships throughout the generated image set.

When Set-Aware Generation (SG) is removed, although the images retain some consistency in their overall visual aesthetic, they suffer from noticeable quality degradation and reduced alignment with the original prompts (as evidenced by the portable gaming console example in the first image). Furthermore, maintaining consistent representation of detailed elements across the image set becomes substantially more challenging.

In contrast, our full method successfully combines both components, demonstrating significant improvements over the ablated versions by achieving high-quality, consistent, and well-aligned image sets that overcome the limitations observed in each individual component.

We further analyze the effect of different step sizes in our divide-and-conquer strategy. The step size N determines the granularity of the divide phase, with the remaining steps allocated to the conquer phase. As shown in Figure [11](https://arxiv.org/html/2506.23275v2#A9.F11), smaller step sizes (N=1) lead to semantic confusion and loss of details, as evidenced by the portable gaming console example (similar to the case without Set-Aware Generation). The initial semantic imprecision makes it difficult to maintain consistency of detailed elements during the later conquer phase. In contrast, with larger step sizes, individual images tend to maintain their own content, resulting in overall inconsistency across the set. This demonstrates the critical importance of balancing step sizes between the two phases. We found that a ratio of 2:20 provides an optimal balance point. Moreover, this ratio can be generalized - similar balanced results can be achieved with proportional ratios like 3:30 and 4:40.

Figure: Figure 8: Example of consistency degradation in 1*5 grid layout. Note how the logo’s appearance shifts across the sequence, with more pronounced differences between the first and last images.
Refer to caption: x7.png

Figure: Figure 10: Example of challenges in maintaining story prompt alignment across frames. While our method achieves overall style and character consistency, aligning detailed sub-stories across multiple frames remains challenging.
Refer to caption: x9.png

Figure: Figure 11: Ablation study visualizing the impact of different components and step sizes. Top row: Results without Structured Recaption (SR) showing inconsistent style and elements. Second row: Results without Set-Aware Generation (SG) showing degraded quality despite some consistency. Third row: Results with step size 1/20 showing semantic confusion. Fourth row: Our full method with optimal step ratio 2/20. Bottom rows: Results with larger step ratios (4/20, 6/20) showing maintained quality but reduced consistency across the set.
Refer to caption: x10.png

## Appendix J Implementation Details of Existing Methods

For evaluating various generation methods, we implement and compare the following approaches:

Compositional Models. We use gemini-2.0-flash Team et al. (2023) to decompose the T2IS instructions into sub-captions, which are then fed into state-of-the-art T2I models including Stable Diffusion 3 Esser et al. (2024), Pixart Chen et al. (2023), Hunyuan Li et al. (2024b), and Flux-1 Labs (2024). For each model, we follow their official implementation settings with consistent configurations: number of inference steps and guidance scale as specified in their default scripts, and a unified resolution of 1024*1024 pixels.

Unified Models. We evaluate Anole Chern et al. (2024), Show-o Xie et al. (2024), and Janus-Pro Chen et al. (2025) using their official implementations and pretrained weights. Among these models, only Anole demonstrates native capability for multi-image generation, while Show-o and Janus-Pro exhibit limitations in comprehending complete T2IS instructions. Due to their relatively weak instruction understanding abilities, we augment their performance by utilizing the sub-captions approach from Compositional Models.

Agentic Models. We evaluate ChatDit Huang et al. (2024d) and ISG-Agent Chen et al. (2024) which employ multi-step planning and generation processes. Instructions are directly fed into their official scripts to obtain the generated image sets for evaluation.

Commercial Models. We also evaluate commercial models including Gemini 2.0 Flash Google Cloud (2025) and GPT-4o OpenAI (2025). For Gemini 2.0 Flash, we utilize their latest available APIs which directly support multi-image generation, allowing us to evaluate the returned image sets uniformly. Since GPT-4o can only generate single images, we modify the instructions by prepending "please generate [num]-grid images" to produce multi-grid outputs that we then manually segment. As GPT-4o’s API was not publicly available during our testing period, we developed automated scripts to interact with their official web interface for input submission and image collection.

## Appendix K Visualization Gallery

This section provides a comprehensive visualization gallery that serves two key purposes: (1) demonstrating the diversity and coverage of T2IS-Bench through carefully curated examples across different categories, and (2) showcasing AutoT2IS’s capabilities in handling these diverse text-to-image set generation scenarios. For each category, we present detailed task descriptions, example instructions, and AutoT2IS-generated image sets to illustrate both the benchmark’s systematic organization and our method’s effectiveness in generating high-quality, consistent image sets.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See the weakness section.
soundness: 3
strengths: - AutoT2IS achieves significant improvements over existing methods through a simple yet effective divide-and-conquer strategy that maximally leverages DiT's in-context capabilities.
- Comprehensive experiments demonstrating competitive or superior performance compared to both generalized approaches and even specialized methods.
- The presentation is clear and easy to follow.
summary: This paper proposes Text-to-ImageSet (T2IS) generation, a new task that requires models to generate coherent image sets that satisfy diverse consistency requirements across identity, style, and logic dimensions. The authors introduce T2IS-Bench, a comprehensive benchmark with 596 instructions spanning 26 subcategories, along with T2IS-Eval, an adaptive evaluation framework that transforms user instructions into multifaceted assessment criteria. They propose AutoT2IS, a training-free approach leveraging pretrained Diffusion Transformers through structured recaptioning and set-aware generation with a divide-and-conquer strategy, demonstrating superior performance over both generalized and specialized baselines.
weaknesses: - It seems the size of the image set (n) ranges from 2 to 5. The paper does not discuss the effect of this size number. For example, how will the size affect the quality and consistency of the images? What will happen when the size is extended to even larger numbers, such as 8 or 10?
- In addition to text-to-image generation works, the authors should also carefully discuss another line of research, i.e., interleaved text-and-image generation, where the MLLM is also tasked with generating a sequence of images and text. Several representative works include InterleavedEval [1] and OpenING [2].

[1] Holistic Evaluation for Interleaved Text-and-Image Generation. EMNLP 2024.

[2] OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation. CVPR 2025.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See above.
soundness: 3
strengths: 1. This paper is well-written and very easy to follow.
2. This paper proposes a novel task: text to image set. This task has application values and is very important for future image generation research. Besides, the newly design evaluation metrics are also reasonable.
3. This paper also proposes a training-free framework for this task, which is novel and promising.
4. Comprehensive evaluation proves this benchmark is important, revealing some limitations of current open-/closed-source models, and providing interesting research direction for future work.
summary: This paper introduces a new research paradigm called Text-to-ImageSet (T2IS) generation, which aims to produce a set of semantically coherent and visually consistent images from a single textual description—going beyond the traditional text-to-image (T2I) paradigm that generates only one image. To systematically study this problem, the authors propose three major contributions: (1) T2IS-Bench, the first large-scale benchmark for this task, covering 596 high-quality prompts across 26 subcategories that span various dimensions of consistency (identity, style, logic, etc.); (2) T2IS-Eval, an automatic evaluation framework leveraging large language and vision-language models to assess multi-dimensional consistency through interpretable, question–answer-based scoring; and (3) AutoT2IS, a training-free generation framework that exploits the in-context capabilities of diffusion transformers via structured recaption and set-aware generation. Experiments on T2IS-Bench demonstrate that AutoT2IS outperforms state-of-the-art models—including Show-o, Janus-Pro, and Gemini+Flux—across identity, style, and logic consistency dimensions, while maintaining generality and efficiency. The work establishes a strong foundation for future exploration in consistent multi-image generation and introduces valuable tools for benchmarking and evaluation in this emerging area.
weaknesses: From my perspective, as a benchmark paper, this is enough. But I still have several questions:
1. How do you make sure the evaluation consistency? The evaluation metrics are VLLM-based. It would be better to prove your proposed evaluation metrics is reasonable. For example, you can have a subset human evaluation and analyze the consistency of your metrics and human evaluation score. 
2. It seems that the proposed framework performs worse than baselines in some cases in Table 2 (e.g., Style Design Generation, Story Generation). Can authors explain the performance gap or give some error analysis on this part?
3. The image generation model is the most important part for the framework. Could you conduct more ablation study on image generation model? In your framework, keep all other parts as the same, but change the image generation model to Flux, Image-Qwen, SD3, also closed-source model such as GPT and Nano Banana, etc. You do not need to conduct on the whole dataset if the time is not enough, instead you can subset 25% samples to conduct this ablation study. This is to analyze in the same framework, how different image generation model influence the final results.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: Please see weakness points above.
soundness: 2
strengths: - The paper proposes a task of Text-to-ImageSet generation, which generates multiple images instead of single images.
- A benchmark and evaluation framework are proposed for this task.
- A training-free method is proposed that leverages LLMs for structured recaption and a set-aware generation strategy.
summary: This paper introduces Text-to-ImageSet generation, the task of creating coherent image sets from a single instruction. The authors propose three contributions: T2IS-Bench, a new benchmark with diverse instructions; T2IS-Eval, an MLLM-based evaluation framework for set-level consistency ; and AutoT2IS, a training-free generation method. AutoT2IS uses an LLM for "Structured Recaption" to create detailed prompts and a "Set-Aware Generation" diffusion strategy to balance individual image content with set-level consistency. Experiments show the proposed method generate high quality image sets.
weaknesses: - The primary concern is the framing of the T2IS problem itself. The paper aggregates several pre-existing, distinct research tasks (e.g., 'Character Generation' , 'Story Generation' , 'Process Generation' ) under a new umbrella, as shown in Table 2. **It is not clearly articulated what new research challenge is unlocked by this aggregation.** The contribution appears to be more of a unified testbed rather than a novel research problem, which raises questions about the work's fundamental research significance.

- The reliability of the T2IS-Eval framework is questionable. The evaluation hinges entirely on an MLLM-as-a-judge (Qwen2.5-VL-7B). The paper's own reliability analysis in Appendix D (Table 5) reports only a 0.61 Pearson Similarity between the model's fine-grained scores and human fine-grained ratings. This moderate correlation suggests the automated metric may be an unreliable proxy for human perception of consistency and may not be robust enough to support the paper's strong quantitative claims.

- The methodological novelty of the AutoT2IS framework appears limited. The "Structured Recaption" component, which uses an LLM to parse a complex prompt into multiple sub-captions, is a common pattern in many recent agentic and compositional generation frameworks. This part of the method does not represent a significant methodological advance for the T2IS problem.

### **Conclusion**
In conclusion, although the paper proposed a task and provides a benchmark which seems new, **it is not of enough research significance**. The 'T2IS' problem is largely an aggregation of existing tasks, which limits its research novelty. This is compounded by an evaluation metric of questionable reliability and a generation method with limited methodological innovation.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: The questions are mainly about the weaknesses:

1. Could the authors elaborate on the position of the proposed dataset? How does it offer a more effective path toward generalization compared to combining existing datasets?
2. Are there many noisy or inconsistent samples in the dataset, and how do they affect the experiments?
2. How can T2IS-Bench be used to promote the training of generalized models rather than just serving as an evaluation benchmark?
soundness: 3
strengths: * The paper proposes a fresh and interesting idea: generating sets of images that maintain both variety and consistency. This direction extends traditional text-to-image generation toward more general and realistic applications.
* The method is described clearly, and the proposed approach that combines prompt concatenation and masked latent operations is creative and well-motivated.
* The experiments are comprehensive, covering different domains and including strong comparisons with both open-source and commercial models. The T2IS-Bench dataset also appears to be well-organized and diverse, which strengthens the results.
summary: This paper introduces a new task called Text-to-ImageSet (T2IS) generation, which aims to create coherent sets of images that share visual consistency while following diverse text instructions. To support research on this task, the authors propose a benchmark dataset named T2IS-Bench, along with an evaluation framework called T2IS-Eval. They also present a training-free approach called AutoT2IS, which leverages pretrained diffusion transformers to generate consistent image sets. The authors demonstrate that AutoT2IS performs effectively across different domains than specifically made models, including multi-view generation, font generation, and character generation.
weaknesses: * The paper could better explain the position and purpose of T2IS-Bench relative to existing datasets. Since the benchmark focuses on various types of visual consistency (such as identity, style, and logic), it would be helpful to clarify why combining existing datasets, such as those for multi-view or character generation, would not achieve the same generalization goal. For example, using datasets all at once like DTH [1], MipNeRF-360 [2], or SerialGen [3] to achieve multi-view or personalized generation. The authors could strengthen their argument by emphasizing what unique gaps T2IS-Bench fills for generalizing models.
* The dataset quality seems uneven. As illustrated in Figure 1, there are noticeable inconsistencies in some samples. For instance, in the multi-view generation example, some visual elements (like the cylinder on the camera) appear in the top-left image but disappear in the bottom-right views, suggesting possible noise or imperfect alignment. Similarly, for character expression generation, the beard is not consistent and even disappears in the end. These inconsistencies might impact the reliability of the experiments and evaluation results.
* The paper mainly discusses AutoT2IS, a training-free approach, but does not explore how the dataset itself could support training generalized models. Since the dataset is presented as a contribution, it would be valuable to include examples or experiments that demonstrate how it can be used for model training or fine-tuning in addition to evaluation.

---

**Overall**: The paper is well-written and presents a compelling and original idea. However, the motivation and positioning of the proposed dataset are not fully convincing, and the discussion on how it might facilitate generalized model training is limited. Addressing these points would make the contribution stronger. At this stage, I would rate it a 6, but I am open to increasing the score if these concerns are clarified.

---

1. Aanæs et al. Large-scale data for multiple-view stereopsis. IJCV, 2016
2. Barron et al. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
3. Xie et al. SerialGen: Personalized Image Generation by First Standardization Then Personalization. CVPR 2025.