Group Diffusion Transformers are Unsupervised Multitask Learners

## Abstract

Abstract While large language models (LLMs) have revolutionized natural language processing with their task-agnostic capabilities, visual generation tasks such as image translation, style transfer, and character customization still rely heavily on supervised, task-specific datasets. In this work, we introduce Group Diffusion Transformers (GDTs) , a novel framework that unifies diverse visual generation tasks by redefining them as a group generation problem. In this approach, a set of related images is generated simultaneously, optionally conditioned on a subset of the group. GDTs build upon diffusion transformers with minimal architectural modifications by concatenating self-attention tokens across images. This allows the model to implicitly capture cross-image relationships ( e.g. , identities, styles, layouts, surroundings, and color schemes) through caption-based correlations. Our design enables scalable, unsupervised, and task-agnostic pretraining using extensive collections of image groups sourced from multimodal internet articles, image galleries, and video frames. We evaluate GDTs on a comprehensive benchmark featuring over 200 instructions across 30 distinct visual generation tasks, including picture book creation, font design, style transfer, sketching, colorization, drawing sequence generation, and character customization. Our models achieve competitive zero-shot performance without any additional fine-tuning or gradient updates. Furthermore, ablation studies confirm the effectiveness of key components such as data scaling, group size, and model design. These results demonstrate the potential of GDTs as scalable, general-purpose visual generation systems.

## 1 Introduction

Figure: Figure 1.1: Group Diffusion Transformers perform a vast array of visual generation tasks in a unified framework termed group generation. Note that NO task-specific dataset and NO additional gradient update is applied. The model is automatically generalized to these tasks after unsupervised training on image groups. For simplicity, textual descriptions of images are omitted here, which can be found in Appendix.
Refer to caption: x1.png

The advent of large language models (LLMs) has brought a paradigm shift in natural language processing (NLP) [, , , , , , , ], enabling a wide range of tasks to be approached in a task-agnostic manner. These models, trained on vast corpora, can generate coherent and contextually relevant content across various domains without the need for task-specific fine-tuning, setting a new standard for what is achievable in NLP. However, this level of task generalization has yet to be fully realized in the field of visual generation. Unlike NLP, visual generation tasks – such as pose transfer [, ], image translation [, ], customization [, ], stylization [, ], and font creation [, ] – remain largely siloed, relying heavily on supervised learning paradigms. These tasks often demand extensive task-specific datasets and additional modules, such as LoRAs [, , ], adapters [, ], visual encoders [, , ], and ControlNets [, ], to achieve satisfactory performance.

Figure: Figure 1.2: When conditioned on a subset of the group data, Group Diffusion Transformers could perform conditional group generation in the inpainting fashion. Note that the model is automatically generalized to these tasks after unsupervised training on image groups. Textual descriptions of images are omitted here (can be found in Appendix), and we summarize them into brief task descriptions.
Refer to caption: x2.png

This reliance on specialized data and architectures presents significant challenges for scalability and generalization. First, it limits scalability by failing to leverage the vast amount of weakly supervised data available on the Internet; creating and curating task-specific datasets is human-laboring. Second, it restricts models’ adaptability to unseen tasks. Third, cross-task adaptation is lacking, particularly in compositional control, where multiple tasks are implicitly managed. For example, consider creating a picture book, characters, environments, and attire must be dynamically adjusted, requiring decisions on which elements to keep consistent and which to vary. Finally, we hypothesize that training on single-task, shallow-domain datasets leads to the lack of generalization in real-world applications. To truly unlock the potential of visual generation, it is crucial to develop models capable of performing a wide range of tasks in a task-agnostic manner. This demands a shift in how we conceptualize and approach these tasks.

Our key insight is that most, if not all, visual generation tasks can be reformulated within a unified framework that we term the group generation problem. In this framework, the objective is to generate a set of correlated data, or a group, optionally conditioned on a subset of this group. For instance, tasks such as generating picture books [, ], font images [, ], or emoticons [] involve producing multiple images with distinct yet related descriptions simultaneously. The inherent correlations are implicitly captured through the relationships among these descriptions. Conversely, tasks like sketching [, ], colorization [, , ], character-specific image generation [, ], and multiview image generation from a single image [, ] can be framed as conditional group generation problems, where a subset of the group data is provided as a reference. Figure 1.1 and  1.2 provide examples of group generation and conditional group generation. By reframing these tasks as group generation problems, we leverage the power of unsupervised learning to address a broad spectrum of tasks without the need for task-specific supervision, simplifying the learning process and broadening applicability.

One of the most compelling advantages of the group generation framework is its natural alignment with the vast amount of data available on the Internet. Multimodal articles, image galleries, and multi-shot videos are just a few examples of readily accessible sources of group data. Each of these sources inherently captures the relationships between different data elements, offering a form of free supervision that is both scalable and diverse. The availability of such abundant group data not only reduces the need for labor-intensive data annotation but also enables the training of models on a wide array of tasks simultaneously, further enhancing generalizability.

To address the group generation problem, we introduce a minimalistic modification to diffusion transformers [, , ], termed Group Diffusion Transformers (GDTs). The core idea is to concatenate self-attention tokens across a group of inputs, allowing the model to learn the correlations and variations within the group. This modification is straightforward, requiring minimal changes to the underlying architecture of diffusion transformers (DiTs), yet it significantly enhances the model’s ability to capture relationships among multiple generated data. To address reference-based generation problems, such as style transfer [, ] and image translation [, ], we incorporate techniques like SDEdit [] and inpainting [, ]. These methods enable the model to generate the remaining elements of a group when conditioned on a subset of inputs. Figure 2.1 provides a detailed architectural overview of GDTs. The straightforward design of GDTs makes it easy to implement and shows promise for efficient scaling.

To evaluate the capabilities of our model, we first introduce a user interface that can automatically convert user instructions into textual descriptions of the target image group to support group generation. Then, we construct a comprehensive benchmark that covers a wide range of visual generation tasks, both with and without reference images. All tasks are performed in a zero-shot setting, without any parameter or architectural modifications. Despite the absence of task-specific supervision during training, our model demonstrates promising performance across most tasks in human rating and quantitative evaluation.
Finally, we conduct ablation studies to examine the impact of key components in our framework, such as data scale, group size, model design and quality tuning, on overall performance.

## 2 Approach

The core of our approach is to reformulate visual generation tasks into a group generation problem and solve it using minimally modified diffusion transformers. We begin by detailing how these tasks are reformulated, followed by a comprehensive introduction to our model, its architecture, the data employed, the training procedure, and the user interface for inference.

### 2.1 Problem Formulation

We propose that a vast array of visual generation tasks can be unified under a single framework we term the group generation problem. In this framework, the objective is to generate a group of $n$ elements $\mathbf{x}=\{\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{n}\}$, where each element is conditioned on its respective context (e.g., image descriptions) $\mathbf{c}=\{\mathbf{c}_{1},\mathbf{c}_{2},\cdots,\mathbf{c}_{n}\}$. The relationships among these elements are implicitly defined by the interdependencies within their contextual conditions. Optionally, a subset of $0\leq m<n$ elements of $\mathbf{x}$ can be provided as reference data, with the task being to generate the remaining $(n-m)$ elements. This formulation naturally encapsulates a variety of tasks:

- •
Text-to-Image: A special case where the group size $n=1$ and the reference subset size $m=0$. The task is to generate a single image from a textual description.
- •
Font Generation: Here, the group size $n>1$ corresponds to the number of characters to generate, with $m=0$.
- •
Picture Book Generation: Similar to font generation, the group size $n>1$ corresponds to the number of picture book pages, with $m=0$. The descriptions capture the connections and variations across the pages.
- •
Identity Preservation: Here, the group size $n>1$ corresponds to the number of photos with the same identities to generate, with $m=0$. Identity-specific information is reflected in the descriptions, such as names or other identifiers.
- •
Local Editing: In this task, the group size is $n=2$ with a reference subset size $m=1$. One reference image is provided, and the model generates the edited image based on the differences captured in their descriptions.
- •
Image Translation: Similarly, the group size is $n=2$ with a reference subset size $m=1$. A reference image from one domain is converted to another domain according to their descriptions.
- •
Subject Customization: The task involves generating $(n-m)\geq 1$ images, where $1\leq m<n$ character images are used as references.
- •
Style Preservation: In this task, $(n-m)\geq 1$ corresponds to the number of stylized images to be generated, with $m=1$ being the reference image guiding the target style.

These examples illustrate just a few of the many tasks that can be naturally expressed within the group generation framework. Across these tasks, the task hints are naturally embedded within the group element descriptions, much like how a human might communicate with a designer. This unified framework simplifies the approach to diverse visual generation tasks and paves the way for scalable, generalized solutions.

### 2.2 Model and Architectures

Figure: Figure 2.1: The overview of Group Diffusion Transformer, which takes minimal adaptations for the encoder-decoder and encoder-only visual generation architectures. We make a straightforward modification on self-attention blocks by concatenating image tokens across group inputs, allowing to learn cross-image correlations.
Refer to caption: x3.png

To tackle the group generation problem, it is crucial to establish connections between multiple group elements during the generation process, allowing the model to perceive and utilize the correlations among these elements. Our approach involves a straightforward modification: concatenating tokens across group inputs within the self-attention blocks of diffusion transformers. This enables tokens from different data elements to interact with one another throughout the model’s layers.

For different text-conditioned visual generation architectures, we make minimal adaptations to accommodate our approach:

- •
Encoder-Decoder: In architectures like PixArt [], each transformer block includes a self-attention operation for the image, cross-attention for interaction between image and text, and a feed-forward network. We choose to concatenate all the image tokens in self-attention blocks, which allows every token attends to all the image tokens within the group. After self-attention operation, concatenated image tokens are split back correspondingly. Then, in cross-attention blocks, each image token attends only to the text embeddings associated with its respective description. This setup is illustrated in Figure 2.1 (b).
- •
Encoder-Only: Examples like Stable Diffusion 3 [] and FLUX [] feature transformer blocks with self-attention blocks and feed-forward networks. We modify the self-attention operation into a masked version,
which is depicted in Figure 2.1 (c). Specifically, image tokens $\mathbf{x}_{i}$ as well as text tokens $\mathbf{c}_{i}$ are first concatenated with each other all over the group. Then, we perform the masked self-attention, where the mask is designed for allowing every image token attends to all tokens across the group while allowing context tokens only attend to their associated image tokens as well as themselves.
Concretely, let $M{(\mathbf{a}_{j},\mathbf{b}_{k})}$ indicate the attention mask for tokens in $\mathbf{a}_{j}$ and $\mathbf{b}_{k}$, where $\mathbf{a},\mathbf{b}\in\{\mathbf{c},\mathbf{x}\},0\leq j,k\leq n$. Then, $M{(\mathbf{a}_{j},\mathbf{b}_{k})}$ is decided by
$M{(\mathbf{a}_{j},\mathbf{b}_{k})}=\begin{cases}1&\text{if }(j=k)\text{ or }(
\mathbf{a}\in\mathbf{x}\text{ and }\mathbf{b}\in\mathbf{x})\\
0&\text{else}\end{cases}.$
(1)

### 2.3 Training Dataset

We focus on image-related tasks in this work, which requires a high-quality, large-scale, and diverse image group dataset. While existing multimodal datasets like MINT-1T [] are large, they fall short of our pretraining needs due to low image quality and biased group type distribution relative to real-world visual generation applications. Thus, we construct our own dataset by sourcing image groups from multimodal Internet articles.

Figure: Figure 2.2: Distribution of group size in our training dataset.
Refer to caption: x4.png

Our dataset creation process involve several key steps: (1) We collect a substantial amount of multimodal data, extracting images while preserving their original order to maintain group integrity. (2) A small subset of these image groups is manually annotated as either positive (suitable for retention) or negative (to be discarded). (3) Using these annotations, we train a binary classifier to score and filter the collected image groups. (4) We perform deduplication across and within groups to eliminate redundant groups and images. After preprocessing, we compile a dataset of approximately 500,000 image groups, with the distribution of group size illustrated in Figure 2.2.

The next crucial step is to generate descriptions that accurately capture the correlations among the images within each group. To achieve this, we utilize our internal multimodal large language models (MLLMs), iteratively testing and refining prompts to ensure the generated descriptions are stable and applicable across different group types. In Figure 2.3, we show the prompt we used, as well as the resulting group image descriptions.

While pretraining on our large-scale dataset provides a solid foundation for learning correlations with Group Diffusion Transformers (GDTs), it is common practice in visual generation tasks to conduct a supervised fine-tuning stage to enhance generation details and aesthetics. To this end, we curate a smaller, high-quality subset of approximately 10,000 image groups. These groups were selected for their strong correlations, high image quality, aesthetic appeal, and diversity. Fine-tuning our pretrained models on this curated dataset significantly improves both the image quality and content consistency in group generation, where the comparison can be found in Section 4.2.5.

Figure: Figure 2.3: Example of our training dataset, where the group images are captioned through prompting our internal MLLMs.
Refer to caption: x5.png

### 2.4 Training Process

We initialize the Group Diffusion Transformers (GDTs) with weights from pre-trained text-to-image models, such as PixArt-$\alpha$ [] and Stable Diffusion 3 []. Since GDTs introduce no additional parameter to the existing diffusion transformers, the pretrained weights are fully compatible. During both pretraining and supervised fine-tuning, we uniformly sample group sizes ranging from 1 to 4, dynamically adjusting the batch size to maintain consistent GPU memory usage. This approach ensures balanced performance across different group sizes. The model undergoes pretraining for approximately 100,000 steps, followed by fine-tuning on a curated dataset for around 5,000 steps. All training is conducted on A100 GPUs. We adopt the same hyperparameter settings as the official models in PixArt-$\alpha$ and Stable Diffusion 3.

### 2.5 User Interface

Considering it is tedious to write a group of prompts in the inference stage, we build a user interface to provide a convenient interaction with GDTs. As illustrated in Figure 3.1, we follow the pipeline of [Instruction] $\rightarrow$ [Group Prompts] $\rightarrow$ [Generated Images] for group generation, and [IMGs] + [Instruction] $\rightarrow$ [Group Prompts] $\rightarrow$ [Generated Images] for conditional group generation. Specifically, we prompt MLLMs to convert the user instruction into group prompts, where the MLLMs are asked to analyze the number of group prompts and the corresponding tasks. For example, if the instruction is “Draw a line sketch of a female character and the corresponding colored photo”, the MLLM can deduce that this instruction should be transformed into two prompts, categorizing the task as sketch coloring.

## 3 Benchmark

Figure: Figure 3.1: We build a user interface that automatically converts the user instruction into group prompts using MLLMs, which is useful in the inference stage of GDTs.
Refer to caption: x6.png

Figure: Figure 3.2: Overview of our benchmark, covering about 30 distinct types of generation tasks.
Refer to caption: x7.png

Given the diverse nature of visual generation tasks, evaluating the performance of our GDTs presents unique challenges. Therefore, we design a benchmark (^1^11Benchmark is available at [Google Drive](https://drive.google.com/drive/folders/1e7muQDqJ-kV4H5yzGMCzCHORpz3ut1CN?usp=sharing).) that spans a wide array of tasks as shown in Figure 3.2. Specifically, our benchmark consists of over 200 instructions, each corresponding to one of 30 distinct types of visual generation tasks. This diversity enables a thorough assessment of the generalization capabilities of GDTs across various scenarios.

This evaluation suit encompasses tasks such as identity preservation, local editing, subject customization, font generation, stylized group generation, and step-by-step generation. Among these coarse-grained categories, further fine-grained tasks are expanded. For example, step-by-step generation contains subtasks like story telling [], painting process [], and growth process. Besides, all the textual descriptions in this benchmark are created through our user interface.

## 4 Results

### 4.1 User Study

**Table 4.1: User study on our benchmark. Human evaluation on three questions in a five-point scale.**
| Models | Q1 | Q2 | Q3 |
| --- | --- | --- | --- |
| group generation |  |  |  |
| PixArt-$\alpha$ | 3.44 | 3.89 | 3.78 |
| Stable Diffusion 3 | 3.20 | 3.35 | 3.29 |
| conditional group generation |  |  |  |
| PixArt-$\alpha$ | 3.15 | 3.56 | 3.68 |
| Stable Diffusion 3 | 3.02 | 3.27 | 3.34 |

Figure: Figure 4.1: Qualitative results of GDTs on our benchmark, including group generation and conditional group generation.
Refer to caption: x8.png

We first qualitatively evaluate the generated results of GDTs on our proposed benchmark as shown in Figure 4.1. GDTs could perform both group generation and conditional group generation according to the user instructions. Note that the task scope of this benchmark is effectively limited by our imagination, but thanks to our task-agnostic pretraining, GDTs can theoretically be generalized to arbitrary visual generation tasks.

In our user study, we mainly adopt human ratings to assess the performance of GDTs on the benchmark. Three questions are included to measure the prompt following ability, content consistency within the image group, and the overall instruction following ability, namely: Q1: Prompt following on each image within the group: Q2: Content consistency among generated group images, regardless of prompts, Q3: Instruction following on the generated group images. Evaluators are asked to rate on three questions in the scale from 1 to 5, where 5 signifies perfection and 1 denotes the lowest quality. The final evaluation score is derived from the average ratings across all tasks, which serves as a robust indicator of the overall performance and its potential for real-world applications. The human-rated results are illustrated in Table 4.1, where GDTs achieve overall satisfaction (higher than 3) on all of the three questions.

### 4.2 Ablation Analysis

#### 4.2.1 Metrics

While our benchmark with over 200 instructions could well evaluate model’s capabilities on a five-point scale, we would like to compare these ablated models in a more nuanced and quantitative manner in our ablation experiments. Therefore, we mainly present the objective metrics like FID and CLIP score. To be specific, we measure image fidelity by calculating FID on the validation set using 50k images. We assess content consistency and prompt adherence within each group by averaging CLIP similarities across every image-image and image-text pairs, respectively.
In terms of reference-based generation, we adopt the same metrics but exclude pairs that involve the reference images themselves, as well as pairs between reference images and their corresponding texts.

**Table 4.2: Performance evaluation on key components of GDTs. We investigate the impacts of data scale, group size, model design, and quality tuning on encoder-decoder and encoder-only models.**
| Settings | PixArt-$\alpha$ (Encoder-Decoder) | Stable Diffusion 3 (Encoder-Only) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| FID-50k | Content<br>Consistency | Prompt<br>Adherence |  | FID-50k | Content<br>Consistency | Prompt<br>Adherence |  |
| Data Scaling |  |  |  |  |  |  |  |
| 5k groups | 8.40 | 0.747 | 0.291 |  | 8.95 | 0.740 | 0.298 |
| 50k groups | 12.06 | 0.767 | 0.293 |  | 10.92 | 0.760 | 0.302 |
| 500k groups | 15.91 | 0.778 | 0.300 |  | 11.30 | 0.761 | 0.305 |
| Group Size |  |  |  |  |  |  |  |
| groupsize = 2 | 15.69 | 0.784 | 0.299 |  | 12.37 | 0.763 | 0.301 |
| groupsize = 4 | 18.19 | 0.761 | 0.291 |  | 13.85 | 0.739 | 0.298 |
| groupsize = 8 | 48.26 | 0.701 | 0.252 |  | 18.28 | 0.701 | 0.290 |
| Inpainting |  |  |  |  |  |  |  |
| SDEdit | 15.71 | 0.702 | 0.299 |  | 12.15 | 0.751 | 0.303 |
| trainable | 10.91 | 0.725 | 0.287 |  | 10.94 | 0.755 | 0.298 |
| Quality Tuning |  |  |  |  |  |  |  |
| before | 15.91 | 0.778 | 0.300 |  | 11.30 | 0.761 | 0.305 |
| after | 12.53 | 0.792 | 0.298 |  | 10.03 | 0.781 | 0.303 |

#### 4.2.2 Data Scaling

Without the demand of task-specific supervision, it is quite easy to acquire a large abundance of group data from the Internet. We scale the training data to 5k, 50k, and 500k groups, to explore the impact of data scale in GDTs. As illustrated in Table 4.2, with the increase of the amount of training data, GDTs behave increasingly better in content consistency and prompt adherence. Interestingly, we find that FID would become lower when training on less data, which may be that it is easier to overfit to small datasets. We plan to further scale up our data to the level of hundreds of millions of groups in the future, in order to fully leverage the potential of GDTs.

#### 4.2.3 Group Size

We gradually increase the upper limit of group size to 2, 4, and 8, and perform inference based on that limit. Note that doubling the group size will, in turn, double the sequence length in self-attention, leading to a corresponding increase in computational complexity, so we cap the maximum group size at 8 in our ablation. From the ablated results in Table 4.2, we find that larger group sizes lead to a more pronounced performance decline in image quality, content consistency, and prompt adherence. The reason may be that it is more difficult to learn the complex relationships across a large group of images. Besides, the scarcity of data of large group sizes prevents the model from being adequately trained. In the future, we would greatly scale our training data.

#### 4.2.4 SDEdit or Inpainting

When conditioned on a subset of the group data, using methods like SDEdit [] or trainable inpainting [, ], GDTs can be instructed to generate the remaining data of the group. Specifically, SDEdit is a training-free technique which provides the reference images that are added with the same noise step as the generated images during the denoising stage. In contrast, trainable inpainting concatenates the reference image to the noised one in channel dimension, allowing the model to “copy” the reference images and generate the remaining ones. In our ablation study, as illustrated in Table 4.2, it is observed that trainable inpainting performs better in image quality and content consistency, while the training-free SDEdit is good at prompt adherence. We adopt the model design of trainable inpainting in our GDTs.

#### 4.2.5 Quality Tuning

While quality tuning is a common practice in visual generation models to enhance aesthetic appeal, we investigate its impact under the paradigm of group generation. As illustrated in Table 4.2, after the supervised fine-tuning on a small subset of high-quality image groups, GDTs exhibit significantly better image quality. We also find that quality tuning helps generating image groups with higher content consistency, while barely compromising the adherence to textual descriptions.

## 5 Related Work

### 5.1 Text-to-Image Generation

The emergence of DDPM [] has catalyzed rapid advancements in text-to-image (T2I) generation. Earlier frameworks focused on T2I generation in pixel space, exemplified by GLIDE [] and Imagen []. In contrast, Stable Diffusion [] introduced latent space for T2I generation, while DALLE-2 (unCLIP)[] expanded this to a multimodal latent space. EMU[] demonstrated that supervised fine-tuning on a small set of appealing images can significantly enhance generation quality. Unlike U-Net architectures, several approaches, including DiT [], Pixart [], HunyuanDiT [], and SD3 [], adopt transformers as their backbone.

### 5.2 Controllable Text-to-Image Generation

Personalization. The personalization task [, , , , ] seeks to capture and utilize concepts as generative conditions, such as subject [, ], person [, , , ], style [, ], and image [, , ]. Textual Inversion [] embeds user-provided concepts into optimized words within the text embedding space, while DreamBooth [] uses low-frequency words to represent concepts, subsequently updating UNet parameters. Subject-driven methods utilize additional encoders to represent facial identity information, as seen in Face0 [] and DreamIdentity [], which employ pre-trained face recognition models. A variety of personalization technologies continue to emerge to satisfy diverse application needs.

Spatial Control. Given the challenge of accurately representing the spatial structure of generated images through text [], such as layout [, , ], human pose, and segmentation masks, controllable T2I generation with various positional signals is garnering increasing attention in AIGC literature [, , , , ]. ControlNet [] introduces an additional encoder with ”zero-initialization,” while UniControl [] encodes conditions using mixtures of experts (MoE) adaptors.

Advanced Control. To address the complex needs of real applications, controlled generation technology is evolving in several new directions. Attend-and-Excite [] refines cross-attention mechanisms to enhance text alignment. Composer [] and Cocktail [] employ joint training for multi-condition generation. Cones [] fine-tunes concept neurons post-personalization to improve quality and enable multi-subject generation. Universal Guidance [], EMU2 [], and FreeDom [] strive to achieve universal controllable T2I generation.

### 5.3 Attention Adjustment in Diffusion Models

Some previous works [, , , ] have shown that adjusting the attention operation in diffusion models could help preserve the content and style across the generated images. Particularly, JeDi [], StyleAligned [], and StoryDiffusion [] modify the self-attention operation to allow multiple images to attend to each other. Although GDTs adopt a similar design, we are fundamentally different with the following capabilities: (1) general-purpose visual generation system; (2) no need for task-specific learning or finetuning; (3) easily scalable, unsupervised visual pretraining; (4) enabling zero-shot task generalization.

## 6 Conclusion and Limitations

We reformulate most visual generation tasks into a group generation problem, thereby introducing a unified framework named Group Diffusion Transformers (GDTs). We present that with scalable, unsupervised, and task-agnostic pretraining on group data, GDTs could achieve competitive zero-shot performance on a vast array of visual generation tasks. Our results demonstrate the potential of GDTs as scalable, general-purpose visual generation systems.

Moreover, we point out that there is still a discrepancy in image quality between GDTs and the state-of-the-art text-to-image models. The amount of group data for pretraining is also not sufficient yet, which has not fully unleashed the model’s capabilities. We are optimistic that with an enlarged group dataset, we can further optimize the model’s performance and reduce the discrepancy. In the future, we also plan to extend the time dimension of GDTs to enable multi-shot video generation, which can be naturally expressed under our group generation framework.

## Appendix A Appendix

**Table A.1: Textual descriptions of the images in Figure 1.1.**
| Image | Description | Image | Description |
| --- | --- | --- | --- |
|  | The image presents a large three-dimensional letter “J”, made of polished metal with a smooth surface and a strong mirror reflection. The interior of the letter reflects a blurry cityscape or mountain range, primarily in dark blue and gray-white tones. The letter is placed against a rough gray cement wall, creating a sharp contrast that enhances the metallic texture and three-dimensionality. |  | The image is dominated by a large three-dimensional letter “I”, made of polished metal with a mirror effect, reflecting the surrounding light. Inside the letter is a clear black and white silhouette of a city, with densely packed buildings. The bottom is dark blue, contrasting with the bright white above. The background is a rough gray cement wall, creating visual tension with the metallic letter. |
|  | The image shows a large three-dimensional letter “Y”, also made of polished metal with a mirror reflection. The letter reflects a relatively clear city outline; the buildings are arranged irregularly, and the overall color is dark. The bottom of the letter transitions to dark blue, contrasting with the gray cement wall in the background, creating an industrial and modern feel. The letter is three-dimensional and has a strong texture. |  | The main subject of the image is a transparent orange letter “B”, with a fine water droplet-like texture on its surface, giving it a crystal-clear texture. The letter “B” is located in the center of the image, surrounded by bright orange rays of light radiating outwards, creating a dazzling light effect. Small orange droplets are scattered in the background, echoing the texture on the surface of the letter. The overall tone is warm, bright, and full of vitality. |
|  | The main subject of the image is a transparent orange letter “O”, with a fine water droplet-like texture on its surface, giving it a crystal-clear texture. The letter “O” is located in the center of the image, surrounded by bright orange rays of light radiating outwards, creating a dazzling light effect. Small orange droplets are scattered in the background, echoing the texture on the surface of the number. The overall tone is warm, bright, and full of vitality. |  | A young woman with long red hair is shown with her eyes closed, wearing a beige sweater with a white collar underneath and round glasses. The background features a purple backdrop, and her hair is blowing in the wind. |
|  | A young woman with long red hair is shown wearing a beige sweater with a white collar underneath and round glasses. She turns her head to look at the camera, with a sunset and a purple backdrop in the background. |  | A young woman with long red hair is shown with her eyes closed, wearing a beige sweater with a white collar underneath and round glasses. The background features a sunset and a purple backdrop. |
|  | In the photo, a young couple is all smiles. The girl is wearing a white slip dress and the boy is wearing a black suit with a white shirt and black tie. They are holding hands and look very sweet and happy. |  | In the photo, a young couple cuddling intimately. The girl is wearing a white slip dress and the boy is wearing a black suit. They are looking into each other’s eyes and the girl has a sweet smile on her face, overflowing with happiness. |
|  | In the photo, a young couple is in an intimate pose. The girl is wearing a white slip dress with her back to the camera, revealing her bare back. The boy is wearing a black suits and looking at the girl’s back affectionately. |  | A young woman floats in the water, her head slightly lifted and her gaze directed upwards. Her dark hair spreads around her, contrasting sharply with her fair skin and vibrant red lipstick. The water surrounds her face and hair, creating a dreamlike atmosphere. |
|  | A young woman floats in the water with her eyes closed. Her dark hair spreads around her, contrasting with her fair skin and vibrant red lipstick. The water surrounds her face, creating a peaceful and serene atmosphere. |  | A young woman leans on a chest, gazing directly at the viewer. She has fair skin, her brown hair falls loosely around her shoulders, and her dark eyes hold a gentle expression. She wears a light-colored shirt, the neckline slightly open, revealing the graceful lines of her neck. |
|  | A young woman stands by the window, her head tilted slightly upwards as if feeling the caress of a gentle breeze. She has fair skin, her brown hair falls naturally, and her dark eyes are filled with hope for the future. She wears a light-colored shirt with loose, comfortable sleeves and dark pants, creating a simple and elegant look. |  | A fashion photography piece showcasing a female model in a pink printed maxi dress layered over with a long white furry coat. The model has long brown hair and sophisticated makeup; she stands against a light-colored backdrop in an elegant pose with her hands in her pockets. The dress has a V-neck design, is lightweight in texture, and features a soft print. The coat is fluffy and soft, with a high-end feel. The model is wearing gold strapped heels. |
|  | Fashion photography showcasing a female model in a champagne-colored strapless A-line dress. The dress is smooth in texture, with a strong drape, and an A-line skirt, cinched at the waist with a golden belt. The model’s hairstyle and makeup are consistent with the previous image, and the background is similarly simple and light-colored. The model’s pose is elegant, with her gaze directed forward. She wears gold heels that complement the dress. |  | A fashion photography image displaying a female model in a burgundy velvet maxi dress. The dress is a halter neck style, with a defined waist, and an asymmetric hemline, trailing on one side while revealing the ankle on the other. The dress is thick in texture, with a rich color, and the skirt flows slightly. The model’s hairstyle and makeup remain consistent, against a simple light-colored background. The model’s posture is elegant, and she wears gold high-heeled shoes. |
|  | A fashion photography piece featuring a female model in a purplish-red velvet slip dress. The dress is simply cut and form-fitting, accented with a thin silver belt at the waist. The model maintains the consistent hairstyle and makeup from other images, against a light-colored background. The model poses elegantly, with a confident smile. Her makeup is refined, with simple earrings. The model wears light gold heeled shoes. |  | A futuristic cyberpunk car faces the camera, showcasing its front design. The car boasts sleek lines, intricate details, and dazzling lighting effects. The backdrop is a technologically advanced cityscape at night. |
|  | A futuristic cyberpunk car is seen from above, showcasing its roof design and overall layout. The car boasts sleek lines, intricate details, and dazzling lighting effects. The backdrop is a technologically advanced cityscape at night. |  | A futuristic cyberpunk car is presented sideways, showcasing its side profile and streamlined body. The car boasts sleek lines, intricate details, and dazzling lighting effects. The backdrop is a technologically advanced cityscape at night. |
|  | The image shows a modern minimalist living room from an overhead perspective. A beige fabric L-shaped sofa occupies the center of the space, complemented by a dark wood coffee table and a light-colored rug. Decorative items are placed on the coffee table. A floor-to-ceiling window dominates one wall, showcasing lush greenery outside. An abstract artwork hangs on the wall, and a wooden side table is visible in a corner. The overall color scheme is soft, creating a relaxed and comfortable atmosphere. |  | The image depicts a modern minimalist living room. A beige fabric sofa is centrally positioned, featuring smooth lines and soft, comfortable material. A dark wood coffee table with a simple and elegant design sits in front of the sofa. A floor-to-ceiling window occupies one wall, revealing a pleasant outdoor view. An abstract artwork hangs on the wall, next to a wooden side cabinet displaying a blue vase. The overall space is spacious and bright, with a clean and sophisticated design style. |
|  | A front view of a Minecraft-style game scene, the picture is full of pixelated style. In the foreground is green grass dotted with colorful flowers. Not far away is a house built of oak, with a path paved with stone bricks in front of the house. Around the house are planted several lush trees with leaves in different shades of green. Several white clouds float in the sky. |  | A side view of a Minecraft-style game scene, the picture is full of pixelated style. In the foreground is green grass dotted with colorful flowers. The grass stretches into the distance, where it meets the blue sky. On the left side of the picture, you can see a house built of oak, with a path paved with stone bricks in front of the house. Several lush trees are planted next to the house, with leaves in different shades of green. Several white clouds float in the sky. |
|  | A top view of a Minecraft-style game scene, the picture is full of pixelated style. In the center of the picture is a green grass with colorful flowers dotted on it. There is a house built of oak on the grass, with a path paved with stone bricks in front of the house. Several lush trees planted around are the house, with leaves in different shades of green. A winding river can be seen beside the house. Several white clouds float in the sky. |  | A fluffy squirrel, dressed in a brown plaid shirt and green pants, stands on the grass. It has a wide grin, wide-open eyes, and holds a giant pine cone, looking delighted. |
|  | A fluffy squirrel, dressed in a brown plaid shirt and green pants, sits on a branch. It has a wide grin, wide-open eyes, and looks happy. A small bird sits on the branch next to it, and another bird sits on a higher branch. |  | A fluffy squirrel, dressed in a brown plaid shirt and green pants, stands on the grass. It has wide-open eyes and a slightly open mouth, looking surprised. |
|  | A fluffy squirrel, dressed in blue overalls, lies on top of a giant pine cone. It has a frown, wide-open eyes, and a slightly open mouth, looking angry. |  | The picture depicts a sudden downpour in the countryside. Large raindrops fall from the sky as people on the hillside run downhill to seek shelter. The strong wind bends the trees and blows fruit to the ground. A car drives through the rain on a country road. |
|  | The picture depicts a bountiful harvest scene in the countryside. Golden wheat fields stretch as far as the eye can see, with rolling mountains and a few houses in the distance. A scarecrow stands in the wheat field, guarding the upcoming harvest. A few birds perch on a fence by the field, seemingly admiring the beautiful scenery. |  | The picture depicts a tranquil rural scene. A country road winds through the picture, flanked by several houses surrounded by lush greenery. On a table by the road lie slices of watermelon with a few birds pecking at the crumbs on the ground. A dog lies under the table, seemingly anticipating the delicious treat. |
|  | The picture depicts a busy harvest scene in the countryside. In the golden wheat fields, farmers are diligently harvesting the ripe wheat. They tie the wheat stalks into bundles and stack them at the edge of the field. In the distance are rolling mountains, with houses and farmland at the foot of the mountains, forming a beautiful rural landscape. |  | A blue-green car is driving on the road with two rabbits in it. |
|  | A red car is driving on the road with two mice in it. |  | A blue car is driving on the road with four animals inside: a giraffe, a tiger, a frog, and a teddy bear. |
|  | A cartoon capybara wearing a blue uniform and hat and carrying a mailbag sits on a bench next to a bus stop sign that reads “BUS”. The capybara is surrounded by flowers and plants. |  | A cartoon capybara wearing a blue uniform and hat and carrying a mailbag sits on a bench next to a bus stop sign that reads “BUS”. The capybara is surrounded by flowers and plants. |
|  | A cartoon capybara wearing a blue uniform and hat and carrying a mailbag stands atop a turtle walking through a garden of tulips and other plants. The capybara has a stem of a plant in its mouth, and an envelope files behind it. |  | An orange fox with closed eyes snuggles next to a pipa, surrounded by blooming blue flowers, with a golden full moon and twinkling stars in the background. |
|  | An orange fox wearing headphones sits with its eyes closed on a grassy field, next to a purple vintage cassette player. Surrounding it are red mushrooms and plants bearing red fruits. |  | An orange fox is peacefully sleeping with its eyes closed in a blue flower bed, next to a golden saxophone. A blue bird is joyfully singing on top of the saxophone. |
|  | A girl stands against a white background,<br>smiling and holding up her right hand in a peace sign. She wears a beanie hat, and her long hair falls over her shoulders. She’s dressed in a vest over a long-sleeved T-shirt, and loose-fitting pants. A bag hangs from her shoulder, and sneakers complete her casual look. |  | A girl stands against a teal background, smiling and holding up her right hand in a peace sign. She wears a blue beanie hat, and her long brown hair falls over her shoulders. She’s dressed in a red and blue striped vest over a white long-sleeved T-shirt, and loose-fitting blue jeans. A blue bag hangs from her shoulder, and white sneakers complete her casual look. |
|  | On a white sheet of paper, a pencil sketch of a bouquet of flowers. The bouquet consists of two blooming sunflowers and some green leavers and small flowers wrapped in wrapping paper and tied with a bow. |  | On a cream-colored sheet of paper, all areas of a bouquet are painted with watercolor paints. The bouquet consists of two blooming sunflowers and small flowers, wrapped in wrapping paper and tied with a bow. |
|  | On a sunny day, on a green grassland, several houses can be seen in the distance, with blue sky and white clouds dotted the sky. In the center of the screen, a small sapling has just been planted in the soil. It looks very fragile, but full of vitality. |  | On a sunny day, on a green grassland, several houses can be seen in the distance, with blue sky and white clouds dotted the sky. In the center of the screen, the once weak sapling has grown much taller, about 1 meter high, and its branches and leaves are even more lush. |
|  | On a sunny day, on a green grassland, several houses can be seen in the distance, with blue sky and white clouds dotted the sky. In the center of the screen, the tree has grown to 2 meters high, with lush branches and leaves, full of vitality. |  | On a sunny day, on a green grassland, several houses can be seen in the distance, with blue sky and white clouds dotted the sky. In the center of the screen, the tree has grown into a 3-meter-tall tall tree, with lush branches and leaves, covering the sky and sun. Standing west, it exudes strong vitality. |
|  | A little girl standing on a cliff edge, looking up at a cluster of red plants with a glowing sphere in the center. Below the cliff is a black forest, with a white background speckled with red plants. |  | A winged girl holding a glowing sphere, surrounded by swirling red lines and stars. Her background is black, dotted with red plants. |
|  | A girl with her eyes closed, a bare tree on her head. There are small houses around the tree, set against a black background adorned with red leaves and triangles. |  | A winged girl, mouth open, looking up at a cluster of red plants with a glowing sphere in the center. Beneath her are small houses and black plants. |
|  | A cartoon polar bear wearing a red floral headscarf and a red bow tie, with a serious expression, stands in the center of a purple striped background. |  | A white drawstring backpack with a cartoon polar bear wearing a red floral headscarf and a snowman with a red scarf and a carrot nose printed on it, with “Merry Christmas” written above. |
|  | The picture shows a cute little dog with fluffy fur, round eyes, and a lively nose, smiling kindly at the camera. The puppy’s fur color is light brown, and its body posture appears very relaxed. Its limbs naturally hang on both sides of its body, and its tail gently sways, as if welcoming its owner’s arrival. The background of the puppy is a green lawn, with sunlight shining through the clouds, illuminating the entire scene. |  | The picture shows a cute cartoon style backpack with a blue themed color and some cute patterns printed on it. On the front of the backpack, there is a photo of a small dog with fluffy fur, round eyes, and a lively nose. It is facing the camera with a friendly smile. The photo of the puppy is printed in the center of the backpack, with a moderate size that matches the overall style of the backpack. The background of the backpack is a bright yellow color, creating a lively atmosphere. |
|  | A carton character, resembling Buddha, wearing a golden lotus headdress and pink monk robes. His hands are hidden in his sleeves. He has a slight smile and closed eyes, showing contentment. |  | A carton character, resembling Buddha, wearing a golden lotus headdress and pink monk robes. His hands are hidden in his sleeves. He has a furrowed brow and downturned mouth, showing sadness. |
|  | A carton character, resembling Buddha, wearing a golden lotus headdress and pink monk robes. His hands are hidden in his sleeves. He has a big smile and closed eyes, showing joy. |  | A carton character, resembling Buddha, wearing a golden lotus headdress and pink monk robes. His hands are hidden in his sleeves. His eyes are closed and his eyebrows are slightly furrowed, showing exhaustion. |
|  | A photo of a square glass bottle containing a light brown liquid. The bottle is sealed with a wooden stopper and has a label with Chinese characters and design elements. The bottle is surrounded by green plants, creating a natural feel. |  | A photo of a square glass bottle containing a light brown liquid. The bottle is sealed with a wooden stopper and has a label with Chinese characters and design elements. It’s set against a dark background and surrounded by scattered orange petals, creating a professional studio ambiance. |
|  | A product photography image with a warm orange background. The main focus is a golden gift box with a fan-shaped cutout pattern and landscape painting decorations; three golden mooncakes are visible inside. Next to the gift box is a red tray, also containing three mooncakes. Abstract white mountain-shaped decorations are on the right, and a traditional Chinese red fan and a teacup with Chinese-style patterns are on the left. The overall tone is warm and festive, highlighting the product’s high-end feel and traditional cultural elements. |  | A product photography image with a warm orange background. The center of the image features a white gift box with a fan-shaped cutout pattern and landscape painting decorations; three golden mooncakes are visible inside. Abstract white mountain-shaped decorations are on the right. On the left is a red rectangular object and a red round tray with a mooncake on it. The overall style of the image is simple and clear, highlighting the elegance and sophistication of the gift box. The color scheme is harmonious, creating a comfortable visual experience. |

**Table A.2: Textual descriptions of the images in Figure 1.2.**
| Image | Description | Image | Description |
| --- | --- | --- | --- |
|  | This painting depicts a retro Volkswagen van with luggage and surfboards loaded on the roof. In front of the van are two lush palm trees, creating a strong summer beach vacation atmosphere. Below the van, there is a blank banner with a budding rose flower below, surrounded by branches and leaves. The entire painting is outlined with clear lines, contrasting black and white, and has a minimalist style. |  | This painting depicts a red and white vintage Volkswagen van with luggage and surfboards loaded on its roof. In front of the van are two lush palm trees, creating a strong summer beach vacation atmosphere. Below the van is a yellow banner that reads “TRAVEL WITH YOU”. Below the banner is a blooming red rose surrounded by branches and leaves. The painting is vibrant and full of energy. |
|  | This is a photo of an underground parking garage with a white Lamborghini sports car parked. The sports car has smooth lines and a dynamic shape, and its body is pure white, which is particularly striking under the light. The floor of the parking lot is smooth and reflects the light. |  | This is a photo of an underground parking garage with a blue Lamborghini sports car parked. The sports car has smooth lines and a dynamic shape, and its body is dark blue, shining with a metallic luster under the light. The floor of the parking lot is smooth and reflects the light. |
|  | In the picture, a little girl wearing a yellow and white striped shirt and blue pants is dancing happily. She wears an orange baseball cap, with her hair tied in a cute side ponytail. She stands on her toes with her left foot, her right leg extended backward, her left arm naturally bent, and her right arm raised high as if trying to touch the sky, a joyful smile on her face. |  | In the picture, a little girl wearing a green and black striped shirt and brown overalls is taking a leisurely stroll. Her long black hair is tied up in a high bun, with a few strands casually falling down. She wears a pair of brown ankle boots, giving her a stylish and playful look. She walks with a brisk pace, slightly turning her body to the side, her eyes curiously gazing into the distance, as if filled with interest in everything around her. |
|  | The central focus is a close-up of a girl’s head and shoulders. She has shoulder-length, slightly wavy brown hair with bangs covering her eyebrows, her eyes are closed, and her expression is calm and serene, with rosy cheeks. She wears a dark blue top with light yellow accents at the collar. The overall style is soft, with warm colors, smooth lines, and visible brushstrokes. The background is pure white, highlighting the main subject. A simple “zom” is printed at the bottom. |  | The central focus is a close-up of a girl’s head and shoulders. She has shoulder-length, slightly wavy brown hair with bangs covering her eyebrows and is wearing round gold-rimmed glasses. Her eyes are wide open, she is smiling, and her cheeks are rosy. She wears a dark blue top with white polka dots and light yellow accents at the collar. The overall style is soft, with warm colors, smooth lines, and visible brushstrokes. The background is pure white, highlighting the main subject. |
|  | A young woman with long, dark hair cascading over her shoulders wears a light-colored, off-the-shoulder dress, her delicate features clearly visible. She sits on a bamboo mat, both hands gently touching her hair, her gaze soft as she looks directly at the camera. |  | A young woman with long, dark hair cascading over hers shoulders wears a light-colored, off-the-shoulder dress, her delicate features clearly visible. She sits on a bamboo mat, her right hand gently touching her hair, her gaze soft as she looks directly at the camera. |
|  | A cartoon-style portrait of a little girl, drawn in pencil sketch style. She is wearing a pointed hat and a pair of bib pants with two buttons. The little girl has short hair to her shoulders, her eyes closed in two curved arcs, her mouth slightly open with a happy smile. |  | A cartoon-style portrait of a little girl, colorfully drawn. She is wearing a sky blue pointed hat with a circle of white stripes. The little girl is wearing a black and white stripped top with sky blue overalls decorated with two buttons. The little girl has short hair to her shoulders, thick brown hair naturally hanging down. The little girl has two big eyes, her mouth slightly open with a happy smile. |
|  | A girl with black bob hair and bangs is wearing a black beret. She has large eyes, a round face, rosy cheeks, and is wearing silver earrings. She is wearing a brown vest with white cloud patterns over a blue turtleneck. |  | A girl with black bob hair and bangs is wearing a black checkered beret. She has large eyes, a round face, rosy cheeks, and is wearing gold earrings. She is wearing a yellow sweater with the words ”byebye” printed on it and green striped collar and cuffs. |
|  | {CJK*}<br>UTF8gbsnA product display image with a pure white background. A cylindrical metal can is placed in the center. The can is white with a delicate textured pattern on its surface, resembling clouds or petals, with a golden metallic sheen. The lid is made of rose gold metal. The center of the can features a golden brand logo, including the brand name “PHYSICAL ART” and the Chinese brand name “悠然”. The bottom of the can is marked with “CONCERT 45ML”. The overall style is simple, elegant, and refined. |  | {CJK*}<br>UTF8gbsnA product display image with a pure white background. Three identical cylindrical metal cans are arranged side by side. The cans are white with a delicate textured pattern on their surface, resembling clouds or petals, with a golden metallic sheen. The lids are made of rose gold metal. The center of each can features a golden brand logo, including the brand name “PHYSICAL ART” and the Chinese brand name “悠然”. The bottom of each can is marked with “CONCERT 45ML”. The overall style is simple, elegant, and refined. |
|  | The center of the image is a chocolate cake. The bottom of the cake is red, the top is covered with thick chocolate sauce, and it’s decorated with fluffy cream and two fresh strawberries. The cake rests on a dark base. Above the image is the title “Strawberry”, with an elegant and flowing font. The background is simple, the overall style is fresh and sweet, creating a comfortable visual experience. The watercolor painting technique gives the image a soft color transition and a light texture. |  | The main subject of the image is a slice of strawberry tart. The tart crust is golden yellow, and it’s topped with bright red strawberries, decorated with a few blueberries and cherries. There’s a golden chocolate decoration on the strawberries. The cross-section of the tart shows a rich layering and the texture of the filling. Above the image is the title S̈trawberry,̈ consistent with the first image. The background is equally simple, and the watercolor painting technique creates a light and dreamy atmosphere. |
|  | This illustration depicts a huge rock standing on the coast in a pen-and-ink sketch style. The surface of the rock is rough, with rich texture details and smooth and natural lines. Behind the rock is the rough sea, and the rolling mountains can be vaguely seen in the distance. The sky is covered with clouds of various shapes. |  | This illustration depicts a huge rock standing on the coast in the style of an engraving. The surface of the rock is rough, with rich texture details, and the use of dense lines to depict the effect of light and shadow. The bottom of the rock is beaten by the surging waves, and the the spray is splashing. In the distance are rolling mountains, and the sky is filled with clouds of various shapes, with sunlight shining through the clouds. |
|  | This is a bustling modern city, with high-rise buildings standing tall and arranged in a staggered manner. The silhouette of the distant high-rise buildings is clearly visible under the afterglow of the sunset. The glass of some buildings reflects golden sunlight, and the streets are bustling with traffic. The sky presents a gradient from orange yellow to pink, the clouds are dyed with brilliant colors, and the city is bathed in a warm sunset. |  | This is a bustling modern city, with high-rise buildings standing tall and arranged in a staggered manner. The silhouette of distant high-rise buildings is clearly visible in the night sky. Most buildings were lit up with scattered lights, and the streets were bustling with traffic, with light trajectories crisscrossing. The sky is dotted with several bright stars, and the night sky is deep blue. The city is immersed in a peaceful and tranquil atmosphere. |
|  | A young Asian woman with long, black hair sits on the floor wearing a pink lace dress with white ribbon decorations tied in bows on her head. She holds a storybook and looks at the viewer with clear eyes. The room features white wood paneling and a white cabinet. |  | A young Asian woman with long, black hair sits on the floor wearing a pink lace dress with white ribbon decorations tied in bows on her head. She embraces a plush toy and looks gently ahead. The room features white wood paneling and a white cabinet. |
|  | A young Asian woman with long, black hair sits on the floor wearing a pink lace dress with white ribbon decorations tied in bows on her head. She rests one hand on her leg and looks directly at the viewer with clear eyes. The room features white wood paneling and a white cabinet. |  | {CJK*}<br>UTF8gbsnThe scene presents a tranquil winter nightscape. Snow-laden stone buildings stand tall on either side, with dense icicles hanging from their roofs. A woman in a reddish-brown coat ascends a stone staircase, carrying a warm lantern. Dim yellowish lights emanate from the buildings, contrasting with the twinkling stars in the sky. Fine snowflakes fall from the sky, and a thin layer of snow covers the ground. The overall color palette is cool, creating a serene and peaceful atmosphere. In the distance, another figure can be vaguely seen standing at a building entrance. A lone, bare tree is visible in the lower right corner. Vertical Chinese characters reading “愿你安好” are displayed at the top. |
|  | The image depicts a cold winter night scene. Tall buildings on both sides are covered with thick snow and long icicles. A stone staircase winds upwards, with a figure in an orange-yellow coat walking along it. The buildings’ lights are warm and yellowish, contrasting with the sparse stars in the night sky. Tiny snowflakes fall from the sky, and the ground is covered in snow. The color palette is cool, but the lights provide a touch of warmth, creating a quiet and slightly mysterious atmosphere. A bare tree and a partial view of the building interiors are visible in the background. The overall style is dreamy and slightly impressionistic. |  | The image presents an adorable chibi girl dressed in dark blue clothing. She wears a dark blue hat, a dark blue jacket, a white top underneath, a dark blue skirt, and black ankle boots. She has two braided pigtails and holds a dark purple folding fan. Her eyes are clear and bright, and her overall style is fashionable with traditional elements. The background is light orange with small dots around the edges, creating a fresh and cute style. |
|  | The image shows an adorable chibi girl in casual attire. She sports a red baseball cap with “DOUDU” printed on it, a yellow jacket over a white top, blue jeans, and white sneakers, along with a dark green crossbody bag. She has shoulder-length brown hair, slightly rosy cheeks, and a somewhat shy expression, creating a youthful and lively overall style. The background is light orange with small dots around the edges, maintaining a fresh and cute style. |  | The center of the image is a yellow, melting-icecream-like character with devil horns, holding a trident. It’s surrounded by various colored ice creams and many small creatures wearing tiny halos or having little devil wings. The background is a soft orange and pink, creating an overall cute and playful style. The ice creams are richly colored, detailed, and present a sweet and dreamy atmosphere. |
|  | The center of the image is a yellow, melting-ice cream-like character with devil horns, holding a trident. It stands on a rock, surrounded by many small creatures wearing tiny halos or having little devil wings. The background is a refreshing blue sky with a sun and moon and some clouds. The overall style is cute and fantastical, with bright colors and lively designs, creating a magical atmosphere. |  | A yellow cartoon cat with black eyes, whiskers, and a red mouth. It wears a red bow tie and a blue coat decorated with patterns like stars, dots, and hearts. At the bottom of the image is the word “LOVE CAT”. |
|  | A pink cartoon pig with black eyes, a red nose, and a red mouth. It wears a red mushroom cap decorated with white dots. At the bottom of the image is the word “MUSH ROOM”. |  | A cartoon character is sitting on the ground with its eyes closed and a smile on its face. It is wearing an orange tiger jumpsuit with a red apple on the hood. It has a pair of white wings and is holding a small orange tiger plush toy. The background is white with yellow stars. |
|  | A cartoon character is sitting on the ground with its mouth open, seemingly eating a piece of white rice cake. Its eyes are narrowed, and it has a smile on its face. It is wearing an orange tiger jumpsuit. The background is white with yellow stars and pink flowers. To the left of the cartoon character, there is a plate with a piece of white rice cake on it. |  | The image shows a small orange-yellow bird with its wings spread, looking down at a little girl in a ballet dress enclosed in a transparent bubble. Surrounding them are blooming pale yellow roses, the background is warm and soft, dotted with white bubbles of varying sizes, creating a peaceful and serene atmosphere. The style is fresh, elegant, with soft colors and smooth lines. |
|  | The image depicts a girl with long, flowing dark hair wearing a pink dress, sitting on the back of a pale purple whale, holding a cup of green drink. The whale swims in light blue water, surrounded by colorful bubbles of different sizes. The background colors gradually change from light blue to light purple. The overall atmosphere is dreamy and romantic, with fresh and soft colors. The style is fresh, elegant, with smooth lines. |  | The image portrays a brown deer with a girl in a yellow dress riding on its back. They stand in a field of pink flowers, against a soft beige background. A large transparent bubble is featured in the center above them, with sunlight streaming through, creating a warm and dreamy atmosphere. The style is fresh, elegant, with soft colors and smooth lines. |
|  | A clear plastic cup with the “alex” logo contains a pink drink. A green leaf and ice cubes are visible within the beverage. The background is white with abstract red shapes and lines. |  | A clear plastic cup with the “alex” logo contains a red drink. The background is white with abstract red shapes and lines. |
|  | This is a 3D rendered image showcasing the back view of a cartoon girl. She has pink hair tied up in a high bun, adorned with a golden spherical ornament. She is wearing a blue top with a white apron tied around her waist, featuring rope-like detailing. Her dress is pink, gradually fading to white at the bottom. She has pointed ears, exhibiting an overall cute and sweet style. |  | This is a 3D rendered image showcasing the front view of a cartoon girl. She has pink hair tied up in a high bun, adorned with a golden spherical ornament. She has large eyes and cute pointed ears. She is wearing a blue short-sleeved top with a white apron. She is wearing a pink dress, gradually fading to white at the bottom. The overall style of the girl is cute and sweet. |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: na
soundness: 2
strengths: - The proposed method is pretty simple and easy to follow.
- The paper is written clearly.
summary: This paper presents group diffusion transformer (GDT), targeting the problem of multi-image generation for multi-task learning. The key idea is to bind the self-attention layers in generative models such as Pixart. Experiments with user study and typical metrics on a newly established benchmark show that GDT is an effective multi-task learner.
weaknesses: - **Evaluation**. The evaluation is only conducted on the authors' own benchmark, and no comparison to other methods is conducted. Besides, the metrics like CLIP-T are not that reliable. The authors are encouraged to conduct experiments on human-aligned image generation benchmarks.
- **Novelty**. While simple, I think the idea of binding multi-image attention is not quite novel. However, I would not say that such exploration should not be encouraged.
- **Solidness**. The authors are training the model on their own dataset. I am not sure if the effectiveness mainly comes from the dataset or the model design. The authors should try to train the model on some other datasets.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please see weakness for questions for clarification and additional results.

Misc:

* line 317. Stable Diffusion 3 citation is wrong.
soundness: 2
strengths: * The paper tackles an important problem that extends the capability of text-to-image models to more diverse set of tasks without requiring specialized data for each new task. 
* The authors proposed a simple technical solution that seems to work.
* I like the idea of solving the multiple visual generation and editing tasks in a unified framework.
summary: The paper presents group diffusion transformer (GDT) that can generate a group of coherent images via text prompt instructions. The technical contribution of the paper is rather simple. To allow the generation of group of coherent images, self-attention tokens across multiple images are concatenated. The paper explains how it is applied to different diffusion transformer architectures (e.g., PixArt that decouples self- and cross-attention, SD3 that couples self- and cross-attention). The important bit of a paper is their data collection. The training dataset is scrawled from the Internet that contains a group of images. After some manual and automatic processing, authors were able to collect 500K image groups and further refined to 10K image groups of higher quality. These image groups are then described using multimodal LLMs to capture correlations among the images within each group. In experimental results, authors provide two metrics -- user study, which rates the goodness of the results by the GDTs, and automated metrics that measures the FID, content consistency or prompt adherence.
weaknesses: * Overall, the paper lacks the comparison to existing works, making it difficult to evaluate how good the proposed method is. Though I understand that the paper might be the first of its kind that presents the unified solution for visual generation and editing tasks, there are many previous works that solve each of these problems, and having no comparison to those works makes the paper not very well grounded.

---
* Details on the training dataset is missing
  * How many tasks do training dataset cover and how do their distributions look like?
  * How many of these collected data are AI generated image sets?
  * How much manual refinement (if exists at all) is requires for generating descriptions for image sets?
  * See comments on zero-shot below.

---

* Several misleading claims:
  * It is misleading to say "unsupervised training on image groups". It is effectively equivalent to the "supervised training on input-output pair / tuples" in the context of this paper. As such, the title of the paper "unsupervised multitask learners" could be misleading. Supervision (or weak supervision at least) is required to construct a set of image groups.
  * Claims on zero-shot: authors should make sure that the training dataset do not contain the data for certain task to claim zero-shot. For example, can GDT perform subject customization if the training data does not contain a group of images that contain the same subject, or image completion from sketch if the dataset does not contain such (sketch, image) pairs? Also, would the data scale matter the zero-shot performance?

---

* Issue in evaluation
  * FID evaluation: while authors hypothesized that training on a small dataset is easier to overfit, thus providing lower FID, this is not particularly convincing. On the other hand it raises a question on the evaluation dataset used for FID calculation. Would love to hear more on the distribution of eval dataset and how overfitting on the small training dataset could result in lower FID. Also, it would be great if authors could provide some generated images from a model trained with different training data scales.
  * Unlike many other user study that evaluates the preference between the two or more images generated from different models, authors asked users to give a score from 1 to 5 for test examples and concluded that the average score of GDTs is higher than 3. However, this score is not calibrated at all, making it difficult to evaluate how good the model is. 
  * Not all tasks considered in this paper is new and there are tasks where abundant previous works exist, such as subject / style customization, image colorization, image variation generation, etc. While I understand that the difference between the GDT and those models that solves a single task, but the quality matters. The proposed user preference study or ablation metrics does not provide any points of comparison to existing works, making it difficult to evaluate the quality of the GDT. Authors are required to provide at least a few additional data points for quality comparisons to existing works on more established tasks.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See above weaknesses.
soundness: 3
strengths: - The presentation of the proposed method is clear and easy to understand.
- The proposed method, GDTs, is novel in its ability to generalize across various visual generation tasks, supporting scalable and task-agnostic pretraining.
- Experiments demonstrate that GDTs exhibit zero-shot capability across different visual generation tasks.
- Ablation studies indicate that GDTs have strong potential for scalable pretraining with larger datasets and a greater number of groups.
summary: The author introduces a novel framework called Group Diffusion Transformers (GDTs), which unifies diverse visual generation tasks by framing them as a group generation problem. In this approach, a set of images is generated simultaneously, optionally conditioned on a subset of the group. GDTs make minimal modifications by concatenating self-attention tokens across images. This group-based design supports scalability, unsupervised learning, and task-agnostic pretraining, achieving strong zero-shot performance in task evaluation settings.
weaknesses: - The evaluation of GDTs performance is limited, as the user study results lack comparisons with other task-specific methods. While the evaluations of GDTs are intended to be zero-shot, including comparisons with task-specific methods would provide a clearer understanding of GDTs’ capabilities.
- The proposed benchmark is not fully utilized. It would be helpful to demonstrate GDT performance across different tasks within the benchmark to highlight its versatility.
- It is unclear whether pretraining in GDTs improves performance when fine-tuned for specific tasks. I suggest providing a few task-specific fine-tuning examples of GDTs, along with comparisons to existing task-specific methods on relevant benchmarks.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: One major question is that whether the purposed GPTs can beat current SOTA methods on any of the visual tasks it claimed it can handle. As an example, can we compare GPTs with Instructional Pix2Pix on instructional based editing, and will GPTs win? I understands that it is infeasible (and unfair) for GPTs to beat SOTA on all claimed visual tasks, but the approach will be more persuasive if it can beat SOTA on some tasks. (And clearing my concern on Weaknesses No.2)
soundness: 3
strengths: 1. One of the major highlight of this work is providing an unsupervised “Group Training” of the specialized diffusion models using a series of correlated images and corresponding captions. The merit of such “Group Training” is that the latter trained model can be widely utilized in up to 30 downstream visual tasks.

2. Preparing a large-scale group date with correlated images and fine-grain prompt is valuable to the community. Not sure whether the author is be able to open-source it or not.

3. Despite the difficulty of evaluating around many visual task, the author did a reasonably effort providing numerical metrics (FID & CLIP score), evaluating the performance of the method, and ablation studies as well.
summary: The author introduce Group Diffusion Transformers (GDTs), in which users can group-solve diverse visual tasks simultaneously (i.e. mainly generating multiple images). GDTs, sourced from SD3 & Pixart-Alpha, applies minor modifications on architecture, claimed to be a scalable method trained unsupervisely on task-agnostic data (i.e. image groups). Later at inference stage, GDTs can achieve zero-shot style transfer / sketching / colorization and many other visual tasks.
weaknesses: 1. As the reviewer understands, the approach still relies heavily on a well-trained MLLM.

Specifically, the interface provided in paper session 2.5 involves first generating a group prompt using LLM and later using the group prompt to generate a series of correlated images.

2. Despite the almighty GDTs claimed by the author, the visualization of many tasks (Fig.1,2) shows that downstream tasks’ performance still has some issues. Explicitly speaking, using GDTs on image editing tasks, the quality of the results shows that preserving the ID/Style/Content remains challenging. The generated image is also not well-aligned with user input. This suggests GDTs will likely downgrade performance like many other all-in-one models.