Return of Unconditional Generation: A Self-supervised Representation Generation Method
Authors: Tianhong Li
Dina Katabi
Kaiming He
Venue: NeurIPS 2024 oral (2024)
Keywords: Unconditional Generation
Representation-Conditioned Generation
Self-supervised Learning
Primary area: generative_models

Abstract: Unconditional generation -- the problem of modeling data distribution without relying on human-annotated labels -- is a long-standing and fundamental challenge in generative models, creating a potential of learning from large-scale unlabeled data. In the literature, the generation quality of an unconditional method has been much worse than that of its conditional counterpart. This gap can be attributed to the lack of semantic information provided by labels. In this work, we show that one can close this gap by generating semantic representations in the representation space produced by a self-supervised encoder. These representations can be used to condition the image generator. This framework, called Representation-Conditioned Generation (RCG), provides an effective solution to the unconditional generation problem without using labels. Through comprehensive experiments, we observe that RCG significantly improves unconditional generation quality: e.g., it achieves a new state-of-the-art FID of 2.15 on ImageNet 256x256, largely reducing the previous best of 5.91 by a relative 64%. Our unconditional results are situated in the same tier as the leading class-conditional ones. We hope these encouraging observations will attract the community's attention to the fundamental problem of unconditional generation. Code is available at [https://github.com/LTH14/rcg](https://github.com/LTH14/rcg).

Return of Unconditional Generation:
A Self-supervised Representation Generation Method
Tianhong Li
Dina Katabi
Kaiming He
MIT CSAIL
Abstract
Unconditional generation-the problem of modeling data distribution without re- lying on numan-annotated labels-is long- long-standing and fundamental challenge in generative models, creating a potential of learning from large-scale unlabeled data. In the literature, the generation quality of an unconditional method has been much worse than that of its conditional counterpart. This gap can be at- tributed the lack of semantic information provided by labels. In this work, we show that one can close this gap by generating semantic representations in the representation space produced by a self-supervised encoder. These represen- tations can be used to condition the image generator. This framework, called cenisentttion-Condtiiirnnn Generation (RCG), provides an effective solution the unconditional generation problem without using labels. Through comprehen- sive experiments, we observe that RCG significantly improves unconditional gen- eration quality: e.g., it achieves 256x256, largely reducing the previous best 5.91 by a relative 64%. Our un- conditional results are situated in the same tier the leading class-conditional ones. We hope these encouraging observations will attract the community's 's atten- tion to the fundamental problem of unconditional generation. Code is available https: / //gi thub. cww. Samm/LTH1//rr.....
state-of- the-art FID of 2.15 ImageNet
1 Introduction
Generative models have been long developed as unsupervised learning methods in the history, e,g., in the seminal works including GAN [27], VAE [39], and diffusion models [57]. These fundamental methods focus on learning the probabilistic distributions of data, without relying on the availabil- ity of human annotations. This problem, often categorized unconditional generation in today's community, is in pursuit of utilizing the vast abundance of unannotated data to learn complex distri- butions.
However, unconditional image generation has been largely stagnant in comparison with its condi- tional counterpart. Recent research [18, 54, 10, 11, 24, 50] has demonstrated compelling image generation quality when conditioned on class labels or text descriptions provided by humans, but its quality degrades significantly without these conditions. Closing the gap between unconditional and conditional generation is a challenging and scientifically valuable problem: it is a necessary step towards unleashing the power of large-scale unannotated data, which is common goal in today's deep learning community.
We hypothesize that such performance gap is because human-annotated conditioning introduces rich semantic information to simplify the generative process. In this work, we largely close this gap by taking inspiration from a closely related area sttaa-uuuueeriiee/effessssuspppp susuprree learning. We
In this paper, the term of "unsupervised learning" implies "not using human supervision". Thus, view self-supervised learning form of unsupervised learning. The distinction between these terminologies beyond the scope of this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Distribution Mapping
Pre-trained
Encoder
Image Distribution
Representation Distribution
(b) Representation Generation
Representation
Generator
Noise Distribution
Representation Distribution
(c) Image Generation
Representation Distribution
Condition
Image
Generator
Noise Distribution
Image Distribution
Figure The Representation-Condittoon Generation (RCG) framework for unconditional generation. RCG consists of three parts: (a) it uses pre-trained self-supervised encoder to map representation distribution; (b) it learns representation generator that
the image distribution samples from noise distribution and generates a representation subject to the representation distri- bution; (c) learns an image generator (e.g., which can be ADM [18], DiT [50], or MAGE [41]) that maps noise distribution to the image distribution conditioned on the representation distribution.
observe that the representations produced by strong self- supervipe encoder (e.g., [30, 12, 8, 14]) can also capture lot semantic attributes without human supervision, as reflected by their transfer learning performance in the literature. These self-supervised representations of conditioning without violating the unsupervised nature of unconditional generation, creating an opportunity get rid of the heavy reliance auman-annotated labels.
form
Based on this observation, we propose to first unconditionally generate a self-supervised represen- tation and then condition this representation generate the images. As preprocessing step (Figure la), we use pre-trained self-supervised encoder (e.g., MoCo [14]) to map the image dis- tribution into the corresponding representation distribution. In this representation space, train a representation generator without any conditioning (Figure 1b). As this space is low-dimensional and compact [65], learning the representation distribution is favorably feasible with unconditional generation. In practice, we implement it as a very lightweight diffusion model. Given this represen- tation space, train second generator that is conditioned these representations and produces images (Figure 1c). This image generator can conceptually be any image generation model. The overall framework, called Representation-Conddie for unconditional generation."
Generation (RCG), provides new paradigm
RCG is conceptually simple, flexible, yet highly effective for unconditional generation. RCG greatly improves unconditional generation quality regardless of the specific choice of the image generator (Figure 2), reducing FID by 71%, 76%, 82%, and 51% for LDM-8, ADM, DiT-XL/2, and MAGE-L, respectively. This indicates that RCG largely reduces the reliance of current generative models on
"The term "unconditional generation" implies "not conditioned on human labels". As such, RCG is an unconditional generation solution.

---

Unconditiona Generation FID
40
39.13
baseline
w/ RCG
30
26.21
27.32
25
15
11.30
6.24
5
4.89
3.44
LDM-8
ADM
DIT-XL/2
MAGE-L
Figure 2: Unconditional Image Generation can be largely improved by our RCG framework. Regardless of the specific form of the image generator (LDM [54], ADM [18]. DiT [50], or MAGE [41]. RCG massively improves the unconditional generation quality. Generation quality is mea- sured by FID on ImageNet with a 256x256 resolution. All comparisons between models without and with RCG are conducted under controlled conditions ensure fairness. The technical details and more metrics are in Section 4.1.
manual labels. On the challenging ImageNet 256 benchmark, RCG achieves an unprecedented 2.15 FID for unconditional generation. This performance only largely outperforms previous unconditional methods, but more surprisingly, can catch up with the strong leading methods that conditional class labels. We hope method and encouraging results will rekindle the community's interest in the fundamental problem of unconditional generation.
Related Work
Generative Models. Generative models aim at accurately modeling data distribution to generate new data point that resembles the original data. One stream of generative models is built top of generative adversarial networks (GANs) [27, 69, 37, 70, 7]. Another stream based on a two- stage scheme [63, 53, 10, 67, 40, 41, 11]: first tokenize the image into a latent space and then apply maximum likelihood estimation and sampling in the token space. Diffusion models [33, 59, 18, 54, 52] have also achieved superior results image synthesis.
The design of generative model is mostly orthogonal how it is conditioned. However, literature has shown that unconditional generation often significantly lags behind conditional generation under same design[18, 41, 10], especially on complex data distributions.
Unconditional Generation. Unconditional generation is the fundamental problem in the realm of generative models. It aims to model the data distribution without relying on human annotations, highlighted by seminal works of GAN [27], VAE [39], and diffusion models [57]. It has demon- strated impressive performance in modeling simple image distributions such as scenes or human faces [23, 10, 18, 54], and has also been successful in applications beyond images where human annotation is challenging or impossible, such as novel molecular design [66, 28, 26], medical image synthesis [71. 16. 47], and audio generation [48, 42, 25J. However, recent research in this domain has been limited, largely due to the notable gap between conditional and unconditional generation capabilities of recent generative models complex data distributions [46, 18, 19, 41, 61].
Prior efforts narrow this gap mainly group images into clusters in the representation space and use the cluster indices underlying class labels to provide conditioning [46, 43, 35). However, these methods assume that the dataset clusterable, the number of classes. Additionally, these methods fall short of generating diverse representations they are unable to produce different representations within the same cluster or underlying class.
the optimal number clusters close to
Representations for Image Generation. Prior works have explored the effectiveness of exploiting representations for image generation. DALL-E [52], a text-conditioaal image generation model, first converts text prompts into image embeddings, and then uses these embeddings as the conditions

---

(a) Representation Generator
Noisy Representation
RDM
Denoised Representation
Representation
Fc block
blolkk
SSL Encoder
(b) Image Generator
Representation
SSL Encoder
Condition
MAGE
VOGAN Tokenizer
Encoder
Decoder
VOGAN nizee
Portollee Voookenize
Masking
Reconstructed Image
Figure 3: RCG's training framework. The pre-trained sule-supe image encoder extracts representations from images and is fixed during training. To train the representation generator, we add standard Gaussian noise to the representations and ask the network to denoise them. To train the MAGE image generator, we add random masking to the tokenized image and ask the network to reconstruct the missing tokens conditioned the representation extracted from the same image.
generate images. In contrast, RCG for the first time demonstrates the possibility of directly gener- ating image representations from scratch, a necessary step to enable conditioning on self-supervised representations in unconditional image generation. Another work, DiffAE [51]. trains image coder an end-to-end manner with a diffusion model as decoder, aiming to learn a meaningful and decodable image representation. However, its semantic representation ability is still limited (e.g., compared to self-supervised models like MoCo [14]. DINO [8]), which largely hinders its perfor- mance in unconditional generation. Another relevant line of work sttrival-auugmetee generative models [5, 4, 9]. where images are generated based on representations extracted from existing im- ages. Such cemi-parametric methods heavily rely on ground grouhd-truth images to provide representations during generation, requirement that is impractical in many generative applications.
3 Method
Directly modeling a complex high-dimensional image distribution challenging task. RCG de- composes it into two much simpler sub-tasks: first modeling the distribution of compact low- dimensional representation, and then modeling the image distribution conditioned on this represen- tation distribution. Figure illustrates the idea. Next, describe RCG and its extensions in detail.
3.1 The RCG Framework
RCG comprises three key components: a pre-trained self-supervised image encoder, a representation generator, and an image generator. Each component's 's design is elaborated below:
Distribution Mapping. RCG employs off-the-shelf image encoder convert the image distribu- tion to representation distribution. This image encoder has been pre-trained using self self-supervised contrastive learning methods, such as MoCo v3 [14], on ImageNet. This approach regularizes the representations on hyper-sphere while achieving state-of-the--s performance in representation learning. The resulting distribution characterized by two essential properties: to be modeled effectively by an unconditional representation generator, and it is rich in high-level semantic content, which is crucial for guiding image generation. These attributes are vital for the effectiveness of the following two components.
simple enough

---

Enabling Guidance in Unconditional Generation. In class-conditional generation, the presence of labels allows not only for class conditioning but can also provides additional "guuddacce" in the generative process. This mechanism is often implemented through classifier- freifferr-reeeeeeee guidance in class- behavior Noisy Represemiatio
Timestep Embed
Figure 4: Representation generator's backbone architecture. Each "Layer" consists of LayerNorm layer [1], a SiLU layer [22], and linear layer. The backbone consists of an input layer that projects the representation den dimension C, followed by N fully- connected (fc) blocks, and an output layer that projects the hidden latent back to the original representation dimen- sion. The diffusion timestep is embed- ded and added every fe block.
Linear
Projection
fe block
Layer
hid-
Layer
Layer
Layer
س Denoised Representation
Representation Generation. In this stage, want to generate abstract, unstructured represen- tations without conditions. To address this issue, we develop diffusion model for unconditional representation generation, which call representation diffusion model (RDM). RDM employs fully-connected network with multiple fully-connected residual blocks as its backbone (Figure 4). Each block consists of input layer, timestep embedding projection layer, and output layer, where each layer consists of a LayerNorm [1]. SiLU [22]. and linear layer. Such an architecture simply controlled by two parameters: the number of blocks, N, and the hidden dimension, C.
RDM follows DDIM [58] for training and inference. As shown in Figure 3a, during training, image representation 2g mixed with standard Gaussian noise variable 24 za vI agc. The RDM backbone is then trained to denoise 2; back to 20- During inference, RDM generates represen- tations from Gaussian noise following the DDIM sampling process [58]. Since RDM operates highly compacted representations, it brings marginal computation overheads for both training and generation (Appendix B. while providing substantial semantic information for the image genera- tor, introduced next.
Image Generation.
The image generator in RCG crafts images conditioned on self-supervised representations. Conceptually, such an image generator can be any modern conditional image gen- erative model by substituting its original conditioning (e-g., class label or text) with representations. In Figure 3b, we take MAGE [41], a paraliel decoding generative model as an example. The image generator is trained to reconstruct the original image from masked version of the image, condi- tioned on the representation of the same image. During inference, the image generator generates images from fully masked image, conditioned on the representation generated by the representa- tion generator.
We experiment with four representative generative models: ADM [18], LDM [54], and DiT [50] diffusion-based frameworks, and MAGE [41] is a parallel decoding framework. Our experiments show that all four generative models achieve much better performance when conditioned on high- level self-supervised representations (Table l).
3.2 Extensions
Our RCG framework can easily be extended to support guidance even in the absence of labels, and to support conditional generation when desired. We introduce these extensions as follows.
conditional generation methods [32, 54, 11, 50]. RCG, the enables to also benefit from such guidance,
in the absence of labels.
Specifically, RCG follows [32, 11] to incorporate guidance into its MAGE generator. During train- ing, the MAGE generator is trained with 10% probability of not being conditioned on image representations, analogous to [32] which has a 10% probability of not being conditioned on labels. For each inference step, the MAGE generator produces representation-condiioon logit, I., and
5

---

Table RCG significantly improves the unconditional generation performance of current gen- erative models, evaluated on ImageNet 256.256. All numbers are reported under the unconditional generation setting.
Unconditional generation FID
ISt
baseline 39.13
22.8
LDM-8 [54]
w/ RCG
101.9 (+-79.1)
baseline 26.21
39.7
ADM [18]
RCG 6.24 (-19.97)
136.9 (+97.2)
baseline 27.32
35,9
DiT-XI.2 [50]
WCGG 4.89 22.433
143.2 (+
baseline 8.67
94.8
MAGE-B [41]
w/ RCG 3.98
(-4.69)
177.8 (+-83.0)
baseline
7.04
123.5
MAGE-L
RCG 3.44 (-3.60)
186.9 (+-63.4)
an unconditional logit, Lus for each masked token. The final logits. lg- are calculated by adjusting le away from lu by the guidance scale, т: lg le 7(e - L.). The MAGE generator then uses l to sample the remaining masked tokens. Additional implementation details of RCG's guidance are provided in Appendix A.
Simple Extension to Class-conditional Generation. RCG scamlessly enables conditional image generation by training a task-specific conditional RDM. Specifically, class embedding is integrated into each fully-connected block of the RDM, in addition to the timestep embedding. This enables the generation of class-specific representations. The image generator then crafts the image conditioned on the generated representation. As shown in Table 3 and Appendix C, this simple modification allows users specify the class of the generated image while keeping RCG's superior generative performance, all without the need to retrain the image generator.
Experiments
We evaluate RCG on the ImageNet 256x256 dataset [17], which is common benchmark for image generation and is especially challenging for unconditional generation. Unless otherwise specified, we do not use ImageNet labels in any of the experiments. We generate 50K images and report the Frechet Inception Distance (FID) [31] and Inception Score (IS) [55] as the standard metrics for assessing the fidelity and diversity of the generated images. Evaluations of precision and recall are included in Appendix B. Unless otherwise specified, we follow the evaluation suite provided by ADM [18]. All ablations and results on other datasets are included in Appendix B.1.
4.1 Observations
We extensively evaluate the performance of RCG with various image generators and compare to the results of state-of-the-art unconditional and conditional image generation methods. Several intriguing properties are observed.
RCG significantly improves the unconditional generation performance of current generative models. In Table 1, we evaluate the proposed RCG framework using different image generators. The results demonstrate that conditioning generated representations substantially improves the performance of all image generators in unconditional generation. Specifically, reduces the FID for unconditional LDM-8, ADM, DiT-XL/2, MAGE-B, and MAGE-L by 71%, 76%, 82%, 54%, and %, respectively. We further show that such improvement is also universal across different datasets, demonstrated by the results CIFAR- 10 and iNaturalist in Appendix B.1. These find- ings confirm that RCG markedly boosts the performance of current generative models in uncondi- labels.
tional generation, significantly reducing their reliance
Moreover, such outstanding performance can be achieved with lower training cost compared to cur- rent generative models. In Figure 5, we compare the training cost and unconditional generation FIDs

---

Generation FID
LDM-8
MAGE-B RCG MAGE-L w/
Figure 5: RCG achieves outstand- ing unconditional generation perfor- mance with less training cost. All numbers are reported under the uncon- ditional generation setting. The train- ing cost is measured using cluster of 64 V100 GPUs. Given that the MoCo v3 VIT encoder is pre-trained and not needed for generation, its training cost is excluded. Detailed computational cost is reported Appendix B.I.
35
30
ADM
DIT-L/Z
DIT-XL/2
Unconditional
15
MAGE-B
10
MAGE-L
4
8
10
12
14
Training Cost (days)
Table 2: RCG largely improves the state-of-the-art in unconditional image generation on Im- ageNet 256x256. All numbers are reported under the unconditional generation setting. Following common practice, we report the number of parameters used during generation. f denotes semi- parametric methods which require ground-truth truth ImageNet images during generation.
Unconditional generation #params FID
1St
BigGAN [19]
~70
38.61
24.7
ADM [18]
554M
26.21
39.7
MaskGIT [10]
227M
20.72
42.1
RCDM [5]
19.0
51.9
IC-GAN' [9]
75M
15.6
59.0
ADDP [61]
176M
8.9
95.3
MAGE-B [41]
176M
8.67
94.8
MAGE-L [41]
439M
7.04
123.5
RDM-IN' [4]
400M
5.91
158.8
RCG (MAGE-B)
239M
3.98 177.8
RCG (MAGE-L)
502M
3.44 186.9
RCG-G (MAGE-B)
239M
3.19 212.6
RCG-G (MAGE-L)
502M
2.15 253.4
of RCG and current generative models. RCG achieves a significantly lower FID with less training cost than current generative models. Specifically, MAGE-B with RCG achieves an unconditional generation FID of 4.87 in less than day when trained on 64 V100 GPUs. This demonstrates that decomposing the complex tasks of unconditional generation into much simpler sub-tasks can signif- icantly simplify the data modeling process.
RCG largely improves the state-of-the-art in unconditional image generation. In Table 2, we compare MAGE with RCG and previous state-of of of-the--rtttttt- methods in unconditional image generation. As shown Figure and Table 2, RCG can generate images with both high fidelity and diversity, achieving an FID of 3.44 and an Inception Score of 186.9. These results are further enhanced with the guided version of RCG (RCG-G), which reaches an FID of 2.15 and an Inception Score of 253.4, significantly surpassing previous methods of unconditional image generation.
RCG's unconditional generation performance rivals leading methods in class-conditional im- age generation. In Table 3, we perform system-level comparison between the unconditional RCG and state-of-the-a art class-conditional image generation methods. MAGE-L with RCG is com- parable to leading caas--oonditional methods, with and without guidance. These results demonstrate that RCG, for the first time, improves the performance plex data distributions to the same level as that of state-oftthe-ar class-conditional generation meth- ods, effectively bridging the historical gap between conditionl and unconditional generation.
unconditional image generation on com-
In Table 4, we further conduct cople-to-apple comparison between the class- condisonniiional versions of LDM-8, ADM, and DIT-XL/2 and their unconditional counterparts using RCG. Surprisingly, with RCG, these generative models consistently outperform their class-conditional versions by noticeable margin. This demonstrates that the rich semantic information from the unconditionally generated representations can guide the generative process more effectively than class labels.
7

---

Table 3: System-level comparison: RCG's unconditional generation performance rivals lead- ing methods in class-conditional image generation on ImageNet 256x256. Following common practice, we report the number of parameters used during generation. Class-conditional results are marked in gray.
w/o Guidance
w/ Guidance
Methods
#params
FID4 ISt
FID ISt
Class-conditional
ADM [18]
554M
10.94
101.0
4.59
186.7
LDM-4 [54]
400M
10.56
103.5
3.60
247.7
U-ViT-H/2-G [2]
501M
2.29
263.9
DiT-XL/2 [50]
675M
9.62
121.5
2.27
278.2
DiffiT [29]
1.73
276.5
BigGAN-deep [6]
160M
6.95
198.2
MaskGIT (10]
227M
6.18
182.1
MDTv2-XL/2 [24]
676M
5.06
155.6
1.58 314.7
CDM [34]
4.88
158.7
MAGVTT-v2 [68]
307M
3.65
200.5
1.78
319.4
RIN [36]
410M
3.42
182.0
VDM+ [38]
2B
2.40
225.3
2.12
267.7
RCG, conditional (MAGE-L)
512M
2.99
215.5
2.25
300.7
Unconditional
RCG (MAGE-L)
502M
3,44
186.9
2.15
253.4
Table 4: Apple-to-apple comparison: RCG's unconditional generation outperforms the class- conditional counterparts of current generative models, evaluated ImageNet x56 MAGE does not report its class-conditional generation performance. Class-conditional results are marked in gray.
Methods
FIDJ ISt
w/ class labels
17.41
72.9
LDM-8 [54]
w/ RCG
11.30
101.9
w/ class labels
10.94
101.0
ADM [18]
w/ RCG
6.24
136.9
w/ class labels
9.62
121.5
DiT-XL/2 [50]
w/ RCG
4.89
143.2
As shown in Table and Appendix C, RCG also supports class-conditional generation with a simple extension. Our representation diffusion model can easily adapt to class-conditional representation generation, thereby enabling RCG to also adeptly perform class-conditional image generation. This result demonstrates the effectiveness of RCG in leveraging its superior unconditional generation performance to benefit downstream conditional generation tasks.
Importantly, such adaptation does not require retraining the representation-condiiisi generator. For any new conditioning, only the lightweight representation generator needs to be re- trained. This potentially enables pre-training of the self-supervised encoder and image generator large-scale unlabeled datasets, and task-specific training of conditional representation generator on a small-scale labeled dataset. We believe that this property, similar to self-supervised learning, allows RCG to both benefit from large unlabeled datasets and adapt to various downstream generative tasks with minimal overheads. We leave the exploration this direction to future work.
image
4.2 Qualitative Insights
In this section, we showcase the visualization results of RCG, providing insights into its superior generative capabilities. Figure 8 illustrates RCG's unconditional image generation results on Ima- geNet 256x 256. The model is capable of generating both diverse and high-quality images without relying on human annotations. The high-level semantic diversity in RCG's generation stems from

---

Reference Image
Generated Images
Deggar
Figure 6: RCG can generate images with diverse appearances but similar semantics from the same representation. We extract representations from reference images and, for each representation, gen- erate a variety of images from different random seeds.
0.1
0.2
0.3
0.4
0.5
0.6
0.8
0.9
1.0
teddy bear
Persian cat
valley
beer glass
Tibetan mastiff
wool
tench
goldfish
Figure 7: RCG's results conditioned on interpolated representations from two images. The seman- ties of the generated images gradually transfer between the two images.
its representation generator, which models the distribution of representations and samples them with varied semantics. By conditioning on these representations, the complex data distribution broken down into simpler, oppessenttionnccondttineed sub-distributions. This decomposition significantly simplifies the task for the image generator, leading to the production high-quality images.
Besides high-quality generation, the image generator can also introduce significant low-level diver- sity the generative process. Figure illustrates RCG's ability generate diverse images that semantically align with each other, given the same representation from the reference image. The images generated by RCG can capture the semantic essence of the reference images while differing in specific details. This result highlights RCG's capability to leverage semantic information in rep- resentations to guide the generative process, without compromising the diversity that is important in unconditional image generation.
Figure further showcases RCG's semantic interpolation ability, demonstrating that the represen- tation space is semantically smooth. By leveraging RCG's 's dependency representations, we can semantically transition between two images by linearly interpolating their respective representations. The interpolated images remain realistic across varying interpolation rates, and their semantic con- tents smoothly transition from one image to another. For example, interpolating between an image of "Tibetan mastiff" and image of "wool" could generate novel image featuring dog wearing a woolen sweater. This also highlights RCG's potential in manipulating image semantics within low-dimensional representation space, offering new possibilities to control image generation.

---

5 Discussion
Computer vision has entered a new era where learning from extensive, unlabeled datasets is becom- ing increasingly common. Despite this trend, the training of image generation models still mostly re- lies on labeled datasets, which could be attributed to the large performance gap between conditional and unconditional image generation. Our paper addresses this issue by exploring Representation- Conditioned Generation, which we propose as a nexus between conditional and unconditional image generation. We demonstrate that the long-standing performance gap can be effectively bridged by generating images conditioned on self-supervised representations and leveraging a representation generator to model and sample from this representation space. We believe this approach has the potential to liberate image generation from the constraints of human annotations, enabling it to fully harness the vast amounts of unlabeled data and even generalize to modalities that are beyond the scope of human annotation capabilities.
Acknowledgements. This work was supported by the GIST MIT Research Collaboration grant funded by GIST. Tianhong Li was also supported by the Mathworks Fellowship. We thank Huiwen Chang, Saining Xie, Zhuang Liu, Xinlei Chen, and Mike Rabbat for their discussion and feedback. We also thank Xinlei Chen for his support on MoCo v3.
References
[1] Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 1oXiv:660700645 2016.
arXiv preprint
[2] Fan Bao, Chongxuan Li, Yue Can, and Jun Zhu. All are worth words: a vit backbone for score-hased diffusion models. In NeurlPS 2022 Workshop on Score-Based Methods, 2022.
[3] Fan Bao, Chongxuan Li, Jiacheng Sun, and Jun Zhu, Why are conditional generative models better than unconditional ones? arXiv preprint arXiv:2212.000 00362, 2022.
[4] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Bjöm Ommer. Retrieval-augmmentdd diffusson models. Advances in Neural Information Processing Systems, 35:15309-15324, 2022.
[5] Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self- supervised representation knows about. arXiv preprint arXiv:21 12.091622111144 2021.
[6] Andrew Brock, Jeff Donahuc, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 18X9v:1809.11 1009, 2018.
[7] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Int. Conf. on Learning Representations (ICLR), 2019.
[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Int. Conference on Computer Vision (ICCV), pp. 9660, 2021.
[9] Arantxa Casanova, Marlene Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero Soriano. Instance-conditioned gan. Advances in Neural Information Processing Systems, 11-27717-27522 2021.
[10] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 11315-11325, 2022.
[11] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 2023.
[12] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for con- trastive learning of visual representations. arXiv preprint arXiv:2000.05709) 2020.
[13] Ting Chen, Simon Kornblith. Mohammad Norouzi, and Geoffrey Hinton. A simple framework for con- trastive learning of visual representations. In icml, pp. 1597-1607. PMLR, 2020.
[14] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self self-super vision trans- formers. In Int. Conference on Computer Vision (ICCV), pp. 9640 9640-9649, 2021.
10

---

[15] Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning. arXiv preprint arXiv:2401.14404 2024.
[16] Pedro Costa, Adrian Galdran, Maria Ines Meyer, Meindert Niemeijer, Michael Abràmoff, Ana Maria Mendonça, and Aurélio Campilho. End-to-end adversarial retinal image synthesis. IEEF transactions on medical imaging, 37(3):781-791, 2017.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchi- cal image datahase. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
[18] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780 8794, 2021.
[19] Jeff' Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in neural information processing systems, 32, 2019.
[20] Alexey Dosovitskiy, Lueas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Int. Conf. on Learning Representations (ICLR), 2021.
[21] DC Dowson and BV666017 Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 12(3):450 455, 1982.
[22] Stefan Elfwing, Eiji Uchibe, and Kenji Doya. gegmoid-weihtee linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
[23] Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image syn- thesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22873-12883 2021.
[24] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusiin transformer a strong image synthesizer. arXiv preprint 0rXiv:2333.143300 2023.
[25] Karan Goel, Alhent Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state-space models. In International Conference on Machine Learning, pp. 7616-7633, PMLR, 2022.
[26] Rafael Kommez-Bombaree Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-LLnnglingg Dennis Sheberla, Jorge eqgilera-Iparrauuirre Timothy Hirzel, Ryan Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 4(2):268 276, 2018.
[27] Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde- Farde-Farlcy Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. 2014.
[28] Gabriel Lima Guimaracs, Benjamin Sanchez-Lenae Lengeling, Carlos Outeiral, Pedro Luis Cunha Farias, and Alán Aspuru-Guzik. Objective-reinforce generative adversarial networks (orgam) for sequence genera- tion models. arXiv preprint a705: 10843, 2017.
[29] Ali Hatamizadeh, Jiaming Song, Guilin Liu. Jan Kautz, and Arash Vahdat. Diflit: Diffusion vision transformers for image generation. arXiv preprint arXiv:2312.0211 2023.
[30] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsuper- vised visual representation leamnin. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 9729 9738, 2020.
[31] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances neural infor- mation processing systems, 30, 2017.
[32] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598
2022.
[33] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion prohabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 6840-6851, 2020.
[34] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffiusion models for high fidelity image generation. The Journal of Machine Learning Re- search, 23(1) 2281, 2022.
11

---

[35] Vincent Tao Hu, David W. Zhang, Yuki M. Asano, Gertjan J. Burghouts. and Cees G. M. Snoek. Self- guided diffuusio models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18413-18422, June 2023.
[36] Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. arXiv preprint arXiv:2212. 11972 2022.
[37] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
[38] Diederik Kingma and Ruiqi Gao. Understanding the diffission objective as weighted integral of elbos. arXiv preprint arXiv:2303.000 00848, 2023.
[39] Diederik P Kingma and Max Welling. Auto auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
[40] Doyup Lcc, Chihcon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image gen- eration using residual quantization. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[41] Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishman. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2142-2152, 2023.
[42] Jen- Liu, Yu-Hua Chen, Yin-Cheng Yeh, and Yi-Hsuan Yang- Unconditional audio generation with generative adversarial networks and cycle regularization. arXiv preprint arXi: 2020.
[43] Steven Liu, Tongzhou Wang, David Bau, Jun-Yar Yan Zhu, and Antonio Torralba. Diverse image generation via self-conditioned gans. In Proceedings of the IEEL/CVF conference on computer vision and pattern recognition, pp. 14286-14295, 2020.
[44] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint wrXiv:1608.0998 2016.
[45] Ilya Loshchilov and Frank Hutter. arX1v:7711.05001.0 2017.
Decoupled weight decay regularization. arXiv preprint
[46] Mario Lučić, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. High- fidelity image generation with fewer labels. In International conference on machine learning, pp. 4183- 4192. PMLR, 2019.
[47] Ali Madani, Mehdi Moradi, Alexandros Karargyris, and Tanveer Syeda-Mahmood, Semi-supervised learning with generative adversarial networks for chest x-ray classification with ability of data domain adaptation. In 2018 IEEE 15th International symposium on biomedical imaging (ISBI 2018), pp. 1038- 1042. IEEE, 2018
[48] Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplernn; An unconditional end-to-end neural audio generation model. arXiv preprint arXiv: /6xiiv:612..0 16.2077837 2016.
[49] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
[50] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
[51] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619-10629 2022.
[52] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 2022.
[53] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq- vae=2. Advances in neurud information processing systems, 32, 2019.
[54] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
12

---

[55] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[56] Dave Salvator, Nvidia developer blog- with-al00-uf32, 2020.
[57] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learn- ing using nonequilibrium thermodynamics. In International conference machine learning, pp. 2256- 2265. PMLR, 2015,
[58] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint urXiv:2010.02502. 2020.
[59] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Int. Conf. on Learning Representations (ICLR), 2021.
[60] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 2826, 2016.
[61] Changyao Tian, Chenxin Tao, Jifeng Dai, Hao Li, Ziheng Li, Lewei Lu, Xiaogang Wang, Hongsheng Li. Gao Huang, and Xizhou Zhu. Addp: Learning general representations for image recognition and generation with alternating denoising diffusion process. arXiv preprint 0rX1v2230.005433, 2023.
[62] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International con- ference on machine learning, pp.
PMLR, 2021.
[63] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[64] Grant Van Horn, Oisin Mac Aodha, Yang Song. Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778 8778, 2018.
[65] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. PMI.R, 2020.
[66] Joseph L Watson, David Juergens, Nathanicl R Bennett, Brian L Trippe, Jason Yim, Helen Eisenach, Woody Ahem, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiftusion. Nature, 620X7976):1088 1100, 2023.
[67] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vegan. arXiv preprint arXiv: 10.04677, 2021.
[68] Lijun Yu, José Lezama. Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen. Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, al. Language model beats diffusion-tokenizer key to visual generation. arXiv preprint a0X0v:2310.05737, 2023.
[69] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolci Huang, and Dimitris Metaxas. StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial net- works. In ICCV, 2017.
[70] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Int. Conference on Machine Learning (ICML), pp. 7354-7363, 2019.
[71] Tianyang Zhang, Huazhu Fu, Yitian Zhao, Jun Cheng, Mengjie Guo, Zaiwang Gu, Bing Yang. Yuting Xiao, Shenghua Gao, and Jiang Liu. Skrgan: Sketching-rendering unconditional generative adversarial networks for medical image synthesis. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2019: 22nd International Conference, Shenghen, China, October 13-17, 2019, Proceedings, Part IV 22, pp. 777-785. Springer, 2019.
[72] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint 1..08221 2021.
13

---

Figure 8: Unconditional generation results of RCG on ImageNet 256x256. RCG can generate
realistic images with diverse semantics without human annotations.
A Implementation Details
In this section, we describe the implementation details of RCG, including hyper-parametens, model architecture, and training paradigm. We also include copy of code the supplementary mate- rial. All codes and pre-trained model weights will be made publicly available.
Image Encoder. We use Vision Transformers (ViTs) [20] pre-trained with MoCo v3 [14] as the default image encoder. We evaluate three ViT variants (VIT-S, ViT-B, and VIT-L) the main paper, each trained ImageNet for 300 epochs. We utilize the image representation after the MLP projec- tion head, favoring its adjustable dimensionality. An output dimension 256 has proven the most effective. The representation of each image is normalized by its mean and variance. Detailed training recipes our pre-trained image encoder can be found in [14].
Representation Diffusion Model (RDM). Our RDM architecture employs backbone of multiple fully connected blocks. We use 12 blocks and maintain consistent hidden dimension of 1536 across the network. The timestep is discretized into 1000 values, each embedded into 256- dimensional vector. For class-conditional RDM, we embed each class label into a 512-dimensional
vector. Both timestep and class label embeddings projected 1536 dimensions using different linear projection layers in each block. Detailed can be found in Table 5.
for RDM's 's training and generation
We experiment with ADM [18]. LDM [54], DiT [50], and MAGE [41] as the im- age generator. For representation-condiiioo
Image Generator.
ADM, LDM and DiT, we substitute the original class embedding conditioning with the image representation. We follow ADM's original training recipe to ADM for 400 epochs. We follow LDM-8's original training recipe, with modifications including batch size 256, a Icarning rate 6.4e-5, and a training duration of 40 epochs. We follow the DiT training scheme in [15]. which trains DiT-XL for 400 epochs with batch size 2048 and linear learning rate warmup for 100 epochs. The 3g the AdamW optimizer is set to 0.95. For representation- condiiioooo MAGE, replace the default "fake" class token embedding [C,] with the image representation for conditioning.
train
During the training of RCG's image generator, the image is resized so that the smaller side is of length 256, and then randomly flipped and cropped to 256x 256. The input to the SSL encoder further resized to x24 224 to be compatible with its positional embedding size. Our implementation of guidance follows Muse [11], incorporating a linear guidance scale scheduling. Detailed hyper- parameters for our representationnccnniiioonn MAGE are provided in Table 6.
14

---

Table RDM implementation details.
config
value
#blocks
12
hidden dimension
1536
#params
63M
optimizer
AdamW [45]
learning rate
5.12c-4
weight decay
0.01
optimizer momentum
B1, 82
0.9, 0.999
batch size
512
learning rate schedule
constant
training epochs augmentation
200
Resize(256)
RandCrop((26)
RandomFlip (0.5)
diffusion steps
1000
noise schedule
linear
DDIM steps 1
250
1.0
Table 6: Repseetaaioon-coniiioneee MAGE implementation details.
config optimizer
value
AdamW [45]
hase learning rate weight decay optimizer momentum
1.5e-4
0.05
$1. 92 = 0.9, 0.95 4096
batch size learning rate schedule
cosine decay [44]
warmup epochs
training epochs gradient clip label smoothing [60]
800
3.0
dropout
0.1
augmentation
Resize(256)
RandCrop(256)
RandomFlip (0.5)
masking ratio min masking ratio max masking ratio mode masking ratio std rep. drop rate parallel-decoding
0.5
1.0
0.75
0.25
0.1
6.0 (B) 11.0
temperature parallel-decoding steps
20
guidance scale (T) guidance scale schedule
1.0 (B) 6.0 (L)
linear [11]
B Additional Quantitative Results
B.1
Ablations
This section provides a comprehensive ablation study of the three core components RCG. Our default setup uses MoCo v3 ViT-B the pre-trained image encoder, RDM with 12-block, 1536-hiddee-dimensioo backbone trained for 100 epochs, and MAGE-B image generator trained for 200 epochs. Unless otherwise specified, all other properties and modules are set to the default settings during each component's individual ablation. The FID in this section is evaluated against the ImageNet validation set.
Distribution Mapping. Table ablates the image encoder. Table 7a compares image encoders trained via various self-supervised learning methods (MoCo v3, DINO, and iBOT), highlighting
15

---

Table 7: Distribution mapping ablation experiments. The default encoder is MoCo v3 ViT-B with 256 projection dimension. Default settings are marked in gray
Method
Model
params
lin.
FID
Projection
FID
15
No conditioo
14.25
52.7
VIESS
23M
73.2
5.77
1308
12
9.14
$1.0
MoCo v3 [14]
5.07
142.5
VIE-B
86M
76.7
64
6.09
119.2
DINO [8]
7,53
1608
VTT.
304M
77.6
5.06
143.3
iDOT [72]
8,05
148.7
256
5.07
1/2.5
DeiT [62] (sspenised))
5.51
211.7
768
6.10
112.7
Pre-training RCG achieves good par- Model size. RCG scales with larger Projection dimension. The dimensionalitt formance with encoders pre-trained with differ- pre-trained encoders with better linear prob- of the image represenlation is important in RCG's cut contrastive learning and supervised learning accuracy.
performancc.
methods.
Table 8: Representation generation ablation experiments. The default RDM backbone of 12 blocks and 1536 hidden dimensions, trained for 100 epochs, and takes 250 sampling steps during generation. The representation Frechet Distance (rep FD) is evaluated between 50K generated repre- sentations and representations extracted from the ImageNet training set by MoCo v3 VIT-B. Default settings are marked gray
ABlocka
rep
Hiddon
Fpochs
rap
#Stops
15
rep
7.33
113.5
0.71
256
12.99
67.3
3.95
10
534
125.4
0.87
5.80
120.3
0.37
5:40
132.9
0.53
512
9.07
99.8
1.19
50
5.21
138.3
0.54
50
5.28
133.0
0.55
5.07
142.5
0.88
1024
5:35
132.0
0.56
100
5.07
142.5
0.48
100
5.15
138
0.48
0.50
142.5
200
0.49
300
Model depth, RDM can improve generation prove generation performance.
A deeger (b) Model A RDM
(c)
Training epochs. Triining
Diffusion steps. sampling steps can improve generation perfoe- mance.
RDM longer improves generation
performance.
performance.
Table 9: Image generation ablation experiments. The default image generator is MAGE-B trained for 200 epochs. Table 9c evaluates different T using MAGE-L with RCG trained for 800 epochs and the FID is evaluated following ADM suite. Default settings are marked in gray
Conditioning
HD
18
Epachs
FII
18
0.0
1.0
30
50
6.0
7.0
No condition
127.7
Cluster label
6.60
121.9
200
5.07
142.5
1S
186.9
228.5
251.
252.7
253.4
252.6
147.3
Generated rep.
5.07
142.5
800
4.15
172.0
Oracle rep.
4.37
149.0
(a) Conditioning. Conditioning on gen- (b) Training epochs, Longer train- (c) Classifier-free guidance scale, 7 6 achieves
erated representations improves
ing improve generation perfor- the best FID and IS for ROG-L.
baselines in FID.
mance.
their substantial improvements over the unconditional baseline. Additionally, an encoder trained with DeiT [62] in supervised manner also exhibits impressive performance, indicating RCG's adaptability to both supervised and self-supervised pre-training approaches.
We also notice that using representations from MoCo v3 achieves better FID than using representa- tions from DINO/iBOT. This is likely because only MoCo v3 uses an InfoNCE loss. Literature has shown that optimizing InfoNCE loss maximize uniformity and preserve maximal information in the representation. The more information the representation, the more guidance it can provide for the image generator, leading
better and more diverse generation. To demonstrate this, we com- pute the uniformity loss on representations [65]. Lower uniformity loss indicates higher uniformity and more information in the representation. The uniformity loss of representations from MoCo v3, DINO, and iBOT is -3.94, -3.60, and -3.55, respectively, which aligns well with their generation performance.
Table 7b assesses the impact of model size on the pre-trained encoder. Larger models with better linear probing accuracy consistently enhance generation performance, although smaller ViT-S model still achieves decent results.
We further analyze the effect of image representation dimensionality, using MoCo v3 VIT-B models trained with different output dimensions from their projection head. Table 7e shows that neither excessively low nor nigh-dimensional representations are ideal too low dimensions lose vital image information, while too high dimensions pose challenges for the representation generator.
Representation Generation. Table 8 ablates the representation diffusion model and its effectiveness in modeling representation distribution. The RDM's depth and width are controlled by the number of fo blocks and hidden dimensions. Table 8a and Table 8b ablate these parameters, indicating an
16

---

Table 10: CIFAR-10 and iNaturalist results. RCG consistently improves unconditional image
generation performance on different datasets. Dataset
Methods
FID
baseline
3.29
CIFAR-10
Improved DDPM [49]
w/ RCG
2.62
w/ class labels
2.89
baseline
8.64
iNaturalist 2021
MAGE-B
w/ RCG 4.49
w/ class labels
4.55
optimal balance at 12 blocks and 1536 hidden dimensions. Further, Table 8c and Table 8d suggest that RDM's performance saturates at 200 training epochs and 250 diffusion steps.
Besides evaluating FID and IS on generated images, we also assess the Frechet Distance (FD) [21] between the generated representations and the ground-truth representations. A smaller FD indicates that the distribution of generated representations more closely resembles the ground-trut -trut distri- bution. Since the MoCo encoder is trained on the ImageNet training set, the representation distribution in the training set can be slightly different from that in the validation set. To establish a better reference point, we compute the FD between 50K randomly sampled representations from the training set and the representations from the entire training set, which should serve as the lower bound the FD for our representation generator. The result is an FD of 0.38, demonstrating that our representation generator (with FD of 0.48) can accurately model the representation distribution.
We also evaluate the representation generator against the validation set, resulting in an FD of 2.73. As reference point, the FD between 50K randomly sampled representations from the training set and the validation set 2.47, which also close to the FD of our representation generator.
Image Generation. Table 9 ablates RCG's image generator. Table 9a experiments with MAGE- under different conditioning. MAGE-B with RCG significantly surpasses the unconditional and clustering- based baselines, and further outperforms the class- dass--oonditioall baseline in FID. This shows that representations could provide rich semantic information to guide the generative process. It is also quite close to the "upper bound" which is conditioned on oracle representations from Im- ageNet real images, demonstrating the effectiveness of the representation generator in producing realistic representations.
We also ablate the training epochs of the image generator and the guidance scale т, as shown in Table 9b and Table 9c, Training MAGE longer keeps improving the generation performance, and 6 achieves the best FID and IS.
B.2 Other Datasets
In this section, we evaluate RCG on datasets other than ImageNet to validate its consistent effective- ness across different datasets. We select CIFAR- 10 and iNaturalist 2021 [64]. CIFAR-10 represents a relatively simple and low-dimensional image distribution, and iNaturalist 2021 represents a more complex image distribution, with 10,000 classes and 2.7 million images. For CIFAR-10, we employ SimCLR [13] trained on CIFAR-10 10 as the image encoder and Improved DDPM [49] as the image generator. The FID is evaluated between 50,000 generated images and the CIFAR-IO training set. For iNaturalist, we employ MoCo v3 ViT-B trained on ImageNet as the image encoder and MAGE-B as the image generator. The FID is evaluated between 100,000 generated images and the iNaturalist validation set, which also consists of 100,000 images.
As shown in Table 10, RCG consistently enhances unconditional image generation performance on both CIFAR-10 and iNaturalist 2021, demonstrating its universal effectiveness across various datasets. Notably, the improvement on complex data distributions such as ImageNet and iNaturalist is more significant than on simpler data distributions such as CIFAR-10. This is because RCG de- composes a complex data distribution into two relatively simpler distributions: the representation distribution and the data distribution conditioned on the representation distribution. Such decompo- sition particularly effective on complex data distributions, such as natural images, paving the way for generative models to model unlabeled complex data distributions.
17

---

Table 11: Computational cost. RCG achieves much smaller FID with similar or less computa- tional cost as baseline methods. The number of parameters. training cost, and the number of training epochs of the representation generator and the image generator are reported separately.
Unconditional Generation #PParams (M) Training Cost (days)
Epochs
Throughput (samples/s)
FID
LDM-8 [54]
395
0.9
39.13
ADM [18]
554
14.3
400
0.05
26.21
DiT-L [50]
458
6.8
400
0.3
30.9
DiT-XL [50]
675
9.1
400
0.2
27.32
MAGE-B [41]
176
5.5
1600
3.9
8.67
MAGE-L [41]
439
10.7
1600
2.4
7.04
RCG (MAGE-B)
634176
0.140.8
100+200
3.6
4.87
RCG (MAGE-B)
63+176
0.243.3
200+800
3.6
3,98
RCG (MAGE-L)
63+439
0.3+1.5
100+200
2.2
4.09
RCG (MAGE-L)
63+439
0.6+6.0
200+800
2.2
3,44
Table 12: RCG's unconditional generation FID, IS, precision and recall on ImageNet 256x256, evaluated following ADM's suite [18].
Methods
FID
ISt
Prec.
Rec.1
RCG (MAGE-B)
3.98 177.8
0.84
0,47
RCG (MAGE-L)
3.44 186.9
0.82
0.52
RCG-G (MAGE-B)
3.19 212.6
0.83
0.48
RCG-G (MAGE-L)
2.15 253.4
0.81
0.53
B.3 Computational Cost
In Table 11, we present a detailed evaluation of RCG's computational cost, including the number of parameters, training costs, and generation throughput. The training cost of all image generators is measured using a cluster of 64 V100 GPUs. The training cost of RDM measured using V100 GPU, divided by 64. The generation throughput is measured ADM measure their generation throughput on a single NVIDIA A100 [54], we convert it to V100 throughput by assuming x2.2 speedup of A100 V100 [56].
single V100 GPU. As LDM and
As shown in the Table 11, RCG requires significantly lower training costs to achieve great per- formance. For instance, it achieves FID of 4.87 in less than one day of training. Moreover, the training and inference costs the representation generator are marginal compared to those the im- age generator. This efficiency potentially enables for lightweight adaptation to various downstream generative tasks by training only the representation generator on small-scale labeled datasets.
B.4 Precision and Recall
In Table 12, we report the unconditional generation precision and recall of RCG, evaluated on Im- ageNet 256x256 following the ADM suite [18]. Larger models as well as incorporating guidance (RCG-G) both improve recall while slightly decreases precision.
C Additional Qualitative Results
We include more qualitative results, including class- dlasndunconi image generation (Figure 9), class-conditional image generation (Figure 10 and Figure 11), and the comparison between gen- eration results with or without guidance (Figure 13). All these results demonstrate RCG's superior performance in image generation. We also include some failure cases Figure 12.
D Limitations and Negative Impact
Limitations, Like any other generative models, RCG can also produce unrealistic or low-quality results (see Appendix C for some examples).
18

---

Figure More RCG class-unconditional image generation results on ImageNet x56x256.
Societal Impact. Despite the rapid advancements in generative models, they also carry potential negative societal impacts. For instance, such models can amplify existing biases present internet data. RCG, being generative model, is not immune to these issues. However, it is important to note that RCG operates within an unconditional generation framework, which does not depend on human- provided labels. This characteristic might possess the potential mitigate the influence of human biases, offering a more neutral approach to data generation compared to traditional condi- tional models.
19

---

Figure 10: RCG class-conditional image generation results on ImageNet 256x256. Classes are 874: trolleybus, 664; monitor, 249: malamute; 952: fig, 968: cup, 256: Newfoundland; 789: shoji, 659: mixing bowl, 681: notebook; 119: rock crab, 629: lipstick, 192: cairn; 359; ferret, 9; ostrich, 277:
red fox.
20

---

Figure 11: RCG class-conditional image generation results on ImageNet 256 256. Classes are 1: goldfish, 388: panda, 279: Arctic fox; 323: monarch butterfly, 292: tiger, 933: cheeseburger; 985;
daisy, 979: valley, 992: agaric
ELS
Figure 12: Similar other generative models on ImageNet, RCG also could face difficulty in gen- crating texts, regular shapes (such as keyboard and wheel), and realistic human.
21

---

w/o CFG
w/ CFG
w/o CFG
w/ CFG
w/o CFG
w/ CFG
Figure 13: Class Class-unconenn unconditional image generation results on ImageNet 256x256. with or without guidance. RCG achieves strong generation performance even without guidance. Incorporating guid- ance further improves the generation quality.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper presents unconditional image generation method that rivals the performance the state-of-the-an class-conditional generation methods.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix D.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificattin, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
This paper does not include theoretical contribution.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yss]
Justification: See Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
Code is available at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: atttps:/nnips.om
for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g,, for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/niiss... ctccc//plllc ccc/publc//aae aeeiic/uuieee gwwggguddee//e/ / Sut///oddeSiiii ssi onPolicy)
more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: Following common practice in the generative modeling literature, do not report error bars in this paper because of the heavy computation overheads.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See subsection B.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tte:///neuuu..eeeeeee ctt.iiss..cppppeeeee http:://harrpp...cppb.t.t//// detellcceeeccuuudelllleeeee ines?
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: see Appendix D.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification:
We will require the users to adhere to usage guidelines for our released mod-
els.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly cite the original assets in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswi thcode. com/ datastts has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "QjcNU6P1yu", "reviewer": "Reviewer_zDRz", "summary": "This paper proposes a very simple unsupervised image generation framework that does not rely on human labeled annotation without compromising generation quality. This framework has two stages: i) representation generator learning and ii) image generator learning. The representation generator is trained in the form of diffusion model training to take a noisy latent image as input and output the corresponding representation encoded by a self-supervised image representation model. The image generator is trained to generate the image corresponding the given representation (encoded by the self-supervised representation model). This paper demonstrates that this simple framework is effective in achieving generation quality comparable to the counterpart supervised learning methods, regardless of architecture type.", "questions": "Please respond the two questions I raised in the weakness section.", "limitations": "This paper properly deals with the limitation and potential societal impact in the supplementary material.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Simple but effective framework for unsupervised image generation. The proposed method is designed to solve difficult problems very simply and intuitively, so it will be able to provide inspiration not only in this field but also in a variety of other fields.\n\n2. Great presentation. It was very helpful in understanding this paper since it explained the information covered in each section in a very informative but concise manner.\n\n3. A variety of experiments can support the authors' claim and the effectiveness of the proposed method.", "weaknesses": "I do not have a major concern for this work. I only have two questions in the method.\n\n1. When training the image generator, why was the representation of the SSL model used instead of the output of the representation generator trained in the previous step? Although the representation generator is trained to distill the generation capability of the SSL model, isn't it more appropriate to train under the same conditions as in the inference in which the output (representation) of the representation generator is used?\n\n2. Is there any way for the two generators to be trained together in an end-to-end manner?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our simple but effective framework, the strong experiment results, and the great presentation of our paper. Below, we address the two questions (***Q***) raised by the reviewer.\n\n***Training the image generator using ground-truth representations vs. generated representations (Q1)***\n\nThe reviewer asks why we use the representations from the SSL model instead of the output of the representation generator during the training of the image generator. This is because the image generator needs both the representation as a condition and the corresponding ground-truth image as supervision during training. For a representation output by the unconditional representation generator, it is challenging to determine the corresponding image. Therefore, if we were to use generated representations as conditions for our image generator, we wouldn’t have the corresponding ground-truth images needed for training. Additionally, our design allows us to fully decouple the training of the representation generator and the image generator, making our framework more flexible to train.\n\n***End-to-end training (Q2)***\n\nThis is an excellent suggestion. Training the representation generator and the image generator together could potentially further enhance the performance of the entire system. However, similar to ***Q1***, one possible issue is that the denoised representation output from the representation generator might not match the ground-truth representation and the corresponding ground-truth image, especially at high noise levels. In this scenario, using the denoised representation as the condition for the image generator while using the ground-truth image for supervision might cause inconsistencies during training. Nonetheless, we believe this would be an interesting future direction to explore.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DojXaSZoI4", "reviewer": "Reviewer_xRbx", "summary": "This paper proposes RCG, a novel framework to enhance unconditional image generation by leveraging self-supervised representations. The main idea is first to generate self-supervised representations unconditionally using a pre-trained encoder and then condition the image generation on these representations. This process does not require human-annotated labels and involves training a lightweight diffusion model to generate representations efficiently. The authors demonstrate the effectiveness of RCG by significantly improving the quality of unconditional generation across various architectures, closing the performance gap between unconditional and conditional generation methods.", "questions": "**[Q]** Is there any intuition as to why MoCo-v3 is the best representation for RCG?", "limitations": "They addressed the limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "**[S1]** The paper is well-motivated, addressing the importance of unconditional image generation for utilizing abundant data, which has remained stagnant compared to conditional generation.\n\n**[S2]** The idea of utilizing self-supervised learning for image generation makes sense and is a novel idea.\n\n**[S3]** The paper shows significant performance improvements.", "weaknesses": "I found no weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our motivation, novel idea, and significant performance improvements. Below we address the question raised by the reviewer.\n\n***Why MoCo-v3 is the best representation for RCG***\n\nWe compare self-supervised representations from different methods in Table 7(a), and the representations from MoCo v3 achieve the best FID. This is likely because MoCo v3 uses an InfoNCE loss, which attracts positive samples and repels negative samples. Literature has shown that optimizing such an InfoNCE loss can maximize uniformity and preserve maximal information in the representation [1], thus providing substantial guidance for the image generator and leading to better and more diverse generation results.\n\nNonetheless, Table 7(a) also demonstrates that RCG achieves substantial improvements over the unconditional baseline using representations from various image encoders, including self-supervised encoders such as iBOT and DINO, as well as the supervised encoder DeiT. This shows that RCG can effectively utilize different self-supervised encoders and consistently improve the performance of unconditional generation.\n\n[1] Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4PAmnTS0Mp", "reviewer": "Reviewer_fB14", "summary": "This paper proposes generative models conditioned on representation obtained from a pre-trained self-supervised encoder to achieve high-quality diverse generation.", "questions": "1. When extending to class-conditional generation, is it required to fine-tune the model?", "limitations": "The limitations should include the discussions on the necessity of a self-supervised encoder and how to obtain such an encoder for datasets, especially for other modalities such as text, speech.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. The writing is clear. \n2. The experimental results demonstrate significant improvement over unconditional generation.", "weaknesses": "1. The idea of generation conditioned on representation is not new. As the authors mentioned, conditioning on instance representations has been well studied in the community. Though the authors argue that they require ground-truth images to provide representations during generation, the issue can be addressed by adopting the idea from VAE to align those instance representations with the prior noise or by incorporating the representation generator proposed in this paper. Therefore, I would say the main contribution of this work is to improves those representation generation works by introducing a generative modeling for the representation, which however is limited.\n2. The deep clustering community has also adopted the technique of self-supervised learning for a more clustering friendly representation and has achieved significant improvement in complex datasets, like ImageNet.  Therefore, generation conditioned on clustering structures also has potential to bring gaps between unconditional generation and conditional generation. I would say the studied problem has been well addressed in the community. Comparisons with conditioning on clustering should be included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our clear writing and strong experiment results. Below we address the weaknesses (***W***) and questions (***Q***) raised by the reviewer.\n\n***Importance of the studied problem***\n\nWe respectfully disagree with the reviewer that “the studied problem has been well addressed in the community.” Unconditional image generation has lagged behind its conditional counterpart for a long time in the literature, especially on complex data distributions such as ImageNet. As shown in Table 2, none of the prior works achieve an unconditional generation FID less than 5 on ImageNet 256x256, and the previous state-of-the-art (RDM-IN) requires the ImageNet training set during generation to achieve a 5.91 FID. In contrast, as shown in Table 3, state-of-the-art class-conditional methods can easily achieve an FID around or below 2, demonstrating a significant gap between unconditional and class-conditional generation.\n\nOther reviewers have also acknowledged the difficulty and significance of the studied problem. Reviewer xRbx noted, “unconditional image generation has remained stagnant compared to conditional generation.” Reviewer zDRz stated, “the proposed method is designed to solve difficult problems very simply and intuitively.” Reviewer pEoh mentioned, “the paper shows very good and solid results on a relevant topic, touches upon the surface of very fundamental questions related to self-supervised methods and the training of generative models.”\n\nOur paper proposes a novel method based on generating self-supervised representations to address this long-standing open problem in the community. The proposed RCG framework significantly improves the quality of unconditional generation, regardless of the specific form of the image generator. It achieves an unprecedented unconditional generation FID of **2.15**, bridging the long-standing performance gap between unconditional and class-conditional generation methods for the first time. We hope the reviewer could reconsider the importance of the studied problem and the rating of our paper in light of this clarification.\n\n***Conditioning on instance representations (W1)***\n\nPrior methods for unconditional generation that use instance representations require ***existing images*** to provide representations during generation, which is impractical for many generative applications. Moreover, none of the prior works use a generative model to accurately model the pre-trained self-supervised representation distribution. Our RCG framework is the first unconditional generation framework that generates pre-trained self-supervised representations ***from scratch*** and uses them as conditions for the image generator. This novel approach significantly boosts unconditional generation performance and rivals class-conditional generation methods, all without the need for any images during the generation process.\n\n***Conditioning on clustering (W2)***\n\nWe agree that using pseudo-labels from clustering methods as class labels can be an option for unconditional generation. In fact, we have included this ablation study in Table 9(a), where we experimented with our image generator under different conditions, including clustering labels obtained from MoCo v3 representations. Conditioning on clustering labels achieves 6.60 FID and 121.9 IS, while conditioning on ground-truth class labels achieves 5.83 FID and 147.3 IS. Conditioning on our generated representations achieves 5.07 FID and 142.5 IS. These results demonstrate that clustering-based conditions perform worse than ground-truth class labels, whereas our generated representations can outperform ground-truth class labels. This is because the generated representations provide richer semantic information to guide the generative process. Furthermore, common clustering methods require the dataset to exhibit clear and distinct groupings or clusters that can be easily identified by clustering algorithms. It also limits the diversity of conditions, as it cannot produce different conditions within the same cluster. We will include this discussion in the revision.\n\n***Extending to class-conditional generation (Q1)***\n\nRCG seamlessly enables class-conditional image generation and achieves competitive performance, as shown in Table 3 (RCG, conditional (MAGE-L)). This is accomplished by training a class-conditional representation generator without the need to retrain or fine-tune the image generator. As shown in Table 11, training the RDM is very lightweight compared to training the image generator.\n\n***Limitations***\n\nWe thank the reviewer for the suggestion. Applying RCG to other data types is beyond this paper's scope and could be an interesting future direction. It requires a pre-trained encoder to extract representations from the data, which are typically available off-the-shelf for common data types such as images, videos, texts, and speech [1, 2, 3, 4, 5, 6]. We will include this discussion in the revision.\n\n[1] An Empirical Study of Training Self-Supervised Vision Transformers\n\n[2] Spatiotemporal Contrastive Video Representation Learning\n\n[3] A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning\n\n[4] SimCSE: Simple Contrastive Learning of Sentence Embeddings\n\n[5] W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training\n\n[6] Speech simclr: Combining contrastive and reconstruction objective for self-supervised speech representation learning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4DcodCgqkG", "reviewer": "Reviewer_pEoh", "summary": "The paper proposes a framework, coined Representation-Conditioned Generation (RCG), that aims to bring the advantages of Conditional Generation techniques to Unconditional Generation settings.\nTo do so, they use the output of a representation generator network instead of class labels.\nThis representation generator is trained in a prior stage to approximate the distribution of image features extracted by a pre-trained self-supervised network (MoCo-v3).\nThey test this technique with a set of very different generative models, including a latent diffusion model, a diffusion transformer, and a masked generative transformer, showing improvement accross the board on the ImageNet Unconditional Generation benchmark.", "questions": "- In Table 8, it is difficult to have an interpretation of the FD as we don't have a reference point. Maybe showing FD between training and validation set could be an option, provided the number of samples for the estimates is handled carefully. Also, FD is evaluated on the training set. It would be good to have validation scores as well.\n\n- In a similar vein, I'd like to see the performance (FID) of RCG on features sampled from training images (similar to Figure 6). This would give another indication of how good the representation generation is. This is important since the representation generation is a major, and arguably the most important, contribution of the paper.\n\n- The authors mention in Appendix B that the FID in the corresponding section are computed on the validation set. Is that also the case for the other values in the main paper?\n\n- For the point raised in weakness b., a precise question I'd like to ask is why, as seen in Table 7a, MoCo v3 features are so much better for this task than DINO or iBOT features that are supposed to be better in other downstream tasks.\n\n- Also linked to weakness b., there are strong connections between generative models and self-supervised learning. In fact, works such as [61, A] or [41] on which the current paper builds a lot of their results are already generative models that explicitly double as representation learners, or even have representation learning as their main objective. Those ties should be discussed in the paper. \nMoreover, if a generative model can be a representation learner, and if using a representation can help achieve a better generative model, how to explain the good results in the submission becomes even less clear. I'd be happy to hear the authors' thoughts on that point.\n\nIn any case, I think the submission is already sound and of interest. It shows very good and solid results on a relevant topic, and opens up interesting research directions. But also, it doesn't provide much in terms of technical novelty or in terms of analysis. The paper touches upon, but barely scratches the surface of very fundamental questions related to self-supervised methods and the training of generative models. I would definitely consider raising my rating to 7 or 8 if it would have a slightly more complete evaluation of the feature generation part and if it provided a deeper analysis of the results.", "limitations": "The authors make an attempt to address limitations by showing failed generation. They could discuss the scope in terms of data type: are there results likely to transfer to data? what hurdles to expect in doing so?\n\nThey also try to adress societal impacts by discussing generative model biases and hypothesize that unsupervised models should significantly mitigate the influence of human bias. I'd appreciate if they would either expand on that point or alternatively refrain from making such an assumption without a stronger backing. ImageNet photos certainly have been produced by humans and are not bias-free.\nSince they are willing to discuss the limitations and impacts of generative models in general, they could also mention potential misuses such as deep fakes and misinformation.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "a. The proposed method is sound, the technical details are delivered clearly and thoroughly. \n\nb. Experiments and ablations are extensive, and results look robust, being confirmed for very different classes of generative models.\n\nc. The paper provides new empirical evidence and numbers on very relevant points:\n  - Diffusion models can generate self-supervised representations convincingly.\n  - Those generated features can serve as good conditioning signals to help train image generation models.", "weaknesses": "a. Technical novelty is limited: [48] also generates images by first generating pretrained features to condition the image generator, and works such as [5,A] manage to generate high-quality images using representations. The remaining differentiator to existing work in terms of technical contribution is then in the choice of the pre-trained feature extractor and image generator.\n\nb. In that light, while thorough ablation results are presented, I find the interpretation of the results lacking. If those choices are important, and it seems to be the case considering the range of FID scores in the ablations, it would be very useful to provide intuitions about how to make those choices and why.\n\n--\n\n[A] SODA: Bottleneck Diffusion Models for Representation Learning. Hudson et al. CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our method, extensive and robust experiments, and the potential to open up interesting research directions. Below we address the weaknesses (***W***) and questions (***Q***) raised by the reviewer. We hope the reviewer could consider raising the score given the additional evaluation in feature generation and the interpretation of the ablation results provided in the rebuttal.\n\n***Technical novelty (W1)***\n\nAs emphasized in the general response, one major technical contribution of this work is the representation generator. Unlike previous works such as [5, A] ([48] not related to image generation) which require ground-truth images to provide the representation during the generative process, our approach does not rely on such impractical requirements and demonstrates the possibility of unconditionally generating pre-trained self-supervised representations.\n\n***Additional evaluation of feature generation (Q1, Q2)***\n\nAs per the reviewer’s request, below we provide the reference points for FD and RCG’s performance using representations sampled from training images.\n\n***Q1: Reference points for FD***\n\nSince the MoCo v3 encoder is trained on the ImageNet training set, the representation distribution in the training set can be slightly different from that in the validation set. To establish a better reference point, we compute the FD between 50K randomly sampled representations from the training set and the representations from the entire training set, which should serve as the lower bound of the FD for our representation generator. The result is an FD of 0.38, demonstrating that our representation generator (with an FD of 0.48) can accurately model the representation distribution.\n\nWe also evaluate the representation generator against the validation set, resulting in an FD of 2.73. As a reference point, the FD between 50K randomly sampled representations from the training set and the validation set is 2.47, which is also close to the FD of our representation generator. We will include both results in the revision.\n\n***Q2: Performance using representations sampled from training images***\n\nIn Table 9(a), we evaluate our image generator under different conditions, including oracle representations from ImageNet training images. The oracle conditioning yields 4.37 FID and 149.0 IS, while conditioning on our generated representations achieves 5.07 FID and 142.5 IS. This further demonstrates the effectiveness of our representation generator in producing realistic and high-quality representations.\n\n***FID evaluation scheme (Q3)***\n\nFor FID in the main paper, we follow ADM’s evaluation suite which computes FID w.r.t. the ImageNet training set. This evaluation suite is widely adopted in prior works, so we need to follow it to make a fair comparison with them. Evaluating on the training set versus the validation set does not significantly change FID, and the ablation results trend remains consistent. To ensure consistency with the main paper, we will revise the FID results in the ablation section accordingly.\n\n***Interpretation of the ablation results (W2)***\n\nMost of the results in Tables 7-9 are standard hyper-parameter sweeps. Through these experiments, we aim to provide insights into which hyper-parameters are important and require tuning for future applications of our system.\n\nTwo results are particularly interesting: Table 9(a), which we have interpreted in our response to ***Q2***, and Table 7(a), which ablates different pre-trained encoders. We will explain the findings from Table 7(a) below.\n\n***MoCo v3 features vs. DINO/iBOT features (W2, Q4)***\n\nIn Table 7(a), using representations from MoCo v3 achieves better FID than using representations from DINO/iBOT. This is likely because only MoCo v3 uses an InfoNCE loss. Literature has shown that optimizing InfoNCE loss can maximize uniformity and preserve maximal information in the representation [1]. The more information in the representation, the more guidance it can provide for the image generator, leading to better and more diverse generation. To demonstrate this, we compute the uniformity loss on representations following section 4.1.2 from [1]. Lower uniformity loss indicates higher uniformity and more information in the representation. The uniformity loss of representations from MoCo v3, DINO, and iBOT is -3.94, -3.60, and -3.55, respectively, which aligns well with their generation performance. We will include this result and discussion in the revision.\n\n[1] Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere\n\n***Generative models and self-supervised learning (W2, Q5)***\n\nWe thank the reviewer for initiating this discussion. The community has a long-standing belief in the synergy between self-supervised learning and generative models: good representations should enhance the generative process, and generative models can learn robust representations. As the reviewer mentioned, several papers have provided evidence supporting the latter. On the other hand, we focus on the former, showing that explicitly providing generated representations can significantly improve unconditional generative models, which also offers new and compelling evidence to support the synergy.\n\n***Limitations and negative societal impacts***\n\nApplying RCG to other data types is beyond this paper's scope and could be an interesting future direction. It requires a pre-trained encoder, typically available off-the-shelf for common data types. We will include this discussion in the revision.\n\nWe acknowledge that image datasets can contain various types of human bias, including biases in data collection and labeling. RCG's unsupervised nature could mitigate labeling bias as it does not rely on human-provided labels. However, we agree that this topic is beyond the paper’s scope and will refrain from making this claim in the revision. We will also include discussions on potential misuses.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for providing lots of insightful and constructive feedback. We will definitely improve our manuscript accordingly. We are glad to see the commonly recognized strengths highlighted by the reviewers:

1. The presentation of the paper is clear and concise (zDRz, fB14, pEoh).

2. The problem studied in the paper is difficult (zDRz), of importance (xRbx), and relevant (pEoh).

3. The introduced framework is novel (xRbx), sound (pEoh), and intuitive (zDRz). It “will be able to provide inspiration not only in unconditional image generation but also in a variety of other fields” (zDRz), “opening up interesting research directions” (pEoh).

4. The empirical results are extensive and robust (zDRz, pEoh). The performance improvement on unconditional generation is significant (xRbx, fB14).

We would like to reemphasize a major technical contribution of this work: we demonstrate the possibility of ***unconditionally generating a representation*** pre-trained by state-of-the-art self-supervised learning methods. These generated representations can be used as conditions to improve the unconditional generation performance of various image generators. Such a representation generator is key to enabling unconditional generation without relying on ground-truth images during the generative process.

Furthermore, we note that the contribution of our work extends beyond the technical aspects. The ground-breaking empirical finding that unconditional generation can rival the performance of conditional generation by generating and conditioning on representations is a significant contribution. We believe this approach and the promising results have the potential to liberate image generation from the constraints of human annotations and rekindle the community’s interest in the fundamental problem of unconditional generation.

As there are no outstanding common questions, we will address each reviewer’s specific questions in separate responses. We are also happy to continue the discussion if the reviewers have any further questions or concerns.

Meta-review: This paper addresses the problem of unconditional image generation using latent representations. An image generator is conditioned on pre-computed image representations, at the generation stage. The idea, though very simple seems to result in high generation quality surpassing several existing baselines. All the reviewers voted for acceptance and I concur with their decision.
Decision: Accept (oral)