Return of Unconditional Generation: A Self-supervised Representation Generation Method

Authors: tianhong  li
dina  katabi
kaiming  he
mit  csail
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 23:03:17

Abstract: Unconditional generation -- the problem of modeling data distribution without relying on human-annotated labels -- is a long-standing and fundamental challenge in generative models, creating a potential of learning from large-scale unlabeled data. In the literature, the generation quality of an unconditional method has been much worse than that of its conditional counterpart. This gap can be attributed to the lack of semantic information provided by labels. In this work, we show that one can close this gap by generating semantic representations in the representation space produced by a self-supervised encoder. These representations can be used to condition the image generator. This framework, called Representation-Conditioned Generation (RCG), provides an effective solution to the unconditional generation problem without using labels. Through comprehensive experiments, we observe that RCG significantly improves unconditional generation quality: e.g., it achieves a new state-of-the-art FID of 2.15 on ImageNet 256x256, largely reducing the previous best of 5.91 by a relative 64%. Our unconditional results are situated in the same tier as the leading class-conditional ones. We hope these encouraging observations will attract the community's attention to the fundamental problem of unconditional generation. Code is available at [https://github.com/LTH14/rcg](https://github.com/LTH14/rcg).

Main paper: 
 introduction: 
 Generative models have been long developed as unsupervised learning methods in the history, e.g., in the seminal works including GAN [27] , VAE [39] , and diffusion models [57] . These fundamental methods focus on learning the probabilistic distributions of data, without relying on the availability of human annotations. This problem, often categorized as unconditional generation in today's community, is in pursuit of utilizing the vast abundance of unannotated data to learn complex distributions.However, unconditional image generation has been largely stagnant in comparison with its conditional counterpart. Recent research [18, 54, 10, 11, 24, 50] has demonstrated compelling image generation quality when conditioned on class labels or text descriptions provided by humans, but its quality degrades significantly without these conditions. Closing the gap between unconditional and conditional generation is a challenging and scientifically valuable problem: it is a necessary step towards unleashing the power of large-scale unannotated data, which is a common goal in today's deep learning community.We hypothesize that such a performance gap is because human-annotated conditioning introduces rich semantic information to simplify the generative process. In this work, we largely close this gap by taking inspiration from a closely related area-unsupervised/self-supervised learning. 1 We Image Distribution (c) it learns an image generator (e.g., which can be ADM [18] , DiT [50] , or MAGE [41] ) that maps a noise distribution to the image distribution conditioned on the representation distribution.observe that the representations produced by a strong self-supervised encoder (e.g., [30, 12, 8, 14] ) can also capture a lot of semantic attributes without human supervision, as reflected by their transfer learning performance in the literature. These self-supervised representations can serve as a form of conditioning without violating the unsupervised nature of unconditional generation, creating an opportunity to get rid of the heavy reliance on human-annotated labels.Based on this observation, we propose to first unconditionally generate a self-supervised representation and then condition on this representation to generate the images. As a preprocessing step (Figure 1a ), we use a pre-trained self-supervised encoder (e.g., MoCo [14] ) to map the image distribution into the corresponding representation distribution. In this representation space, we train a representation generator without any conditioning (Figure 1b ). As this space is low-dimensional and compact [65] , learning the representation distribution is favorably feasible with unconditional generation. In practice, we implement it as a very lightweight diffusion model. Given this representation space, we train a second generator that is conditioned on these representations and produces images (Figure 1c ). This image generator can conceptually be any image generation model. The overall framework, called Representation-Conditioned Generation (RCG), provides a new paradigm for unconditional generation. 2RCG is conceptually simple, flexible, yet highly effective for unconditional generation. RCG greatly improves unconditional generation quality regardless of the specific choice of the image generator (Figure 2 ), reducing FID by 71%, 76%, 82%, and 51% for LDM-8, ADM, DiT-XL/2, and MAGE-L, respectively. This indicates that RCG largely reduces the reliance of current generative models on Regardless of the specific form of the image generator (LDM [54] , ADM [18] , DiT [50] , or MAGE [41] ), RCG massively improves the unconditional generation quality. Generation quality is measured by FID on ImageNet with a 256×256 resolution. All comparisons between models without and with RCG are conducted under controlled conditions to ensure fairness. The technical details and more metrics are in Section 4.1.manual labels. On the challenging ImageNet 256×256 benchmark, RCG achieves an unprecedented 2.15 FID for unconditional generation. This performance not only largely outperforms previous unconditional methods, but more surprisingly, can catch up with the strong leading methods that are conditional on class labels. We hope our method and encouraging results will rekindle the community's interest in the fundamental problem of unconditional generation. 
related work: 
 Generative Models. Generative models aim at accurately modeling data distribution to generate new data point that resembles the original data. One stream of generative models is built on top of generative adversarial networks (GANs) [27, 69, 37, 70, 7] . Another stream is based on a twostage scheme [63, 53, 10, 67, 40, 41, 11] : first tokenize the image into a latent space and then apply maximum likelihood estimation and sampling in the token space. Diffusion models [33, 59, 18, 54, 52] have also achieved superior results on image synthesis.The design of a generative model is mostly orthogonal to how it is conditioned. However, literature has shown that unconditional generation often significantly lags behind conditional generation under the same design [18, 41, 10] , especially on complex data distributions.Unconditional Generation. Unconditional generation is the fundamental problem in the realm of generative models. It aims to model the data distribution without relying on human annotations, highlighted by seminal works of GAN [27] , VAE [39] , and diffusion models [57] . It has demonstrated impressive performance in modeling simple image distributions such as scenes or human faces [23, 10, 18, 54] , and has also been successful in applications beyond images where human annotation is challenging or impossible, such as novel molecular design [66, 28, 26] , medical image synthesis [71, 16, 47] , and audio generation [48, 42, 25] . However, recent research in this domain has been limited, largely due to the notable gap between conditional and unconditional generation capabilities of recent generative models on complex data distributions [46, 18, 19, 41, 3, 61] .Prior efforts to narrow this gap mainly group images into clusters in the representation space and use the cluster indices as underlying class labels to provide conditioning [46, 43, 3, 35] . However, these methods assume that the dataset is clusterable, and the optimal number of clusters is close to the number of classes. Additionally, these methods fall short of generating diverse representationsthey are unable to produce different representations within the same cluster or underlying class.Representations for Image Generation. Prior works have explored the effectiveness of exploiting representations for image generation. DALL-E 2 [52] , a text-conditional image generation model, first converts text prompts into image embeddings, and then uses these embeddings as the conditions to generate images. In contrast, RCG for the first time demonstrates the possibility of directly generating image representations from scratch, a necessary step to enable conditioning on self-supervised representations in unconditional image generation. Another work, DiffAE [51] , trains an image encoder in an end-to-end manner with a diffusion model as decoder, aiming to learn a meaningful and decodable image representation. However, its semantic representation ability is still limited (e.g., compared to self-supervised models like MoCo [14] , DINO [8] ), which largely hinders its performance in unconditional generation. Another relevant line of work is retrieval-augmented generative models [5, 4, 9] , where images are generated based on representations extracted from existing images. Such semi-parametric methods heavily rely on ground-truth images to provide representations during generation, a requirement that is impractical in many generative applications. 
method: 
 Directly modeling a complex high-dimensional image distribution is a challenging task. RCG decomposes it into two much simpler sub-tasks: first modeling the distribution of a compact lowdimensional representation, and then modeling the image distribution conditioned on this representation distribution. Figure 1 illustrates the idea. Next, we describe RCG and its extensions in detail.RCG comprises three key components: a pre-trained self-supervised image encoder, a representation generator, and an image generator. Each component's design is elaborated below: Distribution Mapping. RCG employs an off-the-shelf image encoder to convert the image distribution to a representation distribution. This image encoder has been pre-trained using self-supervised contrastive learning methods, such as MoCo v3 [14] , on ImageNet. This approach regularizes the representations on a hyper-sphere while achieving state-of-the-art performance in representation learning. The resulting distribution is characterized by two essential properties: it is simple enough to be modeled effectively by an unconditional representation generator, and it is rich in high-level semantic content, which is crucial for guiding image generation. These attributes are vital for the effectiveness of the following two components. Each "Layer" consists of a LayerNorm layer [1] , a SiLU layer [22] , and a linear layer. The backbone consists of an input layer that projects the representation to hidden dimension C, followed by N fullyconnected (fc) blocks, and an output layer that projects the hidden latent back to the original representation dimension. The diffusion timestep is embedded and added to every fc block.Representation Generation. In this stage, we want to generate abstract, unstructured representations without conditions. To address this issue, we develop a diffusion model for unconditional representation generation, which we call a representation diffusion model (RDM). RDM employs a fully-connected network with multiple fully-connected residual blocks as its backbone (Figure 4 ). Each block consists of an input layer, a timestep embedding projection layer, and an output layer, where each layer consists of a LayerNorm [1] , a SiLU [22] , and a linear layer. Such an architecture is simply controlled by two parameters: the number of blocks, N , and the hidden dimension, C. [58] for training and inference. As shown in Figure 3a , during training, image representation z 0 is mixed with standard Gaussian noise variable ϵ:z t = √ α t z 0 + √ 1 -α t ϵ.The RDM backbone is then trained to denoise z t back to z 0 . During inference, RDM generates representations from Gaussian noise following the DDIM sampling process [58] . Since RDM operates on highly compacted representations, it brings marginal computation overheads for both training and generation (Appendix B.1), while providing substantial semantic information for the image generator, introduced next.Image Generation. The image generator in RCG crafts images conditioned on self-supervised representations. Conceptually, such an image generator can be any modern conditional image generative model by substituting its original conditioning (e.g., class label or text) with representations. In Figure 3b , we take MAGE [41] , a parallel decoding generative model as an example. The image generator is trained to reconstruct the original image from a masked version of the image, conditioned on the representation of the same image. During inference, the image generator generates images from a fully masked image, conditioned on the representation generated by the representation generator.We experiment with four representative generative models: ADM [18] , LDM [54] , and DiT [50] are diffusion-based frameworks, and MAGE [41] is a parallel decoding framework. Our experiments show that all four generative models achieve much better performance when conditioned on highlevel self-supervised representations (Table 1 ).Our RCG framework can easily be extended to support guidance even in the absence of labels, and to support conditional generation when desired. We introduce these extensions as follows.Enabling Guidance in Unconditional Generation. In class-conditional generation, the presence of labels allows not only for class conditioning but can also provides additional "guidance" in the generative process. This mechanism is often implemented through classifier-free guidance in classconditional generation methods [32, 54, 11, 50] . In RCG, the representation-conditioning behavior enables us to also benefit from such guidance, even in the absence of labels. Specifically, RCG follows [32, 11] to incorporate guidance into its MAGE generator. During training, the MAGE generator is trained with a 10% probability of not being conditioned on image representations, analogous to [32] which has a 10% probability of not being conditioned on labels. For each inference step, the MAGE generator produces a representation-conditioned logit, l c , and an unconditional logit, l u , for each masked token. The final logits, l g , are calculated by adjusting l c away from l u by the guidance scale, τ : l g = l c + τ (l c -l u ). The MAGE generator then uses l g to sample the remaining masked tokens. Additional implementation details of RCG's guidance are provided in Appendix A.Simple Extension to Class-conditional Generation. RCG seamlessly enables conditional image generation by training a task-specific conditional RDM. Specifically, a class embedding is integrated into each fully-connected block of the RDM, in addition to the timestep embedding. This enables the generation of class-specific representations. The image generator then crafts the image conditioned on the generated representation. As shown in Table 3 and Appendix C, this simple modification allows users to specify the class of the generated image while keeping RCG's superior generative performance, all without the need to retrain the image generator. 
experiments: 
 We evaluate RCG on the ImageNet 256×256 dataset [17] , which is a common benchmark for image generation and is especially challenging for unconditional generation. Unless otherwise specified, we do not use ImageNet labels in any of the experiments. We generate 50K images and report the Frechet Inception Distance (FID) [31] and Inception Score (IS) [55] as the standard metrics for assessing the fidelity and diversity of the generated images. Evaluations of precision and recall are included in Appendix B.1. Unless otherwise specified, we follow the evaluation suite provided by ADM [18] . All ablations and results on other datasets are included in Appendix B.1.We extensively evaluate the performance of RCG with various image generators and compare it to the results of state-of-the-art unconditional and conditional image generation methods. Several intriguing properties are observed.RCG significantly improves the unconditional generation performance of current generative models. In Table 1 , we evaluate the proposed RCG framework using different image generators. The results demonstrate that conditioning on generated representations substantially improves the performance of all image generators in unconditional generation. Specifically, it reduces the FID for unconditional LDM-8, ADM, DiT-XL/2, MAGE-B, and MAGE-L by 71%, 76%, 82%, 54%, and 51%, respectively. We further show that such improvement is also universal across different datasets, as demonstrated by the results on CIFAR-10 and iNaturalist in Appendix B.1. These findings confirm that RCG markedly boosts the performance of current generative models in unconditional generation, significantly reducing their reliance on human-annotated labels.Moreover, such outstanding performance can be achieved with lower training cost compared to current generative models. In Figure 5 RCG's unconditional generation performance rivals leading methods in class-conditional image generation. In Table 3 , we perform a system-level comparison between the unconditional RCG and state-of-the-art class-conditional image generation methods. MAGE-L with RCG is comparable to leading class-conditional methods, with and without guidance. These results demonstrate that RCG, for the first time, improves the performance of unconditional image generation on complex data distributions to the same level as that of state-of-the-art class-conditional generation methods, effectively bridging the historical gap between class-conditional and unconditional generation.In Table 4 , we further conduct an apple-to-apple comparison between the class-conditional versions of LDM-8, ADM, and DiT-XL/2 and their unconditional counterparts using RCG. Surprisingly, with RCG, these generative models consistently outperform their class-conditional versions by a noticeable margin. This demonstrates that the rich semantic information from the unconditionally generated representations can guide the generative process even more effectively than class labels. As shown in Table 3 and Appendix C, RCG also supports class-conditional generation with a simple extension. Our representation diffusion model can easily adapt to class-conditional representation generation, thereby enabling RCG to also adeptly perform class-conditional image generation. This result demonstrates the effectiveness of RCG in leveraging its superior unconditional generation performance to benefit downstream conditional generation tasks.Importantly, such an adaptation does not require retraining the representation-conditioned image generator. For any new conditioning, only the lightweight representation generator needs to be retrained. This potentially enables pre-training of the self-supervised encoder and image generator on large-scale unlabeled datasets, and task-specific training of conditional representation generator on a small-scale labeled dataset. We believe that this property, similar to self-supervised learning, allows RCG to both benefit from large unlabeled datasets and adapt to various downstream generative tasks with minimal overheads. We leave the exploration on this direction to future work.In this section, we showcase the visualization results of RCG, providing insights into its superior generative capabilities. Figure 8 illustrates RCG's unconditional image generation results on Ima-geNet 256×256. The model is capable of generating both diverse and high-quality images without relying on human annotations. The high-level semantic diversity in RCG's generation stems from its representation generator, which models the distribution of representations and samples them with varied semantics. By conditioning on these representations, the complex data distribution is broken down into simpler, representation-conditioned sub-distributions. This decomposition significantly simplifies the task for the image generator, leading to the production of high-quality images.Besides high-quality generation, the image generator can also introduce significant low-level diversity in the generative process. Figure 6 illustrates RCG's ability to generate diverse images that semantically align with each other, given the same representation from the reference image. The images generated by RCG can capture the semantic essence of the reference images while differing in specific details. This result highlights RCG's capability to leverage semantic information in representations to guide the generative process, without compromising the diversity that is important in unconditional image generation.Figure 7 further showcases RCG's semantic interpolation ability, demonstrating that the representation space is semantically smooth. By leveraging RCG's dependency on representations, we can semantically transition between two images by linearly interpolating their respective representations. The interpolated images remain realistic across varying interpolation rates, and their semantic contents smoothly transition from one image to another. For example, interpolating between an image of "Tibetan mastiff" and an image of "wool" could generate a novel image featuring a dog wearing a woolen sweater. This also highlights RCG's potential in manipulating image semantics within a low-dimensional representation space, offering new possibilities to control image generation. 
discussion: 
 Computer vision has entered a new era where learning from extensive, unlabeled datasets is becoming increasingly common. Despite this trend, the training of image generation models still mostly relies on labeled datasets, which could be attributed to the large performance gap between conditional and unconditional image generation. Our paper addresses this issue by exploring Representation-Conditioned Generation, which we propose as a nexus between conditional and unconditional image generation. We demonstrate that the long-standing performance gap can be effectively bridged by generating images conditioned on self-supervised representations and leveraging a representation generator to model and sample from this representation space. We believe this approach has the potential to liberate image generation from the constraints of human annotations, enabling it to fully harness the vast amounts of unlabeled data and even generalize to modalities that are beyond the scope of human annotation capabilities. In this section, we describe the implementation details of RCG, including hyper-parameters, model architecture, and training paradigm. We also include a copy of our code in the supplementary material. All codes and pre-trained model weights will be made publicly available.Image Encoder. We use Vision Transformers (ViTs) [20] pre-trained with MoCo v3 [14] as the default image encoder. We evaluate three ViT variants (ViT-S, ViT-B, and ViT-L) in the main paper, each trained on ImageNet for 300 epochs. We utilize the image representation after the MLP projection head, favoring its adjustable dimensionality. An output dimension of 256 has proven the most effective. The representation of each image is normalized by its own mean and variance. Detailed training recipes of our pre-trained image encoder can be found in [14] .Representation Diffusion Model (RDM). Our RDM architecture employs a backbone of multiple fully connected blocks. We use 12 blocks and maintain a consistent hidden dimension of 1536 across the network. The timestep t is discretized into 1000 values, each embedded into a 256dimensional vector. For class-conditional RDM, we embed each class label into a 512-dimensional vector. Both timestep and class label embeddings are projected to 1536 dimensions using different linear projection layers in each block. Detailed hyper-parameters for RDM's training and generation can be found in Table 5 .Image Generator. We experiment with ADM [18] , LDM [54] , DiT [50] , and MAGE [41] as the image generator. For representation-conditioned ADM, LDM and DiT, we substitute the original class embedding conditioning with the image representation. We follow ADM's original training recipe to train representation-conditioned ADM for 400 epochs. We follow LDM-8's original training recipe, with modifications including a batch size of 256, a learning rate of 6.4e-5, and a training duration of 40 epochs. We follow the DiT training scheme in [15] , which trains DiT-XL for 400 epochs with batch size 2048 and a linear learning rate warmup for 100 epochs. The β 2 of the AdamW optimizer is set to 0.95. For representation-conditioned MAGE, we replace the default "fake" class token embedding [C 0 ] with the image representation for conditioning.During the training of RCG's image generator, the image is resized so that the smaller side is of length 256, and then randomly flipped and cropped to 256×256. The input to the SSL encoder is further resized to 224×224 to be compatible with its positional embedding size. Our implementation of guidance follows Muse [11] , incorporating a linear guidance scale scheduling. Detailed hyperparameters for our representation-conditioned MAGE are provided in Table 6 . B Additional Quantitative ResultsThis section provides a comprehensive ablation study of the three core components of RCG. Our default setup uses MoCo v3 ViT-B as the pre-trained image encoder, an RDM with a 12-block, 1536-hidden-dimension backbone trained for 100 epochs, and a MAGE-B image generator trained for 200 epochs. Unless otherwise specified, all other properties and modules are set to the default settings during each component's individual ablation. The FID in this section is evaluated against the ImageNet validation set.Distribution Mapping. (c) Classifier-free guidance scale. τ = 6 achieves the best FID and IS for RCG-L. their substantial improvements over the unconditional baseline. Additionally, an encoder trained with DeiT [62] in a supervised manner also exhibits impressive performance, indicating RCG's adaptability to both supervised and self-supervised pre-training approaches.We also notice that using representations from MoCo v3 achieves better FID than using representations from DINO/iBOT. This is likely because only MoCo v3 uses an InfoNCE loss. Literature has shown that optimizing InfoNCE loss can maximize uniformity and preserve maximal information in the representation. The more information in the representation, the more guidance it can provide for the image generator, leading to better and more diverse generation. To demonstrate this, we compute the uniformity loss on representations [65] . Lower uniformity loss indicates higher uniformity and more information in the representation. The uniformity loss of representations from MoCo v3, DINO, and iBOT is -3.94, -3.60, and -3.55, respectively, which aligns well with their generation performance.Table 7b assesses the impact of model size on the pre-trained encoder. Larger models with better linear probing accuracy consistently enhance generation performance, although a smaller ViT-S model still achieves decent results.We further analyze the effect of image representation dimensionality, using MoCo v3 ViT-B models trained with different output dimensions from their projection head. Table 7c shows that neither excessively low nor high-dimensional representations are ideal -too low dimensions lose vital image information, while too high dimensions pose challenges for the representation generator.Representation Generation. Table 8 ablates the representation diffusion model and its effectiveness in modeling representation distribution. The RDM's depth and width are controlled by the number of fc blocks and hidden dimensions. Besides evaluating FID and IS on generated images, we also assess the Frechet Distance (FD) [21] between the generated representations and the ground-truth representations. A smaller FD indicates that the distribution of generated representations more closely resembles the ground-truth distribution. Since the MoCo v3 encoder is trained on the ImageNet training set, the representation distribution in the training set can be slightly different from that in the validation set. To establish a better reference point, we compute the FD between 50K randomly sampled representations from the training set and the representations from the entire training set, which should serve as the lower bound of the FD for our representation generator. The result is an FD of 0.38, demonstrating that our representation generator (with an FD of 0.48) can accurately model the representation distribution.We also evaluate the representation generator against the validation set, resulting in an FD of 2.73. As a reference point, the FD between 50K randomly sampled representations from the training set and the validation set is 2.47, which is also close to the FD of our representation generator.Image Generation. Table 9 ablates RCG's image generator. Table 9a experiments with MAGE-B under different conditioning. MAGE-B with RCG significantly surpasses the unconditional and clustering-based baselines, and further outperforms the class-conditional baseline in FID. This shows that representations could provide rich semantic information to guide the generative process. It is also quite close to the "upper bound" which is conditioned on oracle representations from Im-ageNet real images, demonstrating the effectiveness of the representation generator in producing realistic representations.We also ablate the training epochs of the image generator and the guidance scale τ , as shown in Table 9b and Table 9c . Training MAGE longer keeps improving the generation performance, and τ = 6 achieves the best FID and IS.In this section, we evaluate RCG on datasets other than ImageNet to validate its consistent effectiveness across different datasets. We select CIFAR-10 and iNaturalist 2021 [64] . CIFAR-10 represents a relatively simple and low-dimensional image distribution, and iNaturalist 2021 represents a more complex image distribution, with 10,000 classes and 2.7 million images. For CIFAR-10, we employ SimCLR [13] trained on CIFAR-10 as the image encoder and Improved DDPM [49] as the image generator. The FID is evaluated between 50,000 generated images and the CIFAR-10 training set. For iNaturalist, we employ MoCo v3 ViT-B trained on ImageNet as the image encoder and MAGE-B as the image generator. The FID is evaluated between 100,000 generated images and the iNaturalist validation set, which also consists of 100,000 images.As shown in Table 10 , RCG consistently enhances unconditional image generation performance on both CIFAR-10 and iNaturalist 2021, demonstrating its universal effectiveness across various datasets. Notably, the improvement on complex data distributions such as ImageNet and iNaturalist is more significant than on simpler data distributions such as CIFAR-10. This is because RCG decomposes a complex data distribution into two relatively simpler distributions: the representation distribution and the data distribution conditioned on the representation distribution. Such decomposition is particularly effective on complex data distributions, such as natural images, paving the way for generative models to model unlabeled complex data distributions. In Table 11 , we present a detailed evaluation of RCG's computational cost, including the number of parameters, training costs, and generation throughput. The training cost of all image generators is measured using a cluster of 64 V100 GPUs. The training cost of RDM is measured using 1 V100 GPU, divided by 64. The generation throughput is measured on a single V100 GPU. As LDM and ADM measure their generation throughput on a single NVIDIA A100 [54] , we convert it to V100 throughput by assuming a ×2.2 speedup of A100 vs V100 [56] .As shown in the Table 11 , RCG requires significantly lower training costs to achieve great performance. For instance, it achieves an FID of 4.87 in less than one day of training. Moreover, the training and inference costs of the representation generator are marginal compared to those of the image generator. This efficiency potentially enables for lightweight adaptation to various downstream generative tasks by training only the representation generator on small-scale labeled datasets.In Table 12 , we report the unconditional generation precision and recall of RCG, evaluated on Im-ageNet 256×256 following the ADM suite [18] . Larger models as well as incorporating guidance (RCG-G) both improve recall while slightly decreases precision.We include more qualitative results, including class-unconditional image generation (Figure 9 ), class-conditional image generation (Figure 10 and Figure 11 ), and the comparison between generation results with or without guidance (Figure 13 ). All these results demonstrate RCG's superior performance in image generation. We also include some failure cases in Figure 12 .Limitations. Like any other generative models, RCG can also produce unrealistic or low-quality results (see Appendix C for some examples). Societal Impact. Despite the rapid advancements in generative models, they also carry potential negative societal impacts. For instance, such models can amplify existing biases present in internet data. RCG, being a generative model, is not immune to these issues. However, it is important to note that RCG operates within an unconditional generation framework, which does not depend on human-provided labels. This characteristic might possess the potential to mitigate the influence of human biases, offering a more neutral approach to data generation compared to traditional conditional models. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Following common practice in the generative modeling literature, we do not report error bars in this paper because of the heavy computation overheads.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: see Appendix D.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We will require the users to adhere to usage guidelines for our released models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly cite the original assets in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.In this paper, the term of "unsupervised learning" implies "not using human supervision". Thus, we view self-supervised learning as a form of unsupervised learning. The distinction between these two terminologies is beyond the scope of this work.The term "unconditional generation" implies "not conditioned on human labels". As such, RCG is an unconditional generation solution. 
 
 Appendix:  


Decision: This paper addresses the problem of unconditional image generation using latent representations. An image generator is conditioned on pre-computed image representations, at the generation stage. The idea, though very simple seems to result in high generation quality surpassing several existing baselines. All the reviewers voted for acceptance and I concur with their decision.

Reviews: review: paper_summary: paper_summary: This paper proposes a very simple unsupervised image generation framework that does not rely on human labeled annotation without compromising generation quality. This framework has two stages: i) representation generator learning and ii) image generator learning. The representation generator is trained in the form of diffusion model training to take a noisy latent image as input and output the corresponding representation encoded by a self-supervised image representation model. The image generator is trained to generate the image corresponding the given representation (encoded by the self-supervised representation model). This paper demonstrates that this simple framework is effective in achieving generation quality comparable to the counterpart supervised learning methods, regardless of architecture type.
strength_weakness: strength_weakness: I do not have a major concern for this work. I only have two questions in the method.

1. When training the image generator, why was the representation of the SSL model used instead of the output of the representation generator trained in the previous step? Although the representation generator is trained to distill the generation capability of the SSL model, isn't it more appropriate to train under the same conditions as in the inference in which the output (representation) of the representation generator is used?

2. Is there any way for the two generators to be trained together in an end-to-end manner?
questions: questions: Please respond the two questions I raised in the weakness section.
limitations: limitations: This paper properly deals with the limitation and potential societal impact in the supplementary material.
paper_summary: paper_summary: This paper proposes RCG, a novel framework to enhance unconditional image generation by leveraging self-supervised representations. The main idea is first to generate self-supervised representations unconditionally using a pre-trained encoder and then condition the image generation on these representations. This process does not require human-annotated labels and involves training a lightweight diffusion model to generate representations efficiently. The authors demonstrate the effectiveness of RCG by significantly improving the quality of unconditional generation across various architectures, closing the performance gap between unconditional and conditional generation methods.
strength_weakness: strength_weakness: I found no weaknesses.
questions: questions: **[Q]** Is there any intuition as to why MoCo-v3 is the best representation for RCG?
limitations: limitations: They addressed the limitations.
paper_summary: paper_summary: This paper proposes generative models conditioned on representation obtained from a pre-trained self-supervised encoder to achieve high-quality diverse generation.
strength_weakness: strength_weakness: 1. The idea of generation conditioned on representation is not new. As the authors mentioned, conditioning on instance representations has been well studied in the community. Though the authors argue that they require ground-truth images to provide representations during generation, the issue can be addressed by adopting the idea from VAE to align those instance representations with the prior noise or by incorporating the representation generator proposed in this paper. Therefore, I would say the main contribution of this work is to improves those representation generation works by introducing a generative modeling for the representation, which however is limited.
2. The deep clustering community has also adopted the technique of self-supervised learning for a more clustering friendly representation and has achieved significant improvement in complex datasets, like ImageNet.  Therefore, generation conditioned on clustering structures also has potential to bring gaps between unconditional generation and conditional generation. I would say the studied problem has been well addressed in the community. Comparisons with conditioning on clustering should be included.
questions: questions: 1. When extending to class-conditional generation, is it required to fine-tune the model?
limitations: limitations: The limitations should include the discussions on the necessity of a self-supervised encoder and how to obtain such an encoder for datasets, especially for other modalities such as text, speech.
paper_summary: paper_summary: The paper proposes a framework, coined Representation-Conditioned Generation (RCG), that aims to bring the advantages of Conditional Generation techniques to Unconditional Generation settings.
To do so, they use the output of a representation generator network instead of class labels.
This representation generator is trained in a prior stage to approximate the distribution of image features extracted by a pre-trained self-supervised network (MoCo-v3).
They test this technique with a set of very different generative models, including a latent diffusion model, a diffusion transformer, and a masked generative transformer, showing improvement accross the board on the ImageNet Unconditional Generation benchmark.
strength_weakness: strength_weakness: a. Technical novelty is limited: [48] also generates images by first generating pretrained features to condition the image generator, and works such as [5,A] manage to generate high-quality images using representations. The remaining differentiator to existing work in terms of technical contribution is then in the choice of the pre-trained feature extractor and image generator.

b. In that light, while thorough ablation results are presented, I find the interpretation of the results lacking. If those choices are important, and it seems to be the case considering the range of FID scores in the ablations, it would be very useful to provide intuitions about how to make those choices and why.

--

[A] SODA: Bottleneck Diffusion Models for Representation Learning. Hudson et al. CVPR 2024
questions: questions: - In Table 8, it is difficult to have an interpretation of the FD as we don't have a reference point. Maybe showing FD between training and validation set could be an option, provided the number of samples for the estimates is handled carefully. Also, FD is evaluated on the training set. It would be good to have validation scores as well.

- In a similar vein, I'd like to see the performance (FID) of RCG on features sampled from training images (similar to Figure 6). This would give another indication of how good the representation generation is. This is important since the representation generation is a major, and arguably the most important, contribution of the paper.

- The authors mention in Appendix B that the FID in the corresponding section are computed on the validation set. Is that also the case for the other values in the main paper?

- For the point raised in weakness b., a precise question I'd like to ask is why, as seen in Table 7a, MoCo v3 features are so much better for this task than DINO or iBOT features that are supposed to be better in other downstream tasks.

- Also linked to weakness b., there are strong connections between generative models and self-supervised learning. In fact, works such as [61, A] or [41] on which the current paper builds a lot of their results are already generative models that explicitly double as representation learners, or even have representation learning as their main objective. Those ties should be discussed in the paper. 
Moreover, if a generative model can be a representation learner, and if using a representation can help achieve a better generative model, how to explain the good results in the submission becomes even less clear. I'd be happy to hear the authors' thoughts on that point.

In any case, I think the submission is already sound and of interest. It shows very good and solid results on a relevant topic, and opens up interesting research directions. But also, it doesn't provide much in terms of technical novelty or in terms of analysis. The paper touches upon, but barely scratches the surface of very fundamental questions related to self-supervised methods and the training of generative models. I would definitely consider raising my rating to 7 or 8 if it would have a slightly more complete evaluation of the feature generation part and if it provided a deeper analysis of the results.
limitations: limitations: The authors make an attempt to address limitations by showing failed generation. They could discuss the scope in terms of data type: are there results likely to transfer to data? what hurdles to expect in doing so?

They also try to adress societal impacts by discussing generative model biases and hypothesize that unsupervised models should significantly mitigate the influence of human bias. I'd appreciate if they would either expand on that point or alternatively refrain from making such an assumption without a stronger backing. ImageNet photos certainly have been produced by humans and are not bias-free.
Since they are willing to discuss the limitations and impacts of generative models in general, they could also mention potential misuses such as deep fakes and misinformation.

Comments: title: Thanks for the responses.
comment: I appreciate authors to address my questions. I will increase my rating.
I thank the authors for the detailed responses, I have no major concerns left.
After going through the paper, all the reviews, and the rebuttals, I believe it is a strong submission that contains significant results and prompts important discussions. As such, I would recommend acceptance and I updated my rating accordingly.
I thank the authors point out their ablation study about conditioning on cluster labels. So, I raised my score. 

However, I still feel that the technical contribution of this paper is marginal as I demonstrated in Weakness 1. Moreover, as shown in their results, the performance of conditioning on cluster labels is relatively good, though a little worse than the proposed method. That proves that generation conditioned on clustering structures has great potential to bring gaps between unconditional generation and conditional generation. Image generation without labels does not lag so behind its conditional counterpart with labels.
Thank you for your response! After reading the other reviews and the authors' comments, I stick to my score and recommend acceptance.