BiGR: Harnessing Binary Latent Codes for Image Generation and Improved Visual Representation Capabilities
Authors: Shaozhe Hao
Xuantong LIU
Xianbiao Qi
Shihao Zhao
Bojia Zi
Rong Xiao
Kai Han
Kwan-Yee K. Wong
Venue: ICLR 2025 Poster (2025)
Keywords: Image generation
Generative model
Representation learning
Primary area: generative models
TL;DR: We present a novel conditional image generation model that unifies generative and discriminative tasks effectively.

Abstract: We introduce BiGR, a novel conditional image generation model using compact binary latent codes for generative training, focusing on enhancing both generation and representation capabilities. BiGR is the first conditional generative model that unifies generation and discrimination within the same framework. 
BiGR features a binary tokenizer, a masked modeling mechanism, and a binary transcoder for binary code prediction. 
Additionally, we introduce a novel entropy-ordered sampling method to enable efficient image generation. 
Extensive experiments validate BiGR's superior performance in generation quality, as measured by FID-50k, and representation capabilities, as evidenced by linear-probe accuracy. 
Moreover, BiGR showcases zero-shot generalization across various vision tasks, enabling applications such as image inpainting, outpainting, editing, interpolation, and enrichment, without the need for structural modifications. Our findings suggest that BiGR unifies generative and discriminative tasks effectively, paving the way for further advancements in the field. We further enable BiGR to perform text-to-image generation, showcasing its potential for broader applications.

Published as a conference paper at ICLR 2025
BIGR: HARNESSING BINARY LATENT CODES FOR IM- AGE GENERATION AND IMPROVED VISUAL REPRE- SENTATION CAPABILITIES
Shaozhe Hao
Xuantong Liu
Xianbiao Qi Shihao Zhao?
Bojia Zi
Rong Xiao
Kai Han
Yean-Yee Wong
'The University of Hong Kong
"Hong Kong University of Science and Technology
3 IIntellifusion
"The Chinese University of Hong Kong
{szhac, shzhao, Kykwong]ecs hku.hk gixianbiao@gmill com kaihanxehku.nk
Project page: https: /htp///aaoosa// github.io/BiGn Code and models: https: /eg//ggttthtt cww./aoozoh.iiiiittco.
Baseline (LlamaGen)
FID-50K
Linear Probe
3.81
64.3
Baseline Durs
Baseline Ours
Figure 1: BiGR generates high-quality images while improving the discriminative capabili- ties of the representations. Left: Generated 512x512 samples, 256x256 samples, and class- conditional editing samples. Right: BiGR vs. LlamaGen (Sun et al., 2024). We visualize image features from 100 classes in ImageNet-IK validation split using t-SNE (van der Maaten Hinton, 2008), where the same color indicates the same class. Our model produces features with greater discriminative separability and enhances both generative and discriminative performance.
ABSTRACT
We introduce BiGR, novel conditional image generation model using compact binary latent codes for generative training. focusing on enhancing both generation and representation capabilities. BiGR is the first conditional generative model that unifies generation and discrimination within the same framework. BiGR features a binary tokenizer, masked modeling mechanism, and binary transcoder for binary code prediction. Additionally, we introduce novel entropy-ordered sam- pling method to enable efficient image generation. Extensive experiments vali- date BiGR's 's superior performance in generation quality, as measured by FID-50k, and representation capahilities, evidenced by linear-probe accuracy. Moreover, BiGR showcases zero-shot generalization across various vision tasks, enabling applications such image inpainting, outpainting, editing, interpolation, and en- richment, without the need for structural modifications. Our findings suggest that BiGR unifies generative and discriminative tasks effectively, paving the way for further advancements in the field. We further enable BiGR to perform text-to- image generation, showcasing its potential for broader applications.
Project lead Corresponding authors

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Image generation is experiencing revolutionary growth driven by the advancements in diffusion models (Ho et al., 2020; Rombach et al., 2022; Peebles Xie, 2023; Ma et al., 2024) and au- toregressive models (Esser et al., 2021; Sun et al., 2024; Tian et al., 2024). While these mod- els have demonstrated impressive performance, their representation capabilities are under-studied. As revealed by Balestriero & LeCun (2024), reconstruction- compelling results but fails to provide strong latent representations for perception. It has been a long-desired goal of the research community to design good image generator which can also serve as a strong feature extractor.
learning often produces visually
Centered around this goal, previous studies (Chen al., 2020a; Li et al., 2023a) representation capabilities of generative models have primarily focused on unconditional generation. Despite con- ditional generation (Peebles & Xie, 2023; Sun al., 2024; Li al., 2024) has emerged as a recent research trend and garnered much attention, investigations of the representation capabilities of con- ditional generative models remain limited. In conditional image generation, conditions are added to guide the generation process. However, this guidance is absent in downstream discriminative tasks. This weakens the relationship between features and categories, and thereby diminishes the repre- sentation capabilities of the features. We validate this limitation using the latest class-conditional image generation model (Sun et al., 2024) (see Fig. (right)), and stress the necessity of improving the representation capabilities of conditional generative models.
We introduce BiGR, novel conditional image generation model that utilizes compact Binary latent codes for Generative tasks with improved Representation capabilities. BiGR is trained exclusively through a generative process by reconstructing tokens without relying on any discriminative losses. We compress
image into sequence of binary codes using lookup-free quantization (Yu et al., 2024; Wang et al., 2023) and utilize our model to predict these binary codes. We emphasize that BiGR is the first conditional image generation model that unifies generative and discriminative tasks, achieving improved performance across both. Below, we describe our model design, generative and discriminative use, and zero-shot generalized applications.
Our framework, built upon the language model architecture, has three major components, namely (1) a binary tokenizer that converts a pixel-level image into a sequence of binary latent codes, (2) a transformer equipped with full bidirectional attention, and (3) a binary transcoder that transforms continuous features into -iittull-ddsstiiuteed binary codes. We train BiGR using the masked mod- eling approach (Bao et al., 2022; Chang et al., 2022; Li et al., 2023a). This modification, deviating from the typical autoregressive approach, expands token interaction without altering the structure of Llama. Paired with tailored inference process and inherent visual representations, BiGR can perform both generative and discriminative tasks.
For generative purpose, we design sampling method that iteratively unmask tokens in sequence, with the order determined by the binary entropy magnitude from the predicted Bernoulli distribution probabilities. This approach requires only small number of sampling iterations which significantly accelerates the inference process. As a result, we achieve high efficiency in image generation com- pared with diffusion models, which require multiple steps to remove noise. Through extensive experiments, we show that BiGR performs on par with, or even surpasses, existing basclines in quantitative metrics.
For discriminative purpose, we perform average pooling on the intermediate features in BiGR. By this straightforward operation, BiGR exhibits significantly stronger representation capabilities than comparable models, which has been empirically validated through linear probe evaluation. Due to the compactness of binary codes and the global information from masked modeling, the feature representations produced by BiGR can more effectively linearly separate visual categories in down- stream discriminative tasks.
Moreover, we explore the zero-shot generalization capabilities of BiGR within the generation do- main. Unlike autoregressive models that must predict tokens in raster order, the masked modeling mechanism offers a huge flexibility during inference, allowing for the design of task-specific strate- gies. As a result, BiGR can perform various vision tasks in a zero-shot manner, without requiring any structural changes parameter fine-tuning. In this paper, we showcase applications of our model in image inpainting, outpainting, editing. interpolation, and enrichment. We further extend BiGR to perform text-to-image generation, highlighting its potential for broader applications. The
2

---

Published as a conference paper at ICLR 2025
generated results are provided in the Appendix. We believe that further applications of BiGR can be unlocked through community efforts.
To summarize, our BiGR possesses the following prominent advantages: (i) Uniformity - -iiRR is the first conditional image generation model that unifies generative and discriminative tasks within the same model. By modeling compact binary latent codes, BiGR delivers strong performance in both tasks compared to existing models. (ii) Efficiency- BiGR generates images at a low time cost, attributed to the small number of sampling steps required in the iterative unmasking process, while still maintaining high generation quality. (iii) Flexibility BiGR can be flexibly employed for var- ious vision applications, such as inpainting, outpainting. editing, interpolation, and enrichment in a zero-shot manner, without the need for task-specific structural changes or parameter fine-tuning. (iv) Scalability- BiGR demonstrates scalability both generative and discriminative tasks, as evi- denced by comprehensive evaluations of both generation quality and linear-probs performance.
2 RELATED WORK
Binary latent code modeling Binary latent code, also known as hashing (Wang al., 2017), has demonstrated great effectiveness in visual representations due to its compactness and discrete- ness (Cakir et al., 2019; Jiang & Li, 2018; Shen et al., 2015; Wei al., 2021; Wu et al., 2019). In the realm of visual generation, the study of binary tokenizers has recently attracted notable atten- tion, referred as look-up free quantization in Yu et al. (2024) and as binary autoencoder in Wang et al. (2023). Binary tokenizers can enhance the codebook utilization for vector- necor-quantization meth- ods (Esser et al., 2021; Van Den Oord et al., 2017), facilitating image and video generation. Wang et al. (2023) introduces Bernoulli diffusion process that operates on Bernoulli Bernoulli-disriiutee distributed vari- ables to generate binary latents. Our work studies this type of tokenizers and we propose a novel generative framework for uniform conditional generation and visual representation.
Generative representation learning Representation learning has long been an important topic, with self-supervised methods (He et al., 2020; Chen et al., 2020b; Caron et al., 2020; Grill et al., 2020; Caron et al., 2021: Zhou et al., 2022) dominating the field in the past few years. Some works learn visual representations through generative modeling. For example, iGPT (Chen al., 2020a) predicts pixels in a manner similar to GPTs (Brown et al., 2020), while MAE (He et al., 2022) and MAGE (Li et al., 2023a) reconstruct masked image regions. ViT-VQGAN (Yu et al., 2022a) studies the representation capabilities of unsupervised generative models. However, these methods involve specialized designs for discriminative tasks and are not directly suited for conditional image generation. Our work broadens this scope by proposing a conditional image generation framework that consistently delivers both high-quality generation and strong representation capabilities.
Conditional image generation Conditional image generation has gained significant attention re- cently. Existing works on this topic can be broadly grouped into two categories: diffission mod- els (Ho et al., 2020; Song et al., 2021; Rombach et al., 2022; Peebles & Xie, 2023; Ma et al., 2024; Chen et al., 2024b), which gradually denoise a random Gaussian noise, and autoregressive mod- els (Esser et al., 2021; Yu et al., 2022b;a; Sun et al., 2024; Tian et al., 2024), which predict the next tokens similarly to language models. The models based on masked prediction (Chang et al., 2022; Li et al., 2023a; Chang et al., 2023) can be classified as autoregressive models, as discussed in (Li et al., 2024). In this paper, for clarity, we use "autoregressive" to specifically refer to models that use causal attention and next-token prediction, and "mask" to refer to models using masked modeling. Although conditional generative models can produce visually compelling images, their representation capabilities have rarely been studied. Our work aims bridge this gap.
METHOD
Our framework based on masked language model that operates directly on binary latent codes derived from images. We train the model by masking a portion of the input tokens and learning to unmask them using predicted output tokens. The prediction is achieved through a Bernoulli diffusion process (Wang et al., 2023), which is well-suited for generating binary latent codes. In sampling, we determine the order of tokens to be unmasked based on the magnitude of entropy computed from the predicted Bernoulli distribution probabilities. To obtain latent representations, perform average pooling on the intermediate features of our model. We present the overview of BiGR in Fig. 2. We describe the details of each of its components below.
3

---

Published as a conference paper at ICLR 2025
Training
Generation
iters
0.8
0.3
0.4
Bemoull sample
Binary Transcoder
Liama (bidirectional)
WBCE
loss
Binary
Projection
Representation
Tokenizer
G
M
M
M
Average
Encode
Masked tokens
Figure 2: Overview of BiGR. For simplicity, we display only bit for each token, although each token actually consists of K bits in length. Left: We outline the training of BiGR. Starting with binary codes from binary tokenizers, we append a condition token and mask partial tokens. These tokens are projected into continuous embeddings and processed by the Llama backbone. The outputs undergo Bernoulli denoising process in the binary transcoder to generate probabilities, penalized by the weighted binary cross-ent cess-py loss (wBCE) masked positions. Right: We illustrate the generation process (detailed in Sec. 3.3) and the representation acquisition via average pooling.
3.1 PRELIMINARY
We first review the binary tokenizer and the Bernoulli diffusion process that underpin our model.
Binary tokenizer An image tokenizer T can encode image E 3 into sequence of latent codes { 52.... .(...."") = Txx), where each Ç* represents the code at specific spatial posi- tion. Binary tokenizers (Yu al., 2024; Wang al., 2023), also known lookup-free quantization, transform the code into binary format by
= (() L((" +},
and a corresponding token index r¹ can be computed by
K
[k:]
(2)
where 2²[k] denotes the k-th bit of the binary code 2", and K is the number of binary bits (i.e, code dimension), resulting in total of 2 token indices. Using Eq. (2), Yu et al. (2024) indexes image tokens with the binary code z² and build a vocabulary of size 2 for generative purposes. In contrast, approach focuses on directly modeling the sequence of binary codes {2
Bernoulli diffuusin We generate binary codes through a Bernoulli diffusion process (Wang et al., 2023), which effectively models permoulli-distribuce variables. Specifically, Bernoulli diffusion process adds Bernoulli noise from the starting point 2 ~ q(29):
(=*1===11~ = B (z/; (1 3¹) 0.5,*)
1=1,2 .T.
Here, B denotes Bernoulli distribution, and the timestep t out of the total T is denoted perscript. We model the denoising process by p(=1~1|2" which predicts the Bernoulli distribution probabilities for the binary code at the previous timestep. By iterating the denoising process, starting with a coin toss B(0.5), we can finally generate binary codes that follow Bernoulli distributions. In model, the binary transcoder is the component that integrates the Bernoulli diffusion process, responsible for denoising Bernoulli noise and predicting binary codes.
3.2 MASKED MODELING ON BINARY LATENT CODES
Backbone We build method upon the transformer- basedd language model Llama (Dubey al., 2024: Touvron et al., 2023b;a). Unlike language, an image not naturally modeled as causal sequence of tokens, but instead, each token should have access all others to better capture global
4

---

Published as a conference paper at ICLR 2025
visual information. Therefore, replace the causal attention commonly used in language models with bidirectional attention, and let the model predict masked tokens instead of next tokens.
Input projection In the input space, instead of looking up embedding vector with token index, we use a simple linear layer that projects the binary code onto the embedding space. This technique has recently been explored for continuous- valuue tokenizers in Tschannen et al. (2023), and we find that it also works well for binary-valuee tokenizers. We maintain standard conditional embeddings and mask embeddings, where the conditional embedding is appended at the start of the sequence, and the mask embedding replaces inputs at masked positions.
Mask-token prediction During training, simply mask a portion of image tokens with learn- able mask token [M]. The fraction of masked tokens follows a cosine schedule, as used in Li et al. (2023a). We compute losses only for the masked positions, where the model predicts the values of the masked tokens. Formally, let fo represent the language model, and { {=in, ) denote the se- quence of binary codes that are partially masked. Here, M {m=} indicates whether the i-th position is masked (m; 1) or left unmasked (m, 0). We obtain outputs the masked positions from the language model {1e) Jm,= = [o({=m, }==1), which are distributed a continuous space.
Binary transcoder We transform the model outputs h into binary codes' z through a Bernoulli diffusion process (Wang al., 2023). In particular, we learn a denoising network Js with a Sigmoid function S to model
(=(=->11=²)= - B S(a (,,))),
which predicts the probabilities of the Bernoulli distribution conditioned the intermediate feature h. Consequently, binary codes can be generated by sampling from these probabilities. Follow- ing Wang et al. (2023); Ho et al. (2020), the training target is the binary residual, Le., 2 @ 20 where @ represents the element-wise XOR operation. The training objective is simply an element-wise weighted binary cross-entropy (wBCE) loss, expressed
1/k = (z' @ [ E {0, 1]
Ph = ((g (1,,, [))] E [0,
(5)
K
L = K
Σ
we (ye log Pk + (1 - yk) log(1-ps)) pp))),
(6)
k=1
K
where
Wkk = (1 yk)
vs wv/KK+ ys- -0 (1-
mn/K) = + 1/K.
km
Here, V/ represents the target, and phe is the predicted probability for the k-th bit in the binary code. The element- loss weight wp is applied to mitigate the imbalance between Os and 1s, calculated based on their respective ratios in a K -ddimensional code. The constant term 1/ added to prevent nearly-zero weights that could impede training. In training, we jointly optimize the language model fe and the denoising network Js using the loss defined in Eq. (6).
Visual representation Once trained, our model inherently possesses strong visual representations. Given an image, we input it into the model without any masks, along with an unconditional token appended. We then perform average pooling the continuous-valued features h to derive the global representation of the given image. We observe that the most discriminative representation originates not from the final layer but from the middle layers within the transformer blocks, line with the findings in Yu et al. (2022a); Chen et al. (2020a). As a result, we use the intermediate features as the final image representation.
3.3 ENTROPY-ORDERED GENERATIVE SAMPLING
For image generation, we design a sampling strategy for our model, enabling it to iteratively predict tokens from fully masked sequence. Unlike in training, where mask positions are randomly chosen at each step, during sampling, the order in which tokens are unmasked follows a predefined criterion.
We arrange the masked tokens according to the binary entropy magnitude calculated from the pre- dicted probabilities. The binary entropy is defined as:
1 Σ
H
Pa loga Ps + (1 - PM) log -px),
(8)
K
Since the operation is pise,ion-wise, we omit the superscript of positions for simplicity.
5

---

Published as a conference paper at ICLR 2025
which ranges from 0 to 1. Here, low value indicates high prediction confidence (i.e., when py is closer to either 1 or 0). Therefore, a confidence score can be derived from 1 - H. illustrating the model's confidence in this prediction. Following Li et al. (2023a), we add noise sampled from random Gumbel distribution multiplied by the temperature r to the confidence score.
At each sampling iteration, we select and unmask proportion of masked positions with the highest confidence scores. To unmask each token, we obtain its binary codes by performing Bernoulli sam- pling from the distribution (pss). The unmasking ratio follows a cosine schedule as used in Chang et al. (2022); Li et al. (2023a). This process operates over N sampling iterations. When the mask ratio drops to zero, the sampling progresses to the last iteration where all tokens are unmasked, marking the completion of the generation process.
4 EXPERIMENT
4.1 IMPLEMENTATION DETAILS
Model contiguration We use the binary autoencoder (B-AE) introduced by Wang et al. (2023) as our binary tokenizer. The downsampling rate of the autoencoder is 16, projecting a x56 image into a 16x 16 token sequence. We train four variants of the binary autoencoders designed with four different binary code dimensions, namely 16, 20, 24, and 32. With the four binary tokenizers, we train our BiGR of three different sizes based on Llama (Touvron et al., 2023a), namely L. (316M), XI. (743M), and XXI. (1.38B). For the binary transcoder, we follow Li al. (2024) to employ an MLP Is with an adaptive LayerNorm, with sizes of 20M, 56M, and 104M respectively. For clarity, we denote the s-sized variant with a B-dim tokenizer as BiGR-S-dB. e.g., BiGR-L-dI6.
Training details We train all models for 400 epochs, with L-sized models using a batch size of 1024 and the others using a batch size of 512. Our L/XL-sized models are trained on A800 GPUs, while XXL-sized models are trained on 32 A800 GPUs. We maintain consistent training settings across all compared models based on the model size.
Sampling Our model inherently supports classifier-free guidance (CFG) (Ho & Salimans, 2022) through the Bernoulli diffusion process. Within our sampling process, four hyperparameters are involved: CFG scale, Gumbel temperature (7), the number of sampling iterations (N), and the number of Bernoulii denoising steps (T)). We identify the optimal hyperparameter setting for all models. We set CFG to 2.5 for all quantitative evaluations, which has shown to be effective across all our models. We set T to 100 as default for all models. See more details in Appendix A.
4.2 UNIFORM PERFORMANCE
Evaluation We evaluate the uniformity of BiGR by concurrently comparing generative and dis- criminative performance. We evaluate generation quality on ImageNet- IK 256 256 by reporting Frechet Inception Distance (FID) the main metric, along with Inception Score (IS), sFID, Preci- sion (Pre.), and Recall (Rec.) as auxiliary metrics. All metrics are obtained using 50K generated samples. We assess representation capabilities through linear-probe evaluation, reporting the top-1 and top-5 accuracies, abbreviated as ACC1 and ACC5, on ImageNet- 1 256x256 validation split. We follow standard practice (He et al., 2022) by using a parameter-free BatchNorm (loffe, 2015) layer and a linear classifier layer to classify the model features. We use the intermediate features from the 10-th layer for L-sized models, the 15-th layer for XL-sized models, and the 32-nd layer for XXL-sized models, as experiments on d16 models demonstrate these configurations yield the best performance. Additionally, we compare the inference speed, specifically the time taken to generate each image using one 4090 GPU with a batch size of 64.
Comparison Starting from the latest autoregressive generation baseline LlamaGen (Sun et al., 2024), we comprehensively analyze two major components in this paper, namely (1) training ob- jectives, specifically categorical loss (cat.) and binary loss (bin.), and (2) modeling types, including masking and autoregressive (AR) approaches. In total, we compare five models in Tab. 1, training four models -0, S1, S2, and with different configurations, excluding LlamaGen. For Llama- Gen, the generative performance results under a fair setting of 256x256 image generation, as reported in their paper. We conduct our own evaluation of linear probe performance using their pretrained model. The inference time of all models is tested the same machines by us.

---

Published as a conference paper at ICLR 2025
Table 1: Uniformity comparison. We compare the generative and discriminative performance of our model against LlamaGen (Sun et al., 2024) and three other settings, varying by tokenizers, training objectives, and modeling types. We use KV cache to accelerate all AR models.
Generative
Discriminative
Model
Tokenizer
Objective
Type
Timel
FID|
ISt
sFID
Pre.f
Rec.t
ACCI
ACCS
LJamaGin
VQGAN
Cat.
AR
0.13
3.81
248.28
8.49
0.83
0.52
40.5
64.4
S0
B-AE
Cat.
AR
0.15
3.21
239.17
5.38
0.83
0.54
23.8
44.2
S1
B-AF
Cat.
Mask
0.10
3.85
261 81
6.10
0.85
0.47
61.1
83.2
S2
B-AE
Bin.
AR
1.04
7,50
164.31
6,56
0.85
0.41
45.2
69.3
(Ours)
B-AE
Mask
0.69
3.17
262.14
5.59
0.86
0.50
64.3
Table 2: Binary transcoder comparison.
Table 3: Sampling order comparison. We in- clude the autoregressive variant for reference.
Geserstiee
Disciiinattve ACCI
Birnry thiestive
FiDj
1S1
.FD0
Nec.T
ACES
Type
Onder
Timei
FED1
151
sFIDL
Pre.1
Rec.1
weo Aeronall Arawising
Binci ICC
5.64
212.34
9.89
0.78
0.52
631
$4.8
1.04
0:41
Borooall dossoding
Raster
8.81
0.54
Predect
274.36
0.7
63.0
Rand.
174.11
Mask
Ouns
Observation As shown in Tab. 1, our model significantly outperforms other methods across all main evaluation metrics. In addition, we have the following observations. (1) By comparing Lla- maGen and SO, using binary autoencoder provides better generative performance and worse dis- criminative performance compared to VQGAN. (2) For generation, AR modeling is better suited for categorical loss, while masked modeling more appropriate for binary loss. (3) For discrim- ination, masked modeling drastically outperforms AR modeling for both losses, with binary loss further enhancing performance. (4) Masked modeling achieves significantly faster inference speed compared AR modeling due to its fewer sampling iterations, with the binary objective taking more time resulting from the diffusion process. This can be further accelerated by reducing sam- pling iterations and diffusion timesteps, as discussed in Fig. 3. To conclude, BiGR, which employs masked modeling on binary latent codes, achieves the best uniform performance on both generative and discriminative tasks, accompanied by an efficient inference runtime.
4.3 MODEL ANALYSIS
We analyze each component of our proposed method below. All experiments are conducted on BiGR-L-d16 unless otherwise specified.
Binary transcoder We apply Bernoulli denoising process (Wang et al., 2023) as our binary transcoder to generate probabilities of Bernoulli distributions, from which the binary codes are sampled. We experiment with two variants, namely (1) predicting the initial clean latent 2o. and (2) predicting the element-wise exclusive OR (XOR) value between the latent at the t-th timestep z' and 2". We find empirically the latter performs better, and thus, we adopt this setting for all of our models. Alternatielly, a naive approach involves using a direct binary cross-entropy (BCE) loss to train the model, replacing the Bernoulli denoising process. We compare these three variants in Tab. 2. Our method outperforms the other two variants across all main metrics. We observe that using direct BCE generates very smooth images which harms the generative performance. XOR prediction yields better generative and discriminative performance compared to 2 prediction.
Sampling strategy In this paper, we propose simple entropy-ordered sampling strategy tailored for the masked training paradigm. We compare our method with two alternative sampling orders, namely (1) a raster-scan order similar to the autoregressive approach, and (2) a random order. Like our strategy, both compared methods are applied to the same trained model. The comparison results of the generative evaluation are reported in Tab. 3. The results indicate that the proposed sampling strategy is the best fit for our model's generative purposes.
Inference hyperparameters We evaluate the impact of two hyperparameters specific to our model on its performance. (1) We first present the FID results and sample time for different numbers of sampling iterations N on the left side of Fig. 3. We observe that larger models generally achieve lower FID values, although they also increase sample time. In addition, more sampling iterations do not guarantee better performance, as different-sizee ations. For example, the L-sized model achieves its best performance with 20 iterations, rather than with larger numbers. (2) On the right side of Fig. 3. we present the results for different numbers of diffusion timesteps T. The results indicate that diffusion timesteps have marginal impact on
models have varying optimal sampling iter-
7

---

Published as a conference paper at ICLR 2025
3.4
Model size
--10
Mindll
3.2
FID-50K
4
XXL
3.0
N4-22
20-23
3-10
2.8
M4-25
8-20
25
100
Sample Time (s/img)
Sample Time (s/img)
Figure 3: Relationships between FID-50K and sample time across varying inference hyper- parameters. We compare different numbers of sampling iterations N (left) and varying diffusion timesteps (right) for three model sizes. All other hyperparameters are kept at their default settings.
70
Linear-probe ACC1
3.0
FID-50K
68
2.5
Model
Model
2.0
XXL
iFID
XXL
16
20
24
32
16
20
32
Dimension of binary codes
Dimension of binary codes
Figure Evaluation of generative and discriminative performance across different model sizes. We report results for all tested tokenizers across four different dimensions of binary codes. We include the reconstruction FID (rFID) for each binary tokenizer for reference (grey points).
generative performance, suggesting that our model can achieve comparable generation quality with fewer diffusion timesteps. This can significantly accelerate generation speed, especially for larger models. For example, with 10 diffusion timesteps, the XXL-sized model can achieve an FII) of 2.73 speed of 0.85s per image.
Model size and code dimension We validate that our model is scalable by testing the performance of different-sized models using tokenizers with various code dimensions. Note that the dimension of the binary codes only alters the number of parameters in the input and output linear projections, resulting in minimal effects on the overall model size. The evaluation results of both generative and discriminative performance are shown in Fig. 4. Our model generally performs better with larger sizes across all code dimensions, as indicated by both generative and discriminative metrics.
Besides, we have the following observations from Fig. 4. (1) When the model size is small, it be- comes challenging to model large-dimensional codes, such as a dimension of 32 for the L-sized model, especially for generative purpose. (2) In contrast, the model size increases, the improve- ment for smaller-dimensional codes is relatively modest, indicating that these codes are easier to model and can be effectively handled by smaller-sized models. (3) An exception arises in the linear- probe evaluation of models with 32-dimensional codes, where our XL-sized model outperforms the XXL-sized model. We hypothesize that this may be due to the optimal transformer layer for fea- ture representation identified in the 16-dimensional model, which might not be the best choice for 32-dimensional models of the XXL size.
Unconditional training Our model is a class- conditonniiional generative model. Intuitively, condi- tional generative training adds condition guidance that is absent in downstream discriminative tasks, which can diminish the representation capabilities of the model. We validate this conjecture by comparing the linear-prohe performance of our model with that of its unconditional counterpart. We train the unconditional model by replacing the class conditional tokens with single unconditional token, and keep the inference process unchanged. We evaluate BiGR-L-d20 alongside its uncon- ditional counterpart, and report the results in Tab. 4. The unconditional counterpart demonstrates better representation capabilities than our conditional model, indicating that discriminative tasks are more challenging for conditional generative models.
Resolution 512x512 Using a binary autoencoder that projects a 512x512 image into 32x32 binary latent codes, enable our model generate 512x512 images by increasing the input
8

---

Published as a conference paper at ICLR 2025
Table 4:
Linear-probs evaluation of condi- tional and unconditional counterparts.
Training
ACC1
ACC5
Cond.
67.5
87.5
Figure 5: Generated 512x511 samples. Table 5: Generative performance comparison Table 6: Linear-prohe top-1 accuracy on ImageNet-IK. I; our evaluation results.
Uncond.
68.3
on 256 256 ImageNet- IK benchmark.
Type
Model
#Params.
FID.
187
Method
#Tokems
Parums
ACCI1
LXT-LZZ (Peehles & Xic, 2023)
458M
5,02
167.2
Maño
375M
6866
Diff
SIT XL2 (ODE) (Ma 2024)
T=T-XI
675M
675M
2,27
2.15
258.1
278.2
SwAV (Carun et at. 2020)
20200) 2020b)
93M
76.5
75.3
MaskCHTT (Chang
S/T-XL/2 (SDK)
227M
182.1
Con.
DYOL (Grill et 2020)
DINO (Cim at. 2021)
375M
85M
78.5
75.3
CAE (Chen 2024c)
301M
78.1
227M
1.40
80.4
CMAE (Huung
73.9
VIE-VQGAN ot 2022a)
1.78
175.1
iBOT
2023)
RQTmi (L.ce sl., 2022)
MAR-dI6(Tian 2024)
310M
3.8
3.30
7.55
274.4
134.0
MIM
MAR et 2022)
RETT 2027)
14x14
304M
73.5
25.8
YAR
VAR-124
VAR 420
600M
1.08
2.57
2,09
302.6
312.9
MAGE (Li et al., 2023a)
BigRKIAN (Brock, 2018)
16x16
328M
344M
61.3
78,9
VAR d30
2.00
1.92
323.1
IGPT-L (Chen 20.00a
32x
60.3
MAR B aLi 2024)
208M
2.31
281.7
1GPT-1.
48x48
65.2
MAR
MAR-L
479M
1.78
296.0
VIT-VQGAN-R (Ya 2022a)
65.1
MAR-HI
943M
1.55
363.7
Gen.
VIT-VQGAN-L
32x32
L.7B
73.2
RCG (13 et al., 20236)
16x 16
304M
77.6
LammaGen B (Sun et al. 2024)
Liumatien
343M
111M
3.81
5,46
248.3
193.6
1DAA (Chen et 20244)
304M
75.0
LamaaGen-XI.
775M
Ilaurafien-11
16x16
40.5
LiumaGen
1.40
MAR-BI Bf (Li al.,
16x16
208M
57.9
LlamaGen-38
3.18
222.3
Cond.
MAR-L"
16x16
479M
59.1
BIGR-L--024
336M
275.7
gmn.
MAR-H
943M
60.0
Ours
BIGR-XIL-224
799M
2.49
278.8
BiGR-L-120 (Ours)
BIGR XXL. 432
1.58
2,35
277.2
RIGI--XI--52 (Oun)
16x16
795M
69.8
quence length to 1024. We train such a binary autoencoder with a code dimension of 32 and train model to accommodate this sequence length. We showcase the generated samples in Fig. 5, with additional samples available in Appendix F.
4.4 SYSTEM-LEVEL COMPARISONS
We re-emphasize that the goal of this work is to propose a uniform conditional generative model that can produce high-quality generations while maintaining strong representation capabilities. There- fore, surpassing -fate-of-the-ar the-art models across all metrics is not within the scope of this research. We provide a more comprehensive comparison in Appendix E.
Conditional image generation We present comparison of the generative performance of our model with leading generative systems in Tab. 5. The recent MAR (Li al., 2024) achieves the lowest FID, and SiT (Ma et al., 2024) sets the state-of-thed oh--tr-rrttttttt among diffusion- based models. Our model maintains top-tier generative quality among the first echelon of approaches. Besides, BiGR significantly outperforms LlamaGen.
We compare the linear- -roobe-prooeeaee results of our model and the previous meth- ods specifically designed for discriminative tasks. The results are shown in Tab. 6. We categorize the compared models into several types: contrastive (Con.). masked image modeling (MIM), gen- erative (Gen.), and conditional generative (Cond. gen.). This classification not entirely precise, as some models may use multiple losses for training. like MAGE (Lii et al., 2023a). Our model fairly compared the conditional generative models, which solely rely on plain reconstruction loss without discriminative designs, such as specialized losses, augmentations, or additional data. For LlamaGen (Sun et al., 2024) that has the same model architecture as ours, we use the feature from the same layer for linear layer training. For MAR (Li et al., 2024), since their structure largely resembles MAE (He et al., 2022), we follow MAE's approach and train the linear layer on top the encoder outputs. BiGR significantly outperforms the other conditional generative models.
Visual representation
4.5 ZERO- SERO GENERALIZED APPLICATIONS
The nature of the masked modeling mechanism allows the of BiGR in a wide range of appli- cations in a zero-shot manner, without the need for task-specific structural changes or parameter fine-tuning. We present the results of BiGR applied across various tasks in Fig. 6.

---

Published as a conference paper at ICLR 2025
Inpainting
Outpainting
Editing
Interpolation
Enrichment
Figure 6: Zero-shot generalization. We present samples of inpainting, outpainting, editing, in- terpolation, and enrichment. The original image is marked with purple border, with a pink box highlighting the masked region. Images without the purple borders are generated by our model.
Inpainting & Outpainting Given an image with mask, we use the unmasked regions to initialize the model inputs, enabling it to generate the remaining masked tokens. This generation process guided by an unconditional token, which ensures that no additional information introduced, allowing the model to focus solely the existing image information. This approach enables high- quality and diverse inpainting and outpainting.
Class-conditional editing Unlike inpainting and outpainting, class-conditional editing is guided by a specific class condition, allowing the model to edit the masked region with a designated class object. Other operations remain consistent with inpainting and outpainting.
Class interpolation We interpolate between two class conditions by calculating a weighted sum in the embedding space. We then use the resulting interpolated embedding to guide the generation process. This interpolation process demonstrates that our model can generalize visual characteristics across different classes rather than merely memorizing each class.
Image enrichment Our model can also enrich visual details in a low-resolution image, a process we call image enrichment. Specifically,, we first upsample 128x 128 image resolution of 256 256 and encode it into a sequence of 16x 16 tokens. This approach leverages the model's generative capabilities to enrich images from low-resolution inputs.
CONCLUSION
We introduce BiGR as the first conditional generative model that unifies generative and discrimina- tive tasks within the same framework. Through extensive experiments, we highlight its uniformity, efficiency, flexibility, and scalability. Our results demonstrate that BiGR achieves decent perfor- mance in both generation quality and linear separability. Additionally, showcase its application in various zero-shot generalized tasks. We believe BiGR has the potential to be adapted for a broader range of applications in the future.
Limitations (1) Our sampling strategy involves numerous hyperparameters to tune, resulting in
substantial search space; thus, the reported models may not represent the optimal settings. (2) The model's sequence length fixed during training, making it inflexible to accommodate inputs of varying lengths. Consequently, generating higher-resolution images requires re-training the model.
10

---

Published as a conference paper at ICLR 2025
Acknowledgement This work is partially supported by the Hong Kong Research Grants Council General Research Fund (Grant No.: 17211024).
REFERENCES
Randall Balestriero and Yann LeCun. How learning by reconstruction produces uninformative fea- tures for perception. In ICML, 2024.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transform- ers. In ICLR,, 2022.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dall---3 pdf. 2023.
Andrew Brock. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
Fatih Cakir, Kun He, Sarah Adel Bargal, and Stan Sclaroff. Hashing with mutual information. 1EEE TPAMI, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments, In NeurlPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming- Hiung-suaan Yang, Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In ICML, 2023.
Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024a.
Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang. James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-$) allphas: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024b.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In JCML, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020b.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo. Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representa- tion learning. IJCV, 2024c.
Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning. arXiv preprint 144. 2024d.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In NeurIPS, 2019.
11

---

Published as a conference paper at ICLR 2025
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In ICLR, 2017.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Jean-Bastier Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurlPS, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In ICML, 2020.
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance. arXiv preprint
MrXiv:2207.12595 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders stronger vision learners. IEEE TPAMI, 2023.
Sergey loffe. Batch normalization: Accelerating deep network training by reducing internal covari- ate shift. arXiv preprint arXiv: 2015.
Qing- Yuan Jiang and Wu-Jun Li. Asymmetric deep supervised hashing. In AAAI, 2018.
Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In CVPR, 2023.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, 2022.
Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In CVPR, 2023a.
Tianhong Li, Dina Katabi, and Kaiming He. Self-conditioned image generation via generating representations. arXiv preprint arXiv:2312.03700 2023b.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint 1xXiv:2406.18833 2024.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- and Sain- ing Xie. SiT: Exploring flow and diffusion- based generative models with scalable interpolant transformers. In ECCV, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
12

---

Published as a conference paper at ICLR 2025
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: datasets. In ACM SIGGRAPH, 2022.
Scaling stylegan to large diverse
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:211 14, 2021.
Fumin Shen, Chunhua Shen, Wei Liu, and Heng Tao Shen. Supervised discrete hashing, In CVPR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Kegiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, al. Journeydb: A benchmark for generative image understanding. In NeurlPS, 2023.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 0.Xiv:2400.006555, 2024.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.08 02905, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint (oXXi:2307.09888 2023b.
Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative infinite-vocabulary transformers. arXiv preprint arXiv:2312.02116, 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurlPS, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008
Jingdong Wang, Ting Zhang, Nicu Sebe, Heng Tao Shen, et al. A survey learning hash. IEEE TPAMI, 2017.
Ze Wang, Jiang Wang, Zicheng Liu, and Qiang Qiu. Binary latent diffusion. In CVPR, 2023.
Xiu-Shen Wei, Yang Shen, Xuhao Sun, Han-Jia Ye, and Jian Yang. -net Learning attribute- aware hash codes for large-scale fine-grained image retrieval. In NeurIPS, 2021.
Dayan Wu, Qi Dai, Jing Liu, Bo Li, and Weiping Wang. Deep incremental hashing network for efficient image retrieval. In CVPR, 2019.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. In ICLR, 2022a.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. TMLRR, 2022b.
13

---

Published as a conference paper at ICLR 2025
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation. In ICLR, 2024.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In ICLR, 2022.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL IMPLEMENTATION DETAILS
Model configuration The configuration settings for the model architecture, training and infer- ence of BiGR across different model sizes are provided in Tab. 7. Unless otherwise specified, our model uses this default setting in the main paper. For the inference hyperparameters, we observe the following patterns:
CFG scale: larger CFG scale produces smoother images with clearer class features, while smaller scale enhances fine-grained details.
2. Gumbel temperature: A higher Gumbel temperature increases generation diversity but reduces quality, whereas a lower temperature improves quality at the cost of diversity.
3. Sampling iterations: 20-30 iterations generally perform well. Using 10 iterations speeds generation but slightly reduces quality, while more than 30 iterations has minimal impact but slows down generation.
4. Diffusion timesteps: 100 steps typically yield good results. Performance remains largely consistent across broad range of 10-200 steps, with only marginal differences.
Table 7: The default configuration settings of three models: BiGR-L, BiGR-XL, BIGR-XXL.
BiGR-L
BiGR-XL
BiGR-XXL
Config
Value
Config
Value
Config
Value
Architecture
Architecture
Architecture
former layers
24
Transformer layers
36
Transforner layers
48
Transformer heads
16
Transformer heads
20
Transtormer heads
24
Transformer dimensions
1024
Transformer dimensions
1280
Transformer dimensions
1536
MLP layers
MLP layers
MLP layers
MLP dimensions
1024
MLP dimensions
1280
MLP dimensions
1536
Training
Training
Training
size
1024
size
Batch size
$12
Epochs
400
Epochs
400
Epochs
400
Weight decay
2e-2
Weight decay
2e-2
Weight decay
2e-2
Learning rale
le-4
Leurning rate
le-4
Learning rale
le-4
Total diffusion timesteps
256
Total diffusion timesteps
256
Total diffusion timesteps
256
Inference
Inference
Inference
CFG scale
2.5
CFG scale
2.5
CFG scale
2.5
Sampling iterations
20
Sampling iterations
25
Sampling iterations
25
Gumbel lemperature
0.17
Gumbel temperature
0.25
Gumbel lemperature
0.30
Diffusion timesteps
100
Ditfusion timesteps
100
Diffusion timesteps
100
Binary transcoder After producing Bernoulli distribution probabilities through Bernoulli denois- ing, there are two ways to obtain binary codes: deterministic and non-deterministic. For deter- ministic method, values are set to 1 if the probability exceeds 0.5, and otherwise. In contrast, for non-deterministic methods, we sample directly from the Bernoulli distribution to obtain 0 and 1 values. We empirically compare these two methods, as shown in Tab. 8 and find that the deterministic approach slightly outperforms its deterministic counterpart. As result, we adopt the non-deterministic approach for all models presented the main paper.
Table 8: Comparison of deterministic and non-deterministic sampling.
Determ.
FID
ISt
sFID
Pre.1
Rec.t
3.19
239.79
6.25
0.84
0.52
X (Ours)
3.17
262.14
5.59
0.86
0.50
Sampling strategy In our sampling strategy, the implementation of Eq, (8) in the main paper may encounter "nan" issue caused by the logarithmic operation. Since we only need to compare the relative magnitudes of different entries, we can instead use a value with the same trend to mimic the exact confidence. We use 2 |ps- 0.5| the final confidence value in our implementation.
Adaptive LayerNorm We empirically find that adaptive LayerNorm (adaLN) has a marginal ef- fect performance. Following the approach in Tian et al. (2024), implement shared adaL.N
15

---

Published as a conference paper at ICLR 2025
that uses single MLP to obtain the shift, scale, and gate values for all transformer layers, which adds only a minimal number of parameters.
Linear probe Following the linear-probe evaluation protocol outlined in (He et al., 2022), the LARS optimizer with a momentum of 0.9. We train the linear head for 100 epochs, using a batch size of 256 along with 8 gradient accumulation steps. We use warm-up period of 10 epochs and set the learning rate to 0.1. An extra BatchNorm layer is added before the linear classifier, without affine transformation. We refrain from using mixup, cutmix, drop path, or color jittering, and the weight decay is set to zero, We use the same linear-probe setting for all compared models in the main paper.
B TEXT-TO-IMAGE GENERATION
We validate the effectiveness of BiGR for text-to-image generation (T2I).
Model adaptation We follow the practice of LlamaGinn to adapt our class- cans-conniiional image gen- eration model into a text-to-image generation model. Most parts of our model remain unchanged, except for replacing the class-condition token embedding with the text token embedding extracted by a text encoder. We use T5-XXL (Raffel et al., 2020)² as our text encoder. The maximum text to- ken length is set to 120, with left fett-padding applied during both training and inference. The extracted text token embeddings are projected through an additional MLP layer and appended to the patch tokens. Conditioning for image generation relies solely on these appended text token embeddings, without using adaptive LayerNorm or cross-attention, Note that this minimal model adaptation is intended validate the model's capability for text-to-image generation. More refined designs could be implemented to enhance generation performance, which we leave for future research.
Training dataset We train our T2I model on a 20M subset of LAION-400M (Schuhmann al., 2021) and JourneyDB (Sun et al., 2023), a large-scale dataset containing 4M high-quality images annotated with corresponding text captions. The training images are center-cropped and resized to resolutions of 256x256 and 512x512.
Implementation details We conduct experiments using BiGR-XL-d24. For simplicity, we train our T2I model in single stage, unlike the two-stage approach used in LlamaGen. The T2I model is initialized with the pretrained weights of our class-conditional generation model. We train the model on 32 A800 GPUs in three stages: (1) We initially train the model for 250K steps with a batch size of 1024 using JourneyDB at a resolution of 256x256 (2) We then finetune it for 450K steps with batch size of 256 on both datasets at resolution of 512 512; (3) Finally, we further finetune the model for 500K steps on JourneyDB at a resolution of 512.
Generation samples We present in Fig. 7 and with long prompts in Fig. 8, showing that our model performs well with both short selected from LlamaGen (Sun et al., 2024) and various T2I works, such as PixArt (Chen et al., 2024a), Imagen (Saharia et al., 2022), and DALL-E 3 (Betker al., 2023). These prompts unseen in the training set, demonstrating that our T2I model gen- eralizes well. Current results reveal the strong potential of BiGR in text-to-image generation. Note that our training data, model size, and training duration are relatively limited, and we only train a 512x512 T21 model, which has relatively low image resolution. We believe that scaling up train- ing data and model size, along with exploring advanced techniques such as incorporating higher resolutions, fine-tuning autoencoders, and using images with higher quality, can further enhance BiGR's s T2I performance. We will explore this in future research.
text-to-image generation samples with short prompts
and long prompts. The prompts
C VISUALIZATION OF THE SAMPLING ORDER
We propose an entropy-ordered sampling process in this paper. To explore this sampling order fur- ther, we visualize the generated results different iterations within the process. The visualization of the process is presented in Fig. 9. We observe that early iterations capture class-level characteristics, while subsequent iterations generate finer object-related details. In the final stages, visual quality steadily improves.
hhttps://huggingeee cog.://non......oggggggggo ee/tl-11
16

---

Published as a conference paper at ICLR 2025
mon looks up at the sturry sky, lonely.
An astronout riding horse.
Crocodile in a sweater.
dog reading thick book.
the black hole the
table tep with vase of flowers on it.
on illustration of teapot.
the Eiffel Tower winter
snowy mountein.
A jellyfish rocket.
chimporzee
on
0 Tiger
photo of four-leaf clover made of water.
A dragon fruit wearing karate belt in the snow.
bottom shallow stream
baby penguin
rabbit, forest spring.
illustration of stylish swardsma.
Chinese pointing of grapes.
parrot with carning, Vermeer style.
made vegetobles
fisheye lens view turtle sitting forest.
A dog that has meditating time.
digital art.
alpaca mode of colorful building blocks, cyberpunk.
small coctus with happy Sahara desert.
A photo of confused grizzly bear calculus class.
wont to supplement vitamin c, please help me paint related food.
pickup truck driving through desert environment.
Figure 7: Text-conditional x12 image generation samples with short prompts. The results are generated by BiGR-XL-d22.
17

---

Published as a conference paper at ICLR 2025
nature vs humon nature, surreal, UHD, 8k, hyper details, rich colors, photograph.
on Emu, focused yet playful, ready for competitive matchup, photorealistic quality with cartoon vibes.
A worker that looks like mixture of cow working hard to type code.
Half human, half robot, repaired human, human flesh warniar, mech display, man in mech, cyberpunk.
troveler navigattng boat countless mountains, Chinese painting.
realistic landscape shot of the Northern Lights doncing snowy mountain ronge in Icelond, with long exposure cupture the motion and vibrant colors.
high controst portrait of very happy fuzzy panda dressed chef in a high end kitchen making dough. pointing of flowers on the wall behind him.
poster machonical cat, technical Schematics viewed from front and side light white bluepnint paper, illustration drafting style, illustration, typogrophy, conceptual art, dark fontasy steampunk, cinematic, dark fartasy.
century Scottish several brightly colored rocks on colorful beach, the style luminous spheres, 3840x2160, emek golan, translucent color, 32k uhd, toyen, captivating
wizard with mysterious smile piencing gaze, enigmatic, photorealistio
incredibly detailed, sharpness, detail, cinematic lighting
phottggraph portrait statue of a pharooh wearing stoampunk glosses, t-shirt and leather jacket.
Rikaahu with angry expression eyes, with lightning around it, hyper realistic style.
Oppenheimer sits chain, watching a nuclear exposition with huge. mushroom cloud, 120mm.
A dair image of lemur wearing red magician hat and blue coat per performing with cards in gorden.
silhouette grand piano ovenlooking dusky cityscape viewed from a top floor panthouse, rendered in the bold vivid style of vintage trovel poster.
A
a fontastical setting, highly desailed furry humonoid skunk piercing eyes confidently peses
Pirate ship trapped in cosmic moelstrom nebulo, rendered cosmic beach whirlpool engine, volumetric lighting, spectaculas ambient lights, light pollution, cinematic atmesphere, art nouveau style, illustration art artwork by Sensei oye. intricate detail.
Art style of snowboarder in mid-gir performs a trick on block rail, wearing blue sweatshirt and black with arms outstretched.
Design a letter A, 3D stereoscopic Ice material Enterior light blue Conceptual product design
An ancient stone Colossus with Stephon Martinière, dark yellow and light emerald, color painting, Denis Sarazhin, dark emerald silver, expresssionsmm high detail.
medium shot, wearing
Futuristic Blind box
animal hide jacket.
The serene snewy landscape background, dotted with frees, complements the scene. angie perspective emphasizes the trick's height and skill.
toy Handcrafted
Exquisite effect
masterfully rendered
Full body display Ultra- high precision.
the character in digital art, copturing the intricate details of fur and clothing texture.
Figure 8: Text-conditional 512x512 image generation samples with long prompts. The results generated by BiGR-XL-d24.
18

---

Published as a conference paper at ICLR 2025
Iter=0 (start)
iter=2
Iter=4
Iter=6
Iter=
iter=15
iter=24 (end)
iter=0 (start)
Iter=2
iter=44
iter=6
iter=10
iter=15
iter=24 (end)
L
Iter=O (start)
Iter=2
iterw4
iter=6
Iter=10
iter=15
iter=24 (end)
Iter=D (start)
Iter=2
Iter=4
Iter=
iter=10
iter=15
iter=24 (end)
Figure 9: Visualization of the entropy-ordered sampling process. For each generation sample, we present the illustration of the unmasked tokens at the top and the corresponding generated images the bottom for each sampling iteration. each iteration, the masked positions are filled with the output tokens from the current iteration. Here, use BiGR-XL-d24, setting the total number of sampling iterations to 25.
D FAILURE CASES
We show some failure cases of our generation results in Fig. 10. It is challenging to generate high- quality, authentic human faces and fingers, clear and accurate numbers and signs, and highly com- plex scenes with many objects. These issues common in conditional generative models, arising from the significant visual quality differences across categories the ImageNet- 1K training dataset and the autoencoder's limited ability reconstruct complex details.
19

---

Published as a conference paper at ICLR 2025
Figure 10: Failure cases in generating high-quality human faces and fingers, clear numbers and signs, and complex scenes.
E ADDITIONAL SYSTEM-LEVEL. COMPARISON
We provide more comprehensive comparison of different leading models. We compare generative performance Tab. and discriminative performance Tab. 10.
ADDITIONAL GENERATED SAMPLES
We provide additional 512x512 samples and 256x 256 samples generated by our model in Fig. We also include uncurated generated samples from various classes in Fig. 12 to 23.
G ETHICS STATEMENT
We recognize the ethical risks of image generation, such potential misuse for harmful content. Our research aims to promote positive uses like creativity and education, with commitment to responsible application. Safeguards and continuous ethical oversight are strongly encouraged.
20

---

MaskGIT-re Published as a conference paper at ICLR 2025
Table Model comparison of generative performance on ImageNet-IK, Metries include Frechet inception distance (FID), inception score (IS), precision (Pre.) and recall (Rec.). All models are tested on x56 ImageNet- benchmark. The suffix "-re" denotes the use of rejection sampling.
Type
Model
#Params.
FID
1St
Pre.t
Rec.t
BigGAN (Brock, 2018)
112M
6.95
224.5
0.89
0.38
GAN
GigaGAN (Kang et al., 2023)
569M
3.45
225.5
0.84
0.61
StyleGanXL (Sauer al., 2022)
166M
2.30
265.1
0.78
0.53
LDM-4 (Rombach et al., 2022)
400M
3.60
247.7
DiT-L/2 (Peebles & Xic, 2023)
458M
5.02
167.2
0.75
0.57
Diffusion
DiT-XL/2
675M
2.27
278.2
0.83
0.57
SiT-XL/2 (ODE) (Ma et al., 2024)
675M
2.15
258.1
0.81
0.60
SiT-XL/2 (SDE)
675M
2.06
277.5
0.83
0.59
MaskGIT (Chang et al., 2022)
227M
6.18
182.1
0.8
0.51
Mask.
227M
4.02
355.6
VQGAN (Esser ct al., 2021)
227M
18.65
80.4
0.78
0.26
VQGAN
1.4B
15.78
74.3
VQGAN-re
1.4B
5.20
280.3
AR
ViT-VQGAN (Yu ct al., 2022a)
1.7B
4.17
175.1
VIT-VQGAN-re
1.7B
3.04
227.4
RQTran, (Lee al., 2022)
3.8B
7.55
134.0
RQTran.-re
3.8B
3.80
323.7
VAR-d16 (Tian et al., 2024)
310M
3.30
274.4
0.84
0.51
VAR-d20
600M
2.57
302.6
0.83
0.56
VAR
VAR-d24
1.0B
2.09
312.9
0.82
0.59
VAR-d30
2.0B
1.92
323.1
0.82
0.59
MAR-B ct al., 2024)
208M
2.31
281.7
0.82
0.57
MAR
MAR-L
479M
1.78
296.0
0.81
0.60
MAR-H
943M
1.55
303.7
0.81
0.62
LlamaGen-B (Sun al., 2024)
111M
5.46
193.6
0.83
0.45
LlamaGen-L
343M
3.81
248.3
0.83
0.52
LlamaGen-XL.
775M
3.39
227.1
0.81
0.54
AR
LlamaGen-XXL
1.4B
3.09
253.6
0.83
0.53
LlamaGen-3B
3.1B
3.05
222.3
0.80
0.58
BiGR- L-d24
336M
2.71
275.7
0.84
0.53
Ours
BiGR-XLL-d24
799M
2.49
278.8
0.84
0.55
BiGR-XXL-BR
1.5B
2.36
277.2
0.83
0.55
21

---

Published as a conference paper at ICLR 2025
Table 10: Linear-probe top-1 accuracy on ImageNet-IK. MIM denotes masked image modeling. : our evaluation results.
Method
#Tokens
Params
ACC11
Contrastive methods
CPC (HenafT, 2020)
303M
71.5
MoCo (He et al., 2020)
375M
68.6
SimCLR (Chen al., 2020b)
375M
76.5
SwAV (Caron et al., 2020)
93M
75.3
DINO (Caron et al., 2021)
85M
75.3
BYOL (Grill et al., 2020)
375M
78.6
CAE (Chen et al., 2024c)
304M
78.1
CMAE (Huang et al., 2023)
86M
73,9
iBOT (7hou et al., 2022)
304M
81.0
MIM
BEiT (Bao al., 2022)
16x16
307M
73.5
MAE (He et al., 2022)
14x
304M
75.8
MAGE (Li et al., 2023a)
16x16
328M
78.9
BiGAN Donahue al. (2017)
138M
31.0
Generative methods
BigBiGAN (Donahue & Simonyan, 2019)
86M
56.6
BigBiGAN
344M
61.3
iGPT-I (Chen el al., 2020a)
32x32
1.4B
60.3
iGPT-L
48x48
1.4B
65.2
ViT-VQGAN-B (Yu et al., 2022a)
32x32
650M
65.1
VIT-VQGAN-L
32x32
1.7B
73.2
RCG (L.i et al., 2023b)
16x
304M
77.6
I-DAE (Chen et al., 2024d)
304M
75.0
LlamaGen-LLF (Sun et al., 2024)
16x 16
343M
40.5
MAR-BE Bf (Li et al., 2024)
16x16
208M
57.9
MAR-LI
16x
479M
59.1
Cond.
MAR-Hi
16x
943M
60.0
BiGR-L-d20 (Ours)
16x16
336M
67.5
BiGR-XL-d33 XL- (Ours)
16x16
799M
69.8
22

---

Published as a conference paper at ICLR 2025
Figure 11: Additional generated 256x256 and 512x512 samples.
23

---

Published as a conference paper at ICLR 2025
Figure 12: Uncurated x56x256 samples.
Figure 13: Uncurated 256 samples. Model: BiGR-XXI-dd3
Model: B32
Class label = "Jay" (17)
Class label = "Loggerhead, loggerhead turtle, Caretta caretta" (33)
24

---

Published as a conference paper at ICLR 2025
Figure 14: Uncurated 256 samples. Model: BiGR-XXI--332
Figure 15: Uncurated 256x256 samples. Model: BiGR-XXI-dd3 Class label "Macaw" (88)
Class label = "Common iguana, Iguana, Igua- na iguana" (39)
25

---

Published as a conference paper at ICLR 2025
Figure 16: Uncurated 256x256 samples.
Figure 17: Uncurated 256x256 samples. Model: BiGR-XXL-d32
Model: BiGR-XXL-d3?
Class label = "Golden retriever" (207)
Class label "Siberian husky" (250)
26

---

Published as a conference paper at ICLR 2025
Figure 18: Uncurated 256 samples.
Figure 19: Uncurated 256 samples. Model: BiGR-XXI--332
Model: BiGR-XXI--d32
Class label = "Otter" (360)
Class label = "Madagascar cat, ring- lemur, Lemur catta" (383)
27

---

Published as a conference paper at ICLR 2025
Figure 20: Uncurated 256x256 samples.
Figure 21: Uncurated 256x256 samples.
Model: BiGR-XXI-da
Model: BiGR-XXL-d32
Class label = "Altar" (406)
Class label "Space Shuttle" (812)
28

---

Published as a conference paper at ICLR 2025
Figure 22: Uncurated 256x256 samples.
Figure 23: Uncurated 256x256 samples.
Model: BiGR-XXI-da
Model: BiGR-XXI-d32
Class label = "Alp" (970)
Class label "Volcano" (980)
29


Reviews:
[{"review_id": "hwgZbdZ4dd", "reviewer": "Reviewer_nzvc", "summary": "BiGR is a novel conditional image generation model that uses compact binary latent codes to enhance both generative and representation capabilities.​ It unifies generative and discriminative tasks within the same framework, featuring a binary tokenizer, a masked modeling mechanism, and a binary transcoder for binary code prediction. BiGR introduces an entropy-ordered sampling method for efficient image generation and demonstrates superior performance in generation quality and representation capabilities.", "questions": "- The SiT architecture reports improved results [1]; could authors clarify more about the SotA claim? \n\n- Could the authors please fix the citation format at L196 by using \\citet{}?\n\n[1] Ma et al. ECCV 2024, SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Paper is clear and well-written.\n2. The binary latent idea is new regarding Image Generation through LLMs.", "weaknesses": "- While the idea introduced is novel, it is hard for me to reason why the design choices used in the paper are leading to improving representation capabilities. It would be great if authors shed light on this much more.\n\n- The idea of the diffusion process in Binary seems interesting, however the motivation of why it should improve the overall results could be clearer.\n\n- The authors claim that they have replaced causal attention with bi-directional attention. I need help understanding how this can be done at the inference stage and what fine-tuning was done to make it work.\n\n- The LlamaGen paper reports better results for the ImageNet (256x256). So could the authors please clarify the discrepancy in the results reported?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yZQu9GstMs", "reviewer": "Reviewer_jehB", "summary": "The paper introduces BiGR, a conditional image generation model that leverages compact binary latent codes to achieve both high-quality image generation and strong visual representation capabilities. BiGR integrates a binary tokenizer, a masked modeling mechanism, and a binary transcoder to generate binary codes, to achieve efficient generation through an entropy-ordered sampling strategy. The model's design allows it to perform favorably in both generative and discriminative tasks. BiGR demonstrates strong performance on generation metrics, e.g., FID-50k and representation tasks evaluated via linear-probe accuracy. Additionally, the proposed method demonstrates its versatility in applications including image editing and zero-shot generalization on several tasks.", "questions": "1.\tHow does BiGR handle scenarios where binary latent codes introduce quantization artifacts? \n2.\tDoes entropy order sampling prioritize representative features (attribute) of an object or class? Is there any relation in order of sampling and semantic characteristics?\n3.\tIt would be insightful to include examples of failure cases.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Unified Framework for Generation and Representation. The proposed method effectively combines generative and discriminative capabilities within a single model, demonstrating strong performance in both areas.\n- Strong Experimental Validation. The model's performance is validated through extensive experiments, showing improvements over previous methods in generation quality and discriminative accuracy.\n- Fast Inference Speed. The model’s entropy-ordered sampling strategy accelerates the generation process by iteratively unmasking tokens in a confidence-guided manner. This is significantly faster compared to autoregressive models.\n- Various Applications. BiGR's ability to perform tasks such as inpainting, outpainting, and image enrichment in a zero-shot setting validates its flexibility and generalization capabilities.\n- Extensive Ablations. The paper provides thorough ablation studies that detail the impact of various components and settings on the model's performance.\n- Well written. The paper's motivation is clear and well-connected to the approach. Although some technical parts can be improved with more detail, the paper is well-written overall.\n- The limitations are discussed in the paper.", "weaknesses": "- Hyperparameter Complexity. The proposed method relies on several hyperparameters for both training and inference, such as the CFG scale, Gumbel temperature, number of sampling iterations, and number of diffusion steps. This complexity increases the time and resources required for tuning. This is discussed in the limitation section of the paper.\n\n- Fixed Sequence Length: The model’s architecture enforces a fixed sequence length during training, which restricts its flexibility to handle inputs of varying sizes. Generating images at different resolutions requires retraining the model with the new sequence length configuration. This is also discussed in the limitation section of the paper.\n\n- The diffusion and denoising process is a bit confusing. It took me a while to figure out where the noise and denoising process is applied. Clarifying that the binary transcoder is the component responsible for denoising the noise introduced in the \"Bernoulli diffusion\" section would make the flow more understandable and easier to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m2Kv3rBESD", "reviewer": "Reviewer_bv3t", "summary": "The paper introduces BiGR, a novel conditional image generation model that unifies generative and discriminative tasks within a single framework. This model is notable for several key advantages:\n\nUniformity: BiGR is the first model to integrate both generative and discriminative tasks, leveraging compact binary latent codes to achieve strong performance in both areas. This unification allows BiGR to handle tasks that typically require separate models.\n\nEfficiency: The model is designed to generate images quickly, making it more efficient than existing models. This efficiency is achieved without compromising the quality of the generated images.\n\nFlexibility and Scalability: BiGR is adaptable to various tasks, including zero-shot generalized tasks, showcasing its potential for a wide range of applications. The model's scalability is demonstrated through its performance across different model sizes and configurations.\n\nPerformance: Extensive experiments show that BiGR delivers decent performance in terms of generation quality and linear separability. The model's performance is evaluated using metrics like FID (Fréchet Inception Distance), and it is shown to perform well compared to other models like LlamaGen.\n\nInference Hyperparameters: The paper discusses the impact of hyperparameters such as the number of sampling iterations and diffusion timesteps on the model's performance. It is noted that larger models tend to achieve lower FID values, but with increased sample time, and that optimal performance varies with model size.\n\nComparison with Other Models: BiGR is compared against other models, including LlamaGen, across different settings involving tokenizers, training objectives, and modeling types. The paper highlights that while the unconditional version of the model shows better representation capabilities, the conditional version excels in generative tasks.\n\nOverall, BiGR represents a significant advancement in the field of image generation by combining generative and discriminative capabilities in a single, efficient model, with promising applications for future research and development.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Please refer to the summary.", "weaknesses": "Lack of Comprehensive Benchmarking: While the paper compares its model against LlamaGen and a few other settings, the scope of comparison is limited. The paper could benefit from a more extensive benchmarking against a wider range of state-of-the-art models to better establish its relative performance.\n\nSampling Strategy Issues: The paper mentions a \"nan\" issue in the sampling strategy due to logarithmic operations. Although a workaround is provided, this indicates potential instability in the model's implementation. A more robust solution to this problem would enhance the reliability of the model.\n\nLimited Exploration of Model Configurations: The paper primarily focuses on a few configurations (S0, S1, S2, S3) and does not explore a broader range of hyperparameters or architectural variations. This limits the understanding of the model's capabilities and its adaptability to different tasks or datasets.\n\nEvaluation Metrics: The paper emphasizes generative performance but does not provide a detailed analysis of other important aspects such as scalability, robustness, or efficiency. Including these metrics would provide a more holistic view of the model's strengths and weaknesses.\n\nAssumptions and Limitations: The paper acknowledges that surpassing state-of-the-art models across all metrics is not the goal, but it does not clearly outline the specific scenarios or applications where the proposed model excels. A clearer articulation of the model's intended use cases and limitations would help in understanding its practical applicability.\n\nTheoretical Justification: While empirical results are presented, the paper could strengthen its theoretical foundation by providing more in-depth explanations or proofs of why certain design choices, such as the non-deterministic binary transcoder, lead to better performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QuDXHs2Qhy", "reviewer": "Reviewer_FMho", "summary": "The authors propose a language model based image generation/discrimination model. Using binary latent code autoencoder, the model can learn binary codes from the image representation. Llama is originally a decoder-only model but this method use it as encoder-only model. The generation of an image is conducted by sampling from the Bernoulli distribution of outputs of the model.", "questions": "- Comments in Weaknesses should be resolved.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The method is easy to follow and the paper is easy to read\n- The architecture of this model seems to work very well on specific tasks such as inpainting, outpainting.", "weaknesses": "- The biggest problem of this approach is that it is unable to conduct text-to-image generation. Early stage of diffusion models were constrained to class-conditional generation but now it is hard to find models that is unable to do t2i generation. Even LlamaGen can receive various types of condition (especially text condition) since it is a decoder-only model.\n- I think that's why binary latent code has been enough to encode image representation. Even VQ-VAE inevitably suffers from loss of information because the latent variable is not continuous. But as the problem setting of this paper is limited to class conditional image generation, the amount of information is not large enough to see the malfunction of the binary code.\n- Also, I think it is not fair to directly compare with LlamaGen since it is designed to handle multiple modalities, not focusing on image generation. And also with an appropriate prompt, LlamaGen is able to conduct image discrimination task as well.\n- In conclusion, limiting the scope of the problem enabled the binary code to work well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)