Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning
Authors: Ting Chen
Ruixiang ZHANG
Geoffrey Hinton
Venue: ICLR 2023 poster (2023)
Keywords: Diffusion Models
Discrete Data

Abstract: We present Bit Diffusion: a simple and generic approach for generating discrete data with continuous state and continuous time diffusion models. The main idea behind our approach is to first represent the discrete data as binary bits, and then train a continuous diffusion model to model these bits as real numbers which we call analog bits. To generate samples, the model first generates the analog bits, which are then thresholded to obtain the bits that represent the discrete variables. We further propose two simple techniques, namely Self-Conditioning and Asymmetric Time Intervals, which lead to a significant improvement in sample quality. Despite its simplicity, the proposed approach can achieve strong performance in both discrete image generation and image captioning tasks. For discrete image generation, we significantly improve previous state-of-the-art on both CIFAR-10 (which has 3K discrete 8-bit tokens) and ImageNet-64x64 (which has 12K discrete 8-bit tokens), outperforming the best autoregressive model in both sample quality (measured by FID) and efficiency. For image captioning on MS-COCO dataset, our approach achieves competitive results compared to autoregressive models.

Published as a conference paper at ICLR 2023
ANALOG BITS: GENERATING DISCRETE DATA USING DIFFUSION MODELS WITH SELF-CONDITIONING
Ting Chen, Ruixiang Zhang', Geoffrey Hinton Google Research, Brain Team
{i amt ingchen, rui xiangz geo geo ffhinton) htwp::inttttoooogoo.ooooooo...
ABSTRACT
We present Bit Diffusiin; a simple and generic approach for generating discrete data with continuous state and continuous time diffusion models. The main idea behind our approach to first represent the discrete data as binary bits, and then train a continuous diffusion model to model these bits as real numbers which we call analog bits. To generate samples, the model first generates the analog bits, which are then thresholded to obtain the bits that represent the discrete variables. We further propose two simple techniques, namely Self-Conditioning and Asymmetric Time Intervals, which lead to significant improvement in sample quality. Despite its simplicity, the proposed approach can achieve strong performance in both discrete image generation and image captioning tasks. For discrete/categoriiall image generation, we significantly improve previous state-of-the-art on both CIFAR- R- 10 (which has 3K discrete 8-bit tokens) and IMAGENET 64 x 64 (which has 12K discrete 8-bit tokens), outperforming the best autoregressive model in both sample quality (measured by FID) and efficiency. For image captioning on MS-COCO dataset, our approach achieves competitive results compared to autoregressive models.
INTRODUCTION
State-of- thr-t--ttt--ttttt art generative models for discrete data, such discrete images and text, are based on
autoregressive modeling (Van den Oord et al. Salimans et al. 2017 Parmar et al. 2018 Child
al. 2019 Roy al., 202 Jun al. 2020 Sutskever et al. [0014] Brown et 2020 Chowdhery
et al. 2022). where the networks, often Transformers (Vaswani et al.) 2017). are trained to predict each token given its preceding ones in a sequential manner or with causal attention masks. One major drawback of such approaches is that they typically require computation and memory that is quadratic to the dimension of data (e.g., sequence length or image size), leading to difficulties in modeling large images sequences. Another drawback is that, during generation, autoregressive models generate one token at a time so the total number of sequential sampling steps is often the same as the dimension of data, making it slow in generating large images or long sequences.
In contrast, diffusion models (Sohl-Dickstein et al. 2015 Ho et al.] 2020 Song et al. 2202)), or score-based generative models (Song& & Ermon 2019 2020 Song et al. (021). can model much higher dimensional data without running into computation and memory issues. During generation, diffusion models iteratively refine samples with high degree of parallelism,
the total number of sequential sampling steps can be much less than the dimension of data. However, state-of- sa-te-of-the-an diffusion models (Dhariwal & Nichol 202 Ho al. 2022 Nichol al.) 2021 Ramesh et al.. 2022] Saharia et al., 2022) can only generate continuous data (mainly real valued pixels), and have not yet achieved results competitive with autoregressive models in generating discrete/categorical data, such
as generating discrete/categorica images (Hoogeboom et al. 2021 Austin et al. 2021
In this work, we propose simple and generic approach for enabling continuous state diffusion models to generate discrete data. The key ingredient in our approach is analog bits: real numbers used to model the bits that represent the discrete data. Analog bits can be directly modeled by continuous state diffusion models, without requiring a discrete state space or re- re-formulation of the continuous
Work done student researcher Google. Code wttp:/uuhh...ooooogleeeeeaaiinii

---

Published as a conference paper at ICLR 2023
training
token_.dd-5
int2bit
cost real numbers
with scale
10
ispul to
Rinalog Bits
difusion models
br22tt
thresholding
generale
sampling
laken_ids5
0101
.105 10) -102
Figure 1: Bit Diffusion: modeling discrete data using continuous diffusion models with analog bits.
diffusion process. At sampling time, the generated analog bits can be decoded into discrete variables by a simple thresholding operation. Our approach, as illustrated in Figure [ is based on the following high-level conjecture. With strong continuous generative models (diffusion models in particular), it should not be too difficult to generate highly concentrated bimodal data where each real-valued analog bit is close the network has to model structures among analog bits that can actually lead to meaningful discrete variables after thresholding.
binary bit. To reduce the prediction loss (such as negative log likelihood),
Besides analog bits, we further propose two simple techniques, namely Self-Conditionii and Asymmetric Time Intervals that greatly improve the sample quality. We evaluate the proposed approach on both discrete image generation, and image-conditional text caption generation. On discrete CIFAR-10 10 and IMAGENET 64x64, the proposed Bit Diffusion model significantly improves both existing discrete diffusion models but also the best autoregressive model. For example, on categorical CIFAR-10, 10, the best autoregressive model (Jun et al., 2020) obtains a FID of 12.75, while our model (with 1/3 of the model size of the autoregressive model, using 100 instead of 3072 sequential inference steps) achieves a much better 6.93. For image captioning on MS-COCO dataset, model achieves a result competitive with strong autoregressive captioner based Transformer.
2 METHOD
Preliminaries We start with a short introduction to diffusion models "Sohl-Dickstein et al. 2015 Ho
let al.) 22200 Song al. 2020 2021). Diffusion models learn series of state transitions to map noise E from a known prior distribution to vo from the data distribution. To learn this (reverse) transition from the noise distribution to the data distribution, forward transition from Ey to œ; is first defined:
xt = (t) æD + - V1-+(t)6 e,
where E ~ .NO, I). (') is continuous variable, and xt) is a monotonically decreasing function from to O. Instead of directly learning neural net model the transition from æ, to at-A. one can learn neural net f(xx,t) to predict ao (or e) from æ;, and estimate - from and estimated To (or é). This training of fxx, t) is based on denoising with a &g regression loss:
La 1B 1I,f ((T(()(())))))))))))))) xo + vI-2(() e, xoll?
To generate samples from learned model, it follows a series of (reverse) state transition ØT -> ær -> -> Zn- This can be achieved by iteratively applying denoising function f on each state at to estimate 2ig and then make a transition to at- with the estimated æo (using transition rules such as those specified in DDPM (Ho al., 2020) DDIM (Song et al., 2020)). Note that state transitions in these diffusion models assume a continuous data space and state space. Therefore, one cannot directly apply it to model and generate discrete/categorical data.
Analog Bits A discrete data variable from an alphabet of size K can be represented using n = [log, KJ bits, 10". Due the discreteness, existing work has to re-formulate continuous diffusion models by adopting a discrete data space and state space (Sohi-Dickstein et al., 2015] Hoogeboom al., 2022T] Austin et al., (202)) In contrast, propose simply cast the binary bits 10.1) into real numbers IR" for the continuous diffusion models [ We term these real numbers analog bits since they learn to share the same bimodal values as binary bits but are modeled as real numbers. To draw samples, we follow the same procedure as sampling in a continuous diffusion model, except that we apply quantization operation the end by simply thresholding the generated analog bits. This yields binary bits which can be then converted into original discrete/categorical variables. Notably, there is no hard constraint to force the model to generate exact binary bits, but we After casting as real numhers, one may also transform them by shifting and scaling from 0, to -b.bb.. b..b.
2

---

Published as a conference paper at ICLR 2023
A++A
Zt
Lt-
3++ 3t 2/-A
30
T
30
(a) Standard reverse diffusion steps.
(b) Confitonnii on the previous æy estimate.
Figure 2: An illustration of reverse diffusion sampling steps (a) without or (b) with Self-Conditioning To denotes the estimation of data sample by the denoising network f at sampling step. We propose to condition the network directly on its previously generated/estimated samples.
expect strong continuous generative model to generate real numbers that exhibit very clear bimodal concentrations and this is what happens our experiments.
For simplicity, use the same regression loss function (Eq. 2 for modeling analog bits. However, it is possible to use other loss functions such as the cross entropy loss. We also note that the binary encoding mechanism for constructing analog bits is extensible well (e.g., one-hot encoding). Extensions of loss functions and binary encoding are described in the appendix
Self-Conditioniin Conditioning is useful technique for improving diffusion models (Nichol & Dhariwal 2021 Ho et al. 2022). ) However, a typical conditioning variable is either from some external sources, such class labels (Nichol & Dhariwal (02T)) low-resolution images from
another network (Nichol & Dhariwal [021] Saharia et al. [021] Ho et al. 2022). Here we propose a technique for the model to directly condition on previously generated samples of its own during the iterative sampling process, which can significantly improve the sample quality of diffusion models.
In typical diffusion sampling process, the model iteratively predicts Zy (or c) in order to progress the chain of mapping noise into data. However, as shown in Figure 2a the previously estimated 2. is simply discarded when estimating 2g from new time step, i.e. the denoising function fa, t) does not directly depend on previously estimated Zn. Here we consider slightly different denoising function of (a,, Do 1) that also takes previous generated samples as input, illustrated in Figure 2b A simple implementation of Self-Conditioning is to concatenate I; with previously estimated zo- Given that To is from the carlier prediction of the model in the sampling chain, this comes at a negligible extra cost during sampling. In order to train the denoising function f f(x,, To t), we make some small changes to the training. With some probability (e.g.. 50%), we set c. = 0 which falls back modeling without Self-Conditioning. At other times, first estimate io fw,, O, t) and then use it for Self-Conditioning. Note that we do not backpropagate through the estimated Tp so the overall increase of training time is small (e.g., less than 25%).
Asymmetric Time Intervals Besides seff-Conditioning. we identify another factor, time step f, that can also impact Bit Diffusion models. Time step is an integral part of both denoising network f( t) as well as the state transitions. During a typical reverse diffusion process, the model takes symmetric time intervals (i.e., A in t >> A) for both the state transition and time reduction itself, resulting in the same/shared t for both arguments of f(x,, t). However, we find that, when taking large reverse steps, using asymmetric time intervals, implemented via simple manipulation of time scheduling at generation, can lead to improved sampling quality for Bit Diffusion models.
More specially, with asymmetric time intervals during the sampling process, we have (a, ('), where t' + & and is small non-negative time difference parameter. Note that training remains unchanged, and the same/shared is used for both arguments of the f(x,, t). Figure iillustrates the effect with a trained Bit Diffusion model, where it asked to take two reversing steps from state I constructed using the forward diffusion, and it shows that asymmetric time intervals reduce the number of noisy pixels (after thresholding and converting back discrete variables).
((
f( ('=0.)
Figure 3: When taking large reverse step from T1=0.6 to Tr in Bit Diffusion with maximum time T 1.0, we see that asymmetric time intervals with a positive time difference & improve the denoising quality of 2 (by reducing the number of noisy pixels).
3

---

Published as a conference paper at ICLR 2023
Putting it together Algorithm and summarize the training and sampling algorithms for the proposed Bit Diffusion model with Analog Bits, Self-Conddtioning. and Asymmetric Time Intervals (via the td parameter). The proposed changes to the existing diffusion models are highlighted in blue. Note that unlike standard diffusion models (Sohl-Dickstein et al. 2015; Ho et al. 2020 Nichol & Dhariwal] 202T), we use continuous time parameterization between 0 and I instead of fixed discrete time for maximal flexibility but they perform similarly, More details of the algorithm (including some important functions) can be found in Appendix A
Algorithm 1 Bit Diffusion training algorithm.
A Algorithm 2 Bit Diffusion sampling algorithm.
def train_loas (x):
def generate (steps, cd-0) (mean-C, std-1)
Binary encoding: discrete to analog bits.
x_t
'rt2bit [x] asstya (flest)
x xreddd like
bitss - (Xbita
1) scale
for step in vange
Corrupt data.
f Get time for gurrent and next states.
t = unitormi0) 1)
L_oww
step steps
eps normal (mean-0, std=1)
t_next max (atep+1+td) / aceps, 0)
crpt agrt (gamma (t) x_bits
sqrt (1 gamma (t]) + aps
t Prodict x_c.
if not self self_cond: x_pred zeros soros_like
Compute self cond estimate.
x x_ered = zeros_ zeros_like (x_ept__ppt]
if self_cord and uniform (0, 11 0.5:
x_eeed - net (cat (Ix_ pzed)
(_aow)
x_pred
(ix_ert,, x_pred], -1), x_pred atop_gradient (x_pred)
Eatimatu x at next.. - ddim piu___or_doo_stepe
pr compute
x_t, x_ared, t_new, t_next)
x_pred = net (cat [ x_predl, 1), t) loss (x_pred x_bits)
Binary decoding discrete data. return bit2int (x_pred 0)
loss.
3 EXPERIMENTS
We experiment with two different discrete data generation tasks, namely discrete/categorical image generation, and image captioning (image-conditional text generation).
3.1 EXPERIMENTAL SETUP AND IMPLEMENTATION DETAILS
Datasets We use CIFAR- 10 Krizhevsky et al., 2009) and IMAGENET 64 64 (Deng et al. 2009
for image generation experiments. We adopt widely used FID (Heusel et al. (017)) the main evaluation metric, and it is computed between 50K generated samples and the whole training set. For image captioning, following (Chen et al., 2022), we use MS-COCO 2017 captioning dataset (Lin
et al. 2014).
Binary encoding Each pixel consists of sub-pixels (RGB channels), and each sub-pixel is an integer in [0, 256) representing the intensity. Standard continuous generative models cast RGB channels as real numbers and normalize them in [-1, 1]. For discrete image generation, we consider three discrete encoding for sub-pixels, namely UINT8, GRAY CODE, and UINT8 (RAND). In UINT8, we use 8-bit binary codes converted from the corresponding sub-pixel integer [0, 256). GRAY CODE, we assign 8-bit binary codes uniquely to each sub-pixel integer such that two adjacent integers only differ by bit. And in UINT8 (RAND), we assign 8-bit binary codes to every sub-pixel integer by randomly shuffling the integer-to-bits mapping in UINT8. The binary codes in UINT8 and GRAY CODE are loosely correlated with its original sub-pixel intensities, while UINT8 (RAND) has no correlation so each sub-pixel is a categorical variable. The details of the binary codes and their correlations with sub-pixel intensity can be found in the appendix C bits from 0, 1 to 1,, 1 for the analog bits.
We shift and scale the binary
For image captioning, we follow (Chen et al. 2022). and use sentencepiece (Kudo & Richardson 2018) with vocabulary of size 32K tokenize the captions. After tokenization, encode each token into 15 analog bits using the binary codes converted from the corresponding integer. We set the maximum number of tokens to 64 the total sequence length is 960 bits. Since we directly model bits, it is also possible to directly work with their byte representations without a tokenizer, but we leave this for future work.
Following (Brock et al. 2018 Nichol & Dhariwal 2021 center crop and area downsample images to 64x64.
4

---

Published as a conference paper at ICLR 2023
Table I: Comparison of FIDs on unconditional and class-conditional CIFAR-10. Note that both UINT8 and GRAY CODE are only partial/weakly ordinal (see Appendix CC) Our Bit Diffusion achieves state-of-the-ar FIDs in generating discrete images, beating the best autoregressive model.
FID (Unconditional)
FID (Conditional)
Method
State space
On continuous pixels (as reference):
DDPM (Ho et al. (220))
Continuous Continuous
3.17
DDPM (our reproductron)
3.14
2.95
On discrete (partial) ordinal pixels:
D3PM Gauss+Logiste (Austin al. 2021
Discrete
7.34
rLDR-10 (Campbell et al. 2022
Discrete
3.74
Bit Diffusion on UINTS
Continuous
3.48
2.72
Bit Diffusion GRAY CODE
Continuous
3.86
2.94
On categorical pixels:
D3PM uniform (Austin et al.
2021
Discrete
51.27
D3PM absorbing (Austin et al.
2021
Discrete
30.97
Autoregressive Transtormer Jun et al
2020
Discrete
12.75
Bit Diffusion on UINT8 (RAND)
Continuous
6.93
6.43
Table 2: Comparison of FIDs on class-conditional IMAGENET 64 64x The corresponding samples
can be found in Figure 4 and
DDPM (our repo.) on continuous pixels
Bit Diffusion on UINT8
Bit Diffusion on GRAY CODE
Bit Diffusion
on UINT8 (RANd)
3.43
4.84
5.14
8.76
Architecture We use the U-Net architecture (Ho et al. 2020; Nichol & Dhariwal 2021 Ronneberger iet 2015) for image generation. For CIFAR- 10, we use a single channel dimension of 256, stages and 3 residual blocks (He al., 2016) per stage, with total of 51M parameters. We only use dropout (Srivastava et al.) (014)) of 0.3 for continuous diffusion models on CIFAR- 10. For IMAGENET 64 x 64, following (Nichol & Dhariwal 2021). we use a base channel dimension of 192, multiplied by 1,2,3,4 in 4 stages and 3 residual blocks per stage, which account for total of 240M parameters For UINT8 (RAND) encoding, we find the following "softmax factorization" architectural tweak on the final output layer can lead to better performance. Instead of using a linear output layer to predict analog bits directly, we first predict probability distribution over 256 classes per sub-pixel (with each class corresponds to one of the 256 different 8-bit codes), and then map class distribution into analog bits by taking weighted average over all 256 different 8-bit codes.
For image captioning. we follow the architecture used in (Chen al., 2021 2022), with a pre-trained image encoder using the object detection task, for both autoregressive bascline as well as the proposed method. Both decoders are randomly initialized 6-layer Transformer (Vaswani al., 2017) decoder with 512 dimension per layer. For the autoregressive decoder, the token attention matrix is offset by the causal masks, but is non-masked all-to-all attention for our Bit Diffusion.
Other settings We train our models with the Adam optimizer (Kingma & Ba 2014). For CIFAR-10 10, we train the model for 1.5M steps with a constant learning rate of 0.0001 and batch size of 128. For IMAGENET 64 x 64, we train the model for 500K steps with a constant learning rate of 0.0002 and batch size of 1024. For Bit Diffusion, we use Self-Condee Self-Conditioning by default, unless otherwise specified. We use an exponential moving average of the weights during training with a decay factor of 0.9999. For our best image generation results, we sweep over few sampling syyper-parameters such as sampler (DDIM vs DDPM), sampling steps in {100, 250, 400, 1000). and time difference in {o., 0.01, 0.1, 0.2, 0.5).
3.2 DISCRETE IMAGE GENERATION
We compare our model against state-of-ther oftthe--t-tt-tttt generative models (Ho et al. 2020 Austin et al.) 2021 Campbell al. 2022 Jun et al. 2020) on generating discrete CIFAR- 10 images in Table Our model
*Our model is about 30M parameters smaller than that used (Nichol Dhariwal 2021 as we drop the middle blocks for convenience, which may have minor effect on performance.
'For VINT8 (RAND) encoding, we use learning rate of 0.0001 instead.
5

---

Published as a conference paper at ICLR 2023
DDPM continuous pixels (FID=3.43)
(b) Bit Diffusion UINT8. (FID=A.8)
(c) Bit Diffusion CODE (F1D=5.14)
(d) Bit Diffusion on UINT8 (RAND) (FID=8.76)
Figure 4: Class-conditional generations on continuous v.s. discrete ImageNet 64x64. Each row represents random samples conditioned on class, and the classes are adopted from (Nichol Dhariwal 2021), namely, 9: ostrich, 11: goldfinch, 130: flamingo, 141: redshank, 154: pekinese, 157: papillon, 97: drake and 28: spotted salamander. More samples from random classes are shown in Figure
achieves better results compared to both existing discrete diffusion models and the best autoregressive model. When compared continuous diffusion models (i.c., DDPM), our Bit Diffusion models on UINT8 and GRAY CODE can achieve similar performance.
Discrete generation of IMAGENET 64 64 is significantly harder than CIFAR- and we have not found other competing methods that report FIDs, so we only compare the proposed method against DDPM on continuous pixels. Results are shown in Table 2 We find that the diffusion model on continuous pixels has the best FID while the diffusion model on UINT8 (RAND), i.e., categorical data, has the worst FID, indicating the increase of hardness when removing intensity/order information in sub-pixels. Note that, in these experiments, there is no extra model capacity to compensate for the loss of intensity/order information since the model sizes are the same. Figure 4 shows generated images of different diffusion models continuous and discrete IMAGENET 64x64. Despite the differences in FIDs, visually these samples look similar.
Ablation of Self- Ceff-Condiionnng Figure 5 shows the effectiveness of the Self-Conditioning technique in both Bit Diffusion and continuous diffusion models. Note that the experiments are performed in three settings, namely CIFAR- 0 with UINT8, CIFAR-10 with UINT8 (RAND), and IMAGENET

---

Published as a conference paper at ICLR 2023
Wio Self-Gond
Wi Saff-Cond
Wo Self-Cond
Wi Belf-Cond
Wio Saf-Cond
W Self-Cond
236.3
6-
2
2:43
2:09
3:46
471 0
Predict
Predict x0
Predict
Predict xD
Predict eps
Predict x0
(a) 10,AR-10, UINT8.
(b) CIFAR-10. 10, UINT8 (RAND).
(c) IMAGENET 64x64.
Figure 5: Self-conditioning is generic technique that not only greatly improves Bit Diffusion but also leads to improved results for continuous diffusion models.
DDMM aleps)
30
DDIM
DDPM (100 aleps)
DDPM (100 steps)
2
20
2 20
DDPP (400 steps)
DDPM (400 sleps)
10
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0,50
0.76
1.00
Time difference
Time difference
(a) UINT8.
(b) UINT8 (RAND).
Figure 6: Effect of time difference in class-conditiona IMAGENET 64x64. Optimal time difference shrinks to zero as the number of sampling steps increases. For 100 sampling steps, non-zero time difference leads to improved FIDs.
64x64 64 with continuous pixels, where the only difference for pairs in each setting is whether the Self-Conditioning
used. For C0, we find that condiCondiiionnng greatly improves the performance across different binary encodings. We also notice that for Bit Diffusion, predicting 2o is much more effective than predicting e. For IMAGENET 64x64, we find that the proposed Self-Conditioning also leads to improved FIDs for continuous diffusion (i.e., DDPM). Therefore, we conclude that Ceff-Condiiinnin by itself is a generic technique that can benefit diffusion models on both continuous and discrete data.
Ablation of asymmetric time intervals ssoww the FID on generated IMAGENET 64 64 samples we vary the time difference parameter during the sampling process. We find that as the number of steps increases (from 100 to 400), the optimal time difference shrinks to 0. For 100 steps, a non-zero time difference leads a significant improvement of FID. We also note that for Bit Diffusion on UINT8 (RAND), using 400 sampling steps actually leads to a drastically worse sample quality than using 100 steps. This is related how the Self-Conditioning is applied and we present alternative Self- Seff-onnditionin sampling strategies in the Appendix G some of which lead to improved FIDs at cost of longer sampling time.
Concentration of generated analog bits Figure 7 visualizes the distribution of generated analog bits from 64 generated images on IMAGENET 64 64. Although there is no hard constraint on the analog bits being binary/ bimodal, the generated ones are highly concentrated on two modes, which makes the thresholding quantization easy and robust.
Pircetoo
Gand oi2E www.og 28
Generated ansiig site
Ganautts unaba Du
(a) UINT8.
(b) GRAY CODE.
(c) UINT8 (RAND).
Figure 7: Histogram distribution (50 bins) of the analog bits from 64 randomly generated IMAGENET x4 samples at Zo. with 100 DDIM steps. Most of the generated analog bits are very concentrated.
7

---

Published as a conference paper at ICLR 2023
3.3 IMAGE CAPTIONING
We compare our Bit Diffusion model with autoregressive Transformer baseline (Chen al. 2022) As mentioned, both models have similar architectures, with an object detection pretrained (Chen et al. 202I) image encoder. and randomly initialized Transformer (Vaswani al. 2017) decoder. Table presents the main comparison. Overall, our model achieves similar performance as the autoregressive model. We find that generally it only needs about 10 steps for the model to achieve good results, despite that there are a total of maximum 960 bits for caption that the model has to model. We find that the asymmetric time intervals play an important role the final performance our model, as demonstrated in Table especially when sampling steps are fewer.
Table 3: Image captioning results on MS-COCO dataset with a randomly initialized text decoder.
Method
BLEU-4
CIDEr
ROUGE-L
Autoregressive Transformer
33.9
1.18
0.57
Bit Diffusion steps)
31.5
1.00
0.55
Bit Diffusion (10 steps)
34.5
1.13
0.57
Bit Diffusion (20 steps)
34.7
1.15
0.58
Bit Diffusion (40 steps)
34.4
1.15
0.57
Table 4: Asymmetric time intervals significantly improves the performance of Bit Diffusion.
Time difference
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
5 steps
17.1
27.8
30.8
31.5
31.6
31.5
31.5
31.5
31.6
10 steps
17.6
26.3
30.7
32.6
33.4
34.0
31.3
34.5
34.6
20 steps
20.0
27.9
30.6
32.0
32.3
33.9
34.4
34.7
34.5
10 steps
20.7
27.5
30.7
32.2
32.9
33.2
33.8
34.4
34.4
5] some generated samples of our model when different inference steps are used. The model makes mistakes when the sampling steps too few, and the mistakes may not always be interpretable due to that the model directly predicts the bits behind the tokenized word pieces and small difference in bits can lead to total different words.
Table 5: Generated image captions under different number of sampling steps.
Steps=1: A group of diets for foraa
Steps=2: A group of elephants in in alufffd.
Steps=5: group of elephants standing a Irish.
Steps=10: A group of elephants standing by fence.
Steps=1: A says man fora You alu0000.
Steps=2: A says man skateboard' aa.
Steps=5: man on skateboard skatehoard.
Steps=10: A man is doing trick skateboard.
RELATED WORK
Autoregressive models for discrete data Autoregressive models have demonstrated state-of-the-art results when it comes to generating discrete data. In particular, text generation, or language modeling, is dominated by autoregressive approaches (Sutskever et al. 2014 Brown et al. 2020: Chowdhery image generation Van den Oord et 2016; Salimans al. [017]] Parmar al.) 2018 Child et al. 2019 Roy et al.) 2021 Jun ct al. 2020 Chen et al.) (2200a), where they work well on small image resolutions. However, the computation cost and memory requirement increase drastically (typically in quadratic relation) as
et al. 2022). Autoregressive models are also applied to discrete/categoree
8

---

Published as a conference paper at ICLR 2023
the size of sequence the image resolution increase, so it becomes very challenging to scale these approaches to data with large dimensions.
Diffusion models for discrete data Sfate-of-the-ar diffusion models (Dhariwal & Nichol, 202 I
Ho et al, 2022 Nichol et al., [02T] Ramesh ct al. 2022 Saharia et al. 2022) cannot generate discrete or categorical data. Existing extensions of these continuous diffusion models to discrete data
based on both discrete data space and state space (Sohl-Dickstein et al.) 2015: Hoogeboom et al.)
2021: Austin et al., 2021; Campbell et al. 2022). Compared discrete state space, continuous state space is more flexible and potentially more efficient. Our approach is also compatible with both discrete and continuous time, and does not require re-formulation of existing continuous models, thus it is simpler and can potentially be plugged into a broader family of generative models.
Another line of discrete diffusion models based on the embedding of discrete data (Li et al. 2022). One can also consider our binary encoding with analog bits as a simple fixed encoder, and the decoding/ quantization of bimodal analog bits is easy and robust via a simple thresholding operation. In contrast, the quantization of real numbers in generated continuous embedding vectors may contain multiple modes per dimension, leading to potential difficulty in threstoodingqqaaniiaaioon
Normalizing Flows for discrete data Normalizing Flows (Rezende & Mohamed, 2015 Dinh et al. 2017 Kingma & Dhariwal, 2018) are powerful family of generative models for high- high-dimensionnl continuous distributions based on some invertible mapping. However, straightforward application of flow-based models on categorical data is limited due to the inherent challenges on discrete support.
Discrete flows (Tran al., 2019 Hoogeboom et al. 2019] Lindt & Hoogeboom| (021)) introduce invertible transformations of random variables in discrete space without the need of computing the
log-determinant of Jacobian. Other works (Lippe & Gavves [021] [Hoogeboom et al. 202 Tan
et al. 2021) introduce various embedding methods for transforming discrete data into continuous space with disjoint support, which can be interpreted as a variational inference problem (Theis et al. 2015) with different dequantization distribution families. Several works (Kingma et al.) 2016
Ziegler& & Rush 2019; Zhang et al.] 2020)) also explore normalizing flows on discrete data under the Variational Autoencoders (Kingma & Welling, 2013) framework by enriching the prior. Compared to our diffusion- bassdd approach, these models suffer from strict invertible restrictions on network architecture, thus limiting their capacity.
Other generative models for discrete data Other generative models, such as Varational Autoen- coders (VAE) (Kingma & Welling, 2013). Generateive Adversarial Networks (GAN) (Goodfellow) ct al. 2014: Yu et 20.7 Che et al. 2017 Hjelm et al. 2017; Fedus et al., 2018) have also been applied to generate discrete data. These methods have not yet achieved the level of performance as autoregressive models on tasks such as discrete image generation or text generation, in terms of sample quality data likelihood. Potentially, the proposed analog bits can also be applied to these continuous generative models, by having the networks directly model and generate analog bits, but it not explored in this work.
Other related work The proposed Self-Conditioning technique shares some similarities with self- modulation in GANs (Chen et al. 2018a) (where the earlier latent state can directly modulate the later latent states) and SUNDAE (aavinov et al. 2021) (where an inference step is incorporated for denoising).
5 CONCLUSION
We introduce a simple and generic technique that enables continuous state diffusion models to generate discrete data. The main idea is to encode discrete or categorical data into bits and then model these bits as real numbers that we call analog bits. We also propose two simple techniques, namely seff-Conditioning (i.e., condition the diffusion models directly on their previously generated samples) and Asymmetric Time Intervals, that lead improved sample quality. We demonstrate that our approach leads to state-of-the-an results in discrete categorical image generation, beating the best autoregressive model. In an condiiccn text generation task on MS-COCO dataset, we also achieve competitive results compared to autoregressive models. One limitation of our approach, similar to other existing diffusion models, is that they still require significant number of inference steps for generating good (image) samples. However, we expect that future improvements from diffusion models for continuous data can also transfer to discrete data using analog bits.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We would like to thank Priyank Jaini, Kevin Swersky for providing helpful feedback to our draft. Our implementation is partially based on the Pix2Seq codebase, and we thank Lala Li, Saurabh Saxena, for their contributions to the Pix2Seq codebase.
REFERENCES
Jacob Austin. Daniel Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models discrete state-spaces. Advances in Neural Information Processing Systems, 34:1 34:17981- 17993, 2021. (Cited on 2
Andrew Brock, Jeff Donahuc, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis, arXiv preprint arXiv: 1809. 1096, 2018. (Cited on 4
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subhiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advonces neural information processing systems, 13:1877-1901, 2020. (Cited on
Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. arXiv preprint arXiv:2205, 14987, 2022. (Cited
on 5
Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. 2017. (Cited on(9)
augmented discrete generative adversarial networks. arXiv preprint arXiv:100.007983,
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pp. 1691-1703. PMLR, 2020a.
(Cited on 8
Ting Chen, Mario Lucie, Neil Houlsby, and Sylvain Gelly, On self modulation for generative adversarial networks. arXiv preprint arXiv:1810.01365 2018a. (Cited on
Ting Chen, Martin Renqiang Min, and Yizhou Sun. Learning k-way d-dimensional discrete codes for compact embedding representations. International Conference on Machine Learning, pp. 854 863, PMLR, 2018b. (Cited on [4]
Ting Chen, Lala Li, and Yizhou Sun. Differentiable product quantization for end-to-end embedding compression. In International Conference on Machine Learning. pp. 1617-1626. PMLR, 2020b. (Cited on 14)
Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2109.1088: 2021. (Cited on 5
Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David J Fleet, and Geoffrey Hinton. A unified sequence interface for vision tasks. arXiv preprint arXiv:2206. 07669, 2022. (Cited on 5
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904 10509, 2019. (Cited on
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham. Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint 2rXiv:2204.02311 2022. (Cited on [
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 225. Icce, 2009. (Cited on
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021. (Cited on g
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In International Conference on Learning Representations, 2017. (Cited on
William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: better text generation via filling in the_ arXiv preprint 180100000 2018. (Cited
10

---

Published as a conference paper at ICLR 2023
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, Advances neural information processing systems, 27, 2014. 4 (Cited on on
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. (Cited
on 5
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017. (Cited on 4
Devon Hjelm, Athul Paul Jacob, Tong Che, Adam Trischler, Kyunghyun Cho, and Yoshua Bengio. Boundary- seeking generative adversarial networks. arXiv preprint 172.:1700..08311, 2017. (Cited on
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop Deep Generative Models and Downstream Applications, 2021. (Cited on
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural
Information Processing Systems, 33:6840-688 6851, 2020. (Cited on 1
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans, Cascaded diffusion models for high fidelity image generation. Mach. Learn. Res., 23:47-1, 2022. (Cited on 3
Emiel Hoogeboom, Jomn Peters, Rianne Van Den Berg, and Max Welling, Integer discrete flows and lossless compression. Advances in Neural Information Processing Systems, 32, 2019. (Cited on !
Emiel Hoogeboom. Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multino- mial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34: 12454-12465, 2021. (Cited on
Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford, and Ilya Sutskever.
Distribution augmentation for generative modeling. In International Conference on Machine Learning, pp. 5006-5019. PMLR, 2020. (Cited 22 g
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6680, 2014. (Cited on
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
(Cited o
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018. (Cited on g
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016. (Cited on 9
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009. (Cited on
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv: 1808. 06226, 2018, (Cited on 4
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang. and Tatsunori Hashimoto. Diffusion-Im improves controllable text generation. arXiv preprint arXiv:2205. 14217, 2022. (Cited on
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014. (Cited on
Alexandra Lindt and Emiel Hoogeboom. Discrete denoising flows. arXiv preprint arXiv:2107.1162. 2021.
(Cited on 9
Phillip Lippe and Efstratios Gavves. Categorical normalizing flows via continuous transformations. In Interna- tional Conference Learning Representations, 2021. (Cited on
Cheng Lu. Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver; A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.0092, 2022. (Cited on
16
11

---

Published as a conference paper at ICLR 2023
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, arXiv preprint arXiv:2112.1077 2021. (Cited on
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In
International Conference on Machine Learning, pp. 8162-8171. PMI.R, 2021. (Cited on
4 S 6
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pp. 1055-40065 PMLR, 2018. (Cited
on
1 8
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2204006/25, 2022. (Cited
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530- 1538. PMLR, 2015. (Cited on 9
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image compuring and computer-assisted intervention, pp. 234-241. Springer, 2015. (Cited on
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68, 2021. (Cited on
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint xxxi:2104.076.66 2021. (Cited
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11457 2022. (Cited on |
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv1701.05517, 2017. (Cited on
Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. Step-unrolled denoising autoencoders for text generation. arXiv preprint arXiv:2112.06749, 2021. (Cited on
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint weXiv:150..07900 2015. (Cited on 14]
Jascha Sohl- bick-siic,,,,n, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015. (Cited on 2
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020. (Cited on 2
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019. (Cited on [
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33: 33:12438-12248, 2020. (Cited on
Yang Song, Jascha Sohl-Dickstein. Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. (Cited on |2
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 2014. (Cited on 5
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014. (Cited on [8
Shawn Tan, Chin-Wei Huang, Alessandro Sordoni, and Aaron Courville. Learning to dequantise with truncated flows. In International Conference on Learning Representations, 2021. (Cited on
12

---

Published as a conference paper at ICLR 2023
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note the evaluation of generative models. arXiv preprint arXiv:1511.01844 2015. (Cited on
Dustin Tran, Keyon Vafa, Kumar Agrawal, Laurent Dinh, and Ben Poole. Discrete flows: Invertible generative models of discrete data, Advances in Neural Information Processing Systems, 32, 2019. (Cited on 9
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 2016. (Cited on
g
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser. and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
(Cited on I
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017. (Cited |
Zijun Zhang, Ruixiang Zhang, Zongpeng Li, Yoshua Bengio, and Liam Paull. Perceptual generative autocncoders. In International Conference on Machine Learning, pp. 11298 11306. PMLR, 2020. (Cited on (
Zachary Ziegler and Alexander Rush. Latent normalizing flows for discrete sequences. International Conference on Muchine Learning, pp. 7673-7682. PMLR, 2019. (Cited on 9
13

---

Published as a conference paper at ICLR 2023
A
MORE DETAILS OF ALGORITHM AND AND 2
Algorithm
and provide more detailed implementations of functions in Algorithm and 2 3
Algorithm 3 Binary encoding and decoding algorithms (in Tensorflow).
import consorflow f
def int2bit n-8):
Convert integers into the corresponding binary bits.
tf.bitwise, wtt.seeerriit___aae_e shif: (tf, expanee dima (x, -1), tf. (E.range
tf.math.mod(x,, return x
bit2int (x)
Convert binary bits into the corresponding integers.
tf.cas: (x, tf.int32)
x.shape [-1]
tf..mth.
tf. renge (n)),
return
Algorithm 4 estimation with DDIM / DDPM updating rules.
def gamma it,, ns=0
schadaling function based cosine function, retuEn numpy. ((tt + na) (1 + da)) * nampy -ppp / 2) xv2
def sdem_step (x_t, x_pred, t_now, t_next); Estimate at t_next with DDIM apdating rule.
Thow gamma (t_now)
gamma (t next)
prrred clip (x_reed, -scale, scale)
cps V----- (x_t Voov x_prod)
xenxtt Thext x_ed vI Thext ops
return x_next
ddem_step (x_t, x_pred, tonow, :_next):
Rstimate x at noxt with DDPX updating relo.
Thou gamma (t_new)
Onew gamma (t (t_now)
gamma (t_next)
dnow scrt (1 (nox)
normal (mean-0, std-l)
pred clipix pred, -scale, acale)
cps VI= * (x_t Vhov x_prod)
x_next your (x_t
VI-
1-am
eps) dnos
return x_next
B ALTERNATIVE BINARY ENCODING AND LOSS FUNCTIONS
B. ANALOG BITS BASED ON ONE-HOT ENCODING
An alternative binary encoding to the base-2 encoding of the discrete data used in the main paper, the one-hot encoding, where discrete variable is represented same as the vocabulary size K, with single slot being and the rest being O. The resulting one-hot vector can be similarly treated analog bits and modeled by continuous state diffusion models. To obtain discrete variables corresponding to the generated analog bits, use arg max operation over all candidate categories, instead the thresholding operation in base-2 analog bits. Note that the one-hot encoding requires K bits, which is less efficient compared to base-2 encoding that only requires [log, K") bits, especially for large K.
vector whose length is the
Athough, one reduce the vocabulary size K by using sub-tokenization (c.g., subword (Sennich al. 2015) ) learned discrete codes (Chen et al. 2018h 2020b
14

---

Published as a conference paper at ICLR 2023
B.2 SIGMOID CROSS ENTROPY LOss
As we use & loss by default for its simplicity and compatibility with continuous diffusion models. The proposed Bit Diffusion models work with other loss functions too. Since the analog bits bimodal, we can use the following sigmoid cross entropy loss:
L = log o (#o ()),
where we assume 2o E {-1,1]", and a is sigmoid function. During the sampling process, we use 2x (f ()) the output of denoising network.
B.3 SOFTMAX CROSS ENTROPY LOSS
For one-hot analog bits, one could also add softmax activation function for the output of denosing network f. and use the following softmax cross entropy loss:
Le..w.,t
Zg log softmax (z;,
where we assume zo {0, 13" which the one-hot representation.
B.4 PRELIMINARY EXPERIMENTS
Table presents FIDs of Bit Diffusion models with different types of analog bits and loss functions on unconditional CIFAR-10 10. Note that it possible some of these results can be improved by more tuning of hyper-parameters tweaks of the network, but we do not focus them in this work.
Table 6: FIDs of Bit Diffusion models with different types of analog bits and loss functions on unconditional CIFAR-10.
8 loss
Logistic loss
Softmax loss
ONE HOT
16.32
26.82
29.49
UINT8
3.48
3.53
CODE
3.86
3.71
UINT8 (RAND)
6.93
49.29
C ON BINARY ENCODING OF PIXELS: UINT8, GRAY CODE, UINT8 (RAND)
In the main paper, we describe three different types of binary encodings of pixels. Here we provide additional detail on how we generate UINT8 (RAND): first apply a random permutation 256 sub-pixel values, and then assign the binary binary bits of permuted integers to the non-permuted integers. For example, assume is mapped 228 after the permutation, the analog bits of 0 would be the binary bits of 228. The random permutation is generated by numpy. I random. seed (42): numpy. random. shuffle (numpy aamppy.arann arange (256)).
Abs alferencce in Intensity
afffrrnneeen Intensityy
Abs aifference in intersity
200
200
350
350
-50
100
300
100
30
50
Hamming
Hamming distance
(a) UINT8.
(b) GRAY CODE.
(c) UINT8 (RAND).
Figure 8: Correlation between (absolute) difference in subpixel intensity and the Hamming distance of the corresponding binary bits.
Figure show the correlation between Hamming distance of three different binary encodings we use and the (absolute) difference of sub-pixel intensity. This is done by taking every pair of subpixel
15

---

Published as a conference paper at ICLR 2023
integers (in [0. 256)), compute their absolute difference, well the Hamming distance between the corresponding binary bits. We find that both UINT8 and GRAY CODE exhibit partial correlation between the two quantities (with different correlation patterns), meaning that these codes partially contain the order information about the original sub-pixel intensity. However, UINT8 (RAND) exhibits no correlation between hamming distance and sub-pixel intensity, indicating the order information is fully removed, thus can be considered as categorical data.
D A TOY EXAMPLE ON CONTINUOUS MODELING OF DISCRETE VARIABLES
An intuitive toy example of how continuous generative model can generate binary data is given in Figure where mapping from prior distribution at æ to data distribution zy is shown. With deterministic sampler (such as DDIM), it is straight-forward how they can represent any Bernoulli distribution by dividing the prior into regions of probability densities corresponding to the Bernoulii distribution. For stochastic samplers, they can achieve similar effect but the mapping from noise data is stochastic. For arbitrary discrete variable, represented m-dimensional Bernoulli distribution, the mapping from continuous noise distribution to the target Bernoulli distribution also exists but it is more complicated (and difficult visualize).
FT
A
(a) Mapping from (V, 1) to Bernoulli distribution with P(zo = 1) = 0.5.
(b) Mapping from (0, 1) Bernoulli distribution with P( 1) = 0.7.
Figure 9: A toy example continuous modeling of discrete variables.
E ON OTHER SAMPLERS FOR CONTINUOUS DIFFUSION MODELS
As our models are based continuous diffusion models, in theory models are able to incorporate faster samplers. To this end, we conduct preliminary exploration of using DPM-Solver (Lu et al.) 2022) for sampling some of our models.
We find that DPM-Solver provides boost to diffusion models based on analog bits, similar to what is able to do for continuous data. This shows potential of our model enjoying faster continuous sampler while other baselines (e.g., D3PM) may not be able to do due to their use of discrete states. Table 7 below shows the FID scores of bit diffusion models ImageNet-644x 64x64 under different able to provide a significant reduction in
binary encoding schemes. We find that the function evaluations for bit diffusion on discrete/categorical data (with 30 NFES gets comparable FIDs as 100 NFEs of DDIM), similar to that in continuous diffusion models.
Furthermore, also find that self-conditioning continues provide boost with DPM-solver. For example, the table g shows FID scores of diffusson models on ImageNet 64x64 (continuous rgb values). And we find that the self-conditioning consistently improves the performance DPM-Solver with fixed number of function evaluations.
F EXTRA RANDOM SAMPLES ON CIFAR-10 AND IMAGENET 64x64
Figure
shows random samples (non cherry -icked) from unconditional diffusion models on CIFAR- 10 with continuous pixels and analog bits.
10
Figure shows random samples (non therry-picked from class-conditional diffusion models on IMAGENET 64 with continuous pixels and analog bits.
16

---

Published as a conference paper at ICLR 2023
Table Comparison of Continuous Diffusion Samplers. FIDs on ImageNet 64 x 64 shown below.
Samplers
UINTS
Gray Code
UINTS (RAND)
DDIM @ 1000 NFE
5,51
5.14
58.45
DDPM 1000 NFE
7.71
6,91
64.08
DDIM @ 400 NFE
5.00
5.52
38.44
DDPM @ 400 NFE
4.84
5.37
40.91
DDIM @ 100 NFE
8.80
11.31
8.76
DDPM 100 NFE
13.04
12.77
9.25
DPM-Solver @ 30 NEE
7.85
9.64
10.39
DPM-Solver @ NFF
6.46
7.61
10.96
Table 8: The effect of Self-Conditioning for sampling with DPM-Solver. FIDs on ImageNet 64 x44 shown below.
Model
DPM-Solver @ 20 NFE
DPM-Solver @ 30 NFE
prediction, w/o self-conditionin prediction, self conditioning
6.10
5.58
4.24
4.15
zo prediction, w/o self-conditioning
12.13
11.05
2p prediction, ceff-conditioning
6.94
6.43
(a) DDPM on continuous pixels (FID=3.14)
(b) Bit Diffusion on UINT8 (FID=3.71)
(c) Bit Diffusion on GRAY CODE (FID=3.88)
(d) Bit Diff. on UINT8 (RAND) (FID=6.93)
Figure 10: Random samples from unconditional models trained on CIFAR- 10. (a) is for continuous image generation, (b), (c), and (d) are for discrete/categoriall image generation.
17

---

Published as a conference paper at ICLR 2023
DDPM continuous pixels (FID=3.43)
(b) Bit Diffusion on UINT8. (FID=4.84)
(c) Bit Diffusion on GRAY CODE (FID=5.14)
(d) Bit Diffusion on UINTS (RANd) (FID=8.76)
Figure 11: Random samples from class-conditional models trained on IMAGENET 64 (a) is for continuous image generation, (b), (c), and (d) are for discrete/categorial image generation.
18

---

Published as a conference paper at ICLR 2023
G ON SAMPLING STRATEGIES WITH SELF-CONDITGNNIG
G.1 METHOD
In this section, we present extensions the default sampling strategy with Self-Conditionin in Algorithm 2 The default sampling strategy utilizes data estimate from the previous step as the conditional input to the denoising network for producing data estimate at the current step. While this is both simple and effective, observe that, for UINT8 (RAND) encoding of pixels, the number of sampling steps increases (with both DDIM or DDPM samplers), the generated samples tend to be over-smoothed. We propose the following two extensions of the default sampling strategy to mitigate the issues and provide improvements when using larger sampling steps.
Self-Cone Conditioning based Momentum Estimate The first extension the default sampling strategy is to adopt an exponential moving average over the previous data estimate to provide a more reliable conditioning imput, similar to momentum optimizer. The detailed procedure is shown in algorithm[5] where the differences from the default sampling strategy are highlighted blue. Note that the default sampling strategy can also be considered special case of this generalized form in that the momentum is set to zero.
Saff-Conditioning based on Self-Guidanes One potential issue with the default sampling strategy is the slight discrepancy of the Self- colf-Conditioning signal during training and inference/sampling Specifically, during training, the Sol--Conditionin signal is the data estimate from the same time step, while, during sampling, it is from the past time step(s). Therefore, here we propose approach that also use the same step data estimate for coffccnndiiining, which comes the cost of extra forward pass over the denoising network at sampling time. Specifically, we conduct two forward passes of denoising network per sampling step, one with zero data estimate and the other with current data step estimate, and then we use weighted combination, similar to (Ho & Salimans] (002)), of both prediction to form the final prediction at the current step. The detailed procedure is given in algorithm 6 with differences to the default sampling strategy highlighted.
Algorithm 5 Sampling with Self-Conditioning based on Momentum Estimate.
def generate (stepa, td-0, mome::um-0. [mean=0, std=1) x_pred lekes_like (xLt) x_accu zeros_like(x_] (x_t)
x_t
for step in range (steps)
Get time for current and states.
step / steps t_next max (step
tonow
td) stepe, 01
Predict x_ (with self-coad).
x momentum x_accu x_pred net (cat [ [x_t, x_acou], -11, t_now)
nomentum) x_pred
Estimate at t_next. x_t aidi__or_ddpm_stspixee
x_pred, t_now, t_rext)
Binazy decoding: analog bits to discrete data. iint bi=2int (X_ored 0)
return x_int
19

---

Published as a conference paper at ICLR 2023
Algorithm 6 Sampling with Self-Conditionig based on Self Belf-nuid
def generate (stepa, td-0, guide_x-3.0 x_t normal Imean=0, std=1) x_pred zeros_
for step parge
Ges time for curcent and next states.
step steps
t_next max
(step + td) steps, 01
Predict x_ wo/ self-cond
pred uncond net (est (Ix_t, zercs zerce like -1), t_ Predict
send-
x _.nred_seefoone = net (cat ([x_t, x_pree__ncond), -1), [_now] Apply feglf-guide dance.
X_pred quide_M x_pree_selfoond (1.0 quide_N) x_pred_
at nextt. adim_or__dpp_ster
x_t
pzed,, nonww t_eext)
Binary decoding: analog bits to discrete data.
x_int bi:2int (x_pred > 0)
return x_int
G.2 EXPERIMENTS
Table reports the best FID scores across various sampling strategies discussed here (as well as samplers, sampling steps, time difference in asymmetric time intervals).
Table 9: Best FIDs of Bit Diffusion models with different Self-Conditioning sampling strategies on conditional IMAGENET 64x64.
UINT8
GRAY CODE
UINT8 (RAND)
Default sampling (momentum= 0)
4.84
5.14
8.76
Momentum Estimate
4.85
5.14
8.51
Self- Saff-Guidance
5.15
5.65
7.87
Figure shows FIDs conditional IMAGENET 64x64 with UINT8 encoding, using Momentum Estimate with different sampling steps. We find that the momentum on the data estimate only helpful when sampling steps are larger.
Figure 13 shows FIDs on conditional IMAGENET 64x64 with UINT8 encoding, using Self-Guidance with different sampling steps. We find that a guidance weight between 3.0 and 5.0 is generally preferable and robust to other hyper-parameter (such sampler choice, sampling steps, and time difference).
G.3
SAMPLES
Figure 14 and 15 provide generated samples from different sampling strategies with 100 and 1000 DDIM sampling steps, respectively.
20

---

Published as a conference paper at ICLR 2023
140
140
120
120
100
100
Score
80
60
T
40-
20
20-
0.0
0.1
0.4
0.5
0.7
0.8
0.1
0.4
0.5
0.7
0.8
(a) DDIM, numher of sampling steps < 500.
(b) DDIM, number of sampling steps > 500.
140
140
120
120
100
100
FD Spore
0.0
0.1
0.2
0.3
0.4
0.5
0.7
0.8
0.9
0.0 0.1 0.2
0.3
0.4 0.5
0.6
0.7 0.8 0.9
Momentum
Momentum
(c) DDPM, number of sampling steps 500.
(d) DDPM, number of sampling steps 500.
Figure 12: FID on conditional IMAGENET 64 with UINT8 (RAND) encoding using self- condition sampling based momentum estimate. The statisties of FID scores in each group aggregated over the number of sampling steps in (100, 200, 400, 600, 800, 1000), time difference in (0.0, 0.2, 0.4, 0.6, 0.83.
70
T
FD Score
0
6.0 1.0 Salf-guidance Weight
9.0
2.0
4.0
9.0 10.0
Self-guidance Weight
(a) DDIM, number of sampling steps < 500.
(b) DDIM, number of sampling steps > 500.
100
100
FID Score
Scone
40
40
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
10.0
1.0 2.0 3.0
4.0
5.0
6.0
7.0
8.0 9.0 10.0
Self-guidance Weight
Self-guidance Weight
(c) DDPM, number of sampling steps 500.
(d) DDPM, number of sampling steps 500.
Figure 13: FID on conditional IMAGENET 64 with UINT8 (RAND) encoding using self-condition sampling based on self-guidance. The statistics of FID scores in each group are aggregated over the number of sampling steps in (100, 200, 400, 600, 800, 1000), time difference in (0.0, 0.1}.
21

---

Published as a conference paper at ICLR 2023
(a) W/o Self-Condition cofdioondiiin,nng FID=90.77
(b) W/ previous estimate (momentum=O), FID=12.3.
(c) W/ previous estimate (momentum=0.5) FID=35.5. (d) W/ prev. estimate (momentum=0.9) FID=132.6.
(e) W/ same-step step estimate (w 3.0), FID=15.6.
(f) W/ same-step estimate (w 5.0), FID=9.3.
Figure 14: Random samples of Bit Diffusion with UINT8 (RAND) on categorical IMAGENET 64x64 using various Self-Conditioning sampling strategies. Different plots share the same set of æT. Sampling with 100 steps of DDIM without asympetric time intervals.

---

Published as a conference paper at ICLR 2023
(a) W/o Scff-Conditioning, FID=73.60.
(b) W/ previous estimate (momentum=0), FID=58.6 6.
(c) W/ previous estimate (momentum=0.5) FID=46.0.
(d) W/ previous estimate (momentum=0.9), FID= 11D-111.. 11.7.
(e) W/ same-stee step estimate (w 3.0), FID=11.8.
(f) W/ same-step estimate (w 5.0), FID=9.6.
Figure 15: Random samples of Bit Diffusion with UINT8 (RAND) on categorical IMAGENET 64x64 using various Self-Conditioning sampling strategies. Different plots share the same set of æT. Sampling with 1000 steps of DDIM without asynymetric time intervals.


Reviews:
[{"review_id": "yZ21-dZ458Z", "reviewer": "Reviewer_h3v1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the simple yet effective idea of using continuous diffusion models to generate discrete data by representing them as binary data and modeling these binary data as real numbers. The authors also propose two techniques, namely Self-Conditioning and Asymmetric Time Intervals, which lead to improvement in sample quality.", "summary_of_the_review": "Overall, the main idea of this paper is novel and simple. The experiments show the good performance of the analog bits idea, and this work paves the way for future works on generating discrete data using continuous diffusion models.", "strength_and_weaknesses": "Strengths:\n* The idea of treating discrete data as continuous bits, is simple yet effective.\n* The proposed self-conditioning and asymmetric time intervals techniques seem to improve the performance significantly and are also applicable to other continuous diffusion models.\n\nWeaknesses:\n* The authors should discuss we the experiments have been over small size images (only 64 by 64). If the computational cost is the main concern, it should be delineated if the expectation is that the same performance will be observed for higher resolutions or not.\n* Harder constraint or regularization to enforce the binary outputs has not been adopted based on the assumption that the model expressivity will take care of it. However, conducting experiments to show additional regularization won't benefit the performance will be valuable.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "maqQGj4gi_", "reviewer": "Reviewer_YYSB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The main idea of this paper is to convert data to bit, and then model these bits using continuous diffusion models. This paper proposes two additional techniques: the self-conditioning to improve the x0 prediction, and the asymmetric time to improve the sampling speed. There are experiments on both images and texts, which show the effectiveness of this method.", "summary_of_the_review": "The method is simple and looks effective under some cases, and meanwhile I have some concerns on the \"Asymmetric Time Intervals\" and the sample quality for categorical data. My recommadation is marginally above the acceptance threshold.", "strength_and_weaknesses": "Strength:\nThe author shows that modeling bits is a simple and effective method. This makes it possible to directly use continuous diffusion models to model discrete data such as texts. Although the self-conditioning technique takes some additional time, it looks very useful to improve the generation performance. This method works for both images and texts.\n\n\nWeakness:\n\n1. In Figure 6 (a), \"Asymmetric Time Intervals\" only improves the sample quality when the number of steps is small. When the number of steps is large, a \"time difference\" of zero performs the best. It needs to tune the \"time difference\" hyperparameter carefully given a specific number of steps. So I wonder would it be more convenient to directly use a fast sampling method, such as [1, 2], for a good sample quality with a small number steps?\n\n2. In Figure 6 (b), using a large number of steps for \"UNIT8 (RAND)\" leads to a worse performance. While the author empirically fixes this issue in Appendix, it would be better to provide some formal or intuitive explanation.\n\n3. Since the text is also categorical data, like the \"UNIT8 (RAND)\", would the text also meets the same problem that a large number of steps lead to a worse performance?\n\n[1] Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models\n\n[2] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NfshG4ZFtr", "reviewer": "Reviewer_ECCX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of this paper introduce a method to use diffusion models to generate discrete data. The crux of the method rests on the observation that discrete data can be represented with binary bits and binary bits can be uses as real numbers to train a continuous diffusion model; the output at the end of the diffusion process can be rounded to obtain binary output bits.", "summary_of_the_review": "Diffusion models have shown strong performance in continuous domains. However, there are many domains that are not continuous (like text) and adapting diffusion models to discrete settings will aid in exploring whether state-of-the-art results can be improved by using diffusion models for discrete domains. The method in this paper also converts discrete domain data to a \"continuous representation\" and therefore additional work on continuous diffusion models can easily be ported and used for the discrete setting. Furthermore, self-conditioning and asymmetric time intervals are useful general techniques that will be of interest to the broader diffusion model community. ", "strength_and_weaknesses": "Strengths:\n* The authors introduce two techniques - self-conditioning and asymmetric time intervals. Both of these seem useful for both continuous and discrete diffusion models. Other concurrent work has also shown that using self-conditioning improves results of diffusion models.\n* Figure 7 shows that the generated bits are concentrated on two modes without adding explicit constraints and this validates the authors' hypothesis that modeling binary bits is easy. \n* For image captioning, the AnalogBits achieves the same performance as auto-regressive models with just 10 diffusion steps!\n\nWeaknesses:\n* Discrete diffusion models are especially suitable for NLP tasks given the discrete nature of text. Despite, proposing a method for discrete data, the authors only presents results on one NLP task (image captioning). Additional experimental results on other NLP texts would help strengthen this paper.\n* For encoding NLP tokens as analog bits, each token is mapped to a random set of 15 analog bits. As a results, there is no semantic meaning in the new representation (similar tokens don't have similar bit representations). The authors show that using UINT8 is much better when compared to using random bits for discrete image generation. Exploring ways to represent semantic meaning in bit representations of tokens would be an interesting extension to the presented work.\n* Gray codes perform worse than UINT8 and no explanation is give as to why this is the case. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "42iPSNgm4pC", "reviewer": "Reviewer_j3eh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel idea of generating images in bits level. The proposed method is shown to be effective on selected dataset. The selected datasets are not simple and includes complex dataset such as COCO(in contrast, some of the works only report results on CIFAR and ImageNet, which are simpler than COCO). The two proposed tricks: 1. self-conditioning and 2 asymmetric time intervals are simple yet shown to be effective also. Overall, I like the paper.", "summary_of_the_review": "Overall, I like the idea behind the paper. Although the proposed methods is not verified in large scale, it shows some promising results on complex tasks already (COCO dataset).", "strength_and_weaknesses": "Strength:\n1. The paper is well written and easy to follow.\n2. The proposed methods are simple yet effective and supported by the experiments.\n3. The idea behind the paper is novel: modeling generative tasks in bits level with diffusion models.\n4. The proposed algorithm is well presented and the is reproducible.\n\nWeakness:\n1. The proposed algorithm is not verified on larger dataset such as LAION-5B wit larger models. It is hard to judge if the proposed method still work when scaled to large dataset. \n2. The two proposed tricks are working well on DDPM sampler. However, recently, a variety of efficient samplers are proposed such as DPM-Solver and PLMS sampler. Does the proposed tricks work on those efficient samplers?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster