Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors of this paper introduce a method to use diffusion models to generate discrete data. The crux of the method rests on the observation that discrete data can be represented with binary bits and binary bits can be uses as real numbers to train a continuous diffusion model; the output at the end of the diffusion process can be rounded to obtain binary output bits.
strength_and_weaknesses: Strengths:
* The authors introduce two techniques - self-conditioning and asymmetric time intervals. Both of these seem useful for both continuous and discrete diffusion models. Other concurrent work has also shown that using self-conditioning improves results of diffusion models.
* Figure 7 shows that the generated bits are concentrated on two modes without adding explicit constraints and this validates the authors' hypothesis that modeling binary bits is easy. 
* For image captioning, the AnalogBits achieves the same performance as auto-regressive models with just 10 diffusion steps!

Weaknesses:
* Discrete diffusion models are especially suitable for NLP tasks given the discrete nature of text. Despite, proposing a method for discrete data, the authors only presents results on one NLP task (image captioning). Additional experimental results on other NLP texts would help strengthen this paper.
* For encoding NLP tokens as analog bits, each token is mapped to a random set of 15 analog bits. As a results, there is no semantic meaning in the new representation (similar tokens don't have similar bit representations). The authors show that using UINT8 is much better when compared to using random bits for discrete image generation. Exploring ways to represent semantic meaning in bit representations of tokens would be an interesting extension to the presented work.
* Gray codes perform worse than UINT8 and no explanation is give as to why this is the case. 
clarity,_quality,_novelty_and_reproducibility: The method is clearly explained and the necessary ablations have been conducted. 
summary_of_the_review: Diffusion models have shown strong performance in continuous domains. However, there are many domains that are not continuous (like text) and adapting diffusion models to discrete settings will aid in exploring whether state-of-the-art results can be improved by using diffusion models for discrete domains. The method in this paper also converts discrete domain data to a "continuous representation" and therefore additional work on continuous diffusion models can easily be ported and used for the discrete setting. Furthermore, self-conditioning and asymmetric time intervals are useful general techniques that will be of interest to the broader diffusion model community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes the simple yet effective idea of using continuous diffusion models to generate discrete data by representing them as binary data and modeling these binary data as real numbers. The authors also propose two techniques, namely Self-Conditioning and Asymmetric Time Intervals, which lead to improvement in sample quality.
strength_and_weaknesses: Strengths:
* The idea of treating discrete data as continuous bits, is simple yet effective.
* The proposed self-conditioning and asymmetric time intervals techniques seem to improve the performance significantly and are also applicable to other continuous diffusion models.

Weaknesses:
* The authors should discuss we the experiments have been over small size images (only 64 by 64). If the computational cost is the main concern, it should be delineated if the expectation is that the same performance will be observed for higher resolutions or not.
* Harder constraint or regularization to enforce the binary outputs has not been adopted based on the assumption that the model expressivity will take care of it. However, conducting experiments to show additional regularization won't benefit the performance will be valuable.
clarity,_quality,_novelty_and_reproducibility: * The paper is well-written with clear exposition of ideas.
* Enough details are provided to enable reproducibility.
summary_of_the_review: Overall, the main idea of this paper is novel and simple. The experiments show the good performance of the analog bits idea, and this work paves the way for future works on generating discrete data using continuous diffusion models.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel idea of generating images in bits level. The proposed method is shown to be effective on selected dataset. The selected datasets are not simple and includes complex dataset such as COCO(in contrast, some of the works only report results on CIFAR and ImageNet, which are simpler than COCO). The two proposed tricks: 1. self-conditioning and 2 asymmetric time intervals are simple yet shown to be effective also. Overall, I like the paper.
strength_and_weaknesses: Strength:
1. The paper is well written and easy to follow.
2. The proposed methods are simple yet effective and supported by the experiments.
3. The idea behind the paper is novel: modeling generative tasks in bits level with diffusion models.
4. The proposed algorithm is well presented and the is reproducible.

Weakness:
1. The proposed algorithm is not verified on larger dataset such as LAION-5B wit larger models. It is hard to judge if the proposed method still work when scaled to large dataset. 
2. The two proposed tricks are working well on DDPM sampler. However, recently, a variety of efficient samplers are proposed such as DPM-Solver and PLMS sampler. Does the proposed tricks work on those efficient samplers?
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. The paper is presenting an idea that the generation tasks can be modeled in bits level with diffusion models. However, the proposed method is not verified on large scale making it unclear on the impacts of those tricks in large scale.
summary_of_the_review: Overall, I like the idea behind the paper. Although the proposed methods is not verified in large scale, it shows some promising results on complex tasks already (COCO dataset).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The main idea of this paper is to convert data to bit, and then model these bits using continuous diffusion models. This paper proposes two additional techniques: the self-conditioning to improve the x0 prediction, and the asymmetric time to improve the sampling speed. There are experiments on both images and texts, which show the effectiveness of this method.
strength_and_weaknesses: Strength:
The author shows that modeling bits is a simple and effective method. This makes it possible to directly use continuous diffusion models to model discrete data such as texts. Although the self-conditioning technique takes some additional time, it looks very useful to improve the generation performance. This method works for both images and texts.


Weakness:

1. In Figure 6 (a), "Asymmetric Time Intervals" only improves the sample quality when the number of steps is small. When the number of steps is large, a "time difference" of zero performs the best. It needs to tune the "time difference" hyperparameter carefully given a specific number of steps. So I wonder would it be more convenient to directly use a fast sampling method, such as [1, 2], for a good sample quality with a small number steps?

2. In Figure 6 (b), using a large number of steps for "UNIT8 (RAND)" leads to a worse performance. While the author empirically fixes this issue in Appendix, it would be better to provide some formal or intuitive explanation.

3. Since the text is also categorical data, like the "UNIT8 (RAND)", would the text also meets the same problem that a large number of steps lead to a worse performance?

[1] Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

[2] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
clarity,_quality,_novelty_and_reproducibility: The paper is very clear and readable. The work is original.
summary_of_the_review: The method is simple and looks effective under some cases, and meanwhile I have some concerns on the "Asymmetric Time Intervals" and the sample quality for categorical data. My recommadation is marginally above the acceptance threshold.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a simple but effective method to extend diffusion models to generate discrete data. The basic idea is to encode a discrete data with a vector of binary variables, which are linked to latent continuous variables that are modeled with continuous diffusion models. Two techniques, including Self-Conditioning and Asymmetric Time Intervals, have also been introduced to improve the underlying diffusion models. While there are some concerns on the level of novelty and the limitation on the proposed approach on naturally-occurred discrete data, all reviewers recognize the contributions in providing a simple and effective solution of generating discrete data in two different types of tasks and are in favor of accepting the paper. 
justification_for_why_not_higher_score: There are some concerns on the level of novelty and the limitation on the proposed approach on naturally-occurred discrete data.
justification_for_why_not_lower_score: The empirical performance on the chosen tasks is convincing. The paper provides a valuable addition to the application of diffusion models.
Decision: Accept: poster