Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: There is a recent emergence of interests in adapting diffusion models to handle discrete data and this work adds to that literature. To understand the main difference from prior work, it is important to note that there are two (almost) equivalent frameworks for learning diffusion models: the variational lower bound framework and the denoising score matching framework. Prior work on discrete diffusion models have mostly focused on the variational lower bound framework because score functions are not defined in discrete domains. This paper replaces the continuous score with a discrete analogy which were studied in previous work on ratio matching/generalized score matching. The authors then design a transformer-based architecture to model this "discrete score" (which is essentially all complete conditionals at time t). Experiments show that the proposed discrete score matching objective is effective in fitting discrete energy-based models and diffusion models, with comparable or worse results on image data but better results on music data. 
strength_and_weaknesses: ## Strengths

* This paper is well-motivated. A well-known fact in the diffusion model community is that score-based objectives often lead to better likelihoods than the ELBO. It is natural to ask if this holds for discrete data. However, the score-based interpretation was overlooked in the prior literation of discrete diffusion models. They mostly mentioned that scores are undefined in this case but ignored the fact that Hyvärinen had written about discrete generalization of score matching (called ratio matching) in his own work. 

* The proposed learning objectives are technically sound. As far as I checked, the derivations are correct. Modeling the complete conditionals also seems a very natural choice to me. 

## Weaknesses

* The paper claims they extend score matching to general categorical data in several places. However, this is not true given the prior work on ratio matching and generalized score matching. The discrete analogy of score has also appeared in these works. From this perspective, the technical novelty seems to be quite limited. 

* Although the motivation suggests that using score-based loss function should work better than ELBOs, the results on image data indicates the opposite. On simple datasets like CIFAR10, the proposed method was outperformed by prior discrete diffusion models, including D3PM, Campbell et al. 

* One reason the paper uses for explaining the superiority of continuous diffusion models over discrete ones is that score matching for discrete variables does not benefit from prior knowledge about ordinal discrete data. However, I believe this is not the case for the "discrete score" definition in (11), which leverages the "sparse neighborhood" knowledge by only looking at states that differ by +-1. 

* Section 4 is confusing. I do not get the reasoning "Such a jump rate depends on both the time t and ... Hence, it is inefficient to simulate the reverse time process X_t exactly". 
clarity,_quality,_novelty_and_reproducibility: This could be solid paper if the weaknesses are sufficiently addressed. Clarity is generally good except Section 4 (see above comments). The work is original in applying discrete variants of score matching to diffusion models, although techniques mostly follow from prior work on ratio matching and the discrete diffusion model formulation by Campbell et al. 

Minor:

* eq. (11), the first \approx should be identity, and the second identity should be \approx
summary_of_the_review: The work investigates an under-explored direction for developing discrete diffusion models. However, the paper is not ready yet due to lack of necessary theoretical developments to address the major problems and insufficient empirical evidence for supporting their claims.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper presents a diffusion-type generative model for categorical data. For this, the authors derive a variational learning scheme to approximate the discrete state space analog of the score of the marginal likelihood. The authors present a forward noise and backward denoising model based on a continuous-time Markov chain model. For learning the authors present three different neural-network-based parametrizations for the variational approximation. The algorithm is extensively evaluated for synthetic and real-world data and shows very good results.
strength_and_weaknesses: Overall, I do not have much to say about this paper, besides that, I really really enjoyed reading it. It is written in an easy-to-follow way, even though continuous-time stochastic process modeling usually requires immense technical machinery. The approximate inference algorithm is derived in a principled way and the empirical results of the paper are very promising.

Some sugestions I have are:
- Including the result of equation (47) in the main text, would have made it easier for me to understand how the learning of the conditionals is related to the ratio in equation (10)
- In Sec 4.1. the transition matrix is given as an aside as $q_{t \mid s}(\cdot \mid \cdot)=\int_s^t \exp(Q_{\tau})\, \mathrm d \tau$. This is not correct as it is the solution to the linear ODE given in equation (7), for which the solution is given by the state-transition function, for which there is generally no decomposition as a matrix exponential, see, e.g., "Peano–Baker series", or, "ordered exponential".
- The results for reversing a CTMC are not new and well known in the context of continuous-time filtering and smoothing. The authors should consider citing the relevant work, see, e.g., [1-3], and the references therein.

[1] Anderson, Brian DO, and Ian B. Rhodes. "Smoothing algorithms for nonlinear finite-dimensional systems." Stochastics: An International Journal of Probability and Stochastic Processes 9.1-2 (1983): 139-165.
[2] Elliott, R. "Reverse-time Markov processes (Corresp.)." IEEE transactions on information theory 32.2 (1986): 290-292.
[3] Van Handel, Ramon. Filtering, stability, and robustness. Diss. California Institute of Technology, 2007.

clarity,_quality,_novelty_and_reproducibility: I had no trouble understanding the paper and I think it is very nicely written. The approach is novel to me and it should be easy to reproduce the results.
summary_of_the_review: The paper presents a new algorithm for generative modeling with categorical data. It is very nicely written and it shows very competitive results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The submission considers score-based generative models with SDEs, but for discrete variables. The main challenge here is that the existing score-based methods do not apply to discrete variables since the log likelihood is not well defined. To deal with this issue, the authors use continuous-time discrete-state Markov process to model the forward process, with the according backward process also being a continuous-time discrete-state Markov process. Under this setting, a score-based estimation method is proposed for the diffusion model, with a sampling scheme for the reverse process. 

The performance of the proposed method is thoroughly studied empirically, demonstrating an improved performance. 
strength_and_weaknesses: Strength

1. The submission is well written. The notations are well defined. The used concepts are background details are clearly provided in the submission, making the paper self-contained. 

2. The empirical analysis is thorough and clear. The protocol, competing methods, expected results and the achieved results are clearly provided.  

3. The methodology is novel. While the whole framework follows the existing framework of score-based generative models with SDEs, the proposed score function for discrete state variables is an extension to existing works. 

Weakness

1. I am confused by the stochastic jump process. To the best of my knowledge, this terminology is more often used for jump processes like Poisson jump process. Specifically, it emphasizes more the discreteness of the path, instead of the discreteness of the variables: many stochastic jump processes have continuous-time variables. The model used in this submission is more a continuous-time discrete-state Markov process. Would it be possible for authors to clarify the rigorous definition of this terminology? 

2. I am wondering whether we can "naively" use a score-based generative models with SDEs for discrete variables, by simply doing some discretization when conducting sampling? This might be a bad idea when the variable has only two categories, but what if it has more than 200 categories like the image example in the experiments? What is the problem of this simple solution? Answering this question can further motivate the submission. 

clarity,_quality,_novelty_and_reproducibility: Clarity

The submission is clearly written and easy to follow. 

Novelty

The submission is novel by extending an existing framework. 

Reproducibility

Implementation details are provided in the submission. 
summary_of_the_review: The submission is well written with lots of technical details. A new method is proposed with empirical results supporting the performance improvement. To the best of my knowledge, the submission extends the existing score-based method to discrete variables, which is a contribution to this area.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work examines the problem of continuous-time discrete diffusion models, and the paper mainly focuses on 1) extending the existing discrete-time discrete diffusion models to the continuous-time via stochastic jump processes, 2) extending the score function to generic categorical variables and deriving a score-matching based learning for the model, 3) different sampling methods for the reverse process. Experimental validations on synthetic data, image and music generation have shown promising results comparing to existing discrete-time and continuous-time discrete diffusion models.
strength_and_weaknesses: *Strengths*

I think this work studies an interesting problem and it is nice to see the authors take the effort to extend the score functions to the categorical data and manage to leverage it to derive a score-matching based objective for estimating the continuous-time diffusion models. The main idea of using score-based objectives instead of ELBO surrogates is well-motivated, and the related works are properly referenced.

The authors have tried many different parametrizations and a new design for the conditional marginals, and it is also nice to see the proposed score-based learning naturally introduces a new analytic sampling algorithm. Experiments with toy 2-D datasets, CIFAR-10, and monophonic music dataset show the soundness of the proposed method, and the results are competitive with previous state-of-the-art methods. 


*Weaknesses*

The presentation is on the weaker side, and I think the paper can be significantly improved by having better clarity.
- The authors extend the ratio test proposed in Hyvarinen (2007) to the categorical variables but point to equation (14) as the "score-matching" objective. However, it does not resemble those used in Hyvarinen (2007) or Song et al. (2020). It would be better if the authors could discuss the relationship of the proposed objective to the classic ones.
- It would be nicer if the authors could discuss more regarding the similarities and differences from Campbell et al. (2022). For example, is there any subtle difference between the derivation of the continuous time modeling in Section 3.1 and those in Section 3.1 in Campbell et al. (2022)? Also, comparisons between the learning objectives would also make it easier to understand the benefits of the proposed method.

The parametrization of the conditional marginal distributions. Even though the authors have tried to design an efficient structure through different architecture choices including the hollow transformers, the current design is more computationally demanding comparing to existing methods and prevents it from directly modeling the CIFAR10 data with resolution as small as 32x32, not to mention other datasets with higher resolution. So that it is not easy to perform an apple-to-apple comparison with existing work.

The experimental validations are not very convincing. Two benefits of the proposed method are discussed in the introduction, including 1) the continuous-time discrete diffusion formulation is more flexible comparing to the discrete-time counterparts and may admit more effective estimation and generation; 2) the score-based learning yields superior estimation quality comparing to the ELBO surrogate used in Campbell et al. (2022) for the continuous-time discrete diffusion models. However, neither of them is convincingly validated in the experiment section. First, there is no formal comparison of the computations used during training and generation between the proposed model and the discrete-time ones. Second, the noise schedule adopted here is different from what used in Campbell at al. (2022). It has been shown that noise schedule can have noticeable impact on the final performance. Thus, this difference may question the validity of the conclusion that the better performance of the proposed model can be solely attributed to the score-based learning process. Third, it would be desirable to perform some ablation studies to validate the designs. For example, comparisons among all three parametrizations in the synthetic experiments and comparisons between the sampling methods discussed in Section 4.2 and 4.3.
clarity,_quality,_novelty_and_reproducibility: The extension of the ratio test for binary variables proposed in Hyvarinen (2007) to the categorical counterpart is straightforward but make it working with the proposed model is a non-trivial task and requires a lot of architectural designs. Unfortunately, the presentation of this paper makes it difficult to understand. Besides issues mentioned above, a few more:
- In Section 6.3, it states that "we use the simulation time $\tau=1e^{-3}$ for all methods". Is this $\tau$ the same as the $\epsilon$ used in Section 4,2 and 4.3? Or is this something not referenced previously? Which simulation time is used for the CIFAR10 data?
- In Appendix B.2, how does equation (34) equal to equation (35)? is this a typo?
- Do we have the similar SDE formulations as in equation (5,6) for the discrete data since we already have the score function? Is it just a simple replacement of the proposed score function?

For the reproducibility, there are certain experimental details presented in the main text and appendix, but I think it is still not quite easy to reproduce the results presented in Section 6. Hopefully the authors would release their code.
summary_of_the_review: I think the score-based continuous-time extension of the discrete diffusion models is interesting and the paper makes a novel contribution to it. However, I am leaning towards the negative side due to the weak presentation and not-so-convincing experiments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a generative diffusion model for categorical data. The paper claims to propose the first categorical score matching method and the first application of score matching to diffusion models.

Three reviewers vote for accepting the paper. The fourth reviewer criticises limited technical novelty and votes for reject. 
pro:
- The reviewers find the main idea of using score-based objectives instead of ELBO surrogates well-motivated and novel.

The following weaknesses were identified by the reviewers: 
-	One reviewer states that categorical score matching is not novel. (authors extend to categorical – not novel) but no reference is provided by the reviewer. The authors have refuted this criticism to the best of their knowledge. Unfortunately, the reviewer did not respond to the authors comments.
-	No experiment showing that continuous time formulation is better. This criticism could be addressed during the rebuttal phase, as the authors have provided additional experiments that show that the continuous time formulation indeed outperforms simpler discrete-time models.
- computationally demanding comparing to existing methods and prevents it from directly modeling the CIFAR10 data with resolution as small as 32x32. The inefficiency of the method was acknowledged by the authors.

After the rebuttal phase only the third one remains. Yet, the reviewers find that the methodical contribution outweighs the inefficiency of the method.

justification_for_why_not_higher_score: Overall the paper is a marginal accept. Two reviewers have increased their score during the rebuttal phase towards marginally above the acceptance threshold.
justification_for_why_not_lower_score: Only one review has voted towards rejection. As we did not find evidence for the negative criticism, there is no reason to reject the paper.
Decision: Accept: poster