Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a novel scheduled sampling algorithm to guide the training of diffusion models for markup-to-image generation. Unlike the traditional diffusion models, which sample the training data from the Markov chain quantified Gaussian distribution Q(yt|y0), the paper take the m earlier predictions into consideration and thus can address the exposure bias issues in the image generative models to some extent. 
strength_and_weaknesses: Strength:
+ The proposed algorithm seems to be effective. 

Weakness:
- The algorithm is not well explained. I believe that readers require a lot of background in diffusion model to understand the paper content. Some notions are proposed without any explanation or reference, which makes the paper even harder to follow. To name just a few, starting from Eqn. 1 (Sec.3), the paper talks about the Markov chain Q and its sampling. However, what is Q and how Q is related to diffusion model is not discussed. How to derive Eqn.1, and Eqn. 2 are also unknown. I would like to suggest the authors either briefly introduce the pipeline and refer readers to a more detailed deduction, or carefully explain the terms to avoid ambiguity. 
- The overall network pipeline is unclear. I think including a pipeline figure and an algorithm flow tables for both training and sampling procedures are necessary.
- Lacking essential experiments. The paper only show results from their own and don't include any comparison results with state-of-the-art (SOTA) text-to-image generative models. Considering the large amount of SOTAs, I believe that a comprehensive comparison result is needed. 
clarity,_quality,_novelty_and_reproducibility: I think there're still plenty of aspects that the authors need to address before getting accepted.
summary_of_the_review: In general, the paper proposed a novel sampling strategy for diffusion models to handle exposure issues. Even though the paper provided some reasonable results, lacking essential comparison experiments and poor paper organization make me hesitate to give it a high rank.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes the task of markup-to-image generation and adapts a procedure of scheduled sampling to improve diffusion model performance. Here, the objective is to synthesize images from latex, musical notation, or HTML snippets, or any markup language that defines an output image in a deterministic fashion. In this context, scheduled sampling is applied to incorporate intermediate outputs of the diffusion model during training. Empirically, this results in some ability for the model to self-correct when it begins to generate erroneous outputs. The method is evaluated on a few different proposed datasets (HTML, music notation, molecule diagrams, latex) with an array of ablation studies. Overall the results show convincing performance, and this research direction may be significant for advances in markup compilers that are robust to typos or which incorporate generative capabilities.
strength_and_weaknesses: The paper is mostly well-written and I found the description of the method to be clear. The paper makes a convincing case for the markup-to-image task, and I think this direction has potential for impact in the community. Moreover, the evaluation seems thorough, and while there is still room for improvment in the results, the manuscript and method are of high quality.

While I think the paper is fairly strong, I do have some reservations about the evaluation and clarity of the writing in a few sections as I detail below.
- The main quantitative results shown in Table 2 do not necessarily show the most convincing argument for the benefits of scheduled sampling. Some metrics are very close between the baselines, and scheduled sampling seems to worsen performance in other metrics (e.g., RASE). The paper also seems to be missing an analysis of how the performance changes with choosing m>1 (the number of scheduled sampling steps to use). Since I perceive the scheduled sampling to be the main technical contribution, I was expecting a more thorough analysis on this point. 
- I see only one qualitative result that shows the effect of including scheduled sampling (Fig. 2). I think including more such results could help make a more convincing argument for using this procedure. 
- I couldn't follow the perturbation analysis section. The paper repeatedly refers to a "gold" image without explaining what this is.

clarity,_quality,_novelty_and_reproducibility: Overall, I found the paper to be mostly clear and of high quality. While the key ideas of the main technical innovation (i.e., scheduled sampling) exist in some form in the literature, I believe the application to diffusion models is novel, as is the task of markup-to-image generation to some extent. I don't forsee any problems with reproducibility.

I have a few additional suggestions for improving the clarity of the manuscript:
- Fig. 2 is showing the steps of the reverse diffusion, but the convention is that the steps number the forward diffusion process, i.e., step 0 is the clean image and the image converges to Gaussian noise with increasing steps. I think the opposite numbering convention is used here, which conflicts with the notation elsewhere. 

- Apply consisting bolding in Table 2.

- Implementation details: uses M=1 but I think the parameter is given as "m" and not "M" previously in the manuscript.

- The paper should describe what is meant by a "gold" image, as this is not a standard term in my experience. Without this, I could not follow the perturbation analysis. I also didn't understand how the symbols are removed in the perturbation analysis. Are they removed from the input text conditioning?


summary_of_the_review: Overall the paper is well-written and contributes an interesting new technical idea for improving diffusion models as well as a new and potentially impactful task of markup-to-image generation. While there are areas where the clarity of the manuscript and the evaluation could be improved, I think the paper meets the bar for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper experiments on a deterministic generation (renderining) markup-to-image problem with diffusion model. They show that standard diffusion training process suffers exposure bias and using scheduled sampling algorithm is effective in fixing generation issues. 
strength_and_weaknesses: Strength
1. This paper explores a unique generation problem with diffusion model where the mapping between input and generation target is deterministic, this provides a new angle for analyzing and evaluating generative models. 
2.  Authors noticed that standard diffusion training process for the markup-to-image problem often generate duplicate or misplaced symbols similar to widely studied exposure bias, so they adapt scheduled sampling by training diffusion model with its own generations and showed improved evaluation metrics on four small-scaled datasets: Math, Tables, Music and Molecules. 


Weaknesses
1. Diffusion model has been proven effective in distribution learning, if the learning obejctive is a deterministic mapping, would regression based methods be more appropriate ? 
2. Comparisons are only made among authors' designs, are there previous works to compare with ? 
3. The authors already hinted varying performance on different domains, I would like to see more analysis on it. 
clarity,_quality,_novelty_and_reproducibility: Clarity: High.

Quality: In general the quality on introducing the problem, challenge, and solution is high. The experiments could probably be better designed: especially the evaluation metrics, and if the evaluation metrics are not to be trusted or revealing, more analysis on the patten of failure cases and improvments would be helpful. 

Novelty: The chosen methods (diffusion, scheduled sampling algorithm) are existing works, but the perspective of applying the methods to deterministic markup-to-image problem is novel.

Reproducibility: High.
summary_of_the_review: The  paper is clearly written. Treating markup-to-image as a generative process is a debatablly meaningful but novel point, especially when considering evaluating generative models. The authors use diffusion model as the generative model and adapt scheduled sampling to improve generation results. The methods are evaluated on four datasets with limited copmarison to other works if exist. The analysis of results is not totally convincing though, especially when the evaluation metrics are not convincing enough, more analysis on qualitative results would be helpful.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes markup-to-image generation as a task to measure the quality of image generation models, and examines the quality of the images generated by diffusion-based models for the task. They considered four domains, math, table, sheet music, molecules. As an improvement to the diffusion-based models for the task, they propose to use scheduled sampling. The experiments confirm that it improves the performance. The code and data are open-sourced.
strength_and_weaknesses: Strengths:

There is a value to introduce the markup-to-image generation task for image generation models. It can assess the aspects of the models that cannot be assessed by the existing tasks and there is a potential that the new task brings fundamental improvements to the models.

It is reasonable to use scheduled sampling for diffusion-based models and the effectiveness is confirmed by the experiments.

Weaknesses:

It more or less just applied a diffusion-based image generation model to the markup-to-image generation task with a technique inspired by a known technique. We can see the results by the latest image generation algorithm on the task, but it is not clear what scientific value it has to show the results by the models optimized for other tasks. It could have a deeper discussion on why the new task is needed and why the existing method does not work well yet, and a scientific contribution based on the deeper insight.

Probably, the experiments on assessing the absolute quality of the model are not enough. The "Perturbed" results in Figure 4 seem to have been generated by removing symbols for the amount in the x-axis. This can only tell how the evaluation metric is affected by random deletion errors. I do not feel it tells much to compare the simulated results with the ones by the diffusion-based models. The readers do not know how critical it is to miss a symbol to a particular renderer. How about insertion and substitution errors?
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is easy to follow although there were some typos. I find there is originality and novelty in the work but the discussion and the content of the work could go deeper as mentioned above. More meaningful experiments could be done to assess the quality of the latest image generation models.
summary_of_the_review: The introduction of the new task seems valuable and it could potentially have a huge impact by resulting in a number of following studies. However, the discussion and the content in the paper are rather shallow and the quality of the paper could be further improved. I do not see any concerns and fine to see it accepted, but it is not a very strong support.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes using scheduling sampling when training a diffusion model. It starts by introducing the markup-to-image generation task, which is a task of generating an image from the markup description of the image. This task has a one-to-one mapping between the input and the output. It is particularly suitable for evaluation. In this task, they compare training diffusion models with scheduling sampling or not. Experiment results suggest that with scheduling sampling, the trained diffusion models produce higher-quality images.

The paper receives four reviewers. Three reviewers consider the paper above the bar, citing the task novel, the idea of using scheduling sampling well-motivated, and the presented results convincing. One reviewer rates the paper below the bar. It considers the presented algorithm not well-explained and the presented results lacking those from state-of-the-art methods.

After consolidating the paper, review, and rebuttal, the AC decides to side with the majority. The AC agrees the presented task is interesting and the proposed method well-motivated. While the AC agrees that the algorithm can be better explained and the authors could have included more stronger baselines, the AC would not hold them against acceptance of the paper. The AC believes the paper has enough value to be published as an ICLR paper.
justification_for_why_not_higher_score: The paper presentation and experiment results are a bit lacking.
justification_for_why_not_lower_score: The idea is novel enough and the evaluation is sufficient.
Decision: Accept: poster