Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: Authors propose a diffusion model that adds a mask as an additional inputs to noisy data. The mask controls which positions in the input should be denoised, in order to provide control for partial inference with controllable positions. Authors experiment on polyphonic music modeling and language modeling.
strengths_and_weaknesses: # Strengths:
* The task of controllable generation is interesting, particularly to allow humans to interact with generative models. Allowing users to choose which pieces to improve is a particularly simple yet effective way of providing intuitive control and interaction.


# Weaknesses:
* The writing could be considerably improved. In particular, the introduction is very long and has several flaws such as:
  * The first paragraph is too vague to be of any use and I recommend authors remove it
  * The role of the introduction is to give context around the paper’s contribution, why it is useful and why the author's method is worth considering. However, the motivation of authors and the logical premises to their contribution are completely absent of the introduction. The introduction should at least finish with something akin to the first paragraph of the method section, which introduces the contribution in context.
* The method section is written such that a reader who is not an expert with SUNDAE (a previous work which we cannot expect the average ML researcher to know) will need to go back and forth between papers. While I acknowledge there are several opinions on this, I personally think the paper should be as self-contained as possible, and should provide much more context around the method.
* The motivation of the authors is not clear to me: unrolling and using a time-dependent mask are presented as key components of the method, yet the model is only used with 2 steps (without ablation) and a constant mask, which is a bit underwhelming and does not support author’s claims that unrolling is useful (claim in L221). Ablation studies would be a minimum.
* Experiments are performed on three datasets, yet their presentation is minimal. Results on polyphonic music modeling are just presented as a big table with zero comment on the metric, the baselines, and the variants of the model. Similarly, experiments on the news dataset are not commented on, baselines shown in the plot are not even cited. Authors should definitely revamp the experiments section to better highlight the advantages of their method.

In summary, while the topic is of wide interest for NeurIPS, the presentation of the author’s method and experiments is currently too flawed to be worth publication at NeurIPS.

questions: * Please number equations
* The github repository linked in appendix is empty 
* L110: why introducing m_t as a time-dependent mask while as soon as L116 it’s limited to a constant mask?
* L160: is there a specific motivation for sampling the difficulty variable at the batch level?
* L169: why not using the same notation as in the loss equation?
* L172: schemes -> scheme
* L185: or -> for
* L186: trained model -> the trained model
* L213: papers to use -> papers use
* L306: underflying -> underlying
limitations: Authors adress ethical concerns of music and language modeling in Appendix A, and current inference runtime issues in Appendix F.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduced a new generative modelling method for structured data and demonstrated its effectiveness on symbolic polyphonic music and character level and word-level text modelling. This method also offers the opportunity for subtle control at inference time, which could be useful for downstream human-in-the-loop applications. 
strengths_and_weaknesses: Strengths

Originality: This paper seems fairly original to me, but I am not sure since I am not familiar with prior work.

Significance: This paper seems fairly significant to me as author’s method is overall the best compared to other closely related works (even though I don’t know whether those are SOTA methods on this task). 

Quality: This work seems to be of high quality as there seems to be significant engineering effort put into the models to make them work well. I always have respect for such engineering efforts.  

Weaknesses
 
Clarity: To me, some places in this paper feel a lack of context. Since this paper is built on prior works like SUNDAE and Coconet, it’d be better to detail those two important prior works. This could be used to replace sections 1.1 and 1.2, as most people in NeurIPS community should at least know the gist of autoregressive models and non-autoregressive models. However without the context of SUNDAE and Coconet, it is hard to understand how this method differ from those works and the new insights in this work.   

questions: In Table 1, sometimes SD-AT seems to be doing better (e.g. for Note and Rhythm and slightly better for Repeated Sequence”, why is SD-AT not bolded for those and instead SMc-T is bolded? I understand that overall SMc-T performed the best, but it’s a bit confusing to me because bolded numbers are not always the best. 

DO you think including Figure 3 in the main paper would be necessary? Personally it is very hard for me to imagine the sound merely by looking at the visuals. A project page with generated music would be a much better media to demonstrate this, but given the limitation of the paper format, perhaps those figures could be moved to the appendix. With the space saved, authors could consider including more details about closely related works such as SUNDAE and Coconet. 

limitations: Yes, authors have discussed in detail the ethical implications of their work in appendix section A.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes SUNMASK, which models discrete sequences without ordering assumptions. The authors emphasized the fine-grained control during the iterative generation. Experiments on generating symbolic music and language model show the efficacy of the proposed SUNMASK.

strengths_and_weaknesses: In short, this paper is a combination of SUNDAE and Coconet. The major contribution of the paper seems to be the controlled unrolling procedure which is essentially adding a mask over the SUNDAE loss formulation. The authors include the lots of details for training and implementation which is very helpful for the readers to understand and potentially reproduce. 

However, this paper has several major shortcomings:
- The organization of the writing is very loose, making it very confusing for the readers. E.g. sections are named as a mere word like “ trade-offs”, “inference”, which does not offer any additional clues to readers beside its literal meanings.
- The paper is heavily based on SUNDAE, and it is not much of an improvement compared to SUNDAE if we look at, for instance, figure 2. Since fig 2 is the same task as the original task of SUNDAE, I believe it is a apple-to-apple comparison. 
- The analysis is not very convincing both quantitatively and qualitatively. Quantitatively, I think Bach Mock grading function is biased towards its own task, and it is less generaliziliby than BLEU score on the LM task, which SUNMASK does not outperform existing models. Qualitatively, the cherry-picked “ode to joy” example is good itself, but too specific, maybe it would be good we combine some human subject study?

questions: Figures4-9 are hard to understand, with very verbose description, maybe would be good to provide a high-level idea first?

limitations: The authors did not really cover too much about limitations...
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new non-autoregressive generative model for discrete sequences which combines a recent work SUNDAE with masking. The mask is provided to the network as additional conditioning - unlike e.g. BERT which masks out the tokens themselves. The advantages of the proposed technique are demonstrated on a polyphonic symbolic music generation benchmark JSB, where it outperformes another non-autoregressive method CoCoNet and the original SUNDAE. Additionally, the paper provides some experiments on text generation on EMNLP'17 News and text-8 datasets. Finally, the paper qualitatively investigates music in-painting with the proposed method. Extensive ablation studies are provided which compare different architectures and different sampling methods, e.g. finding typical sampling to be beneficial.
strengths_and_weaknesses: \+ The whole paper is well-written, illustrations are nice, music samples in the supplementary are helpful.

\+ Contribution is reasonably novel.

\+ Experiments are well-executed, with sufficient ablations to understand what changes lead to most improvement.

\+ Improvement over relevant baselines in polyphonic music generation is significant.

\- Language results are worse than SUNDAE.
questions: + Is there an AR baseline in Table 1? Is it BachMock?
+ figure 1: do correct predictions become unmasked? or do you keep mask constant for the unrolls?
+ table 1: is SMc-T better than everything else across all metrics or just overall? In the latter case, maybe use bold for the best method on the particular metric. Also, standard deviation is normally denoted as ±.
+ standard deviations in Table 1 are quite big, any way to reduce them?
+ L285-286: Negative BLEU/Self-BLEU in generated corpora of 1000 sentences - while it's understandably quite expensive to compute it for 10K samples x many temperatures (as was done in SUNDAE & ScratchGAN papers from DeepMind), keep in mind that the results are usually a bit different between 1K and 10K samples. It is certainly worth adding a disclaimer.
+ some questions (like the one about the compute) are addressed in checklist but not in the paper. Would it make sense to include those discussions in the main text?
+ Are there plans to open-source the code? Even non-runnable code is useful for people reproducing prior work.
limitations: Limitations are clear from the work itself.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: A generative sequence modelling based on masked unrolled denoising autoencoders, called SUNMASK, is proposed in this manuscript. The proposed separates the role of masking and correction by conditioning predictions on the mask. Compared to the baseline, the authors claim SUNMASK has a better performance on music modelling. The experiment also proves this contribution.

strengths_and_weaknesses: Generative modelling is a hot topic, since a lot of application can get the benefit. This idea is very close to SUNDAE proposed by DeepMind this year, which is claimed on page 6. But in Section 3.1, they mainly discuss the difference between the proposed methods and the coconut and the discussion about SUNDAE is ignored. This makes the originality not very clear. Hope the authors can give the discussion deeply and move the related\closed work to section 2. The researchers would like to know the difference when they get this manuscript since it is too closed to SUNDAE. The quality of this manuscript is good. Their designed experiments can prove their contribution and the manuscript is easy to follow. However, there are some points that could be further improved. For example, the font size of Table 1 is too small and Fig 4-9 could be shown in fig. 

questions: I hope the author can clearly claim their contributions, especially to discuss the differences between SUNDAE and proposed method.
limitations: Although the experiments have shown the proposed method has great performance, the discussion about the experiment is limited. For example, Fig 6-9 is just shown on page9 without discussion.
review_title: null

Meta-review: metareview: This paper introduces SUNMASK for modeling discrete sequences. It builds upon previous works such as SUNDAE, Coconet and order-agnostic NADE, but uses a masking scheme that enables fine-grained or human-in-the-loop control during the generation. The qualitative experiments about musical inpainting and masking terms in language modeling do support this motivation to some extent. However, the reviewers are mainly concerned with both the algorithmic novelty and experiments of the paper.
 
Regarding the algorithmic novelty, some reviewers are concerned that the method is a straightforward combination of SUNMASK and Coconet. I tend to agree with this. Also the reviewers are concerned that the paper could have done a better job in the introduction and background section by putting the method in a better context and better describing related methods such as SUNDAE. This could help highlight the novelty of the paper.
 
Regarding the experiments, some reviewers are concerned about language modeling (Fig. 2) experiments, and that they do not show much improvement over SUNDAE. I tend to agree, and this is also shown in the results of Table 3 where it is clear that the quality of the language model is not on a par with the recent developments.
 
Also some of the motivations of the paper, especially the arguments about "high trust / low trust" interpretation of the mask, was unclear to me.
 
In short, I believe the paper should be clarified and improved by addressing the above concerns.
confidence: Certain
Decision: Reject