SUNMASK: Mask Enhanced Control in Step Unrolled Denoising Autoencoders
Authors: Kyle Kastner
Tim Cooijmans
Yusong Wu
Aaron Courville
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Diffusion
Generative Modeling
Music Generation
Non-autoregressive Sequence Modeling
Transformer
Convolutional Neural Network

Abstract: This paper introduces SUNMASK, an approach for generative sequence modeling based on masked unrolled denoising autoencoders. By explicitly incorporating a conditional masking variable, as well as using this mask information to modulate losses during training based on expected exemplar difficulty, SUNMASK models discrete sequences without direct ordering assumptions. The addition of masking terms allows for fine-grained control during generation, starting from random tokens and a mask over subset variables, then predicting tokens which are again combined with a subset mask for subsequent repetitions. This iterative process gradually improves token sequences toward a structured output, while guided by proposal masks. The broad framework for unrolled denoising autoencoders is largely independent of model type, and we utilize both transformer and convolution based architectures in this work. We demonstrate the efficacy of this approach both qualitatively and quantitatively, applying SUNMASK to generative modeling of symbolic polyphonic music, and language modeling for English text.

SUNMASK: Mask Enhanced Control in Step Unrolled Denoising Autoencoders
Anonymous Author(s)
Affiliation
Address
email
Abstract
This paper introduces SUNMASK, an approach for generative sequence modeling based on masked unrolled denoising autoencoders. By explicitly incorporating a conditional masking variable, as well as using this mask information to modulate losses during training based on expected exemplar difficulty, SUNMASK models discrete sequences without direct ordering assumptions. The addition of masking terms allows for fine-gra gre-grained control during generation, starting from random tokens and mask over subset variables, then predicting tokens which are again combined with subset mask for subsequent repetitions. This iterative process gradually improves token sequences toward structured output, while guided by proposal masks. The broad framework for unrolled denoising autoencoders largely independent of model type, and utilize both transformer and convolution based architectures in this work. We demonstrate the efficacy this approach both qualitatively and quantitatively, applying SUNMASK generative modeling of symbolic polyphonic music, and language modeling for English text.
10
12
14
Introduction
16
Generative modeling approaches can stratified into different modeling approaches based on factoriza- tion to form two broad categories, autoregressive modeling (AR) and non-autoregressive modeling (NAR). We introduce SUNMASK, a NAR generative model for structured sequences.
18
19
Autoregressive Models
AR modeling with deep neural networks has been dominant approach to generative modeling 21 and feature learning [38 70 73
76 74) which has many crucial advantages in both training and inference. One key concern is the necessity of defining a "dependency chain" in the form of a
39
29 a (typically) directed acyclic graph (DAG). Sampling during inference can be accomplished straightforward manner using ancestral sampling sampling from the first variable variables in the DAG, using those to conditionally estimate probability distribution for subsequent variables.
25
Many applications have straightforward orderings in which to define this chain of variables, based on domain knowledge. For example following the flow of time for timeseries modeling often
27
28 logical choice, allowing models to make predictions into the future from the past. However in
29 many other domains, for example images, language, or music, the process defining a dependency chain over input variables (e.g. pixels, characters, words, notes) is far from straightforward, as
31 for any arbitrary ordering there can frequently be examples where this ordering creates long-ter term dependencies, otherwise makes satisfaction of dependencies during training and evaluation more 39 difficult than another alternative ordering.
34 This divide becomes further compounded in many creative applications to these domains, as creators
35 typically iterate repeatedly: forming a concept, applying an initial sequence steps create the
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

36
framing of the concept, and seeing where the creative flow may lead to alterations in the original concept. thus altering future steps. Though the resulting output may be perceived in a time-ordered fashion (for example, reading a book or listening to a song), the initial creation was performed globally and holistically. This global view is often critical to creating elements such as foreshadowing and tension which make the resulting output interesting enjoyable. This iterative process is directly 40
37
38
39
41 at odds with a strict AR factorization, and requires well trained AR models to cope with a high degree of uncertainty and multi-modality for long range dependencies, which can lead to logical mistakes or
43
other errors.
1.2 Non-Autoregressive Models
45
An alternative methodology for generative modeling is non-autoregressioon (NAR), broadly covering a large number of different modeling approaches which attempt remove assumptions about variable ordering, instead either hand-defining per-exemplar orderings, or modeling variables jointly without resorting chain rule factorization. One way to define ordering over variables is via masking of imputs or intermediate network representations 122 72 (77 (3 [5]] and indeed modern AR approaches such as transformers [75] use an autoregressive mask internally to define the chain of variables order. These masks can either be constant over all training (as in standard AR transformers and PixeICNN 73 or dynamic per example (as in MADE 22
46
47
49
50
51
52
When masks are dynamic per example, we begin to see the relationship between enforeing AR via masking and NAR methods, as although some ordering assumed this ordering is no longer constant, and it becomes possible to use the same trained model to evaluate the probability of a particular output variable under multiple possible orderings.
53
54
55
56
57
Closely linked to masking methods are so called diffusion models, which relax the variable ordering problem through noise prediction [67 69 30). Rather than predicting a new variable variables given previous ones in an arbitrarily chosen DAG, diffusion models focus on predicting a less noisy version of many variables jointly, given set of noisy input variables. Iteratively applying this learned denoising improvement operator should eventually result in predicting fully clean output estimate, given either a noisy version of the target domain, or even starting from pure noise. Given this framing it is clear that diffusion models are closely linked to denoising methods in general. specifically denoising autoencoders, as well as modern density modeling approaches such as generative adversarial networks (GAN (33). variational autoencoders (VAE [41), flow-base
58
59
60
61
62
63
64
65
66
models (NICE [.5), RealNVP [6], Normalizing Flows [6]. IAF [42] MAF (57) ) iterative canvas sampling (DRAW (24), and noise contrastive estimation (NCE 27 Particular applications of
67
68
this denoising philosophy such as BERT [14,, WaveGrad (9. and GLIDE [56), have resulted in
large quality improvements for feature learning and data generation for text, images, and audio
70
66
60
71
1.3
Trade-offs Between AR and NAR Approaches
72
The choice between AR and NAR methods is not clear-cut. For many domains, high-quality models exist using both approaches but we can define some crucial parameters. Some NAR methods such as GAN or VAE are capable of generating output in only one inference step. however they are typically hard train on certain data modalities (e.g. text data) comparing to AR counterparts. Other NAR methods such as diffusion models typically allow for choosing a diffusion length during inference, which is independent of that used at training. Choosing low diffusion length can frequently lead to poor sample quality, and tuning this setting (among many others) is critical to high quality generation. However if the tuned diffusion length for given sequence (of length T) is shorter than the length of those sequences, the NAR method has a computational advantage over the equivalent AR model (which would require T steps for T length sequence). In addition, the ability to tune this diffusion length can be useful in interactive applications, or when a variety of output is desirable. This
73
74
76
77
78
79
80
81
82
83 setting can also be a curse, as even well-trained models perform poorly with improper diffusion
84 settings. Several branches of current research are focused on improving guarantees
[35 68 40 and
85
convergence speed for diffusion models
36 79).
1.4 SUNMASK, non-autoreereesive sequence model
87 We introduce SUNMASK, a NAR sequence model which uses masks over noised, discrete data to learn a self-improvement operator to transition from categorical noise to the data distribution in

---

89
iterated steps. Given a target data representation, we train model which can map from a noisy version of input data to a corrected form of the imput. In this work, we use multinomial noise- namely entries are corrupted to of P possible values (for given set size P), with the number of noised entries defining the relative noise level for the training example. This is similar to many diffusion approaches at high level, and particularly shown to be an effective tool in SUNDAR [65] and Coconet [33]. In addition to the use of multinomial noise, we also form a mask representing where the data was noised, feeding this mask alongside the input data to form conditional probability distribution.
90
91
92
93
94
Training Data
Model Input
Denoise Step-1
Denoise Step-2
SUNMASK
Loss
Corrupt
Meek
Stop Gredient
Figure 1: Step-unrolled denoising training for SUNMASK on polyphonic music, unrolled step length 2. Training data (left) consists of four voices corrupted by sampling a random mask per voice and replacing the masked data (red) with random pitches (green). SUNMASK takes both mask and corrupted training data as input, predicting denoised original data as output. In the second step, the model takes a sampled version of the model step predictions and the same mask as input, outputting another prediction of the original data.
96
Method
97
The relationship between discrete diffusion and denoising autoencoders has been explored in previous work [I 65 3 32]. We build upon this foundation, combined with many insights from prior
99
100
orderless modeling approaches, crucially Orderless NADE (72,, Coconet (33 (which is a more
modern variant of Orderless NADE), and SUNDAE 655
101
102
SUNMASK is built around process Tt fe(-(xx-15 m. space X = {1, u} of arrays of categorical variables. This parametric transition function fo takes an additional argument m E 0, 1 During training, m indicates variables that were not corrupted, and as a consequence we can use during inference to tell so which variables to trust.
103
104
106
Given a sequence of masks mo..... m1-11 the generating distribution of our model derives from a prior Po (typically uniform noise) and repeated application of fa:
107
pr pT (#T; mo...., MT-I)
Σ II (i PO
(1)
-11..EE-IEXXtE t=1
108
In practice, po is typically elementwise iid uniform noise, and the masks mo.. 1TT1 are drawn according to a schedule and may be held constant for several steps.
109
110
To train Se. we take a training example x ~ Pdata and draw a mask m. We apply the corruption procedure 20 ~ 9 [r; m) obtain zo which equals a where the mask m is true and uniform random values elsewhere. Then we iterate It ~ fe(-1z;-1; m) with the aim of reconstructing x.
111
112
As in SUNDAE, the transition fo models the variables as conditionally independent of one another. However SUNDAE has no direct concept of masking. SUNMASK thus combines past insights from
114
the masked NAR models Orderless NADE and Coconet with existing concepts from SUNDAE, along
116 with new model classes and inference schemes to form a powerful generative model. Similar to
117 SUNDAE, our objective is to minimize (L()) + (()) where
(5) Σ (1 m ) log 16 (10 (zx 15
T 10 (0)
E. E 1 E
Pdata ql-la, mo) fa(-I mo) (ol-(xs;m:))
E, 1 m (
m
folt aim; 2)
(2)
3

---

118 is the reconstruction loss for the elements of x that were cortupted. As in Orderless NADE 722 and Coconet [33. we weigh each term according to the size of the mask, to ensure that the overall weight on each conditional f(6)) is uniform across i. Unlike previous methods, we target only masked variables in the loss. In practice we choose my
119
120
121
14-1 during training and = 2. Since we only go to t = 2, keeping the mask constant is a close enough approximation to the masking schedule used in inference. The choice of t is driven by the ablation study SUNDAE, where = 2 was found to account for nearly all performance gains in translation experiments, with higher unrollings showing no additional benefit. addition higher values of unrolling generally increase memory usage, making the training of high order unrollings complicated.
122
123
124
125
126
Coupled with multi-step unrolling, the SUNMASK training scheme encourages learning complex relationships between the mask and the data, allowing the potential for multi-level trust over the input data: variables with a mask value of which appear correct (given context), variables of mask value I which appear incorrect, variables of mask value O which appear correct, and variables of mask value 0 which appear incorrect. Denoising only methods (such as SUNDAE (65) would need to form an internal, controllllel mask in order to disentangle these states, and 0 mask models (such as Coconet 33 have controllable input masks but combine both masked states.
128
129
130
131
132
133
134
SUNMASK allows for direct control at inference using both proposal masks and noising of variables, combining elements of both SUNDAE and Coconet. We show high level example of the unrolled training scheme, mask proposals, and input data processing in Figure] T
135
136
137
2.1
SUNMASK, SUNDAF, and Coconet Comparison
138
The overall unrolled mask and iterative inference setting is largely independent of architecture choice, and as long as the internal architecture does not make any ordering assumption over the input data we can incorporate it into SUNMASK. We use two primary archetypes for the internal model in this paper: Attentional U-Net and Relative Transformer. Detailed description of the respective architectures can be seen in the Appendix.
139
140
141
142
143
SUNMASK uses an unrolled training scheme, similar that shown in SUNDAE, as well as a mask which is input to the model and defines manipulated variables as in Coconet. The loss is masked based on this manipulation mask, unlike Coconet or SUNDAE. The SUNMASK loss is further weighted by
144
145
148
the total amount of masked variables. Comparisons of various high level modeling features between
147
SUNMASK, Coconet, and SUNDAE are shown in Table
Table 1: Comparison of model features for SUNMASK, Coconet, and SUNDAE
Model
SUNMASK
Coconet
SUNDAE
Mask input to model
Masked loss
X
X
Re-weighted loss
J
X
Unrolled loss
X
Inference mask schedule
X
X
Sampling rejection step
V
Mask control preserves data
X
X
148
2.2
Model Training
During training, the internal architecture is combined with step unrolled training procedure, as highlighted by SUNDAE (65). Rather than directly randomizing positions, we re-write this as a masking scheme, first sampling mask (with 0 randomize, 1 keep, which we denote O-active format) then performing randomization to one of P possibilitics. for the masked subset of K variables. This random masking procedure is equivalent to the approach from SUNDAE, but using mask allows us to further combine the mask information with the input data, in order to form a conditional probability estimate. In addition, this O-active masking scheme makes direct comparison masking schemes with absorbing states (such as OrderlessNADE [72]. Coconet [33. VQ-Diffusion [5] and AR-ARDM (1)) simpler, as the mask can be directly multiplied with the data in a O-active format.
150
152
154
156
157
Convolutional SUNMASK incorporates mask information with a one-hot data representation by concatenation along the channel axis. Transformer SUNMASK uses slightly different setting-
158

---

160
assuming input transformer data (T, B), with T timesteps, B batch elements, and vocabulary size P transformed to (T, B, L) dimensional embedding, we repeat the mask along the embedding axis I times, downweighting the values in the mask by VT. for numerical reasons. Concatenating this reduced mask with the input embedding along the last axis is sufficient to form the desired conditional probability distribution. This stretched and reduced mask format provides more stable training than other schemes such as separately embedding the mask, then concatenating or summing with the input
161
162
163
164
165
186
data embedding.
167
Each training batch is randomly sampled from the training dataset, and corresponding noise value drawn from rand(N) for NN examples in the minibatch. This per-example noise value is then used to derive per-step mask over T timesteps. by comparing noise rand(N) < rrad(N (N, T). During training, this means some examples have a high per-example noise value (9)), and thus many values masked and noised in the training, while other examples may have a low noise value (e.g. .01) drawn instead. Combined with a training loss which learns to denoise the input and focuses on imputing information about masked corrupted inputs, the overall model will learn chain to go from more noisy data to less noisy step-wise, resulting in a learned improvement operator 32 65
168
170
172
173
174
175
This improvement operator can be applied to noisy data or pure noise, and iterate toward predictive sample from the training distribution. See Multinomial Diffusion 32 and SUNDAE (65) for more detail on this proof, well fundamental work on denoising autoencoders [1] In SUNMASK, we combine the mask used to noise the input with the input data itself, while modifying the loss to predict only masked variables. In addition, we downweight the loss by 1- for each batch element, meaning that losses for heavily masked entries are downweighted compared losses on examples with little masking. in a form of curriculum weighting based on expected estimation difficulty.
176
178
179
180
181
182
While
step denoising scheme can be sufficient for learning the data manifold M. unrolling this denoising scheme into a multi-step process can have performance benefits. SUNMASK directly uses the unrolled loop scheme described in (3). using a step value of 2. For detailed description of
189
184
185
the step unrolled training scheme, see Appendix or the overview description from SUNDAE (65].
186
The masked and unrolled training can be seen as container for any internal model which does
187
not make ordering assumptions, and we utilize both convolutional U-Net (a variant of GLIDE [56]
188
U-Net) and Relative Transformer
59 models for various experiments, shown in Section
2.3
Inference Specific Settings
190
Well-trained SUNMASK models should be applicable to full content generation, as well as a variety of partially conditional generative tasks such as infilling and numan-in-the-loge creation. Basic sampling involves creating set of variables, with all variables randomly set to 1 of P values in the domain (or partial randomization in the case of infilling) along with an accompanying mask, which is initially all 0 for full generation, or mixed is and Os for partial generation tasks. Given this data and mask as input, the trained model then predicts a probability distribution over all possible values, for all variables. Despite the use of masked losses in training, we sample these prediction distributions for all variables. These predictions are then accepted or rejected from the original set, resulting in a new variable set. We then sample new mask (based on a predefined schedule) and combine it with the initial mask, then iterate this overall process, updating at least some of the variables at each step.
191
192
193
195
196
197
198
199
200
During inference we use several key techniques to improve generative quality. We use typicality sampling [52) on the output probability distribution and a variable number of diffusion steps, on the order of 100 to 2000. Masks are randomly sampled using the schedule defined in [33] which linearly decreases the number of masked variables over time according to an = max((minn Qmax
201
202
203
33 along with an optional triangular linear ramp-up and ramp-down schedule for the probability of accepting predictions from the model into the current variable set at each step, as shown in [65] Active balance, by increasing the probability of updating variables which have been updated less often, is another inference time option. Variables can also be re-noised at any step, randomly resetting any variable with a O value for the mask at that diffusion step. Some problems (primarily symbolic music modeling) showed increased variability from active balance and re-noising, but the usefulness of these options is task
NN (Omax 0min)) with amin = 0001, Olmax .999, and v = 3/4 as in
206
208
210
211
dependent.
212 We caution that tuning hyperparameters for inference is critical to success, as improper settings can
drastically lower the performance of SUNMASK, see Section 4 for variance over various inference
5

---

settings in different tasks. For human in-the-loop applications, the existence of these controls can allow number of ane-grained workflows to emerge, driven by expert users to create and curate 215
214
216
interesting output [8] 11, demonstrated in Figure 3
3
Related Work
217
218
We state here some key related approaches, as well as how our method differentiates from these previous settings. A number of recent publications on diffusion models and feature learning have
219
220
incorporated masks as part of their overall training scheme [31 7 22), however these papers use
221
masks for blanking, rather than as indicators over stochastic variables. Many infilling models [7
222
37
14 50 10], and masked image models [29] feature conditional modeling with mask (blank) token, predicting the variables masked from the input for feature learning or generative modeling.
223
224
XLNet (77] combines the infilling and autoregressive paradigms, learning arbitrary permuted orders over masked out variables, using blank-out masking and randomly generated autoregressive ordering similar to OrderlessNADE and Coconet. Conditional diffusion generators [53] 63 64 and GAN generators [20] have the combination of mask indicators as well as preserving stochasticity of the masked variables. However these methods do not use an unrolled training scheme, and generally target image related tasks, with the notable exception of maskGAN. Many models use a concept of a working canvas, and do repeated inference steps for generation correction of data [24] 45 21]
225
227
229
231
551. SUNMASK differs from these models due to architecture choices, training scheme, and loss
232
weighting, as well as application domain [54 58
62
25
56
60
233
3.1
Convolutional SUNMASK
234
SUNMASK is most closely related to coconet (33 and SUNDAE 665. Coconet (as an instance of OrderlessNADE), trains by sampling a random mask per training example, using this mask set part of the input (in one hot format) to zero. The mask is further concatenated to the zeroed data along the channel axis, and this combined batch is passed through a deep convolutional network with small 3 x 3 kernels. Convolutional SUNMASK uses downweighted loss over only variables masked in the input. However, SUNMASK additionally uses the unrolled training scheme, as well as a different inference procedure due to preserving the values of masked out variables during training and sampling.
235
236
237
238
239
240
241
242
Our best performing convolutional SUNMASK architecture takes hints from recent image transformer and vector quantized generators, exchanging the small kernels used in Coconet for extremely large kernels of shape 4 x P over the time and feature dimensions, somewhat analogous to input patches, removing the model's 's translation invariance over the feature axis by setting kernel dimension equal to the total feature size. However this makes the number of parameters per convolutional layer extremely large. Convolutional SUNMASK adopts an attentional U-Net structure which reduces only across the time axis, modified from GLIDE [56], rather than the deep residual convolution network used by Coconet. Combined with the addition of step unrolled training, we are only able to train convolutional SUNMASK with a batch size of (expanded to effective batch size 2 due to step unrolling) on commodity GPU hardware with 1 16GB VRAM.
243
244
245
246
247
248
249
251
252
Due to the design choice of extremely large kernel sizes which depend on the size of the domain, we only use convolutional SUNMASK for polyphonic music experiments, see Section for more details. Exact specification of the convolutional U-Net architecture can be seen in the Appendix.
253
254
255
3.2
Transformer SUNMASK
256
Transformer SUNMASK relates closely to the transformer used in SUNDAE. The architecture uses a relative multi-head attention [4] and has no autoregressive masking. SUNMASK transformer also uses larger batch sizes, typically 20 or larger, though this is far smaller than the batch sizes seen in the experiments of SUNDAE. Sequence length and data iterator strategy were both a critical aspect for training transformer SUNMASK. We found short sequences (from 32 to 128) worked best, along with iteration strategies that were example based. In the language experiments, padding each example to some max length resulted in more stable training than the typical language modeling
257
258
259
260
261
262
approach of treating the corpus as one long sequence and slicing into even sized chunks, or iterating sequentially. The stability gap between padded sequences and non-overlapping chunking became especially apparent at sequence lengths above 128 with transformer SUNMASK.
264

---

Table 2: Quantitative results from the Bach Mock grading function [9]
Lower values represent better chorales.
Model
Note
Rhythm
Paraliel Errors
Harmonic Quality
Intervals
Intervals
Intervals
B Intervals
Repeated
Overall
Sequence
GT
0.24 10.15
0.13
0.0:10069
0411:02
0.4710.28
0:49:10:22
0.53:10.24
0.69::04
1.29
1.6
BachMock
0.37=0022
0.26=0.14
2.16=3.22
0.54=0.31
0.53=0.35
0.71+00.44
0.73+:0.38
0.89+ 68
1.86+2881
8.9444.64
SMC-T-BEST20-200
0.
0.23
0.0:10.81
0.68
0.59
0.88:10:42
0.80:10.20
0.71::0.27
1.42
7.16.10.77
AugGen
8.02+2.92
Coconer
0,24
1.85=0.39
2.61
0.39
0.70
0.86
0,86-1-0.42
1.02:-0.38
6.07-::.76
17.00-4-6.88
0.59+1182
0.93=0.84
6.42-4.11
0.98+00.7
1.17=5.09
2.65-14008
1,574:5 68
2.57: 28
2.45+22.39
23.25-221.44
SD-T
0.63:1:244
0.60=0.96
3.82.14.98
0.961:0.64
1:203
3,40:144.99
3.02:1.5.02
2.36.13.99
1.52:13.43
23.09:122.88
SD-AT
0.524:2.42
0.60=0.95
3. 11185.110 10
0.96=00.44
1.24=5.00
3.93:-5003
2.22+
2.00:
1.80
18 18.90
SMc
0.87::2.05
0.63=0.77
1.38=6600
0,2
5.22
2.02+-4.36
1.94::5772
2.91=:4994
2.32++2.31
22.47=20.80
SMc-A
1.02-1:2.22
0.47
3.92:13.91
0.91
2.32.15.23
3.54114498
2.74:15.30
5.96:1:4.99
2.23:1:3.82
27.82 18.82
SMc-T
1.79
0.69
3.273
0.93
0.8011.551
0.99114.01
1.20-14.68
1.40.13.91
1.81.10.83
13,43
SMc-AT
1.66+t1.90
0.55=0.29
2.76=33.63
0.94=0047
0.91#4.11
1.10+44.00
1.26-1-.22
1.55+4.56
2.05++0.96
16.50= 17
SMc-ATN
2.24+2.366
0.58=0,49
6.82=4.81
1.56+0.54
6.46#4.14
8.51+:4.43
7,21+4.28
7.60+
1.47+10022
18.85-28841
SMI
1.00
0.74=0.90
0
164=0070
8.9444.66
6.49-14.99
8.47455588
7,72444.41
3.10+22.97
42.87
SMI-A
3.00+11.85
0.74=0.90
0.00=1105
0,6
8.9444.66
6.49:44999
8,47435558
7.72+
3.10+2..77
42,87
SMI-T
3.74422.16
0.58=0.56
0.00=22.56
1.73+0.73
8.75
6.22:43399
8.05-14.77
7.95
2.35+1.79
46.21117.30
ST
3.74
0.58 0.56
0.00 2.56
1,73
8.75
6.22.1.3.9
8.07.14.73
7.95.14.49
1.35
46.21 17.30
We list the hyperparameters for the transformer SUNMASK models in the Appendix. Transformer SUNMASK was trained on every dataset used in this paper, and we show performance in Section 4 as well comparisons to convolutional SUNMASK on symbolic polyphonic music modeling. Both convolutional and transformer based SUNMASK use the Adam optimizer, with gradient clipping by value at 3. Inference hyperparameter types and general sampling strategies used are the same with both models, though specific hyperparameter values may change between datasets.
267
269
271
4
Experiments
273
4.1
Quantitative Results
We demonstrate the use of SUNMASK for polyphonic symbolic music modeling the JSB dataset 2
275
5. The JSB dataset consists of 382 four-part chorales, originally written by Johann Sebastian Bach.
These chorales are quantized at the 16th note interval, cut into overlaaalgp chunks of length 1 128, skipping chunks which would cross the end of a piece. This processing results in a training dataset of 4956 examples, with each example being size (4, 128). We train convolutional and transformer versions of both SUNMASK and SUNDAE for comparison, as well as the pretrained Coconet [33]
277
279
280
For polyphonic music, the quantized data was rasterized in soprano, alto, tenor, bass (SATB) order, as in Music Transformer 34 and BachBot (47). then chunked into non-overlapping training examples.
281
282
Results are shown in Table 2 These results evaluated Bach ground truth data (Bach GT),
283
BachMock Transformer (BachMock [19 49) (closely related to the decoder from VQ-CPC (28).
284
Coconet, SUNDAE (SD), and SUNMASK convolutional (SMc) and SUNMASK transformer (SMt).
285
Model variants are indicated with Active Balance (A), Typical Sampling (T), and Noise (N).
The grading function used for evaluation, referred to BachMock here, is designed specifically to correlate with expert analysis on Bach. In particular using this metric to choose correct examples in a paired comparison test, outperforms novice, intermediate, and expert listeners by varying margins [19]. This indicates that scoring well on the aggregate metric should correlate to high sample quality. The metric has many sub-parts, ranking various musical attributes crucial to codifying the style of J.S. Bach. AugGen 49 incorporated this metric into an iterative training and sampling scheme which improved final generative capability for a fixed model, showing the effectiveness of BachMock in practice for ranking machine generated samples. For every grading function in Bach Mock grading function, we show the median value and + standard deviation as well as the overall grade, and lower values are better. We see the strongest results for convolutional SUNMASK with typicality sampling. Combined with final top-N (N 20) selection out of a candidate set of 200 samples, the overall sample quality outperforms strong baselines. In addition, the massive performance gain from top-N selection indicates that the variance is likely driven by failures during sampling, rather than more
287
289
290
291
282
293
294
285
296
297
298
299
fundamental modeling errors.
The EMNLP 2017 News dataset common benchmark for word-level language modeling []] containing large number of news article sentences [51). Preprocessing steps collapse to sentences
300
302
containing the most common 5700 words, resulting in a training set of 200k sentences with a test set
of 10k. The overall maximum sentence length is 51. Common processing for this dataset includes
7

---

304
padding all sentences up to this maximum length, different than the standard long sequence chunking commonly used in other language modeling tasks.
305
We show the results of several SUNMASK models for generating sentences similar to 306
MMMNLP2017Neee comparing benchmarks using the standard Negative BLEEU/elf-BLLU evalua- over generated corpora of 1000 sentences in Figure 2 This set of scores, varied across
308
tion 80 6
309
temperature, is compared against baseline scores [48] 78
13 75), similar to the evaluation 65 These reference benchmarks used 10000 sentences to form performance
8
26
310
shown in SUNDAE
estimates.
1.0
1+0.2
5-0.6
ARt Model
SUNDAE
SUNMASK
SUNMAAS 1010
SUNMASK 5-15
05-
0
LeaKSAN
b-0.8
ParkEAN @ SEOEKN
0.-
-0.6
-0.5
-0.4
-0.2
-0.1
BLEU (iqualit))
Figure 2: Negative BLEU/SeIf-BLES scores EMNLP2017 News. Left (x-axis) is better, lower (y-axis) is better. Quality/variation is controlled by changing the temperature (1), and varying diffusion schedule (s). For SUNMASK, typical sampling results [52) are shown.
311
200
101000
42
Figure 3: A
Figure 4: B
Figure 5: C
SUNMASK harmonization (bass, tenor, alto) of an existing melody (A)prano)A), with mask which highlights the left half (0 64) soprano voice (B), left half mask with right half replacement (C)
312
4.2
Qualitative Study of Masking For General Task Control
Given the flexibility of masking inference, we perform number of qualitative queries to inspect how the model adapts based on noise and mask value. Figures 3 and demonstrate the use of SUNMASK for musical inpainting, holding the top voice (soprano) either fully or partially fixed to the well-known melody "Ode to Joy", by Ludwig van Beethoven. We see the trained model is more pre-defined mask.
314
316
than capable of inpainting based
318
test masks which hold the whole soprano fixed, masks which cover only parts of the soprano but do not allow randomization away from those notes, and masks which cover parts of the soprano but
320
allow rewriting of the non-masked parts of the soprano, as well as rewriting all other voices. The use
of masks to focus on subsets of variables while preserving underlying intermediate predictions is

---

322
unique to SUNMASK, as SUNDAE does not have an explicitly controllable input mask and Coconet does not have the ability to mask without also blanking the underlying variable.
323
324
This control is also demonstrated in Table 3 where masking is used to variably increase or decrease the weight on various pre-specified terms, held fixed throughout inference. The combination of these words, and their mask status can be seen to influence the overall tone of the selected text passages which showed the strongest effect in a particular inference batch. Though the generation quality is flawed, we clearly see a relationship between the masked word and the emergent surrounding context,
325
328
327
328
329
for example highlighting war draws forth divorce, attack, and leave, while play instead references Premier Cup and excitement. We see similar results on a batch scale, and full demonstration of the text samples can be seen in the Appendix.
330
331
5
Conclusion
332
333
We introduce SUNMASK, a method for masked unrolled denoising modeling of structured data. SUNMASK separates the role of masking and correction by conditioning predictions on the mask, allowing for fine-gres grained control at inference. When applied to text as well as symbolic polyphonic music, SUNMASK is competitive with strong baselines, outperforming reference baselines music modeling. Leveraging the separation of mask and noise allows for subtle control at inference, paving the way for variety of domain specific applications and generative pipelines for human-in-the-loop creation.
335
336
337
338
339
Table 3: Qualitative samples using masks to emphasize the influence of particular words.
Samples from a SUNMASK Transformer trained on the word level MAANLL20117Nere dataset.
think want to leave success the end of the disaster but because that a nice to say it s not good to be the challenge and this is a very good thing <eos>
Success unmasked disaster masked
That was the job was success to have pay my disaster but hopefully I have been able to
pull playing in the first couple of the season ve been happy to go through this
team, he said <eos>
Although more than 80, 000 success have been displaced in the disaster since the last
Success masked disaster unmasked
year, more than 700, 000 lives have been injured in the country, and 70 of them were killed, according to the UN media <eos>
haven had a success at the league, the disaster and picked running with the door ago
we have Champions and I was couple of pressure... ... and it was lot of times <eos> This is a fact on the celebration is such a really good crime, but it can be some of the most
Celebration unmasked crime masked
good people around the world and think it must be the good way to do it <eos> It s part of those celebration at the start of the crime, and it s lot of good pride over the past few years it s going to be more happy to play through this world <eos>
Last year, the numbers of celebration applications have been adopted in crime since the
Celebration masked crime unmasked
Middle since since year has watched a rate of more than 95 per cent in the UK since 2011 2015 to 45 per cent <eos>
The Prime Minister has been celebration to the course of the crime deal which and to have a relationship of the European Union, with the rest of the European Union has before the scandal <eos>
The Prime Minister David Cameron said war had not be hard to play the divorce of the European Union. and determined would mark a divorce between the majority of the UK and leave the bloc of the European Union <eos>
War unmasked play masked
It may be really more special war try to play it, and I hope we re going to be able to attack this team so we have do it <eos>
But I was proud of the war I ve got to play to it but I wish that is because I want to do. and
War masked play unmasked
I' m pretty excited before the end of the season, he said <eos>
We are in the Premier Cup war and we want to keep play with their top six that which we need
to play the World Champions and at the end of the season that we have to be it well <eos>
By the time. the driver had been deployed to lie out to the incident, an new official said that the
Unconstrained generations
woman had not been found a them <eos>
According The Wall Post survey poll found that 80 per cent of eligible older registered who, they thought he would be the rate for 10 per cent less likely to vote, while 16 per cent of those said they would still less likely to happen <eos>
9

---

References
340
341
[1] Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data- generating distribution. The Journal of Machine Learning Research, 15(1):3563-3593 2014.
[2] Moray Allan and Christopher Williams. Harmonising chorales by probabilistic inference. Advances in neural information processing systems, 17, 2004.
344
345
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:1 17991-11999, 2021.
347
348
[4] Philip Bachman and Doina Precup. Data generation as sequential decision making, Advances in Neural Information Processing Systems, 28, 2015.
[5] Nicolas dependencies in high-dimensional sequences: application to polyphonic music generation and transcription. In Proceedings of the 29th International Coference International Conference on Machine Learning, pages 1881-1888, 2012.
Yoshua Bengio, and Pascal Vincent. Modeling temporal
351
353
354
66] Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short. arXiv preprint arXiv: /8/
355
1. 1.02549, 2018.
[7] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In The IEEE Conference on Computer Vision and Patlern Recognition (CVPR). June 2022.
357
358
[8] Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. Maximum-likelihod augmented discrete generative adversarial networks. arXiv preprint (tfd::/70000700 2017.
380
361
382
[9] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2020.
383
384
385
[10] Kevin Clark, Minh- Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators, arXiv preprint arXiv:2003. 2020.
386
357
[1I] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance, 2022.
388
389
370
[12] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint 19Xiv:1901.02866 2019.
371
372
373
[13] Cyprien de Masson d' Autumme Shakir Mohamed, Mihaela Rosca, and Jack Rae. Training language gans from scratch. Advances in Neural Information Processing Systems, 32, 2019.
374
375
[14] Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18X0::8800 2018.
376
377
378
[15] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv: 10. 16, 2014.
380
[16] Laurent Dinh, Jascha
and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 08800, 2016.
381
382
[17] Chris Donahue, Mina Lee, and Percy Liang. Enabling language models to fill in the blanks. arXiv preprint arXiv:2005.053339, 2020.
383
[18] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for hegl-resolution image synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 12873- 12883, June 2021.
385
10

---

387
[19] Alexander Fang, Alisa Liu, Prem Seetharaman, and Bryan Pardo. Bach or mock? a grading function for chorales in the style js bach. arXiv preprint arXiv:2006.13329, 2020.
388
389
[20] William Fedus, lan Goodfellow, and Andrew M. Dai. Maskgan: Better text generation via filling in the
380
2018.
[21] Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, SM Ali Eslami, and Oriol Vinyals. Synthe- sizing programs for images using reinforced adversarial learning. In International Conference on Machine Learning, pages 1666-1675. PMLR, 2018.
382
394
[22] Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle, Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning, pages 881-889.
396
PMLR, 2015.
[23] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
398
400
[24] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In International Conference on Machine Learning, pages 1462-1471. PMLR, 2015.
402
[25] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. arXiv preprint WIXi:2211.114222 2021.
404
406
[26] Junliang Guo, Linli Xu, and Enhong Chen. Jointly masked sequence-to-sequee model for non-autoree neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 376-385, 2020.
408
[27] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international con- ference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
410
412
413
[28]
Gaëtan Hadjeres and Léopold Crestel. Vector quantized contrastive predictive coding for template-based music generation. arXiv preprint arXiv:2004. 201100 2020.
414
415
[29]
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. wwXiv:2111.06377 2021.
416
417
[30]
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-66 6850,, 2020.
418
419
[31] Emiel Hoogeboom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. arXiv preprint arXiv:21 2021.
420
421
[32] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34, 2021.
422
423
424
[33] Cheng-Zhi Zhi Anna Huang, Tim Cooijmans, Adam Roberts, Aaron Courville, and Douglas Eck. Counterpoint by convolution, arXiv preprint arXiv: MXXiv:9933007227 2019.
425
426
[34] Cheng-ZKi Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. In International Conference on Learning Representationss, 2018
427
429
[35] Chin-Wei Huang, Jae Hyun Lim, and Aaron C Courville. A variational perspective on diffusion- based generative models and score matching. Advances in Neural Information Processing Systems, 34, 2021.
431
11

---

433
[36] Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. Fast- diff: A fast conditional diffusion model for high-quality speech synthesis. arXiv preprint arXiv:220.009934 2022.
434
436
[37] Daphne Ippolito. Anna Huang, Curtis Hawthorne, and Douglas Eck. Infiling piano perfor- mances. In NIPS Workshop on Machine Learning for Creativity and Design, 2018.
437
438
[38] Nal Kalchbrenner, Ivo Danihelka, and Alex Graves. Grid long short-term memory. arXiv preprint arXiv: 2015.
439
440
[39] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pages 2410-2419. PMLR, 2018.
441
442
443
444
[40]
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems, 2021.
446
[41] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
447
448
[42] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
449
450
451
[43] Zhifeng Kong, Wei Ping, Jiaji Huang. Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2020.
452
453
454
[44] Max WY Lam, Jun Wang, Rongjie Huang, Dan Su, and Dong Yu. Bilateral denoising diffusion models. arXiv preprint MrXivv2208... /1544, 2021.
456
[45] Alex M Lamb, Devon Hjelm, Yaroslav Ganin, Joseph Paul Cohen, Aaron C Courville, and Yoshua Bengio. Gibbsnet: Iterative adversarial inference for deep graphical models. Advances in Neural Information Processing Systems, 30, 2017.
457
459
[46] Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In EMNLP, 2018.
460
461
[47] Feynman Liang. Bachbot: Automatic composition in the style of bach chorales. University of Cambridge, 4:19-48, 2016.
[48] Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. Adversarial ranking for language generation. Advances in neural information prucessing systems, 30, 2017.
464
465
[49] Alisa Liu, Alexander Fang, Gaëtan Hadjeres, Prem Seetharaman, and Bryan Pardo. Incorporat- ing music knowledge in continual dataset augmentation for music generation. arXiv preprint arXiv:2006. 13331, 2020.
466
467
468
[50] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907. 11692, 2019.
469
470
471
[51] Sidi Lu, Yaoming Zhu, Weinan Zhang, Jun Wang, and Yong Yu. Neural text generation: Past, present and beyond. arXiv preprini
2018.
[52] Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Typical decoding for natural language generation. arXiv preprint urXiv:2202.00666 2022.
474
475
[53] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.0 01073, 2021.
477
478 [54] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint arXiv:2103.16099, 2021.
12

---

480
[55]
Alex Nichol. Vq-draw: A sequential discrete vae. arXiv preprint arXiv:2003.01599 2020.
[56]
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Hya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprini ar 2021.
482
484
[57] George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
485
486
[58] Ashis Pati, Alexander Lerch, and Gaëtan Hadjeres. Learning to traverse latent spaces for musical score impainting. arXiv preprint arXiv: arXiv:1907.01164 2019.
[59]
Christine Payne. Musenet. opwnnai.com//lo//maaeeiio
2019.
[60]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022.
490
481
[61]
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR, 2015.
482
483
[62]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
434
485
[63]
Chitwan Saharia, William Chan, Huiwen Chang, Chris A Lee, Jonathan Ho, Tim Salimans, David J Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. arXiv preprint arXiv:2111.00 1.02111..0826.6 2021.
498
497
498
[64] Chitwan Saharia, Jonathan Ho. William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2/04.07636 2021.
499
500
[65] Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. Step-unrolled denoising autoencoders for text generation. arXiv preprint / 2021.
502
503
[66] Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. Latent-variable non- autoregressive neural machine translation with deterministic inference using a delta posterior. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8846-8853. 2020.
505
507
508
[67] Jascha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
510
511
(68] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34, 2021.
512
[69] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
514
515
[70] I Lucas Theis and Matthias Bethge. Generative image modeling using spatial Istms. Advances in neural information processing systems, 28, 2015.
516
517
[71] Benigno Uria, Marc-Alexandre Côté, Karol Gregor, lain Murray, and Hugo Larochelle. Neural autoregressive distribution estimation. The Journal of Machine Learning Research, 17(1):7184, 7220, 2016.
518
519
520
[72] Benigno Uria, lain Murray, and Hugo Larochelle. A deep and tractable density estimator, In International Conference on Machine Learning, pages 467-475. PMLR, 2014.
521
522
[73]
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pages 1747-1756. PMLR, 2016.
523
524
[74] Sean Vasquez and Mike Lewis. Melnet: A generative model for audio in the frequency domain.
arXiv preprint arXiv: 01080,0000000000 2019.
13

---

528
[75] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
527
528
529
[76] Francesco Visin, Kyle Kastner, Kyunghyun Cho, Matteo Matteucci, Aaron Courville, and Yoshua Bengio. Renet: A recurrent neural network based alternative to convolutional networks. arXiv preprini arXiv: N:0Nv05505.00000 2015.
530
532
[77] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
533
535
[78] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu, Seggan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
536
538
[79] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204. 00044/39000 13902, 2022.
$39
540
[80]
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang. and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1097-1100, 2018.
542
Checklist
544
545
1. For all authors...
546
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [TODO][Yes]
548
(b) Did you describe the limitations of your work? [TODO][Yes] Throughout the paper we list downsides to including the mask in the input, as well as the complexity of our inference pipeline
549
$50
551
(c) Did you discuss any potential negative societal impacts of your work? [TODO] Yes] We plan to address this in the supplementary material and appendix due to space limitations
552
553
554
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [es]
556
2. If you are including theoretical results...
557
(a) Did you state the full set of assumptions of all theoretical results? [TODO][[Ys] Our theorems state the relevant conditions
(b) Did you include complete proofs of all theoretical results? [TODO]][es) We also cite relevant prior work on said theorems
560
561
3.
If you ran experiments...
562
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL.? [TODO][No] We will include (likely non-runnable) version of the experimental code in the supplemental material, and work toward a more generally usable release during the review cycle
563
584
585
586
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [TODO][No] To fully disclose this, requires full experimental code
567
566
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [OODO][Yes] Variance in the scoring table
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [TODO][No We discuss the GPU type, but do not do full accounting of resources used. All experiments were trained on commodity hardware experiment run
571
572
873
using single GPUs, for less than 24 hours an
574
4. If you using cxisting assets (c.g., code, data, models) or curating/releasing new assets...
14

---

578
(a) If your work uses existing assets, did you cite the creators? [TODO][ Yes]
(b) Did you mention the license of the assets? [TODO][No] All material is extensively used in prior work
578
578
(c) Did you include any new assets either in the supplemental material or as a URL?
[TODO][No] No new assets
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [TODO][No] No new assets, and assets used are public domain (c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [TODO][No]
584
585
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [TODO][N/A]
587
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [TODO][N/A]
589
590
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [TODO][NA]
15


Reviews:
[{"review_id": "avqO09Ign6e", "reviewer": "Reviewer_y5UN", "summary": "Authors propose a diffusion model that adds a mask as an additional inputs to noisy data. The mask controls which positions in the input should be denoised, in order to provide control for partial inference with controllable positions. Authors experiment on polyphonic music modeling and language modeling.", "questions": "* Please number equations\n* The github repository linked in appendix is empty \n* L110: why introducing m_t as a time-dependent mask while as soon as L116 it’s limited to a constant mask?\n* L160: is there a specific motivation for sampling the difficulty variable at the batch level?\n* L169: why not using the same notation as in the loss equation?\n* L172: schemes -> scheme\n* L185: or -> for\n* L186: trained model -> the trained model\n* L213: papers to use -> papers use\n* L306: underflying -> underlying", "limitations": "Authors adress ethical concerns of music and language modeling in Appendix A, and current inference runtime issues in Appendix F.", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n* The task of controllable generation is interesting, particularly to allow humans to interact with generative models. Allowing users to choose which pieces to improve is a particularly simple yet effective way of providing intuitive control and interaction.\n\n\n# Weaknesses:\n* The writing could be considerably improved. In particular, the introduction is very long and has several flaws such as:\n  * The first paragraph is too vague to be of any use and I recommend authors remove it\n  * The role of the introduction is to give context around the paper’s contribution, why it is useful and why the author's method is worth considering. However, the motivation of authors and the logical premises to their contribution are completely absent of the introduction. The introduction should at least finish with something akin to the first paragraph of the method section, which introduces the contribution in context.\n* The method section is written such that a reader who is not an expert with SUNDAE (a previous work which we cannot expect the average ML researcher to know) will need to go back and forth between papers. While I acknowledge there are several opinions on this, I personally think the paper should be as self-contained as possible, and should provide much more context around the method.\n* The motivation of the authors is not clear to me: unrolling and using a time-dependent mask are presented as key components of the method, yet the model is only used with 2 steps (without ablation) and a constant mask, which is a bit underwhelming and does not support author’s claims that unrolling is useful (claim in L221). Ablation studies would be a minimum.\n* Experiments are performed on three datasets, yet their presentation is minimal. Results on polyphonic music modeling are just presented as a big table with zero comment on the metric, the baselines, and the variants of the model. Similarly, experiments on the news dataset are not commented on, baselines shown in the plot are not even cited. Authors should definitely revamp the experiments section to better highlight the advantages of their method.\n\nIn summary, while the topic is of wide interest for NeurIPS, the presentation of the author’s method and experiments is currently too flawed to be worth publication at NeurIPS.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UY1carN03Y0", "reviewer": "Reviewer_EWiy", "summary": "This paper introduced a new generative modelling method for structured data and demonstrated its effectiveness on symbolic polyphonic music and character level and word-level text modelling. This method also offers the opportunity for subtle control at inference time, which could be useful for downstream human-in-the-loop applications. ", "questions": "In Table 1, sometimes SD-AT seems to be doing better (e.g. for Note and Rhythm and slightly better for Repeated Sequence”, why is SD-AT not bolded for those and instead SMc-T is bolded? I understand that overall SMc-T performed the best, but it’s a bit confusing to me because bolded numbers are not always the best. \n\nDO you think including Figure 3 in the main paper would be necessary? Personally it is very hard for me to imagine the sound merely by looking at the visuals. A project page with generated music would be a much better media to demonstrate this, but given the limitation of the paper format, perhaps those figures could be moved to the appendix. With the space saved, authors could consider including more details about closely related works such as SUNDAE and Coconet. \n", "limitations": "Yes, authors have discussed in detail the ethical implications of their work in appendix section A. ", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nOriginality: This paper seems fairly original to me, but I am not sure since I am not familiar with prior work.\n\nSignificance: This paper seems fairly significant to me as author’s method is overall the best compared to other closely related works (even though I don’t know whether those are SOTA methods on this task). \n\nQuality: This work seems to be of high quality as there seems to be significant engineering effort put into the models to make them work well. I always have respect for such engineering efforts.  \n\nWeaknesses\n \nClarity: To me, some places in this paper feel a lack of context. Since this paper is built on prior works like SUNDAE and Coconet, it’d be better to detail those two important prior works. This could be used to replace sections 1.1 and 1.2, as most people in NeurIPS community should at least know the gist of autoregressive models and non-autoregressive models. However without the context of SUNDAE and Coconet, it is hard to understand how this method differ from those works and the new insights in this work.   \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M2QDXQ3F5Qp", "reviewer": "Reviewer_cQyt", "summary": "A generative sequence modelling based on masked unrolled denoising autoencoders, called SUNMASK, is proposed in this manuscript. The proposed separates the role of masking and correction by conditioning predictions on the mask. Compared to the baseline, the authors claim SUNMASK has a better performance on music modelling. The experiment also proves this contribution.\n", "questions": "I hope the author can clearly claim their contributions, especially to discuss the differences between SUNDAE and proposed method.", "limitations": "Although the experiments have shown the proposed method has great performance, the discussion about the experiment is limited. For example, Fig 6-9 is just shown on page9 without discussion. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Generative modelling is a hot topic, since a lot of application can get the benefit. This idea is very close to SUNDAE proposed by DeepMind this year, which is claimed on page 6. But in Section 3.1, they mainly discuss the difference between the proposed methods and the coconut and the discussion about SUNDAE is ignored. This makes the originality not very clear. Hope the authors can give the discussion deeply and move the related\\closed work to section 2. The researchers would like to know the difference when they get this manuscript since it is too closed to SUNDAE. The quality of this manuscript is good. Their designed experiments can prove their contribution and the manuscript is easy to follow. However, there are some points that could be further improved. For example, the font size of Table 1 is too small and Fig 4-9 could be shown in fig. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ID1qM0TGIjC", "reviewer": "Reviewer_hRGZ", "summary": "This paper proposes SUNMASK, which models discrete sequences without ordering assumptions. The authors emphasized the fine-grained control during the iterative generation. Experiments on generating symbolic music and language model show the efficacy of the proposed SUNMASK.\n", "questions": "Figures4-9 are hard to understand, with very verbose description, maybe would be good to provide a high-level idea first?\n", "limitations": "The authors did not really cover too much about limitations...", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In short, this paper is a combination of SUNDAE and Coconet. The major contribution of the paper seems to be the controlled unrolling procedure which is essentially adding a mask over the SUNDAE loss formulation. The authors include the lots of details for training and implementation which is very helpful for the readers to understand and potentially reproduce. \n\nHowever, this paper has several major shortcomings:\n- The organization of the writing is very loose, making it very confusing for the readers. E.g. sections are named as a mere word like “ trade-offs”, “inference”, which does not offer any additional clues to readers beside its literal meanings.\n- The paper is heavily based on SUNDAE, and it is not much of an improvement compared to SUNDAE if we look at, for instance, figure 2. Since fig 2 is the same task as the original task of SUNDAE, I believe it is a apple-to-apple comparison. \n- The analysis is not very convincing both quantitatively and qualitatively. Quantitatively, I think Bach Mock grading function is biased towards its own task, and it is less generaliziliby than BLEU score on the LM task, which SUNMASK does not outperform existing models. Qualitatively, the cherry-picked “ode to joy” example is good itself, but too specific, maybe it would be good we combine some human subject study?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "207BpZrj73a", "reviewer": "Reviewer_b2LK", "summary": "This paper proposes a new non-autoregressive generative model for discrete sequences which combines a recent work SUNDAE with masking. The mask is provided to the network as additional conditioning - unlike e.g. BERT which masks out the tokens themselves. The advantages of the proposed technique are demonstrated on a polyphonic symbolic music generation benchmark JSB, where it outperformes another non-autoregressive method CoCoNet and the original SUNDAE. Additionally, the paper provides some experiments on text generation on EMNLP'17 News and text-8 datasets. Finally, the paper qualitatively investigates music in-painting with the proposed method. Extensive ablation studies are provided which compare different architectures and different sampling methods, e.g. finding typical sampling to be beneficial.", "questions": "+ Is there an AR baseline in Table 1? Is it BachMock?\n+ figure 1: do correct predictions become unmasked? or do you keep mask constant for the unrolls?\n+ table 1: is SMc-T better than everything else across all metrics or just overall? In the latter case, maybe use bold for the best method on the particular metric. Also, standard deviation is normally denoted as ±.\n+ standard deviations in Table 1 are quite big, any way to reduce them?\n+ L285-286: Negative BLEU/Self-BLEU in generated corpora of 1000 sentences - while it's understandably quite expensive to compute it for 10K samples x many temperatures (as was done in SUNDAE & ScratchGAN papers from DeepMind), keep in mind that the results are usually a bit different between 1K and 10K samples. It is certainly worth adding a disclaimer.\n+ some questions (like the one about the compute) are addressed in checklist but not in the paper. Would it make sense to include those discussions in the main text?\n+ Are there plans to open-source the code? Even non-runnable code is useful for people reproducing prior work.", "limitations": "Limitations are clear from the work itself.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\\+ The whole paper is well-written, illustrations are nice, music samples in the supplementary are helpful.\n\n\\+ Contribution is reasonably novel.\n\n\\+ Experiments are well-executed, with sufficient ablations to understand what changes lead to most improvement.\n\n\\+ Improvement over relevant baselines in polyphonic music generation is significant.\n\n\\- Language results are worse than SUNDAE.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject