SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We here investigate sample diversity and novelty of our unconditional generative models. We measure uniqueness by computing the chemical graph corresponding to each generated sample and measure what proportion of the 10000 produced samples have a unique chemical graph amongst this set of 10000 as is done in [8]. We show our results in Table 7 and find our TDDM method to have slightly lower levels of uniqueness when compared to the fixed dimension diffusion model baseline.
Table 5: The 10 conditioning tasks used for evaluation. The number of each atom type required for the task is given in columns 2 - 5 whilst the average number of atoms in molecules that meet this condition in the training dataset is given in the 6th column.
Task	Carbon	Nitrogen	Oxygen	Fluorine	Mean Number of Atoms
1	4	1	2	1	11.9
2	4	3	1	1	13.0
3	5	2	1	1	13.9
4	6	0	1	1	14.6
5	5	3	1	0	16.0
6	6	3	0	0	17.2
7	6	1	2	0	17.7
8	7	1	1	0	19.1
9	8	1	0	0	19.9
10	8	0	1	0	21.0
Figure 8: Distribution of molecule sizes for each conditioning task. Tasks  $1 - 5$  are shown left to right in the top row and tasks  $6 - 10$  are shown left to right in the bottom row. We show the unconditional size distribution from the dataset in blue vertical/horizontal hashing, the size distribution of our conditionally generated samples in orange diagonal hashing and finally the size distribution for molecules in the training dataset that match the conditions of each task (the ground truth size distribution) in green no hashing.
Measuring novelty on generative models trained on the QM9 dataset is challenging because the QM9 dataset contains an exhaustive enumeration of all molecules that satisfy certain predefined constraints [46], [8]. Therefore, if a novel molecule is produced it means the generative model has failed to capture some of the physical properties of the dataset and indeed it is found in [8] that during training, as the model improved, novelty decreased. Novelty is therefore not typically included in evaluating molecular diffusion models. For completeness, we include the novelty scores in Table 7 as a comparison to the results presented in [8] Appendix C. We find that our samples are closer to the statistics of the training dataset whilst still producing 'novel' samples at a consistent rate.
D.2 Video
D.2.1 Dataset
We used the  $\mathrm{VP^2}$  benchmark, which consists of 35000 videos, each 35 frames long. The videos are evenly divided among seven tasks, namely: push {red, green, blue} button, open {slide, drawer}, push {upright block, flat block} off table. The 5000 videos for each task were collected using a scripted task-specific policy operating in the RoboDesk environment [40]. They sample an action vector at every step during data generation by adding i.i.d. Gaussian
Figure 9: Samples generated by our model when conditional diffusion guidance is applied. Each row represents one task with task 1 at the top, down to task 10 at the bottom. For each task, 10 samples are shown in each row.
noise to each dimension of the action vector output by the scripted policy. For each task, they sample 2500 videos with noise standard deviation 0.1 and 2500 videos with standard deviation 0.2. We filter out the lower-quality trajectories sampled with noise standard deviation 0.2, and so use only the 17500 videos (2500) per task with noise standard deviation 0.1. We convert these videos to  $32 \times 32$  resolution and then, so that the data we train on has varying lengths, we create each training example by sampling a length  $l$  from a uniform distribution over  $\{2, \dots, 35\}$  and then taking a random  $l$ -frame subset of the video.
D.2.2 Forward Process
The video domain differs from molecules in two important ways. The first is that videos cannot be reasonably treated as a permutation-invariant set. This is because the order of the frames matters. Secondly, generating a full new component for the molecules with a single pass autoregressive network is feasible, however, a component for the videos is a full frame which is challenging for a single pass autoregressive network to generate. We design our forward process to overcome these challenges.
Figure 10: Interpolations showing a sequence of generations for linearly increasing polarizability from  $39\mathrm{Bohr}^3$  to  $66\mathrm{Bohr}^3$  with fixed random noise. Each row shows an individual interpolation with  $\mathrm{Bohr}^3$  increasing from left to right.
Table 6: Ablation of when to set the forward rate to 0 on the conditional molecule generation task. We report dimension error as the average Hellinger distance between the generated and ground truth conditional dimension distributions as well as average sample quality metrics. Metrics are reported after 620k training iterations.
Method	Dimension Error	% Atom stable	% Molecule Stable	% Valid
→λt<0.03T=0	0.227±0.16	91.5±3.7	56.5±9.8	72.0±11
→λt<0.1T=0	0.162±0.071	92.4±2.8	53.9±12	72.7±9.6
→λt<0.3T=0	0.266±0.11	92.0±3.2	53.5±13	66.6±12
Table 7: Uniqueness and novelty metrics on unconditional molecule generation. We produce 10000 samples for each method and measure validity using RDKit. Uniquenesss is judged as whether the chemical graph is unique amongst the 10000 produced samples. Amongst the valid and unique molecules, we then find the percentage that have a chemical graph not present in the training dataset.
Method	% Valid	% Valid and Unique	Percentage of Valid and Unique Molecules that are Novel
FDDM [8]	91.9	90.7	65.7
TDDM (ours) →	92.3	89.9	53.6
TDDM, const λt	86.7	84.4	56.9
TDDM, λt<0.9T=0	89.4	86.1	51.3
TDDM w/o Prop. 3	87.1	85.9	63.3
We define our forward process to delete frames in a random order. This means that during generation, frames can be generated in any order in the reverse process, enabling more conditioning tasks since we can always ensure that whichever frames we want to condition on are added first. Further, we use a non-isotropic noise schedule by adding noise just to the frame that is about to be deleted. Once it is deleted, we then start noisig the next randomly chosen frame. This is so that, in the backward direction, when a new frame is added, it is simply Gaussian noise. Then the score network will fully denoise that new frame before the next new frame is added. We now specify exactly how our forward process is constructed.
We enable random-order deletion by applying an initial shuffling operation occurring at time  $t = 0$ . Before this operation, we represent the video  $\mathbf{x}$  as an ordered sequence of frames,  $\mathbf{x}_0 = [\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_{n_0}]$ . During shuffling, we sample a random permutation  $\pi$  of the integers  $1, \dots, n_0$ . Then the frames are kept in the same order, but annotated with an index variable so that we have  $\mathbf{x}_{0^+} = [(x_{0^+}^{(1)}, \pi(1)), (x_{0^+}^{(2)}, \pi(2)), \dots, (x_{0^+}^{(n_0)}, \pi(n_0))]$ .
We will run the forward process from  $t = 0$  to  $t = 100N$ . We will set the forward rate such we delete down from  $n_t$  to  $n_t - 1$  at time  $(N - n_t + 1)100$ . This is achieved heuristically by setting
$$
\overrightarrow {\lambda} _ {t} (n _ {t}) = \left\{ \begin{array}{l l} 0 & \text {f o r} t <   (N - n _ {t} + 1) 1 0 0, \\ \infty & \text {f o r} t \geq (N - n _ {t} + 1) 1 0 0. \end{array} \right.
$$
We can see that at time  $t = (N - n_t + 1)100$  we will quickly delete down from  $n_t$  to  $n_t - 1$  at which point  $\vec{\lambda}_t(n_t)$  will become 0 thus stopping deletion until the process arrives at the next multiple of 100 in time. When we hit a deletion event, we delete the frame from  $\mathbf{X}_t$  that has the current highest index variable  $\pi(n)$ . In other words
$$
K ^ {\mathrm {d e l}} (i | \mathbf {X} _ {t}) = \left\{ \begin{array}{l l} 1 & \text {f o r} n _ {t} = \mathbf {x} _ {t} ^ {(i)} [ 2 ], \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
where we use  $\mathbf{x}_t^{(i)}[2]$  to refer to the shuffle index variable for the  $i$ th current frame in  $\mathbf{x}_t$ .
We now provide an example progression of the forward deletion process. Assume we have  $n_0 = 4$ ,  $N = 5$  and sample a permutation such that  $\pi(1) = 3$ ,  $\pi(2) = 2$ ,  $\pi(3) = 4$ , and  $\pi(4) = 1$ . Initially
the state is augmented to include the shuffle index. Then the forward process progresses from  $t = 0$  to  $t = 500$  with components being deleted in descending order of the shuffle index
$$
\begin{array}{l} \mathbf {x} _ {0 ^ {+}} = \left[ \left(\mathbf {x} _ {t} ^ {(1)}, 3\right), \left(\mathbf {x} _ {t} ^ {(2)}, 2\right), \left(\mathbf {x} _ {t} ^ {(3)}, 4\right), \left(\mathbf {x} _ {t} ^ {(4)}, 1\right) \right] \\ \mathbf {x} _ {1 0 0 ^ {+}} = \left[ \left(\mathbf {x} _ {t} ^ {(1)}, 3\right), \left(\mathbf {x} _ {t} ^ {(2)}, 2\right), \left(\mathbf {x} _ {t} ^ {(3)}, 4\right), \left(\mathbf {x} _ {t} ^ {(4)}, 1\right) \right] \\ \mathbf {x} _ {2 0 0 ^ {+}} = \left[ \left(\mathbf {x} _ {t} ^ {(1)}, 3\right), \left(\mathbf {x} _ {t} ^ {(2)}, 2\right), \left(\mathbf {x} _ {t} ^ {(4)}, 1\right) \right] \\ \mathbf {x} _ {3 0 0 ^ {+}} = \left[ \left(\mathbf {x} _ {t} ^ {(2)}, 2\right), \left(\mathbf {x} _ {t} ^ {(4)}, 1\right) \right] \\ \mathbf {x} _ {4 0 0 ^ {+}} = [ (\mathbf {x} _ {t} ^ {(4)}, 1) ] \\ \end{array}
$$
In this example, due to the random permutation sampled, the final video frame remained after all others had been deleted. Note that the order of frames is preserved as we delete frames in the forward process although the spacing between them can change as we delete frames in the middle.
Between jumps, we use a noising process to add noise to frames. The noising process is non-isotropic in that it adds noise to different frames at different rates such that the a frame is noised only in the time window immediately preceding its deletion. For component  $i \in [1, \dots, n_t]$ , we set the forward noising process such that  $p_{t|0}(\mathbf{x}_t^{(i)}|\mathbf{x}_0^{(i)}, M_t) = \mathcal{N}(\mathbf{x}_t^{(i)}; \mathbf{x}_0^{(i)}, \sigma_t(\mathbf{x}_t^{(i)})^2)$  where  $\mathbf{x}_0^{(i)}$  is the clean frame corresponding to  $\mathbf{x}_t^{(i)}$  as given by the mask  $M_t$  and  $\sigma_t(\mathbf{x}_t^{(i)})$  follows
$$
\sigma_ {t} (\mathbf {x} _ {t} ^ {(i)}) = \left\{ \begin{array}{l l} 0 & \text {f o r} t <   (N - \mathbf {x} _ {t} ^ {(i)} [ 2 ]) 1 0 0, \\ 1 0 0 & \text {f o r} t > (N - \mathbf {x} _ {t} ^ {(i)} [ 2 ]) 1 0 0, \\ t - (N - \mathbf {x} _ {t} ^ {(i)} [ 2 ]) 1 0 0 & \text {f o r} (N - \mathbf {x} _ {t} ^ {(i)} [ 2 ]) 1 0 0 \leq t \leq (N - \mathbf {x} _ {t} ^ {(i)} [ 2 ] + 1) 1 0 0 \end{array} \right.
$$
where we again use  $\mathbf{x}_t^{(i)}[2]$  for the shuffle index of component  $i$ . This is the VE-SDE from [3] applied to each frame in turn. We note that we only add noise to the state values on not the shuffle index itself. The SDE parameters that result in the VE-SDE are  $\vec{\mathbf{b}}_t = 0$  and  $\vec{g}_t = \sqrt{2t - 2(N - \mathbf{x}_t^{(i)}[2])100}$ .
D.2.3 Sampling the Backward Process
When  $t$  is not at a multiple of 100, the forward process is purely adding Gaussian noise, and so the reverse process is also purely operating on the continuous dimensions. We use the Heun sampler proposed by [16] to update the continuous dimensions in this case, and also a variation of their discretisation of  $t$  - specifically to update from e.g.  $t = 600$  to  $t = 500$ , we use their discretization of  $t$  as if the maximum value was 100 and then offset all values by 500.
To invert the dimension deletion process, we can use Proposition 3 to derive our reverse dimension generation process. We re-write our parameterized  $\overleftarrow{\lambda}_t^\theta$  using Proposition 3 as
$$
\stackrel {\leftarrow} {\lambda} _ {t} ^ {\theta} (\mathbf {X} _ {t}) = \vec {\lambda} _ {t} (n _ {t} + 1) \mathbb {E} _ {p _ {0 | t} ^ {\theta} (n _ {0} | \mathbf {X} _ {t})} \left[ \frac {p _ {t | 0} (n _ {t} + 1 | n _ {0})}{p _ {t | 0} (n _ {t} | n _ {0})} \right]
$$
At each time multiple of 100 in the backward process, we will have an opportunity to add a component. At this time point, we estimate the expectation with a single sample  $n_0 \sim p_{0|t}^\theta(n_0|\mathbf{X}_t)$ . If  $n_0 > n_t$  then  $\overleftarrow{\lambda}_t^\theta(\mathbf{X}_t) = \infty$ . The new component will then be added at which point  $\overleftarrow{\lambda}_t^\theta(\mathbf{X}_t)$  becomes 0 for the remainder of this block of time due to  $n_t$  becoming  $n_t + 1$ . If  $n_0 = n_t$  then  $\overleftarrow{\lambda}_t^\theta(\mathbf{X}_t) = 0$  and no new component is added.  $\overleftarrow{\lambda}_t^\theta(\mathbf{X}_t)$  will continue to be 0 for the remainder of the backward process once an opportunity to add a component is not used.
When a new frame is added, we use  $A_{t}^{\theta}(\mathbf{y}^{\mathrm{add}}, i|\mathbf{X}_{t})$  to decide where the frame is added and its initial value. Since when we delete a frame it is fully noised,  $A_{t}^{\theta}(\mathbf{y}^{\mathrm{add}}, i|\mathbf{X}_{t})$  can simply predict Gaussian noise for the new frame  $\mathbf{y}^{\mathrm{add}}$ . However,  $A_{t}^{\theta}(\mathbf{y}^{\mathrm{add}}, i|\mathbf{X}_{t})$  will still learn to predict a suitable location  $i$  to place the new frame such that backward process is the reversal of the forward.
We give an example simulation from the backward generative process in Figure 11.
Figure 11: An example simulation of the backward generative process conditioned on the first and last frame. Note how the process first adds a new frame and then fully denoises it before adding the next frame. Since the first and last frame are very similar, the process produces a short video.
D.2.4 Network Architecture
Our video diffusion network architecture is based on the U-net used by [42], which takes as input the index of each frame within the video, and uses the differences between these indices to control the interactions between frames via an attention mechanism. Since, during generation, we do not know the final position of each frame within the  $\mathbf{x}_0$ , we instead pass in its position within the ordered sequence  $\mathbf{x}_t$ .
One further difference is that, since we are performing non-isotropic diffusion, the standard deviation of the added noise will differ between frames. We adapt to this by performing preconditioning, and inputting the timestep embedding, separately for each frame  $\mathbf{x}_t^{(i)}$  based on  $\sigma_t(\mathbf{x}_t^{(i)})$  instead of basing them on the global diffusion timestep  $t$ . Our timestep embedding and pre- and post-conditioning of network inputs/outputs are as suggested by [16], other than being done on a per-frame basis. The architecture from [42] with these changes applied then gives us our score network  $s_t^\theta$ .
While it would be possible to train a single network that estimates the score and all quantities needed for modelling jumps, we chose to train two separate networks in order to factorize our exploration of the design space. These were the score network  $s_t^\theta$ , and the rate and index prediction network modeling  $p_{0|t}^\theta (n_0|\mathbf{X}_t)$  and  $A_{t}^{\theta}(i|\mathbf{X}_{t})$ . The rate and index prediction network is similar to the first half of the score network, in that it uses all U-net blocks up to and including the middle one. We then flatten the  $512\times 4\times 4$  hidden state for each frame after this block such that, for an  $n_t$  frame input, we obtain a  $n_t\times 8192$  hidden state. These are fed through a 1D convolution with kernel size 2 and zero-padding of size 1 on each end, reducing the hidden state to  $(n_{t} + 1)\times 128$ , which is in turn fed
through a ReLU activation function. This hidden state is then fed into three separate heads. One head maps it to the parameters of  $A_{t}^{\theta}(i|\mathbf{X}_{t})$  via a 1D convolution of kernel size 3. The output of size  $(n_{t} + 1)$  is fed through a softmax to provide the categorical distribution  $A_{t}^{\theta}(i|\mathbf{X}_{t})$ . The second head averages the hidden state over the "frame" dimension, producing a 128-dimensional vector. This is fed through a single linear layer and a softmax to parameterize  $p_{0|t}^{\theta}(n_0|\mathbf{X}_t)$ . Finally, the third head consists of a 1D convolution of kernel size 3 with 35 output channels. The  $(n_{t} + 1)\times 35$  output is fed through a softmax to parameterize distributions over the number of frames that were deleted from  $\mathbf{X}_0$  which came before the first in  $\mathbf{x}_t$ , the number of frames from  $\mathbf{X}_0$  which were deleted between each pair of frames in  $\mathbf{x}_t$ , and the number deleted after the last frame in  $\mathbf{x}_t$ . We do not use this head at inference-time but found that including it improved the performance of the other heads by helping the network learn better representations.
For a final performance improvement, we note that under our forward process there is only ever one "noised" frame in  $\mathbf{x}_t$ , while there are sometimes many clean frames. Since the cost of running our architecture scales with the number of frames, running it on many clean frames may significantly increase the cost while providing little improvement to performance. We therefore only feed into the architecture the "noised" frame, the two closest "clean" frames before it, and the two closest "clean" frames after it. See our released source code for the full implementation of this architecture.
D.2.5 Training
To sample  $t$  during training, we adapt the log-normal distribution suggested by [16] in the context of isotropic diffusion over a single image. To apply it to our non-isotropic video diffusion, we first sample which frames have been deleted, which exist with no noise, and which have had noise added, by sampling the timestep from a uniform distribution and simulating our proposed forward process. We then simply change the noise standard deviation for the noisy frame, replacing it with a sample from the log-normal distribution. The normal distribution underlying our log-normal has mean  $-0.6$  and standard deviation 1.8. This can be interpreted as sampling the timestep from a mixture of log-normal distributions,  $\frac{1}{N}\sum_{i=0}^{N-1}\mathcal{LN}(t - 100i; -0.6, 1.8^2)$ . Here, the mixture index  $i$  can be interpreted as controlling the number of deleted frames.
We use the same loss weighting as [16] but, similarly to our use of preconditioning, compute the weighting separately for each frame  $\mathbf{x}_t^{(i)}$  as a function of  $\sigma_t(\mathbf{x}_t^{(i)})$  to account for the non-isotropic noise.
D.2.6 Perceptual Quality Metrics
We now verify that our reverse process does not have any degradation in quality during the generation as more dimensions are added. We generate 10000 videos and throw away the 278 that were sampled to have only two frames. We then compute the FID score for individual frames in each of the remaining 9722 videos. We group together the scores for all the first frames to be generated in the reverse process and then for the second frame to be generated and so on. We show our results in Table 8. We find that when a frame is inserted has no apparent effect on perceptual quality and conclude that there is no overall degradation in quality as our sampling process progresses. We note that the absolute value of these FID scores may not be meaningful due to the RoboDesk dataset being far out of distribution for the Inception network used to calculate FID scores. We can visually confirm good sample quality from Figure 5.
Table 8: FID for video frames grouped by when they were inserted during sampling.
1st	2nd	3rd	3rd last	2nd last	last
34.2	34.9	34.7	34.2	34.1	34.4
In this work, we presented a general method for performing generative modeling on datasets of varying dimensionality. We have not focused on applications and instead present a generic method.
Along with other generic methods for generative modeling, we must consider the potential negative social impacts that these models can cause when inappropriately used. As generative modeling capabilities increase, it becomes simpler to generate fake content which can be used to spread misinformation. In addition to this, generative models are becoming embedded into larger systems that then have real effects on society. There will be biases present within the generations created by the model which in turn can reinforce these biases when the model's outputs are used within wider systems. In order to mitigate these harms, applications of generative models to real world problems must be accompanied with studies into their biases and potential ways they can be misused. Further, public releases of models must be accompanied with model cards [47] explaining the biases, limitations and intended uses of the model.