SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Glossary of variables and symbols used in this paper. Note that for DYffusion, the reverse process starts at  $n = 0$  and progresses forward (together with the corresponding interpolation timestep,  $i_n$ ) as  $n \rightarrow N$  ending with the final output  $\mathbf{s}^{(N)}$ . For conventional diffusion models, this notation is reversed and the final output of the reverse diffusion process corresponds to  $\mathbf{s}^{(0)}$ .
B Methodology
B.1 DYffusion
Algorithm 3 DYffusion, Two-stage Training, all details
Input: networks  $F_{\theta}, \mathcal{I}_{\phi}$ , norm  $||\cdot||$ , horizon  $h$ , schedule  $[i_n]_{i=0}^{N-1}$ , loss coefficients  $\lambda_1, \lambda_2$ , forecaster conditioning  $\mathbf{c}$   
Stage 1: Train interpolator network,  $\mathcal{I}_{\phi}$   
1. Sample  $i \sim$  Uniform  $(\{1, \dots, h-1\})$   
2. Sample  $\mathbf{x}_t, \mathbf{x}_{t+i}, \mathbf{x}_{t+h} \sim \mathcal{X}$   
3. Optimize  $\min_{\phi} ||\mathcal{I}_{\phi}(\mathbf{x}_t, \mathbf{x}_{t+h}, i) - \mathbf{x}_{t+i}||^2$   
Stage 2: Train forecaster network (diffusion model backbone),  $F_{\theta}$   
1. Freeze  $\mathcal{I}_{\phi}$  and enable inference stochasticity (e.g. dropout)  
2. Sample  $n \sim$  Uniform  $(\{0, \dots, N-1\})$  and  $\mathbf{x}_t, \mathbf{x}_{t+h} \sim \mathcal{X}$   
3.  $\hat{\mathbf{x}}_{t+h}^{(1)} \gets F_{\theta}(\mathcal{I}_{\phi}(\mathbf{x}_t, \mathbf{x}_{t+h}, i_n), i_n, \mathbf{c}(\mathbf{x}_t, n))$   
4.  $\hat{\mathbf{x}}_{t+h}^{(2)} \gets F_{\theta}(\mathcal{I}_{\phi}(\mathbf{x}_t, \hat{\mathbf{x}}_{t+h}^{(1)}, i_{n+1}), i_{n+1}, \mathbf{c}(\mathbf{x}_t, n+1))$  if  $n \leq N-2$  else  $\mathbf{x}_{t+h}$   
5. Optimize  $\min_{\theta} \lambda_1 ||\hat{\mathbf{x}}_{t+h}^{(1)} - \mathbf{x}_{t+h}||^2 + \lambda_2 ||\hat{\mathbf{x}}_{t+h}^{(2)} - \mathbf{x}_{t+h}||^2$
In Alg. 3 we give the full training algorithm for DYffusion. The only differences to the truncated training algorithm in the main text (Alg. 1) are that we also include:
- One-step look-ahead loss term trick,  $\lambda_2\| F_\theta (\mathcal{I}_\phi (\mathbf{x}_t,\hat{\mathbf{x}}_{t + h},i_{n + 1}),i_{n + 1}) - \mathbf{x}_{t + h}\| ^2$  whenever  $n + 1 < N$ . Essentially, the look-ahead loss term simulates one step of the sampling process (Alg. 2) and backpropagates through it, so that the network is trained with an objective that is closer to and partially mimics the iterative sampling process. In all experiments, we use  $\lambda_{1} = \lambda_{2} = \frac{1}{2}$ . We believe that it would be interesting to explore alternative approaches to this loss term, such as fine-tuning the interpolator network in the second-stage too.
- Optional conditioning of the forecaster network on a noised or clean version of the initial conditions,  $\mathbf{x}_t$ . We explore three different approaches: 1) No conditioning at all, i.e.  $\mathbf{c}(\cdot ,\cdot) = \mathbf{nu11}$ ; 2) Clean initial conditions, i.e.  $\mathbf{c}(\mathbf{x}_t,\cdot) = \mathbf{x}_t$ ; 3) Noised initial conditions, where we sample random noise  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  of the same shape of  $\mathbf{x}_t$ , and then define  $\mathbf{c}(\mathbf{x}_t,n) = \frac{n}{N - 1}\mathbf{x}_t + (1 - \frac{n}{N - 1})\epsilon$ . The second variant is analogous to the conditioning of conventional diffusion models, which DYffusion does not necessarily require since the initial conditions indirectly influence the main input of the forecaster net (i.e.  $\hat{\mathbf{x}}_{t + i_n} = \mathcal{I}_{\phi}(\mathbf{x}_t,\mathbf{x}_{t + h},i_n)$ ). The motivation for the third variant stems from trying to give the forecaster net a more explicit conditioning on the initial conditions when the reverse process goes farther away from  $t$  (i.e. as  $n\to N$  and  $\hat{\mathbf{x}}_{t + i_n}\rightarrow \mathbf{x}_{t + h}$ ). An ablation for the choice of  $\mathbf{c}$  is reported in D.5.4.
B.2 Connection to standard diffusion models
Our work builds upon cold diffusion [2], which "paves the way for generalized diffusion models that invert arbitrary processes". The cold sampling algorithm, which we use to sample from DYffusion (see Alg. 2), is a generalization of DDIM sampling for "generalized diffusion models" (see Appendix A.6 of [2]) and a key ingredient to make DYffusion work (see D.5.1). Our proposed forward and reverse processes are specifically designed to fall under this framework. Given this context, it becomes clear that DYffusion is a generative model for forecasting that falls in the category of "generalized diffusion models".
Training objectives. To see the similarity between standard diffusion models and DYffusion in terms of training objectives, we write out the corresponding equations Eq. 1 and Eq. 3. For a standard "generalized" diffusion model, we can rewrite the main objective inside Eq. 1 as follows:
$$
\left\| R _ {\theta} \left(D \left(\mathbf {s} ^ {(0)}, n\right), \mathbf {x} _ {t}, n\right) - \mathbf {s} ^ {(0)} \right\| ^ {2} = \left\| R _ {\theta} \left(\mathbf {s} ^ {(n)}, \mathbf {x} _ {t}, n\right) - \mathbf {s} ^ {(0)} \right\| ^ {2}, \tag {7}
$$
where  $\mathbf{s}^{(0)} = \mathbf{x}_{t + 1:t + h}$ , and  $\mathbf{s}^{(n)}$  is a noisy version of  $\mathbf{s}^{(0)}$  (its level of corruption increases with  $n$ ).
In the context of DYffusion's forecasting network, we can express its training objective (Eq. 3) in the following manner:
$$
\begin{array}{l} \left| \left| F _ {\theta} \left(\mathcal {I} _ {\phi} \left(\mathbf {x} _ {t}, \mathbf {x} _ {t + h}, i _ {n}\right), i _ {n}\right) - \mathbf {x} _ {t + h} \right| \right| ^ {2} = \left| \left| F _ {\theta} \left(\mathcal {I} _ {\phi} \left(\mathbf {x} _ {t}, \mathbf {s} ^ {(N)}, i _ {n}\right), i _ {n}\right) - \mathbf {s} ^ {(N)} \right| \right| ^ {2} (8) \\ = \left\| F _ {\theta} \left(\mathbf {s} ^ {(n)}, i _ {n}\right) - \mathbf {s} ^ {(N)} \right\| ^ {2}, (9) \\ \end{array}
$$
where  $\mathbf{s}^{(N)} = \mathbf{x}_{t + h}$  and  $\mathbf{s}^{(n)}\approx \mathbf{x}_{t + i_n}$  is now a stepped backward in time version of  $\mathbf{s}^{(N)}$ . Note that the diffusion step indexing (superscript  $n$ ) for DYffusion is reversed so that it aligns with the temporal indexing (subscript  $t$ ), such that e.g.  $\mathbf{s}^{(n)}\approx \mathbf{x}_{t + i_n}$  temporally precedes  $\mathbf{s}^{(n + 1)}\approx \mathbf{x}_{t + i_{n + 1}}$ . Accounting for the reversed indexing of the diffusion steps, it is easy to see the equivalence of the denoising network,  $R_{\theta}$ , in a conventional diffusion model, and its forecasting network counterpart,  $F_{\theta}$ , in DYffusion. The reason why DYffusion's forecaster network does not necessarily require the initial conditions,  $\mathbf{x}_t$ , as additional input is because the diffusion state,  $\mathbf{s}^{(n)}$ , depends on them. The central difference is the design and semantic meaning of the diffusion processes accompanying them. We visualize this in the diagrams in Fig. 1.
C Diffusion process as ODE
In this section, we treat  $\mathbf{x}$  as a continuous function of time, i.e.  $\mathbf{x}:\mathbb{R}\to \mathbb{R}^n$ ,  $s\mapsto \mathbf{x}(s)$ . Here  $s$  denotes the time variable (since  $t$  is already used to define the index of temporal data snapshots).
During prediction,  $\mathbf{x}_t$  is given,  $\phi$  is fixed, and  $\mathcal{I}_{\phi}$  only depends on  $\hat{\mathbf{x}}_{t + h}$  and  $i$ . Therefore, we simplify notations by writing  $\mathcal{I}_{\phi}(\mathbf{x}_t,\hat{\mathbf{x}}_{t + h},i) = \mathcal{I}_{\phi ,\mathbf{x}_t}(\hat{\mathbf{x}}_{t + h},i)$ . We will further omit the subscripts  $\phi$  and  $\mathbf{x}_t$ .
C.1 Cold Sampling from the Euler method
In this section, we show that Cold Sampling is an approximation of the Euler method for (5).
The Euler method for integrating  $\mathbf{x}$  is
$$
\mathbf {x} _ {s + \Delta s} = \mathbf {x} _ {s} + \Delta s \frac {d \mathcal {I} \left(F _ {\theta} \left(\mathbf {x} _ {s} , s\right) , s\right)}{d s} \tag {10}
$$
for a small  $\Delta s$ . We do not have access to  $\frac{d\mathcal{I}(F_{\theta}(\mathbf{x}_{s},s),s)}{ds}$ . However, since we know  $F_{\theta}$  and  $\mathcal{I}_{\phi}$ , we can approximate  $\frac{d\mathcal{I}(F_{\theta}(\mathbf{x}_{s},s),s)}{ds}$  by its first-order Taylor expansion around  $s$ :
$$
\Delta s \frac {d \mathcal {I} \left(F _ {\theta} \left(\mathbf {x} _ {s} , s\right) , s\right)}{d s} \approx \mathcal {I} \left(F _ {\theta} \left(\mathbf {x} _ {s + \Delta s}, s + \Delta s\right), s + \Delta s\right) - \mathcal {I} \left(F _ {\theta} \left(\mathbf {x} _ {s}, s\right), s\right) \tag {11}
$$
This step can also be interpreted as evaluating the integral in (6) using the fundamental theorem of calculus. Then the Euler method becomes
$$
\mathbf {x} _ {s + \Delta s} = \mathbf {x} _ {s} + \mathcal {I} \left(F _ {\theta} \left(\mathbf {x} _ {s + \Delta s}, s + \Delta s\right), s + \Delta s\right) - \mathcal {I} \left(F _ {\theta} \left(\mathbf {x} _ {s}, s\right), s\right) \tag {12}
$$
Note that  $\mathbf{x}_{s + \Delta s}$  on the right-hand side is unknown because it is the quantity we want to approximate in this step. A reasonable way to approximate  $F_{\theta}(\mathbf{x}_{s + \Delta s}, s + \Delta s)$  is to replace it by  $F_{\theta}(\mathbf{x}_s, s)$  because they both predict  $\mathbf{x}(h)$  and use nearby points (assuming  $\Delta s$  is small and  $\mathbf{x}$  behaves nicely around  $s$ ). The resulting update,
$$
\mathbf {x} _ {s + \Delta s} = \mathbf {x} _ {s} + \mathcal {I} \left(F _ {\theta} \left(\mathbf {x} _ {s}, s\right), s + \Delta s\right) - \mathcal {I} \left(F _ {\theta} \left(\mathbf {x} _ {s}, s\right), s\right), \tag {13}
$$
is exactly the Cold Sampling algorithm (Alg. 2). By formulating the diffusion process as an ODE, we have provided a new theoretical explanation for Cold Sampling.
Note that we made the approximation that  $F_{\theta}(\mathbf{x}_{s + \Delta s}, s + \Delta s) \approx F_{\theta}(\mathbf{x}_s, s)$  to obtain the update rule (previous equation). The error introduced by this approximation is expected to be larger when  $s$  is small. The intuition is as follows. When  $s$  is small, the distance between  $s$  and  $t + h$  is large, and  $F_{\theta}(\mathbf{x}_s, s)$  has to make a prediction farther ahead. Predicting far into the future is generally harder than predicting the near future. Therefore, the prediction error  $F_{\theta}(\mathbf{x}_s, s) - \mathbf{x}_{t + h}$  is expected to be larger when  $s$  is small. The larger uncertainty may lead to larger difference between  $F_{\theta}(\mathbf{x}_{s + \Delta s}, s + \Delta s)$  and  $F_{\theta}(\mathbf{x}_s, s)$ .
To reduce the error brought by this approximation, it makes sense to sample more densely around the early part of the prediction window.
C.2 Why is cold sampling better than naive sampling?
The cold sampling algorithm (adapted for DYffusion in Alg. 2) was proposed as an improved version over so-called "naive" sampling in the original Cold Sampling paper [2] for generalized diffusion model forward processes. Both
Algorithm 4 Adapted Naive Sampling [2]
1: Input: Initial conditions  $\hat{\mathbf{x}}_t \coloneqq \mathbf{x}_t$ , schedule  $[i_n]_{i=0}^{N-1}$
2: for  $n = 0,1,\ldots ,N - 1$  do
3:  $\hat{\mathbf{x}}_{t + h} \gets F_{\theta}(\hat{\mathbf{x}}_{t + i_n}, i_n)$
4:  $\hat{\mathbf{x}}_{t + i_{n + 1}} = \mathcal{I}_{\phi}\left(\mathbf{x}_t,\hat{\mathbf{x}}_{t + h},i_{n + 1}\right)$  # difference w.r.t. Alg. 2
5: end for
sampling algorithms can be directly used to sample from DYffusion. For completeness, we adapt the simpler, naive sampling algorithm to our notation in Alg. 4, where the only difference to cold sampling (Alg. 2) lies in the missing error-correction terms in line 4 of the algorithms. In our ablation experiments, we find that cold sampling indeed outperforms naive sampling by a large margin (see Naive sampling row in Table 7), especially for the SST dataset where we use auxiliary diffusion steps. We explain by analyzing the discretization errors in the two sampling algorithms. In cold sampling, the discretization error per step is bounded by a term proportional to the step size  $\Delta s$ . Naive sampling does not have this property.
The true value of  $\mathbf{x}$  at  $s + \Delta s$  according to Equation (6) is
$$
\begin{array}{l} \mathbf {x} (s + \Delta s) = \mathbf {x} (s) + \int_ {s} ^ {s + \Delta s} \frac {d \mathcal {I} _ {\phi} (F _ {\theta} (\mathbf {x} , s) , s)}{d s} \\ = \mathbf {x} (s) + \mathcal {I} \left(F _ {\theta} (\mathbf {x} (s + \Delta s), s + \Delta s), s + \Delta s\right) - \mathcal {I} \left(F _ {\theta} (\mathbf {x} (s), s), s\right). \tag {14} \\ \end{array}
$$
Recall from Alg. 2 that given  $\mathbf{x}(s)$ , cold sampling predicts  $\mathbf{x}(s + \Delta s)$  as
$$
\hat {\mathbf {x}} (s + \Delta s) = \mathbf {x} (s) + \mathcal {I} _ {\phi} (\mathbf {x} _ {t - l: t}, F _ {\theta} (\mathbf {x} (s), s), s + \Delta s) - \mathcal {I} _ {\phi} (\mathbf {x} _ {t - l: t}, F _ {\theta} (\mathbf {x} (s), s), s). \tag {15}
$$
The discretization error  $e(\mathbf{x})$  of one step of cold sampling is the difference between the exact and predicted  $\mathbf{x}(s + \Delta s)$ :
$$
\begin{array}{l} e (\mathbf {x}, \Delta s) = \mathbf {x} (s + \Delta s) - \hat {\mathbf {x}} (s + \Delta s) \\ = \mathcal {I} \left(F _ {\theta} (\mathbf {x} (s + \Delta s), s + \Delta s), s + \Delta s\right) - \mathcal {I} \left(F _ {\theta} (\mathbf {x} (s), s), s + \Delta s\right). \tag {16} \\ \end{array}
$$
The following proposition states that  $e(\mathbf{x})$  is bounded by a term proportional to the step size  $\Delta s$ .
Proposition C.1. Assume that  $F_{\theta}(\mathbf{x}(s), s)$  is Lipschitz in  $s$ . Assume also that  $\mathcal{I}_{\phi}(\mathbf{x}_{t + h}, s)$  is Lipschitz in  $\mathbf{x}_{t + h}$ . The norm of the cold sampling discretization error,  $\| e(\mathbf{x}, \Delta s)\|_2$ , is bounded by  $O(\Delta s)$ .
Proof. The proof relied on applying definitions of Lipschitz functions twice. Let  $L_{1}$  be the Lipschitz constant for  $F_{\theta}(\mathbf{x}(s), s)$  in  $s$ . Let  $L_{2}$  be the Lipschitz constant for  $\mathcal{I}(\mathbf{x}, s)$  in  $\mathbf{x}$ . Since  $F_{\theta}(\mathbf{x}(s), s)$  is Lipschitz in  $s$ , we have  $||F_{\theta}(\mathbf{x}(s + \Delta s), s + \Delta s) - F_{\theta}(\mathbf{x}(s), s)||_{2} \leq L_{1}\Delta s$ . Since  $\mathcal{I}(\mathbf{x}, s)$  is Lipschitz in  $\mathbf{x}$ , we have  $||\mathcal{I}(F_{\theta}(\mathbf{x}(s + \Delta s), s + \Delta s), s + \Delta s) - \mathcal{I}(F_{\theta}(\mathbf{x}(s), s), s + \Delta s)||_{2} \leq L_{2}L_{1}\Delta s$ . Therefore  $||e(\mathbf{x})|| \leq L_{2}L_{1}\Delta s$ , which means the discretization error is bounded by a first-order term of the step size.
Under the same Lipschitz assumptions, the discretization error of the naive sampling is not guaranteed to be in the first order of step size. In naive sampling, the predicted  $\mathbf{x}$  at time  $s + \Delta s$  is
$$
\hat {\mathbf {x}} (s + \Delta s) = \mathcal {I} \left(F _ {\theta} (\mathbf {x} (s), s), s + \Delta s\right). \tag {17}
$$
The discretization error of one step of naive sampling is  $\mathbf{x}(s + \Delta s)$ :
$$
\begin{array}{l} e (\mathbf {x}, \Delta s) = \mathbf {x} (s + \Delta s) - \hat {\mathbf {x}} (s + \Delta s) \\ = \mathcal {I} \left(F _ {\theta} (\mathbf {x} (s + \Delta s), s + \Delta s), s + \Delta s\right) - \mathcal {I} \left(F _ {\theta} (\mathbf {x} (s), s), s + \Delta s\right) \\ + \mathbf {x} (s) - \mathcal {I} \left(F _ {\theta} (\mathbf {x} (s), s), s\right). \tag {18} \\ \end{array}
$$
Note that the first two terms are the same as the discretization error in cold sampling. However, the last two terms are not bounded by first-order terms of  $\Delta s$ . Hence, naive sampling can have larger discretization errors.
D.1.1 SST data preprocessing
We create a new sea surface temperatures (SST) dataset based on NOAA OI SST V2 [30], which comes at a daily time scale. These data are available from 1982 to the present at a resolution of  $1/4^{\circ}$  degrees. For training, we use the years 1982-2018, for validation 2019, and for testing 2020. We have preprocessed the NOAA OI SST V2 dataset as follows:
1. First, the globe is divided into  $60 \times 60$  latitude  $\times$  longitude grid tiles,
2. all tiles with less than  $95\%$  of ocean cover are filtered out,
3. standardize the raw SSTs using daily means and standard deviations (computed on the training set only, i.e. 1982-2018),
4. replace continental NaNs with zeroes (after standardization), and
5. we subsample 11 grid tiles (covering mostly the eastern tropical Pacific, as shown in Fig. 5).
Figure 5: Visualization of the SST dataset that we created. It divides the globe into  $60 \times 60$  latitude  $\times$  longitude grid tiles. We only use the subset delineated in red, i.e. boxes 84-89 and 108-112.
Table 5: The hyperparameters used for each dataset. For the learning rates, we sweep over each value and report the best set of runs based on their validation CRPS computed on 50 samples. DYffusion  $k$  refers to the number of artificial diffusion steps used, see Fig. 3. For architectural details, see D.3.
Hyperparameters for each dataset
Hyperparameter	SST	Navier-Stokes	Spring Mesh
Batch size	64	32	64
Accumulate gradient batches	4	2	1
Max. Epochs	50	200	300
Gradient clipping (norm)	1.0	1.0	1.0
Learning rate(s)	7e-4, 3e-4, 5e-5, 1e-5	7e-4, 3e-4	4e-4
Weight decay	1e-5	1e-4	1e-4
AdamW β1	0.9	0.9	0.9
AdamW β2	0.99	0.99	0.99
DYffusion k	25	0	0
D.1.2 Navier-Stokes and spring mesh datasets
We refer to the physical systems benchmark paper for more details regarding the Navier-Stokes and spring mesh benchmark datasets [44]. We follow the same evaluation splits and procedure (only complemented by the probabilistic skill metrics that we employ too). We always use the full training set provided by [44]. We use the Navier-Stokes dataset with four obstacles.
D.2 Implementation Details
The set of hyperparameters that we use for each dataset, such as the learning rate and maximum number of epochs, can be found in Table 5. For all experiments, we use a floating point precision of 16 and do not use a learning rate scheduler. All diffusion models, including DYffusion, are trained with the L1 loss, while all barebone UNet/CNN networks are trained on the L2 loss. We use three different dropout rates for the SST UNet: 1) before the query-key-value projection of each attention layer,  $dr_{at}$ ; 2) After the first sub-block of each ResNet block,  $dr_{bl_1}$ ; 3) After the second sub-block of each ResNet block,  $dr_{bl_2}$ , where the first ResNet sub-block consists of convolution  $\rightarrow$  normalization  $\rightarrow$  time-embedding scale-shift  $\rightarrow$  activation function, and the second sub-block is the same but without the time-embedding scale-shift.
Perturbation baseline We perturb the initial conditions,  $\mathbf{x}_t$ , with small amounts of Gaussian noise  $\epsilon \sim \mathcal{N}(0,\sigma_{\epsilon}\mathbf{I})$ . We found that  $\sigma_{\epsilon}^{*} = 0.05$  gave the lowest CRPS scores among all variances that we tried,  $\sigma_{\epsilon} \in \{0.01, 0.03, 0.05, 0.07, 0.1, 0.15, 0.2\}$ . We note that choosing larger variances results in better SSR scores, but significantly lower CRPS and MSE scores. Inference dropout was disabled for this baseline variant.
Dropout baseline For this baseline, we enable the barebone model's dropout during both training and inference. For the SST dataset, similarly to the interpolator network of DYffusion, we found that using high dropout rates results in better performance. An explanation could be that the SST UNet has more capacity than the other backbone architectures. Concretely, the following SST UNet dropout rates resulted in the best performance:  $dr_{at} = dr_{bl_2} = 0.6$ ,  $dr_{bl_1} = 0.3$ . For Navier-Stokes and spring mesh, there is only one dropout hyperparameter, and the corresponding best model uses 0.2 and 0.05 as dropout rate, respectively (selected from a sweep over {0.05, 0.1, 0.15, 0.2, 0.25, 0.3}).
DDPM We found that the cosine (linear) noise schedule gives better results for the SST (Navier-Stokes) dataset, and always uses the "predict noise" objective. For the SST dataset, the best performing DDPM is trained with 5 diffusion steps, while for Navier-Stokes it is trained with 500 steps. For Navier-Stokes, we found that while a DDPM with 5 or 10 diffusion steps can give good validation scores (or even better ones than the 500-step DDPM), it ends up diverging at test time after a few autoregressive iterations when used to forecast full trajectories.
MCVD [67] We train MCVD with 1000 diffusion steps for all datasets, as we were not able to successfully train it with fewer diffusion steps. We use a linear noise schedule (we found the cosine schedule to produce inferior results) using the "predict noise" objective. Due to the inference runtime
complexity of using 1000 diffusion steps, we only report one MCVD run in our main SST results. Note that in none of the experiments we use the UNet-based diffusion model backbone originally used by MCVD. In preliminary experiments on the SST dataset, however, we found that wrapping MCVD around the SST UNet resulted in slightly improved scores over wrapping it around the original MCVD UNet.
**DYffusion** For the SST dataset, we use 25 artificial diffusion steps (analogous to the schedule in green in Fig. 3), while for the Navier-Stokes and spring mesh datasets, we do not use any, i.e.  $S = [j]_{j=0}^{h-1}$ . Furthermore, we found that the refinement step of Alg. 2 did not improve performance for the SST dataset so we did not use it there, whereas it did improve performance for Navier-Stokes and spring mesh (see D.5.2). As for the choice of the interpolator network,  $\mathcal{I}_{\phi}$ , we conduct a sweep over the dropout rates for each dataset (analogous to the "Dropout" baseline). This is an important hyperparameter since we found that stochasticity in the interpolator is crucial for the overall performance of DYffusion. The interpolator network is selected based on the lowest validation CRPS. For the SST dataset, the selected  $\mathcal{I}_{\phi}$  uses  $dr_{at} = dr_{bl_2} = 0.6$ ,  $dr_{bl_1} = 0$ . The dropout rates for Navier-Stokes and spring mesh are 0.15 and 0.05, respectively. Generally, we found that the optimal amount of dropout for any given dataset strongly correlates between the "Dropout" multi-step forecasting baseline and DYffusion's interpolator network,  $\mathcal{I}_{\phi}$ . Thus, for a new dataset or problem, it is a valid strategy to sweep over the dropout rate for just one of the two model types. Motivated by the intuition that temporal interpolation is a simpler task than forecasting, the channel dimensionality of  $\mathcal{I}_{\phi}$  is only 32 (instead of 64) for the first downsampling block of the SST UNet. To feed the two inputs  $\mathbf{x}_t$  and  $\mathbf{x}_{t+h}$  into the interpolator, we simply concatenate these snapshots along the channel dimensions, effectively doubling the number of input channels.
Evaluation All experiments are usually run over three random seeds when compute constraints allow. We use a 50-member ensemble for all reported results unless noted otherwise to compute the CRPS, mean squared error (MSE), and spread-skill ratio (SSR). The MSE is computed on the ensemble mean prediction. The SSR is defined as the ratio of the square root of the ensemble variance to the corresponding ensemble RMSE. It serves as a measure of the reliability of the ensemble, where values smaller than 1 indicate underdispersion (i.e. the probabilistic forecast is overconfident in its forecasts), and larger values overdispersion [16, 18]. We use the implementation in the xskillscore<sup>4</sup> Python package to compute the CRPS of the ensemble forecasts. The CRPS penalizes both the absolute skill as well as the sharpness of the ensemble of forecasts. It rewards small spread (i.e. sharpness) if the forecast is accurate, and measures the difference between the forecasted and observed cumulative distribution function (CDF). For a deterministic forecast, the CRPS reduces to the mean absolute error. For CRPS and MSE lower is better, for SSR closer to 1 is better.
D.3 Neural Architecture Details