SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\langle \mathcal {A} \sum_ {i = 0} ^ {N - 1} X _ {t} [ i / N ] \xi^ {i}, \zeta^ {k} \right\rangle = \left\langle \sum_ {i = 0} ^ {N - 1} X _ {t} [ i / N ] \mathcal {A} \frac {1}{N} \sum_ {l = 0} ^ {N - 1} e ^ {l} \exp \left(- j 2 \pi l \frac {i}{N}\right), \zeta^ {k} \right\rangle = \\ \left\langle \sum_ {i = 0} ^ {N - 1} X _ {t} [ i / N ] \frac {1}{N} \sum_ {l = 0} ^ {N - 1} b ^ {l} e ^ {l} \exp \left(- j 2 \pi l \frac {i}{N}\right), \zeta^ {k} \right\rangle = \\ \sum_ {i = 0} ^ {N - 1} X _ {t} [ i / N ] \frac {1}{N} \sum_ {l = 0} ^ {N - 1} b ^ {l} \exp (j 2 \pi l ^ {k} / N) \exp (- j 2 \pi l ^ {i} / N) = \\ \sum_ {l = 0} ^ {N - 1} b ^ {l} \exp (j 2 \pi l ^ {k} / N) \mathfrak {F} \left(X _ {t} [ ^ {i} / N ]\right) ^ {l} = \\ \Im \left(b ^ {l} \mathfrak {F} (X _ {t} [ ^ {i} / N ]) ^ {l}\right) ^ {i}. \\ \end{array}
$$
The noise term  $\mathrm{d}W_{t}\left[{^{k}} / N\right]$  is approximated as:
$$
\mathrm {d} W _ {t} \left[ ^ {k} / N \right] = \langle \mathrm {d} W _ {t}, \zeta^ {k} \rangle = \langle \sum_ {i = 0} ^ {\infty} \mathrm {d} W _ {t} ^ {i} e ^ {i}, \zeta^ {k} \rangle = \sum_ {i = 0} ^ {\infty} \mathrm {d} W _ {t} ^ {i} \exp \left(j 2 \pi i \frac {k}{N}\right) \simeq \mathfrak {F} (\mathrm {d} W _ {t} ^ {i}) ^ {k},
$$
where we are truncating the sum. The score term has expression:
$$
\begin{array}{l} s _ {\boldsymbol {\theta}} \left(\sum_ {i} X _ {t} [ i / N ] \xi^ {i}, t\right) = - (\mathcal {S} (t)) ^ {- 1} \left(\sum_ {i} X _ {t} [ i / N ] \xi^ {i} - \exp (t \mathcal {A}) n \left(g \left(X _ {t} [ i / N ]\right), t, \boldsymbol {\theta}\right)\right) = \\ - \sum_ {k} \frac {\overbrace {\sum_ {i} X _ {t} [ i / N ] \left\langle \xi^ {i} , (e ^ {k}) ^ {\dagger} \right\rangle} ^ {= \mathfrak {F} \left(X _ {t} [ i / N ]\right) \stackrel {\mathrm {d e f}} {=} C _ {t} ^ {k}} - \exp \left(b ^ {k} t\right) \left\langle n \left(g \left(X _ {t} [ i / N ]\right), t, \boldsymbol {\theta}\right), (e ^ {k}) ^ {\dagger} \right\rangle}{s ^ {k}} e ^ {k} = \\ - \sum_ {k} \frac {C _ {t} ^ {k} - \exp (b ^ {k} t) \langle n (g (X _ {t} [ i / N ]) , t , \boldsymbol {\theta}), \exp (- j 2 \pi k p) \rangle}{s ^ {k}} e ^ {k} \simeq \\ - \sum_ {k} \frac {C _ {t} ^ {k} - \exp (b ^ {k} t) \left(N ^ {- 1} \sum_ {r} n (g (X _ {t} [ i / N ]) , t , \boldsymbol {\theta}) \left[ \frac {r}{N} \right] , \exp (- j 2 \pi k \frac {r}{N})\right)}{s ^ {k}} e ^ {k}, \\ \end{array}
$$
where the approximation is due to the substitution of explicit scalar product with the discretized version through  $\mathfrak{F}$ . When evaluated on the grid of interest:
$$
\begin{array}{l} s _ {\boldsymbol {\theta}} \left(\sum_ {i} X _ {t} [ ^ {i} / N ] \xi^ {i}, t\right) [ ^ {i} / N ] = \\ - \sum_ {k} \frac {\left(C _ {t} ^ {k} - \exp (b ^ {k} t) \left(N ^ {- 1} \sum_ {r} n (g \left(X _ {t} [ i / N ]\right) , t , \boldsymbol {\theta}) \left[ \frac {r}{N} \right] , \exp (- j 2 \pi k \frac {r}{N})\right)\right)}{s ^ {k}} \exp (j 2 \pi k ^ {i} / N) = \\ - \Im \left(\frac {\mathfrak {F} \left(X _ {t} [ i / N ]\right) - \exp \left(b ^ {k} t\right) \mathfrak {F} \left(n \left(g \left(X _ {t} [ i / N ]\right) , t , \boldsymbol {\theta}\right) [ i / N ]\right)}{s ^ {k}}\right). \\ \end{array}
$$
The value of  $\tilde{\gamma}_{\theta}$ , Equation (33) and the expression of the backward process, Equation (34), are obtained similarly, considering the above results.
E Implementation Details and Additional Experiments
In all experiments we use the complex Fourier basis for the Hilbert spaces, indexed by  $k$ . This extends to the 2-dimensional case what we described in Appendix D.1. As stated in the main paper, our practical implementation sets  $f = 0$ : then, we only need to specify the value for the parameters  $b^{k}, r^{k}$ . In our implementation we consider an extended class of SDEs that include time-varying multiplying coefficients in front of the drift and diffusion terms, as done for example in the Variance Preserving SDE originally described by Song & Ermon (2020). This can be simply interpreted as the time-rescaled version of autonomous SDEs.
E.1 Architectural details
INR-based score network. In our implementation, we use the original INR architecture (Sitzmann et al., 2020). For the specific denoising task we consider in our model, we extend the input of the network architecture to include the corrupted version of the input sample and the diffusion time  $t$ , in addition to the spatial coordinates. We emphasize that our architectural design is simple, and does not require self-attention mechanisms (Song & Ermon, 2020). The non-linearity we use in our network is a Gabor wavelet activation function (Saragadam et al., 2023). Furthermore, we found beneficial the inclusion of skip connections.
As stated in the main paper, we consider the modulation approach to INRs. In particular, we implement the meta-learning scheme described by Dupont et al. (2022b); Finn et al. (2017). The outer loop is dedicated to learning the base parameters of the model, while the inner loop focuses on refining the base parameters for each input sample. In the outer loop, the optimization algorithm is AdaBelief (Zhuang et al., 2020), sweeping the learning rate over 1e-4, 1e-5, 1e-6. We found the use of a cosine warm-up schedule to be beneficial for avoiding training instabilities and convergence to sub-optimal solutions. The inner loop is implemented by using three steps of stochastic gradient descent (SGD).
Transformer-based score network. In our experiments with the Transformer architecture for score modeling, we employed the UViT backbone Bao et al. (2022). This backbone processes all inputs, be they temporal or noisy image patches, as tokens. Rather than utilizing UViT's default learned positional embeddings, we adapted it to integrate 2D sinusoidal positional encodings. For the noisy input images, patch embeddings transform them into a sequence of tokens. Notably, we chose a patch size of 1 to fully harness the functional properties of our framework. Time embeddings are computed based on the time and then concatenated with the image tokens.
Our chosen transformer architecture comprises 7 layers, with each layer composed of a self-attention mechanism with 8 attention heads and a feedforward layer. Furthermore, we use long skip connections between the shallower and deeper layers, as outlined by Bao et al. (2022).
For optimization during our training, we utilized the AdamWLoschilov & Hutter (2017) algorithm with a weight decay of 0.03. We employed a cosine warm-up schedule for the learning rate, which ends at a value of 2e-4.
E.2 Additional results
E.2.1 A Toy example.
We here present some qualitative examples on a synthetic data-set of functions  $\in L([-1,1])$ , and therefore consider the settings described in Appendix D. The Quadratic data is generated as in (Phillips et al., 2022), i.e.  $X_0[p] = qp^2 + \epsilon$ , where  $\epsilon \sim \mathcal{N}(0,0.1)$  and  $q$  is a binary random variable that take values  $\{-1,1\}$  with equal probability. Concerning the design of the forward SDE, we select  $b^{k} = \min(\sqrt{k}, 10)$  and  $r^{k} = k^{-2}$  (thus satisfying Corollary 1). The real data is generated considering a grid of 100 equally spaced points. We can see in Figure 3 some qualitative results. On the left real (red) and generated through FDP (blue) samples show good agreement. Center and right plots depict some example of diffused samples for times 0.2 and 1.0 respectively.
E.2.2 MNIST data-set
We evaluate our approach on a simple data-set, using MNIST  $32 \times 32$  (LeCun et al., 2010). In this experiment, we compare our method against the baseline score-based diffusion models from Song et al. (2021), which we take from the official code repository https://github.com/yang-song/ score_sde. The baseline implements the score network using a U-NET with self-attention and skip connections, as indicated by current best practices, which amounts to  $O(10^{8})$  parameters.
Instead, our method uses a score-network/INR implemented as a simple MLP with 8 layers and 128 neurons in each layer. The activation function is a sinusoidal non-linearity (Sitzmann et al., 2020). Our model counts  $O(10^{5})$  parameters. We consider an SDE with parameters  $r^{k,m} = \frac{176}{k^2 + m^2 + 2},$  and  $b^{k,m} = \min((k^2 + m^2 + 0.3)^{-1} + \left(\frac{r^{k,m}}{33}\right)^{\frac{1}{4}}, 3.6)$ . These values have been determined empirically by observing the power spectral density of the data-set, to ensure a well-behaved Signal to Noise ratio evolution throughout the diffusion process for all frequency components.
4Strictly speaking, the sum of the series  $r^{k,m}$  is not convergent. We experimented changing the decay to ensure convergence, but we observed no numerical difference with the settings we the setting we used. It is an interesting avenue for future work to study if this approximation has an impact for higher-resolution data-sets.
In Figure 4 we report un-curated samples generated according to our FDP. In Figure 5 we present instead various "intermediate" noisy versions of the training data, to illustrate the kind of noise we use to train the score network, and the output of the denoising INR. We also report the Fréchet Inception Distance (FID) score computed using 16k samples (lower is better). For the baseline we obtain  $\mathrm{FID} = 0.05$ , whereas for the proposed method we obtain  $\mathrm{FID} = 0.43$ . Although the FID score is in favor of the baseline, we believe that our results – obtained with a simple MLP – are very promising, as further corroborated by experiments on a more complex dataset, which we show next.
Super resolution. We demonstrate how an INR trained at a  $32 \times 32$  resolution on MNIST can be seamlessly applied to increase the resolution of the generated data points. Given that the INR establishes a mapping between a grid and its corresponding value, we initiated the diffusion process from a grid at a  $32 \times 32$  resolution. The diffusion process continued until the final step, where we used the last learned parameters to extrapolate outputs at a higher resolution. A significant advantage of using INRs is the ability to conduct the resource-intensive sampling at a lower resolution and then effortlessly transition between resolutions with just a single forward call to the model. Figure 6, shows our results at different resolutions.
E.2.3 CELEBA data-set
For the CELEBA data-set we considered the same SDE as for the MNIST experiment. Results reported in the main paper have been obtained using a numerical integration scheme of a variant of the predictor-corrector scheme of (Song & Ermon, 2020), which we adapted to the SDEs we consider in our work. In Figure 8 and Figure 9 we report additional un-curated samples obtained with the INR and Transformer respectively. We proceed to describe further experiments in the following section.
Conditional generation. In the following, we consider three use-cases for conditional generation: in-painting, de-blurring, and colorization, which we describe next. All these additional experiments
were completed using the same architecture and configuration of the unconditional generation described above.
In-painting. We perform in-painting experiments by adopting the same approach described by Song & Ermon (2020), and report results in Figure 10. Original images (left-column of Figure 10) are masked (center-column of Figure 10), where we set the value corresponding to the missing pixels to 0. The right column of Figure 10 shows the results of the in-painting scheme where, qualitatively, it is possible to observe that the conditional generation is able to fill the missing portion of the images while maintaining good semantic coherence.
De-blurring. Our FDPs are naturally suited for the de-blurring use-case, as shown in Figure 12. In this experiment, we take the original images (left column of Figure 12) and filter them with a low pass filter (center column of Figure 12). The de-blurring scheme is implemented as the in-painting approach described by Song & Ermon (2020), where the only difference is that the masking at each update is applied in the frequency domain. The right column of Figure 12 shows that our technique gracefully recovers missing details and is capable of producing high quality images conditioned on the distorted inputs.
Colorization. In this use-case, we adapt the approach from (Song & Ermon, 2020) to our setting. Figure 11 depicts qualitative results of the colorization experiment, confirming the flexibility of the proposed scheme.
E.2.4 SPOKEN DIGIT data-set
To demonstrate the versatility of our framework, we conducted preliminary experiments on an audio dataset, specifically the Spoken Digit Dataset. This dataset comprises recordings of spoken digits, stored as wav files at an  $8\mathrm{kHz}$  sampling rate, with each recording trimmed to minimize silence at the beginning and end. The dataset features five speakers who have contributed to a total of 2,500 recordings, providing 50 recordings of each digit per speaker. The dataset is publicly available on the TensorFlow Dataset Catalog. As preprocessing, each sample was either padded with zeros or truncated to a maximum duration of one second. Subsequently, the data was normalized using the effectsnormalize function from the pydub library, and each sample was scaled to a range of  $[-1, +1]$  by dividing it by the dataset's maximum intensity. For the audio experiments, we fed the raw audio data directly into the transformer model, without converting it to log mel spectrograms, which is a common practice in audio processing tasks. The transformer model was configured with a patch size of 2, an embedding size of 512, 13 layers, and 8 heads. We employed the AdamW optimizer with a weight decay of 0.03 and a cosine warmup schedule that decays at a value of 1e-5. The preliminary examination of the audio generated by our model reveals its ability to effectively generate spoken digits. Upon listening to the model's generated samples, we were able to recognize the digits accurately, showcasing the model's potential in audio generation tasks. Figure 7 provides a comparative analysis of the waveforms generated by our model against real examples from the dataset.