SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Generator hyperparameters The generator is a conditional Neural SDE. Stochastic integration was again taken in the Itô sense and we used the Euler method. The noise size was set to  $w = 8$ , the size of the hidden state was taken  $y = 16$ . The MLPs governing the vector fields were the same as per the rBergomi and multidimensional unconditional examples, except we increased the width of the layers in the neural networks to 64 neurons. We conditionalized the input paths via the truncated log-signature of order 5. In order to estimate the batched loss, we need to specify the size of the conditional distribution  $\mathbb{P}_{X^{\theta}}(\cdot |x)$  output by the conditional generator, which we set to 32 paths. Finally, we applied the time normalisation and lead-lag transformations to the input paths  $x$  before taking their truncated log-signature.
Discriminator hyperparameters We trained with respect to  $\phi_{\mathrm{sig}}$ . Again we lifted paths via the RBF kernel and chose the smoothing parameter  $\sigma = 1$ . We used a dyadic refinement level of 1 for the PDE solver associated to  $k_{\mathrm{sig}}$ . All paths had the time normalisation transformation applied to them before having the loss evaluated.
Training hyperparameters We set the batch size equal to 128 and trained the conditional generator for 10000 steps. We set the learning rate  $\eta_{G} = 2\times 10^{-6}$  and used the Adam optimisation algorithm in PyTorch. No learning rate annelears were used.
Results Results are presented in the body of the paper.
B.5 Simulation of limit order books
The Neural SPDE model introduced in [SLG22] extends Neural SDEs to model spatiotemporal dynamics by parametrising the differential operator, drift and diffusion of SPDEs of the type
$$
d Y _ {t} = \left(\mathcal {L} Y _ {t} + \mu \left(Y _ {t}\right)\right) d t + \sigma \left(Y _ {t}\right) d W _ {t} \tag {12}
$$
where both  $\mu$  and  $\sigma$  are local operators acting on the function  $Y_{t}$  that is,  $\mu (Y_t)(x)$  and  $\sigma (Y_t)(x)$  only depend on  $Y_{t}(x)$ . Moreover, it is assumed that  $\mathcal{L}$  is a linear differential operator generating a semigroup  $e^{t\mathcal{L}}$  which can be written as a convolution with a kernel  $\kappa_{t}$ .
Let  $D \subset \mathbb{R}^d$  be a bounded domain. Let  $W:[0,T] \to L^{2}(D,\mathbb{R}^{d_{w}})$  be a Wiener process and  $a$  an  $L^2 (D,\mathbb{R}^{d_a})$ -valued Gaussian random variable. The values  $d_w, d_a \in \mathbb{N}$  are hyperparameters describing the size of the noise. A Neural SPDE is a model of the form
$$
Y _ {0} (x) = \ell_ {\theta} (a (x)), \quad Y _ {t} = \mathcal {K} _ {t} * Y _ {0} + \int_ {0} ^ {t} \mathcal {K} _ {t - s} * (\mu_ {\theta} (Y _ {s}) + \sigma_ {\theta} (Y _ {s}) \dot {W} _ {s} ^ {\epsilon}) d s, \quad X _ {t} ^ {\theta} (x) = \pi_ {\theta} (Y _ {t} (x)).
$$
for  $t \in [0, T]$  and  $x \in D$  where  $Y: [0, T] \to L^2(D, \mathbb{R}^{d_y})$  is the mild solution, if it exists to the SPDE in Equation (12) with regularised driving noise  $W^\epsilon$  and where  $*$  denotes the convolution in space with the kernel  $\mathcal{K}_t: D \times \bar{D} \to \mathbb{R}^{d_y \times d_y}$  (see [SLG22] for more details). Similarly to the Neural SDE model,
$$
\ell_ {\theta}: \mathbb {R} ^ {d _ {a}} \to \mathbb {R} ^ {d _ {y}}, \quad \mu_ {\theta}: \mathbb {R} ^ {d _ {y}} \to \mathbb {R} ^ {d _ {y}}, \quad \sigma_ {\theta}: \mathbb {R} ^ {d _ {y}} \to \mathbb {R} ^ {d _ {y} \times d _ {w}}, \quad \pi_ {\theta}: \mathbb {R} ^ {d _ {y}} \to \mathbb {R} ^ {d _ {x}}
$$
are feedforward neural networks. Imposing globally Lipschitz conditions (by using ReLU or tanh activation functions in the neural networks  $\mu_{\theta}$  and  $\sigma_{\theta}$ ) ensures the existence and uniqueness of the mild solution  $Y$ . Finally, we note that in [SLG22], the authors propose two distinct algorithms to evaluate the Neural SPDE model based on two different parameterisations of the kernel  $\mathcal{K}$ .
Next, we provide more details on how we trained such a Neural SPDE model to generate Limit Order Book (LOB) dynamics  $\left[\mathrm{GPW}^{+}13\right]$ . The increasing availability of LOB data has instigated a significant interest in the development of statistical models for LOB dynamics. In recent years, new models based on SPDEs have been proposed to accurately describe and analyse these complex dynamics [HKN20, CM21].
Data processing and hyperparameters We used real LOB data from the NASDAQ public exchange made publicly available in  $\left[\mathrm{NMK}^{+}18\right]$  which consists of about 4 million timestamped events over 10 consecutive trading days with  $L = 10$  price levels on each side (bid and ask) of the LOB. Three versions of this dataset are provided, each normalised using a different technique. We used the data normalised with z-scores and split the LOB trace into sub-traces of length  $T = 30$ .
Generator hyperparameters The generator is a Neural SPDE driven by a cylindrical Wiener process  $W$  with  $d_w = 2$ . The vector fields  $\mu_{\theta}$  and  $\sigma_{\theta}$  are taken to be single layer perceptrons with  $d_y \in \{16,32\}$  followed by batch normalization and tanh activation function. Thus we obtain
$$
\mu_ {\theta} \in \mathcal {N N} (d _ {h}, d _ {h}, \text {B a t c h N o r m , t a n h}), \quad \sigma_ {\theta} \in \mathcal {N N} (d _ {h}, d _ {h} \times d _ {w}, \text {B a t c h N o r m , t a n h})
$$
We used the second evaluation method proposed in [SLG22, Section 3.3] with 4 Picard's iterations and maximum number of frequency modes in  $\{10,20\}$  in the spatial direction and fixed to 20 in the temporal direction. Instead of sampling the initial condition  $a$  from a  $L^2 (D,\mathbb{R}^{d_a})$ -valued Gaussian, we simply used the samples from  $X_0^{\mathrm{true}}$ , in which case  $d_{a} = 1$ .
Discriminator hyperparameters We integrated in time the output trajectories from the generator, as we observed this yielded more stable kernel scores. We mapped the path state values into  $\mathcal{H}_{\kappa}$  where  $\kappa$  denotes a SE-T kernel on  $L^2 (D)$  with  $D = [0,1]$ , that is, a kernel defined for all  $f,g\in L^{2}(D)$  by  $\kappa (f,g) = e^{-\frac{1}{2\sigma^2}\| T(f) - T(g)\|_{\mathcal{V}}^2}$  where  $T:L^{2}(D)\to \mathcal{V}$  is a Borel measurable, continuous and injective map. We considered three SE-T kernels respectively termed ID, SQR and CEXP:
1. (ID) SE-T kernel with  $T: L^2(D) \to L^2(D)$  defined for all  $f \in L^2(D)$  by  $T(f) = f$
2. (SQR) SE-T kernel with  $T: L^2(D) \to L^2(D) \oplus L^2(D)$  defined by  $T(f) = (f, f^2)$ .
3. (CEXP) SE-T kernel with  $T: L^2(D) \to L^2(D)$  defined by  $T(f) = C_{F,l}(f)$  where  $C_{F,l}$  is the covariance operator associated to the kernel  $k_{F,l}$  defined for all  $x, x' \in D$  by
$$
k _ {F, l} (x, x ^ {\prime}) = e ^ {- \frac {1}{2 l ^ {2}} (x - x ^ {\prime}) ^ {2}} \sum_ {n = 0} ^ {F - 1} \cos (2 \pi n (x - x ^ {\prime}))
$$
For ID and SQR, we used  $\sigma \in \{1, 10\}$ , and for CEXP we used  $(\sigma, l, F) \in \{(1, 1, 5), (10, 10, 5)\}$ . We then used a dyadic order of 1 to compute the signature kernel.
Training hyperparameters We set the learning rate of the generator  $\eta_G$  to be  $1\times 10^{-3}$  and trained it for a maximum number of 1500 epochs. We used a batch size of 64 due to memory constraints and the Adam optimizer with the default parameters of PyTorch.