Efficient Training of Neural Stochastic Differential Equations by Matching Finite Dimensional Distributions

# Efficient Neural SDE By Matching Finite Dimensional Distributions

# Introduction to Stochastic Differential Equations (SDEs)

- •
Ordinary Differential Equations (ODEs): Describe the deterministic evolution of systems over time.
$\frac{dx}{dt}=f(x,t)$
- •
But there are systems with randomness:
–
Finance: stock prices.
–
Physics: Describing particle motion in fluids .
–
Biology: Population dynamics under random environmental changes.

# From ODE to SDE

- •
ODE:
$\frac{dy}{dt}=f(t,y)$

- •
Adding Gaussian Noise:
$\frac{dy}{dt}=f(t,y)+\epsilon\eta_{t}$
where $\eta_{t}$ is Gaussian noise with mean 0 and variance 1, and $\epsilon$ is a small parameter.
- •
Limiting Behavior:
$dX_{t}=\mu(t,X_{t})dt+\sigma(t,X_{t})dB_{t}$
As $\epsilon\to 0$, the ODE with Gaussian noise converges to an SDE with a Brownian Motion $B_{t}$.
- •
Key Idea: SDEs = ODEs + Random Noise
The term $dB_{t}$ represents a Brownian motion, a process with independent Gaussian increment.

# Neural Stochastic Differential Equations (Neural SDE)

- •
SDE is a continuous process defined on [0, T]:
$dX_{t}=\mu(t,X_{t})dt+\sigma(t,X_{t})dB_{t}$
- •
Definition of Neural SDE: SDEs where the drift $\mu$ and diffusion $\sigma$ terms are parameterized by neural networks.
- •
Formulation:
$dZ_{t}^{{\color\theta}}=\mu^{{\color\theta}}(t,Z_{t}^{{\color\theta}})\,dt+
\sigma^{{\color\theta}}(t,Z_{t}^{{\color\theta}})\,dB_{t}$
- •
Applications: continuous time-series generative modeling.

# Challenges in Training Neural SDE

- •
GAN-based Training (kidger21b):
–
Use a GAN to distinguish $Z_{t}^{true}$ from $Z_{t}^{{\color\theta}}$.
–
Instability and mode collapse.
- •
Signature Kernels (issa2023sigker):
–
Training by minimizing a distance measure.
–
Evaluating the distance measure requires solving a linear PDE.
–
Time complexity: $O(T^{2})$, where $T$ is the number of discretization steps of the PDE solver.
- •
Truncated signature kernel (Bonnier24truncated):
–
Extract features from the sequence with the truncated signature transform.
–
Space complexity: $O(d^{N})$ where $d$ is the dimension of feature space and $N$ is the truncation size.
- •
None has been tested on discretizations with length $>100$.
- •
Our contribution: an efficient training objectives.

# Technical Background: Markov Process

- •
Markov Process: a process only depends on the most recent state. $i.e.\mathbb{P}(X_{s}|X_{u},\forall u\leq t)=\mathbb{P}(X_{s}|X_{t})$.
- •
$e.g.$ For a discrete process, $\mathbb{P}(X_{n+1}|X_{n},\dots,X_{1})=\mathbb{P}(X_{n+1}|X_{n})$.

# Technical Background: Scoring Rule

- •
A scoring rule $s(P,\omega)$ maps a predictive distribution $P$ and a sample $w$ to a real number.
- •
$s$ is strictly proper if $S(P,Q)=\mathbb{E}_{\omega\sim Q}[s(P,\omega)]$ satisfies $S(P,Q)\leq S(Q,Q)$ and the equality is taken $\iff$ $P=Q$.
- •
Example Gneiting07ProperScore: let $k$ be the rbf kernel,
$\displaystyle s(P,y)$
$\displaystyle=\frac{1}{2}\mathbb{E}_{X,X^{\prime}\sim P}k(X,X^{\prime})-
\mathbb{E}_{X}k(X,y)$
$\displaystyle S(P,Q)$
$\displaystyle=\frac{1}{2}\mathbb{E}_{X,X^{\prime}\sim P}k(X,X^{\prime})-
\mathbb{E}_{X,Y\sim Q}k(X,Y)$
- •
If $P^{\theta}$ is parameterized by generative neural networks $\theta$ and $Q$ is the data distribution we’d like to learn, maximizing $S(P^{\theta},Q)$ will yield a $P^{\theta}=Q$.
- •
Scoring rule for finite-dimensional distributions is well-studied but not for random processes.

# Main Theorem

Statement in English: we can convert a strictly proper scoring rule $s$ for finite-dimensional distributions to a strictly proper scoring rule for contiuous Markov processes.

Statement in Math: Let markov processes $X,Y$ on $[0,T]$ take values in $\mathbb{R}^{d}$. Let $s$ be any strictly proper scoring rule defined on $\mathbb{R}^{2d}$. Define the scoring rule for continuous Markov processes

$$ $\bar{s}(\mathbb{P}_{X},y)=\mathbb{E}_{(t_{1},t_{2})\sim Uniform([0,T]^{2})}s({ (X_{t_{1}},X_{t_{2}})},(y_{t_{1}},y_{t_{2}})),$ $$

###### Theorem 1 .

For any $\mathbb{R}^{d}$-valued Markov processes $X,Y$ with laws $\mathbb{P}_{X},\mathbb{P}_{Y}$, respectively, $\bar{S}(\mathbb{P}_{X},\mathbb{P}_{Y})\leq\bar{S}(\mathbb{P}_{Y},\mathbb{P}_{Y})$ with equality achieved only if $\mathbb{P}_{X}=\mathbb{P}_{Y}$.

###### Theorem 1 .

# Example:

Let $k:\mathbb{R}^{2d}\times\mathbb{R}^{2d}\to\mathbb{R}$ be the RBF kernel, $X^{\theta}$ be the neural SDE, and $Y$ be the data process, then optimizing

$$ $\displaystyle\bar{S}(\mathbb{P}_{X^{\theta}},\mathbb{P}_{Y})=$ $\displaystyle\mathbb{E}_{(t_{1},t_{2})\sim U([0,T]^{2})}\big{[}\frac{1}{2} \mathbb{E}_{X,X^{\prime}}k([X_{t_{1}},X_{t_{2}}],[X^{\prime}_{t_{1}},X^{\prime }_{t_{2}}])$ $\displaystyle-\mathbb{E}_{X,Y}k([X_{t_{1}},X_{t_{2}}],[Y_{t_{1}},Y_{t_{2}}]) \big{]}$ $$

over $\theta$ leads to $\mathbb{P}_{X^{\theta}}=\mathbb{P}_{Y}$, where $[\cdot,\cdot]$ is the concatenation of random vectors.

# Experiments: Forex

- •
We test the model on some financial datasets and models.
- •
We use the default hyperparameters in issa2023sigker for all models and haven’t done any tuning bias towards our model.
- •
Along each dimension and for marginals at different times, we report the Kolmogorov-Smirnov test scores at a 5%-confidence level, averaged across different batches, as well as the type-I error, where the null hypothesis states that the two marginal distributions are equal.

**Table 1: Average KS test scores and chance of rejecting the null hypothesis at 5%-significance level on marginals for different currency pairs, trained on paths evenly sampled at 64 timestamps.**
| Dim | Model | t=6 | t=19 | t=32 | t=44 | t=57 |
| --- | --- | --- | --- | --- | --- | --- |
| EUR/USD | Signature Kernel | 0.1224, 8.3% | 0.1394, 19.1% | 0.1402, 19.9% | 0.1342, 15.9% | 0.1300, 14.0% |
| Truncated Signature | 0.2206, 94.8% | 0.2721, 99.4% | 0.3037, 99.6% | 0.3224, 99.7% | 0.3434, 99.9% |  |
| SDE-GAN | 0.1700, 45.9% | 0.2002, 78.9% | 0.1982, 80.1% | 0.2185, 88.3% | 0.2921, 99.0% |  |
| FDM | 0.1293, 13.7% | 0.1120, 6.6% | 0.1098, 5.8% | 0.1123, 6.6% | 0.1133, 6.5% |  |
| USD/JPY | Signature Kernel | 0.1141, 4.6% | 0.1239, 8.0% | 0.1276, 10.8% | 0.1268, 11.3% | 0.1264, 12.1% |
| Truncated Signature | 0.2061, 86.3% | 0.2466, 98.9% | 0.2791, 99.3% | 0.3005, 99.7% | 0.3215, 99.9% |  |
| SDE-GAN | 0.1432, 19.7% | 0.1647, 40.5% | 0.1787, 56.5% | 0.1850, 62.2% | 0.1924, 69.0% |  |
| FDM | 0.1201, 10.4% | 0.1108, 5.5% | 0.1090, 5.0% | 0.1100, 5.7% | 0.1093, 5.4% |  |

# Experiments: Forex

**Table 2: Average KS test scores and chance of rejecting the null hypothesis at 5%-significance level on marginals for different currency pairs, trained on paths evenly sampled at 256 timestamps.**
| Dim | Model | $t=25$ | $t=76$ | $t=128$ | $t=179$ | $t=230$ |
| --- | --- | --- | --- | --- | --- | --- |
| EUR/USD | Signature Kernel | 0.5351, 100.0% | 0.5345, 100.0% | 0.5359, 100.0% | 0.5457, 100.0% | 0.5395, 100.0% |
| Truncated Signature | 0.1374, 18.9% | 0.1844, 67.1% | 0.2520, 99.6% | 0.2900, 100.0% | 0.3184, 100.0% |  |
| SDE-GAN | 0.1343, 21.6% | 0.4109, 100.0% | 0.5685, 100.0% | 0.5478, 100.0% | 0.3379, 99.9% |  |
| FDM (ours) | 0.1360, 23.0% | 0.1118, 5.7% | 0.1230, 12.7% | 0.1322, 17.8% | 0.1413, 26.6% |  |
| USD/JPY | Signature Kernel | 0.5346, 100.0% | 0.5339, 100.0% | 0.5350, 100.0% | 0.5376, 100.0% | 0.5405, 100.0% |
| Truncated Signature | 0.1143, 7.1% | 0.1515, 28.3% | 0.1994, 82.8% | 0.2316, 97.9% | 0.2417, 99.2% |  |
| SDE-GAN | 0.2014, 72.6% | 0.3340, 99.9% | 0.4069, 100.0% | 0.4047, 100.0% | 0.3378, 100.0% |  |
| FDM (ours) | 0.1236, 13.8% | 0.1124, 6.3% | 0.1178, 6.9% | 0.1216, 9.0% | 0.1147, 6.2% |  |

# Experiments: Forex

**Table 3: Average KS test scores and chance of rejecting the null hypothesis at 5%-significance level on marginals for different currency pairs (EUR/USD and USD/JPY), trained on paths evenly sampled at 1024 timestamps. A thread limit error is encountered during the training of the Signature Kernel (issa2023sigker).**
| Dim | Model | $t=102$ | $t=307$ | $t=512$ | $t=716$ | $t=921$ |
| --- | --- | --- | --- | --- | --- | --- |
| EUR/USD | Signature Kernel | - | - | - | - | - |
| Truncated Signature | 0.4763, 100.0% | 0.7184, 100.0% | 0.9931, 100.0% | 0.9959, 100.0% | 0.8872, 100.0% |  |
| SDE-GAN | 0.2795, 98.4% | 0.8181, 100.0% | 0.9627, 100.0% | 0.8464, 100.0% | 0.8049, 100.0% |  |
| FDM (ours) | 0.1173, 11.1% | 0.1167, 9.0% | 0.1384, 25.1% | 0.1525, 36.2% | 0.1909, 66.5% |  |
| USD/JPY | Signature Kernel | - | - | - | - | - |
| Truncated Signature | 0.7661, 100.0% | 0.7433, 100.0% | 0.6700, 100.0% | 0.9982, 100.0% | 1.0000, 100.0% |  |
| SDE-GAN | 0.5277, 100.0% | 0.2913, 100.0% | 0.3886, 100.0% | 0.5298, 100.0% | 0.6550, 100.0% |  |
| FDM (ours) | 0.1379, 20.9% | 0.1241, 14.3% | 0.1501, 32.1% | 0.1990, 74.9% | 0.2603, 97.9% |  |

# Experiments: Rough Bergomi

- •
Rough Bergomi model capture market volatility but is hard to simulate.
- •
We generate multi-dimensional paths using existing implementation of Rough Bergomi model and build a generative model upon it.

**Table 4: Average KS test scores and chance of rejecting the null hypothesis at 5%-significance level on marginals across all dimensions, trained on paths evenly sampled at 64 timestamps from a 16-dimension rough Bergomi model.**
| Model | $t=6$ | $t=19$ | $t=32$ | $t=44$ | $t=57$ |
| --- | --- | --- | --- | --- | --- |
| Signature Kernel | 0.1122, 6.6% | 0.1181, 7.8% | 0.1239, 10.8% | 0.1320, 16.3% | 0.1444, 25.5% |
| Truncated Signature | 0.4497, 100.0% | 0.4578, 100.0% | 0.4616, 100.0% | 0.4612, 100.0% | 0.4597, 100.0% |
| SDE-GAN | 0.3081, 99.8% | 0.3737, 99.4% | 0.3926, 99.5% | 0.4063, 99.6% | 0.4295, 99.7% |
| FDM | 0.1130, 7.2% | 0.1155, 7.8% | 0.1185, 8.8% | 0.1243, 11.8% | 0.1306, 15.8% |

# Experiments: Rough Bergomi

**Table 5: Average KS test scores and chance of rejecting the null hypothesis at 5%-significance level on marginals across all dimensions, trained on paths evenly sampled at 64 timestamps from a 32-dimension rough Bergomi model. Truncated Signature runs out of GPU memory.**
| Model | t=6 | t=19 | t=32 | t=44 | t=57 |
| --- | --- | --- | --- | --- | --- |
| Signature Kernel | 0.1200, 11.1% | 0.1372, 18.5% | 0.1486, 26.5% | 0.1573, 35.3% | 0.1675, 45.2% |
| Truncated Signature | - | - | - | - | - |
| SDE-GAN | 0.2840, 99.8% | 0.2875, 99.7% | 0.2976, 99.8% | 0.3114, 99.9% | 0.3262, 100.0% |
| FDM | 0.1170, 9.1% | 0.1194, 10.2% | 0.1217, 11.4% | 0.1243, 13.0% | 0.1276, 15.4% |

# Experiments: Gas Prices

- •
The latest $20\%$ data in time is reversed for test to verify if the model can generalize into future.

**Table 6: Average KS test scores and chance of rejecting the null hypothesis at 5%-significance level on marginals of oil prices, trained on paths evenly sampled at 64 timestamps. ”BRENT”, ”DIESEL”, ”GAS”, and ”LIGHT” stand for US Brent crude oil, gas oil, natural gas, and US light crude oil, respectively.**
| Dim | Model | t=6 | t=19 | t=32 | t=44 | t=57 |
| --- | --- | --- | --- | --- | --- | --- |
| BRENT | Signature Kernel | 0.1448, 21.8% | 0.1450, 22.1% | 0.1415, 19.9% | 0.1396, 18.3% | 0.1300, 13.1% |
| Truncated Signature | 0.2100, 88.7% | 0.2231, 94.8% | 0.2342, 97.1% | 0.2538, 98.8% | 0.2697, 99.4% |  |
| SDE-GAN | 0.4925, 100.0% | 0.8832, 100.0% | 0.9837, 100.0% | 0.9975, 100.0% | 1.0000, 100.0% |  |
| FDM | 0.1290, 14.1% | 0.1292, 13.8% | 0.1355, 17.6% | 0.1347, 17.0% | 0.1341, 16.2% |  |
| DIESEL | Signature Kernel | 0.1409, 17.6% | 0.1546, 29.7% | 0.1578, 32.5% | 0.1581, 32.6% | 0.1601, 35.4% |
| Truncated Signature | 0.1812, 61.8% | 0.2059, 86.7% | 0.2225, 94.9% | 0.2343, 97.5% | 0.2616, 99.0% |  |
| SDE-GAN | 0.1690, 44.9% | 0.2332, 89.3% | 0.3192, 99.4% | 0.4222, 100.0% | 0.5233, 100.0% |  |
| FDM | 0.1243, 11.3% | 0.1172, 7.6% | 0.1173, 8.5% | 0.1224, 9.9% | 0.1203, 9.7% |  |
| GAS | Signature Kernel | 0.1775, 56.3% | 0.1896, 69.9% | 0.1845, 61.5% | 0.1797, 56.7% | 0.1682, 44.2% |
| Truncated Signature | 0.2747, 99.6% | 0.3125, 99.7% | 0.3375, 99.8% | 0.3699, 99.9% | 0.3832, 99.9% |  |
| SDE-GAN | 0.2627, 98.9% | 0.2726, 99.2% | 0.2814, 99.4% | 0.3942, 99.9% | 0.5649, 100.0% |  |
| FDM | 0.1157, 6.4% | 0.1230, 10.1% | 0.1298, 15.2% | 0.1338, 19.3% | 0.1351, 20.2% |  |
| LIGHT | Signature Kernel | 0.1608, 35.9% | 0.1560, 31.4% | 0.1461, 23.6% | 0.1384, 17.4% | 0.1237, 9.7% |
| Truncated Signature | 0.2472, 98.8% | 0.2656, 99.4% | 0.2787, 99.4% | 0.2984, 99.6% | 0.3153, 99.8% |  |
| SDE-GAN | 0.1991, 81.4% | 0.2180, 89.7% | 0.3701, 99.6% | 0.5798, 100.0% | 0.7610, 100.0% |  |
| FDM | 0.1250, 11.5% | 0.1273, 12.9% | 0.1377, 18.9% | 0.1401, 20.8% | 0.1447, 24.0% |  |

# Experiments: Bonds

- •
The latest $20\%$ data in time is reversed for test to verify if the model can generalize into future.

**Table 7: Average KS test scores and chance of rejecting the null hypothesis at 5%-significance level on marginals of bonds, trained on paths evenly sampled at 64 timestamps. ”BUND”, ”UKGILT”, and ”USTBOND” stand for Euro Bund, UK Long Gilt, and US T-BOND, respectively.**
| Dim | Model | t=6 | t=19 | t=32 | t=44 | t=57 |
| --- | --- | --- | --- | --- | --- | --- |
| BUND | Signature Kernel | 0.1365, 17.0% | 0.1559, 30.6% | 0.1691, 45.9% | 0.1794, 59.0% | 0.1780, 58.1% |
| Truncated Signature | 0.2160, 91.7% | 0.2519, 99.0% | 0.2856, 99.5% | 0.3112, 99.9% | 0.3313, 99.9% |  |
| SDE-GAN | 0.2905, 98.9% | 0.4420, 100.0% | 0.5458, 100.0% | 0.6184, 100.0% | 0.6946, 100.0% |  |
| FDM | 0.1105, 6.3% | 0.1121, 5.4% | 0.1234, 7.7% | 0.1319, 12.8% | 0.1297, 12.5% |  |
| UKGILT | Signature Kernel | 0.1134, 7.4% | 0.1138, 5.8% | 0.1223, 8.2% | 0.1274, 10.9% | 0.1282, 11.6% |
| Truncated Signature | 0.1641, 40.0% | 0.2113, 89.9% | 0.2561, 99.0% | 0.2860, 99.6% | 0.3108, 99.7% |  |
| SDE-GAN | 0.2825, 97.6% | 0.6705, 100.0% | 0.9487, 100.0% | 0.9928, 100.0% | 0.9969, 100.0% |  |
| FDM | 0.1285, 13.0% | 0.1124, 6.3% | 0.1080, 4.9% | 0.1086, 5.2% | 0.1076, 5.2% |  |
| USTBOND | Signature Kernel | 0.1340, 15.3% | 0.1310, 10.8% | 0.1416, 18.3% | 0.1462, 21.7% | 0.1499, 25.9% |
| Truncated Signature | 0.2531, 99.0% | 0.2744, 99.6% | 0.3117, 99.8% | 0.3374, 99.9% | 0.3579, 100.0% |  |
| SDE-GAN | 0.4624, 100.0% | 0.8672, 100.0% | 0.9741, 100.0% | 0.9990, 100.0% | 1.0000, 100.0% |  |
| FDM | 0.1282, 12.3% | 0.1196, 8.8% | 0.1129, 5.7% | 0.1134, 5.7% | 0.1113, 5.1% |  |

# Experiments: Efficiency

**Table 8: Training time of different methods on forex data with different lengths in terms of hours.**
| Method | 64 Timestamps | 256 Timestamps | 1024 Timestamps |
| --- | --- | --- | --- |
| Signature Kernel | 0.66 | 7.80 | thread limit error |
| Truncated Signature | 0.31 | 1.34 | 5.61 |
| SDE-GAN | 0.64 | 4.21 | 20.72 |
| FDM | 0.27 | 1.21 | 5.43 |

**Table 9: Training time of Rough Bergomi model with different data dimensions in terms of hours.**
| Method | 16 Dim | 32 Dim |
| --- | --- | --- |
| SDE-GAN | 1.41 | 1.58 |
| FDM | 0.40 | 0.54 |
| Signature Kernel | 4.11 | 6.74 |
| Truncated Signature | 6.86 | - |

# Limitations and Future Work

- •
Limitation: Our method only works for continuous Markov processes.
- •
Future Work: Experiment on a couple of more datasets, compare against the recent functional diffusion model.
- •
Future Extensions: Non-Markov processes ($e.g.$ path-dependent SDE). Non-continuous processes ($e.g.$ abilene network data).

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: In Theorem 2, does the result hold for any scoring rule $s$, or does $s$ also need to be strictly proper? Could you clarify if there are specific conditions on $s$ required for Theorem 2 to apply?
soundness: 2
strengths: This paper addresses an important problem: the high computational complexity (quadratic in time steps) associated with training Neural SDEs using scoring rules. The authors propose a reduced complexity method, aiming for linear complexity to enhance performance. They also provide a theoretically grounded approach in designing a new scoring rule. However, despite this strong motivation, the results are somewhat unconvincing due to certain weaknesses noted below.
summary: This paper proposes a new method called Finite Dimensional Matching (FDM) for training Neural SDEs by identifying a class of strictly proper scoring rules for comparing continuous Markov processes. Using this scoring rule, they claim to reduce the training complexity from quadratic in discretization timesteps to linear.
weaknesses: This paper has several notable weaknesses. While the authors propose a reduced complexity approach for training Neural SDEs, they do not adequately explain key concepts, making the paper difficult to follow, especially for readers without a strong background in this area. For example, despite experience with SDEs in score-based generative models and deep learning theory, I found the explanations lacking in detail and context and some results are not convincing.

The authors do not provide sufficient preliminary material on scoring rules or background on how these rules are used to measure divergence between two Markov processes. A concrete example of the scoring rule $s(P, z)$ with an RBF kernel, presented early on, would have improved clarity.

Additionally, the complexity reduction claim is unconvincing. For example, at the beginning, the authors claim they reduce the complexity to linear using $D$ to denote the time steps, however, this notation $D$ is never used again in the rest of the paper. Instead, in Section 4.2 Algorithm, they compare two stochastic processes and use $B$ to denote the total number of time steps. However, the nested summations in the top equation on page 5 suggest quadratic rather than linear complexity, i.e., $B^2$.

Finally, the paper uses the **ICLR 2024 format rather than the ICLR 2025** format.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: The main concern I had was in regard to the proof. Please help me to understand or fix the issue. 

If this and the writing issues I raised above can be addressed, I will happily give you at least a 6. However, since I feel that your main result is wrong, I have to give a low score for now.

**Update: I was wrong about the validity of the proof. The result is sound. **

If this (or a variant of the) scoring rule for Markov processes is indeed correct, I think the authors could improve the paper by exploring the sensitivity properties of this scoring rule; for example, what kernel to use and how the score behaves when P and Q are close? Is there a simple formula to compute the gradient?
soundness: 4
strengths: The method is very simple and elegant and the implication of this result, **if true**, could be impactful to the neural SDE and generative modeling community.

**Update: I was wrong about the validity of the proof. The result is sound. **
summary: This paper introduces a very simple yet elegant way of comparing the similarity of the laws of two homogeneous Markov processes. The idea follows from the fact that the distribution of the process is completely determined by the transition kernel. Hence the ability to compare the transition distribution should give enable a comparison between the laws. 

The authors then use this finding as a scoring rule to learn to simulate from a neural SDE, given the observations from a ground truth process. Then, they test their scoring rule and their FDD matching procedure in numerical experiments, showing that their method outperforms existing law matching techniques.
weaknesses: I think there is a significant issue in the proof that could invalidate your main theorem 2. In the middle part of page 14, you have the integral over $t_1,t_2$ of the $S(P_{t_1,t_2}, P_{t_1,t_2}')$ scoring rules being equal, but then you conclude that $$S(P_{t_1,t_2},P_{t_1,t_2}') = S(P_{t_1,t_2},P_{t_1,t_2})$$ a.e.. This is not true or I am missing something? 

**Update: I was wrong about the validity of the proof. The result is sound. **

My intuition is that this is not an easy fix: You want to match the distribution over all $t_1,t_2$, so, in some sense, you want the expectation equality to hold for all test measures $\nu$ instead of just one particular $\nu$. I would be very happy to raise the score and rewrite my review if I am wrong. However, it does seem that proving a result like yours is possible, given that the generator or the resolvent will determine the law of the Markov process (see Either and Kurtz). 

I appreciate the rigor in defining the math notations and results. But the writing and explanations in the paper need improvements. For example, what do you mean by "Update the model parameters $\theta$ through backpropogation to maximize $\hat S$" (inside the algorithm)? Also, there is no explanation of what the data is. What are the "Average KS test scores"? Are you repeating the experiment across multiple batches and producing the percentage of rejection "chance of rejecting the null hypothesis (%) at 5%-significance level on marginal"? 

Other minor issues include: 
1. I think you need $\mathcal{E}$ to be Polish. 
2. The radial basis function (RBF) kernel is not defined. 
3. The $\pi$ notation seems a little distracting, why not just use $(x_t,x_s)$ for $\pi_{t,s}(x)$. 
4. It would be nice to emphasize that the Markov processes you define are homogeneous Markov processes.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: * Please include confidence intervals in your tables: variability of your results is a very important aspect.

* Could you please include vizualizations of the full generated time series, rather than only plotting the time marginals ?

* Could you add experiments on at least one high dimensional and one non euclidian real-world dataset ?

I would considered increasing my score if a significant number of concerns on the experiments are addressed. Similarly, I could consider lowering my score during the discussion phase if other reviewers relativise the strength of the theoretical contribution --- which again seems sufficient to me.
soundness: 4
strengths: * The paper is well written and structured, making it easy to read and to follow. The introduction is sound and features a thorough literature review.
 
* The main technical contribution is Theorem 2, which allows to convert any scoring rule on a generic space. I believe this contribution to be non trivial and novel, although simple to prove.  

* The experiments are carried out on a vast array of time series datasets and show overall superior performance of the proposed approach. The authors compare themselves to all relevant baselines to the best of my knowledge. Large generative models such as diffusion models are not included ; however, I do believe that they do not belong to the same class of models and do not require a comparable computational budget.
summary: The authors propose a scoring rule for continuous time stochastic processes that is directly derived from a scoring rule on a generic space. They show that this rule is proper (i.e. injective from the space of paths to the space of laws), which is a non-trivial contribution. Experiments show that this method outperforms existing concurrent methods based on signature kernels and SDE-GANs. I stress that the experiments are carried out on a vast array of datasets.
weaknesses: * While the contribution made in Theorem 2 is elegant and novel, one could object that it is slightly insufficient --- this is not my point of view, but another reviewer might disagree and I am willing to discuss this point. In order to strengthen their theoretical part, I encourage the authors to consider for instance the sample complexity of their kernel i.e. how fast does the empirical divergence they define through a kernel on $\mathcal{E}$ converge through the expected divergence ? Do sample complexities in $\mathcal{E}$ carry over to the space of paths ? See Gretton (2012) for an example. 

* A blind spot of the paper is in my opinion the choice of the kernel. The authors do not seem to consider other kernels for $\mathbb{R}^d$ valued processes, which could yield an interesting extension. This might especially be interesting since some kernels are sometimes used because of their specific properties (invariances, ...). I would suggest that the authors consider this point, at least in the Appendix. 

* This approach nicely extends to kernels defined on any space - this could be graphs, images, etc. and could allow to generate time series in these spaces. This would provide, in my sens, a extremely valuable extension to the paper. 

* The experimental section is hard to read and a tad unstructured. I encourage the authors to use less tables, add more comments and broaden the analysis of their experiments. Also, a notable restriction of this part is that experiments are only carried out on $2$ dimensional time series. I strongly encourage the authors to extend their experiments to high dimensional datasets. Also, there are no confidence intervals in the tables. 

* Regarding this last point, I believe that a valuable extension could be to consider random feature approximations to the kernels for high dimensional generation, which is still a major hurdle in the field. Similarly, the authors could consider sliced kernels on $\mathcal{E}$ when the dimension is high.  

* Concerning experiments, an interesting task to consider could be the augmentation of a time-series dataset, and the analysis of the gain in performance for any model trained on this dataset. 

* Concerning experiments, I believe that it would be highly valuable to extend applications beyond finance. Generating time series is a major hurdle in many domains with great social impact such as neuroscience, healthcare, biology, climatology, economics ... 

* A valuable extension of this work would be the investigate the use of the devised score for other purposes than training generative models, such as two-sample tests for instance.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see weaknesses.
soundness: 1
strengths: The paper tackles a crucial challenge in training neural SDEs by introducing a new scoring rule that optimizes efficiency for continuous Markov processes. The proposed FDM method is backed by mathematical proofs, providing a strictly proper scoring rule that extends from finite-dimensional distributions to continuous Markov processes. This theoretical contribution is valuable to the literature on neural SDE training methods. Experiments show that FDM offers computational efficiency gains, reducing training complexity from quadratic to linear in the number of discretization steps. The approach outperforms prior methods in computational efficiency, as shown in multiple experimental benchmarks.
summary: This paper presents a novel approach to training neural stochastic differential equations by introducing Finite Dimensional Matching, a new scoring rule designed for continuous Markov processes. The proposed method leverages the Markov property of stochastic processes to reduce the computational complexity of training neural SDEs, with the goal of enhancing both efficiency and generative quality. Theoretical contributions establish that the new scoring rule provides a strictly proper method for comparing two-time joint distributions. Experimental results demonstrate improved performance in training efficiency and generative quality when evaluated against competing methods.
weaknesses: The paper’s main theorem relies on strong assumptions regarding the Markovian properties and continuity of the processes involved. These assumptions may limit the applicability of the FDM algorithm in more complex, non-Markovian stochastic processes or those with jumps, which are common in real-world scenarios. Explicitly discussing these limitations and potential ways to relax these assumptions would make the contributions more transparent.

Also, the writing template is for ICLR 2024, not ICLR 2025.

Meta-review: In this work, the authors show that proper scoring rules on distributions can be extended to proper scoring rules on processes through their finite dimensional distributions. Using this result, a method called Finite Dimensional Matching (FDM) is proposed to bypass pain points with fitting stochastic processes to data. In a large number of numerical experiments, the authors demonstrate the advantages of this approach over competing methods. Reviewer opinions are mixed: two are positive, one mixed, and one negative. The reviewers stating positive opinions have stood firm on their stance that the paper should be accepted, while the more negative reviewers have failed to adequately engage in the discussion period. This is unfortunate, as the discussion has been considerable!

My own stance agrees with the positive feedback: the results are simple in retrospect, but profound in development and application. This is a strong contribution to the literature, and I believe it is worthwhile to the community even in its current state. **However**, in addition to reviewer feedback, I object to several aspects of the presentation: text in figures is unacceptably small (should be close to same size as surrounding text), avoid repeating the same reference multiple times in a paragraph, values in tables might be better separated by brackets e.g. .137 (17.0), some mistaken capitalization, placement of footnote 1 is strange, missing spacing around some references, no space between tables and table legends, paragraph discussing figures is too far away from the figures themselves. I implore the authors to address these points in their next revision. I also agree that providing the most general possible form of Theorem 2, even if in supplementary material, would be ideal. 

With this in mind, I give a tentative recommendation for acceptance.

Decision: Accept (Poster)