Simulation-Free Training of Neural ODEs on Paired Data
Authors: Semin Kim
Jaehoon Yoo
Jinwoo Kim
Yeonwoo Cha
Saehoon Kim
Seunghoon Hong
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural ODE
simulation-free training
flow matching
Primary area: diffusion_based_models
TL;DR: We propose simulation-free training method for Neural ODEs by adopting flow matching objective with learnable embeddings.

Abstract: In this work, we investigate a method for simulation-free training of Neural Ordinary Differential Equations (NODEs) for learning deterministic mappings between paired data. Despite the analogy of NODEs as continuous-depth residual networks, their application in typical supervised learning tasks has not been popular, mainly due to the large number of function evaluations required by ODE solvers and numerical instability in gradient estimation. To alleviate this problem, we employ the flow matching framework for simulation-free training of NODEs, which directly regresses the parameterized dynamics function to a predefined target velocity field. Contrary to generative tasks, however, we show that applying flow matching directly between paired data can often lead to an ill-defined flow that breaks the coupling of the data pairs (e.g., due to crossing trajectories). We propose a simple extension that applies flow matching in the embedding space of data pairs, where the embeddings are learned jointly with the dynamic function to ensure the validity of the flow which is also easier to learn. We demonstrate the effectiveness of our method on both regression and classification tasks, where our method outperforms existing NODEs with a significantly lower number of function evaluations. The code is available at https://github.com/seminkim/simulation-free-node.

Simulation-Free Training of Neural ODEs on Paired Data
Semin Kim
Jachoon Yoo
Jinwoo Kim Seunghoo Hong'
Yeonwoo Cha
Saehoon Kim 2
KAIST
2 Kakao Brain
Abstract
In this work, we investigate a method for simulation-Fir free training of Neural Ordi- nary Differential Equations (NODEs) for learning deterministic mappings between paired data. Despite the analogy of NODEs as continuous-depth residual networks, their application in typical supervised learning tasks has not been popular, mainly due to the large number of function evaluations required by ODE solvers and nu- employ the
merical instability in gradient estimation. To alleviate this problem, flow matching framework for simulation- frreennn-reeeeee training of NODEs, which directly regresses the parameterized dynamics function predefined target velocity field. Contrary to generative tasks, however, we show that applying flow matching directly between paired data can often lead an ill-defined flow that breaks the coupling of the data pairs (e.g.. . due to crossing trajectories). We propose a simple extension that applies flow matching in the embedding space of data pairs, where the embeddings are learned jointly with the dynamic function to ensure the validity of the flow which is also easier learn. We demonstrate the effectiveness of our method on both regression and classification tasks, where our method outperforms existing NODEs with a significantly lower number of function evaluations. The code available at https:://// / http//gggtttttt.e./eee com/ com/seeinne iokkiiii/m simulation -ooo-frre-odee
Introduction
Continuous-depth models 3 g have received growing attention an alternative to deep feedforward networks composed of a stack of discrete layers. As they approximate continuous, infinite-depth models with a constant set of model parameters, they are parameter-efficient models that reuse their parameters across depth. Additionally, by controlling the number of function evaluations (NFEs) during inference, we can choose the optimal trade-off between performance and computational cost. This trade-off can be adjusted without retraining the model, unlike conventional neural networks that require separately trained models of different capacities to achieve similar flexibility.
prominent class of continuous-depth models is Neural Ordinary Differential Equations (NODEs) [8]. which utilize continuous-tin differential equations to describe evolutions of intermediate states. In NODEs, a parameterized dynamics function learns the time derivative of the continuous transforma- tion of a state. NODEs can be interpreted as continuous limit of residual networks 8 [5], and hence they offer a versatile framework similarly to ResNet [17). As a result, NODEs have been successfully
applied to tasks such as physically informed modeling [37 41), time series modeling 55] 23 36] and
generative modeling with normalizing flows 112 14]
However, despite their success in various tasks, the application of NODEs to learn deterministic mappings between paired data, such as regression or classification, remains under-explored. This largely due to the substantial computational burden of training NODEs, since numerical ODE solving during training inherently serial, slow, and requires large NFEs.
"qqual Contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Recently, an alternative, sinulation-free training method has been introduced for ODE-based genera- tive models 28 29.. Known as flow matching, this approach eliminates the need for the expensive ODE solving during training by directly regressing the model on presumed velocity field. While this method significantly improves training efficiency by requiring only one function evaluation per training step, its effectiveness on deterministic tasks has not been well explored.
In this work, we investigate a simulation-fice training method for continuous-depth models in learning deterministic mapping between paired data. We first identify potential problems that occur when applying flow matching objective to NODEs, and then propose a method to mitigate them. With our simulation- ffree training scheme, we can significantly reduce the computational demands of NODE training while maintaining competitive performance deterministic tasks. We demonstrate the advantages of our method on both regression and classification problems, providing empirical evidence of its effectiveness and versatility in common supervised learning settings. Furthermore, by leveraging simple flows, our method can achieve superior performance compared to NODEs in low-NFE regime.
Preliminary
NODEs Continuous-Depth Models We aim to learn deterministie mapping between paired ]im with continuous-depth model. To this end, we consider Neural Ordinary Differential Equations (NODEs) R which can be regarded as a continuous limit of residual networks. Formally, single layer of residual network transforms hidden state with residual connection, i.e. Zt+1 = 2, hig (2x). This update resembles a single step Euler solver, which uses parameterized dynamics function hgg to approximate the derivative of with respective to I. Taking this discretization to the continuous limit, residual networks are equivalent to the following ODE:
data D = {zz()) y
dæ ho( t).
= dt
To train a NODE with a loss defined on the state at time 12, we first solve an initial value problem starting from known initial state 21,;
212 Tti + 1-²
he (o t)dt ODESolve(Z,,))..o......... he t2)
For the task of fitting paired dataset, each data-label pair (x, y) is placed on the state-space of the ODE as endpoints (zo, ) for time interval [0, 1]. This requires matching the dimensions of data x and label y to states 2, which is done by projecting data initial state 20 = (x) and the solved final state 21 to predicted label ÿ = (z1). Then, NODES are trained end-to-end to minimize the loss C(j. y). However, this approach is inherently slow and computationally intensive, as requires a large number of sequential function evaluations by the ODE solver [8] [2, (34). There is also a hidden cost in integrating the ODE backward in time for gradient estimation (i.e., adjoint method
[8), which
necessitates additional serial function evaluations and is numerically unstable [43). Also, common choices of adaptive-step ODE solvers tend to make NODEs learn arbitrarily complex trajectories 42 making both training and inference computationally expensive.
Flow Matching for Simulation-Free Training Instead of optimizing Eq. ①I via the expensive
initial value problem of Eq. ( we can alternatively employ the flow matching framework [ 28 29 to directly regress the dynamics function he to the velocity field i by:
E, [1 t) - =x11
Since the intermediate state 2, and its velocity v; are generally unknown and intractable to compute, existing works [ 28] 29 employ predefined, tractable conditional velocity fields defined per sample. Specifically, a closed form expression of 2, is given as an interpolant of two endpoints and time (i.e. = a₁ =o + 1,)2) and a target velocity field is constructed as time derivative of the interpolant. One simple instantiation of an interpolant and its corresponding target velocity field is linear dynamics with constant speed [28] 29]:
21 = t)20 + t21,
it = 21 20-
A typical choice for f in the literature is the identity mapping (i.e., 20 2).

---

GT Asociation
MSE- NFE-62
MSE-1.10M 107, NFE-1
MSE-0.000 NFE-1
2p
20
(a) Ground Truth
(b) Neural ODE
(c) Flow Matching (FM) (d) Embed. + FM (Ours)
Figure I: Comparison of the learned trajectories. The final train loss (MSE) and training NFE are shown above each plot. (a) We consider deterministic regression task of four data pairs, each of which is represented by two circles (filled and empty circles) connected by dotted lines. (b) NODEs can correctly associate the pairs but through complex paths (solid lines) that require large NFEs. (c) Flow matching with linear velocity can greatly reduce the training NFEs by simulation-free training, but fails to associate the correct pairs due the crossing trajectories induced by predefined dynamics. (d) The proposed method can alleviate the problems by learning the embeddings for data jointly with the flow matching.
Since we can obtain i at arbitrary time i, this allows the dynamics function to be trained in simulation-free manner, where the regression of the vector field is performed parallel in time. Simulation free training in the flow matching models is intriguing, since there is no need for serialized function evaluations during training as well as backpropagation through time. Employing a simple velocity field such as Eq. (4) also greatly simplifies the trajectory of the dynamics function, reducing the number of function evaluations for inference. However, flow matching has been mainly studied for generative modeling, which aims to find transportation map between two marginal distributions p(20) and p(21) while learning arbitrary per-sample couplings between 2g ~ (Zz)) and 21 ~ p(21). It makes it difficult to be applied in deterministic regression tasks where the per-sample coupling is defined by data pairs (zo. =1) ~ (x,y). We elaborate on this issue in the next section.
3 Challenges in Flow Matching for Paired Data
We seek to adopt the simulation-free free training objective in Eq. (3) to learn continuous depth model of Eq. (I). However, we find that careful consideration is required when employing the flow-matching objective to model a deterministic mapping between paired data.
To illustrate this point, we provide a toy example in Fig. [ Here, consider simple regression task with swo-dimensional inputs and outputs, translating four points at æ = -1 to the ones at x = 1 but in different order in the y axis. For ease of analysis, we consider simple linear velocity field in Eq. (4) for flow matching.
First, we observe that NODE successfully learns to associate inputs and outputs by solving Eq. (Fig. (b)). However, the learned trajectories are highly non-linear and complex which leads to a large number of function evaluations for training and inference.
The flow matching can greatly improve the computation cost by optimizing the simulation-free objective in Eq.. (3). However, we observe that it fails to infer the correct input-output correspondence
according to the learned trajectories (Fig. (c)). This is because some target trajectories induced by the predefined velocity field crossing each other, which cannot be modeled by ODEs [40]. As
result, the learned dynamics function at the intersection of crossing trajectories produces their mean velocity, which makes the inferred trajectory fail to arrive at the correct output.
Note that the crossing trajectories induced by the predefined flow are less problematic in generative tasks. It is because their objective is to transport between two marginal distributions p(20) and p(21) while allowing arbitrary association between samples žn ~ p(z0) and ž ~ p(21). is evident from Fig. I (c), where the model fails to preserve the initial coupling in the dataset but still yields valid marginal distribution at the output. However, it is not desirable for deterministic regression, since the goal is to preserve per-sample correspondence between data and label.
3

---

Data Space
Embedding Space
Data Encoder
Target
Trajectory Crossing
Label Encoder
Label Decoder dy
Figure 2: An overview of our framework. We avoid the crossing trajectory problem data space by introducing learnable encoders that project data and label to embedding space. In the learned embedding space, the presumed dynamics induce valid target velocity field.
4 End-to-End Latent Flow Matching
Previous discussions suggest that predefined flow between paired data can yield crossing trajectories that cannot be modeled by the well-defined ODE, which leads to invalid paths by the dynamics function that break the coupling of data and label. As simple solution, we propose to learn the embeddings of data and label jointly with dynamics function, in such way that the predefined flow induces non-crossing trajectories in the embedding space (Fig. (d)).
Fig. illlstrates the overall framework. The proposed framework comprises the data encoder for label encoder 9pp. label decoder de ~ 9p and dynamics function hg. Given a data pair (I, y). our model first projects the data and label using respective encoders by (zo, 21) = (ff(x). (v)). Then, for all E [0, 1]. the state 2₁ and the corresponding target velocity U2 are obtained by some predefined dynamics (e.g., Eq. (4). We train the encoders and dynamics function jointly using the flow matching loss, while an additional label autoencoding objective is applied to the label encoder and decoder. The inference procedure remains similar to that of NODEs: given the input embedding 20 = f (x), we obtain the state & by solving ODE in Eq. 2 and decode to a label by the label decoder (21). Below, we describe the learning objective and optimization of our method in detail.
Flow Loss With the learnable projections žo = fe(x), = (y(() and any simple dynamics assumption 2, = 20 + Beär, our flow matching objective is defined by:
Cio (0, 6, sp) Be(rr,))= (2,, v, 1
(5)
In contrast to conventional flow matching (Eq. (3) that optimizes the dynamics function given the fixed endpoints 2n and 21, the loss in Eq. (5) optimized jointly with two encoders fg and 9.- Since the loss has the optimum at flow (0", Ø (0") = O when the dynamics function he- perfectly fits to the velocity field u, optimizing Eq. (5) guarantees that the optimal encoders fa- and IG-- induce non-crossing target trajectories in the embedding space.
Conceptually, our approach resembles the reflow procedure in Rectified Flow 29), which straightens the target trajectories by recursively rewiring 20 and žj according to the trajectory obtained by solving approach straightens the trajectory by learning the encoders such that the
an ODE. In contrast, trajectories defined by the predefined flow do not cross in the embedding space, which does not involve ODE solving and, most importantly, preserves the initial coupling (Zu, (1).
However, contrary to Eq. (3). we observe that optimizing the flow matching objective with learnable encoders in Eq. (5) can lead to trivial degenerate solutions. One such example is when both encoders collapse to output constant ignoring data, which produces trivial targets for the dynamic model (e.g., F0 21 ho(:-)) = 0). To avoid trivial solutions, both encoders require strong regularization to be relevant to the inputs. Fortunately, such regularization is naturally provided by the learning objective for the label decoder dg, which is described below.

---

Label Autoencoding Loss We define the label autoencoding loss as simple mean squared error (MSE) between the label y and its reconstruction obtained by label encoder and decoder:
Liabel ar (v/, vp) B(lld.((p(()) vl
(6)
The primary purpose of Eq. (6) is to provide learning signals for the label decoder dg. which is used decode the predicted 21 to the label de (2)) inference. However, Eq. (6) also functions
regularization to avoid trivial solutions of Eq. (5) by preventing the label encoder de from ignoring
the input lahel y. Regularizing the label encoder is sufficient to prevent trivial solutions in principle, since it eliminates the trivial targets for both data encoders fg and the dynamics function hg.
Objective Function The overall objective function for the proposed framework is given by combin- ing the two losses introduced above: min Lioow (0, o, sp) + Liabel_ae aee 0.0.P.1
Theoretically. we can show that optimizing the combination of flow loss and autoencoding loss in Eq. 7 can indeed mitigate the problem of target trajectory crossing:
Proposition 1. There exist S and gup that induce non-crossing target trajectory for any data pair in D while minimizing Lla
Proof. The proof can be found in App. A.1
Proposition 2. If g is injective, the following equivalence holds: f 9up and hg minimize L to zero for all & [0, 1) if and only if So and Dup induce non-crossing target trajectory and hg perfectly fits the induced target velocity, for any data pair in D.
Proof. The proof can be found in App. A.2
Proposition D ensures the existence of encoders that not induce crossing the target trajectory, while Proposition 2 suggests that the encoders are optimal when flow loss is minimized, assuming the label encoder is injective. Since the label autoencoding task enforces 9.p to be injective, combining the two propositions implies that optimizing the objective function in Eq. (7) prevents the encoders from inducing target trajectory crossing and enables the dynamics function to accurately fit the induced trajectories.
Optimization During the optimization of Eq. 7. we find that two additional regularizations on the encoders usefuul in further preventing suboptimal solutions and stabilizing training and inference.
First, although the autoencoding loss in Eq. O prevents degenerate encoders in principle, we find that flow matching loss in Eq. (5) can still induce an ill-behaved local minima for the data encoder fa- For instance, under linear velocity field (Eq. 4), collapsed, constant data encoder, e.g., 2u
makes both intermediate state and target velocity depend only on at / E (0, (0, e.g., 2t (2] and et 21. In this case, a dynamics function that scales with time, he (=, 1) z/1, can fit the target velocity field for all E (0. 1] although it does not yield meaningful mapping between (x. y). We find that explicitly sampling
with a certain probability during training effectively resolves the issuc.
Second, we empirically find that the label encoder tends to reduce the scale of the output embeddings optimize the flow matching loss. Although this is not fundamental problem in principle, we observe that it often affects the generalization performance by making the model prone to small numerical errors at inference, such as prediction errors in the dynamics function discretization errors in the ODE solver. To address this, we encourage the label embeddings to repel each other, so that they construct a robust destination point for ODE solving during inference. Specifically, during training we add random noise (N0 o²) to label embedding and let the label decoder to
reconstruct original label from it, iee, we minimize [[[ld (y) + <) --]?]] instead Eq. (6). We empirically find these techniques helpful for better optimization, shown in Sec. 6.3
5

---

5 Related Work
Enhancing Efficiency of NODEs Several previous works have addressed the problem of high numbers of function evaluations (NFEs) in the forward process of Neural Ordinary Differential Equations (NODEs). Most approaches involve imposing regularization on the learned trajectory,
such penalizing higher-order derivatives [20] incorporating kinetic regularization 11] Similar
effects can be achieved through weight decay [14], augmenting dimensions [IT]. or using internal
solver heuristics [34]. Additionally, sampling the end time of the integration interval [13 has also been explored as a simple solution. These methods encourage the model to learn simpler trajectories, thereby effectively reducing the NFE required to solve the ODE. However, with the common choice of adaptive-step solvers, there often remain dozens of sequential function evaluations during single training step, making the training of NODEs slow and computationally intensive.
Simulation-Free Training on Paired Dataset Some studies have explored simulation- free training methods for fitting dynamics functions on paired datasets, primarily within the context of diffusion probabilistic models (DPMs) [8] DPMs achieve simulation-free training by learning to denoise data at multiple noise levels in parallel [3]. During inference, DPMs generate data from standard Gaussian noise through iterative denoising. These models have been applied to various vision tasks aimed at learning deterministic mappings, including segmentation [2] 4 [9]. object detection [7]. and image restoration tasks [27 33 39], among others. While these applications are impressive, they are tailored specifically to their target tasks and do not represent general methods for paired data with diverse label structures.
One notable work in this area is CARD [16], which introduced for classification and regression tasks, making it applicable to arbitrary regression and classification data. Although these works, based on diffusion models trained with denoising objectives, align more closely with neural SDEs rather than ODEs, we include a comparison with CARD in our main experiments to provide comprehensive evaluation.
conditional diffusion process
Experiments
6.1 Experimental Setup
Baselines and Datasets We validate the effectiveness of our method on various datasets for both regression and classification tasks. For baselines, we compare our method with the standard NODE g and previous works that utilize regularization to reduce NFEs. Specifically, we compare against STEER [3, which introduces stochasticity in the integration interval, and RNODE [12. which regularizes the norm of the velocity field. Additionally, we include a comparison with CARD [66), classification and regression model based on diffusion. Following prior work [I] [3, we use MNIST (26). SVHN (33), and CIFARIO 10 (55 for image classification experiments. For regression tasks, use UCI regression datasets (11. adhering to the protocol used by CARD.
Evaluation We report classification accuracy and root mean square error (RMSE) the main performance metrics. To quantify computational costs, we report average per-sample NFEs along with training throughput measured by the total training iterations divided by training time. For all NODE baselines, we use the dopri5 (10) adaptive-step solver implemented in torchdiffeq [6] package for both training and inference. Additionally, solver. For CARD, we perform few-step inference by periodically skipping intermediate steps, similar to DDPM in few-step inference [8] (38, and report the metric of full-step inference in the place of adaptive-step solver in NODEs.
report few-step inference results using the Fuler
Implementation Details We use the same network architecture across baselines, employing an MLP-based architecture for MNIST and UCI and convolutional architecture for SVHN and CI- FAR10. For classification tasks, we use one-hot encoded labels for y and assign the predicted label ğ by applying argmax on the channel dimension. To handle the significant memory requirements of training NODE-based baselines, we use the adjoint sensitivity method g for SVHN and CIFARIO experiments. When using the adjoint method, we report the total NFE by summing the number of function evaluations in both the forward and backward passes. For our model, we primarily use

---

Table 1: Experiment results image classification. Training and few-/full-step performances are reported in three datasets. For classification accuracy, numbers indicate the number function evaluations with Euler solver, where co denotes the result of dopri5 adaptive-step solver. For CARD, we report the 1000-step decoding results instead of using the adaptive solver, as the model was trained on discrete timesteps. CARD" is trained with 4 times longer steps.
Training Cost
Accuracy NFE (%)
Throughput (Batch / sec.)
Dataset
Model
NFB
10
00
NODE
354
0.93
14.20
7.71
8.81
10.83
98.36
STEER
90
3.22
24.79
29.39
49.57
62.68
99.23
MNIST
RNODE
43
3.27
99.35
99.35
99.35
99.35
99.35
CARD
28.95
9.72
15.88
98.92
99.12
98.83
Ours
29.41
99.33
99.25
99.35
99.34
99.30
NODE
87.72
91.89
95.18
95.16
95.09
STEER
110
0,27
30.55
65.00
93.22
94.44
94.55
RNODE
130
0.14
93.36
95.01
95.37
95.39
95.39
SVHN
CARD
7.94
9.95
16.01
76.87
79.49
65.31
CARD
7.94
9.95
35,91
95.36
95.31
95.23
Ours
9.48
96.16
96.14
96.03
96.03
96.12
NODE 91
0.34
81.18
82.72
86.25
86.33
86.30
STEER 96
0.33
76.80
77.78
83.55
84.24
84,51
RNODE
89
0.19
79.61
81.62
85.68
85.96
86.08
CIFARIO
CARD
1
7.67
10.22
18.78
84,48
84.68
81.77
CARD*
7.67
10.61
33.69
86.67
86.54
86.42
Ours
9.00
88.87
88.85
88.71
88.88
88.89
Our
NODE
STEER
RNODE
CARD
10
Bosten
Conente
Energy
XinAnen
Nival
RMSS
10
10"
10
2
10'
10"
10
1,00
100
Power
Yaclt
YeaPeediittenMSD
10"
TMSSB
16
10-
10
10'
10
10
100
1000
10
100
1000
10
100
1000
10
100
1000
1.0
100
1000
NFE
NFE
NFE
NFE
NFE
Figure 3: RMSE over NFEs on UCI regression tasks. To control the NFE, we use Euler solver for the
evaluation. By assuming lincar dynamies, our model shows better performance in low NFE regime.
a simple linear dynamics assumption, which is shown to be effective according to the analysis in Further experimental details can be found in App. B
Sec. 6.3
6.2 Main Results
Training Cost and Performance In Tab. we report the training cost and classification accuracy on MNIST, SVHN, and CIFARIO. MODE-bbsed baselines suffer from large NFEs -engiing from tens to bundreds-during training, resulting in low throughput. In contrast, simulation- frre training method (i.e., ours and CARD) require only a single function evaluation per training step, significantly boosting training speed.
The trade-off for faster training in simulation-fra methods a constraint on the dynamics that can be learned. However, we observe that this reduced flexibility does not lead to significant performance degradation. Compared to NODE-based baselines, our method shows only minor degradation on MNIST and even improvements SVHN and CIFAR10 in terms of final classification accuracy. On SVHN and CIFARIO, 10, NODE-based baselines are trained with the adjoint sensitivity method [ to
meet memory requirements, which is known to suffer from inaccurate gradient estimation 42 43
7

---

Thus we hypothesize that the performance gains on SVHN and CIFARIO may be due to the accurate gradient calculation achieved through direct backpropagation in our model, which is a positive byproduct of utilizing simulation-free training.
When compared to the diffusion-based baseline, our model tends to show better performance and faster convergence. Specifically, our model outperforms CARD on all three datasets, even compared CARD variants trained for longer iterations. While diffusion models also benefit from simulation- free training, they are based on stochastic differential equations that induce stochastic and non-linear trajectories. We conjecture that this characteristic not ideal for few-step inference and also contributes to slower convergence.
Few-Step Inference Ideally, our model trained with linear dynamics will yield a perfectly straight solution trajectory, which can be accurately estimated even with one-step Euler solver. Consequently, with our linear dynamics assumption, we can significantly enhance inference speed by utilizing few- step inference while maintaining competitive performance compared to many-step solving. To demonstrate this, we report few-step inference results with the Euler solver Tab. T Our model, by avoiding crossing points in the learnable embedding space, produces linear trajectory and thus exhibits superior few-step performance. This observation aligns with findings in flow matching models 28]29, which highlight the advantage of linear dynamies for generating high quality samples with low inference cost.
Regression We further analyze the effectiveness of our method in regression tasks, as shown in Fig. See App. for full results. Despite differences in label structure, we observe similar trends for both classification and regression tasks: our method significantly reduces computational burden during training and demonstrates superior performance in few-step inference with linear dynamics. Similar to the original NODE, our model can be effectively applied to wide range of common supervised learning settings, regardless of whether the labels are categorical or continuous.
6.3 Analysis and Discussion
Learning Encoders with Flow Loss To support our key claim that learning encoders with flow loss allows our model to avoid crossing trajec- tories, we compare our model with two variants that do not utilize flow loss for learning encoders. Specifically, we consider: (1) ANODE+ FM which augments the data and label to same dimensional- ity by zero-pare paro-padding and learns dynamics function with flow matching; and (2) Autoencoder+ FM which employs a two-stage approach by first learn- ing the embedding space using an independent autoencoding objective for both data and labels, and then learning the dynamics function on the fixed embedding spacd" To quantify the cross- ing points in trajectories, we train all models with the linear dynamics assumption and measure the proportion of samples where the predicted labels from one-step Euler solver and adaptive-step solver "
Table 2: The effectiveness of learning encoders with flow loss. Training accuracy and the pro- portion of disagreement in prediction between a one-step Euler solver and an adaptive-step solver are shown. Simply augmenting dimensions (AN- ODE+FM) does not effectively prevent trajec- tory crossing. Furthermore, learning encoders without flow loss (Autoencoderr+FM) also fails to preserve the original coupling due to crossing trajectories.
Training
Disagreement
Accuracy
ANODE + FM Autoencoder FM
47.30%
30.23%
11.56%
55.73%
Ours
0.02%
99.80 %
We report the disagreement ratio and training accuracy in Tab. 2
As discussed in Sec. our results indicate that merely augmenting the dimension (ANODE+FM) does not resolve the issue of crossing trajectories induced by the predefined dynamics, resulting in poor performance (Autoencoder+FM) partially reduces disagreements, it still fails to fit the training data properly. Such crossing trajectories can be eliminated by learning encoders with the flow matching loss, allowing our model to fit successfully to training data with high accuracy.
training data. Although employing more sophisticated encoders
'The samples reconstructed by the pretrained data autoencoder are presented in App. D.1 We also analyze the relationship between time and the occurrence of trajectory crossing in App. D.2
8

---

Ours
w/o Explicitly Sampling 0-0
w/ Adding Noise to Label Autoencoding
Training Flow loss on CIFARIO
Training Accuracy on CIFARIO
Test Accuracy on CIFAR10
16-
Accuracy (%)
Accuracy (%)
Flow Loss
08-
800
4%
906
206
80
64
66x
Training Steps
Training Steps
Training Steps
Figure 5: Ablation study of optimization techniques on CIFARIO. Explicitly sampling 0 in train-
ing prevents suboptimal solutions while adding noise to label autoencoding improves generalization.
Role of the Learned Dynamics Function A potential concern is that learning a nonlinear transfor- mation embed data might lead the data encoder to perfectly predict the label. This issue, noted in previous work [32], warns that learning a complex input transformation could result in a collapse where the dynamics function becomes& a simple identity map. To investigate whether this issue occurs in our case, we conduct 1-NN classification using the learned data encoder on CIFARTO 10 image classification. The I-NN classification accuracy with the learned data embedding 2g 65.66%, which is significantly lower than the accuracy of 88.47% when we utilize the learned dynamics function to obtain the predicted label embedding 2 This result indicates that the learned data embedding is not yet linearly separable enough, and the learned flow can further enhance accuracy. Based on this observation, we conclude that the collapse scenario, where the velocity field becomes an identity map, does not occur in our model. Instead, we find that the learned dynamics function clearly plays role in processing the data, thereby avoiding the aforementioned pitfall.
Nonlinear Predefined Dynamics Our method em- braces not only the linear dynamics but also dynamics having fixed form of 2, 0g + 3,21 in general. Here, we investigate the effect of utilizing different dynamics assumptions, including nonlinear ones. To be specific, we choose three different dynamics that easy to implement:
Dynomics
Boston
RMSE
10'
10
Concave: az- coss( f1, = sin (t)
NFF
Linear: a = (1 -1-11) 31 =t
Figure 4: Analysis on predefined dynamics. (Left) Change of coefficients in interpolant with respective to time. (Right) Prediction RMSE over NFE on UCI Boston dataset.
Convex: O2 = 1 - sin( (1,,
sin() B, = cos (t).
We visualize the difference of dynamics and their effect on final performance in Fig. g
As it shows, the performance of the nonlinear variants (i.e., convex and concave) clearly improves with an increased number of function evaluations. For these variants, the many-step inference performance increases as invest more NFEs and then later saturates, indicating that the model's approximation of an infinite-depth model becomes sufficiently accurate as discretization error diminishes. In contrast, our model trained on the linear dynamics shows consistently good performance, even with a few function evaluations. This behavior is somewhat expected, as the ideal linear trajectory can be already accurately inferred using one-step Euler solver. Surprisingly, we also empirically observe that the choice of linear dynamics leads to better performance, compared to more complex choice of dynamics assumption. Thus, similar to the claims of Liu et (2.2)) 22). we believe that linear dynamics should be considered as a default choice unless specific constraints on hidden states are required.
Ablation on Optimization Techniques We conduct an ablation study on CIFAR10 dataset to investigate the effects of optimization techniques introduced in Sec. and report the results in Fig. 5 Similar ablation studies were also conducted on other datasets, detailed in App. D.3 The variant without explicitly sampling t = O fails to fit on training data, despite the convergence of the flow loss. Furthermore, the variant without adding noise to the label autoencoding objective succeeds

---

in fitting the training set, but its test accuracy significantly degrades compared to the version with noise in label autoencoding, Thus, as discussed Sec. we conclude that explicitly sampling t and introducing noise in label autoencoding effectively regularize both encoders and produces label embedding that is robust to test-time errors.
Limitations and Future Work
In this work, primarily study the problem of adopting flow matching by imposing fixed and simple dynamics assumption, observing that a fixed, closed-form equation for the intermediate state already sufficient bring the advantages of flow matching. Although employing simple dynamies (e.g. linear) may seem overly restrictive, this approach can be justified the context Koopman operator theory, which aims to find embeddings that globally linearize the dynamics. Since method of flow matching with linear dynamics shares high-level concept with Koopman operator theory, exploring the relationship between the two could be promising direction for future research. We leave further discussions in App. C
On the other hand, while studied simple and predefined dynamics, the form of dynamics sumption could be further generalized to be a learnable component. To be specific, introducing a learnable target dynamics that determines per-sample dynamics would be promising future direction to study. This would be advantageous for handling inputs with varying complexity, by efficiently and adaptively allocating more computation hard samples.
Furthermore, extending our method to a broader range of paired data applications might be a useful future direction. The reduced computational burden achieved through simulation-free training could offer several benefits of continuous-deptt models diverse applications. For example, applying our method to model compression knowledge distillation could be particularly promising, leveraging the parameter efficiency of continuous- models.
Conclusion
In this work, adopted flow matching objective achieve simulation-free training of continuous- depth models for learning deterministic mappings between paired data. We proposed learning an embedding space where flow matching occurs, which identified a erucial component for ensuring the validity of the target velocity field. Our proposed method significantly reduces the computational burden of training NODEs while maintaining competitive performance. Additionally, we found that our method, leveraging simple linear dynamics, demonstrates impressive performance on low-NFE regime.
Acknowledgments and Disclosure of Funding
This work was in part supported by the National Research Foundation of Korea (RS-2024-00351212 and RS-2024-00436165) and Institute of Information and communications Technology Planning and Evaluation (IITP) grant RRS-2022-I1220922 11190075) funded by the Korean govemmeet (MSIT). We thank Youngmin Ryou (KAIST) for his valuable input in formulating the theoretical analysis presented in this work.
RS-2024-00509279, R8-2021-1121268,, and RS-2019-
10

---

References
[1] Michael S. Albergo and Eric Vanden- Eijjeen.. 2023. Building Normalizing Flows with Stochas tic Interpolants In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
[2] Tomer Amit, Eliya Nachmani, Tal Shaharabany, and Lior Wolf. 2021. SegDiff: Image Segmen- tation with Diffusion Probabilistic Models. CoRR, abs/21
[3] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. 2019. Deep Equilibrium Models. In NeurlPS, pages 689-699.
[4] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko.
2022. Lafel-Efficient Semantic Segmentation with Diffusion Models. In ICLR.
[5] Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. 2019. GRU-ODE-Bayes Continuous Modeling of Sporadically- Time Series. In NeurIPS.
[6] Ricky T. Q. Chen. 2018. torchdiffeq
[7]
Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. 2023. DiffusionDet: Diffusion Model for Object Detection, In IEEE/CVF International Conference Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 19773-19786. IEEE.
[8] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. 2018. Neural Ordinary Differential Equations] In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 6572-6583.
[9] Ting Chen, Lala Li, Saurabh Saxena, Geoffrey F. Hinton, and David J. Fleet. 2023. A Generalist Framework for Panoptic Segmentation of Images and Videos. In ICCV.
[10] J.R. Dormand and P.J. Prince. 1980. A family of embedded Runge-Kutta formulae Journal of Computational and Applied Mathematics, 6(1):19-26.
[11] Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. 2019. Augmented Neural ODEs In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 31 3134-3144.
[12] Chris Finlay, Train Your Neural ODE: the World of Jacobian and Kinetic Regularization. In JCML, volume 119 of Proceedings of Machine Learning Research, pages 3154-310 3164. PMLR.
Jacobsen, Levon Nurbekyan, and Adam M. Oberman. 2020. How to
[13] Arnab Ghosh, Harkirat Behl, Emilien Dupont, Philip Torr, and Vinay Namboodiri. 2020. STEER Simple Temporal Regularization For Neural ODE, Advances in Neural Information Processing Systems, volume 33, pages 18831-14443 Curran Associates, Inc.
[14] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. 2019. FFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models. In ICLR. OvenReview.net.
[15] Eldad Haber and Lars Ruthotto. 2017. Stable architectures for deep neural networks Inverse Problems, 34(1):014004.
[16] Xizewen Han, Huangjie Zheng, and Mingyuan Zhou. 2022. CARD: Classification and Regres sion Diffusion Models, In Advances in Neural Information Processing Systems 35: Annual Conference on Neurul Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA USA, November 28- December 9, 2022.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR, pages 770-778. IEEE Computer Society.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Probabilistic Models. In NeurIPS.

---

[19] Sergey loffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In JCML.
[20] Jacob Kelly, Jesse Bettencourt, Matthew J. Johnson, and David Duvenaud. 2020. Learning Differential Equations that are Easy Solve. In NeurIPS.
[21] Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. 2023. The UCI machine learning repository.
[22] Patrick Kidger. 2022. On Neural Differential Equations. CoRR, abs/2202.02435.
[23] Patrick Kidger, James Morrill, James Foster, and Tery J. Lyons. 2020. Neural Controlled Differential Equations for Irregular Time Series, In NeurIPS.
[24] Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR (Poster).
[25] Alex Krizhevsky, Geoffrey Hinton, et al. 2009, Learning multiple layers of features from tiny images.
[26] Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. 1989. Handwritten Digit Recognition with Back-Propagation Network. In NIPS, pages 396-404. Morgan Kaufmann.
[27] Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. 2022. SRDiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 479:47-59.
[28] Yaron Lipman, Ricky T. Q. Chen, Heli Hem-Hamu, Maximilian Nickel, and Matthew Le. 2023.
Flow Matching for Generative Modeling In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OvenReview.net.
[29] Xingchao Liu, Chengyue Gong, and Qiang Liu. 2023. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
[30] Calvin Luo. 2022. abs/2208. 11970.
Understanding Diffusion Models: A Unified Perspective.
CoRR,
[31] Bethany Lusch. Nathan Kutz, and Steven Brunton. 2018. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1):4950.
[32] Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. 2020. Dissecting Neural ODEs. In NeurIPS.
[33] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. 2011. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning. Granada, Spain.
[34] Avik Pal, Yingbo Ma, Viral B. Shah, and Christopher Vincent Rackauckas. 2021. Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristies. In ICML.
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR.
[36] Yulia Rubanova, Tian Qi Chen, and David Duvenaud. 2019. Latent Ordinary Differential Equations for Trregularly-Sampler Time Series. In NeurIPS.
[37] Alvaro Sanchez-CGonzaler Victor Bapst, Kyle Cranmer, and Peter W. Battaglia. 2019. Hamilto- nian Graph Networks with ODE Integrators. CoRR, abs/1909.12707
[38] Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021. Denoising Diffusion Implicit Models. In ICLR.
12

---

[39] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. 2022. Deblurring via Stochastic Refinement. CVPR.
[40] Laurent Younes. 2010. Shapes and Diffeamorphisms, volume 171. Springer Science & Business
Media.
[41] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. 2020. Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control. In JCLR.
[42] Juntang Zhuang, Nicha C. Dvorekk, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James S. Duncan. 2020. Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE. In ICML.
[43] Juntang Zhuang, Nicha C. Dvorekk, Sekhar Tatikonda, and James S. Duncan. 2021. MALI: A memory efficient and reverse accurate integrator for Neural ODEs. ICLR. OpenReview.net.
13

---

Appendix
A Proofs
In this section, provide full proofs for the propositions in Sec. 4 We first start with formal definition of target trajectory crossing. Assume that we have a data encoder fa and label encoder 9e that transform data æ E R and labels y E Rdy to latent 20: 21 E R where 20 = fz( and
21 9- (v), respectively. We also choose predefined dynamics F F(zu, 21, a, + S:21 2t-
Under the assumption of d > dg. d, Dand both az, Be being smooth and nonzero except for 0 and
t == I, we define the target trajectory crossing as follows:
Definition 1 (Target Trajectory Crossing), The encoders (fo. are said to induce target trajectory crossing if there exists a tuple (t, x. y, ".x'.'' s/) such that a1)o(z) + 8.0 (v) ftf((x') + (1' for x + x' and + y/.
We now provide the proofs for each proposition in the following subsections.
A.1 Proof of Proposition I (Section
Proposition 1. There exist (fp) D while minimizing _la
9xp) that induces non-crossing target trajectory for any data pair in
Proof. Let the latent space constructed by a set of basis I (ci, C2 ...... eaj. Since d > dy. we can find a label encoder Рp such that utilizes h basis J = {ei. E2. ..., { (d > k 2 dy ) and minimizes can find data encoder fo such that (r (x) = (K)) fa('') (x') if and only if x = x', where K. = (ex+1, ed))
the autoencoding loss (i.e., 9 (v) (3') if and only if y¹). Also,
Then, suppose that there exists a tuple (t, x. y. x', ./) such that fofo(x) + (1) = anfo(z') + (1gg(1)) i.e., (x) - fa(x')) (g (x(1') O.
Since gp((y) - (1/) = 0 and only if y = s/ and projspen (f)(2) - fa(x')) = if and only I = x' by construction, such a tuple does not exist. Therefore, there exists for Iup such that does not induce target trajectory crossing, while minimizing the autoencoding loss.
A.2 Proof of Proposition] 2 (Section 4
Proposition 2. If is injective, the following equivalence holds: (Sos 92, ho) minimizes Livon to zero for all E [0, 1) if and only if (f 9.) induces non-crosee non-crossing target trajectory and hi perfectly fits the induced target velocity, for any data pair in D.
Proof. (f) (Ser 9p) induces non-crossing target trajectory for any data pair in D. there is a well-defined target velocity # at every which is continuous on t. If he perfectly fits this target velocity for all (zz, t), the flow loss zero.
(=) We prove by contradiction. Suppose the flow loss is zero but there is a crossing trajectory, i.e. there exists tuple (f, y, x', v') that 2, 2 for * x' and y # y'. Since the loss is zero for all d t [0, 1), the dynamics function he must output di F(2o, 21, t) 21, and di (2/1 21, t) at 2/. This is contradiction since at the point of crossing we have z z but di F( 21, # d F(/ o t). O
B Experiment Details
This section describes the implementation detail our experiments (Sec. 6).
$ As de and d, are dimensions in observation space, we also conjecture that the latent dimension d can be made smaller if the data lives low- wwwddimensioaal manifold.
14

---

B.1 Experiment Details for Classification Tasks
Network Architecture (MNIST) We employ an MLP-based architecture for MNIST dataset. 34-ndimnsiional embedding, using three-layer MLP with
The data encoder maps each image a hidden dimension 784 and BatchNorm [19]. The dynamics function consists of a three-layer MLP with 2048 hidden dimension. Following NODE [8] we concatenate the time variable to the input of each layer. Consistent with common practices for NODES [22], normalization layers are not included in the dynamics function. For the label autoencoder, class labels are converted into one-hot vectors and encoded with a single linear layer, while the label decoder utilizes a two-layer MLP with BatchNorm.
Network Architecture (SVHN, CIFARIO) We adopt CNN-based architecture for both the SVHN and CIFARIO datasets. The data encoder utilizes 7 convolutional layers with hidden dimension of 64, By default, we employ 3x3 convolution kernels, while the final two layers utilize 4x4 kernels with stride of for downsampling, resulting in the data being encoded into states with dimensions of 7x7664. The dynamics function consists of 6 convolutional layers with 3x3 kernels and hidden dimension of 256. At each layer, the time variable is concatenated to the input. We utilize a single linear layer for label encoding, reshaping the output to match the size of embedding. In the label decoding process, we average the feature map over the spatial dimension and apply a single linear layer.
Training We train all models for 100,000 iterations using the Adam optimizer [4 with a cosine learning rate scheduler. For all classification experiments, we utilize batch size of 1024. Additionally, we set maximum training time of 48 hours for the feasibility of experiments. By default, le-3 for MNIST and 3e-4 for CIFARIO and SVHN. For our method, we set the ratio of explicitly sam- pling / = 0 to 10% for all datasets. Regarding the noise introduced to the label autoencoder, standard deviation o to for MNIST, 7 for SVHN, and 10 for CIFARIO. In cases where training of NODE basclines fails, adjust the learning rate accordingly. The failure cases of NODE baselines are illustrated in Fig.
Los
NIEE
set the learning rate
Slep
Figure 6: Failure cases of NODEs.
set the
B.2 Experiment Details for Regression Tasks
Network Architecture We utilize a 2-layer MLP with a hidden dimension of 64 for the data encoder, and a 7-layer MLP with the same hidden dimension for the dynamics function. In the dynamics function, we concatenate the time variable to the input of each layer. On the label side, we employ single linear layer for encoding and another single linear layer for decoding.
Training We trained all models for 100,000 iterations using the Adam optimizer with a constant learning rate of 3e-3. Additionally, we split the training set into a train-validation split with a ratio of 6:4, and utilized the validation metric for early-stopping. Early stopping was implemented by measuring the validation metric every 1,000 iterations and setting the patience level to 10. For our model, we sample the noise added to the label autoencoding from (0, 32) and the proportion of explicitly sampling = 0 as 10%.
B.3 Experiment Details about Baselines
Following the tolerance values used in NODE g and ANODE [1I. we used dopri5 IT] solver with the absolute and relative tolerance of 1e-3 for both training and inference.
NODE 8
STEER [3
STEER introduces new hyperparameter that controls the integration interval.
Instead of integrating the dynamics function from to 1, STEER integrates from 0 to / Uniform(1-
1 + b) during training. We follow the default configuration of using = 0.99 for MNIST classification. For other tasks, set to 0.1 since higher values resulted in training failures.
15

---

RNODE introduces two hyperparameters used for the coefficients of regularization terms. Specifically, regularizes the Jacobian norm and the kinetic energy of the dynamics function to encourage the model to learn straight and constant-speed dynamics. The coefficient of 0.01 was generally used throughout the experiments in the original paper. Therefore, we set both coefficients for the Jacobian norm and kinetic energy as 0.01 for our experiments.
RNODE 12
CARD utilizes discrete timesteps and hyperparameter to schedule the noise level for diffusion modeling. Following the paper's 's approach, we use 1000 discrete timesteps and set linear noise schedule from 31 =le-4 to f1000 = 0.02.
CARD 16
B.4 Computation Resources
We conducted experiments on our internal cluster with two types of machine. We list their specifica- tions below.
1. Intel Xeon Gold 6330 CPU and NVIDIA RTX A6000 GPU (with 48GB VRAM)
2. Intel Xeon Gold 6230 CPU and NVIDIA RTX 3090 GPU (with 24GB VRAM)
We utilized the first machine for image classification experiments, and latter one for regression exper- iments. We expect training our model will take about 90 min., 270 min, 230 min. for classification experiments on MNIST, SVHN and CIFARIO, 10, respectively. For regression tasks, we estimate training cost, summing up for all splits and datasets, would cost about 288 GPU hours.
C Relation to Koopman Autoencoder
Our model with linear dynamics shares the high-level motivation with Lusch et al. (2018) [1], which aims to find an embedding space that yields linear dynamics between source and target. Regardless of the theoretical background, both flow matching and Koopman operator theory are promising approaches that seek to interpret nonlinear system within well-studied linear framework.
At the same time, we identify several differences between our work and the line of research based on Koopman operator theory. While those works mainly focus systematic way obtain linearized representation of the underlying nonlinear dynamics (with eigenfunctions), our work aims to find way to learn it in simulation-free manner, avoiding the heavy computation of forward simulation (e.g., which appears in Liin of Lusch et al. (2018) (1I from an initial state to an end state. Additionally, compared to the discrete depth neural networks that have single linear layer processor, our proposed method is generally applicable to any nonlinear dynamics that connects two endpoints 2n and 21, exemplified as conver or concave discussed in Sec. 6.3 This implies that in our case, is possible to have a latent trajectory as a curve in on-Fuulideen geometry whenever the interpolated state 2, is tractable.
D Additional Results
In this section, we extend the discussions in Sec. with additional results.
D.1 Additional Result of Learning Encoders with Flow Loss
We present the reconstructed images from the trained data autoencoder in Fig. which are used to analyze the effectiveness of the flow loss as reported in Tab. 2 While the pretrained au- toencoder reconstructs the images holistically with minimal information loss (Tab. 2 the em- bedding space learned without the flow loss fails to maintain the coupling with the predefined dy- namics.
Recon. Input
Figure 7: Reconstriction from the autoencoder.
16

---

ANODE FM
Autoencoder FM
Ours
cosine similarity
1.0
0.9
0.8
0.5
0.5
Figure 8: Cosine similarity between target velocity and predicted velocity over time t. Similar to the flow loss, we measure the cosine similarity between target velocity and predicted velocity. Low cosine similarity near 1 = 0 and f = indicates the occurrence of target trajectory intersection near the endpoints.
Ours
w/o Explicitly Sampling 0
w/o Adding Noise to Label Autoencoding
Training Flow loss on MNIST
Training Accuracy on MNIST
Test Accuracy on MNIST
Accuracy (%)
Accuracy (%)
Flow Loss
/
Mygh
0.01
3/6
10c
306
160c
18c
Training Steps
Training Steps
Training Steps
Training Flow loss SVHN
Training Accuracy SVHN
Test Accuracy SVHN
K00-
Accuracy
Flow
24
84
Training Steps
Training Steps
Training Steps
Figure 9: Ablation study of optimization techniques on MNIST and SVHN datasets, corresponding
Fig. 5 Explicitly sampling prevents suboptimal solutions, while adding noise label autoencoding improves generalization performance.
D.2
Location of Crossing Points
In Fig. g we extend the analysis in Sec. 6.3 to identify where target trajectory intersections occur. Specifically, we measure the cosine similarity between target velocity and predicted velocity across By measuring cosine similarity instead of MSE, we can ignore the effect of absolute scale in the embedding space and accurately compare ANODE+FM, Autoencoder+FM, and our method. Model variants that do not learn encoders with flow loss particularly suffer from trajectory crossing near both endpoints, showing low prediction accuracy for the direction of target velocity. In contrast, our proposed method shows consistently high cosine similarity, mitigating the issue of target trajectory intersections near these regions. Since both endpoints are constructed from encoders, the result also supports our claim that the encoders should trained with flow loss to effectively penalize such intersections.
D.3 Additional Results of Ablation on Optimization Techniques
We provide additional results on the optimization techniques in Fig. 9 Consistent with the findings on CIFAR10 (Fig. 5). the model without explicitly sampling at = O converges to suboptimal solutions, while introducing noise during label autoencoding improves test accuracy.
D.4 Additional Experiment Result for UCI Regression Tasks
The full results on UCI regression tasks with various solvers are shown in from Tab. 3 to discussed the main text, our method with linear dynamics assumption shows superior performance compared to baselines in low-NFE (1-2 steps) regime, which aligns with our observations in image classification experiments.
As
17

---

Table 3: Experiment results on UCI regression tasks with Euler 1-step solver.
Dataset
Ours
NODE
STHER
RNODE
CARD
Boston
3.52:10.79
5,951:5.10
552.1011638.55
3.50:10.44
130.23113891
Concrete
5.59:10.58
2922.76:112615.25
1056.15 .27
1.98111.66
198.91172.58
Energy
0.5811.111
253.57:1707.63
164.6
0.83110.41
63.87:112.81
Kin8nm
7.61=0.02
39.04+3331
1694.59=770.1
23.69+0.82
238.18+6.44
Naval
0.08+00000
8447.16+2968.35
3011.13+:90.99 .95
0.16+0.01
15.42+0.59
Power
4.03+0.19
>> 10"
14.7
4.06+0.35
179.91
Protein
4.26.10.08
103.14.1141.5
102.171177.5
7.70.11.88
92.82.123.84
Wine
0.67110.05
1.99-14.31
19.087136.97
0.67:00.05
10.42:12.87
Yacht
0.92110.42
62.19:1
29.06:141.13
2.71
111.21-:6.53
Year
9.15+NA
1366.06=NA
2452.93+NA
22.37+NA
157.11+NA
Table 4: Experiment results on UCI regression tasks with Euler 2-step solver.
Dataset
Ours
NODE
STEER
RNODE
CARD
Boston
3.50
3.81=22004
278.23:+827.35
3.41
12.26
Concrete
5.58:10.57
1560.21
1437.21
5.59:10.80
28.49:17.008
Energy
0.57:10.11
114.46:1310.32
36.50-1-2..1
0.61+00111
8.15
Kin8nm
7.59++0.02
14.62++1.23
813.00-1338.36
9.774.0.17
18.87
Naval
0.07
10*
>> 10
0.08+00.00
1.81
Power
4.03
> 10
5,46:6.52
3.86+0023
10.58
Protein
4.26=t0.08
30,85=224.65
77.94=77.06
4.40:L0.35
14.17=6.997
Wine
0.67+00.55
3.50-
24.95-146.23
0.66++0.04
1.49
Yacht
0.93++0:43
34.66-62.06
17.13+131.04
1.79+1.001
19.40
Year
9.17+NA
6053,32+NA
542.82+NA
9.67+NA
42.18+NA
Table 5: Experiment results on UCI regression tasks with Euler 10-step solver.
Dataset
Ours
NODE
STEER
RNODE
CARD
Boston
3.49=0.79
3.20:10.82
56.541164.66
3.39=0.80
3.32:10.89
Concrete
5,57=0.54
294.5411120.66
18.00=190075
5,59=0.77
5.47-0067
Energy
0.58=0.12
21.24:157.69
15.6
0.52=0.06
0.58=00.09
Kin8nm
7.59=0.02
7.46++0.02
159.73+-6.92
7.43=0.02
7.71+0.02
Naval
0.07=0.00
>> 10*
> 10*
0.03=0.00
0.09+40.00
Power
4.03-0.18
2144.77++9543008
4.10=0.71
3.82-0.21
4.57=+0.19
Protein
4.26=0.08
7.38116.01
7.92
3.87=0.05
5.66110.12
Wine
0.68=0.05
0.82:1044
>> 10
0.66=0.04
0.75
Yacht
0.96-0.45
2.58
1.86=1.93
1.21
0.95
Year
9,25+NA
139.85+NN
69.53+NA
8.87-NA
10.85
Table 6: Experiment results on UCI regression tasks with Euler 20-step solver.
Dataset
Ours
NODE
STEER
RNODE
CARD
Boston
3.49+:0779
3.20-1-.82
29.29-80.54
3.39--0.80
3.32-40.89
Concrete
5.58
148.43:1629.86
85.15
5.59-10.77
5,43:10.66
Energy
0.58110.13
10.43 288
2.23
0.52-10.06
0.53:10.09
Kin8nm
7.62:10.02
7.39100.02
81.51=31.91
7.38100.2
7.95110.03
Naval
0.07+00.0
>> 10a
> 104
0.03++0.00
0.09-+0.00
Power
4.04=0018
1073.63++476..25
3.97=0.19
3.82+00.21
4.81
Protein
4.27+0.08
5.55+2.65
5.33=2.38
3.86+0.04
5.90+E0.07
Wine
0..05
0.71
1.10-0.72
0.66.10.04
0.75
Yacht
0.94
1.29
0.65
1.1740..0
0.89-10.36
Year
9.30-LNA
40.97
33.251NA
8.86
11.40-LNA
18

---

Table 7: Experiment results on UCI regression tasks with Euler 100-step solver.
Dataset
Ours
NODE
STEER
RNODE
CARD
Boston
3.49
3.20:::.82
1.58-+13.13
3,40=10.80
3.44
Concrete
5.63=10.55
33.72::124.75
10.41
5.60+1.77
5.59:00.1
Energy
0.59=10.14
2.12:1:.92
0,6410052
0,52:10.06
0.55
Kin8nm
7.77=0002
7.37+00.2
20.30=5.69
7.36100.03
8.38+:0.02
Naval
0.07=+0.00
2095.27--667.36
116.16=43.60
0.03++0.00
0.12++0.00
Power
0.08==016
216.78++950.18
3.95+0.18
3.82++0.1
5.03++0.18
Protein
4.30=0009
0..29
4,20+0.06
3.86+0.004
6.13+00.9
Wine
0.68
0.66.10.04
0.7110.07
0.66110.04
0.7810.06
Yacht
0.9611.46
1.12.10.54
1,23
1.1610.60
0.98
Year
9.35.LNA
8,94
9,41
8.86
11.87
Table 8: Experiment results on UCI regression tasks with Euler 1000-step solver.
Dataset
Ours
NODE
STEER
RNODE
CARD
Boston
3.50++..00
3.20+:082
3.44
0.00
3.34+0.90
Concrete
5.66+0.55
8.29+11.12
5.70
5.60
5.56+0.52
Energy
0.59:10.14
0.52+40.08
0.53+10.07
0.52-00.06
0.58
Kin8nm
7.87+10.02
7.36+10.02
7.69-10.06
7.36+-0.03
8.53-10.03
Naval
0.07+-0.00
19.23+55.63
10.21+3.75
0.03++0.00
0.13+00.00
Power
4.14:10.14
24.24:190.84
3.95
3.82:0021
5.1520.12
Protein
4.36110.09
4.33:10.23
4.20110066
3.86.10.04
6.17:0.11
Wine
0.69110055
0.66110.04
0.70
0.66100.0
0.7610.00
Yacht
0.96+0046
1.1240.54
1,23+0048
116+0.60
0.95+0.34
Year
9,39+NA
8.95+NNA
8.95+NA
8.86+NA
12.05++NA
Table 9: Experiment results UCI regression tasks with dopri5 solver.
Dataset
Ours
NODE
STEER
RNODE
CARD
Boston
3.50
3.20.1081
3.44.10.69
3.40-10.80
3.34.10.90
Concrete
5.6710.55
5.76110.9
5.70.1.557
5.60.10.77
5.56.10.52
Energy
0.60.10.14
0.52.10.08
0.53
0.51100.66
0.58.00.07
Kin8nm
7.89+:002
7.35=0.02
7.56+0.02
7.37+20.03
8.53+0.03
Naval
0.0740.00
0.05-0.00
0.10+0..00
0.03+00.0
0.13+00.00
Power
4.1740011
3.91=0.19
3.95+t0.17
3.83+0021
5.15+0.12
Protein
4.37--0.11
4.33-10.3
4.20-10.66
3.86+10.04
6.17+00.11
Wine
0.69+10.05
0.66+10.44
0.70++0.05
0.66+L0.04
0.76+L0.06
Yacht
0.96+-0.6
1.12-0.54
1,23
1.16+00.00
0.95+
Year
9.411NA
8.95+NA
8,95
8.88
NA
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations of work in Sec. 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
We provide the full set of assumptions and proof for theoretical analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide all the information needed to reproduce our work.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: We release the code with instructions at https://githut www.smmmmmmmiccceeemmmm..e...e semi / n
node
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https wwt.sSuwwisssssppoooooooooo
/nipss.cc/pulli//guui
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specified all the experiment details either in the main text or in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We reported either 95% CI or standard deviation for UCI regression tasks. However, we were not able to repeat image classification experiment due to the limited computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided sufficient information on the computer resources in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[NA]
Justification: observed no particular societal impact to address.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not have experiments with dataset having such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cited all the datasets and assets that used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provided the code with documentation in our submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "sqgRe66thG", "reviewer": "Reviewer_YTiC", "summary": "This paper develops the Flow Matching (FM) algorithm to connect paired data. Due to the issue of crossing trajectories, FM in the data space cannot perfectly match associated pairs. To address this, the authors perform FM in an embedded space. Ultimately, they encode source and target data through an encoder end-to-end and learn FM loss in the embedded space. The main application is image classification.", "questions": "- In the real world scenario (on the classification/regression task presented in the paper), I am not sure if the crossing trajectory is crucial point that influence the performance. I am curious that if authors can compare their models with FM (well, there may be issues on matching source and target dimensions equal when impelmenting FM).", "limitations": "See the Weakness section.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper begins with a reasonable motivation and is well-presented, making it easy to follow.", "weaknesses": "- While the motivation is to avoid incorrect pair connections due to crossing trajectories in the data space, embedding data into a latent space does not guarantee the prevention of trajectory crossings. I recommend authors to visualize (on toy data, or on real data) that the proposed method experimentally prevents crossing trajectories. Moreover, it would be valuable if authors show trajectory crossing occurs in real world scenarios by comparing the results with FM. For example, it would support the motivation and also the proposed method if the proposed method outperforms FM in connecting paired data (in real-world data).\n\n- The improvement in accuracy may be due to the additional embedding networks rather than resolving the issue of trajectory crossings. Therefore, it is recommended that the authors conduct experiments and visulalize/demonstrate crossing issue can be resolved in a learned latent space on simple low-dimensional toy data.\n\n- The comparison group is weak. There are algorithms like ANODE [1] and FFJORD [2] designed to connect paths with simpler trajectories. It would be beneficial to compare these algorithms as well. \n\n[1] Augmented Neural ODEs, NeurIPS, 2019.\n[2] FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models, ICLR, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1.** Embedding data into a latent space does not guarantee the prevention of trajectory crossings. The improvement in accuracy may be due to the additional embedding networks rather than resolving the issue. Please show that the proposed method experimentally prevents the issue.\n\n**A1.** We would like to first clarify that while an embedding space alone does not prevent trajectory crossings, the flow loss (Eq. (5)) encourages non-crossing trajectories. As described in our paper (L128-130), this objective is optimized when encoders induce non-crossing target trajectories. As the flow loss remains high if the target trajectory has intersections (a dynamics function should struggle to fit on multiple velocities simultaneously), jointly optimizing the encoders with the dynamics function lets the encoders to relax trajectory crossings by adjusting the embeddings.\n\nTo visualize this, we have extended the 2D toy experiment from Fig. 1, comparing trajectories learned by NODE, flow matching, and our method. Results are presented in Fig. R.1 of the rebuttal PDF. The ground truth coupling (Fig. R.1(a)) with linear predefined dynamics induces multiple points of intersection, causing naive flow matching with an identity encoder to fail in preserving the original coupling (Fig. R.1(c)). Our method (Fig. R.1(d)) successfully learns an embedding space inducing non-crossing target trajectories, correctly fitting the data with proper coupling.\n\nAs demonstrated in the toy experiment, we believe that our model mainly benefits from solving trajectory crossing, rather than additional architectural components. In fact, we use the same data encoder $f_\\phi$ (which precedes the dynamics function) for all baselines in our experiments (Sec. 6.2-6.3), to ensure fair comparison. This results in a roughly same architecture for all methods in the experiment, where the only difference comes from the label encoder used in our method only to achieve simulation-free training, and not used during inference. The ablation study in our paper (L257-276) also supports our claim that accuracy improvement results from avoiding crossing trajectory. We further extended the study in **Q3**, to clearly show that our proposed method benefits from mitigating crossing trajectory.\n \nLastly, we kindly note that the performance gain compared to NODE baselines may come from a precise gradient calculation (L232-234), as NODE baselines using adjoint sensitivity methods can suffer from inaccurate gradient estimation [1, 2]. \n\n>**Q2.** The comparison group is weak. (Comparison with ANODE and FFJORD)\n\n**A2.** For more comprehensive comparison, we compared with ANODE [3] baseline, which utilizes zero-padding to increase data dimension (Tab. R.2). Similar to as discussed in Sec. 6.2, our method consistently outperforms ANODE in training cost, test accuracy, and performance in the low NFE regime. We will include these results in our final manuscript.\n\nRegarding FFJORD [4], we found that its main focus is on improving continuous normalizing flow in terms of computational efficiency, rather than encouraging simpler trajectories. Thus, we would be happy to hear from the reviewer about how further baselines could be added, so that we can improve the presentation of empirical results. Besides, we believe that RNODE [5], which explicitly regularizes trajectories, already serves as a strong baseline, demonstrating fairly good performance in the low NFE regime for SVHN image classification (Tab. 1).\n\n>**Q3.** In the real world scenario, I am not sure if the crossing trajectory is a crucial point that influences the performance. (comparison with FM)\n\n**A3.** To address the reviewer’s concern about the importance of crossing trajectories in real-world scenarios, we compared two naive FM baselines on CIFAR10 while matching the source and target dimensions: one using a zero-padding encoder (ANODE+FM) and another using a learnable encoder which is trained by autoencoding objective then frozen during flow loss optimization (Autoencoder+FM). The analysis below expands our analysis in Sec. 6.3 (L257-276).\n\nTab. R.1 shows that the disagreement ratio and classification accuracy (measured on the training set) are significantly affected by learning encoders with our proposed objective. As discussed in our paper (L267-272), we could identify a trajectory crossing by high disagreement ratio between one-step and multi-step prediction result. Our observation that naive FMs show high disagreement ratios suggests that they cannot properly resolve target trajectory crossing, thereby failing to fit the training dataset (low accuracy). Our model, however, achieves high accuracy by preventing most trajectory crossings, showing low disagreement ratio.\n\nWe also measured the accuracy of predicted velocity, similar to the flow loss (Eq. (5)), by replacing MSE with cosine similarity to disregard the magnitude of the target velocity, which depends on the learned embedding space. If a target trajectory crossing problem occurs, the velocity prediction near the intersection point shows low cosine similarity. As shown in Fig. R.2, naive FM baselines suffer from crossings near endpoints, resulting in low cosine similarity. Our model mitigates this issue, consistently showing high cosine similarity across the entire range of $t$.\n\nBy comparing our method with naive FM baselines, we observe that trajectory crossing is indeed crucial in real-world scenarios, and our proposed method benefits from effectively preventing it.\n\n[1] Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE, Zhuang et al. (2020)\n\n[2] MALI: A memory efficient and reverse accurate integrator for Neural ODEs, Zhuang et al. (2021)\n\n[3] Augmented Neural ODEs, Dupont et al. (2019)\n\n[4] FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models, Grathwohl et al. (2019)\n\n[5] How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization, Finlay et al. (2020)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WsgXzoeK8G", "reviewer": "Reviewer_4GBu", "summary": "The authors propose to use the flow matching loss, which directly matches the dynamics of a neural ODE (NODE) model to the pre-defined (simple) vector field, for supervision tasks. While the flow matching with simple linear vector fields is efficient, it cannot work well for supervision tasks because the paired data structure can require a crossing trajectories, which cannot be achieved by using the ODE + linear vector field. To overcome this issue, the authors propose to use the input and label encoders, and learning simple linear vector field in the latent space. To prevent learning trivial dynamics (e.g., ignoring data), they also introduce the label decoder and label reconstruction loss, which makes the output latent signal to be meaningful. The authors validate their approach with various supervision tasks, and show the proposed framework outperforms other competitors in terms of the cost-performance trade-off.", "questions": "The authors experimentally demonstrated that input/label encoders do not become arbitrarily complicated (i.e., learn all the information on the given task), thus the latent dynamics play a sufficiently significant role for solving the task. Can the authors intuitively explain how this is possible?", "limitations": "The authors mention some limitations (e.g., assumming the underyling dynamics is linear) on the proposed method.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "At first glance, the proposed method seems to be too simple; using the auto-encoders to match the latent dynamics with simple one is somewhat straight-forward. However, such a simple approach can remarkably improve NODE-based models for supervision tasks, with a significant margin compared to baseline NODEs.\n\nThe paper is extremely well-written and easy to follow.", "weaknesses": "While this paper provides useful insights into NODEs, such as the crossing trajectory problems and the use of latent dynamics, these are well-known topics within the community of NODEs. Therefore, I believe this paper should be evaluated based on its practical application rather than its theoretical aspects. \n\nFrom the practical perspective, while the proposed technique outperforms some popular NODE-based supervision models, it exhibits significantly lower performance compared to the standard no-NODE-based baselines; the classification accuracy of 88.89% for CIFAR10 is not great. The fact that all NODE-based models failed at this task, and that the proposed model is better at least, does not bring much satisfaction (note that diffusion-based CARD model can estimate the uncertainty, though its performance is not great also). \n\nI do not think that every model needs to achieve SOTA performance to be published. However, to at least have readers consider trying the proposed NODE-based framework instead of conventional finite-depth models (given that supervision is generally not approached using NODEs), I believe the proposed model should at least be compared to similar-sized non-NODE-based MLP and ResNet models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1.** While this paper provides useful insights into NODEs, such as the crossing trajectory problems and the use of latent dynamics, these are well-known topics within the community of NODEs. Therefore, I believe this paper should be evaluated based on its practical application rather than its theoretical aspects.\n\n**A1.** We agree with the reviewer that the problem of crossing trajectory and corresponding solution of using latent that augments data dimension is already discussed in previous NODE literature [1, 2]. They mainly discuss the approximation capability of NODEs when it has insufficient data dimension, where NODEs without dimension augmentation are revealed not to be a universal function approximator [2, 3]. \n\nOur observation, however, concerns **a different type of crossing trajectory problem** that arises when applying flow matching for simulation-free training of NODEs. This issue stems from **intersections in target trajectories induced by predefined dynamics**, rather than from an inherent limitation of NODEs.\n\nThe toy experiment in Fig. 1 (or Fig R.1 of the rebuttal PDF) illustrates this difference. As shown in Fig. 1(b), NODE successfully fits the data even without a dimension augmentation. However, when we introduce flow matching for simulation-free training, the predefined linear dynamics lead to trajectory crossing (Fig. 1(c)). Our key contribution is resolving this issue by inducing a valid velocity field (Fig. R.1 (d)) for the dynamics function to regress on.\n\n>**Q2.** To at least have readers consider trying the proposed NODE-based framework instead of conventional finite-depth models (given that supervision is generally not approached using NODEs), I believe the proposed model should at least be compared to similar-sized non-NODE-based MLP and ResNet models.\n\n**A2.** \nTo address this concern, we conducted an additional experiment comparing our method with a ResNet model. Our main experiments primarily aimed to compare our proposed method with NODE baselines, using a simple CNN backbone following the convention of NODEs [4]. While this CNN model sufficiently demonstrates our method and allows comparison with NODE baselines, we found that we can boost the performance of our model by using stronger backbones.\n\nWe customized the ResNet-18 architecture for our model and compared it with a ResNet model. Both models have approximately the same number of parameters (11.2M). Our model achieved a classification accuracy of 94.5%, matching the ResNet-18 model's performance of 94.6% with similar training costs.\n\nUnlike conventional neural networks, NODEs have advantages of learning smooth, bijective continuous transformations between source and target. For instance, this property makes NODE-based classification models robust to adversarial data perturbation, as studied in TisODE [5]. However, it was previously difficult to consider NODEs as replacements for conventional neural networks in supervised tasks due to performance issues from inaccurate gradient estimation [6, 7] and high training costs associated with numerical ODE solvers. As our method matches the performance and training cost of conventional neural networks, we believe NODEs with simulation-free training could be now viable alternatives to consider.\n\n>**Q3.** The authors experimentally demonstrated that input/label encoders do not become arbitrarily complicated (i.e., learn all the information on the given task), thus the latent dynamics play a sufficiently significant role for solving the task. Can the authors intuitively explain how this is possible?\n\n**A3.** \nWe appreciate your thoughtful comment. Intuitively, the dynamics function plays a crucial role in solving tasks as it is a main component to be used in solving an ODE initial value problem from $z_0$​ to $z_1$​ during inference. Fitting the dynamics function to the target vector field is essential, while input and label encoders support this by constructing an embedding space that prevents target trajectory crossings with given data pairs and predefined dynamics. Therefore, the dynamics function effectively remains as the key element in solving the task.\n\n[1] Augmented Neural ODEs, Dupont et al. (2019)\n\n[2] Dissecting Neural ODEs, Massaroli et al. (2020)\n\n[3] Approximation Capabilities of Neural ODEs and Invertible Residual Networks, Zhang et al. (2020)\n\n[4] Neural Ordinary Differential Equations, Chen et al. (2018)\n\n[5] On Robustness of Neural Ordinary Differential Equations, Yan et al. (2019)\n\n[6] Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE, Zhuang et al. (2020)\n\n[7] MALI: A memory efficient and reverse accurate integrator for Neural ODEs, Zhuang et al. (2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XFBP9QXqkC", "reviewer": "Reviewer_RQFX", "summary": "This paper presents an approach for instantiating a flow matching method for paired data $(x, y)$ without relying on iterative ODE solvers. The method uses an input encoder with a pair of target decoder and encoder to project the original data into a latent space. By imposing the form of the dynamics in latent space, the trajectory of the latent vector between $x$ and $y$ can be represented using a closed-form equation. This approach demonstrates competitive or superior performance compared to methods that require iterative ODE solvers and diffusion-based models.", "questions": "- This is an extra point for me: I am curious about the differences between this approach and discrete depth neural networks. If the dynamics between $z_0$ and $z_1$ can be expressed with an interpolation, we could say that this architecture is very similar to the discrete depth neural networks, where the processor in the latent space is a single linear layer with or without a skip connection. An example of such an architecture can be seen in Lusch et al. (2018). Did the authors try to compare the proposed method with such an architecture?\n\nReferences: \n- Lusch et al. (2018), Deep learning for universal linear embeddings of nonlinear dynamics, Nature Communications, https://www.nature.com/articles/s41467-018-07210-0", "limitations": "I confirm that the authors have addressed sufficiently the limitations of their approach.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea and motivation are clearly exposed, with sufficient detail to understand the intuition behind the approach.\n- This work is a good example of the effort to link intuitions from various domains together to create a clear and simple method.\n- The results are interesting, providing evidence that nonlinear dynamics in latent space can be eliminated without compromising prediction accuracy.", "weaknesses": "- I found that the presentation of the main Table 1 is not entirely clear to me. The main messages (smaller NFE vs. competitive results) are conveyed, but it is still a bit confusing to see the number of NFEs directly in the table for the methods requiring only one NFE.\n- Minor issues: many abbreviations are defined multiple times throughout the text. Please be careful if the authors used a writing assistant to help with the drafting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1.** Presentation issues in Table 1 and repeated definition of abbreviations in the main text.\n\n**A1.** We thank the reviewer for highlighting these presentation issues. We acknowledge that some abbreviations (e.g., NFE or NODE) are defined repetitively. In the final version of our manuscript, we will revise the presentation of Tab. 1 and improve the readability of the main text to address these concerns.\n\n>**Q2.** Comparison with the discrete depth neural networks, where the processor in the latent space is a single linear layer with or without a skip connection.\n\n**A2.** We appreciate the reviewer for bringing our attention to this interesting related work. Our model with linear dynamics shares the high-level motivation with Lusch et al. (2018) [1], which is to find an embedding space that yields linear dynamics between source and target. It is interesting to see that two different approaches, namely flow matching (with optimal transport) and Koopman operator, converge on the same point. Regardless of the theoretical background, both approaches are appealing as they seek to interpret a nonlinear system within a well-studied linear framework.\n\nAt the same time, we have identified several differences between our work and the line of research based on Koopman operator theory. While those works mainly focus on a systematic way to obtain a linearized representation of the underlying nonlinear dynamics (with eigenfunction), our work aims to find a way to learn it in a simulation-free manner, avoiding the heavy computation of forward simulation (e.g., which appears in $\\mathcal{L}_{lin}$ of [1]) from an initial state to an end state. Additionally, compared to the discrete depth neural networks that have a single linear layer processor, our proposed method is generally applicable to any nonlinear dynamics that connects two endpoints $z_0$ and $z_1$, exemplified as 'convex' or 'concave' in our paper (L289-L298). This implies that in our case, it is possible to have a latent trajectory as a curve in non-Euclidean geometry whenever the interpolated state $z_t$ is tractable.\n\n[1] Deep learning for universal linear embeddings of nonlinear dynamics, Lusch et al. (2018)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bevh1w4d6m", "reviewer": "Reviewer_bNYt", "summary": "The paper revisits using neural ordinary differential equations (NODEs) for modeling deterministic maps on paired data, e.g., maps that solve regression and classification problems. The authors propose utilizing flow-matching (FM), a recent simulation-free training method for NODEs, to overcome the computational overhead of traditional NODE training and inference. The paper first presents the problems with simply using FM for learning maps on paired data since the learned map by FM is not guaranteed to preserve the coupling presented in training but only to preserve the distributions. Second, the authors propose to solve the coupling preservation problem by adding a learned encoder-decoder on the label space and an additional encoder on the data space to \"rewire\" the trajectories so that they do not cross. Then, the coupling will be preserved through FM training. The authors demonstrate the efficacy of their approach on classification and regression tasks and show that learning with FM also facilitates learning linear maps, which can be inferred in a single function evaluation - alleviating both training and inference shortcomings of traditional NODEs.", "questions": "- Can the authors provide a comparison to augmented NODEs+FM training? or provide and explanation as to why they think this may not work?\n\n- Regarding the second point in the method weaknesses, I would be happy to extend the discussion on this point and think if there's some experiment to justify the use of NODEs here better.\n\n\nTypos:\n- In Figure 1 (c), the title says `NFE=1`, but trajectories are curved. Could there be a typo in the plot title?\n- Figure 1 caption labels (c) and (d) are mislabeled as (b) and (c).\n- Table 1, CIFAR10, RNODE, Throughput seems like a typo, value is like simulation free methods while it should be lower.", "limitations": "yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper presents an interesting approach towards end-to-end learning of NODEs with flow matching when preserving the training set coupling is required.\n- Experiments validate the efficiency of the approach, achieving SOTA performance - both in metrics and runtime.\n\n**Presentation**\n\n- Ideas and concepts in the paper are often presented with provided intuition, guiding the reader to the logic behind the algorithmic choices.", "weaknesses": "**Presentation**\n\n- Although intuitive explanations were mentioned as a strength, in some cases, they come in short. The paper lacks formal and rigorous explanations of the method and experimental settings. For instance:\n  - Adding noise to labels (L164) - I am unsure I understand the setting here. Is the noise added to the ground truth labels in the L2 loss? Or are they added to the embeddings $z_1$? An equation explicitly stating the noise addition would greatly help clarify this. \n- In the preliminaries section, the introduction of flow-matching is inaccurate and lacks the main point that FM does not regress to the marginal velocity field but rather regresses to a conditional velocity field, named linear dynamics in the paper, while the marginal is not necessarily linear. \n\n\n\n**Method**\n\n- A naive solution to the crossing trajectories problem, would be to augment the dimension of the learned flow such as suggested and discussed in [1,2] and is in a sense similar to training a conditional model, where the condition is the initial point. The paper misses the sanity check and comparison with the most naive baseline, combining augmented NODEs with flow matching. I wonder, if one only learns $d_\\psi$ and $g_\\varphi$, $f_\\phi$ is set to be identity, and the learned flow is trained on an augmented space with the initial point, would that achieve better/worse results? According to [2], augmenting the flow model with the initial point would solve the crossing trajectories problem and allow the couplings to be preserved. \n- I find the motivation for using NODEs for regression and classification rather weak when the learned map is linear (i.e., solved by a single NFE). In a sense, the motivation for using NODEs was to utilize \"infinite\" depth, in a shared weights manner, by learning a time-dependent function. But in the case presented in the paper, most of the \"heavy lifting\" in learning the representation is already done by the encoder-decoder neural nets, while the learned velocity could be thought of as a last layer represented by some non-linear function, separating the data, which is not linearly separable as shown by experiments in section 6.3, since as it appears from Table 1, there's not much of a performance gain in using NFE$>1$.\n\n\n\n[1] [Augmented Neural ODEs](https://arxiv.org/abs/1904.01681), Dupont et al. (2019)\n\n[2] [Augmented Bridge Matching](https://arxiv.org/abs/2311.06978), De Bortoli et al. (2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1.** The paper lacks formal and rigorous explanations of the method and experimental settings. For instance, adding noise to labels (L164).\n\n**A1.** We appreciate the comment and will revise the presentations in the method section to be more clear in our final version of the manuscript. For adding noise to labels (L164), we let the label decoder reconstruct from noisy label embedding. Formally, \\\n$\\mathcal{L}(\\psi, \\varphi) = \\mathbb{E} [||d_{\\psi}( g_{\\varphi}(y) + \\epsilon) - y||_2^2],$ where $ \\epsilon \\sim \\mathcal{N}(0,\\sigma^2)$. \n\n>**Q2.** The introduction of flow-matching is inaccurate and lacks the main point. There are typos in Fig.1 and Tab. 1.\n\n**A2.** We appreciate the reviewer’s effort to help make our manuscript rigorous and accurate. We will revise the description of Eq. (3) in Sec. 2, to clearly convey that the target velocity field $v_t$ is not a marginal velocity field but a conditional velocity field, which is defined by a per-sample basis. For typos:\n- Fig. 1.(c) shows NFE for training, which is 1 in flow matching since it does not require a full trajectory to be calculated during training. We will revise the caption to clearly inform this.\n- In Tab. 1, the throughput of RNODE on CIFAR10 is 0.19.\n\n>**Q3.** The paper misses the sanity check and comparison with the most naive baseline, combining augmented NODEs with flow matching.\n\n**A3.** To address the reviewer's concern, we have extended our analysis in Sec. 6.3, adding ANODE+FM (augmented NODE [1] with flow matching) to Tab. R.1 of the rebuttal PDF. Our model demonstrates higher classification accuracy and a lower disagreement ratio compared to this baseline. This is because our model can relax target trajectory crossing, which is a consequence of learning encoders by flow loss. Since the fundamental reason for crossing arises from predefined dynamics rather than insufficient dimensionality, simply allowing encoders to augment the dimension (like ANODE) doesn't effectively prevent the issue of target trajectory crossing.\n\n>**Q4.** The motivation for using NODEs for regression and classification is rather weak when the learned map is linear (i.e., solved by a single NFE).\n\n**A4.** We agree with the reviewer’s point that the motivation for using NODEs becomes weaker when the learned trajectory is solved by a single NFE. However, our method encompasses not only the linear dynamics but also any nonlinear dynamics that connects two endpoints $z_0$ and $z_1$. As demonstrated in Fig. 4 of our paper, utilizing nonlinear dynamics (e.g., convex and concave) yields models that clearly benefit from additional NFEs.\n\nIn these cases, the motivation for using NODE as an infinite depth model remains valid. NODEs offer unique advantages such as parameter sharing across depth and the ability to freely trade off performance and computation without retraining. These properties make NODEs appealing compared to conventional neural networks, indicating their value as a research topic.\n\nWhile our preliminary analysis of nonlinear predefined dynamics (L289-310) concluded that linear dynamics perform better than our nonlinear choices (i.e., convex and concave), this work can be extended by carefully designing new nonlinear dynamics that outperform linear ones. Future research could even make the dynamics a learnable, data-dependent component (L325-328), allowing the model to choose between simpler dynamics for inference cost and more complex dynamics for performance. We believe our work can serve as a starting point for such attempts by addressing the primary concern of heavy computation costs during NODE training.\n\n[1] Augmented Neural ODEs, Dupont et al. (2019)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

We appreciate the constructive feedback provided by all reviewers, which has significantly contributed to the improvement of our paper. We are encouraged by the positive recognition our paper has received, including:
- "begins with a reasonable motivation and is well-presented" (YTiC),
- is "a simple approach that can remarkably improve NODE-based models for supervision tasks" (4GBu),
- is "a good example of the effort to link intuitions from various domains together to create a clear and simple method" (RQFX),
- "presents an interesting approach" for "NODEs with flow matching when preserving the training set coupling is required" (bNYt).

In response to your valuable feedback, we have thoroughly revised our manuscript, addressing your concerns across several aspects.
Please find our reviewer-specific feedback below. We look forward to any further comments or discussion.

Meta-review: The reviewers agreed that the paper summary was as follows: a new training scheme for neural ODEs using “flow matching” with a pairwise loss on an embedding base is proposed and applied to image classification problems.

The reviewers were broadly on the threshold of a borderline accept, with one reviewer at borderline reject, and one review flipping between borderline reject to borderline accept in light of the extra results. All reviewers agreed that the paper was well written and easy to follow.

Reviewer 4GBu and YTiC noted that the theoretical contribution would be limited. Reviewer 4GBu agreed it improved against NODE benchmarks, which was sufficient even though NODEs do get SOTA. (The AC notes that NODE methods can reach SOTA on CIFAR-10 and CIFAR-100 when having comparable parameter counts to the benchmark discrete NNs.)

The most favorable review (by RQFX) was the least thorough and the reviewer did not respond during the discussion period, and as such provided the AC with less evidence for the decision, but the review was not discounted.

The biggest issue the AC sees in the review is subtle but potentially major: (YTiC)
> The improvement in accuracy may be due to the additional embedding networks rather than resolving the issue of trajectory crossings. 

There was an in depth discussion with reviewer YTiC on the effectiveness of the proposed approach and attribution of the gains. The reviewer was not convinced by proposed experiments, and did not reply to an additional proof. This particular AC cannot judge the accuracy of the proof, but the AC is familiar with NODE models relying more on the non-NODE parts of the network than on the NODE part, and the reviewer’s concerns are convincing. 

The authors added the following additional elements during the discussion period:
- The authors added one additional page of results to the rebuttal with two plots and two tables, which made 4GBu increase to a borderline accept.
- A proof was added in the discussion to address the reviewer’s YTiC concern.
- A second table of experiment results was added in response to bNYt.

The additional results and promised proof can be added into the camera ready version without constituting a major revision of the paper. The AC emphasizes to the authors that the additional results were necessary to increase the score.

Because most reviewers believe this paper is over the borderline of acceptance, and there are no major disqualifying issues, the AC believes the paper can be accepted. The issues highlighted by YTiC are subtle and could be settled by future investigators.
Decision: Accept (poster)