Title: Simulation-Free Training of Neural ODEs on Paired Data
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural ODE

simulation-free training

flow matching
Abstract: In this work, we investigate a method for simulation-free training of Neural Ordinary Differential Equations (NODEs) for learning deterministic mappings between paired data. Despite the analogy of NODEs as continuous-depth residual networks, their application in typical supervised learning tasks has not been popular, mainly due to the large number of function evaluations required by ODE solvers and numerical instability in gradient estimation. To alleviate this problem, we employ the flow matching framework for simulation-free training of NODEs, which directly regresses the parameterized dynamics function to a predefined target velocity field. Contrary to generative tasks, however, we show that applying flow matching directly between paired data can often lead to an ill-defined flow that breaks the coupling of the data pairs (e.g., due to crossing trajectories). We propose a simple extension that applies flow matching in the embedding space of data pairs, where the embeddings are learned jointly with the dynamic function to ensure the validity of the flow which is also easier to learn. We demonstrate the effectiveness of our method on both regression and classification tasks, where our method outperforms existing NODEs with a significantly lower number of function evaluations. The code is available at https://github.com/seminkim/simulation-free-node.

Simulation-Free Training of Neural ODEs on Paired Data

Abstract

In this work, we investigate a method for simulation-free training of Neural Ordinary Differential Equations (NODEs) for learning deterministic mappings between paired data. Despite the analogy of NODEs as continuous-depth residual networks, their application in typical supervised learning tasks has not been popular, mainly due to the large number of function evaluations required by ODE solvers and numerical instability in gradient estimation. To alleviate this problem, we employ the flow matching framework for simulation-free training of NODEs, which directly regresses the parameterized dynamics function to a predefined target velocity field. Contrary to generative tasks, however, we show that applying flow matching directly between paired data can often lead to an ill-defined flow that breaks the coupling of the data pairs ( e.g. , due to crossing trajectories). We propose a simple extension that applies flow matching in the embedding space of data pairs, where the embeddings are learned jointly with the dynamic function to ensure the validity of the flow which is also easier to learn. We demonstrate the effectiveness of our method on both regression and classification tasks, where our method outperforms existing NODEs with a significantly lower number of function evaluations. The code is available at https://github.com/seminkim/simulation-free-node .

1 Introduction

Continuous-depth models [3, 8] have received growing attention as an alternative to deep feedforward networks composed of a stack of discrete layers. As they approximate continuous, infinite-depth models with a constant set of model parameters, they are parameter-efficient models that can reuse their parameters across depth. Additionally, by controlling the number of function evaluations (NFEs) during inference, we can choose the optimal trade-off between performance and computational cost. This trade-off can be adjusted without retraining the model, unlike conventional neural networks that require separately trained models of different capacities to achieve a similar flexibility.

Aprominent class of continuous-depth models is Neural Ordinary Differential Equations (NODEs) [8], which utilize continuous-time differential equations to describe evolutions of intermediate states. In NODEs, a parameterized dynamics function learns the time derivative of the continuous transformation of a state. NODEs can be interpreted as a continuous limit of residual networks [8, 15], and hence they offer a versatile framework similarly to ResNet [17]. As a result, NODEs have been successfully applied to tasks such as physically informed modeling [37, 41], time series modeling [5, 23, 36], and generative modeling with normalizing flows [12, 14].

However, despite their success in various tasks, the application of NODEs to learn deterministic mappings between paired data, such as regression or classification, remains under-explored. This is largely due to the substantial computational burden of training NODEs, since numerical ODE solving during training is inherently serial, slow, and requires large NFEs.

Recently, an alternative, simulation-free training method has been introduced for ODE-based generative models [1, 28, 29]. Known as flow matching, this approach eliminates the need for the expensive ODE solving during training by directly regressing the model on a presumed velocity field. While this method significantly improves training efficiency by requiring only one function evaluation per training step, its effectiveness on deterministic tasks has not been well explored.

In this work, we investigate a simulation-free training method for continuous-depth models in learning deterministic mapping between paired data. We first identify potential problems that occur when applying flow matching objective to NODEs, and then propose a method to mitigate them. With our simulation-free training scheme, we can significantly reduce the computational demands of NODE training while maintaining competitive performance in deterministic tasks. We demonstrate the advantages of our method on both regression and classification problems, providing empirical evidence of its effectiveness and versatility in common supervised learning settings. Furthermore, by leveraging simple flows, our method can achieve superior performance compared to NODEs in low-NFE regime.

2 Preliminary

NODEs as Continuous-Depth Models We aim to learn a deterministic mapping between paired data D = { x ( i ) , y ( i ) } N i =1 with a continuous-depth model. To this end, we consider Neural Ordinary Differential Equations (NODEs) [8], which can be regarded as a continuous limit of residual networks. Formally, a single layer of a residual network transforms a hidden state z with a residual connection, i.e. z t +1 = z t + h θ ( z t ) . This update resembles a single step of Euler solver, which uses parameterized dynamics function h θ to approximate the derivative of z with respective to t . Taking this discretization to the continuous limit, residual networks are equivalent to the following ODE:

To train a NODE with a loss defined on the state at time t 2 , we first solve an initial value problem starting from a known initial state z t 1 :

For the task of fitting a paired dataset, each data-label pair ( x, y ) is placed on the state-space of the ODE as endpoints ( z 0 , z 1 ) for time interval [0 , 1] . This requires matching the dimensions of data x and label y to states z , which is done by projecting data x to initial state z 0 = f ϕ ( x ) , 2 and the solved final state z 1 to predicted label ˆ y = d ψ ( z 1 ) . Then, NODEs are trained end-to-end to minimize the loss L (ˆ y, y ) . However, this approach is inherently slow and computationally intensive, as it requires a large number of sequential function evaluations by the ODE solver [8, 22, 34]. There is also a hidden cost in integrating the ODE backward in time for gradient estimation ( i.e. , adjoint method [8]), which necessitates additional serial function evaluations and is numerically unstable [43]. Also, common choices of adaptive-step ODE solvers tend to make NODEs learn arbitrarily complex trajectories [42], making both training and inference computationally expensive.

Flow Matching for Simulation-Free Training Instead of optimizing Eq. (1) via the expensive initial value problem of Eq. (2), we can alternatively employ the flow matching framework [1, 28, 29] to directly regress the dynamics function h θ to the velocity field v t by:

Since the intermediate state z t and its velocity v t are generally unknown and intractable to compute, existing works [1, 28, 29] employ predefined, tractable conditional velocity fields defined per sample. Specifically, a closed form expression of z t is given as an interpolant of two endpoints and time ( i.e. z t = α t z 0 + β t z 1 ), and a target velocity field is constructed as a time derivative of the interpolant. One simple instantiation of an interpolant and its corresponding target velocity field is linear dynamics with a constant speed [28, 29]:

z

0

=

x

GT Association

z

0

=

x

MSE=0.001, NFE=62

z

0

=

x

MSE=1.107, NFE=1

z

0

=

fφ

(

x

)

MSE=0.000, NFE=1

z

0

=

x

GT Association

z

0

=

x

MSE=0.001, NFE=62

z

0

=

x

MSE=1.107, NFE=1

z

0

=

fφ

(

x

)

MSE=0.000, NFE=1

(b) Neural ODE

z

0

=

x

GT Association

z

0

=

x

MSE=0.001, NFE=62

z

0

=

x

MSE=1.107, NFE=1

z

0

=

fφ

(

x

)

MSE=0.000, NFE=1

(c) Flow Matching (FM)

z

0

=

fφ

(

x

)

MSE=0.000, NFE=1

(d) Embed. + FM (Ours)

Since we can obtain v t at arbitrary time t , this allows the dynamics function to be trained in a simulation-free manner, where the regression of the vector field is performed parallel in time. Simulation-free training in the flow matching models is intriguing, since there is no need for serialized function evaluations during training as well as backpropagation through time. Employing a simple velocity field such as Eq. (4) also greatly simplifies the trajectory of the dynamics function, reducing the number of function evaluations for inference. However, flow matching has been mainly studied for generative modeling, which aims to find a transportation map between two marginal distributions p ( z 0 ) and p ( z 1 ) while learning arbitrary per-sample couplings between z 0 ∼ p ( z 0 ) and z 1 ∼ p ( z 1 ) . It makes it difficult to be applied in deterministic regression tasks where the per-sample coupling is defined by data pairs ( z 0 , z 1 ) ∼ ( x, y ) . We elaborate on this issue in the next section.

3 Challenges in Flow Matching for Paired Data

We seek to adopt the simulation-free training objective in Eq. (3) to learn a continuous-depth model of Eq. (1). However, we find that careful consideration is required when employing the flow-matching objective to model a deterministic mapping between paired data.

To illustrate this point, we provide a toy example in Fig. 1. Here, we consider a simple regression task with two-dimensional inputs and outputs, translating four points at x = -1 to the ones at x = 1 but in a different order in the y axis. For ease of analysis, we consider simple linear velocity field in Eq. (4) for flow matching.

First, we observe that NODE successfully learns to associate inputs and outputs by solving Eq. (2) (Fig. 1 (b)). However, the learned trajectories are highly non-linear and complex which leads to a large number of function evaluations for training and inference.

The flow matching can greatly improve the computation cost by optimizing the simulation-free objective in Eq. (3). However, we observe that it fails to infer the correct input-output correspondence according to the learned trajectories (Fig. 1 (c)). This is because some target trajectories induced by the predefined velocity field are crossing each other, which cannot be modeled by ODEs [40]. As a result, the learned dynamics function at the intersection of crossing trajectories produces their mean velocity, which makes the inferred trajectory fail to arrive at the correct output.

Note that the crossing trajectories induced by the predefined flow are less problematic in generative tasks. It is because their objective is to transport between two marginal distributions p ( z 0 ) and p ( z 1 ) while allowing arbitrary association between samples z 0 ∼ p ( z 0 ) and z 1 ∼ p ( z 1 ) . It is evident from Fig. 1 (c), where the model fails to preserve the initial coupling in the dataset but still yields valid marginal distribution at the output. However, it is not desirable for deterministic regression, since the goal is to preserve per-sample correspondence between data and label.

Data Space

z

0

x

y

Target

Trajectory

Crossing

z

1

f

𝜙

g

φ

𝑑𝜓

Data Encoder

4 End-to-End Latent Flow Matching

Previous discussions suggest that predefined flow between paired data can yield crossing trajectories that cannot be modeled by the well-defined ODE, which leads to invalid paths by the dynamics function that break the coupling of data and label. As a simple solution, we propose to learn the embeddings of data and label jointly with dynamics function, in such a way that the predefined flow induces non-crossing trajectories in the embedding space (Fig. 1 (d)).

Fig. 2 illustrates the overall framework. The proposed framework comprises the data encoder f ϕ , label encoder g φ , label decoder d ψ ≈ g -1 φ , and dynamics function h θ . Given a data pair ( x, y ) , our model first projects the data and label using respective encoders by ( z 0 , z 1 ) = ( f ϕ ( x ) , g φ ( y )) . Then, for all t ∈ [0 , 1] , the state z t and the corresponding target velocity v t are obtained by some predefined dynamics ( e.g., Eq. (4)). We train the encoders and dynamics function jointly using the flow matching loss, while an additional label autoencoding objective is applied to the label encoder and decoder. The inference procedure remains similar to that of NODEs: given the input embedding z 0 = f ϕ ( x ) , we obtain the state ˆ z 1 by solving ODE in Eq. (2) and decode it to a label by the label decoder ˆ y = d ψ ( z 1 ) . Below, we describe the learning objective and optimization of our method in detail.

Flow Loss With the learnable projections z 0 = f ϕ ( x ) , z 1 = g φ ( y ) and any simple dynamics assumption z t = α t z 0 + β t z 1 , our flow matching objective is defined by:

In contrast to conventional flow matching (Eq. (3)) that optimizes the dynamics function given the fixed endpoints z 0 and z 1 , the loss in Eq. (5) is optimized jointly with two encoders f ϕ and g φ . Since the loss has the optimum at L flow ( θ ∗ , ϕ ∗ , φ ∗ ) = 0 when the dynamics function h θ ∗ perfectly fits to the velocity field v t , optimizing Eq. (5) guarantees that the optimal encoders f ϕ ∗ and g φ ∗ induce non-crossing target trajectories in the embedding space.

Conceptually, our approach resembles the reflow procedure in Rectified Flow [29], which straightens the target trajectories by recursively rewiring z 0 and z 1 according to the trajectory obtained by solving an ODE. In contrast, our approach straightens the trajectory by learning the encoders such that the trajectories defined by the predefined flow do not cross in the embedding space, which does not involve ODE solving and, most importantly, preserves the initial coupling ( z 0 , z 1 ) .

However, contrary to Eq. (3), we observe that optimizing the flow matching objective with learnable encoders in Eq. (5) can lead to trivial degenerate solutions. One such example is when both encoders collapse to output constant ignoring data, which produces trivial targets for the dynamic model ( e.g. , z 0 = z 1 = h θ ( · , · ) = 0 ). To avoid trivial solutions, both encoders require strong regularization to be relevant to the inputs. Fortunately, such regularization is naturally provided by the learning objective for the label decoder d ψ , which is described below.

Label Autoencoding Loss We define the label autoencoding loss as a simple mean squared error (MSE) between the label y and its reconstruction obtained by label encoder and decoder:

The primary purpose of Eq. (6) is to provide learning signals for the label decoder d ψ , which is used to decode the predicted z 1 to the label ˆ y = d ψ ( z 1 ) at inference. However, Eq. (6) also functions as a regularization to avoid trivial solutions of Eq. (5) by preventing the label encoder g φ from ignoring the input label y . Regularizing the label encoder is sufficient to prevent trivial solutions in principle, since it eliminates the trivial targets for both data encoders f ϕ and the dynamics function h θ .

Objective Function The overall objective function for the proposed framework is given by combining the two losses introduced above:

Theoretically, we can show that optimizing the combination of flow loss and autoencoding loss in Eq. 7 can indeed mitigate the problem of target trajectory crossing:

Proposition 1. There exist f ϕ and g φ that induce non-crossing target trajectory for any data pair in D while minimizing L label _ ae .

Proof. The proof can be found in App. A.1.

Proposition 2. If g φ is injective, the following equivalence holds: f ϕ , g φ and h θ minimize L flow to zero for all t ∈ [0 , 1) if and only if f ϕ and g φ induce non-crossing target trajectory and h θ perfectly fits the induced target velocity, for any data pair in D .

Proof. The proof can be found in App. A.2.

Proposition 1 ensures the existence of encoders that do not induce crossing in the target trajectory, while Proposition 2 suggests that the encoders are optimal when flow loss is minimized, assuming the label encoder is injective. Since the label autoencoding task enforces g φ to be injective, combining the two propositions implies that optimizing the objective function in Eq. (7) prevents the encoders from inducing target trajectory crossing and enables the dynamics function to accurately fit the induced trajectories.

Optimization During the optimization of Eq. (7), we find that two additional regularizations on the encoders are useful in further preventing suboptimal solutions and stabilizing training and inference.

First, although the autoencoding loss in Eq. (6) prevents degenerate encoders in principle, we find that flow matching loss in Eq. (5) can still induce an ill-behaved local minima for the data encoder f ϕ . For instance, under linear velocity field (Eq. (4)), a collapsed, constant data encoder, e.g. , z 0 = 0 , makes both intermediate state and target velocity depend only on z 1 at t ∈ (0 , 1] , e.g. , z t = tz 1 and v t = z 1 . In this case, a dynamics function that scales with time, h θ ( z, t ) = z/t , can fit the target velocity field for all t ∈ (0 , 1] although it does not yield meaningful mapping between ( x, y ) . We find that explicitly sampling t = 0 with a certain probability during training effectively resolves the issue.

Second, we empirically find that the label encoder tends to reduce the scale of the output embeddings to optimize the flow matching loss. Although this is not a fundamental problem in principle, we observe that it often affects the generalization performance by making the model prone to small numerical errors at inference, such as prediction errors in the dynamics function or discretization errors in the ODE solver. To address this, we encourage the label embeddings to repel each other, so that they construct a robust destination point for ODE solving during inference. Specifically, during training we add random noise ϵ ∼ N (0 , σ 2 ) to label embedding and let the label decoder to reconstruct original label from it, i.e. , we minimize E [ || d ψ ( g φ ( y ) + ϵ ) -y || 2 2 ] instead of Eq. (6). We empirically find these techniques helpful for better optimization, as shown in Sec. 6.3.

5 Related Work

Enhancing Efficiency of NODEs Several previous works have addressed the problem of high numbers of function evaluations (NFEs) in the forward process of Neural Ordinary Differential Equations (NODEs). Most approaches involve imposing regularization on the learned trajectory, such as penalizing higher-order derivatives [20] or incorporating kinetic regularization [12]. Similar effects can be achieved through weight decay [14], augmenting dimensions [11], or using internal solver heuristics [34]. Additionally, sampling the end time of the integration interval [13] has also been explored as a simple solution. These methods encourage the model to learn simpler trajectories, thereby effectively reducing the NFE required to solve the ODE. However, with the common choice of adaptive-step solvers, there often remain dozens of sequential function evaluations during a single training step, making the training of NODEs slow and computationally intensive.

Simulation-Free Training on Paired Dataset Some studies have explored simulation-free training methods for fitting dynamics functions on paired datasets, primarily within the context of diffusion probabilistic models (DPMs) [18]. DPMs achieve simulation-free training by learning to denoise data at multiple noise levels in parallel [30]. During inference, DPMs generate data from standard Gaussian noise through iterative denoising. These models have been applied to various vision tasks aimed at learning deterministic mappings, including segmentation [2, 4, 9], object detection [7], and image restoration tasks [27, 35, 39], among others. While these applications are impressive, they are tailored specifically to their target tasks and do not represent general methods for paired data with diverse label structures.

One notable work in this area is CARD [16], which introduced a new conditional diffusion process for classification and regression tasks, making it applicable to arbitrary regression and classification data. Although these works, based on diffusion models trained with denoising objectives, align more closely with neural SDEs rather than ODEs, we include a comparison with CARD in our main experiments to provide a comprehensive evaluation.

6 Experiments

6.1 Experimental Setup

Baselines and Datasets We validate the effectiveness of our method on various datasets for both regression and classification tasks. For baselines, we compare our method with the standard NODE [8] and previous works that utilize regularization to reduce NFEs. Specifically, we compare against STEER [13], which introduces stochasticity in the integration interval, and RNODE [12], which regularizes the norm of the velocity field. Additionally, we include a comparison with CARD [16], a classification and regression model based on diffusion. Following prior work [11, 13], we use MNIST [26], SVHN [33], and CIFAR10 [25] for image classification experiments. For regression tasks, we use UCI regression datasets [21], adhering to the protocol used by CARD.

Evaluation We report classification accuracy and root mean square error (RMSE) as the main performance metrics. To quantify computational costs, we report average per-sample NFEs along with training throughput measured by the total training iterations divided by training time. For all NODE baselines, we use the dopri5 [10] adaptive-step solver implemented in torchdiffeq [6] package for both training and inference. Additionally, we report few-step inference results using the Euler solver. For CARD, we perform few-step inference by periodically skipping intermediate steps, similar to DDPM in few-step inference [18, 38], and report the metric of full-step inference in the place of adaptive-step solver in NODEs.

Implementation Details We use the same network architecture across baselines, employing an MLP-based architecture for MNIST and UCI and a convolutional architecture for SVHN and CIFAR10. For classification tasks, we use one-hot encoded labels for y and assign the predicted label ˆ y by applying argmax on the channel dimension. To handle the significant memory requirements of training NODE-based baselines, we use the adjoint sensitivity method [8] for SVHN and CIFAR10 experiments. When using the adjoint method, we report the total NFE by summing the number of function evaluations in both the forward and backward passes. For our model, we primarily use

1

10

100

1000

10

1

10

2

RMSE

Boston

1

10

100

1000

10

1

10

2

Concrete

1

10

100

1000

10

0

10

1

Energy

1

10

100

1000

10

1

10

0

Kin8nm

1

10

100

1000

10

3

10

2

10

1

10

0

Naval

1

10

100

1000

NFE

10

1

10

2

10

3

RMSE

Power

1

10

100

1000

NFE

10

1

10

2

Protein

1

10

100

1000

NFE

10

0

10

1

Wine

1

10

100

1000

NFE

10

0

10

1

Yacht

1

10

100

1000

NFE

10

1

10

2

Ours

NODE

STEER

RNODE

CARD

a simple linear dynamics assumption, which is shown to be effective according to the analysis in Sec. 6.3. Further experimental details can be found in App. B.

6.2 Main Results

Training Cost and Performance In Tab. 1, we report the training cost and classification accuracy on MNIST, SVHN, and CIFAR10. NODE-based baselines suffer from large NFEs-ranging from tens to hundreds-during training, resulting in a low throughput. In contrast, simulation-free training method ( i.e. , ours and CARD) require only a single function evaluation per training step, significantly boosting training speed.

The trade-off for faster training in simulation-free methods is a constraint on the dynamics that can be learned. However, we observe that this reduced flexibility does not lead to significant performance degradation. Compared to NODE-based baselines, our method shows only minor degradation on MNIST and even improvements on SVHN and CIFAR10 in terms of final classification accuracy. On SVHN and CIFAR10, NODE-based baselines are trained with the adjoint sensitivity method [8] to meet memory requirements, which is known to suffer from inaccurate gradient estimation [42, 43].

Thus we hypothesize that the performance gains on SVHN and CIFAR10 may be due to the accurate gradient calculation achieved through direct backpropagation in our model, which is a positive byproduct of utilizing simulation-free training.

When compared to the diffusion-based baseline, our model tends to show better performance and faster convergence. Specifically, our model outperforms CARD on all three datasets, even compared to CARD variants trained for longer iterations. While diffusion models also benefit from simulationfree training, they are based on stochastic differential equations that induce stochastic and non-linear trajectories. We conjecture that this characteristic is not ideal for few-step inference and also contributes to slower convergence.

Few-Step Inference Ideally, our model trained with linear dynamics will yield a perfectly straight solution trajectory, which can be accurately estimated even with a one-step Euler solver. Consequently, with our linear dynamics assumption, we can significantly enhance inference speed by utilizing fewstep inference while maintaining competitive performance compared to many-step solving. To demonstrate this, we report few-step inference results with the Euler solver in Tab. 1. Our model, by avoiding crossing points in the learnable embedding space, produces a linear trajectory and thus exhibits superior few-step performance. This observation aligns with findings in flow matching models [28, 29], which highlight the advantage of linear dynamics for generating high quality samples with low inference cost.

Regression We further analyze the effectiveness of our method in regression tasks, as shown in Fig. 3. See App. D.4 for full results. Despite differences in label structure, we observe similar trends for both classification and regression tasks: our method significantly reduces computational burden during training and demonstrates superior performance in few-step inference with linear dynamics. Similar to the original NODE, our model can be effectively applied to a wide range of common supervised learning settings, regardless of whether the labels are categorical or continuous.

6.3 Analysis and Discussion

Learning Encoders with Flow Loss To support our key claim that learning encoders with flow loss allows our model to avoid crossing trajectories, we compare our model with two variants that do not utilize flow loss for learning encoders. Specifically, we consider: (1) ANODE+FM which augments the data and label to same dimensionality by zero-padding and learns dynamics function with flow matching; and (2) Autoencoder+FM which employs a two-stage approach by first learning the embedding space using an independent autoencoding objective for both data and labels, and then learning the dynamics function on the fixed embedding space 3 . To quantify the crossing points in trajectories, we train all models with the linear dynamics assumption and measure the proportion of samples where the predicted labels from a one-step Euler solver and an adaptive-step

Table 2: The effectiveness of learning encoders with flow loss. Training accuracy and the proportion of disagreement in prediction between a one-step Euler solver and an adaptive-step solver are shown. Simply augmenting dimensions (ANODE+FM) does not effectively prevent trajectory crossing. Furthermore, learning encoders without flow loss (Autoencoder+FM) also fails to preserve the original coupling due to crossing trajectories.

solver disagree 4 . We report the disagreement ratio and training accuracy in Tab. 2.

As discussed in Sec. 3, our results indicate that merely augmenting the dimension (ANODE+FM) does not resolve the issue of crossing trajectories induced by the predefined dynamics, resulting in a poor performance even on training data. Although employing more sophisticated encoders (Autoencoder+FM) partially reduces disagreements, it still fails to fit the training data properly. Such crossing trajectories can be eliminated by learning encoders with the flow matching loss, allowing our model to fit successfully to training data with high accuracy.

0k

20k

40k

60k

80k

100k

Training Steps

0.00

0.05

0.10

0.15

0.20

Flow Loss

Training Flow loss on CIFAR10

0k

20k

40k

60k

80k

100k

Training Steps

0.2

0.4

0.6

0.8

1.0

Accuracy (%)

Training Accuracy on CIFAR10

0k

20k

40k

60k

80k

100k

Training Steps

0.2

0.4

0.6

0.8

1.0

Accuracy (%)

Test Accuracy on CIFAR10

Ours

w/o Explicitly Sampling t=0

Role of the Learned Dynamics Function A potential concern is that learning a nonlinear transformation to embed data might lead the data encoder to perfectly predict the label. This issue, noted in previous work [32], warns that learning a complex input transformation could result in a collapse where the dynamics function becomes a simple identity map. To investigate whether this issue occurs in our case, we conduct a 1-NN classification using the learned data encoder on CIFAR10 image classification. The 1-NN classification accuracy with the learned data embedding z 0 is 65.66%, which is significantly lower than the accuracy of 88.47% when we utilize the learned dynamics function to obtain the predicted label embedding ˆ z 1 . This result indicates that the learned data embedding is not yet linearly separable enough, and the learned flow can further enhance accuracy. Based on this observation, we conclude that the collapse scenario, where the velocity field becomes an identity map, does not occur in our model. Instead, we find that the learned dynamics function clearly plays a role in processing the data, thereby avoiding the aforementioned pitfall.

Nonlinear Predefined Dynamics Our method embraces not only the linear dynamics but also dynamics having fixed form of z t = α t z 0 + β t z 1 in general. Here, we investigate the effect of utilizing different dynamics assumptions, including nonlinear ones. To be specific, we choose three different dynamics that is easy to implement:

Concave: α t = cos( π 2 t ) , β t = sin( π 2 t )

Linear: α t = (1 -t ) , β t = t

Convex: α t = 1 -sin( π 2 t ) , β t = 1 -cos( π 2 t ) .

We visualize the difference of dynamics and their effect on final performance in Fig. 4.

As it shows, the performance of the nonlinear variants ( i.e. , convex and concave) clearly improves with an increased number of function evaluations. For these variants, the many-step inference performance increases as we invest more NFEs and then later saturates, indicating that the model's approximation of an infinite-depth model becomes sufficiently accurate as discretization error diminishes. In contrast, our model trained on the linear dynamics shows consistently good performance, even with a few function evaluations. This behavior is somewhat expected, as the ideal linear trajectory can be already accurately inferred using a one-step Euler solver. Surprisingly, we also empirically observe that the choice of linear dynamics leads to better performance, compared to more complex choice of dynamics assumption. Thus, similar to the claims of Liu et al.(2022) [29], we believe that linear dynamics should be considered as a default choice unless specific constraints on hidden states are required.

Ablation on Optimization Techniques We conduct an ablation study on CIFAR10 dataset to investigate the effects of optimization techniques introduced in Sec. 4, and report the results in Fig. 5. Similar ablation studies were also conducted on other datasets, as detailed in App. D.3. The variant without explicitly sampling t = 0 fails to fit on training data, despite the convergence of the flow loss. Furthermore, the variant without adding noise to the label autoencoding objective succeeds

in fitting the training set, but its test accuracy significantly degrades compared to the version with noise in label autoencoding. Thus, as discussed in Sec. 4, we conclude that explicitly sampling t = 0 and introducing noise in label autoencoding effectively regularize both encoders and produces label embedding that is robust to test-time errors.

7 Limitations and Future Work

In this work, we primarily study the problem of adopting flow matching by imposing a fixed and simple dynamics assumption, observing that a fixed, closed-form equation for the intermediate state is already sufficient to bring the advantages of flow matching. Although employing simple dynamics ( e.g. linear) may seem overly restrictive, this approach can be justified in the context of Koopman operator theory, which aims to find embeddings that globally linearize the dynamics. Since our method of flow matching with linear dynamics shares a high-level concept with Koopman operator theory, exploring the relationship between the two could be a promising direction for future research. We leave further discussions in App. C.

On the other hand, while we studied simple and predefined dynamics, the form of dynamics assumption could be further generalized to be a learnable component. To be specific, introducing a learnable target dynamics that determines per-sample dynamics would be a promising future direction to study. This would be advantageous for handling inputs with varying complexity, by efficiently and adaptively allocating more computation on hard samples.

Furthermore, extending our method to a broader range of paired data applications might be a useful future direction. The reduced computational burden achieved through simulation-free training could offer several benefits of continuous-depth models to diverse applications. For example, applying our method to model compression or knowledge distillation could be particularly promising, leveraging the parameter efficiency of continuous-depth models.

8 Conclusion

In this work, we adopted a flow matching objective to achieve simulation-free training of continuousdepth models for learning deterministic mappings between paired data. We proposed learning an embedding space where flow matching occurs, which we identified as a crucial component for ensuring the validity of the target velocity field. Our proposed method significantly reduces the computational burden of training NODEs while maintaining competitive performance. Additionally, we found that our method, leveraging simple linear dynamics, demonstrates impressive performance on low-NFE regime.

Acknowledgments and Disclosure of Funding

This work was in part supported by the National Research Foundation of Korea (RS-2024-00351212 and RS-2024-00436165), and Institute of Information and communications Technology Planning and Evaluation (IITP) grant (RS-2022-II220926, RS-2024-00509279, RS-2021-II212068, and RS-2019II190075) funded by the Korean government (MSIT). We thank Youngmin Ryou (KAIST) for his valuable input in formulating the theoretical analysis presented in this work.

References

Michael S. Albergo and Eric Vanden-Eijnden. 2023. Building Normalizing Flows with Stochastic Interpolants. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net.

Tomer Amit, Eliya Nachmani, Tal Shaharabany, and Lior Wolf. 2021. SegDiff: Image Segmentation with Diffusion Probabilistic Models. CoRR , abs/2112.00390.

Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. 2019. Deep Equilibrium Models. In NeurIPS , pages 688-699.

Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2022. Label-Efficient Semantic Segmentation with Diffusion Models. In ICLR .

Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. 2019. GRU-ODE-Bayes: Continuous Modeling of Sporadically-Observed Time Series. In NeurIPS .

Ricky T. Q. Chen. 2018. torchdiffeq.

Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. 2023. DiffusionDet: Diffusion Model for Object Detection. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 , pages 19773-19786. IEEE.

Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. 2018. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 6572-6583.

Ting Chen, Lala Li, Saurabh Saxena, Geoffrey E. Hinton, and David J. Fleet. 2023. A Generalist Framework for Panoptic Segmentation of Images and Videos. In ICCV .

J.R. Dormand and P.J. Prince. 1980. A family of embedded Runge-Kutta formulae. Journal of Computational and Applied Mathematics , 6(1):19-26.

Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. 2019. Augmented Neural ODEs. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 3134-3144.

Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam M. Oberman. 2020. How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization. In ICML , volume 119 of Proceedings of Machine Learning Research , pages 3154-3164. PMLR.

Arnab Ghosh, Harkirat Behl, Emilien Dupont, Philip Torr, and Vinay Namboodiri. 2020. STEER : Simple Temporal Regularization For Neural ODE. In Advances in Neural Information Processing Systems , volume 33, pages 14831-14843. Curran Associates, Inc.

Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. 2019. FFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models. In ICLR . OpenReview.net.

Eldad Haber and Lars Ruthotto. 2017. Stable architectures for deep neural networks. Inverse Problems , 34(1):014004.

Xizewen Han, Huangjie Zheng, and Mingyuan Zhou. 2022. CARD: Classification and Regression Diffusion Models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 .

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR , pages 770-778. IEEE Computer Society.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Probabilistic Models. In NeurIPS .

Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML .

Jacob Kelly, Jesse Bettencourt, Matthew J. Johnson, and David Duvenaud. 2020. Learning Differential Equations that are Easy to Solve. In NeurIPS .

Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. 2023. The UCI machine learning repository.

Patrick Kidger. 2022. On Neural Differential Equations. CoRR , abs/2202.02435.

Patrick Kidger, James Morrill, James Foster, and Terry J. Lyons. 2020. Neural Controlled Differential Equations for Irregular Time Series. In NeurIPS .

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR (Poster) .

Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images.

Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. 1989. Handwritten Digit Recognition with a Back-Propagation Network. In NIPS , pages 396-404. Morgan Kaufmann.

Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. 2022. SRDiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing , 479:47-59.

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. 2023. Flow Matching for Generative Modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net.

Xingchao Liu, Chengyue Gong, and Qiang Liu. 2023. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net.

Calvin Luo. 2022. Understanding Diffusion Models: A Unified Perspective. CoRR , abs/2208.11970.

Bethany Lusch, J Nathan Kutz, and Steven L Brunton. 2018. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications , 9(1):4950.

Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. 2020. Dissecting Neural ODEs. In NeurIPS .

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. 2011. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning . Granada, Spain.

Avik Pal, Yingbo Ma, Viral B. Shah, and Christopher Vincent Rackauckas. 2021. Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics. In ICML .

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR .

Yulia Rubanova, Tian Qi Chen, and David Duvenaud. 2019. Latent Ordinary Differential Equations for Irregularly-Sampled Time Series. In NeurIPS .

Alvaro Sanchez-Gonzalez, Victor Bapst, Kyle Cranmer, and Peter W. Battaglia. 2019. Hamiltonian Graph Networks with ODE Integrators. CoRR , abs/1909.12790.

Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021. Denoising Diffusion Implicit Models. In ICLR .

Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. 2022. Deblurring via Stochastic Refinement. In CVPR .

Laurent Younes. 2010. Shapes and Diffeomorphisms , volume 171. Springer Science & Business Media.

Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. 2020. Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control. In ICLR .

Juntang Zhuang, Nicha C. Dvornek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James S. Duncan. 2020. Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE. In ICML .

Juntang Zhuang, Nicha C. Dvornek, Sekhar Tatikonda, and James S. Duncan. 2021. MALI: A memory efficient and reverse accurate integrator for Neural ODEs. In ICLR . OpenReview.net.

Appendix

A Proofs

In this section, we provide full proofs for the propositions in Sec. 4. We first start with formal definition of target trajectory crossing. Assume that we have a data encoder f ϕ and label encoder g φ that transform data x ∈ R d x and labels y ∈ R d y to latent z 0 , z 1 ∈ R d , where z 0 = f ϕ ( x ) and z 1 = g φ ( y ) , respectively. We also choose a predefined dynamics F ( z 0 , z 1 , t ) = α t z 0 + β t z 1 = z t . Under the assumption of d > d x , d y 5 and both α t , β t being smooth and nonzero except for t = 0 and t = 1 , we define the target trajectory crossing as follows:

̸

Definition 1 (Target Trajectory Crossing) . The encoders ( f ϕ , g φ ) are said to induce a target trajectory crossing if there exists a tuple ( t, x, y, x ′ , y ′ ) such that α t f ϕ ( x ) + β t g φ ( y ) = α t f ϕ ( x ′ ) + β t g φ ( y ′ ) for x = x ′ and y = y ′ .

̸

We now provide the proofs for each proposition in the following subsections.

A.1 Proof of Proposition 1 (Section 4)

Proposition 1. There exist ( f ϕ , g φ ) that induces non-crossing target trajectory for any data pair in D while minimizing L label _ ae .

Proof. Let the latent space constructed by a set of basis I = { e 1 , e 2 , ..., e d } . Since d > d y , we can find a label encoder g φ such that utilizes k basis J = { e 1 , e 2 , ..., e k } ( d > k ≥ d y ) and minimizes the autoencoding loss ( i.e. , g φ ( y ) = g φ ( y ′ ) if and only if y = y ′ ). Also, we can find a data encoder f ϕ such that proj span ( K ) f ϕ ( x ) = proj span ( K ) f ϕ ( x ′ ) if and only if x = x ′ , where K = { e k +1 , ..., e d } .

Then, suppose that there exists a tuple ( t, x, y, x ′ , y ′ ) such that α t f ϕ ( x ) + β t g φ ( y ) = α t f ϕ ( x ′ ) + β t g φ ( y ′ ) , i.e. , α t ( f ϕ ( x ) -f ϕ ( x ′ )) + β t ( g φ ( y ) -g φ ( y ′ )) = 0 .

Since g φ ( y ) -g φ ( y ′ ) = 0 if and only if y = y ′ and proj span ( K ) ( f ϕ ( x ) -f ϕ ( x ′ )) = 0 if and only if x = x ′ by construction, such a tuple does not exist. Therefore, there exists f ϕ , g φ such that does not induce target trajectory crossing, while minimizing the autoencoding loss.

A.2 Proof of Proposition 2 (Section 4)

Proposition 2. If g φ is injective, the following equivalence holds: ( f ϕ , g φ , h θ ) minimizes L flow to zero for all t ∈ [0 , 1) if and only if ( f ϕ , g φ ) induces non-crossing target trajectory and h θ perfectly fits the induced target velocity, for any data pair in D .

Proof. ( ⇐ = ) If ( f ϕ , g φ ) induces non-crossing target trajectory for any data pair in D , there is a well-defined target velocity d dt z t at every z t which is continuous on t . If h θ perfectly fits this target velocity for all ( z t , t ) , the flow loss is zero.

̸

̸

( = ⇒ ) We prove by contradiction. Suppose the flow loss is zero but there is a crossing trajectory, i.e. , there exists a tuple ( t, x, y, x ′ , y ′ ) that z t = z ′ t for x = x ′ and y = y ′ . Since the loss is zero for all t ∈ [0 , 1) , the dynamics function h θ must output d dt F ( z 0 , z 1 , t ) at z t , and d dt F ( z ′ 0 , z ′ 1 , t ) at z ′ t . This is a contradiction since at the point of crossing we have z t = z ′ t but d dt F ( z 0 , z 1 , t ) = d dt F ( z ′ 0 , z ′ 1 , t ) .

B Experiment Details

This section describes the implementation detail in our experiments (Sec. 6).

̸

B.1 Experiment Details for Classification Tasks

Network Architecture (MNIST) We employ an MLP-based architecture for the MNIST dataset. The data encoder maps each image to a 784-dimensional embedding, using a three-layer MLP with a hidden dimension of 784 and BatchNorm [19]. The dynamics function consists of a three-layer MLP with 2048 hidden dimension. Following NODE [8], we concatenate the time variable to the input of each layer. Consistent with common practices for NODEs [22], normalization layers are not included in the dynamics function. For the label autoencoder, class labels are converted into one-hot vectors and encoded with a single linear layer, while the label decoder utilizes a two-layer MLP with BatchNorm.

Network Architecture (SVHN, CIFAR10) Weadopt a CNN-based architecture for both the SVHN and CIFAR10 datasets. The data encoder utilizes 7 convolutional layers with a hidden dimension of 64. By default, we employ 3x3 convolution kernels, while the final two layers utilize 4x4 kernels with a stride of 2 for downsampling, resulting in the data being encoded into states with dimensions of 7x7x64. The dynamics function consists of 6 convolutional layers with 3x3 kernels and a hidden dimension of 256. At each layer, the time variable is concatenated to the input. We utilize a single linear layer for label encoding, reshaping the output to match the size of embedding. In the label decoding process, we average the feature map over the spatial dimension and apply a single linear layer.

Training We train all models for 100,000 iterations using the Adam optimizer [24] with a cosine learning rate scheduler. For all classification experiments, we utilize a batch size of 1024. Additionally, we set a maximum training time of 48 hours for the feasibility of experiments. By default, we set the learning rate to 1e-3 for MNIST and 3e-4 for CIFAR10 and SVHN. For our method, we set the ratio of explicitly sampling t = 0 to 10% for all datasets. Regarding the noise introduced to the label autoencoder, we set the standard deviation σ to 3 for MNIST, 7 for SVHN,

0

200

400

600

800

1000

1200

1400

1600

Step

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Loss

0

200

400

600

800

1000

1200

1400

1600

Step

0

1000

2000

3000

4000

5000

6000

NFE

and 10 for CIFAR10. In cases where training of NODE baselines fails, we adjust the learning rate accordingly. The failure cases of NODE baselines are illustrated in Fig. 6.

B.2 Experiment Details for Regression Tasks

Network Architecture We utilize a 2-layer MLP with a hidden dimension of 64 for the data encoder, and a 7-layer MLP with the same hidden dimension for the dynamics function. In the dynamics function, we concatenate the time variable to the input of each layer. On the label side, we employ a single linear layer for encoding and another single linear layer for decoding.

Training We trained all models for 100,000 iterations using the Adam optimizer with a constant learning rate of 3e-3. Additionally, we split the training set into a train-validation split with a ratio of 6:4, and utilized the validation metric for early-stopping. Early stopping was implemented by measuring the validation metric every 1,000 iterations and setting the patience level to 10. For our model, we sample the noise added to the label autoencoding from N (0 , 3 2 ) and the proportion of explicitly sampling t = 0 as 10%.

B.3 Experiment Details about Baselines

NODE[8] Following the tolerance values used in NODE [8] and ANODE [11], we used dopri5 [10] solver with the absolute and relative tolerance of 1e-3 for both training and inference.

STEER [13] STEER introduces a new hyperparameter b that controls the integration interval. Instead of integrating the dynamics function from 0 to 1, STEER integrates from 0 to t ∼ Uniform (1 -b, 1 + b ) during training. We follow the default configuration of using b = 0 . 99 for MNIST classification. For other tasks, we set b to 0.1 since higher values resulted in training failures.

RNODE [12] RNODE introduces two hyperparameters used for the coefficients of regularization terms. Specifically, it regularizes the Jacobian norm and the kinetic energy of the dynamics function to encourage the model to learn straight and constant-speed dynamics. The coefficient of 0.01 was generally used throughout the experiments in the original paper. Therefore, we set both coefficients for the Jacobian norm and kinetic energy as 0.01 for our experiments.

CARD [16] CARD utilizes discrete timesteps and a hyperparameter β t to schedule the noise level for diffusion modeling. Following the paper's approach, we use 1000 discrete timesteps and set a linear noise schedule from β 1 = 1e-4 to β 1000 = 0 . 02 .

B.4 Computation Resources

We conducted experiments on our internal cluster with two types of machine. We list their specifications below.

Intel Xeon Gold 6330 CPU and NVIDIA RTX A6000 GPU (with 48GB VRAM)

Intel Xeon Gold 6230 CPU and NVIDIA RTX 3090 GPU (with 24GB VRAM)

We utilized the first machine for image classification experiments, and latter one for regression experiments. We expect training our model will take about 90 min., 270 min, 230 min. for classification experiments on MNIST, SVHN and CIFAR10, respectively. For regression tasks, we estimate training cost, summing up for all splits and datasets, would cost about 288 GPU hours.

C Relation to Koopman Autoencoder

Our model with linear dynamics shares the high-level motivation with Lusch et al. (2018) [31], which aims to find an embedding space that yields linear dynamics between source and target. Regardless of the theoretical background, both flow matching and Koopman operator theory are promising approaches that seek to interpret a nonlinear system within a well-studied linear framework.

At the same time, we identify several differences between our work and the line of research based on Koopman operator theory. While those works mainly focus on a systematic way to obtain a linearized representation of the underlying nonlinear dynamics (with eigenfunctions), our work aims to find a way to learn it in a simulation-free manner, avoiding the heavy computation of forward simulation ( e.g. , which appears in L lin of Lusch et al. (2018) [31]) from an initial state to an end state. Additionally, compared to the discrete depth neural networks that have a single linear layer processor, our proposed method is generally applicable to any nonlinear dynamics that connects two endpoints z 0 and z 1 , exemplified as convex or concave as discussed in Sec. 6.3. This implies that in our case, it is possible to have a latent trajectory as a curve in non-Euclidean geometry whenever the interpolated state z t is tractable.

D Additional Results

In this section, we extend the discussions in Sec. 6 with additional results.

D.1 Additional Result of Learning Encoders with Flow Loss

We present the reconstructed images from the trained data autoencoder in Fig. 7, which are used to analyze the effectiveness of the flow loss as reported in Tab. 2. While the pretrained autoencoder reconstructs the images holistically with minimal information loss (Tab. 2), the embedding space learned without the flow loss fails to maintain the coupling with the predefined dynamics.

Input

Recon.

0

0.5

1

t

0.7

0.8

0.9

1.0

cosine similarity

ANODE + FM

0

0.5

1

t

Autoencoder + FM

0

0.5

1

t

Ours

D.2 Location of Crossing Points

In Fig. 8, we extend the analysis in Sec. 6.3 to identify where target trajectory intersections occur. Specifically, we measure the cosine similarity between target velocity and predicted velocity across t . By measuring cosine similarity instead of MSE, we can ignore the effect of absolute scale in the embedding space and accurately compare ANODE+FM, Autoencoder+FM, and our method. Model variants that do not learn encoders with flow loss particularly suffer from trajectory crossing near both endpoints, showing low prediction accuracy for the direction of target velocity. In contrast, our proposed method shows consistently high cosine similarity, mitigating the issue of target trajectory intersections near these regions. Since both endpoints are constructed from encoders, the result also supports our claim that the encoders should be trained with flow loss to effectively penalize such intersections.

D.3 Additional Results of Ablation on Optimization Techniques

We provide additional results on the optimization techniques in Fig. 9. Consistent with the findings on CIFAR10 (Fig. 5), the model without explicitly sampling at t = 0 converges to suboptimal solutions, while introducing noise during label autoencoding improves test accuracy.

D.4 Additional Experiment Result for UCI Regression Tasks

The full results on UCI regression tasks with various solvers are shown in from Tab. 3 to 9. As discussed in the main text, our method with linear dynamics assumption shows superior performance compared to baselines in low-NFE (1-2 steps) regime, which aligns with our observations in image classification experiments.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the limitations of our work in Sec. 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the full set of assumptions and proof for theoretical analysis.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all the information needed to reproduce our work.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We release the code with instructions at https://github.com/seminkim/ simulation-free-node .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specified all the experiment details either in the main text or in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We reported either 95% CI or standard deviation for UCI regression tasks. However, we were not able to repeat image classification experiment due to the limited computational resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provided sufficient information on the computer resources in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We observed no particular societal impact to address.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not have experiments with dataset having such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cited all the datasets and assets that we used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provided the code with documentation in our submission.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper revisits using neural ordinary differential equations (NODEs) for modeling deterministic maps on paired data, e.g., maps that solve regression and classification problems. The authors propose utilizing flow-matching (FM), a recent simulation-free training method for NODEs, to overcome the computational overhead of traditional NODE training and inference. The paper first presents the problems with simply using FM for learning maps on paired data since the learned map by FM is not guaranteed to preserve the coupling presented in training but only to preserve the distributions. Second, the authors propose to solve the coupling preservation problem by adding a learned encoder-decoder on the label space and an additional encoder on the data space to \"rewire\" the trajectories so that they do not cross. Then, the coupling will be preserved through FM training. The authors demonstrate the efficacy of their approach on classification and regression tasks and show that learning with FM also facilitates learning linear maps, which can be inferred in a single function evaluation - alleviating both training and inference shortcomings of traditional NODEs.\n\nSTRENGTHS:\n- The paper presents an interesting approach towards end-to-end learning of NODEs with flow matching when preserving the training set coupling is required.\n- Experiments validate the efficiency of the approach, achieving SOTA performance - both in metrics and runtime.\n\n**Presentation**\n\n- Ideas and concepts in the paper are often presented with provided intuition, guiding the reader to the logic behind the algorithmic choices.\n\nWEAKNESSES:\n**Presentation**\n\n- Although intuitive explanations were mentioned as a strength, in some cases, they come in short. The paper lacks formal and rigorous explanations of the method and experimental settings. For instance:\n  - Adding noise to labels (L164) - I am unsure I understand the setting here. Is the noise added to the ground truth labels in the L2 loss? Or are they added to the embeddings $z_1$? An equation explicitly stating the noise addition would greatly help clarify this. \n- In the preliminaries section, the introduction of flow-matching is inaccurate and lacks the main point that FM does not regress to the marginal velocity field but rather regresses to a conditional velocity field, named linear dynamics in the paper, while the marginal is not necessarily linear. \n\n\n\n**Method**\n\n- A naive solution to the crossing trajectories problem, would be to augment the dimension of the learned flow such as suggested and discussed in [1,2] and is in a sense similar to training a conditional model, where the condition is the initial point. The paper misses the sanity check and comparison with the most naive baseline, combining augmented NODEs with flow matching. I wonder, if one only learns $d_\\psi$ and $g_\\varphi$, $f_\\phi$ is set to be identity, and the learned flow is trained on an augmented space with the initial point, would that achieve better/worse results? According to [2], augmenting the flow model with the initial point would solve the crossing trajectories problem and allow the couplings to be preserved. \n- I find the motivation for using NODEs for regression and classification rather weak when the learned map is linear (i.e., solved by a single NFE). In a sense, the motivation for using NODEs was to utilize \"infinite\" depth, in a shared weights manner, by learning a time-dependent function. But in the case presented in the paper, most of the \"heavy lifting\" in learning the representation is already done by the encoder-decoder neural nets, while the learned velocity could be thought of as a last layer represented by some non-linear function, separating the data, which is not linearly separable as shown by experiments in section 6.3, since as it appears from Table 1, there's not much of a performance gain in using NFE$>1$.\n\n\n\n[1] [Augmented Neural ODEs](https://arxiv.org/abs/1904.01681), Dupont et al. (2019)\n\n[2] [Augmented Bridge Matching](https://arxiv.org/abs/2311.06978), De Bortoli et al. (2023)\n\nQUESTIONS:\n- Can the authors provide a comparison to augmented NODEs+FM training? or provide and explanation as to why they think this may not work?\n\n- Regarding the second point in the method weaknesses, I would be happy to extend the discussion on this point and think if there's some experiment to justify the use of NODEs here better.\n\n\nTypos:\n- In Figure 1 (c), the title says `NFE=1`, but trajectories are curved. Could there be a typo in the plot title?\n- Figure 1 caption labels (c) and (d) are mislabeled as (b) and (c).\n- Table 1, CIFAR10, RNODE, Throughput seems like a typo, value is like simulation free methods while it should be lower.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach for instantiating a flow matching method for paired data $(x, y)$ without relying on iterative ODE solvers. The method uses an input encoder with a pair of target decoder and encoder to project the original data into a latent space. By imposing the form of the dynamics in latent space, the trajectory of the latent vector between $x$ and $y$ can be represented using a closed-form equation. This approach demonstrates competitive or superior performance compared to methods that require iterative ODE solvers and diffusion-based models.\n\nSTRENGTHS:\n- The idea and motivation are clearly exposed, with sufficient detail to understand the intuition behind the approach.\n- This work is a good example of the effort to link intuitions from various domains together to create a clear and simple method.\n- The results are interesting, providing evidence that nonlinear dynamics in latent space can be eliminated without compromising prediction accuracy.\n\nWEAKNESSES:\n- I found that the presentation of the main Table 1 is not entirely clear to me. The main messages (smaller NFE vs. competitive results) are conveyed, but it is still a bit confusing to see the number of NFEs directly in the table for the methods requiring only one NFE.\n- Minor issues: many abbreviations are defined multiple times throughout the text. Please be careful if the authors used a writing assistant to help with the drafting.\n\nQUESTIONS:\n- This is an extra point for me: I am curious about the differences between this approach and discrete depth neural networks. If the dynamics between $z_0$ and $z_1$ can be expressed with an interpolation, we could say that this architecture is very similar to the discrete depth neural networks, where the processor in the latent space is a single linear layer with or without a skip connection. An example of such an architecture can be seen in Lusch et al. (2018). Did the authors try to compare the proposed method with such an architecture?\n\nReferences: \n- Lusch et al. (2018), Deep learning for universal linear embeddings of nonlinear dynamics, Nature Communications, https://www.nature.com/articles/s41467-018-07210-0\n\nLIMITATIONS:\nI confirm that the authors have addressed sufficiently the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to use the flow matching loss, which directly matches the dynamics of a neural ODE (NODE) model to the pre-defined (simple) vector field, for supervision tasks. While the flow matching with simple linear vector fields is efficient, it cannot work well for supervision tasks because the paired data structure can require a crossing trajectories, which cannot be achieved by using the ODE + linear vector field. To overcome this issue, the authors propose to use the input and label encoders, and learning simple linear vector field in the latent space. To prevent learning trivial dynamics (e.g., ignoring data), they also introduce the label decoder and label reconstruction loss, which makes the output latent signal to be meaningful. The authors validate their approach with various supervision tasks, and show the proposed framework outperforms other competitors in terms of the cost-performance trade-off.\n\nSTRENGTHS:\nAt first glance, the proposed method seems to be too simple; using the auto-encoders to match the latent dynamics with simple one is somewhat straight-forward. However, such a simple approach can remarkably improve NODE-based models for supervision tasks, with a significant margin compared to baseline NODEs.\n\nThe paper is extremely well-written and easy to follow.\n\nWEAKNESSES:\nWhile this paper provides useful insights into NODEs, such as the crossing trajectory problems and the use of latent dynamics, these are well-known topics within the community of NODEs. Therefore, I believe this paper should be evaluated based on its practical application rather than its theoretical aspects. \n\nFrom the practical perspective, while the proposed technique outperforms some popular NODE-based supervision models, it exhibits significantly lower performance compared to the standard no-NODE-based baselines; the classification accuracy of 88.89% for CIFAR10 is not great. The fact that all NODE-based models failed at this task, and that the proposed model is better at least, does not bring much satisfaction (note that diffusion-based CARD model can estimate the uncertainty, though its performance is not great also). \n\nI do not think that every model needs to achieve SOTA performance to be published. However, to at least have readers consider trying the proposed NODE-based framework instead of conventional finite-depth models (given that supervision is generally not approached using NODEs), I believe the proposed model should at least be compared to similar-sized non-NODE-based MLP and ResNet models.\n\nQUESTIONS:\nThe authors experimentally demonstrated that input/label encoders do not become arbitrarily complicated (i.e., learn all the information on the given task), thus the latent dynamics play a sufficiently significant role for solving the task. Can the authors intuitively explain how this is possible?\n\nLIMITATIONS:\nThe authors mention some limitations (e.g., assumming the underyling dynamics is linear) on the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops the Flow Matching (FM) algorithm to connect paired data. Due to the issue of crossing trajectories, FM in the data space cannot perfectly match associated pairs. To address this, the authors perform FM in an embedded space. Ultimately, they encode source and target data through an encoder end-to-end and learn FM loss in the embedded space. The main application is image classification.\n\nSTRENGTHS:\nThe paper begins with a reasonable motivation and is well-presented, making it easy to follow.\n\nWEAKNESSES:\n- While the motivation is to avoid incorrect pair connections due to crossing trajectories in the data space, embedding data into a latent space does not guarantee the prevention of trajectory crossings. I recommend authors to visualize (on toy data, or on real data) that the proposed method experimentally prevents crossing trajectories. Moreover, it would be valuable if authors show trajectory crossing occurs in real world scenarios by comparing the results with FM. For example, it would support the motivation and also the proposed method if the proposed method outperforms FM in connecting paired data (in real-world data).\n\n- The improvement in accuracy may be due to the additional embedding networks rather than resolving the issue of trajectory crossings. Therefore, it is recommended that the authors conduct experiments and visulalize/demonstrate crossing issue can be resolved in a learned latent space on simple low-dimensional toy data.\n\n- The comparison group is weak. There are algorithms like ANODE [1] and FFJORD [2] designed to connect paths with simpler trajectories. It would be beneficial to compare these algorithms as well. \n\n[1] Augmented Neural ODEs, NeurIPS, 2019.\n[2] FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models, ICLR, 2019.\n\nQUESTIONS:\n- In the real world scenario (on the classification/regression task presented in the paper), I am not sure if the crossing trajectory is crucial point that influence the performance. I am curious that if authors can compare their models with FM (well, there may be issues on matching source and target dimensions equal when impelmenting FM).\n\nLIMITATIONS:\nSee the Weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}