DFM: Interpolant-free Dual Flow Matching

Authors: denis  gudovskiy
tomoyuki  okuno
yohei  nakata
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-21 02:12:04

Abstract: Continuous normalizing flows (CNFs) can model data distributions with expressive infinite-length architectures. But this modeling involves computationally expensive process of solving an ordinary differential equation (ODE) during maximum likelihood training. Recently proposed flow matching (FM) framework allows to substantially simplify the training phase using a regression objective with the interpolated forward vector field. In this paper, we propose an interpolant-free dual flow matching (DFM) approach without explicit assumptions about the modeled vector field. DFM optimizes the forward and, additionally, a reverse vector field model using a novel objective that facilitates bijectivity of the forward and reverse transformations. Our experiments with the SMAP unsupervised anomaly detection show advantages of DFM when compared to the CNF trained with either maximum likelihood or FM objectives with the state-of-the-art performance metrics.

Main paper: 
 introduction: 
 Discrete-time (DNF) and continuous-time (CNF) normalizing flows have been previously extensively studied and compared in details [Ruthotto and Haber, 2021] . With the pros and cons in each approach, we are motivated to apply CNFs in real-world generative and density estimation applications. Theoretically, an infinite-length architecture with arbitrary parameterization can lead to a significant advantages of CNFs when compared to DNF shortcomings. Meanwhile, CNFs with the ordinary differential equation (ODE) integration step endure higher computational complexity, numerical instabilities and approximation errors [Chen et al., 2018] . In particular, the latter is crucial in density estimation which requires accurate estimates of the Jacobian matrix trace.Recent flow matching (FM) framework [Lipman et al., 2023] simplifies the training phase in CNFs by introducing a new regression objective. This objective minimizes mean square difference of a parameterized vector field model and an interpolated vector field between two data distributions. While the former is a conventional neural network with time-dependent conditioning, the latter relies on certain assumptions about modeled data distributions. As a result, there is an extensive line of research that proposes various forms of the interpolated vector fields and the corresponding probability paths. We summarize recent works in Table 1 with the formal introduction in Section 2. Diffusion models [Song and Ermon, 2019, Ho et al., 2020] that solve stochastic differential equations (SDEs) can also be generalized using the FM framework [Tong et al., 2024] .In this paper, we analyze current interpolation-based FM approach and its inherent limitations i.e. the Gaussian probability path assumption between two data distributions [Lipman et al., 2023] . To address this limitation, we propose a novel interpolant-free dual flow matching (DFM) method. Specifically, we accomplish interpolant-free FM using an additional parameterized reverse vector field model. For simplicity, we model the reverse vector field using exactly the same architecture as for the forward one. Then, we optimize our DFM using an objective that enforces transformation bijectivity of the modeled forward and the reverse vector fields.Table 1 : Generalization of probability paths for diffusion and flow matching methods by the [Tong et al., 2024] framework. Unlike ours, these methods rely on an interpolation of the probability paths.Probability Path q(z) µ t (z) σ tVar. Exploding SDE [Song and Ermon, 2019] q(x 1 )x 1 σ 1-tVar. Preserving SDE [Ho et al., 2020] q(x 1 )α 1-t x 1 1 -α 2 1-tFM [Lipman et al., 2023] q(x 1 ) tx 1 tσ -t + 1 Rectified FM [Liu et al., 2023] q(x 0 )q(x 1 ) tx 1 + (1 -t)x 0 0 Var. Preserving FM [Albergo and Vanden-Eijnden, 2023 ] q(x 0 )q(x 1 ) cos(πt/2)x 0 + sin(πt/2)x 1 0 I-CFM [Tong et al., 2024] q(x 0 )q(x 1 )tx 1 + (1 -t)x 0 σ2 CNF Preliminaries and Prior FM Methods Continuous normalizing flows. We follow Lipman et al. [2023] and Tong et al. [2024] notation.We consider a pair of data distributions q(x 0 ) and q(x 1 ) over R D with densities p(x 0 ) and p(x 1 ), respectively. Often, the p 0 = p(x 0 ) density represents a known prior distribution while the data density p 1 = p(x 1 ) is not given with only access to an empirical q(x 1 ) and p 1 needs to be estimated.Then, there are a probability density path p : [0, 1] × R D → R> 0, which is a time-dependent probability density function p t (x) with t = [0, 1] such that p t (x)dx = 1, and a Lipschitz-smooth time-dependent vector field u :EQUATIONwhere ϕ t (x) is the ODE solution with ϕ 0 (x) initial condition that transports x from time 0 to time t.On the other hand, ϕ t induces a push-forward p t = [ϕ t ] # (p 0 ) that transports the density p 0 from time 0 to time t. The time-dependent density p t is characterized by the continuity equation written byEQUATIONwhere the divergence operator, div, is defined as the sum of derivatives of f t (x) ∈ R D w.r.t. all elements x d or, simply, the Jacobian matrix trace: div(f (x)) = D d=1 ∂f d (x)/∂x d = Tr(J ). The vector field u t (ϕ t (x)) is often modeled without ϕ t (x) invertability requirement by an arbitrary neural network v θ (t, x t ) with the learnable weight vector θ. Then, the continuity equation in (2) for (1) neural ODE can be written using the instantaneous change of variables [Chen et al., 2018] asEQUATIONThe (3) neural ODE can be solved both for a point x 0 and the log-likelihood change as integrationx 0 log(p 1 /p 0 ) = t=0 t=1 v θ (t, x t ) -Tr(∂v θ (t, x t )/∂x T t )dt, and initiallyx t log(p 1 /p t ) = x 1 0 . (4)Then, the CNF maximizes likelihood (MLE) during training and uses its estimate at the evaluation asEQUATIONwhere log p 0 is the likelihood of a point x 0 from (4) when evaluated using a known prior q(x 0 ).Flow matching training. Typically, solving the ODE (4) for the MLE objective ( 5) is computationally expensive [Grathwohl et al., 2019 , Zhuang et al., 2020] . The FM framework [Lipman et al., 2023] proposes an alternative objective that regresses the v θ (t, x t ) to u t by conditioning the latter by a vector z = x 1 . This has been extended by the conditional FM (CFM) framework [Tong et al., 2024] where the u t (x|z) and p t (x|z) are conditioned on a more general z ∼ q(z) such that the marginal probability density path and the corresponding marginal vector field are defined asEQUATIONThe Gaussian conditional probability path in (6) has the unique conditional vector field such thatEQUATIONwhere the mean µ t (z) and the standard deviation σ t (z) functions parameterize the path p t (x|z).Finally, the CFM objective for the simplified CNF training using (7) result can be written as arg min θ L CFM := E t∼U (0,1),xt∼pt(x|z),x∼N (0,I),z∼q(z) ∥v θ (t,EQUATION)Recently proposed µ t (z) and σ t (z) for interpolation of Gaussian conditional probability path p t (x|z) in ( 7) are given in Table 1 . The above CFM framework is inherently limited by the (6) interpolation assumption and (7) path choice. To overcome this, Chen and Lipman [2024] extend the Euclidean FM to Riemannian manifolds. Kapusniak et al. [2024] introduce the metric FM that learns parameterized interpolants. There are attempts to support FM for categorical data Stark et al. [2024] , Cheng et al.[2024], Campbell et al. [2024] and mixed categorical-continuous data [Dunn and Koes, 2024] . In this paper, we are motivated by a different perspective: is it feasible to avoid the interpolation in FM framework with a reasonable computational cost?3 The Proposed Interpolant-free Dual Flow MatchingFigure 1 : The CFM (top) regresses the Gaussianinterpolated forward vector field by a neural network with the affine transformation ϕ t (x). Our DFM (bottom) has two neural networks with the free-form transformations with only the bijectivity objective x t = ϕ -1 λ (ϕ θ (x t )) for an arbitrary vector field and a probability path.Dual CNF via a reverse vector field. Let's extend the CNF framework that is defined in (1-4). We introduce a dual CNF where its first part, implemented as the above non-invertible v θ (t, x t ), approximates the vector field u t (ϕ t (x)). In addition, we employ an extension v λ (t, y t ) with learnable parameters λ that models a reverse vector field model u t (ϕ -1 t (y)). In other words, there is the forward transformation x t = ϕ t (x) and the inverse transformation y t = ϕ t (y) = ϕ -1 t (x) of the bijective map ϕ t . Then, we can reformulate the equations (4) for v λ (t, y t ) model with minor modifications. The neural ODE in (3) can be solved in reverse simultaneously for a point x 1 and the log-likelihood change with the initial condition y t ∼ q(x 0 ) and log(p 0 /p t ) = 0 as integrationx 1 log(p 0 /p 1 ) = t=1 t=0 v λ (t, y t ) -Tr ∂v λ (t,yt) ∂y T t dt. (9)Interestingly, the (9) approach with the modified maximum likelihood L MLE (λ) is known in the DNF literature as a reverse divergence objective [Papamakarios et al., 2021] . When the target data p 1 cannot be analytically evaluated, the ( 9) is impractical for CNF training with the MLE objective.Interpolant-free DFM. On the other hand, the proposed dual CNF with the reverse model in ( 9) can be used for the interpolant-free flow matching. Instead of the less expressive affine transformation (ϕ t (x|z) = µ t (z)+σ t x, x ∼ N (0, I)) induced by the Gaussian interpolation (7), the proposed DFM only requires bijectivity of the free-form transformations ϕ t and ϕ -1 t produced by, correspondingly, the forward v θ (t, x t ) and the reverse v λ (t, y t ) vector field models.Then, the proposed dual CNF with the bijective ϕ t can be expressed as ODEs expressed byEQUATIONAssuming the ϕ t (y) = ϕ -1 t (x) bijectivity in a neighborhood of t for x and y, (10) can be rewritten using the univariate inverse function theorem by substituting the top to bottom asEQUATIONThe ( 11) objective can be achieved by minimizing the cosine distance between two unit vectors asEQUATIONwhere this loss with vector normalization is more numerically stable in practice.Once the (12) loss is minimized, the density estimation i.e. log p1 can be performed using the conventional MLE approach (5) without the extension v λ (t, y t ). On the other hand, the extension can be used to improve log p1 by integrating (9) from t = 1 to t = 0. We use the latter strategy. While we yet to accomplish sampling experiments, we expect DFM to outperform previous FM methods due to the enforced bijectivity which is a common issue in ODEs [Gholami et al., 2019] . 
experiments: 
 Benchmark. We employ real-world SMAP [Hundman et al., 2018] time series benchmark for unsupervised anomaly detection. The soil moisture active passive satellite (SMAP) dataset contains soil samples and telemetry information from the Mars rover with 135K and 428K data points in the training (without anomalies) and test sets, respectively. SMAP data has 25 data dimensions collected from 55 entities. We follow Su et al. [2019] and transform the regression task into a classification task using sliding windows (window size = 8) and replication padding [Tuli et al., 2022] .Flow models. We report experimental results for the Glow-type DNF [Kingma and Dhariwal, 2018] from [Gudovskiy et al., 2024] with the state-of-the-art baselines. Second, we experiment with the vanilla CNF from Section 2 and the CNF trained using the CFM framework i.e., the FM from [Lipman et al., 2023] and I-CFM from [Tong et al., 2024] . All CNF models have exactly the same U-Net architecture [Ronneberger et al., 2015] , learnable N (µ, σ 2 I) prior and identical evaluation using (5).Evaluation. We follow Su et al. [2019] and report precision (P), recall (R), AuC and F 1 score. We provide results when we solve ODE using the fixed-step (F) Euler method with 4 steps and the variable-step (V) Dopri5 method (atol=1e-1, rtol=1e-2) from the [Zhuang et al., 2021] library. We use the Hutchinson stochastic estimator of the Jacobian matrix trace [Hutchinson, 1990] . Quantitative results. We compare flow models to other popular baselines: OmniAnomaly [Su et al., 2019] , CAE-M [Zhang et al., 2021] , TranAD [Tuli et al., 2022] . It is common in these baselines to train and evaluate a separate model for each SMAP entity. In contrast, all our flow models use a single model for all entities in Table 2 i.e. they are entity-unconditional.We can derive several important conclusions from Table 2 results. First, continuous-time normalizing flows models, if properly trained and evaluated, are able to outperform discretetime normalizing flows as well as other nonflow models in this density estimation task. Second, recent integration-free FM training methods using (8) perform similarly or better than the CNF trained by computationally-expensive MLE from (5). Third, the proposed DFM significantly outperforms prior FM methods with only the 2× complexity increase. In particular, DFM increases the non-saturated metrics such as precision and F 1 score by, correspondingly, 6.5 (88.2% →94.7% ) and 3.1 (93.3% →96.4% ) percentage points. 
conclusions: 
 In this paper, we analyzed limitations of the interpolation-based flow matching framework that allows to efficiently train a CNF model. To address the limitations, we proposed the interpolant-free dual flow matching method. Our experiments with the SMAP benchmark showed that our DFM achieves state-of-the-art results for the entity-unconditional unsupervised anomaly detection. In future, the DFM objective (11) and the practical loss (12) can further be extended to multivariate case. 
 
 Appendix: Preprint. 


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: This paper proposes an interpolant-free flow matching method, DFM, using a reverse vector field and conducts experiments to evaluate its effectiveness.
main_review: main_review: ##### Summary of Strengths

- The writing is clear and easy to follow. The related works are well-organized.
- The proposed DFM method is grounded in solid theoretical proof.

##### Summary of Weaknesses

- There are no significant weaknesses. It would be beneficial to compare the proposed method with other flow matching techniques in the domain of image generation.
title: title: Good idea to use a reverse network.
main_review: main_review: This paper proposes using a neural network to replace the fixed reverse process during Flow Matching. Experiments on anomaly detection show that it is comparable to baseline Conditional Normalizing Flow and Flow Matching methods. The precision and the F1 score are significantly higher on the SMAP dataset. I still have several questions:
1. Regarding motivation, I don't understand the logic of "solving an ODE is expensive". Isn't it more expensive to learn an additional neural network? If so, perhaps the last sentence in Section 2 is more suitable.
2. I don't understand the advantage of the cosine distance. Why does equation 12 use cosine distance and CFM in equation 8 use L2 distance? A more detailed discussion of selecting a good metric could be extended.
3. The reproducibility and fairness of comparisons are potential issues. More experimental settings should be provided. For example, can DFM be compared with baselines using the L2 metric?
Overall, the paper is well presented. More datasets can be studied to ensure the method's generality and increase its significance.
title: title: An interesting extension of the flow matching framework
main_review: main_review: DFM: Interpolant-free Dual Flow Matching

The paper proposes an objective to train flow matching models without needing to pre-specify a probability path, by parameterizing both the forward path and the reverse path.

Strength: 1. The paper provides an interesting extension of the flow matching framework. 2. Some empirical evidence was provided.

Weakness: 1. Parameterizing both the forward path and the reverse path leads to some computational overhead. 2. It is not clear how the method performs in some more complex cases, e.g. in high dimensional generative modeling tasks. 3. One benefit of flow matching is, by specifying the desired probability path, it is possible to impose some desired properties on the model, for instance preferring straight paths; see e.g. [1]. It is not clear what the learned probability paths of the proposed method are like.

[1] Improving and generalizing flow-based generative models with minibatch optimal transport, Tong et al.
title: title: Dual Flow Matching to improve continuous normalizing flows by challenging the Gaussian probability path assumption in Flow Matching approach
main_review: main_review: # Strengths
1. **Strong Theoritical Foundation**: The mathematical introduction for CNF and FM is very clear. It is easy to why interpolation is the bottleneck and how DFM fills this gap.
2. **Proposed Approach**: The novel objective function of enforcing bijectivity is mathematically grounded and straightforward.
3. **Experiments**: The empirical experiments demonstrate the consistent improved performance of DFM over the baselines.
4. **Overall Paper**: The paper is well written, with clear notations and definitions. It was easy to follow.

# Weaknesses
I think the paper is solid for the extended abstract track.

# Suggestions
1. It would be nice to have some visualizations of the flow, maybe in Appendix.