Training-free linear image inverses via flows
Authors: Ashwini Pokle
Matthew J. Muckley
Ricky T. Q. Chen
Brian Karrer
Venue: Accepted by TMLR ()

Abstract: Solving inverse problems without any training involves using a pretrained generative model and making appropriate modifications to the generation process to avoid finetuning of the generative model. While recent methods have explored the use of diffusion models, they still require the manual tuning of many hyperparameters for different inverse problems. In this work, we propose a training-free method for solving linear inverse problems by using pretrained flow models, leveraging the simplicity and efficiency of Flow Matching models, using theoretically-justified weighting schemes, and thereby significantly reducing the amount of manual tuning. In particular, we draw inspiration from two main sources: adopting prior gradient correction methods to the flow regime, and a solver scheme based on conditional Optimal Transport paths. As pretrained diffusion models are widely accessible, we also show how to practically adapt diffusion models for our method. Empirically, our approach requires no problem-specific tuning across an extensive suite of noisy linear inverse problems on high-dimensional datasets, ImageNet-64/128 and AFHQ-256, and we observe that our flow-based method for solving inverse problems improves upon closely-related diffusion-based methods in most settings.

Published in Transactions on Machine Learning Research (06/2024)
Training-free Linear Image Inverses via Flows
Ashwini Pokle
Carnegie Mellon University
apokle@andrew. cmu. edu
Matthew J. Muckley
FAIR at Meta
mmuckley@meta.. com
Ricky T. Q. Chen
FAIR Meta
rtqicchenmeta..c com
Brian Karrer
FAIR at Meta
briankarrergmeta.coo
Reviewed OpenReview: http:///htttsss/ppeeeereeee
Abstract
Solving inverse problems without any training involves using pretrained generative model and making appropriate modifications to the generation process to avoid fine-tuning of the generative model. While recent methods have explored the use of diffusion models, they still require the manual tuning of many hyperparameters for different inverse problems. In this work, we propose a training-free method for solving linear inverse problems in non-blind setting by using pretrained flow models, leveraging the simplicity and efficiency of Flow Matching models, using theoretically- justified weighting schemes, and thereby significantly reducing the amount of manual tuning. In particular, we draw inspiration from two main sources: adopting prior gradient correction methods to the flow regime, and solver scheme based on conditional Optimal Transport paths. As pretrained diffusion models are widely accessible, we also show how to practically adapt diffusion models for our method. Empirically, our approach requires no problem-spe specific tuning across an extensive suite of noisy linear inverse problems on high-dimensional datasets, ImageNet-64/122 and AFHQ-256, and we observe that flow-based method for solving inverse problems improves upon closely-related diffusion-based methods in most settings.
1 Introduction
Solving inverse problem involves recovering clean signal from noisy measurements generated by known degradation model. Many interesting image processing tasks can be cast as inverse problem. Some instances of these problems are super-resolution, inpainting, deblurring, colorization, denoising etc. Diffusion models or score- based generative models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song & Ermon, 2019; Song et al., 2021d) have emerged as a Ieading family of generative models for solving inverse problems for images (Saharia et al., 2022b;a; Wang et al., 2022; Chung et al., 2022a; Song et al., 2022; Mardani et al., 2023). However, sampling with diffusion models known to be slow, and the quality of generated images is affected by the curvature of SDE/ODE solution trajectories (Karras et al., 2022). While Karras et al. (2022) observed ODE sampling for image generation could produce better results, sampling via SDE is still common for solving inverse problems, whereas ODE sampling has been rarely considered, perhaps due to the use of diffusion probability paths.
Work done during internship with FAIR Meta

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Gaussian deblur
(b) 2x Super-resolution
(c) Inpainting
(d) Inpainting
Figure 1: Corrected images by solving linear inverse problems with flow models. For each pair of images, we show the noisy measurement (left) and the reconstruction (right).
Continuous Normalizing Flow (CNF) (Chen et al., 2018b) trained with Flow Matching (Lipman et al., 2022) has been recently proposed as a powerful alternative to diffusion models. CNF (hereafter denoted flow model) has the ability to model arbitrary probability paths, and includes diffusion probability paths special case. Of particular interest are Gaussian probability paths that correspond to optimal transport (OT) displacement (McCann, 1997). Recent works (Lipmmn et al., 2022; Albergo & Vanden-Eijndenn 2022; Liu et al., 2022; Shaul al., 2023) have shown that these conditional OT probability paths are straighter than diffusion paths, which results in faster training and sampling with these models. Due to these properties, conditional OT flow models are an appealing alternative to diffusion models for solving inverse problems.
In this work, we introduce a training-free method to utilize pretrained unconditional flow models for solving linear inverse problems. Our approach adds gradient- based adaptation term the unconditional vector field that takes into account knowledge from the degradation model and converts algorithm that extends IIDMM (Song et al., 2022) gradient adaptation ODE sampling with affine Gaussian probability path. Given the wide availability of pretrained diffusion models trained with diffusion probability paths, we also present way to convert these models to other affine Gaussian probability paths. Empirically. we observe images restored via a conditional OT path exhibit perceptual quality better than that achieved by the model's original diffusion path, as well as recently proposed diffusion approaches such as IIGDM (Song et al., 2022) and RED-Diff (Mardani et al., 2023), particularly noisy settings. To summarize, our key contributions are:
a conditional vector field. Specifically, we introduce
We present training-free free approach to solve linear inverse problems in non-blind setting that can be applied any continuous-timg time diffusion flow model under affine Gaussian probability paths that extends IIGDM gradient adaptation to this more generic setting.
We explain the subtleties converting models between different affine Gaussian probability paths. Specifically, we enable the of pre-trained continuous-time diffusion models with conditional OT probability paths by adjusted initialization procedure.
We demonstrate that images restored via our ODE algorithm using conditional OT probability paths have perceptual quality that is largely on par with, or better than that achieved by diffusion probability paths, and other recent methods like IGDDM and RED-Diff, without the need for problem-specific hyperparameter tuning.
2

---

Published in Transactions on Machine Learning Research (06/2024)
2 Preliminaries
We introduce relevant background knowledge and notation from conditional diffusion and flow modeling, as well as training-free inference with diffusion models.
Notation. Both diffusion and flow models consider two distinct processes indexed by time between [0. 1] that convert data to noise and noise to data. Here, we follow the temporal notation used prior work (Lipman et al., 2022) where the distribution at 1 is the data distribution and 0 is a standard Gaussian distribution. Note that this is opposite of the prevalent notation used in diffusion model literature (Song & Ermon, 2019; Ho al., 2020; Song et al., 20222021a We let æ denote a real-valued vector at time t, without regard to which process (i.e., diffusion or flow) it was drawn from. The probability density for the data to noise process is denoted and the parameterized probability density for the noise to data process is denoted po- Expectations with respect to q are denoted via E., where the relevant random variables are noisy æ,, clean data Ø1, and conditioning y with density g( 31, y) = q g<<///01 (21, y))(())))))))))))))))))) 1. y). Here g( y) is unknown and (x y) will be a modeling choice. Conditional diffusion or flow models aim to produce samples #1 q(2111/)) ( We generally keep function arguments of implicit (i.e. [ 1) is informally written f(x))).)
Conditional diffusion models. Suppose we have samples Φ1 (e.g. an image) and conditioning (e.g. a distorted image) drawn from a data distribution ((11, y). Conditional diffusion models use latent variables T0:1 = { It E [o. 1)} to model the joint distribution pe(20:1 31 for the noise to data process pa. The data to noise process approximates the posterior g ( 1 y) and is defined as a Markov chain² that adds Gaussian noise to data, which continuous time satisfies a stochastic differential equation (SDE)(Song & Ermon, 2019; Ho et al., 2020; Song et al., 2021d). The parameters of pe are learned via minimizing regression loss derived from the variational bound negative log- likelihood with respect to æi:
= L.¹ (() (qre)) wtt.ee/lll............... [ v) x2 11] dl,
Laiffusson
where w()) are positive weights (Kingma et al., 2021; Song et al., 2021b; Kingma & Gao, 2023), and 2i is deterministic parametrized denoiser. The optimal solution for 2i with the squared L2 error in Eq. (1) is Ez, fee (x. [2t (m v). For brevity as well as ease of readability, henceforth we will typically write the expectations with respect to q such as the one that appears in Eq. (1) in short as EE,, Many equivalent parameterizations exist for the loss in Eq. (1) and have known conversions to denoising. Sampling using pe proceeds by starting from Do ~ po ( ly) and integrating the SDE using 3i to f 1. If po (xoly) = ((ro y). the SDE integrated exactly, and ii (are. y) = E.g (an |xx, y). the resulting Ø1 q(x1 10111 (y) as desired.
Conditional flow models Alternatively, continuous normalizing flow models (Chen et al., 2018b) define the data generation process through an ODE. This leads to simpler formulations and does not introduce extra noise during intermediate steps of sample generation. Recently, simulation free training algorithms have been designed specifically for such models (I (L-ppman et al., 2022; Liu et al., 2022; Albergo & Vanden-Fijndenn 2022), an example being the Conditional Flow Matching loss (Lipman et al., 2022),
1/'' Eiz (m) v),
Leti i
(p)))))))) .. [|]((, y) - v( V, 2111 dt,
(2)
where 6(,,, y) denotes a parameterized vector field defining the ODE
dx: dt (zx, y),
(3)
and data-condittonal vector field v( y, x1) is determined by modeling choice g( [ar, trained perfectly, the marginal distributions of x, from ODE integration, denoted pa( ly). will match the marginal distributions of qa (gy). Hence sampling from g( ly) as desired can be achieved by sampling initial value æy ~ q( ly) and integrating the ODE from t' to 1. Typically, one samples from t' 0 since (x is tractable distribution.
In discrete time, we can write this joint distribution as po (0 1l)) (mool))) I. po (ax A (m, y) where A denotes an appropriate step size of time discretization in [o, 1].
II discrete time, the forward process 9 11, y) = II, ( 12.
3

---

Published in Transactions on Machine Learning Research (06/2024)
Gaussian probability paths. The time-dependent densities g( y) are referred to conditional probability paths. We focus on the class of afiine Gaussian probability paths of the form
((1122122 y) = ( = -(
(4)
where non-negative or and o, are monotonically increasing and decreasing respectively. This class includes the probability paths for conditional diffusion as well as the conditional Optimal Transport (OT) path (Lipman et al., 2022), where a = and Ø; = 1 --6 L. The conditional OT path used by flow models has been demonstrated to have good empirical properties, including faster inference and better sampling in practice, and has theoretical support in high- dimensions (Shaul al., 2023). As emphasized in Lin et al. (2023), desirable property for probability paths, obeyed by conditional OT but not commonly used diffusion paths, is to ensure g( is known (i.e. (C, I)), as otherwise one cannot exactly sample To which can add substantial error. When using these affine Gaussian probability paths with conditional flow model, one sets (Lipman et al., 2022)
da dt 21 +
do:
(x: 2 di
v(zr, y, æ+)
(5)
at
Converting between flow and diffusion models. In our framing for affine Gaussian probability paths, a model is identified as flow or diffusion by whether an ODE or SDE is used for sampling respectively. For this class of paths though, we can convert directly between flow and diffusion models. To see this, note that the optimal v( y) for the Conditional Flow Matching loss in Eq. (2) is Ey [v([,,,, y, x1 ( y]. which for affine Gaussian probability paths using Eq. 5 is
day Eg [21 (xt, y) + do. ( xx at Egg |0 y
v(E,; y) = di
(6)
di
Tt
This equivalence has been noted by Karras et al. (2022), leveraging a more complex conversion from SDE to probability flow ODE from Song et al. (2021d). Rearranging Eq. (6), a diffusion model's denoiser क (xx,, y) trained using affine Gaussian probability path q can be interchanged with a flow model's ((x,, y) with the same path via
din(aa//o)) ( at
d dn
i
T +
dt
dt
Training-free conditional inference using unconditional diffusion. Given pretrained unconditional diffusion models that are trained to approximate Ey|[2 [x,]. training-free approaches for conditional inference aim to approximate Eg [ (w,, y) without any fine-tuning of the unconditional model. Under affine Gaussian probability paths, the two terms are related by Tweedie's identity (Robbins, 1992) which expresses Eg [z) [x,, y] = (æ, + ofV a. In (/))/)/)a.. Applying this identity (twice for both [21 art, y) and E., [an (ar))) and simplifying gives
+ o² at
E [2,, y) = Ex [
V., In (yy(:))
(8)
Following Eq. 8, past approaches (e.g., Chung et al. (2022a); Song et al. (2022)) have used the pretrained model for the first term and approximated the second intractable term to produce an approximate 21 (x,,,)) 3). Diffusion posterior sampling (DPS) (Chung et al., 2022a) proposed to approximate q (y))))) via g = Ti (x:)). Later, Pseudo- inverse Guided Diffusson Models (IIGDM) (Song et al., 2022) improved upon DPS for linear noisy observations where y = Ax + dyt. where A is some measurement matrix and e ~ (O, I). by approximating ((yla;)) as N(Ae (z.) of r AAT), derived via first approximating q (zx) as (Z where r, is an appropriate time- dependenn standard deviation. TIGDM also suggested adaptive weighting, replacing of /or with another function of time to account for the approximation. While these past approaches have used Eq. (8) for diffusion probability paths, this equation is valid for any affine Gaussian probability path.
3 Solving Linear Inverse Problems without Conditional Training via Flows
In the standard setup of linear inverse problem, observe measurements y E R such that
y
Ax +
(9)
4

---

Published in Transactions on Machine Learning Research (06/2024)
where æ1 E R is drawn from unknown data distribution g(x1), A R= is known measurement matrix, and (O, T) is unknown i.idd Gaussian noise with known standard deviation .y. Given pretrained flow model with o(zz) that can sample from g(æ1), and measurements y, our goal is to produce clean samples from the posterior ((2)(y) xc (( without training problem-sppeiftt conditional flow model defined by ((z y). In this section, we motivate and propose our approach to solving this problem using flows.
3.1 Adapting the vector field of unconditional flow models for conditional sampling
To solve linear inverse problems without any training via flow models, we derive an expression similar to Eq. that relates conditional vector fields under Gaussian probability paths to unconditional vector fields.
Let q be a Gaussian probability path described by Eq. Assume we observe y ~ 4 x1) for arbitrary 2()) and v (x)) is a vector field enabling sampling æ; g Note that Eq. (6) without y also holds for optimal unconditional u (2)). Inserting Eq. (8) into Eq. (6) and taking the difference (v(22,;) y) v(x;)) yields
ad ddIn(as/or)) V., Ie.Ingg In d (zt. y) v(x:)) + of
(10)
We use Eq. (10) in our training-free algorithm for solving linear inverse using flows by incorporating IIGDM's adaptation. In particular, given ((x,) (or ti (xx)), our approximation will be
(d(n((n/o))) dt 7/ Vn, In g "PP (y)a))
((., y) = 0 (a..) of
(11)
where q0tp"(y (z:)) denotes an approximation for ((,) and % denotes dime-dependent weights. We refer to % as unadaptive and other choices as adaptive weights. In general, we view adaptive weights 7 * 1 as an adjustment for error in ( (a)
Approximating u(yl.x). The update for adapting the unconditional vector field in Eq. (10) requires ((ylax)) which is intractable to compute as it involves marginalization over æ
q(yl / q(y(21))0 x1 ( xz 1x)1
(12)
In this equation, the first term (1)(2)) is tractable as is equal to ((a1,, T). However, it is computationally expensive to estimate the second term g( (æx)) with a flow model or a diffusion model. We therefore use an approximation for ((y) and refer this approximation as qmPp (y)(a)) Following IIGDM, we set
(xx) Re (z (1)
(13)
where Tt is appropriately chosen time dependent standard deviation. We can now compute ( in closed form gPP (1) (x)) xe (AAi) of r? AAT which gives the following approximation for Va, In "PP ())((y)
V az In x (y ATi) ( (AA + - Oæ dz.
(14)
Note that this is a vector-Jacobian product and can computed efficiently with packages for automatic differentiation. With this we generalize ITGDM to any Gaussian probability path described by Eq. 4 by using an alternate r?. We choose r by following IGDM's derivation which assumes that ((x1) is (0, I) to derive r². We have ( (x1) N(aæ of Thus by Bayes' rule, can write the posterior as
g<<1)æ;) x q(()))))))) (N()))-(000
(gg a
o 1)
(15)
of
2 2
ai +
With this, we approximate r
of o +
(16)
When ot 1, we recover TIGDM's v? expected under their Variance-Fxploding path specification.
5

---

Published in Transactions on Machine Learning Research (06/2024)
3.2 Converting between affine Gaussian probability paths
To complete our derivation, we demonstrate how one can train with path q' and perform sampling with alternative path q. This conversion is crucial to enable sampling with any probability path, including particularly the conditional OT probability path, without training given an existing pre-trained model. Such conversions have been noted previously by Karras et al. (2022) using an SDE perspective. Our derivation exposes important subtleties when converting between affine Gaussian probability paths.
Consider two affine Gaussian probability paths, with joint densities q and q', defined by Eq. 4 with dg ot and of of respectively. Define t' (t) as the unique solution to 0s/o, / when it exists for given t. The solution for (t) is unique due to the monotonicity requirements of both a and a. By definition, joint densities & and o' share the same distribution over data 11. g<<1ly) = q' (x1 ly). Then for affine Gaussian probability paths, q = x x1/2011 y) = q' (X2 (x) = te / [0 y) when t' (t) exists. Since the joint densities are identical, the conditional distributions over œ1 used by the optimal denoiser and vector fields are also identical at these values.
So ij trained under o' can be used for sampling under q via evaluating at Ti (o t' (t(), y) (with explicit time for clarity) whenever (t) exists, with identical argument changes for vector fields. In particular, sampling uses the conditional OT probability path, we have
( (1- 1 1-t)
(t)) = SNR (SNR, = (
(17)
where Signal-t-noiss ratio SNR(t)- = a1/o The main avenue for non non-existence for (t() is if the model under q' is trained using a minimum SNR above zero, which induces minimum for which ('(1) exists. When a minimum exists, we can only perform sampling with q starting from æ; ~ g(xx, 1y) Approximating this sample is entirely analogous to approximating Zo of (zoly). This error already exists for q' because o' (xoly) is not (V I) unless q' is trained to zero SNR. An initialization problem cannot be avoided if q' has limited SNR range by switching paths to q. This problem is relevant when converting pre-trained diffusion models as typical diffusion paths have a nonzero minimum SNR.
3.3 Our algorithm for solving linear inverse problems
Starting flow sampling at time t > 0. Initializing conditional diffusion model sampling at t > 0 has been proposed by Chung et al. (2022c). For flows, we similarly want 2; ~ q(>aly) at initialization time t. In our experiments, we examine (approximately) initializing at different times t > O using
xt ony + OrE
(18)
for e ~ (O, I) when y is the correct shape. For super- We also consider using A y as ablation in the Appendix B (where A' is the pseudo-inverse of A (Song et al., 2022)). We may be forced to use this initialization for flow sampling due to converting a diffusion model not trained to zero SNR. However as shown in (Chung et al., 2022c) for diffusion, this initialization can improve results more generally. Conceptually, if the resulting o; is closer to x; ~ g( ly) than achieved via starting from an earlier time t' and integrating, then this initialization can result in less overall error.
nearest-neighbor interpolation y instead.
Algorithm summary. Putting this altogether, our proposed approach using flow sampling and conditional OT probability paths is succinetly summarized in Algorithm 1, derived via inserting 0₁ f and o, = 11- t. This algorithm uses the unconditional vector field adaptation proposed in Eq. (1I) and uses this vector field adaptation @adapted to integrate the ODE from some initial time to to 1 to get the final corrected image I1, We can integrate the ODE by using any standard numerical methods such as Euler method, Runge-Kurt Kutta method etc. As an example, an intermediate update of Euler method from time t to t + At during ODE integration is given by æı+ = x; + @adapted Unlike IIGDM, we propose unadaptive weights % 1. By default, we set initialization time to 0.2. The algorithm therefore has no additional hyperparameters tune over traditional diffusion or flow sampling. In Appendix A, we detail our algorithm for other Gaussian probability paths, and the equivalent formulation when a pretrained vector field is available instead.

---

Published in Transactions on Machine Learning Research (06/2024)
Algorithm 1 Solving linear inverse problems via flows using conditional OT probability path
Require: Pretrained denoiser @ (a:+) converted conditional OT probability path, noisy measurement y, measurement matrix A, initial time to, and std σy
1: Initialize Ltg = foy + (1 fo)c, where 6 ~ No, I)
> Initialize T,, Eq. (18)
2: Dt Dio
3: for each time step f of ODE integration do (1-1)²
Integrate ODE from = to to 1.
r
Value of r from Eq. (16)
[1-
5: i ((c)
Convert 3j to vector field, Eq. (7)
6: g = yy-
ATi (r (r?AAT + T) 1
D Va, In g
( x.))
O2e
7: Undapted 1 + : 9
1-1
D Adapt the unconditional vector field o Eq. (11) 4 One step of ODE solver to update æ;
8: Ar+ ODESolverStep(22r: Vndapted)
9: end for
10: return 211
b This is the solution of ODE integration.
4 Experiments
Datasets. We verify the effectiveness of our proposed approach on three datasets: face-blurred ImageNet 64 x 64 and 128 x 128 (Deng et al., 2009; Russakovsky et al., 2015; Yang et al., 2022), and AnimalFacesHHQ (AFHQ) 256 x 256 (Choi et al., 2020). We report our results on 10K randomly sampled images from validation split of ImageNet, and 1500 images from test split of AFHQ.
Tasks. We report results on the following linear inverse problems: inpainting (center-crop), Gaussian deblurring, super-resolution, and denoising. The exact details of the measurement operators are: 1) For inpainting, we use centered mask of size 20 x 20 for ImageNet-64, 40 40 for ImageNet-122 128, and 80 80 for AFHQ. In addition, for images of size 256 x 256, also use free-form masks simulating brush strokes similar to the ones used in Saharia et al. (2022a); Song et al. (2022). 2) For super-resolution, we apply bicubic interpolation to downsample images by 4x for datasets that have images with resolution 256 x 256 and downsample images by 2 otherwise. 3) For Gaussian deblurring, we apply Gaussian blur kernel of size 61 x 61 with standard deviation of for ImageNet-64 and Image 128, and 61 x 61 with standard deviation 3 for AFHQ. 4) For denoising, we add i.i.d. Gaussian noise with σy = 0.05 to the images. For tasks besides denoising, we consider i.i.d. Gaussian noise with og = and 0.05 to the images. Images æ1 are normalized to range [..
Implementation details. We trained our own continuous time conditional VP-SDE model, and conditional Optimal Transport (conditional OT) flow model from scratch on the above datasets following the hyperparameters and training procedure outlined in Song et al. (2021d) and Lipman et al. (2022). These models are conditioned on class labels, not noisy images. All derivations hold with class label since qyy/c, æ1) q(3) (21) (i.e. the noisy image is independent of class label given the image). We use the open-source implementation of the Euler method provided in torchdiffeq library (Chen, 2018) 8) to solve the ODE in our experiments. Our choice of Euler is intentionally simple, as we focus on flow sampling with the conditional OT path, and not on the choice of ODE solver.
Metrics. We follow prior works (Chung al., 2022a; Kawar et al., 2022) and report Fréchet Inception Distance (FID) (Heusel et al., 2017). Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018), peak signal-to- noise ratio (PSNR), and structural similarity index (SSIM). We use open-source implementations these metrics in the TorchMetrics library (Detlefsen et al., 2022).
Methods and baselines. We use our two pretrained model checkpoints conditional OT flow model and continuous VP-SDE diffusion model, and perform flow sampling with both conditional OT and Variance- Preaaccereeerrving (VP) paths, labeling our methods as OT-ODE and VP-ODE respectively. Because qualitative results are identical and quantitative results similar, we only include the VP-SDE diffusion model in the main text, and include the conditional OT flow model in Appendix C. We compare our OT-ODE and VP-ODE methods against IIGMM (Song et al., 2022) and RED-Diff (Mardani al., 2023) relevant baselines. We selected these baselines because they achieve state-of-
7

---

Published in Transactions on Machine Learning Research (06/2024)
10²
0.5
LPIPS
0.3
0.2
0.1
INET-54
INET-128
APHO SR
INET-
INET-128 cB
APHO GB
INET IPC
INET- INET-129 IPC
APHG IPC
APHQ IPF
INET-B4 ON
INET NET-128 DN AFHQ
INET-64 SR
INET INET-128 SR
AFHQ SR
INET INET-64 GB
INET-128 GB
APHO GB
INET IPC
INET. IPC
AFHO IPC
AFHO IPF
INET NET--4
INET-128 DN
AFHO DN
38
PSNR 34
SSIM f
0.5
REDDIM
MGUUM
30
OT-ODE
INET-R 484 SR
INET- SR
AFHG SR
WET-64 GB
INET-128 GB
AFHO GB
INE IPC
INE NEET-12 128 IPC
APHG IPC
AFHG IPF
INE--4 DW
INET-128 DW
APHG DW
INET-64 SR
INET. SR
INET- APHG SR
INT GB
GB
INET 64 IPC
INET 128 IPC
AFHO IPC
INET-R AFHQ IPF
ON
APHG ON
AFHO
INE 128
Figure 2: Quantitative evaluation of pretrained VP-SDE model for solving linear inverse problems super-resolution (SR), gaussian deblurring (GB), inpainting with centered (IPC) and freeform mask (IPF), and denoising (DN) with dg 0.05. We present results on face-blurred ImageNet-64 (INET-64), face-blurred ImageNet-128 (INET-128) and AFHQ.
the-art performance in solving linear inverse problems using diffusion models. The code for both baseline methods is available on github, and we make minimal changes while reimplementing these methods in our codebase. fair comparison between methods requires considering the number of function evaluations (NFEs) used during sampling. We utilize at most 100 NFEs for our OT-ODE and VP-ODE sampling (see Appendix C), and utilize 100 for IIGDM as recommended Song et al. (2022). We allow RED-Diff 1000 NFEs since it does not require gradients of ø1. For OT-ODE following Algorithm 1,
% and initial 0.2 for all datasets and tasks. For VP-ODE a and initial = 0.4 for all datasets and tasks. Ablations afta?
following Algorithm 2 in the Appendix, we use 7t- /
of these mildly tuned hyperparameters are shown in Appendix B. We extensively tuned hyperparameters for RED-Diff and ITGDM as described in Appendix E, including different hyperparameters per dataset and task.
4.1 Experimental Results
We report quantitative results for the VP-SDE model, across all datasets and linear measurements, in Figure for = 0.05, and Figure within Appendix for og = 0. Additionally, we report results for the conditional OT flow model in Figure 11 and Figure 10 for o, 0.05 and og = O. respectively, in Appendix C. Exact numerical values for all the metrics across all datasets and tasks can also be found in Appendix C. Qualitative images have been selected for demonstration purposes.
Gaussian deblurring. We report qualitative noisy results for the VP-SDE model in Figure 3 and for the conditional OT flow (cond-OT) model in Figure 12. We observe that OT-ODE and VP-ODE outperforms ITGDM and RED-Diff. both qualitatively and quantitatively, across all datasets for Gg = 0.05. As shown in these figures, IIGDM tends to sharpen the images, which sometimes results in unnatural textures in the images. Further, we also observe some unnatural textures and background noise with RED-Diff for dy 0.05. For o, 0. OT-ODE has better FID and LPIPS, but IGDDM shows improved PSNR and SSIM. Figure 21 and Figure 18 show qualitative examples for ay = 0.
8

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Reference
(b) Distorted
(c) OT-ODE
(d) VP-ODE
(c) IIGDM
(f) REDDiff
Figure Results for Gaussian debturring with VP-SDE model and By 0.05 for (first row) ImageNet-64, (second row) ImageNet-122 and (third row) AFHQ.
(a) Reference
(b) Distorted
(c) OT-ODE
(d) VP-ODE
(c) IIGDM
(1) REDDiff
Figure 4: Results for super-resolution with VP-SDE model and oy 0.05 for (first row) ImageNet-64 2x, (second row) ImageNet-122 128 2x, and (third row) AFHQ 4x.
Super-resolution. We report qualitative noisy results for the VP-SDE model in Figure and for the cond-OT model in Figure 13. OT-ODE consistently achieves better FID, LPIPS and PSNR metrics compared to other methods for oy 0.05 (See Figure 2 and 11). Similar to Gaussian deblurring, IIGDM tends to produce sharper edges. This is certainly desirable to achieve good super-resolution, but sometimes this results in unnatural textures in the images (See Figure 4). RED-Diff for o, = 0.05 gives slightly blurry images. In our experiments, we observe RED- Diff is sensitive to the values of y, and we get good quality results smaller values of ag, but the performance deteriorates with increase in value of oy. For og O. as shown Figure 19 and Figure 22, all the methods achieve comparable performance and the method declared best varies per metric and dataset.
9

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Reference
(b) Distorted
(c) OT-ODE
(d) VP-ODE
(e) IIGDM
(1) REDDiff
Figure 5: Results for inpainting (centered mask) with VP-SDE model and og 0.05 for (first row) ImageNet-64,
(second row) ImageNet-1258 and (third row) AFHQ.
Inpainting. For centered mask inpainting, OT-ODE outperforms IIGDM and RED-Diff in terms of LPIPS, PSNR and SSIM across all datasets at oy 0.05. Regarding FID, OT-ODE performs comparably to better than VP-ODE (See Figure 2 and 11). Similar observations hold true for inpainting with freeform mask AFHQ. We present qualitative noisy results for the VP-SDE model in Figure 5 and the cond-OT model in Figure 14. As evident in these images, OT- ODE can result in more semantically meaningful inpainting (for instance, the shape of bird's neck, and shape of hot-dog bread in Figure 5). In contrast, the inpainted regions generated by RED-Diff tend to be blurry and less semantically meaningful. However, we note that OT-ODE (and VP-ODE) inpainting occasionally produces artifacts the inpainted region as the resolution of image increases. We show examples of such negative inpainting results in Appendix C.2. Empirically, we observe that performance of RED-Diff and IIGDM improves as oy decreases. For oy 0, RED-Diff achieves higher PSNR and SSIM, but performs worse than OT-ODE in terms of FID and LPIPS (Refer Figure 9). OT-ODE's 's tendency produce inpainting artifacts for higher resolution images remains for dg 0, and occur for the same images as og 0.05. These artifacts can significantly degrade the pixel- based metrics PSNR and SSIM more than the perceptual metrics such as FID and LPIPS. We further note that noiseless inpainting for OT-ODE can be improved by incorporating null-space decomposition (Wang et al., 2022). We describe this adjustment in Appendix D.
5 Related Work
The challenge of solving noisy linear inverse problems without any training has been tackled in many ways, often with other solution concepts than posterior sampling (Elad et al., 2023). Utilizing a diffusion model has host of recent research that we build upon. Our state-of-the-arr baselines ITGDM (Song et al., 2022) and RED-Diff (Mardani al., 2023) correspond lines of research in gradient- adaptations and variational inference.
Earlier gradient-based based adaptations that approximate V,, gn (z,) in various ways include Diffusion Posterior Sam- pling (DPS) (Chung al., 2022a), Manifold Constrained Gradient (Chung et al., 2022b), and annealed approxi- mation (Jalal et al., 2021). IIGDM out- out-performs earlier methods combining adaptive weights and Gaussian posterior approximation with discrete-time denoising diffusion implicit model (DDIM) sampling (Song et al., 2021a). Here we adapt IIGDM to all Gaussian probability paths and to flow sampling. Our results show adaptive weights are unneces- sary for strongly performing conditional OT flow sampling. Denoising Diffusion Null Models (DDNM) (Wang al., 2022) proposed an alternative approximation E, [z1 Fe, y] using a null-space decomposition specific linear inverse problems, which has been explored in combination with our method in Appendix D.
10

---

Published in Transactions on Machine Learning Research (06/2024)
RED-Diff (Mardani et al., 2023) approximates intractable g(z1l)) directly using variational inference, solving for parameters via optimization. RED-Diff was reported to have mode-seeking behavior confirmed by our results where RED-Diff performed better for noiseless inference. Another carlier variational inference method is Denoising Diffusion Restoration Models (DDRM) (Kawar et al., 2022). DDRM showed SVD can be memory-efficien for image applications, and we adapt their SVD implementations for super-resolution and blur. DDRM incorporates noiseless method ILVR (Choi et al., 2021), and leverages a measurement-deeendee forward process (i.e. g( y) # g(az/()))) like earlier SNIPS (Kawar al., 2021). SNIPS collapses in special cases variants proposed in Song & Ermon (2019); Song et al. (2021d); Kadkhodaie & Simoncelli (2020) for linear inverse problems. Additional related work can be found in Appendix G.
6 Discussion, Limitations, and Future Work
We have presented a training-free approach to solve linear inverse problems using flows that can leverage either pretrained diffusion or flow models. The algorithm is simple, stable, and does not require any problem-specific hyperparameter tuning when used with conditional OT probability paths. Our method combines past ideas from diffusion including IIGM and early starting with the conditional OT probability path from flows, and our results demonstrate that this combination can solve inverse problems for both noisy and noiseless cases across a variety of datasets. Our algorithm using the conditional OT path (OT-ODE) produced results superior to the VP path (VP-ODE) and also to IIGDM and REDDiff for noisy inverse problems. For the noiseless case, the perceptual quality from OT-ODE is on par with IIGDM for super-resolution and gaussian deblurring, but lagging for inpainting due to image artifacts.
Another important limitation, shared with most of the past related research, is restriction to linear observations with sealar variance. Our method can extend to arbitrary covariance, but non-linear observations are more complex. Non- linear observations oceur with image inverse tasks when utilizing latent, not pixel-space, diffusion or flow models. Applying approach to such measurements requires devising alternative (1) a2)). Another shared limitation is that we consider the non-blind setting with known A and dy.
Future research could tackle these limitations. For non-linear observations in latent space, we could perhaps build upon Rout et al. (2023) that uses a latent diffusion model for linear inverses. For the blind setting, we might start from blind extensions to DPS and DDRM (Chung et al., 2023a; Murata et al., 2023). As demonstrated here, we may be able to adapt and possibly improve these approaches via conversion flow sampling using conditional OT paths.
References
Shiv Agrawal, Hwanwoo Kim, Daniel Sanz-Alonso, and Alexander Strang. variational inference approach to inverse problems with gamma hyperpriors. SIAM/ASA Journal Uncertainty Quantification, 10(4):1533-1559, 2022. 48
Michael S Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. arXiv preprint arXiv:2209. 2022. 2, 3
Martin Benning and Martin Burger. Modern regularization methods for inverse problems. Acta numerica, 27:1-111, 2018. 48
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image impainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 417-424, 2000. 48
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using generative models. In International conference on machine learning, pp. 537-546. PMLR, 2017. 48
Stanley H Chan, Xiran Wang, and Omar Elgendy. Plug-and-play admm for image restoration: Fixed-point convergence and applications. IEEE Transactions on Computational Imaging, 3(1):84 98, 2016. 48
Ricky T. Q. Chen. torchdiffeq, 2018. URL https:://tttttttte httt:p///g.tttaaa.ooo///t/cce/br
com/
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018a. 47

---

Published in Transactions on Machine Learning Research (06/2024)
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018b. 2,
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2 2021. 11
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209. (466:2 2022a. 1, 4, 7, 10, 48
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint urXiv:2206.0000 2094000660001... 2022b. 10, 48
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-diffusee faster. Accelerating conditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 22413-12422, 2022c. 6
Hyungjin Chung, Jeongsol Kim, Schui Kim, and Jong Chul Ye. Parallel diffusion models of operator and image for blind inverse problems. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6059-6069) June 2023a. 11
Hyungjin Chung, Jeongsol Kim, Sehui Kim, and Jong Chul Ye. Parallel diffusion models of operator and image for blind inverse problems. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 6059-6069, 2023b. 48
Regev Cohen, Michael Flad, and Peyman Milanfar. Regularization by denoising via fixed-point projection (red-pro). SIAM Journal on Imaging Sciences, 14(3):1374-14066 2021. 48
Giannis Daras, Joseph Dean, Ajil Jalal, and Alexandros Dimakis. Intermediate layer optimization for inverse
problems using deep generative models. arXiv preprint arXiv:2102.07364 2021. 48
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li. Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-244822 255. leee, 2009. 7
Nicki Skafte Detlefsen, Jiri Borovec, Justus Schock, Ananya Harsh Jha, Teddy Koker, Luca Di Liello, Daniel Stancl, Changsheng Quan, Maxim Grechkin, and William Falcon. prrchmetrics-messune reproducibility in pytorch. Journal of Open Source Software, 7(70):110,, 2022. 7
Junyu Dong, Ruiying Yin, Xin Sun, Qiong Li, Yuting Yang, and Xukun Qin. Inpainting of remote sensing sst images with deep convolutional generative adversarial network. IEEE geoscience and remote sensing letters, 16(2):173-177, 2018. 48
Michael Elad, Bahjat Kawar, and Gregory Vaksman. Image denoising: The deep learning revolution and beyond a survey paper. SIAM Journal Imaging Sciences, 16(3):1594-1644 2023. 10
Weijie Gan, Yuyang Hu, Jiaming Liu, Hongyu An, Ulugbek Kamilov, et al. Block coordinate plug-and-play methods for blind inverse problems. Advances in Neural Information Processing Systems, 36, 2024. 48
Davis Gilton, Greg Ongie, and Rebecca Willett. Neumann networks for linear inverse problems in imaging. JEEE Transactions on Computational Imaging, 6:328-343, 2019. 48
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-pi and--aad priors.
Advances in Neural Information Processing Systems, 35:14715- 14728, 2022. 48
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint
2018. 47
12

---

Published in Transactions on Machine Learning Research (06/2024)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017. 7
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Neural Information Processing Systems (NeurIPS), 2020. 3,
Seongmin Hong, Inbum Park, and Se Young Chun. On the robustness of normalizing flows for inverse problems in
imaging. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10745-10755, 2023. 48
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances Neural Information Processing Systems, 34: 54:14938-1495, 2021. 10
Kyong Hwan Jin, Michael T McCann, Emmanuel Froustey, and Michael Unser. Deep convolutional neural network for inverse problems in imaging. IEEE Iransactions on image processing, 26(9):4009-4522, 2017. 48
Zahra Kadkhodaie and Eero P Simoncelli. Solving linear inverse problems using the prior implicit in denoiser. arXiv preprint arXiv:2007. 13640, 2020. 11
Ulugbek S Kamilov, Hassan Mansour, and Brendt Wohlberg. plug-and-play priors approach for solving nonlinear imaging inverse problems. IEEE Signal Processing Letters, 24(12):1872-1876 2017. 48
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basee generative
models. Advances Neural Information Processing Systems, 35:26565-26577 2022. 1,4,
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems, 34:21757-21769 2021. 11
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 11-23593-23600 2022. 7, 11, 48
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems,
2021. 3
Diederik P. Kingma and Ruiqi Gao. Vdm++: Variational diffusion models for high-quality synthesis, 2023. 3
Vladimir M Krasnopolsky. Neural network applications to solve forward and inverse problems in atmospheric and oceanic satellite remote sensing. In Artificial Intelligence Methods in the Environmental Sciences, pp. 191-205. Springer, 2009.
Vladimir M Krasnopolsky and Helmut Schiller. Some neural network applications in environmental sciences. part forward and inverse problems in geophysical remote measurements. Neural Nenworks, 16(3-4):321-334 2003. 48
Dilip Krishnan and Rob Fergus. Fast image deconvolution using hyper-laplacian priors. Advances in neural information processing systems, 22, 2009. 48
Charles Laroche, Andrés Almansa, and Eva Coupete. Fast diffusion em: diffusion model for blind inverse problems with application deconvolution. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 527 1-5281, 2024. 48
Dong Liang, Jing Cheng, Ziwen Ke, and Leslie Ying. Deep magnetic resonance image reconstruction: Inverse problems meet neural networks. IEEE Signal Processing Magazine, 31115:4 2020.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. arXiv preprint arXiv:2305.0888) 2023. 4
Yaron Lipman, Ricky TQ Chen, Heli modeling. arXiv preprini 1rXiv:2210.02747, 2022. 2, 16, 47
Maximilian Nickel, and Matt Le. Flow matching for generative
Gongye Liu, Haoze Sun, Jiay Li. Fei Yin, and Yujiu Yang. Accelerating diffusion models for inverse problems through shortcut sampling, arXiv preprint arXiv:2305. 16965, 2023a. 48
13

---

Published in Transactions on Machine Learning Research (06/2024)
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint wwXiv:2209.030000 2022. 2,
Xuan Liu, Yaoqin Xie, Songhui Diao, Shan Tan, and Xiaokun Liang, A diffusion probabilistic prior for low-dose ct image denoising. arXiv preprint 2rXiv:23055/5888 2023b. 48
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. arXiv preprint (rXiv:2305.0439/ 2023. 1, 2, 7, 10, 11. 43, 48
Robert J McCann. A convexity principle for interacting gases. Advances in mathematics, 128(1))153-179 1997.
Tim Meinhardt, Michael Moller, Caner Hazirbas, and Daniel Cremers. Learning proximal operators: Using denoising networks for regularizing inverse imaging problems. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1781-1790, 2017. 48
Naoki Murata, Koichi Saito, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. Gibbsddrm: A partially collapsed gibbs sampler for solving blind inverse problems with denoising diffusion restoration, 2023.
Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(11):7474-7489 2021. 48
Ankit Raj, Yugi Li, and Yoram Bresler. Gan-based projector for faster recovery with convergence guarantees in linear inverse problems. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5602-5611, 2019. 48
Alejandro Ribes and Francis Schmitt. Linear inverse problems in imaging. IEEE Signal Processing Magazine, 25(4): 84-99, 2008. 48
JH Rick Chang, Chun-Liang Li, Barnabas Poczos, BVK Vijaya Kumar, and Aswin C Sankaranarayanan. One network to solve them all all-solving linear inverse problems using deep projection models. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1888-5897, 2017. 48
Herbert Robbins. An empirical bayes approach to statistics. Breakthroughs in Statistics: Foundations and basic theory, pp. 388-394. Springer, 1992. 4
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal on Imaging Sciences, 10(4): 1044)180448844, 2017. 48
Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alexandros G Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. arXiv preprint arXiv:2307.00619, 2023. 11
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252 2015. 7
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super- resolution via iterative refinement. arxiv. arXiv preprint arXiv:2 104.::2004.07666, 2, 2021. 48
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a. 1, 7,
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super- resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713- 4726, 2022b. 1
14

---

Published in Transactions on Machine Learning Research (06/2024)
Viraj Shah and Chinmay Hegde. Solving linear inverse problems using gan priors: An algorithm with provable guarantees. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4609- 4613. IEEE, 2018. 48
Neta Shaul, Ricky TQ Chen, Maximilian Nickel, Matthew Le, and Yaron Lipman. On kinetic optimal probability paths for generative models. In International Conference on Machine Learning, pp. 30883-30907. PMLR, 2023. 2, 4
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (ICML), 2015. 1
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021a. 3, 10
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guiide diffusion models for inverse problems. In International Conference on Learning Representations, 2022. 1, 2, 4, 6, 7, 10, 42, 48
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Neural Information Processing Systems (NeurIPS), 2019. 3, 11
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415-1428, 2021b. 3
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08005 2021c. 48
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021d. I, 3, 7,
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 9446-9454, 2018. 48
Dave Van Veen, Ajil Jalal, Mahdi Soltanolkotabi, Eric Price, Sriram Vishwanath, and Alexandros G Dimakis. Com- pressed sensing with deep image prior and learned regularization. arXiv preprint
2018. 48
Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEEE global conference signal and information processing, pp. 945-948. IEEE, 2013. 48
Ana Fernandez Vidal, Valentin De Bortoli, Marcelo Pereyra, and Alain Durmus. Maximum likelihood estimation of regularization parameters in high- sish-diiennional inverse problems: An empirical bayesian approach part i: Methodology and experiments. SIAM Journal on Imaging Sciences, 1 13(4): 1945-1999 2020. 48
Yinhuai Wang, Jiwen Yu, and Jian Zhang. shot image restoration using denoising diffusion null-space model.
arXiv preprint arXiv:2212222 12 12.0:229222000990 2022. 1, 07,37,40 48
Jay Whang, Qi Lei, and Alex Dimakis. Solving inverse problems with flow-based noise model. In International Conference on Machine Learning, pp. 11 11146-11157. PMLR, 2021a. 48
Jay Whang, Erik Lindgren, and Alex Dimakis. Composing normalizing flows for inverse problems. In International Conference on Machine Learning, pp. 11158-11169 PMLR, 2021b. 48
Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE transactions image processing, 19(11):2861
2010. 48
Kaiyu Yang, Jacqueline H Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky, study of face obfuscation in imagenet.
In International Conference Machine Learning, pp. 15313-25330 PMLR, 2022. 7
Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep cnn denoiser prior for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3929-3938, 2017. 48
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep
features a perceptual metric. In CVPR, 2018.
15

---

Published in Transactions on Machine Learning Research (06/2024)
A Our method for any Gaussian probability path
Algorithm 1 in the main text is specific to conditional OT probability paths. Here provide Algorithm 2 for any Gaussian probability path specified by Eq. 4. Algorithm and Algorithm 2 are written assuming denoiser æi (i(x;) is provided from a pretrained diffusion model. For completeness, we also include equivalent Algorithm 3 that assumes (2x)) provided from a pretrained flow model. In all cases, the vector field or denoiser is evaluated only once per iteration.
Our VP-ODE sampling results correspond a and o; given from the Variance- path, which can be found in (Lipman et al., 2022).
Algorithm 2 A training- free approach to solve inverse problems via flows with a pretrained denoiser
Require: Pretrained denoiser Ti (ax) converted to Gaussian probability path with ar and de noisy measurement y, measurement matrix A, initial time fu, adaptive weights "tt and std dg 1: Initialize Tt = Onty + o where e ~ (0,
b Initialize ær. Eq. (18)
2: x; = =tt
3: for each time step & of ODE integration do ??
Integrate ODE from = to to
r ofta? /o((
Value of rî from Eq. (16)
dt
31
d di
Dt.
Convert Ti to vector field, Eq. (7)
g yy A
AAT T) 1 A
822 Om
Va, In g aP (y) xtt)
Budapted of dlin(as/or /0e) 96
Adapt unconditional vector field 8, Eq. (11) > One step of ODE solver update
8: Cr+ ODESolverStep( Dt; Vndapted)
9: end for
10: return 211
b This is the solution of ODE integration.
Algorithm A training-free approach to solve inverse problems via flows with a pretrained vector field
Require: Pretrained vector field ((x;) converted to Gaussian probability path with and ot, noisy measurement y, measurement matrix A, initial time to. adaptive weights "r, and std σg
1: Initialize Zto = Ct,y + oG where e ~~ NO
b Initialize æ,, Eq. (18)
2: at = IPoo
for each time step f of ODE integration do
Integrate ODE from to to
D Br is value of w, at time during ODE integration
5:
r .o?
Value of r from Eq. (16)
6:
T = (te
dln(ac/os)
(0
d ds x:) dt
> Convert vector field to Ti. Eq. (7)
dt
7: g = (y A (A?ATT- + +
0221 8ec
In PP (y) xx)
8: Vadapted = + 07 ddin(a./os). 7g
b Adapt unconditional vector field 0 Eq. (11) b One step of ODE solver update æt
de
9: F++At ODESolverStep((r: Vadaptos)
10: end for
11: return 27
This is the solution of ODE integration.
16

---

Published in Transactions on Machine Learning Research (06/2024)
B
Ablation Study
Choice of initialization. We initialize the flow at time >>>>>>>>>0 0 as a,y + o,c (y-init) where <~ NO0 r). Another choice of initialization is to use 2; = o,A'y J₁C. However, empirically we find that this initialization performs worse that y-init on cond-OT model with OT-ODE sampling. We summarize the results our ablation study in Table We find that on Gaussian deblurring, initialization with A y does worse than y-init, while the performance both the initializations is comparable for super-resolution. all our experiments, we use y-init, due its better performance on Gaussian deblurring.
Table I: Quantitative evaluation of choice of initialization for conditional OT flow model with OT-ODE sampling on AFHQ dataset. We find that y-init outperforms A y on Gaussian deblurring.
Gaussian deblur, Fy 0.05
SR 4x, 0g
0.05
Initialization
Start time
NFEs
FID 4
LPIPS +
PSNR
SSIM
FID
LPIPS 4
PSNR
SSIM
init
0.2
100
7.57
0.268
30.28
0,626
6.03
0.219
31.12
0.739
A' y
0.1
100
41.22
0.449
28.79
0.392
12.93
0.292
30.46
0.664
Aly
0.2
100
56.42
0.554
28.11
0.249
6.09
0.219
31.12
0.739
Ablation over "/ for VP-ODE sampling. We compare the performance of "/ against "t-
We show
of+o?
results of VP-ODE sampling with VP-SDE model in Table 2 and Table 3. As seen our choice of n outperform 7/t across all the metrics face-blurred ImageNet- 128.
Table 2: Quantitative evaluation of value of Tt in VP-ODE sampling with VP-SDE model face-blurred ImageNet- 128 dataset.
SR 2x, By
0.05
Gaussian deblur, By 0.05
Start time
NFEs
FID
LPIPS
PSNR
SSIM f
FID |
LPIPS
PSNR f
SSIM
1
0.4
60
32.66
0.371
29.06
0.530
29.31
0.346
29.12
0.554
0.4
60
9.14
0.167
32.06
0.838
10.14
0.196
31.59
0.800
a7+07
Table 3: Quantitative evaluation of value of 7/ in VP-ODE sampling with VP-SDE model on bluurbl ImageNet-
128 dataset.
Inpainting Beainting--entte. dg = 0.05
Denoising, dy = 0.05
7
Start time
NFEs +
FID
LPIPS
PSNR
SSIM t
FID 4
LPIPS
PSNR t
SSIM
0.3
70
53.03
0.285
31.55
0.737
28.37
0.238
31.63
0.786
0.3
70
8.47
0.129
34.43
0.876
5.83
0.087
35.85
0.938
02
Variation of performance with NFEs. We analyze the variation performance of OT-ODE, VP-ODE and IIGDM for solving linear inverse problems as NFEs are varied. The results have been summarized in Figure 6. We observe that OT-ODE consistently outperforms VP-ODE and IIGDM across all measurements in terms of FID and LPIPS metrics, even for NFEs small as 20. We also note that the choice of starting time matters to achieve good performance with OT-ODE. For instance, starting at = 0.4 outperforms = 0.2 when NFEs are small, but eventually as NFEs is increased, = 0.2 performs better. We also note that IIGM achieves higher values of PSNR and SSIM at smaller NFEs for super-resolution but has inferior FID and L.IPS compared to OT-ODE.
Choice of starting time. We plot the variation in performance of OT-ODE and VP-ODE sampling with change in start times for conditional OT model and VP-SDE model AFHQ dataset in Figure 7 and Figure 8, respectively.
17

---

Published in Transactions on Machine Learning Research (06/2024)
Super-resolution 4x
Super-resolution 4x
10
10
FDD
6x
100
Super-resolution
Super-resolutton 4x
3.15 1155x10
7x10-1
PSNR
2.05 10
SSIM
3x
VP-DGE
OT-GDE t=0.4
OT-ODE t-0.2
2.9
20 40 60
80 100
150
200
20
40
60
80 100
150
200
NF
NFEs
Gaussian Deblur
Gaussian Deblur
4x 10
5X
10"
F1D
2x
LP1P 121P5
4xx10
20
40
60
80
100
150
200
20
40
60
80
100
150
200
NFEs
NFEs
Gaussian Deblur
Gaussian Deblur
3x
PSNR
2.95 10
SSIM
NGOM (VP)
x.0
RGDM (VE)
VP-DGE
OT-DDE
NFES
Es
Inpainting-Cente
Inpainting- Center
3x 10
2.2 10
FID
x0
10a
1.2x 10
10-1
20
200
150
NFEs
NFEs
Inpainting Center
Inpainting
90
3.4 10
PSNR
3.3
SSIM
3.2 10
3.1
VP ODE
8x
ODE t-0.2
80 100
150
200
20
40
60
80
100
150
200
NFEs
NFEs
Figure 6: Performance of different procedures for solving linear inverse problems with variation in NFEs AFHQ dataset. We use pretrained conditional OT model and set σy 0.05. The legends VP and VE indicate the choice of r used in ITGDM (See Appendix E. 1). Time f = 0.2 and 0.4 indicates the starting time of sampling with OT-ODE.
18

---

Published in Transactions on Machine Learning Research (06/2024)
We note that in general. OT-ODE sampling achieves optimal performance across all measurements and all metrics at = 0.2 while VP-ODE sampling achieves optimal performance between start times of / 0.3 and 0.4. In this work, for all the experiments, we use = 0.2 for OT-ODE sampling and t = 0.4 for VP-ODE sampling.
Goussian Deblur
Caussian Deblur
Goussian Deatur
Gauss Deblur
30.4
0.38
20.2
0.99
0.00
SSIM
0.99
0.30
26.4
0T:00
25-2
start tlmee
start time
start time
start time
Super-rrssolution
Super-resslution
Super
Super-resolution
0.20
10.0
0.26
0.00
30.4
0.23
36.2
0.64
00x
0.1
0.2
0.3
0.4
3.5
0.6
6.1
0.2
0.3
0.4
0.5
0.5
0.1
0.2
0.3
0.4
0.5
0.6
0.1
0.2
0.3
0.4
0.5
0.6
start
start time
start time
start time
Inpainting Center
iypainting-Crrtte
Inpaisting Canter
Inpainting Center
vP.00r
0.15
35.2
S51M4
0.090
0.12
35.1
0.005
35.0
0.11
0.000
start
time
start time
Figure 7: Performance of OT-ODE and VP-ODE solving linear inverse problems with varying start times on AFHQ dataset. We use pretrained cond-OT model and set σy = 0.05.
19

---

Published in Transactions on Machine Learning Research (06/2024)
Gaussion Deblur
Gaussian
Goussian Deblur
Deblur
F10
200
25.5
0.3
V0.00E
27.5
0.2
0.3
0.6
0.5
8.6
0.2
0.3
0.4
0.5
0.6
0.2
0.3
0.4
0.5
0.6
0.2
0.3
041
0.5
0.6
start time
start
start time
start time
Supen-resolution dx
Super resolution 4x
Supper resollttion 4x
Supen resolution 4x
0.7
31.0
30.5
10
LPIPS
20.5
100
20.0
50
01:002
28.0
WP-006
time
start time
time
time
Impeinting-
Inpeinting-Center
Iepeinting-Center
Inpeinting
0.30
0.850
PSNR
0.20
0.775
01-006
0.18
0.2
0.4
0.8
0.6
0.5
0.2
0.6
9.4
start time
start time
start time
start time
Figure 8: Performance of OT-ODE and VP-ODE in solving linear inverse problems with varying start times AFHQ dataset. We use pretrained VP-SDE model and set σy 0.05.
20

---

Published in Transactions on Machine Learning Research (06/2024)
C
Additional Empirical Results
The main text includes Figure 2 with By = 0.05 produced with the denoiser from the continuous-time VP-SDE diffusion model showing plots of various metrics across all datasets and tasks. Here we provide the same for our pre- pre-trained conditional OT flow matching model using Algorithm in Figure 11, and noiseless figures for both models Figure 9 and 10. To save compute, for the flow model we only include our IIGDM baseline as RED-Diff required extensive hyperparameter tuning. The qualitative results using the flow model instead diffusion model checkpoint are identical.
This section also includes tables containing the numerical values of metrics across all datasets and tasks. The tables are hierarchically organized by noise, dataset, and task in consistent ordering. Noisy results with By 0.05 are in Table 4 to 9 and noiseless results with og 0 are in Table 8 to 13.
0.35
0.30
10.0
LPIPS
7.5
5.0
0.10
2.5
INET-84 SR
INET-128 BR
AFNQ BR
INET GB
INET-128 GB
AFHQ 08
INET-64 IPC
INET-128
AFHQ IPC
AFHQ IPF
INET-84 SR
INET-128
AFHQ SR
INET GB
INET ce
AFHO GB
INET-64 IPC
INET-128 IPC
AFHQ IPC
AFHO IPF
38
0.9
PSNR
SSIM
0.8
REDDIFf
32
0.7
NGDM
VP-ODE
30
OT-ODE
INET-84 SR
INET-128 SR
AFHO SR
INET-84 GB
NEE GB
INE 744 IPC
INT-128 IPC
AFHQ IPC
APHQ IPF
INET-64 SR
INET-128 SR
AFHQ SR
INET-84 GB
AFHQ GB
INET-84 IPC
INET-128 IPC
AFHO IPC
APHQ IPF
AFHQ
INET INET-128
Figure 9: Quantitative evaluation of pretrained VP-SDE model for linear inverse problems on super-resolution (SR).
gaussian deblurring (GB), image inpainting centered mask (IPC) and inpainting free-form (IPF) with og = We
show results face-blurred ImageNet-64 (INET-64). face-blurred ImageNet- 128 (INET-128) and AFHQ-22 256 (AFHQ).
Table 4: Quantitative evaluation of linear inverse problems face-blurred ImageNet-64 64
SR 2x, 0yy
0.05
Gaussian deblur, By 0.05
Model
Inference
NFEs 1
FID 1
LPIPS
PSNR
SSIM 1
FID L
LPIPS
PSNR f
SSIM
OT
OT-ODE
80
6.07
0.157
30.88
0.799
6.83
0.185
30.51
0.773
OT
VP-ODE
80
7.82
0.163
30.75
0.792
8.72
0.190
30.40
0.765
OT
TIGDM
100
6.52
0.168
30.54
0.753
55.19
0.374
28.74
0.516
VP-SDE
OT-ODE
80
5.57
0.155
30.88
0.799
6.33
0.181
30.52
0.773
VP-SDE
VP-ODE
80
7.40
0.160
30.75
0.792
8.16
0.187
30.42
0.766
VP-SDE
IIGMM
100
6.84
0.174
30.48
0.743
54.77
0.376
28.74
0.511
VP-SDE
RED-Diff
1000
23.02
0.187
31.22
0.839
51.20
0.236
30.19
0.776
21

---

Published in Transactions on Machine Learning Research (06/2024)
0.30
0.25
LPIPS
0.20
0.10
INET-84 SR
INET-128 SR
INET-128 G8
IETT-54 IPC
INET-128 IPC
AFHO IPC
APHO GB
INET-64 IPC
INET-128 IPC
APHO IPC
AFHQ IPF
APHO
INET-64
APHO
AFHQ
INET-84
AFHQ
INET-64
INET-128
INET
PSNR
SSIM
34
FIGOM
VP-ODE
OT-ODE
INET-84 SR
INET-128 SR
AFHO SR
INET-84 GB
WE GB
AFHO G8
INT-64
INE IPC
AFHQ IPC
AFHQ IPF
INET-64 SR
INET-128 SR
AFHQ SR
WET--5 GB
INET-188 GB
APHO GB
IEET-44 IPC
INET 128 IPC
AFHO IPC
AFHQ IPF
Figure 10: Quantitative evaluation of pretrained conditional OT model for linear inverse problems on super-resolution
(SR), gaussian deblurring (GB), image inpainting centered mask (IPC) and inpainting freeform (IPF) with 0. We show results on face-blurred ImageNet-64 (INET-64), face-blurred ImageNet- 128 (INET- 128), and AFHQ-25 (AFHQ).
Table 5: Quantitative evaluation of linear inverse problems on bluurbdd ImageNet-64 64
Inpainting- Center, σy =
0.05
Denoising, o, = 0.05
Model
Inference
NFEs 1
FID |
LPIPS
SSIM f
PSNR 1
FID |
LPIPS 4
SSIM
PSNR
OT
OT-ODE
80
5.45
0.101
34.21
0.870
2.91
0.044
35.96
0.968
OT
VP-ODE
80
5.70
0.105
33.87
0.865
3.54
0.049
35.37
0.960
OT
HIGDM
100
9.25
0.111
34.13
0.863
16.59
0.102
34.60
0.906
VP-SDE
OT-ODE
80
5.03
0.098
34.25
0.872
2.76
0.042
36.02
0.969
VP-SDE
VP-ODE
80
5.26
0.103
33.93
0.866
3.29
0.048
35.45
0..61
VP-SDE
TIGDM
100
9.75
0.113
34.03
0.860
17.19
0.107
34.25
(),91
VP-SDE
RED-Diff
1000
12.18
0.119
33.97
0.881
6.02
0.041
35.64
0.964
Table 6: Quantitative evaluation of linear inverse problems on face-blurred ImageNet- 128 x 128
SR 2x, Fy
0.05
Gaussian deblur, oy 0.05
Model
Inference
NFEs +
FID
LPIPS
PSNR
SSIM
FID L
LPIPS
PSNR f
SSIM
OT
OT-ODE
70
3.22
0.141
32.35
0.820
4.84
0.175
31.94
0.821
OT
VP-ODE
70
7.52
0.162
32.24
0.847
8.49
0.191
31.76
0.809
OT
HIGMM
100
4.38
0.148
32.07
0.831
30.30
0.328
29.96
0.606
VP-SDE
OT-ODE
70
3.21
0.139
32.40
0.855
4.49
0.173
32.02
0.824
VP-SDE
VP-ODE
70
9.14
0.166
32.06
0.838
9.35
0.193
31.66
0.804
VP-SDE
IIGDM
100
7.55
0.183
31.61
0.785
55.61
0.463
28.57
0.414
VP-SDE
RED-Diff
1000
10.54
0.182
31.82
0.852
21.43
0.229
31.41
0.807
22

---

Published in Transactions on Machine Learning Research (06/2024)
0.3
FID
LPIPS 4
10
0.2
INET-128 BR
AFHG SR
IWET-84 GB
INET-128 Ge
AFHO GB
INET-64 IPC
INE IPC
AFHQ IPC
APHO IPF
INET-84 ON
INET-84 BR
INET-128 DR
AFHQ SR
INET-84 GB
INET-128 GB
APHQ GB
INET-64 IPC
INET-128 IPC
APHQ IPC
AFHG IPF
INET-64 ON
INET-128 ON
INET-64
INET 128
AFHO
AFHQ
PSNR
SSIM
0.5
vP-ODE OT-ODE
INET-64 GR
INET-128 SR
APHQ SR
INET-64 GB
INET-128 GB
INET-84 IPC
INET. IPC
AFHO AFHQ IPC
INET TETT-4 DN
WNTT-122 DN
AFHQ DN
INET-44 GR
INET-128 SR
AFHQ SR
INET-84 GB
INET-125 GB
AFHQ GB
NET-128P IPC
APHQ IPC
AFHO IPF
INE INET-84 DN
INET-128 DN
AFHQ ON
AFHO
INET-64
Figure 11: Quantitative evaluation of pretrained conditional OT model for linear inverse problems on super-resolution
(SR), gaussian deblurring (GB), image inpainting centered mask (IPC) and denoising (DN) with J, = 0.05. We show results on blurred ImageNet-64 (INET-64), face-blurred ImageNet- 128 (NNE-1288) and AFHQ-256 (AFHQ).
Table 7: Quantitative evaluation of linear inverse problems on face-blurred ImageNet- x 128
Inpainting- Centter, σy = 0.05
Denoising, Ty = 0.05
Model
Inference
NFEs +
FID
LPIPS
PSNR
SSIM
FID |
LPIPS
PSNR 1
SSIM
OT
OT-ODE
70
6.58
0.121
35.00
0.881
3.21
0.063
37.35
0.964
OT
VP-ODE
70
6.44
0.127
34.47
0.871
3.98
0.075
36.26
0.948
OT
IIGDM
100
7.99
0.122
34.57
0.867
9.60
0.107
35.11
0.903
VP-SDE
OT-ODE
70
6.39
0.120
35.04
0.882
3.25
0.062
37.41
0.965
VP-SDE
VP-ODE
70
8.47
0.129
34.43
0.876
5.83
0.087
35.85
0.938
VP-SDE
IIGM
100
9.75
0.130
34.45
0.858
10.69
0.124
34.72
0.882
VP-SDE
RED-Diff
1000
14.63
0.171
32.42
0.820
9.19
0.105
33.52
0.895
Table Quantitative evaluation of linear inverse problems on AFHQ-256 256
SR 4x, By
0.05
Gaussian deblur, og 0.05
Model
Inference
NFEs
FID 4.
IPIPS +
PSNR 1
SSIM 1
FID J
LPIPS
PSNR 1
SSIM
OT
OT-ODE
100
6.03
0.219
31.12
0.739
7.57
0.268
30.27
0.626
OT
VP-ODE
100
6.81
0.229
31.01
0.728
7.80
0.276
30.21
0.616
OT
IIGDM
100
12.69
0.285
30.18
0.665
24.60
0.383
28.93
0.429
VP-SDE
OT-ODE
100
7.28
0.238
30.83
0.714
8.53
0.276
30.37
0.641
VP-SDE
VP-ODE
100
8.02
0.243
30.96
0.727
10.21
0.289
30.21
0.621
VP-SDE
IIGDM
100
77.49
0.469
29.34
0.469
116.42
0.535
28.49
0.313
VP-SDE
RED-DifT
1000
20.84
0.331
29.97
0.675
15.81
0.341
30.15
0.645
23

---

Published in Transactions on Machine Learning Research (06/2024)
Table 9: Quantitative evaluation of linear inverse problems on AFHQ-256 256
Inpainting Centter By
0.05
Denoising. Oy 0.05
Model
Inference
NFEs +
FID 4
LPIPS
PSNR
SSIM
FID 4
LPIPS 4
PSNR t
SSIM
OT
OT-ODE
100
8.98
0.104
35.32
0.897
2.48
0.061
37.18
0.965
OT
VP-ODE
100
7.48
0.107
35,02
0.892
3.38
0.075
37.41
0,954
OT
IIGDM
100
19.09
0.153
34.20
0.855
22.87
0.237
32.93
0.823
VP-SDE
OT-ODE
100
9.93
0.107
35.18
0.892
2.17
0.060
37.95
0.963
VP-SDE
VP-ODE
100
8.78
0.107
35.12
0.891
3,08
0.071
37.68
0,959
VP-SDE
IIGDM
100
57.46
0.239
32.40
0.773
81.15
0.451
29.62
0.639
VP-SDE
RED-Diff
1000
11.02
0.124
34.97
0.893
4.93
0.112
34.18
0.899
Table 10: Quantitative evaluation of linear inverse problems on face-blurred ImageNet-64 64
SR 2x, By 0
Gaussian deblur, dy
Model
Inference
NFEs 4
FID 4
LPIPS
PSNR
SSIM 1
FID 4
I.PIS +
PSNR t
SSIM
OT
OT-ODE
80
6.46
0.119
31,59
0.839
2.59
0.038
35.31
0.961
OT
VP-ODE
80
8.29
0.147
31.20
0.817
6.13
0.083
33.31
0.929
OT
IIGDM
100
6.89
0.115
32.02
0.853
4.53
0.051
35.88
0.963
VP-SDE
OT-ODE
80
6.32
0.118
31.60
0.839
2.61
0.037
35.45
0.963
VP-SDE
VP-ODE
80
7.76
0.145
31.21
0.817
5.68
0.080
33.37
0.931
VP-SDE
IIGMM
100
6.47
0.113
32.03
0.853
4.35
0.049
35.95
0.964
VP-SDE
RED-Diff
1000
11.74
0.224
30.12
0.798
15.39
0.134
31.99
0.879
Table 11: Quantitative evaluation of linear inverse problems face-blurred ImageNet-64 64
Inpainting CCenter, By
0
Model
Inference
NFEs
LPIPS 4
PSNR
SSIM t
OT
OT-ODE
80
4.94
0.080
37.42
0.885
VP-ODE
7.85
0.120
34.24
0.858
OT
IIGDM
100
6.09
0.082
36.75
0.901
VP-SDE
OT-ODE
80
4.85
0.079
37.64
0.887
VP-SDE
VP-ODE
7.21
0.117
34.33
0.860
VP-SDE
HIGDM
100
5.79
0.081
36.81
0.902
VP-SDE
RED-Diff
1000
7.29
0.079
39.14
0.925
Table 12: Quantitative evaluation of linear inverse problems on face-blurred ImageNet-128 128
Inpainting Center, dg
0
Model
Inference
NFEs
FID
LPIPS 4
PSNR
SSIM t
OT
OT-ODE
70
5.88
0.095
37.06
0.894
VP-ODE
8.63
0.144
34.48
0.864
OT
IIGDM
100
5.82
0.097
36.89
0.908
VP-SDE
OT-ODE
70
5.93
0.094
37.31
0.898
VP-SDE
VP-ODE
8.08
0.142
34.55
0.865
VP-SDE
IIDM
100
5.74
0.095
37.01
0.911
VP-SDE
RED-Diff
1000
5.40
0.068
38.91
0,928
24

---

Published in Transactions on Machine Learning Research (06/2024)
Table 13: Quantitative evaluation of linear inverse problems on face-blurred ImageNet-128 128
SR 2x, By = 0
Gaussian deblur, dy
Model
Inference
NFEs 4
FID 4
LPIPS
PSNR
SSIM 1
FID +
LPIPS
PSNR t
SSIM
OT
OT-ODE
70
4.46
0.097
33.88
0.903
2.09
0.048
37.49
0.961
OT
VP-ODE
70
7.69
0.144
32.93
0.871
6.02
0.108
34.73
0.925
OT
HIGDM
100
6.09
0.105
34.28
0.910
4.28
0.066
37.56
0.961
VP-SDE
OT-ODE
70
4.62
0.096
33.95
0.906
2.26
0.046
37.79
0.967
VP-SDE
VP-ODE
70
7.91
0.144
32.87
0.869
5.64
0.105
34.81
0.928
VP-SDE HIDM
100
6.02
0.104
34.33
0.911
4.35
0,065
37.70
0.963
VP-SDE
RED-Diff
1000
3,90
0.082
34.47
0.92
4.19
0.085
34.68
0.929
Table 14: Quantitative evaluation of linear inverse problems on AFHQ-256 256
SR 4x, Ty = 0
Gaussian deblur, By
Model
Inference
NFEs 4
FID 1
LPIPS
PSNR
SSIM 1
FID |
LPIPS
PSNR 1
SSIM 1
OT
OT-ODE
100
5.75
0.169
32.25
0.792
6.63
0.213
31.29
0.722
OT
VP-ODE
100
6.14
0.194
31.93
0.773
7.38
0.231
31.10
0.705
OT
HIGDM
100
8.89
0.173
32.57
0.812
9.78
0.209
31.54
0.743
VP-SDE
OT-ODE
100
6.58
0.178
32.18
0.789
8.24
0.226
31.21
0.717
VP-SDE
VP-ODE
100
8.00
0.225
31.48
0.742
9.19
0.252
30.91
0.688
VP-SDE
TIGDM
100
10.85
0.189
32.52
0.811
11.46
0.228
31.47
0.738
VP-SDE
RED-Diff
1000
8.65
0.191
32.21
0.801
11.67
0.268
31.30
0.731
Table 15: Quantitative evaluation of linear inverse problems on AFHQ-256 x 256
Inpainting- Inpainting-Center, By
Inpainting- Fireinting-FFre--fommm, By
Model
Inference
NFEs +
FID |
LPIPS
PSNR
SSIM
FID L
LPIPS
PSNR 1
SSIM
OT
OT-ODE
100
8.87
0.061
37.45
0.921
4.98
0.097
36.15
0.889
OT
VP-ODE
100
9.18
0.106
35.63
0.898
6.92
0.135
34.72
0.869
OT
HIGM
100
7.36
0.080
37.45
0.933
6.52
0.100
36.58
0.913
VP-SDE
OT-ODE
100
9.95
0.064
37.49
0.918
5,39
0.099
36.15
0.887
VP-SDE
VP-ODE
100
10.50
0.112
35.59
0.893
7.36
0.139
34.65
0.865
VP-SDE
IIGDM
100
8.61
0.088
37.27
0.925
7.25
0.109
36.37
0.906
VP-SDE
RED-Diff
1000
8.53
0.050
38.89
0.951
7.27
0.090
36.88
0.892
25

---

Published in Transactions on Machine Learning Research (06/2024)
C.1 Additional qualitative results
(a) Reference
(b) Distorted
(c) OT-ODE
(d) VP-ODE
(e) IIGDM
Figure 12: Gaussian-deblle with conditional OT model and 0g 0.03 for (first row) face-blurred ImageNet-64, (second and third row) face-blurred ImageNet-122 288 and fourth and fifth row) AFHQ.
26

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Reference
(b) Distorted
(c) OT-ODE
(d) VP-ODE
(e) ITGDM
Figure 13: Super-resolution with conditional OT model and By 0.05 for (first row) face-blurred ImageNet-64 2x, 128
(second row) face-blurred ImageNet-129 128 2x, and (third row) AFHQ 4x.
27

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Reference
(b) Distorted
OT-ODE
(d) VP-ODE
TIGDM
Figure 14: Inpainting (Center mask) with conditional OT model and o, 0.05 for (first row) face-blurred ImageNet- 64, (second row) face-blureed ImageNet- 128, and (third row) AFHQ.
28

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Reference
(b) Distorted
(c) OT-ODE
(d) VP-ODE
(c) TTGDM
Figure 15: Inpainting (Free-form mask) with conditional OT model and o, = 0.05 for AFHQ.
29

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Reference
(b) Distorted
(c) OT-ODE
(d) VP-ODE
(c) IIGDM
Figure 16: Denoising with conditional OT model σy 0.05 for (first row) face-blurred ImageNet-64, (second row) face-blurred ImageNet- 128, and (third row) AFHQ.
(a) Reference
(b) Distorted
(c) OT-ODE
(d) VP-ODE
(c) ITGDM
(f) RED-Diff
Figure 17: Denoising with pretrained VP-SDE model and oy = 0.05 for (first row) face-blurred ImageNet-64, (second row) face-blurred ImageNet-128, and (third row) AFHQ.
30

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Reference
(b) Distorted
OT-ODE
(d) VP-ODE
IIGDM
Figure 18: Gaussian deblurring with conditional OT model and By for (first row) face-blurred ImageNet-64,
(second row) face-blurred ImageNet- - 128 and (third row) AFHQ.
(a) Reference
(b) Distorted
OT-ODE
(d) VP-ODE
IIGDM
Figure 19; Super-resolution with conditional OT model and oy 0 for (first row) face-blurred ImageNet-64, (second row) face-blurred ImageNet- 128 128 and (third row) AFHQ.
31

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Reference
(b) Distorted
OT-ODE
(d) VP-ODE
TIGDM
Figure 20: Inpainting (centered mask) with conditional OT model and o, for (first row) face-blurred ImageNet-64, (second row) face-blurred ImageNet-122 128 and (third row) AFHQ.
32

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Reference
(b) Distorted
(c) OT-ODE
(d) VP-ODE
(c) IIGDM
(f) RED-Diff
Figure 21: Gaussian deblurring with VP-SDE model and σg = 0 for (first row) face-blurred ImageNet-64, (second row) face-blurred ImageNet- 128 and (third and fourth row) AFHQ.
(a) Reference
(b) Distorted
(c) OT-ODE
(d) VP-ODE
(c) IIGDM
(f) RED-Diff
Figure 22: Super-resolution with VP-SDE model and By for (first row) face-blurred ImageNet-64, (second row) face-blurred ImageNet- 128 and (third row) AFHQ.
33

---

Published in Transactions on Machine Learning Research (06/2024)
XI
XI
XI
XI
XII
XI
(a) Reference
(b) Distorted
(c) OT-ODE
(d) VP-ODE
(c) TIGDM
(f) RED-Diff
Figure 23: Inpainting (centered mask) with VP-SDE model and ay = 0 for (first and second row) face-blurred ImageNet-64, (third row) face-blurred ImageNet- 128 and (fourth row) AFHQ.
(a) Reference
(b) Distorted
(c) OT-ODE
(d) VP-ODE
(e) L1GDM
(1) RED-Diff
Figure 24: Inpainting (freeform mask) with VP-SDE model and o, for AFHQ.
34

---

Published in Transactions on Machine Learning Research (06/2024)
C.2 Negative results from Inpainting
(a) Reference
(b) Masked-noisy
Corrected-noisy
(d) Masked 0,
Corrected d,
Figure 25: Negative results for inpainting with OT-ODE on AFHQ. We can observe artifacts in high-resolution images where the masked region is not inpainted correctly and there are patches the inpainted region that are semantically incorrect. The observed artifacts present in both the noiseless (e) and noisy (c) columns.
35

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Reference
(b) Masked-noisy
(c) Corrected-noisy
(d) Masked oy 0 (e) Corrected σr
Figure 26: Negative results for inpainting with OT-ODE on face-blurred ImageNet- 128. We can observe artifacts in high-resolution images where the masked region is not inpainted correctly and there are patches in the inpainted region that are semantically incorrect. The observed artifacts are present in both the noiseless (e) and noisy (c) columns.
36

---

Published in Transactions on Machine Learning Research (06/2024)
D Noiseless null and range space decomposition
When of O, can produce vector field approximation with even lower Conditional Flow Matching loss by applying a null-space and range-space decomposition motivated by DDNM (Wang et al., 2022). In particular, when y = Az,, we have that A = A 1 (where A the pseudo-inverse of A) and so
Ey , y = A [[A"TA A + A A y] A (I - A" A' (a ( y).
(19)
So when o = O, is only necessary to approximate the second term, as the first term is known through y. The regression loss is minimized for the first term automatically and æ (0t, y) is only responsible for predicting the second term.
In our experiments, we find that null space decomposition helps in inpainting but not other measurements, We summarize the results in Table 16 to 21 and show qualitative results for inpainting in Figure 27 29.
Table 16: Comparison of performance OT-ODE sampling and OT-ODE sampling with null and range space decom- position (NRSD) on face-blurred ImageNet-64 x 64. For inpainting, OT-ODE sampling with null and range space decomposition outperforms simple OT- sampling.
Inpainting Center, 0y
Model
Inference
NFEs
FID 4
LPIPS
PSNR f
SSIM
OT
OT-ODE
80
4.94
0.080
37.42
0.885
OT
OT-ODE-NRSD
80
3.84
0.072
38.23
0.888
OT VP-ODE
80
7.85
0.120
34.24
0.858
VP-SDE VP-SDE
OT-ODE
80
4.85
0.079
37.64
0.887
OT-ODE-NRSS
80
3.77
0.072
38.24
0.888
VP-SDE
VP-ODE
80
7.21
0.117
34.33
0.860
Table 17: Comparison of performance OT-ODE sampling and OT-ODE sampling with null and range space decomposi- tion (NRSD) on bluerblurred ImageNet-64 64. For tasks like super-resolution and Gaussian deblurring, OT-ODE sampling without null and range space decomposition outperforms other methods.
SR 2x, Ty
Gaussian deblur, By 0
Model
Inference
NFBs
FID
LPIPS
PSNR f
SSIM
FID
LPIPS
PSNR f
SSIM
OT
OT-ODE
80
6.46
0.119
31.59
0.839
2.59
0.038
35.31
0.961
OT
OT-ODE-NRSD
80
7.37
0.134
31.05
0.799
3.05
0.044
35.19
0.956
OT
VP-ODE
80
8.29
0.147
31.20
0.817
6.13
0.083
33.31
0.929
VP-SDE VP-SDE
OT-ODE
80
6.32
0.118
31.60
0.839
2.61
0.037
35.45
0.963
OT-ODE-NRSD
80
7.13
0.133
31.06
0.798
2.99
0.044
35.24
0.956
VP-SDE
VP-ODE
80
7.76
0.145
31.21
0.817
5.68
0.080
33.37
0.931
Table 18: Comparison of performance OT-ODE sampling and OT-ODE sampling with null and range space decomposi- tion (NRSD) on face-blurred ImageNet-125 x 128.
SR 2x, Ty =
Gaussian deblur, oy = O
Model
Inference
NFFs
FID J.
I.PPS J
PSNR 1
SSIM
FID +
I.PPS 4
PSNR 1
SSIM 1
OT
OT-ODE
70
4.46
0.097
33.88
0.903
2.09
0.048
37.49
0.961
OT-ODE-NRSD
70
3.62
0.099
33.24
0.876
1.42
0.036
38.35
0.969
OT
VP-ODE
70
7.69
0.144
32.93
0.871
6.02
0.108
34.73
0.925
VP-SDE
OT-ODE
70
4.62
0.096
33.95
0.906
2.26
0.046
37.79
0.967
VP-SDE
OT-ODE-NRSD
70
3.44
0.098
33.28
0.877
1.36
0.035
38.44
0.969
VP-SDE
VP-ODE
70
7.91
0.144
32.87
0.869
5.64
0.105
34.81
0.928
37

---

Published in Transactions on Machine Learning Research (06/2024)
Table 19: Comparison of performance OT-ODE sampling and OT-ODE sampling with null and range space decomposi-
tion (NRSD) on face-blurred ImageNet-128 x 128
Inpainting Center, By 0
Model
Inference
NFEs 4
4
LPIPS +
PSNR t
SSIM
OT-ODE
70
5.88
0.095
37.06
0.894
OT
OT-ODE-NRSD
70
3,95
0.074
38.27
0.906
OT
VP-ODE
70
8.63
0.144
34.48
0.864
VP-SDE
OT-ODE
70
5.93
0.094
37.31
0.898
VP-SDE
ST-ODE-NRSI
70
3,84
0.073
38.27
0.906
VP-SDE
VP-ODE
70
8.08
0.142
34.55
0.865
Table 20: Comparison of performance OT-ODE sampling and OT-ODE sampling with null and range space decomposi-
tion (NRSD) on AFHQ-256 x 256
SR 4x, Jy =
Gaussian deblur, σy 0
Model
Inference
NFES
FID 4
I.PISS +
PSNR f
SSIM
FID +
I.PPS +
PSNR 1
SSIM
OT
OT-ODE
100
5.75
0.169
32.25
0.792
6.63
0.213
31.29
0.722
OT OT-ODE-NRSS
100
5.73
0.179
31.69
0.753
0.237
30.72
0.665
OT
VP-ODE
100
6.14
0.194
31.93
0.773
7.38
0.231
31.10
0.705
VP-SDE VP-SDE
OT-ODE
100
6.58
0.178
32.18
0.789
8.24
0.226
31.21
0.717
OT-ODE-NRSD
100
6.99
0.195
31.65
0.752
10.19
0.255
30.66
0.662
VP-SDE
VP-ODE
100
8.00
0.225
31.48
0.742
9.19
0.252
30.91
0.688
Table 21: Comparison of performance OT-ODE sampling and OT-ODE sampling with null and range space decomposi-
tion (NRSD) on AFHQ-255 x 256
Inpainting Center, By 0
Aupainting-Free-foree σg 0
Model
Inference
NFEs
FID L
LPIPS +
PSNR f
SSIM
FID 1
LPIPS +
PSNR f
SSIM f
OT
OT-ODE
100
8.87
0.061
37.45
0.921
4.98
0.097
36.15
0.889
OT-ODE-NRSD
100
7.95
0.046
38.01
0.921
4.12
0.083
36.62
0.890
OT
VP-ODE
100
9.18
0.106
35.63
0.898
6.92
0.135
34.72
0.869
VP-SDE
OT-ODE
100
9.95
0.064
37.49
0.918
5.39
0.099
36.15
0.887
VP-SDE
OT-ODE- -RRD
100
10.96
0.052
37.95
0,916
4.87
0.089
36,52
0.884
VP-SDE
VP-ODE
100
10.50
0.112
35.59
0.893
7.36
0.139
34.65
0.865
38

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Reference (b) Distorted
(c) OT-ODE
(d) OT-
(0) HIGDM
Figure 27: Comparison of inpainting (center mask) via OT-ODE sampling with and without null and range space decomposition (NRSD). We use conditional OT model and o, 0 for (first and second row) face-blurred ImageNet- 64, (third row) face-blurred ImageNet- 128, and (fourth row) AFHQ.
39

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Reference
(b) Distorted
OT-ODE
(d) OT-ODE-NRSD
TIGDM
Figure 28: Comparison of inpainting (center mask) via OT-ODE sampling with and without null and range space decomposition (NRSD) for (first row) face-blurred ImageNet-64, (second row) face-blurred ImageNet-128, and (third row) AFHQ. We use VP-SDE model and o, = 0.
40

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Reference
(b) Distorted
OT-ODE
(d) OT-ODE-NRSD
TIGDM
Figure 29: Comparison of inpainting (free-form mask) via OT-ODE sampling with and without null and range space decomposition (NRSD) for AFHQ. We use conditional OT model and σg 0.
41

---

Published in Transactions on Machine Learning Research (06/2024)
E
Baselines
E.1 ITGDM
Implementation details. We closely follow the official code available on github while implementing IIGDM. For noisy case, we closely follow the Algorithm the appendix of Song et al. (2022). We use adaptive weighted guidance for both noiseless and noisy cases as in the original work. We always use uniform spacing while iterating the timestep over 100 steps. We use ascending time from 0 to Note that the original paper uses descending time from to 0. According to the notational convention used in this paper, this is equivalent to ascending time from 0 to 1. For the choice of rt, consider the values derived from both variance exploding formulation and variance preserving formulation.
Value of IIGDM sets the value of of. T+of-, for VE-SDE, where (( æ1) W We follow the same procedure as outlined in Song et al. (2022), and solve for rē in closed form VP-SDE. know for that VP-SDE, (()))<1) (a (1 (1-ca
e [(()) [()) S ((s)ds, and ((s) is the noise scale function. Using equation 16 for VP-SDE gives r3 = ai We can also obtain an alternate r? by plugging in value of of for
,,, where
1-aî
VP-SDE into the expression of r derived for VE-SDE, which evaluates to r
2-af-. Empirically, we find that r for
VE-SDE marginally outperforms VP-SDE. We report performance of IIGDM with both choices of r in Table 22 24.
Table 22: Relative performance of IIGDM on face-blurred ImageNet-64 with VE and VP derived r with σg 0.05
VE
Measurement
Model
FID |
LPIPS
PSNR t
SSIM
FID
LPIPS
PSNR 1
SSIM
SR 2x
OT
6.52
0.168
30.54
0.753
5.91
0.160
30.60
0.762
Gaussian deblur
OT
55.19
0.374
28.74
0.516
39.36
0.326
29.00
0.572
Inpainting- Centtrr
O9
9.25
0.111
34.13
0.863
8.70
0.109
34.17
0.864
Denoising
OT
16.59
0.102
34.60
0.906
16.44
0.101
34.64
0.907
SR Gaussian deblur
VP-SDE
6.84
0.174
30.48
0.743
6.11
0.166
30.54
0.753
VP-SDE
54.77
0.376
28.74
0.511
39.14
0.329
28.99
0.567
Inpainting- Center
VP-SDE
9.75
0.113
34.03
0.860
9.36
0.112
34.06
0.862
Denoising
VP-SDE
17.19
0.107
34.25
0.901
15.54
0.102
34.41
0.906
Table 23: Relative performance of ITGDM face-blurred ImageNet-125 with VE and VP derived rf with oy 0.05
VP
VE
Measurement
Model
FID |
LPIPS 1
PSNR 1
SSIM
FID J
LPIPS
PSNR 1
SSIM
SR 2x
OT
4.38
0.148
32.07
0.831
4.26
0.145
32.12
0.834
Gaussian deblur
OT
30.30
0.328
29.96
0.606
22.42
0.296
30.17
0.642
Inpainting- Center
OT
7.99
0.122
34.57
0.867
7.64
0.120
34.61
0.869
Denoising
OT
9.60
0.107
35.11
0.903
9.30
0.104
35.21
0.906
SR 2x Gaussian deblur
VP-SDE
7.55
0.183
31.61
0.785
6.14
0.168
31.79
0.803
VP-SDE
55.61
0.463
28.57
0.414
41.69
0.404
28.98
0.493
Inpainting- Center Denoising
VP-SDE
9.75
0.130
34.45
0.858
9.46
0.129
34.49
0.859
VP-SDE
10.69
0.124
34.72
0.882
10.11
0.119
34.92
0.886
Choice of starting time. For OT-ODE sampling and VP-ODE sampling, we observe that starting at time / 0 improves the performance. We therefore perform an ablation study on IIGDM baseline, and vary the start time to verify whether starting at > helps to improve the performance. We plot the metrics for three different measurements in Figure 30. We observe that starting later at time > 0 consistently leads worse performance compared starting at time / 0. Therefore, for all our experiments with IGDM,, always start time
42

---

Published in Transactions on Machine Learning Research (06/2024)
Table 24: Relative performance of LIGDM on AFHQ with VE and VP derived rî with By = 0.05
VP
VE
Measurement
Model
FID 4
LPIPS
PSNR
SSIM t
FID
LPIPS 4
PSNR f
SSIM
SR 4x
OT
12.69
0.285
30.18
0.665
12.31
0.282
30.23
0.672
Gaussian deblur
OT
24.60
0.383
28.93
0.429
19.66
0.355
29.16
0.475
Inpainting-Cenne Center
OT
19.09
0.153
34.20
0.855
16.51
0.145
34.40
0.863
Denoising
OT
11.20
0.159
34.49
0.876
10.92
0.153
34.78
0.883
SR 4x Gaussian deblur
VP-SDE 77.49
0.469
29.34
0.469
54.12
0.413
29.73
0.549
VP-SDE
116.42
0.535
28.49
0.313
95.09
0.493
28.74
0.368
Inpainting-Centt
VP-SDE
57.46
0.239
32.40
0.773
56.86
0.238
32.42
0.775
Denoising
VP-SDE
81.15
0.451
29.62
0.639
35.33
0.278
31.72
0.776
Goussian Deblur
Gaussian Deblur
Gaussien Deatur
Gaussian Deblar
Io3M (UP)
0.50
20.0
0.45
nooo (V6)
0.45
20.0
28.6
0.35
0.50
0.0
0.1
0.2
0.5
0.4
0.0
0.1
5.2
0.5
0.4
0.0
0.2
0.2
0.3
0.4
0.0
0.3
0.2
0.5
0.4
start
time
start time
Super Super.resolutton resolltion 4
Super.resoluton
Super resolutton 4x
ruperr-rsolltion 4x
0.28
20.2
0.33
F10
28.0
0.32
0.575
24.4
0,550
0.28
25.2
0.535
start time
start time
start time
start time
Inpainting Center
Inpainting Center
Inpainting Center
Inpainting Center
0.58
34.4
nG2N (UP)
24.2
noore (ve)
24.0
0
190
0.18
PSNR
22.8
0.04
23.6
17.6
0.15
0.0
0.1
0.2
0.5
0.4
0.0
0.1
0.2
0.5
0.4
0.0
0.2
0.2
0.3
0.4
0.0
0.1
0.2
0.5
0.4
time
time
Figure 30: Variation in performance ITDM sampling with variation in start times on AFHQ dataset. We use pretrained conditional OT model and set og 0.05. We observe similar trends with VP-SDE checkpoint. We plot metrics for both choices of r? that can be derived from variance preserving and variance exploding formulations.
E.2 RED-Diff
Implementation details. We use VP-SDE model for all experiments with RED-Diff. We closely follow the official code available on github while implementing RED-Diff. Similar (Mardani et al., 2023), we always use uniform spacing while iterating the timestep over 1000 steps. We use ascending time from 0 to 1. Note that the original paper uses descending time from T to 0. According to the notational convention used in this paper, this is equivalent to ascending time from 0 to 1. We use Adam optimizer and use the momentum pair (0.9, 0.99) similar to the original work. Further, we use initial learning rate of 0.1 for AFHQ and ImageNet-128 128, as used in the original work, and learning rate of 0.01 for ImageNet-64. We use batch size of 1 for all the experiments. Finally, we extensively tuned the regularization hyperparameter to find the value that results in optimal performance across all metrics. We summarize the results of our experiments in Table 25 30. We note that more extensive tuning may be able find better performing hyperparameters but this goes against the intent of training-free algorithm.
43

---

Published in Transactions on Machine Learning Research (06/2024)
Table 25: Hyperparameter search for RED-Diff on face-blurred ImageNet-64 x 64 with dy = 0.05. We use learning
rate of 0.01.
SR 2x, By = 0.05
Gaussian deblur, dg = 0.05
FID 4
LPIPS 4
PSNR
SSIM
FID 4
LPIPS 4
PSNR
SSIM
0.1
34.09
0.224
30.12
0.798
46.76
0.254
29.29
0.715
0.25
28.45
0.206
30.40
0.814
51.20
0.236
30.19
0.776
0.75
23.02
0.187
31.22
0.839
73.76
0.287
30.47
0.750
1.5
32.35
0.243
30.80
0.792
82.26
0.335
30.29
0.705
2.0
40.33
0.284
30.41
0.750
86.48
0.358
30.17
0.683
Inpainting Center, By
0.05
Denoising, dy
0.05
FID 4
LPIPS 4
PSNR
SSIM
FID 4
LPIPS 4
PSNR
SSIM
0.1
15.71
0.155
31.74
0.840
12.47
0.085
32.24
0.907
0.25
15.56
0.155
31.73
0.839
11.80
0.083
32.36
0.908
0.75
13.31
0.139
32.65
0.857
8.43
0.062
33.65
0.932
1.5
12.18
0.119
33.97
0.881
6.11
0.041
35.34
0.958
2.0
12.87
0.119
34.19
0.886
6.02
0.041
35.64
0.964
Table 26: Hyperparameter search for RED-Diff face-blurred ImageNet-64 x 64 with dy 0. We use learning rate
of 0.01.
SR 2x, 0y =
Gaussian deblur, dy
FID 4
LPIPS 4
PSNR
SSIM
FID J
LPIPS $
PSNR
SSIM 1
0.1
11.74
0.224
30.12
0.798
15.39
0.134
31.99
0.879
0.25
12.65
0.130
32.34
0.886
29.56
0.236
30.19
0.776
0.75
20.36
0.187
31.22
0.839
55.43
0.287
30.47
0.750
1.5
33.13
0.243
30.80
0.792
71.64
0.335
30.29
0.705
2.0
41.56
0.288
30.46
0.752
78.55
0.358
30.22
0.685
Sraintiin-Centerr 0g =
À
FID 4
LPIPS
PSNR
SSIM
0.1
7.29
0.079
39.14
0.925
0.25
7.40
0.155
31.73
0.839
0.75
8.47
0.083
38.59
0.922
1.5
10.75
0.095
37.42
0.916
2.0
12.54
0.119
34.19
0.886
44

---

Published in Transactions on Machine Learning Research (06/2024)
Table 27: Hyperparameter search for RED-Diff on face-blurred ImageNet-128 x 128 with og = 0.05. We use learning
rate of 0.1.
SR 2x, dy = 0.05
Gaussian deblur, dy 0.05
FID
LPIPS 4
PSNR
SSIM
FID |
LPIPS 4
PSNR
SSIM 1
0.1
23.25
0.272
30.12
0.731
37.83
0.42
28.54
0.473
0.75
14.56
0.224
30.71
0.782
21.43
0.229
31.41
0.807
1.5
10.54
0.182
31.82
0.852
22.85
0.247
31.65
0.809
2.0
11.65
0.187
31.93
0.859
24.71
0.259
31.61
0.802
Inpainting- Center, By
0.05
Denoising, By
0.05
A
FID +
LPIPS 4
PSNR
SSIM
FID 4
LPIPS 4
PSNR
SSIM 1
0.1
19.68
0.191
31.75
0.795
12.83
0.134
32.27
0.854
0.75
19.03
0.202
31.36
0.779
12.69
0.14
32.09
0.846
1.5
16.33
0.189
31.81
0.794
10.67
0.121
32.89
0.874
2.0
14.63
0.171
32.42
0.819
9.19
0.105
33.52
0.895
Table 28: Hyperparameter search for RED-Diff on face-blurred ImageNet-125 x 128 with = 0. We use learning
rate of 0.1.
SR 2x, 0g 0
Gaussian deblur, dy =
FID +
LPIPS 4
PSNR
SSIM
FID 4
LPIPS 4
PSNR
SSIM 1
0.1
3.90
0.082
34.47
0.922
4.19
0.085
34.68
0.929
0.75
6.52
0.105
33.54
0.905
12.59
0.177
32.71
0.864
1.5
10.46
0.142
32.98
0.894
19.29
0.225
32.15
0.831
2.0
13.08
0.165
32.65
0.884
22.57
0.245
31.94
0.816
Inpainting Center, By
0
Inpainting Frretiinn,--reommmmmmm Ty
A
FID +
LPIPS |
PSNR
SSIM
FID |
LPIPS 1
PSNR
SSIM 1
0.1
5.39
0.068
38.91
0.928
8.94
0.162
35.54
0.830
0.75
5.52
0.073
38.11
0.924
9.26
0.166
35.05
0.826
1.5
6.09
0.079
37.32
0.920
10.13
0.172
34.58
0.821
2.0
6.68
0.083
36.87
0.917
10.87
0.176
34.30
0.818
45

---

Published in Transactions on Machine Learning Research (06/2024)
Table 29: Hyperparameter search for RED-Diff AFHQ with By 0.5. We use learning rate of 0.1.
SR 4x, 0y = 0.05
Gaussian deblur, dy = 0.05
FID 4
LPIPS 4
PSNR
SSIM
FID 4
LPIPS +
PSNR
SSIM
0.1
21.59
0.385
29.51
0.607
17.36
0.379
29.95
0.639
0.25
22.47
0.374
29.66
0.635
15.81
0.341
30.15
0.645
0.75
20.84
0.331
29.97
0.675
25.41
0.366
29.76
0.588
1.5
22.46
0.355
29.68
0.642
38.66
0.409
29.34
0.525
2.0
25.02
0.376
29.49
0.618
45.01
0.427
29.18
0.500
Inpainting- Center, By
0.05
Denoising, By
0.05
FID 4
LPIPS 4
PSNR
SSIM
FID 4
LPIPS 4
PSNR
SSIM
0.1
28.39
0.216
31.53
0.756
8.32
0.159
32.18
0,827
0.25
28.85
0.217
31.51
0.755
8.35
0.161
32.16
0.826
0.75
28.80
0.218
31.64
0.759
7.94
0.156
32.35
0.833
1.5
28.74
0.205
32.19
0.784
6.63
0.138
33.12
0.862
2.0
28.55
0.190
32.63
0.802
5.71
0.124
33.70
0.882
2.5
28.71
0.177
32.99
0.818
4.93
0.111
34.18
0.899
Table 30: Hyperparameter search for RED-Diff on AFHQ with 0g = 0. We use learning rate (Ir) of 0.1 unless mentioned
otherwise.
SR 4x, TT =
Gaussian deblur, Jy = 0
A
FID
LPIPS 4
PSNR
SSIM f
4
LPIPS
PSNR
SSIM 1
0.005
11.67
0.197
32.93
0.837
14.69
0.278
31.73
0.760
0.05
8.65
0.191
32.21
0.801
11.67
0.268
31.30
0.731
0.1
9.65
0.204
31.84
0.781
11.53
0.273
31.05
0.711
0.25
11.65
0.222
31.53
0.768
13.22
0.293
30.63
0.675
0.75
14.98
0.274
30.72
0.726
23.34
0,351
29.91
0.598
1.5
19.40
0.332
29.95
0.665
36.96
0.402
29.39
0.529
2.0
22.72
0.361
29.65
0.632
43.64
0.422
29.22
0.504
Inpainting Center, By O, 1r=0.01
Inpainting-
6g
A
FID
LPIPS 4
PSNR
SSIM f
FID
LPIPS
PSNR
SSIM 1
0.005
8.53
0.050
38,89
0.951
7.22
0.091
36.89
0.892
0.05
8.53
0.050
38.89
0.951
7.27
0.090
36.88
0.892
0.1
8.53
0.050
38.88
0.951
7.23
0.091
36.82
0.891
0.25
8.53
0.050
38.83
0.950
7.32
0.094
36.69
0.889
0.75
8.88
0.056
38.60
0.948
7.74
0.102
36.26
0.884
1.5
10.32
0.071
38.04
0.942
8.41
0.112
35.69
0.877
2.0
11.62
0.084
37.54
0.937
8.76
0.119
35.37
0.872
46

---

Published in Transactions on Machine Learning Research (06/2024)
F
Additional Background
In this section, we follow the notation used in the prior work by Lipman et al. (2022),
Continuous Normalizing Flows (CNFs), A Continuous Normalizing Flow (Chen et al., 2018a) is time-dependent
diffeomorphic map Φi [0,1] x R > Rd that is defined by the ODE:
0((z) = (o((()))); Φv(②) = x
(20)
dt
where Rd and v, [0,1] x R + Rd is time-dependent vector field that is usually parametrized with neural network. The generative process of CNF involves sampling from simple prior distribution ~ po(o)) (e.g. standard Gaussian distribution) and then solving the initial value problem defined by the ODE in Eq. (20) to obtain a sample from the target distribution æ ~ p1 (21). Thus, a CNF reshapes a simple prior distribution po to more complex distribution Pt. via a push-forward equation based on the instantancous change of variables formula.
Pt = [0].Po
(21)
TØ,1 Ox (z)
[o].Po pu(o, 'xx))
(22)
CNFs are usually trained by optimizing the maximum likelihood objective. As shown in Chen et al. (2018a), the exact likelihood computation can be done via relatively cheap operations despite the Jacobian term. However, this requires restricting the architecture of the neural network to constrain the Jacobian term. FFJORD (Grathwohl et al., 2018) improves upon this by proposing a method that uses Hutchinson's 's trace estimator to compute log density, and allows CNFs with free-form Jacobians, thereby removing any restrictions on the architecture. This approach has difficulties for high- digh-dimensional images where the trace estimator is noisy. Flow Matching provides an alternative, scalable approach to training CNFs with arbitrary architectures.
Flow Matching. Suppose we have samples from an unknown data distribution g( ). Let pa denote a probability path from the prior distribution to the data distribution that is approximately equal to q. Flow Matching loss is defined as
LFM = px (2; 0) -u, (x))2
(23)
where u. (x) is a vector field that generates the probability path (x)). and denotes trainable parameters of the CNF. In practice, we usually do have any prior knowledge on Pt and u,, and thus this objective is intractable. Inspired by diffusion models, Lipman et al. (2022) propose Conditional Flow Matching, where both the probability paths and the vector fields are conditioned on the sample æ1 g(a1). The exact objective for Conditional Flow matching is given by
LCFM (.. (l [::
0) un (x(xx)))1
(24)
where, (=( denotes conditional probability path, and u, (=: denotes the corresponding conditional vector field that generates the conditional probability path. Interestingly, both the loss objectives in Eq. (24) and Eq. (23) have identical gradients w.r.t. 0. More importantly, past research has proven that ur(x) = E[u,(1 (2))) ; = x). The optimal solution to the conditional Flow Matching recovers (z)) and therefore (z; 0) generates the desired probability path pi (x)) (a)..TTus Thus, we can train a CNF without access to the marginal vector field ux(x) or probability path (z)) Compared to the prior approaches to train flow models, Flow Matching allows simulation-fri training with unbiased gradients, and scales easily high dimensions.
47

---

Published in Transactions on Machine Learning Research (06/2024)
G Additional Related Work
Inverse problems are ubiquitous in various domains like image processing (Krishnan & Fergus, 2009; Rick Chang et al., 2017; Gilton et al., 2019; Bertalmio et al., 2000; Yang et al., 2010), medical imaging (Ribes & Schmitt, 2008; Jin et al., 2017; Liang et al., 2020; Song et al., 2021c), and remote sensing (Krasnopolsky, 2009; Krasnopolsky & Schiller, 2003; Dong et al., 2018). Many approaches have been developed over years to solve inverse problems. Variational methods (Agrawal et al., 2022) formulate the inverse problem as an optimization task with a regularization term (Benning & Burger, 2018) for certain desirable properties in the solution. Some of the well-known frameworks in this category include plug-and-play prior (P²) (Venkatakrishnan et al., 2013; Chan et al., 2016; Kamilov et al., 2017; Meinhardt et al., 2017; Zhang et al., 2017: Vidal et al., 2020), deep image prior (DIP) (Ulyanov et al., 2018; Van Veen et al., 201 8), and regularization by denoising (RED) (Romano et al., 2017; Cohen et al., 2021). Subsequent works, inspired by these prior works, have extended these frameworks to include flow models (Whang et al., 2021a:b), optimal transport (Vidal et al., 2020), and more recently, diffusion models (Mardani et al., 2023; Graikos et al., 2022; Liu et al., 2023b) as prior. Optimization-basee inversion methods were also extended to include GANs (Bora et al., 2017; Shah & Hegde, 2018; Raj et al., 2019; Daras al., 2021; Pan et al., 2021). Optimization-bana based approaches for solving inverse problems, despite their widespread popularity, have certain drawbacks. These methods are often computationally expensive as they involve optimizing an objective, which might require many steps to converge to solution. Further, designing the optimization objective itself can be challenging. In addition, these methods are sensitive to the choice of hyperparameters like regularization parameter, as noted in our experiments with RED-Diff (Mardani et al., 2023).
With emergence of diffusion models, another family of gradient-based approaches for inverse problems have emerged. These approaches do not explicitly optimize an objective, i.e., they are training-fres frree, but they use gradients to guide the sampling process with diffusion model as prior. These approaches usually involve iterative denoising through a SDE and gradient-based correction that is applied each step of the process. Some of the approaches in this category include Diffusion Posterior Sampling (DPS) (Chung et al., 2022a), Manifold Constraint Gradient (MCG) (Chung et al., 2022b), IIGDM (Song et al., 2022), and shortcut sampling for diffusion (SSD) (Liu et al., 2023a). Our proposed approach for solving linear inverse problems with flow models also falls under this category. Computing gradients at each step of denoising can be expensive. There are many gradient-free iterative methods for inversion that utilize diffusion models as generative prior. Some prominent approaches in this category are denoising diffusion restoration models (DDRM) (Kawar et al., 2022) and denoising diffusion null-space model (DDNM) (Wang et al., 2022). We have covered important distinctions between these approaches briefly in Sec. of the main paper.
The aforementioned approaches for solving inverse problems with diffusion models use pre-trained diffusion models and are not specific to a particular measurement operator. There are works such as (Saharia et al., 2021; 2022a) that train a conditional diffusion model to solve a specific inverse problem. This approach for solving inverse problems is more computationally expensive as it involves training a model from scratch. Further, the resulting model is specific to the measurement operator used in the training data and cannot be reused to solve inverse problems with a different measurement operator.In addition the above, there also a line of research that considers the more general setting of blind inverse problem where the method solve an inverse problem is agnostic to the measurement operator. Some works that have advanced this line of research are Chung al. (2023b); Gan et al. (2024); Laroche et al. (2024). Finally, we note that there are previously proposed methods such as Whang et al. (2021a;b); Hong et al. (2023) which solve inverse problems using CNFs. As noted in these prior works, using CNFs for solving inverse problems presents computational challenges as well as challenges due to restricted architecture. In this work, we consider CNFs that are trained with flow matching (or similarly converted diffusion models) which are more computationally more efficient and do not suffer from drawbacks observed due to restricted architectures.
48


Reviews:
[{"review_id": "csnEp2jy10", "reviewer": "Reviewer_RAno", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is reasonably well written and clean.\n- The main experimental section is reasonably convincing, and ablation studies quite comprehensive. \n\nWeaknesses:\n- The theoretical contributions appear to reasonably minor. The main analysis flow pretty directly from $\\Pi$GDM with some small variation in the terms to encompass Gaussian probability paths. \n- While I appreciate the presentation, I do wonder if there is a better way of presenting the main derivation. It relies on reference to $\\Pi$GDM at many points, rather than standing alone. A better format may be to refer to $\\Pi$GDM less for critical steps and write a self containing presentation. To be more concrete, removing the constant references to $\\Pi$GDM between the start of 3.2 and equation 16, making this section self consistent, and just after this writing a subsection on \"Relationship to $\\Pi$GDM\" to make clear the links to this work. \n\n- It also seems possible to improve the presentation in sections 3.1 and 3.2 to make the derivations less dependent on the flow matching point of view, as the derived method could be applied easily to diffusion models with Gaussian probability paths also.\n- The contribution \"We offer a way to convert between flow models and diffusion models\" is well known and not novel to this work, nor is it to be pedantic true - there are many flow models that do not have a diffusion counterpart. Showing how the conditional sampling framework derived applies to both (under the gaussian probability path assumption) would be novel.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work adapts the $\\Pi$GDM framework for training-free inverse problem conditional sampling of diffusion models for solving linear inverse problems to a restricted class of flow matching models, namely those with Gaussian probability paths, that intersect exactly with those achievable by a similarly restricted class of diffusion noising processes.\n\nThere are some experiments performed to show this method performs reasonably on tasks compared to a diffusion counterpart.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I believe a refining of 3.1/3.2 to be slightly less flow/diffusion specific would be beneficial, and then to show how the derived results apply to both cases would be a good refinement.\n\n- I think making 3.2 more standalone with less reliance on referring to $\\Pi$GDM would benefit readability, and including a short section on the similarities afterwards.\n\n- The paper presents itself as focusing on OT flows specifically. The results derived apply to all diffusions and flows with Gaussian probability paths. I would recommend that the authors separate out the theoretical claims, to show this method has wider applicability than to OT flows, and the specific findings in the experiments section that OT flows perform best. I think this will expand your readership scope and appeal a great deal. I would also highlight that your results can be applied to diffusion models as well, under the Gaussian probability path assumption. For example, the first contribution \n\n    \"We present a training-free approach for solving linear inverse problems with pretrained conditional OT flow models that incorporates the ΠGDM gradient adaptation, proposed for diffusion models, to flow sampling.\" \n\n(incidentally, this sentence looks malformed and needs fixing), could be stated much more generally. For example, one could say \n\n    \"We derive a training-free approach to solving linear inverse problems that can be applied to any diffusion or flow-based model, under the assumption that they use Gaussian probability paths, that extends the work of $\\Pi$GDM to this more generic setting\".\n\nThe empirical claim \n\n    \"We demonstrate that images restored via our algorithm using conditional OT probability paths ...\" \n\ncould be restated more widely as \n\n    \"We empirically test a range of different flow and diffusion models with different Gaussian probability paths. Empirically we find that OT probability paths in flow matching models match or outperform the diffusion setting of $\\Pi$GDM and RED-Diff, demonstrating the usefulness of this more framework for general Gaussian probability paths.\"\n\nThese are meant as examples, not hard suggestions.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "N5GFlTx9TL", "reviewer": "Reviewer_XMQw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThis is an interesting paper that adapts techniques in diffusion-based posterior sampling to flow models. The proposed method can reduce the need for hyperparameter tuning, which is beneficial in practice. The conversion formula between diffusion models and flow models looks novel and interesting.\n\n\n**Weaknesses**\n\n1. In the reviewer's humble opinion, some claims made in the paper are too absolute or not accurate.\n\n- \"without training\"/\"training-free\"\n\nThe authors stated multiple times that their method solves inverse problems \"without (any) training\" or is \"training-free\", for example,\n\"The challenge of solving noisy linear inverse problems without any training has been tackled in many ways...\"\n\nThe reviewer does not completely agree with this wording. The method is still data-driven and requires training a generative model, so it's not \"training-free\". It might be better to use phrases like \"without retraining\", \"problem agnostic\", \"model agnostic\", or \"not trained on specific inverse problems\", following conventions in inverse problem literature (e.g., see the PIGDM paper.)\n\n- \"no hyperparameter tuning\", for example,\n\n\"The algorithm is simple, stable, and requires no hyperparameter tuning when used with conditional OT probability paths.\"\n\nThe reviewer is not sure if this is accurate. Based on \"Methods and baselines\" and appendix, the updating rule of gamma_t, the initial time step and NFEs are all tunable.\n\n\n2. The performance of compared methods seems not as good as they should be. For example,\n\n- Figure 3: row 1, columns (e) and (f); row 2, column (e). These results either exhibit strong (rippling) artifacts or are oversmoothed. The authors might want to double check the hyperparameters for these methods (e.g., step sizes) to make sure the results are representative of these methods' true performance. In the reviewer's opinion, these results do not exhibit the representative performance of the state-of-the-art for these tasks.\n\n- Similar comments for Figure 4: Row 1, column (e) (strong local artifacts); Figure 5, column (f) (over-smoothed). These are not representative of a state-of-the-art method.\n\n\n3. Most of Section 3.2 is hard to follow for the reviewer. Some statements are vague, inexact, or not self-contained. Some notations seem to be borrowed from another paper (PIGDM) but are not defined clearly in this paper. For example,\n\n- Eq 12.: q^{app} seems undefined. What is the relationship between q^app and q? \n\n- Following the above, \"In general, we view adaptive weights γt ̸= 1 as an adjustment for error in qapp(y|xt).\" What error is there in q^app?\n\n- \"For qapp(y|xt), we generalize ΠGDM to any Gaussian probability path described by Eq. 5 via updating r_t^2.\" Again, r_t^2 seems undefined.\n\n- \"We choose r2t by following ΠGDM’s derivation, noting that ...\" How is r2t chosen exactly? What is the PIGDM derivation exactly?\n\n- \"if q(x1) is N(0, I) and q(xt|x1) is N(αtx1, σ2(\\sigma_t I), then rt2 = sgimat2 / (sigmat2 + alphat2)\" How is this (eq 13) derived? Why is it reasonable to assume the data distribution q(x1) is a Gaussian? (The distribution of most real-world data is more complext than a Gaussian.)\n\n\nIn short, it is clear at a high level that the authors are trying to adapt PIGDM to flow model. However, the writting is a little bit informal and inconcrete, making the exact methodology unclear. Perhaps a reader very familiar with the notations in PIGDM paper can understand this fairly easily, but for a general reader, this paragraph is a little bit hard-to-follow.\n\n\n\n4. \"Vector field correction\"/\"correcting the vector field of flow models\"\n\nThe reviewer feels that the term \"correction\" is a little bit misleading. \"Correction\" implies the original vector field has some \"error\" and the proposed method tries to fix the error. However, based on the reviewer's understanding, the original vector field is correct, but intended for unconditional sampling. The method aims to adapt/modify it to a new vector field enabling sampling conditioned on the measurements y.\n\n\nOther comments:\n\n- \"For Gaussian deblurring, we apply Gaussian blur kernel of size 61 × 61 with intensity value 1 for ImageNet-64 and ImageNet-128, and 61 × 61 with intensity value 3 for AFHQ.\" The 61x61 kernel size is too large for blurring. Is it a typo? What does intensity value mean here?\n\n- In Algorithm 1, what's the final output of the algorithm? After obtaining the \"corrected\" vector field, how is xt updated? Also, it might be better to denote the initial time point as t0, rather than t.\n\n- Figure 2: The reviewer is not sure if connecting the dots is appropriate, since in this figure there is no ordinal relation in x.\n\n- \"Under Gaussian probability paths, the two terms are related as Tweedie’s identity (Robbins, 1992) expresses...\" Sentence reads wrong.\n\n- \"In contrast, the inpainted regions generated by RED-Diff tend be blurry and less semantically meaningful.\" Should be \"tend to\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method for solving inverse problems using flow models. It presents theory and algorithms for adapting a pretrained flow-based generative model to approximately sample from the posterior in an inverse problem given measurements and forward operator, using techniques from diffusion-based inverse solvers (specifically, PIGDM). The method is problem-agnostic (able to use the same model to solve different inverse tasks without retraining) and requires less hyperparameter tuning than diffusion-based approaches.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The reviewer would suggest the authors make the following changes to make their claims well-supported.\n\nMake their claims more accurate (see weakness 1). \n\nImprove writing on methodology (section 3.2).\n\nMake sure the performance of compared methods are sufficiently finetuned and optimized. Make sure the results are representative of state-of-the-art and the comparison is fair.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CPFBW1hEXZ", "reviewer": "Reviewer_GG8v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Post-response update: After the discussion period and author revisions the paper is currently excellently written, provides a good contribution in linking flow matching and diffusion models, and provides a strong contribution in solving inverse problems using existing networks without further pre-training. This is a good paper that I'd be happy to see accepted.\n\n------\n\nS/W: The results show that the method works, and gives good performance. However, there are only 2 competing methods. This is not sufficient: there are tons of guided diffusion methods, and even a simple paper needs to have a wider comparison.\n\n~~W: I found the paper to be poorly written with frustratingly ambiguous math. I think I understood the main ideas, but I’m not totally sure. The contributions and novelties are a bit unclear, and the paper claims contributions that to me seem trivial. For example, the second contribution bullet claims that paper offers a way to convert between diffusion and flow models (by just following the ODE part). This does not seem novel.~~\n\n~~W: I think a main contribution is the PiGDM correction, but there is no ablation that shows if it does anything. This needs to be added: it’s one of the main claims, and currently it lacks evidence. Similarly the unguided-to-guided conversion seems to be a main contribution, but I can’t see any experiments about it. (Maybe these are in the paper, I’m not sure)~~", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to couple a fixed linear guidance to pre-trained diffusion/flow models for the purpose of image cleaning. I think the main contributions are turning unguided models into guided ones, and some ODE correction. I found the contributions to be vague, so I'm not totally sure.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Post-response update: The authors have addressed all of my comments, and made extensive improvements to the paper. I have no further comments, and would be happy to see this paper get accepted. \n\n-------\n\n~~To be accepted, the paper needs to clarify its contributions or claims, and pair them with empirical evidence, and clarify why the stuff that is happening is novel, significant or interesting for the community. Currently (i) I’m a bit confused of what are the contributions; (ii) I believe the correction and unguided-to-guided stuff is not backed by empirical evidence; and (iii) I’m not yet convinced that the paper’s material is interesting enough for the community, given that there is so much stuff that seems just common knowledge. Finally, lots of presentation clarifications are needed. I’m looking forward to author clarifications.~~\n\n**Below comments are outdated after revisions**\n\n- Before eq 1 it was stated that p_theta is noise-to-data, while in eq 1 q is noise-to-data. This is confusing.\n- In eq1 we integrate over q. What does this mean? What is q? What do we integrate over? What variables are free and what are fixed? The denoiser is deterministic (I assume, not sure, is it?) so the expectation can only be over x_t. Ok, but this then can mean two different things. Either we keep x_t fixed and integrate over x_1; or we keep x_1 fixed and integrate over different x_t’s. Or maybe we integrate over both x_t and x_1 (which means in reality integrating over eps and x_1). Also the role of q is confusing, since it can be either noise-to-data or data-to-noise. Can you explain what is q, and how should we interpret it, and what are its inputs?\n- Can you also explain what is p_theta, what are its inputs, and why its not present in almost any of the equations? Do we assume some kind of one-way diffusion models similar to GAN? Or is the denoiser hat{x} representing the p_theta?\n- Can you also comment on the nature of xhat: is it deterministic or stochastic?\n- The eq 1 continues by saying that we sample x0 (noise) from the data-to-noise process p. Not sure what this means. So do we start from a clean image, and turn it into pure noise? But why do we need to do this, why not just do pure noise right away? I think the two endpoint densities (pure noise, pure data) are special cases that need clearer treatment in the notation.\n- Or maybe eq 1 is trying to imply that there is an SDE hidden inside xhat, and maybe q refers to that?\n- In eq 2 I’m again confused what is q, or what variables it covers or touches. Does it give both xt and x1, or only xt?\n- In eq 2 the LHS has only v and q, but RHS has x_1, y, x_t. This then implies that the variables (x_1, y, x_t) are all coming from somewhere, and the only thing that can give them is the q. But I’m not sure if this is true. I suspect that the notation is incorrect, and we should add some conditioning to the LHS by saying L_cfm(v, q, x1, y) or something like that. Can you explain what is the nature of (x_1, x_t, y) and where do they come from, and which of them are fixed and which are variable?\n- What are the “marginal” distributions marginalised over? I assume we marginalise x1 and epsilon, is this true?\n- What’s the difference between p(x|y) and q(x|y)? Somehow the presentation is difficult to follow since the notation is flipped wrt usual style, and lots of stuff is undefined. I would guess that q(x|y) represents the noising distribution marginalised over epsilon and x1, and p(x|y) represents denoising distribution marginalised over x0? Can you clarify?\n- Before eq 4 we finally see that q is the Gaussian noising process, but earlier it was stated that p_theta is the noising process. It is also implied that q is mainly about x_t, which means that the E_q’s in previous equations are not integrating over the data distribution. Are these single-datapoint losses? What is the “q” in L_cfm(v,q)?\n- In eq 4 the ds/dt(x-mu/s) looks like a function evaluation, but it also might be just a product. Can you clarify?\n- I don’t understand what E[ x1 | xt ] means before eq 6. What is the q and expectation over? I think this is an average denoising given different xt’s, but that would then assume that we have some x1 reference that produces the xt’s. This is missing here. Or maybe this refers to an expectation over x1, which would be all possible denoising from a single fixed xt. But x1 usually means data, so maybe its not that. x1 can mean either denoising, or data. Or maybe the expectation is over the dataset, such that we look at all images, and see how likely each image is to produce a single fixed xt. But this also makes no sense: it does not matter how we arrived at xt. If we believe the eq 5, then this opens up as $E_q[x_1 | x_t] = \\int q(x_t | x_1, y) p(x_1 | x_t) dx_t$, but this is circular since it has x1 twice. Can you help me understand?\n- It is stated that “training-free approaches” aim to approximate E[x1|xt,y] from E[x1|xt]. Err… what? So training-free refers to adding the class/guidance conditionals. Is this really what you mean?\n- I don’t understand what probability paths mean. Is it an infinite sequence ( q_t(x_t|..))_t=0..1 or something like that? What is the path over? Is it a path of images x_t or densities q_t?\n- In eq 6 it seems that we obtain guided denoising by taking an unguided denoising, and adding a classifier gradient. Not sure if this makes sense. So we add the classifier shift only once? Isn’t this strange? Don’t we want to add the shift along the ODE or SDE?\n- I’m also confused what E_q[ x_1 | x_t] means. I don’t know what is q over, or what are we integrating over. I would guess that x_t is fixed as the conditioning variable, but then it means that we are integrating over x_1’s? Not sure what this means either. I’m also confused if x_1 means observed datapoints, or perhaps denoisings. I think the only sensible interpretation is that this is the expected denoising given a single noisy image x_t. But the denoiser is usually deterministic, so I guess this becomes Dirac. That doesn’t feel sensible. Another option is that we take an expectation over x_t’s, but which ones? Are these all x_t’s from one source image x_1, or maybe all x_t’s that could ever emerge from any image? I can’t find any interpretation that makes sense.\n- In eq 7 I don’t understand what measurements means. Measurement of what? Do we measure something from a sample x ~ q(x_1)? Or is this some general property vector we wish to fulfil? Do we actually go and measure something in the real world? I think the y was supposed to be a noisy image (eg. blur), so I wonder why is this called measurement. Can you help me understand the setting?\n- The density q(x1|y) implies that a joint distribution q(x1,y) exists. Can you elaborate on what would this represent? Can we define this joint? It would help a lot to give comprehensive model definitions.\n- I don’t understand the density q(x1|y). It looks like it represent the probability of all images assuming they are class y. But earlier we had a definition that q(x|..) is a simple Gaussian noising process. How does this then imply a density over clean images? Are the q in eq 5 and q in eq 7 different q’s? In sec 3.1. we are introducing even more q’s: Now we have q’ and q (not sure what these are). Also, is x1 an observed image from the dataset? Or is it a denoised image? Or just any image? Do we observe (x1,y) pair?\n- Overall it would be very useful to write the joint densities and their conditional out in full, and describe what they mean, and also remark how many different densities there are.\n- Lemma 1 is missing context. “optimal solution is known”, but wrt what? Of what? Are we talking about cfm loss eq 2? What does it mean that solution is “known”? Does it have a closed form solution, or does the solution just generally exist? This is too vague, and the assumptions and model setting has not been described precisely enough.\n- What is \\alpha_t and \\alpha_t’ in eq 9? So we have two alphas? Err… what? I’m totally lost. Are we changing the schedule for some reason so that we have one diffusion model with two schedules (whatever that means..)? Do we train two diffusion models with two different schedules? Why would we do that? Or are these the alpha value at two different timepoints as \\alpha_t and \\alpha_{t’}? I’m struggling to decrypt the paper.\n- What is X’_t’(t)? The notation is getting quite complicated. I think this is just showing that we can “stretch” xt from one schedule to equivalent xt on a slower/faster schedule. Is this true? But why do we care about this stuff in the first place? The section 3.1. seems to end without explaining why we did this stuff. I think the motivation could be that we want to use a “slow” schedule diffusion model, and sample it using a “fast” schedule, but I can’t see any usecases for this.\n- In eq 6 we move from xt to x1, and add the guidance correction term once. This implies that the time is somehow discretised into windows, and each has its own single correction. For example, if we go from x0 to xt, and then from xt to x1, we apply guidance twice. The eq 11 is supposed to be a version of eq 6, but in eq 11 we have the correction as part of the ODE time derivative, which means that we apply the correction infinitely many times. Do you see a discrepancy in this, and can you explain it?\n- Is theorem 1 novel?\n- What is q^app? Where does this come from? Why did we ditch the q(y|xt)? I can’t follow this in eq 12. I also can’t follow why we add the gamma. So we added the q^app and then add more stuff to fix the errors that q^app brings. Err… why not just use the original q(y|xt)?\n- So it seems that in eq 12 we use a linear-Gaussian regression from x1 to y. Why is this a good idea? I assume x1 refers to images and its pixels, and I guess y does the same (maybe?).   Surely a linear function is extremely bad at approximating any useful “measurements” of the image? Can you comment on this?\n- Why do we choose the N( xhat, r) approximation for q(x1|xt)? This again looks bad: we make a clean prediction and add gaussian noise around it. Surely this results in either unnecessarily noisy denoisings, or it’s not able to represent any useful uncertainty in the denoising. I’m not really following why we are doing any of this stuff.\n- The text says that q(x1) can be N(0,i). Err… wasn’t x1 supposed to be clean data? This is super confusing. Why would we ever assume data is pure noise?\n- In eq 14 the y now needs to be same size as x. I think this is confusing. The paper started by defining y as noisy image, then in the middle y was ambiguous, and now y is again noisy image. It would be cleaner to keep y same through the paper.\n- Can you describe the novelties of this work? Is this work just adding adaptive weighting to PiGDM?\n- In alg 1 I don’t think the rows 6 and 7 have been explained by the paper. Alg 1 seems to use z while rest of the paper uses x.\n- In experiments it seems that you use two conditional generative models. But wasn’t the whole point of the paper that you take unconditional models, and turn them to conditional ones by your method? What are you trying to demonstrate in the experiments?\n- How do you get the A matrix? What does the A matrix represent or mean? Is the A-matrix dependent on the corruption type or just some global matrix? Do you train these somehow?\n- How did you choose the adaptive weights? Based on what?\n- In the end it seems that you method is taking a diffusion model, and following denoiser ODE while being also pushed by a linear regression towards y. Is this accurate? How come this hasn’t been done already, or has it? Isn’t this just standard classifier-guidance?\n- I also frankly did not even understand what the “correction” is in this paper. Is it the q^app? Or gamma? Or maybe just eq 11? What do we “correct”? If we don’t correct, is there some error somewhere?", "broader_impact_concerns": "No issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The text is almost ready and the revision is merely mentioned to ensure that the suggestions of the reviewers are included in the final version. 

Concretely, I encourage the authors to follow **all** the recommendations of the reviewers and appropriately adjust their contributions. For instance, as the reviewers point out, the link between diffusion models and flow models have emerged before. Therefore, it would be recommended to make the claims match the findings in the literature. More importantly though, the text should be finalized by including all the comments and suggestions of reviewers in the regular text.
Decision: Accept (poster)