Generalized Consistency Trajectory Models for Image Manipulation
Authors: Beomsu Kim
Jaemin Kim
Jeongsol Kim
Jong Chul Ye
Venue: ICLR 2025 Poster (2025)
Keywords: Consistency Models
Image Manipulation
Primary area: generative models

Abstract: Diffusion-based generative models excel in unconditional generation, as well as on applied tasks such as image editing and restoration. The success of diffusion models lies in the iterative nature of diffusion: diffusion breaks down the complex process of mapping noise to data into a sequence of simple denoising tasks. Moreover, we are able to exert fine-grained control over the generation process by injecting guidance terms into each denoising step. However, the iterative process is also computationally intensive, often taking from tens up to thousands of function evaluations. Although consistency trajectory models (CTMs) enable traversal between any time points along the probability flow ODE (PFODE) and score inference with a single function evaluation, CTMs only allow translation from Gaussian noise to data. Thus, this work aims to unlock the full potential of CTMs by proposing generalized CTMs (GCTMs), which translate between arbitrary distributions via ODEs. We discuss the design space of GCTMs and demonstrate their efficacy in various image manipulation tasks such as image-to-image translation, restoration, and editing. Code is available at https://github.com/1202kbs/GCTM.

Published as a conference paper at ICLR 2025
GENERALIZED CONSISTENCY TRAJECTORY MODELS FOR IMAGE MANIPULATION
Beomsu Kim', Jaemin Kim", Jeongsol Kim & Jong Chul Ye
KAIST (beomsu. kim, kjm981 995, jeongsol, jong-ye we)ekkaitsttt.ckk
ABSTRACT
Diffusion models (DMs) excel in unconditional generation, as well as on applica- tions such as image editing and restoration. The success of DMs lies in the iter- ative nature of diffusion: diffusion breaks down the complex process of mapping noise to data into a sequence of simple denoising tasks. Moreover, we are able to exert fine- ane--graiedd control over the generation process by injecting guidance terms into each denoising step. However, the iterative process is also computationally in- tensive, often taking from tens up to thousands of function evaluations. Although consistency trajectory models (CTMs) enable traversal between any time points along the probability flow ODE (PFODE) and score inference with a single func- tion evaluation, CTMs only allow translation from Gaussian noise to data. This work aims to unlock the full potential of CTMs by proposing generalized CTMs (GCTMs), which translate between arbitrary distributions via ODEs. We discuss the design space of GCTMs and demonstrate their efficacy in various image ma- nipulation tasks such as to-gae translation, restoration, and editing. Code is available at https: githtb....b/...ggggg com/ 1 1202kbs//eeeeecc /GCTM.
INTRODUCTION
Diffusion-based generative models (DMs) learn the scores of noise-perturbed data distributions, which can be used to translate samples between two distributions by numerically integrating an SDE or a probability flow ODE (PFODE) (Ho et al., 2020; Dhariwal & Nichol, 2021; Song et al., 2021). They have achieved remarkable progress over recent years, even surpassing well-known generative models such as Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) or Variational Autoencoders (VAEs) (Kingma & Welling, 2014) in terms of sample quality. Moreover, diffusion models have found wide application in areas such image-to-image translation (Saharia et al., 2022), image restoration (Chung et al., 2022: 2023), image editing (Meng et al., 2022), etc.
The success of DMs can largely be attributed to the iterative nature of diffusion, arising from its foundation on differential equations multi-step generation grants high-quality image synthesis by breaking down the complex process of mapping noise to data into composition of simple denoising steps. We are also able to exert fine- ggained control over the generation process by injecting minute guidance terms into each step (Chung et al., 2022; Ho & Salimans, 2022). Indeed, guidance is an underlying principle behind numerous diffusion- image editing and restoration algorithms.
However, its iterative nature is also curse, diffusion inference often demands from tens thou- sands of number of neural function evaluations (NFEs) per sample, rendering practical usage diffi- cult. Consequently, there is now large body of works improving the inference speed of DMs. Among them, distillation refers to methods which train a neural network to translate samples along PFODE trajectories generated by pre-trained teacher DM in one or two NFEs. Representative dis- tillation methods include progressive distillation (PD) (Salimans & Ho, 2022), consistency models (CMs) (Song et al., 2023), and consistency trajectory models (CTMs) (Kim al., 2024b).
In contrast to PD or CMs which only allow traversal to the terminal point of the PFODE, CTMs enable traversal between any pair of time points along the PFODE as well as score inference, all in single inference step. Thus, in theory, CTMs are more amenable to guidance, and are applicable to wider variety downstream image manipulation tasks. Yet, there a lack of works exploring the effectiveness of CTMs in such context.
"Equal Contribution

---

Published as a conference paper at ICLR 2025
(0) ODE velocity
cvaluation
Distribution g(xo)
(c) Improved & accelerated gubded generation
(b) Teaversal between arbitracy time Intervals
(o)) Traveral from K
Distribution ((x,)
(43) Gradicnt guidance
backgrop
(c2) Calcubil Juss
(cl) Projection to
Stochastic generation frosn statkc iaputs
Flow Matching ODE: dr, Zegenulmm (Ro) di
1
((x,)
(xgg
((xy)
Figure I: An illustration of GCTM and its applications solid arrows can be implemented by a single forward pass of the GCTM network. GCTMs learn to traverse the Flow Matching ODE which is capable of interpolating two arbitrary distributions ((xo) and ((x1). GCTMs allow (a) one-step inference of ODE velocity, (b) one-step traversal between arbitrary time intervals of the ODE, (c) improved gradient-guidance by using exact posterior sample instead of posterior mean, and (d) one-step generation of varying outputs æn given a fixed input æ1.
In this work, we take a step towards unlocking the full potential of CTMs. To this end, we first pro- pose generalized CTMs (GCTMs) which generalize the theoretical framework behind CTMs with Flow Matching (Lipman et al., 2023; Pooladian et al., 2023; Tong et al., 2024) to enable translation between arbitrary distributions. Next, discuss the design space of GCTMs, and how each design choice influences the downstream task performance. Finally, we demonstrate the power of GCTMs variety of image manipulation tasks. Our contributions be summarized as follows.
Generalization of theory. We propose GCTMs, which uses conditional flow matching theory to enable one-step translation between two arbitrary distributions (Theorem 1). This stands in contrast to CTMs, which is only able to learn PFODEs from Gaussian to data. In fact, we prove CTM is special case of GCTM when one side is Gaussian (Theorem 2).
Elucidation of design space. We clarify the design components of GCTMs, and explain how each component affects downstream task performance (Section 4.... In particular, flexible choice of couplings enable GCTM training in both unsupervised and supervised settings, allowing us to accelerate zero-shot and supervised image manipulation algorithms. Empirical verification. We demonstrate the potential of GCTMs on unconditional genera- tion, mmage-to-image translation, image restoration, image editing, and latent manipulation. We show that GCTMs achieve competitive performance even with NFE = 1.
RELATED WORK
Diffusion model distillation. Despite the success of diffusion models (DMs) in generation tasks, DMs require large number of function evaluations (NFEs). As a way to improve the inference speed, the distillation method is proposed to predict the previously trained teacher DM's output, egg. score function. Progressive distillation (PD) (Salimans & Ho. 2022) progressively reduces the NFEs by training the student model to learn predictions corresponding to two-steps the teacher model's deterministic sampling path. Consistency models (CMs) (Song et al., 2023) perform distillation by reducing the self-consistency function over the generative ODE. The above methodologies only consider the output of the ODE path. In contrast, consistency trajectory model (CTM) (Kim et al., 2024b) simultancously learns the integral and infinitesimal changes of the PFODE trajectory. Our paper extends CTM to leam the PFODE trajectory between two arbitrary distributions.
2

---

Published as a conference paper at ICLR 2025
Zero-shot image restoration via diffusion. Image restoration such super-resolution deblurring, and inpainting can be formulated as inverse problems, which obtain true signals from given obser- vations. With the advancements in DMs serving as powerful priors, diffusion based inverse solvers have been explored actively. DDRM (Kawar et al., 2022) performs denoising steps on the spectral space of linear corrupting matrix. DPS (Chung et al., 2022) and ITGDM (Song et al., 2022) pro- pose posterior sampling by estimating the likelihood distribution through Jensen's 's approximation and Gaussian assumption, respectively. While diffusion- basoodd inverse solvers facilitate zero-shot image restoration, they often need prolonged sampling times. CoSIGN (Zhao et al., 2024) addresses this problem by using CMs as generative priors, but we show that GCTMs can be better priors.
Image translation via diffusion. The seminal work Pix2Pix (Isola et al.. 2017) achieves image- to-image translation with conditional GANs. SDEdit (Meng et al., 2022) avoids mode collapse and learning instabilities with GANs by utilizing DMs to translate edited images along SDEs. Palette (Saharia et al., 2022) proposed conditional DMs for image-to-image translation tasks. To address the Gaussian prior constraint with DMs, Schrödinger bridge (SB) (Liu et al., 2023; Kim et al., 2024a), direct diffusion bridge (DDB) (Delbracio & Milanfar, 2023), or denoising diffusion implicit bridge (DDIB) (Su et al., 2023) methods have been proposed learn SDEs or ODEs between arbitrary two distributions. However, such models often require large NFEs. This has inspired models which distill conditional ODE trajectories (Mei al., 2024; Xiao al., 2024), DDB trajectories (He al., 2024), or DDIB trajectories (Starodubcev et al., 2024). GCTMs are more general in the sense that they enable velocity evaluation and translation between two arbitrary timesteps.
BACKGROUND
3.1 DIFFUSION MODELS (DMS)
DMs (Song et al., 2021; Ho al., 2020) learn reverse the process of corrupting data into Gaussian noise. Formally, the corruption process can be described by forward SDE
dæ, 22 dw,
(1)
defined on the time interval T E (0, oo). Given To distributed according to data distribution p( (x)) (1) sends To to Gaussian noise as T increases from 0 to co. The reverse of the corruption process can be described by the reverse SDE
dæ, = 2T V logp(x;; dr + 2T dw,
(2)
or its deterministic counterpart, the probability flow ODE (PFODE)
dx, -VV - p(xx) = -x, Ep( (moo) (30)) dr
(3)
where p(a.,) is the distribution of m, following (I), and in, the standard Wiener process in reverse-time. Given noise ~ (Nx==00, 42 I) R p(x+)) for some large f, x, following (2) or (3) distributed p(x,)) as 7 decreases from é to Thus, DMs are able to generate data from noise by approximating the scores V poz,( via score matching and numerically integrating (2) or (3).
3.2 CONSISTENCY TRAJECTORY MODELS (CTMs)
CTMs (Kim et al., 2024b) learn to translate samples between arbitrary time points of PFODE tra- jectories, i.e., the goal of CTMs is to learn the integral of the PFODE
G( T. == x. J7 u (ax. (p..)) [xx]) du
(4)
for a S T, where the terminal distribution p(æx)) is assumed to be Gaussian. The parametrization
G( T,0) 0 xr + g(x,,T,,)) = Mr + T-a J u - - (I, (e (aoo]) du
o(xt,,,0)
(5)
enables both traversal along the PFODE as well as score inference, since
lim g( T. = Em(mo)a [zo] o
(6)
so we may define g(mr, T) == Ep x.) (20]-
3

---

Published as a conference paper at ICLR 2025
Given pre-trained DM, CTMs approximate with neural net ge by simultancously minimizing distillation loss and a denoising score matching (DSM) loss. The distillation loss is
(0) EogosuersoBnto.) [d (T, Geg(o) (Ir-+u U, (.)
where Gy is the G-function with go in place of g. d(:,,) :) is a measure of similarity between inputs, sg is the stop-gradient operation, and - is the integral of the PFODE from time T to u starting from , using score estimates from the pre-trained diffusion model. Minimization of (7) causes Gg to adhere to PFODE trajectories generated by the pre-trained diffusion model. The DSM loss is
(D) == Bosr Wetrsstpepeee (x(e
|| go 7,
(8)
where p(æ, ((,,,,,, E) = - (xx,) and (-) is a Dirac delta at y. Minimization of (8) causes go to satisfy (6) (Vincent, 201 1). This loss acts as a regularization which improves score accuracy, and is crucial for sampling with large NFEs (Kim et al., 2024b). Thus, the final CTM training objective is
(0) + ADSMCDSM
(9)
and it possible to further improve sample quality by adding a GAN loss.
3.3 FLOW MATCHING (FM)
FM (Lipman et al., 2023; Tong et al., 2024; Pooladian et al., 2023) is another technique for learning PFODEs between two distributions g(<r)) and g(m)). Specifically, let g æ1) be a joint distribu-
tion of ((xo) and q(x1). Define
(0, Do, x1) == () (zx), g(xx,) =
x1)]
(10)
where / E (0, 1). Then, by Theorem 3.1 Tong et al. (2024), the ODE given by
dx; Ey(an
(x:) (m.)) xo] dt
(11)
generates the probability path g(x)), i.e. with terminal condition æ ~ g(z)), ). æz following (11) is distributed according to q(æ:)) Analogous to DSM, the velocity term in (11) can be approximated
by neural network Us which solves regression problem (see Theorem 3.2 in Tong al. (2024))
min E ( [[[( zo) u2 +)]
(12)
However, unlike diffusion whose terminal distribution p(æ*a) is Gaussian, g(xr)) can be arbitrary. We provide a complete proof of correctness of this section Appendix C.I.
4 GENERALIZED CONSISTENCY TRAJECTORY MODELS (GCTMs)
We now present GCTMs, which generalize CTMs to enable translation between arbitrary distribu- tions. We begin with a crucial proposition which proves we can parametrize the solution to the FM ODE (11) in a form analogous to CTMs. The proof is deferred to Appendix C.2.
Proposition 1. The ODE (11) equivalent to
dx; = (xt Ey (x [20o] dt
(13)
defined on t E (0, 1). Hence, we can express the solution to (11) as
G(In,t,) t, | + (1 10)))(()) t, g((,,,t,)) = De I-s Sea u- (x. Eeycoolee)) (zo)) du.
(14)
There are two differences between (5) and (14), First, the time variables and s now lie in the x.)) is replaced with 9 g The second difference is what enables translation between arbitrary distributions, as q(zo(zx,) recovers clean images 2o given images x. perturbed by arbitrary type of vectors (e.g., Gaussian noise, images, etc.), while p(æo|ar..) recovers clean images æn only for Gaussian-perturbhe samples æ.. We call a neural network ge which approximates in (14) a GCTM, and we can train such a network by optimizing the FM counterparts of LCTM and LDSM:
unit interval (0. 1) instead of oo), and second,
(GCTM(0) = Eo (ossss<rsiEE(mn)) [d (Go(zz, t, s), (eg( u, s))]
(15)
(0) = Eoses 1 (xn, m B / mm)) [||00 g0 t
(16)
The next proposition shows that the PFODE (3) learned by CTMs is a special case of the ODE (13) learned by GCTMs, GCTMs indeed generalize CTMss The proof is deferred to Appendix C.3.
4

---

Published as a conference paper at ICLR 2025
Proposition 2. Consider the choice of q( p(æv) (x 0. Let
t== 7 7); Dt == -z +)
(17)
where 7 E (0, 20) and x, follows the PFODE (3). Then
Pp(e xx) [0)] y xz m
(18)
and æe follows the ODE
dx, t (x. Ln (xe) (x)) dt
(19)
on L (0,1). Furthermore, let GCTM, ICTM denote CTM solutions and let GGOTM, JGCTM denole GCTM solutions. Then with s s= 0 (.),
(er,( (a, (1 8),
(20)
(T,, T, = (),
In short, (18) shows the equivalence of scores, and (19) shows the equivalence of ODEs. Thus, given go trained with LrM and LGCTM with the setting of Prop. 2, we are able to evaluate diffusion scores and simulate diffusion PFODE trajectories with a simple change of variables (17), as shown in (20).
Given GCTM's capability to replicate CTM, we will now outline the key components of GCTM that enable its significant extension for various downstream tasks. This flexibility offers a notable advantage of GCTM over CTM.
4.1 THE DESIGN SPACE OF GCTMS
Coupling g( 1) In contrast to diffusion which only uses the trivial coupling ((0o, æ1) ((x0)g(21) in (D), FM allows us to use arbitrary joint distributions of q(xu) and q(>1) in (f). Intuitively, ((Zo. æ1) encodes our inductive bias for what kind of pairs (Zo. æ1) we wish the model to learn, since FM ODE is distributed g(x;;) at each time I, and g(x:)) the distribution of (1 - 1) + ix, for (æo, 21) ~ (( 21). Here, we list three valid couplings of GCTM as examples (see Alg. for code). In contrast, CTM only uses a special case the independent coupling.
Independent coupling:
g 211 ((xo))(<=))
(21)
This coupling reflects no prior assumption about the relation between and æ; As shown earlier, diffusion models use this type of coupling with standard normal q(x1).
Minibatch entropic optimal transport (EOT) coupling: in practice, FM loss (16) approx- imated by an average over minibatch of pairs (zo.21) We can consider minibatch EOT coupling samples (Pooladian al., 2023) which are generated by sampling (a1 K from g(2o), sampling {=i y=1 from ((x1) running the Sinkhorn algorithm (Cuturi, 2013) (see Alg. 3) to create doubly-stochastic EOT matrix PEOT between the two batches, and sampling (Do, œ1) pairs from PEOT As observed by Pooladian et al. (2023), minibatch EOT coupling can accelerate flow matching optimization by reducing gradient variance, and we expect similar benefits for GCTM training as well.
Supervised coupling:
(Go, σ1) = q( (6) (x1)dH dH
(22)
where H q( H (o) is random operator, possibly dependent on æp, which maps ground- truth data Dg to observations I,, i.e., x = Hz For instance, in the context of learning inpainting model, H is could be a random masking operator.
Gaussian perturbation. The cardinality of the support of g(x1) must be larger than equal to the cardinality the support of q(xo) for there to be a well-defined ODE from q(x1) to q(zo). This is because the ODE trajectory given an initial condition is unique, so single sample T1 q(x1) cannot be transported to multiple points in the support of g(z)). A simple way to address this problem is to add small Gaussian noise to ((x)) samples such that ((x)) is supported everywhere.
5

---

Published as a conference paper at ICLR 2025
Algorithm ((20, m1) Sampling
Algorithm GCTM Training
1: Assume m 1,... MM, Batch size M
while training do
2: Coupling is Independent then
2:
Sample times {t 1(th 8 u 2n)
(arg Fm ~ ((zg). {a{* Jm ((m1)
3:
With Alg. 1, { {(ag) at m (1 2i)
Return {(=]]. #f(") )m
4:
pm + (1 (m))a + t"
else Coupling is Minibatch soT then
t Jay" + aa
Return SK{ (2g (m
( m. >- g(3og, {4T) (((1)
Jmo {x]" [m"")) Tm.
6:
(F) M Em 10 go go e em)
Th Gzgton (1 -mmm a u 8
else Coupling Supervi
then
(20 hm ~ gz H ~ (HH (x))))
8: LOCTM (0) or 2 (Gaaaag s
10:
Return {(x]]) H x )m
9: Minimize (acTM( XPM
1I:
end if
10: end while
We emphasize that Gaussian perturbation allows GCTMs to achieve one-to-many generation when we use the supervised coupling. Concretely, consider the scenario where there are multiple labels To ~ g(<oolz) (1)) which correspond to an observed 21. Then, the perturbation e added to æ1 acts as source of randomness, allowing the GCTM network to map Φ + to distinct labels Fg for distinct This stands in contrast simply regressing the neural network output of Φ to corresponding labels wo q( (x: 1) with &g loss, as this will cause the network to map æ to the blurry posterior mean gng m1) [aoo] instead of a sharp image Eg. Indeed, in Section 5.2, observe blurry outputs we use regression instead of GCTMs.
Time discretization. In practice, to optimize LGCTM (0), sample time variables s. t, u from discretization lo < l < ... < iN of the unit interval [0, 1). and simulate zz-s4 with respect to the discretization well. Given the success of the EDM time discretization Karras et al. (2022) for fast sampling of diffusion models, we propose using the EDM time discretization converted to FM time discretization via change of variables in Proposition 2,
to = 0, 4 . where T = 1/ + 5 (amax (unn
(23)
We fix م and min = 0.002 proposed in Karras et al. (2022), and control mmax- We note that o mmax controls the amount of emphasis on time near 1, i.e., larger max places more time discretization points near 1.
EXPERIMENTS
We now explore the possibilities of GCTMs on unconditional generation, image-to-image transla- tion. image restoration, image editing, and latent manipulation. In particular, GCTM admits NFE- sampling via H, Gø(a,, t, 0). Due the similarities between CTMs and GCTMs as detailed in Thm. 1, GCTMs can be trained using CTM training methods. In fact, we run Alg. 2 with the method in Section 5.2 of (Kim al., 2024b) to train all GCTMs without pre-trained teacher models. A complete description of training settings are deferred to Appendix A.
5.1 FAST UNCONDITIONAL GENERATION
In the scenario where we do not have access to data pairs, must resort either the independent coupling the OT coupling. Here, we show that the optimal transport coupling can significantly accelerate the convergence speed of GCTMs during training, especially when we use smaller number of timesteps N in time discretization during GCTM training (see Section 4. 4). Using small N may be of interest when we wish trade-off training speed for performance, since per-iteration training cost of GCTMs increases linearly with N. For in- stance, when
60
N - 4, Independent
50
RINSs)) NFF
40
30
Speodup
20
20
40
60
80
100
Iteration (x1k)
Figure 2: Training acceleration.
and u s = 0 in the GCTM loss (15), we need to integrate along the entire time interval (0, 1). which requires N steps of ODE integration.
Method
Teacher
FID
CTM
V
5.28
x
In Figure 2, observe up to x2.5 acceleration in terms of training iterations when we use OT coupling instead inde- pendent coupling. Indeed, in Figure 3, OT coupling samples are visually sharper than independent coupling samples. We postulate this is because (1) OT coupling leads to straighter
CM
V
3.55
X
iCM
X
2.51
GCTM (OT)
Table 1: FID NFE 1.

---

Published as a conference paper at ICLR 2025
(a) Indep. N = 4, FID 24.7
(b) OT N = 4, FID = 18.2
(c) OT N = 32, FID = 5.32
Figure 3: CIFARIO unconditional samples with NFE =
Time (ms)
Edges Shoess
Night Day
Facades
Method
NFE
4
LPIPS 4
IPIPS
FID 4
I.PISS
Regression
87
54.3
3.41
0.100
189.2
1.85
0.373
121.8
3.28
0.274
Pix2Pix (Isola et 2017)
33
77.0
3.17
0.208
158.0
1.68
0.418
134.1
2.74
0.288
Palette (Saharia al., 2022)
334.1
0.707
259.3
2.47
0.394
L's et al., 2023)
284
3.23
0.154
145.8
1.79
0.376
135.2
2.51
0.269
GCTM
87
40.3
3.54
0.097
148.8
2.00
0.317
111.3
2.99
0.230
Table 12I translation results (64 x 64 resolution). Best is in bold, second best is underlined.
ODE trajectories,
accurately integrate ODEs with smaller N, and (2) lower variance from OT pairs leads to smaller variance in loss gradients, as discussed in (Pooladian et al., 2023).
In Table compare the Fréchet Inception Distance (FID) (Heusel al., 2017) of GCTM and relevant baselines on CIFARIO with NFE = 1. the setting where we do not use pre-trained teacher diffusion model, GCTM with OT coupling outperforms all methods with the exception of iCM (Song & Dhariwal, 2024), which is an improved variant of CM. Moreover, GCTM is on par with CTM trained with teacher. Here, the numbers for CTM our reproduced results without GAN loss for fair comparison. We speculate that further tuning of byper-parameters or addition of a GAN loss could push the performance of GCTMs to match that of iCMs.
5.2 FAST IMAGE-TO- IMAGE TRANSLATION
Unlike previous distillation methods such as CM or CTM, GCTM can learn ODEs between arbitrary distributions, enabling image-to-image translation. To numerically validate this theoretical improve- ment, we train GCTMs on three translation tasks Edges->hhees, Night-Daay, and Facades (Isola al., 2017), scaled to 64 64, with the supervised coupling. We consider three bascline methods: fz-regression, Pix2Pix (Isola et al., 2017), Palette (Saharia et al., 2022) and PSB(Liu et al., 2023). To evaluate translation performance, we use FID and Inception Score (IS) (Barratt Sharma, 2018) to rate translation quality and LPIPS (Zhang al., 2018) to assess faithfulness to input. We control NFES such that all methods have similar inference times, and calculate all metrics on validation set.
In Table 2, we see GCTM shows strong performance on all tasks. SDE-based methods ISBB and Palette show poor performance at low NFEs, even when trained with pairs. Qualitative results in Figure 4 are in line with the metrics. Basclines produce blurry nonsensical samples, while GCTM produces sharp and realistic images that are faithful to the input. Table 5, we show results on higher resolution (256 x 256) data, and observe similar trends.
5.3 FAST IMAGE RESTORATION
We consider two settings on the FFHQ 64 x 64 dataset, where we either know or do not know the corruption operator. In the former case, we train unconditional GCTM with the independent coupling, with which we implement three zero-shot image restoration algorithms: DPS, CM-based image restoration, and the guided generation algorithm illustrated in Figure where the loss is given as inconsistency between observations (see Append. B.2 for pseudo-codes and a detailed discussion of the differences). In the latter case, train GCTM with the supervised coupling

---

Published as a conference paper at ICLR 2025
Pix2Pix
Palette
FBSB
Regression
GCTM
37
31
Figure 4: Qualitative evaluation of 12I translation (64 x 64 resolution) on Edges Bigges>Showe Shhess (top), Night +Day (middle) and Facades (bottom). NFE = for SB and Palette.
Method
NFR
Time (ms)
SR2 Bicubic
Deblur Gaussian
Inpaint - Center
PSNR
SSIM
LPIPS
LPIPS
Shot
DPS
32
1079
31.19
0.935
0,015
27.88
0.878
0.041
24.69
0.876
0.042
CM
32
1074
30.80
0.930
0.010
27.85
0.871
0.027
23.02
0.857
0.050
GCTM
32
1382
31.61
0.939
0.015
28.19
0,885
0.037
24.47
0.876
0,042
Regression
87
33.46
0.964
0.015
31.19
0.942
0.015
28.76
0.922
0.028
Supory.
Palette
166
17.88
0.556
0,234
17.81
0.571
0.234
16.12
0.489
0.357
PSB
5
284
26.74
0,869
0.033
26.20
0.853
0.038
26.01
0.874
0.038
GCTM
32.37
0.935
0.009
Table 3: Quantitative evaluation of image restoration FFHQ (64 64 resolution).
and fg-regression, 1SBB and Palette for comparison. Notably, GCTM is the only model applicable to both situations, thanks to the flexible choice of couplings. We again control NFEs such that all methods have similar inference speed.
Table presents the numerical results in both settings. In the zero- GCTM outperforming both DPS and CM. In particular, CM is slightly worse than DPS. Sample quality degrada- tion due to error accumulation for CMs at large NFEs have already been observed in unconditional generation (e.g., see Fig. 9 in (Kim al., 2024b)), and we speculate similar problem occurs for CMs in image restoration as well. On the other hand, GCTMs avoid this problem, as they are able to traverse
X1
Xo
shot setting,
smaller time using the ODE velocity approximated via go- In the supervised setting, we see re- gression attains the best PSNR and SSIM. This is a natural conse- trade-off. Specifically, regression
quence of perception-distortion minimizes the MSE loss, so it leads best distortion metrics (Del- bracio & Milanfar, 2023) while producing blurry results. GCTM, which provides best results if we exclude regression on distortion Figure 5: Reg. vs. GCTM.
Regression
GCTM
metrics (PSNR and SSIM) and best results perception metrics (LPIPS), strikes the best balance between perception and distortion. For instance, Fig. 5 inpainting results, regression sample lacks detail (e.g.. wrinkles) while GCTM sample is sharp. We show more samples in Appendix E. In particular, in Table 6, we demonstrate image restoration task of GCTM on ImageNet with higher resolution (256 x 256 resolution) images demonstrate its scalability.
5.4 FAST IMAGE EDITING
In this section, we demonstrate that GCTM can perform realistic and faithful image editing without any special purpose training. Figure 6 shows image editing with an Edges~ model and an unconditional FFHQ model. On Edges edit image, user creates edited input,
8

---

Published as a conference paper at ICLR 2025
Image-to-Image GCTM
Edited
Output
Stroke
Filling
Shape Change Filling
Patch
Noise-to-Image GCTM
Source
Edited
Output
Lipstick
Mustache
Glasses
Hair
Figure Image editing with GCTM, NFE
Guidance Strength
Latent
Red Blas
Greem Bis
Blts
(a) Controlling latent vector strength y
(b) Mixing latent vectors
Figure 7: Latent manipulation with image-to-image GCTM, NFE 1.
which is an edge image painted to have desired color and or modified to have a desired outline. then interpolate the edited input and the original edge image to certain time point t = 8 and send it to time é with GCTM produce the output. On FFHQ, analogous to SDEdit (Meng et al.. 2022), we interpolate an edited image with Gaussian noise and send it to time t = 0 with GCTM to generate the output. In contrast to previous image editing models such as SDEdit, GCTM requires only a single step to edit an image. Moreover, we observe that GCTM faithfully preserves source image structure while making the desired changes to the image.

---

Published as a conference paper at ICLR 2025
5.5 FAST LATENT MANIPULATION
In this section, we demonstrate that GCTMs have a highly controllable latent space. Since there are plenty of works on latent manipulation with unconditional diffusion models, we focus on latent manipulation with GCTMs trained for image-to-image translation. For an image-to-image transla- tion GCTM trained with Gaussian perturbation in Section 4.1, we assert that the perturbation added to 21 can manipulated to produce desired outputs øg. In other words, the perturbation acts as "latent vector" which controls the factors of variation in To- To test this hypothesis, in Figure 7, we display outputs Gø(æ1 + 7e, 1, for particular choices of E. In the left panel, we observe generated outputs increasingly adhere to the texture of latent as increase guidance strength y. Interestingly, GCTM generalizes well to latent vectors unseen during training, such as leopard spots or the color red. In the right panel, we explore the effect of linearly combining red, green, and blue latent vectors. We see that the desired color change is reflected faithfully in the outputs. These observations validate our hypothesis that image-to-image GCTMs have interpretable latent space.
5.6 ABLATION STUDY
We now perform an ablation study on the design choices of Section 4.1. We have already illustrated the power of using appropriate couplings in previous sections, so we
100
X Perturb, Oney 80
Perturb. Omax 80
explore the importance of maax A robust choice for max
FID(Sk) @ NFE
Perturb, Orw 500
for unconditional generation is well-known to be max 80 (Karras et al., 2022; Kim et al., 2024b), and we found using this choice to perform sufficiently well for GCTMs when learning to translate noise data with independent or OT couplings. So, we restrict our attention to image- to-image translation.
40
20
0
100
Iteration (x1k)
In Figure 8, we display the learning curves on Figure 8: Ablation study of GCTM. Edges-tShoes for GCTMs trained without and with Gaussian perturbation, and max E {80, We observe that GCTM trained without pertur- bation and max = S0 exhibits unstable dynamics, and is unable to minimize the FID below 30. On other hand, GCTM trained with perturbation and Fmax 80 surpasses the model trained without perturbation. This demonstrates Gaussian perturbation is indeed crucial for one-to-many generation, noted in the last paragraph of Section 4.1. Finally, GCTM with both perturbation and 7max 500 minimizes FID the fastest. This shows high-curvature regions for image-to-image ODEs lie near 21, so we need to use large max which places more discretization points near
CONCLUSION
Our work marks significant advancement in the realm of ODE-based generative models, par- ticularly the transformative capabilities of Consistency Trajectory Models (CTMs). While the iterative nature of diffusion has proven to be powerful foundation for high-quality image synthesis and nuanced control, the computational demands associated with numerous neural function eval- uations (NFEs) per sample have posed challenges for practical implementation. Our proposal of Generalized CTMs (GCTMs) extends the reach of CTMs by enabling one-step translation between arbitrary distributions, surpassing the limitations of traditional CTMs confined to Gaussian noise to data transformations. Through an insightful exploration of the design space, we elucidate the impact of various components on downstream task performance, providing comprehensive understanding that contributes to broadly applicable and stable training scheme. Empirical validation across di- verse image manipulation tasks demonstrates the potency of GCTMs, showcasing their ability to accelerate and enhance diffusion-basee algorithms. In summary, our work not only contributes to theoretical advancements but also delivers tangible benefits, showcasing GCTMs as a key element in unlocking the full potential of diffusion models for practical, real-world applications in image synthesis, translation, restoration, and editing.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea under Grant RS-2024- 00336454 and by the Institute for Information & Communications Technology Planning & Evalu- ation (IITP) grant funded by the Korea government (MSIT) (RS-2019-1119005, Artificial Intelli- gence Graduate School Program, KAIST).
REFERENCES
Shane Barratt and Rishi Sharma. Note on the Inception Score. arXiv preprint arXiv:/801.019733 2018.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. ICLR, 2022.
Hyungjin Chung, Jeongsol Kim, and Jong Chul Ye. Direct diffusion bridge using data consistency for inverse problems. NeurlPS, 2023.
Hyungjin Chung, Suhycon Lee, and Jong Chul Ye. Decomposed Diffusion Sampler for Accelerating Large- Scale Inverse Problems. ICLR, 2024.
Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. NeurIPS, 2013.
Mauricio Delbracio and Peyman Milanfar. Inversion by direct itcration: An alternative to denoising diffusion for image restoration. TMLR, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. NeurIPS, 2021.
Ian J. Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. NeurIPS, 2014.
Guande Kaiwen Zheng, Jianfei Chen, Bao, Zhu. Consistency Diffusion Bridge Models. NeurIPS, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. NeurlPS, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffussio guidance. arXiv preprint arXiv2200..25998 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 2020.
Phillip Isola, Jun- Yan Zhu Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. CVPR, 2017.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. generative models. NeurlPS, 2022.
Elucidating the design space of diffusion-based Bahjat Kawar, Michael Elad, Stefano Ermon, Jiaming Song. NeurlPS, 2022.
Denoising diffusion restoration models.
Beomsu Kim, Gihyun Kwon, Kwanyoung Kim, and Jong Chul Ye. Unpaired image- image-to-imant to-image translation via neural Schrödinger bridge. ICLR, 2024a.
Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He. Yuki Mitsufuji, and Stefano Ermon. Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion. ICLR, 2024b.
Diederik P. Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization. ICLR, 2015.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. ICLR, 2014.
Bo Li, Kaitao Xue, Bin Liu, and Yu-Kun Lai. BBDM: Image-to tnaggeto-image Translation with Brownjan Bridge Diffusion Models. CVPR, 2023.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamm,, Maximilian Nickel, and Mathew Le. Flow matching for generative modeling. ICLR, 2023.
Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A Theodorou, Weili Nie, and Anima Anandkumar. SBB Image-to-Image Image Schrödinger Bridge. ICMI, 2023.

---

Published as a conference paper at ICLR 2025
Kangfu Mei, Mauricio Delbracio, Hossein Talebi, Zhengzhong Tu, Vishal M. Patel, and Peyman Milanfar. and Faster Image Generation. In CVPR, 2024,
CoDi: Conditional Diffusion Distillation for
Chenlin Meng, Yutong He, Yang Song. Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. ICLR, 2022.
Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Semmingo-Enrich, Brandon Amos, Yaron Lipman, and Ricky Q. Chen. Multisample matching: Straightening flows with minibatch couplings. ICML, 2023,
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho. Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In SIGGRAPH, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. ICLR, 2022.
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guidel diffusion models for inverse problems. ICLR, 2022.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. ICLR, 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. ICLR, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency Models. ICML, 2023.
Nikita Starodubeev, Mikhail Khoroshikh, Artem Babenko, and Dmitry Baranchuk. Invertible Consistency Distillation for Sext-Guided Image Editing in Around 7 Steps. arXiv preprint arXiv:2406. 14539, 2024.
Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual Diffusion Implicit Bridges for Image-to- Image Translation. In ICLR, 2023.
Alexander Tong. Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. TMLR, 2024.
Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7):1661-74, 2011.
Jie Xiao, Kai Zhu, Han Zhang, Zhiheng Liu, Yujun Shen, Zhantao Yang, Ruili Feng, Yu Liu. Xueyang Fu, and Zheng-Jun Zha. CCM: Seal-Time Controllable Visual Content Creation Using Text-to-Image Consistency Models. ICML,, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. CVPR, 2018.
Jiankun Zhao, Bowen Song, and Liyue Shen. CoSIGN: Few-Step Guidance of ConSIstency Model to Solve General INverse Problems. ECCV, 2024.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. EGSDE: Unpaired image-to to-image translation via energy- guided stochastic differential equations. NeurlPS, 2022.
12

---

Published as a conference paper at ICLR 2025
A
FULL EXPERIMENT SETTINGS
A.1 TRAINING
In this section, introduce training choices which provided reliable performance across all exper- iments in our paper.
Bootstrapping scores. all our experiments, we train GCTMs without pre-trained score model. So, analogous to CTMs, we use velocity estimates given by an exponential moving average OEMA of 0 solve ODEs. We use exponential moving average decay rate 0.999.
Time discretization. In practice, we discretize the unit interval into a finite number of timesteps J where
to 0<< <...<ty=
(24)
and learn ODE trajectories integrated with respect to the discretization schedule. EDM (Karras al., 2022), which has shown robust performance on variety of generation tasks, solves the PFODE on the time interval (amin mmax) for m < max according the discretization schedule
0n
(o 1/p (n/N)(o )
(25)
for 0, .,...,
..., N and p 7. Thus, using the change of time variable (17) derived in Theorem 1, we convert PFODE EDM schedule to FM ODE discretization
to =0=0, 0, fa = / (1++0) ox) for n = N ty=
(26)
In our experiments, we fix min = 0.002 and control max- We note that max controls the amount of emphasis on time near
i..., larger max places more time discretization points near
Number of discretization steps N. CTMs use fixed N = 18. In contrast, analogous to iCMs, we double N every 100k: iterations, starting from N
Time i distribution. For unconditional generation, we sample
= / +)), log ~ ( 1.2?)
(27)
in accordance with EDM. For image-to-image translation, we sample
~~ beta 1).
(28)
Network conditioning. We use the EDM conditioning, following CTMs.
Distance d. CTMs use defined as
d(x, ..) LPIPS(GeeMAA (Iz,, t, 0), GoEMa (ā,,t,0)
(29)
which compares the perceptual distance of samples projected to time O. In contrast, following iCMs, we use the pseudo-huber loss
d(æ,, (z) V ||, 2el12 + c²
(30)
where c = 0.00054v d, where d is the dimension of æ;.
Batch size. We use batch size 128 for 32 32 resolution images and batch size 64 for 64 x 64
resolution images.
Optimizer. We use the Adam optimizer (Kingma & Ba, 2015) with learning rate
7 0.0002/(128/batcch.ss
(31)
and default (31,32) (0.9,
Coefficient for (0). We use ÄPM = 0.1 for all experiments.
Network. We modify SongUNet provided https: / / / thub. wwwpwwwwwc.//ccccoo... accept two time conditions and s by using two time embedding layers.
ODE Solver. We the second order Heun solver calculate (CCTM(B).
Gaussian perturbation. We apply a Gaussian perturbation from normal distribution multiplied by 0.05 to sample 1,, excluding inpainting task.
13

---

Published as a conference paper at ICLR 2025
A.2 EVALUATION
In this section, we describe the details of the evaluation to ensure reproducibility of our experiments.
Datasets. In unconditional generation task, we compare our GCTM generation performance us- ing CIFARIO 10 training dataset. In image-to-image translation task, we evaluate the performance of models using test sets of Edges->Shoes, Night->Day, Facades from Pix2Pix. image restoration task, we use FFHQ and apply following corruption operators H from SB to obtain measurement: bicubic super-resolution with factor of 2, Gaussian deblurring with o 0.8, and center inpainting with Gaussian. We then assess model performance using test dataset.
For 121 translation tasks (64 x 64 resolution), we com- pare three baselines: Janspiry/Palette- -nnttttee--mma---t--------- Lt Diffusion-Moder https://githine wwt.//oabs//aa//
Baselines.
Pix2Pix from https://gittu// com junyanzu Palette model from pettps:/ggithie
com/
and I from In the case of Pix2Pix, during training, we optimize the generator and discriminator with 256 256 resolution images using the recommended hyper-parameters. During inference, we apply bilinear interpolation to generator output images to resize them to 64 64. For other baselines, we modify the model image resolution to 64 x 64 and use the recommended hyper-parameters. Same configuration is used in supervised image restoration task.
Metries details. We calculate FID htt.h//iigg/////gaaaaao////////a and IS from https : /ttthbb.ttuuutgg htt.a//...............aa//aaaata models/incept:an
py. We assess LPIPS from generation task, we employ the en- PerceptuulSSmiiarrs
with AlexNet version 0.1.
tire training dataset to obtain FID scores, and in the other task, we sample 5,000 test datasets. To obtain PSNR and SSIM, we convert the data type of model output to uint& and normalize We use https::p//.cmc //ggttttb..../////gg//..g...hh com/ http:/sikiitiiii wwwiimmaaaiiii.. -iaage for PSNR and SSIM.
Sampling time. To compare inference speed, we measure the average time between the model taking in one batch size as input and outputting it.
B ALGORITHMS
B.1 OPTIMAL TRANSPORT
Algorithm 3 Sinkhorn Sinkhorm-Knopp Knopp (SK)
1: Input: {x0] M mm=1 {xr M /m=1.
2: Compute cost matrix C E R M MM subject to Ci = ||قا x111
3: Solve PROT = arg minp (P, C) - (H(P) such that P1 = P
1 with Algorithm 1 in
Cuturi (2013)
4: Treat PEOT as a discrete distribution over (1,...,M x (1....,M)
5: Sample {(iT "('", " PEOT
6: Return: {(x)"" [ }M=
B.2 IMAGE RESTORATION
In Alg. describe three zero-shot image restoration algorithms, DPS, CM, and GCTM. DPS uses the posterior mean Ey(asla,,) [xto] ] to both traverse to smaller time 4-1 and to approximate measurement inconsistency. As the posterior mean generally do not lie in the data domain, using it to calculate measurement inconsistency can be problematic. Indeed, approximation error in DPS closely related to the discrepancy between the posterior mean and æ',-00 -u (see Theorem 1 in (Chung et al., 2022) for a formal statement). On the other hand, CM uses the ODE terminal point x/ - to traverse smaller time f and to approximate measurement inconsistency. While CM can have better guidance gradients as 2/_- lie within the data domain, using x/ to traverse to f can accumulate truncation error and degrade sample quality. For instance, Figure 9 (a) in (Kim al.,
14

---

Published as a conference paper at ICLR 2025
Algorithm 4 Zero-shot Image Restoration
Input: Measurement æ1, corruption H, discretization [4)]2
2: Fta. NO0 r)
3: for i M to 1 do
4: E~ 1)
5: if' Method DPS then
6:
Ly =00 = t4
7:
x/, (1 (1-14-1 1--14-1 + 1
8:
else if Method is CM then
9:
By Go (1 12 0)
10:
Vú = (1 4-11 2g + 4--16
11:
else if Method is CCTM then
12:
Evaluate score and ODE endpoint in parallel by (t4,t;), (f:, 0):
13:
Lo, do go (1// li, 4:), (a 4; 0)
14:
li-1 30 + 4-11
15:
end
if
16:
+ 2 A Vad o,, ||1 H
17:
end for
Return: 2/
2024b). GCTM mitigates both problems by enabling parallel evaluation posterior mean and ODE endpoint, shown in Line 12-13 of Alg.
B.3 IMAGE EDITING
Algorithm 5 Image Editing
1: Input: (Eo, 211 ~ g(Zo, æ1), t
2: d; = (1 E)Edit(aro) + tæ
3: Return: Go(@r, 1, 0)
C PROOFS
C.I CORRECTNESS OF SECTION 3.3
We show that our exposition in Section 3.3 adheres Conditional Flow Matching (CFM) theory. Specifically, the notations
g(z), Pr(z12), (x(x)2)
(32)
in Section 3 of Tong et al. (2024) are expressed in our paper as
(za, 21), q( x1), q(>,) mo. æ1), 21 Co.
(33)
respectively. follows that (x) and us (x) in Tong et al. (2024) are expressed in our notation as
(x(x)= = pelal pe(x)=)g(=)dzz / u( o (2 xx) dayy da = g(xx))
(34)
and
(x(2))))(x
xa)g(2x 211
ux(x) == Egtz)
Ex(
Em - Øc]
(x)
g(x,))
(35)
respectively. By Theorem 3.1 in Tong et al. (2024), the ODE Eq. (11) indeed generates q(x;). and Eq. (12) is equivalent to the CFM objective Eq. (10) in Tong al. (2024).
15

---

Published as a conference paper at ICLR 2025
C.2 PROOF OF PROPOSITION 1
Proof. We observe that the velocity term in (11) may be expressed as
E [0))) Øol = Eg (ar [(=] - [t)/t]
(36)
Eglan [(x; 0o)/t]
(37)
(ax (e [2to))/t
(38)
since #1 is determined given 2y and zt. This shows the equivalence between (11) and (13). Eq.
(14) is then straightforward consequence of the equivalence between ODEs.
C.3 PROOF OF PROPOSITION 2
Proof. We first show equivalence of scores. We note that
x, + Zt.
(39)
a bijective transformation, so by change of variables, q(x, (2o) (1 + 7) Na æo, rf) (1++) p( (20)
(40)
and marginalizing Øo. get
q( (1+ p(x,))
(41)
It follows by Bayes' rule that
(o))))) pr pn (((2))
(42)
p(xx))
(1 + -)- g( (20))
(43)
(1 1
(a22/220)) (200))))))((
(44)
(
= =x0 (ac:)
(45)
and thus
E Epteo (e, [mo]] Eny (as.) [20]]. for all J E (0, 20) and æ,, We now show the equivalence of ODEs. Diffusson PFODE is
(46)
dx, Kr E Ep( [ol] dr.
(47)
With the change of variable
21 = x1/(1 + T)
(48)
we have
1
dx:
dr +
1 dx,
(49)
(1 + 7)2
2r Ep( [200] dt
dr +
(50)
(1 + 7)2
T(1 + T)
(1 + +)xx
E Epiro [oo] dr
Tt dr +
(51)
+
t(1 t)
Et Eppeea (z)) [Ta]] dr
(52)
7(1 + 7)
32 By(ees/ae)) [20]] dr
(53)
7(1 + 7)
16

---

Published as a conference paper at ICLR 2025
where we have used equivalence of scores at the last line. We then make the change of time variable
1 ==T/(1+7) di (1 (1++)² r)2 dr
(54)
which gives us
21 E (20 (iamolee (m
[2o]
dæ;
dt
(55)
7 7/(1 T)
Dt Ey00 o0.))
[20]
dt.
(56)
t
Foe first equality in (20), transform PFODE variables (æ,, T) into FM ODE variables (æ,, with (17), transport æg to v, with GGeTM, and then transform FM ODE variables (æ,, 8) into PFODE variables (zer o) with the inverse of (17). Second equality in (20) follows directly from (18).
D LIMITATION, SOCIAL IMPACTS, AND REPRODUCIBILITY
Limitations. GCTMs are yet unable to reach state-of-the-as art unconditional generative performance. We speculate further tuning of hyper-parameters in the manner of iCMs could improve the perfor- mance, and leave this for future work.
Social impacts. GCTM generalizes CTM achieve fast translation between any two distributions. Hence, GCTM may be used for beneficial purposes, such fast medical image restoration. How- ever, GCTM may also be used for malicious purposes, such as generation malicious images, and this must be regulated.
We open-source our code at https: com/ hemm/120kbbb /CTM in- cluding training code for unconditional generation, image-to-image translation, and supervised im- age restoration models.
Reproducibility.
E ADDITIONAL EXPERIMENTS
E. COMPARING 121 PERFORMANCE OTHER BASELINE MODELS
We compare the imagee (12I) performance of model with two baseline approaches: EGSDE (Zhao et al., 2022) and BBDM (Li et al., 2023). Since BBDM, an 12I framework based the Brownian Bridge process, leverages latent diffusion model, train it with a pixel-space diffusion model for a fair comparison. Both BBDM and EGSDE are trained on the Edges dataset. As shown in Table 4, our GCTM outperforms all baselines across various metries, even when evaluated with fewer sampling steps.
In addition, we visualize the image editing results in Fig. 9. While EGSDE generates realistic images, it fails to faithfully preserve the given conditions. BBDM, on the other hand, struggles perform robustly on (unseen) conditional images. In contrast, GCTM produces realistic images while accurately maintaining the original conditions.
Method
Time (ms)
IS
LPIPS |
BBDM (Li et al., 2023)
43.7
3.43
0.099
EGSDE (Zhao al., 2022) GCTM
2590
198.1
2.87
0.476
87
40.3
3.54
0.097
Table 4: Evaluation of 12I translation Edges Shoes with other baslines.
17

---

Published as a conference paper at ICLR 2025
Edited
GCTM
EGSDE
BBDM
Edited
GCTM
EGSDE
BBDM
Figure 9: Comparison image editing with GCTM and other baselines
E.2 CONTROLLABLE IMAGE EDITING
In this section, demonstrate that effectiveness of image editing can be controlled. In Algorithm 5, we control the time point t. to determine how much of the edited image to reflect. In Fig. 10, the results visualize how t effect the output of model output. We observe that the larger t, the more realistic the image, and the smaller t, the more faithful the edit feature. We set t = 0.95 and = 0.4 supervised coupling and independent coupling, respectively.
0
0.95
0.97
0.99
0.999
Stroke
t-0
t- 0.1
t-2
0.3
t- 0.4
0-5
0.6
0-7
t-0.8
t-
0.9
t-1
+ Hair
Glasses
Figure 10: Controllability of image editing by t.
18

---

Published as a conference paper at ICLR 2025
E.3 RIGH-RESOLUTION IMAGE-TO-IMAGE TRANSLATION
To verify the robustness of our framework to scalability of resolution, we experiment with image-to- image translation of the Facades dataset with 256 256 resolution (Fig. 11). In Table 5, that GCTM achieves high performance despite using fewer NFEs compared to the baselines, generating realistic, diverse, and faithful translated images.
Facades-250
Method
NFF
FID 4
IS
LPIPS
Pix2Pix (Isola al., 2017)
1
117.2
1.60
0,414
Palette (Saharia al., 2022)
5
396.7
1.14
1.089
ILsu et al., 2023)
5
128.6
2.23
0.454
GCTM
107.0
2.24
0.426
Table 5: Quantitative evaluation of I2I translation with 256 x 256 resolution images. Best bold, and second best is underlined.
11
Pix2Pix
Palette
PSB
GCTM
X
Figure 11: Qualitative comparison of 12I task Facades 256 256.
19

---

Published as a conference paper at ICLR 2025
E.4 HIGH- RIGHLRESOLUTION IMAGE RESTORATION
In Table 6, we demonstrate image restoration task of GCTM on ImageNet with higher resolution images. As diffusion-based solvers for inverse problems, both DPS (Chung et al., 2022) and DDS (Chung et al., 2024) require sufficient number of NFEs to achieve effective reconstruction. Specif- ically for DPS. using significantly fewer NFEs than the 1000 NFEs suggested in the original paper, combined with the absence of measurement noise. disrupts the reconstruction process, resulting in outputs that are worse than GCTM. Although DDS shows improved performance compared to DPS, it requires more NFEs to achieve comparable performance with GCTM, which emphasizes the efficiency of GCTM solving image restoration tasks.
SR4 Bicubic
Dehlur Gaussian
Inpaint Center
Method
NFE
PSNR
LPIPS
PSNR
SSIM
LPIPS
PSNR
LPIPS |
DPS
10
10.37
0.357
0.727
10.27
0.256
0.830
9.98
0.247
0.841
16.15
0.392
0.654
19.19
0.520
0.523
13.61
0.526
0.522
22.36
0.327
26.29
0.739
0.246
18.53
0.681
0.288
DDS
19.79
0,491
21.12
0.634
0,531
21.25
0.571
0.409
23.33
0.704
0.245
13.57
0.485
0.511
GCTM
26.70
0.771
0.223
34.65
0.948
0.032
21.56
0.808
0.229
Table GCTM evaluation of image restoration on ImageNet with 256 x 256 resolution. Best is in bold, and second best is underlined.
DPS 50
DPS 1000
DDS 50
GCTM
X
Gaussian Super-resolation
Inpainting
Figure 12: Qualitative comparison of image restoration task on ImageNet 256x256.
20

---

Published as a conference paper at ICLR 2025
E.5 ADDITIONAL IMAGE-TO- TA-AGE TRANSLATION SAMPLES
Edges SShes
Facades
Night -ay
X1
Pix2Pix
Palette
FSB
Regression
GCTM
Xo
Figure 13: Additional results on image-to-image translation task Edges (top), Facades (middle) and Day (bottom).
21

---

Published as a conference paper at ICLR 2025
E.6
ADDITIONAL IMAGE RESTORATION SAMPLES
SR2
Deblur
Inpainting
T
GCTM
27g
Figure 14: Additional results of supervised image restoration task on FFHQ 64x
22

---

Published as a conference paper at ICLR 2025
SRC
Deblur
Impainting
X1
GCTM
X0
Figure 15: Additional results of zero-shot image restoration task on FFHQ 64x64.
X
X1
X
X1
X
Deblur
Inpainting
2R2
Figure 16: Qualitative comparison zero-shot algorithms on FFHQ 64x64.
23


Reviews:
[{"review_id": "KC59Tcx01u", "reviewer": "Reviewer_oui9", "summary": "Consistency trajectory models (CTM) are a recent technique for accelerated sampling of diffusion models, which involves training a model to predict any intermediate point of the Probability Flow ODE (PFODE) trajectory allowing traversal between any time points along the PFODE. This work extends CTMs to develop generalized CTMs (GCTMs) using conditional flow matching to enable one-step translation between two arbitrary distributions via ODEs instead of only Gaussian noise-to-data transformation in CTMs. They utilize flow matching to allow more flexible couplings between starting and target distributions, including independent coupling used by diffusion models as a special case. This broadens the applicability of GCTMs to handle arbitrary image-to-image translation tasks in addition to unconditional generation, and reduces the computational costs associated with these tasks by reducing the need for multiple neural function evaluations.", "questions": "Can you provide the details of computational costs, and training time required for GCTM, and compare this with  CTM, and the  teacher model?\n\n\nIn CTMs [Kim et al., 2024b], adversarial training can be incorporated to further enhance the quality of samples. Can adversarial training also be incorporated in the proposed GCTMs?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This is overall a nice submission with significant novel technical contributions.\n\nThe proposed  generalized CTM is formulated in a principled manner and is explained well.\n\nThe paper nicely extends consistency trajectory models to allow translation between arbitrary distributions via flow matching.\n\nThe design space is systematically examined, studying the effects of different couplings,  Gaussian perturbation and $\\sigma_{max}$ for  stable training.\n\nExperiments are performed on a variety of image restoration and editing tasks in both zero-shot and  supervised manner. In image restoration, the proposed method outperforms consistency models in zero-shot setting, and provides results competitive with supervised regression with a higher perceptual quality.", "weaknesses": "I do not find any major concerns in the paper.\n\nIn Fig.6, the noise to image GCTM  editing results with NFE = 1 look a little blurred, with background details washed out.\n\nThe authors could consider citing and discussing the following parallel works which also aim to perform image-to-image tasks with few NFEs:\nMei etal. CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation. In CVPR 2024\nZhao etal. CoSIGN: Few-Step Guidance of ConSIstency Model to Solve General INverse Problems. In ECCV 2024\nHe etal. Consistency Diffusion Bridge Models. In NeurIPS 2024.\nXiao etal. CCM: Real-Time Controllable Visual Content Creation Using Text-to-Image Consistency Models. In ICML 2024\nStarodubcev etal. Invertible Consistency Distillation for Text-Guided Image Editing in Around 7 Steps. Arxiv June 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JRygkPQ77p", "reviewer": "Reviewer_AdFa", "summary": "This paper proposes Generalized Consistency Trajectory Models (GCTMs), which extend Consistency Trajectory Models (CTMs) to support sampling from arbitrary distributions rather than being restricted to Gaussian distributions. By reparameterizing the flow matching (FM) ODE in a way analogous to CTMs, the authors provide a general framework that includes CTMs as a special case when the sampled distributions is Gaussian. The paper also explores the design space of the proposed method. The proposed method achieves state-of-the-art performance in multiple image generation and manipulation tasks without relying on pre-trained teacher models.", "questions": "Please refer to the weakness section above.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is theoretically sound and inspring in that: \n(1) The reparameterization of the FM ODE to resemble the CTM form allows the proposed GCTM to perform consistent trajectory sampling between arbitrary distributions, which is highly flexible and useful.\n(2) GCTM is rigorously shown to generalize CTMs. This makes it possible to train the GCTM using the same training stratagies for training CTMs. \n\n* The paper includes exploration of the design space for the proposed model, such as OT coupling for accelerated sampling (~2.5x as the baseline), Gaussian perturbation for more diverse generation. \n\n* GCTM achieves state-of-the-art results across various applications, such as image restoration and translation, without needing a pre-trained teacher model.", "weaknesses": "*Gaussian Perturbation in Section 4.1:\n\nThe introduction of Gaussian perturbation seems contradictory to the main goal of GCTM - sampling from arbitrary distributions. Specifically, in the case of multiple labels corresponding to a single observation (L302-303), Gaussian perturbation does not apply necessarily, as these label variations are not expected to follow an iid Gaussian distribution.\n\n*Lack of teacher model distillation training scheme:\n\nThe reason for the absence of distillation from a teacher network is not fully addressed, although the method is claimed to be a generalization of CTM method. Given that distilling from a teacher could potentially improve training efficiency, an explanation for this choice would strengthen the paper.\n\n*Performance gap with iCM:\n\nGCTM does not outperform improved Consistency Models (iCM), but the reason for this gap is not fully explored. A deeper analysis of this performance difference would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g9cdXNbqcb", "reviewer": "Reviewer_PKWr", "summary": "This work presents \"GENERALIZED CONSISTENCY TRAJECTORY MODELS\" a generative model framework that allows consistency trajectory models for arbitrary coupling of image distribution while the original approach of CTM is restricted to the classical (image, pure noise) independent coupling.", "questions": "What is the value for the batch size M used for OT-based couplings ?\n\nTable 1: The CTM paper [Kim, Lai et al 2024] reports FID of 1.98 for Diffusion Models – Distillation Sampling with NFE = 1. What is the 5.28 FID reported here?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is a mixed approach of CTM and Conditional Flow Matching that is rigorous.\nThe theoretical presentation is new as far as I know.\n\nWhile this is surely of interest, in its current state, the quality of the presentation and the quality of the experiments are not high enough for a pulication at ICLR.", "weaknesses": "The main issue of this paper is that CTM is not used on high-resolution images.\nThe community does not need methods for 64x64 image processing, this resolution is not high enough to assess image to image translation or image restoration.\nIn comparison,\n\nUnpaired Image-to-Image Translation via Neural Schrödinger Bridge\nDownload PDF\nBeomsu Kim, Gihyun Kwon, Kwanyoung Kim, Jong Chul Ye\nICLR 2024\nuses 256x256 and 512x512 and discuss difficulty for other methods to be of higher resolutions.\n\nAs another recent example,\nExtremal Domain Translation with Neural Optimal Transport\nMilena Gazdieva, Alexander Korotin, Daniil Selikhanovych, Evgeny Burnaev\nNeurIPS 2023\nuses 64x64 and 128x128 examples (already to low in my opinion).\n\nAlso comparison with the original pix2pix GAN method is not fair using 64x64 resolution since it is based on a local patch-based discriminator (less resolution means less patches to analyze).\n\n\nl. 425  \"we demonstrate image restoration task of GCTM on ImageNet with higher resolution (256 × 256 resolution) to demonstrate it scalability.\"\nThere is no comparison done in the appendix.\nComparison with few NFE and many NFE methods (eg DPS, [Chung & Kim et al ICLR 2023]) is necessary to assess the performance.\n\n\nAll the background section is written without references.\nEspecially, Section 3.3 FLOW MATCHING (FM) seems far from the original paper by Lipman et al, at least in notation.\nIs it just a reinterpretation of CFM using diffusion-like notation?\n\n\nAnother main issue is the use of Optimal Transport coupling (l. 255 (Entropy-regularized) Optimal transport coupling).\nThis is a chicken and egg problem. If one knew the OT coupling presented by Equation (24) this would be a very good generative model (the best one in a sense) and we would not require a CTM.\nWhat is done in practice is Algorithm 3 Sinkhorn-Knopp (SK) (l. 760) with two batches of size M.\n(on a side remark, I don't understand why using entropy regularization here instead of computing an optimal assignment (eg using the emd function from the POT library) unless the batch size M is larger than 1000. This would ensure to use all the data.)\nAnyway, there is a big gap between equation (24) which is continuous and discrete algorithm restricted on two discrete batches. Expressing the coupling in continuous space resulting from this discrete procedure is probably intractable (and far from Equation (24)).\nThis is linked with l. 361: We postulate this is because (1) OT coupling leads to straighter ODE trajectories.\n\n\nSampling algorithm and details are missing. This sentence is hard to understand \"especially when we use a smaller number of timesteps N (l. 323)\". Is gamma sampling from [Kim, Lai et al 2024] used?\n\n\nTable 1: The CTM paper [Kim, Lai et al 2024] reports FID of 1.98 for Diffusion Models – Distillation Sampling with NFE = 1. What is the 5.28 FID reported here?\n\n\nThe Theorems are more Propositions since their proofs only involves reparameterization of ODE/expectation.\n\n\nMinor corrections / suggestions:\n* l. 170 eq (9): There is a cohabitation of [0,T] and [0,1] convention (change x1 in xT)\n* l. 245: joint distributions of q(x0) and q(x1) (speak of q(x0,x1) instead ?)\n* l. 253:  add with normal distribution\n* l. 269 is hard to inderstands. An ODE as t tends to 0 does not have a rigorous meaning.\n* Many references point to ArXiv papers while published at conferences or journals (l. 544, l. 547, l. 553, ...)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)