Implicit Bridge Consistency Distillation for One-Step Unpaired Image Translation

Authors: Suhyeon Lee, Kwanyoung Kim, Jong Chul Ye
Primary area: generative models
Keywords: image translation, consistency distillation, unpaired, one-step, diffusion models

Abstract:
Recently, diffusion models have been extensively studied as powerful generative tools for image translation. However, the existing diffusion model-based image translation approaches often suffer from several limitations: 1) slow inference due to iterative denoising, 2) the necessity for paired training data, or 3) constraints from learning only one-way translation paths. To mitigate these limitations, here we introduce a novel framework, called Implicit Bridge Consistency Distillation (IBCD),  that extends consistency distillation with a diffusion implicit bridge model that connects PF-ODE trajectories from any distribution to another one. Moreover, to address the challenges associated with distillation errors from consistency distillation, we introduce two unique improvements: Distribution Matching for Consistency Distillation (DMCD) and distillation-difficulty adaptive weighting method. Experimental results confirm that IBCD for bidirectional translation can achieve state-of-the-art performance on benchmark datasets in just one step generation.

Decision: reject
Reviewer scores: 3, 5, 8

Reviews and discussion:
<thread>
creation date:
1729944659412

thread_type:
official_review

<thread_object>
creator: reviewer_etdt

<content>
summary:
The paper proposes to apply consistency distillation (CD) on previous DDIB, achieving a one-step generative model for unpaired image-to-image translation. The authors manage to extend the CD theory, which is applicable to two arbitrary distributions. The novel distribution matching and adaptive weighting techniques further stabilize and facilitate the training process. Both qualitative and quantitative experiments confirm the efficacy of the pipeline and the outperformance.

strengths:
- The paper provides a versatile pipeline for unpaired image-to-image translation within only one step, which outperforms most previous methods even with large NFEs.

- The theory part is clear and intuitive, and the toy data showcases the instability of vanilla IBCD clearly.

- The experimental results are convincing and impressive, demonstrating directly the outperformance.

- The novel adaptive weighting is interesting and effective, encouraging further study in diffusion model distillation with insight.

weaknesses:
- CD highly bases on PF-ODE, i.e., it needs to follow the score function trajectory. In Eq. (6), two PF-ODEs starting from different domains are connected together at $\sigma_{max}$, how to guarantee the smoothness of score function (i.e., gradient) at this point (since one directly uses noisy $x_a$ to solver attached to domain B)? If not smooth, how will the error be like? The authors may provide analysis here similar to original CD paper.

- In L261, the authors claim one of the challenge is to employ only local consistency. However, CTM [1] refers to local consistency as the case when applying PF-ODE solver with extremely smaller step. On the contrary, when using two adjacent timesteps, CTM names it global consistency, similar to original CD. So in the paper, this should also be called a global consistency. I can hardly understand why such strategy is a challenge, given that most distillation works use such a loss.

[1] Learning Probability Flow ODE Trajectory of Diffusion. Kim et al., ICLR 2024.

- The authors state that vanilla IBCD faces mean prediction phenomenon, but provides no convincing analysis on it. Original CD seems not to face such challenge. Does it come from the mismatch of two PF-ODEs? The visualization in Fig. 3(a) fails to convince me. The synthesized samples are not at the mean of domain B. Besides, I cannot see the efficacy of DMCD and cycle loss.

- The ablation study is somewhat confusing. Why vanilla IBCD is only a point rather than a broken line like the others? From Tab. 3 and Fig. 6, it seems that adaptive weighting may harm the performance, which is not consistent with conclusion in toy data. Conversely, DMCD is helpful in real data but fails in toy data. The authors may need further clarification.

questions:
- The authors propose to use one generator for two domains, which may be unreasonable or hard to achieve in practice. I think the whole pipeline is compatible with two independent pre-trained DMs, i.e., one ADM on LSUN Cat and one ADM on ImageNet with some specific class.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730465603799

thread_type:
official_review

<thread_object>
creator: reviewer_cdyb

<content>
summary:
This paper proposes a framework called Implicit Bridge Consistency Distillation (IBCD) for unpaired image to image translation. IBCD connects PF-ODE trajectories from any distribution to another one by extending consistency distillation with a diffusion implicit bridge model. It introduces Distribution Matching for Consistency Distillation (DMCD) and distillation-difficulty adaptive weighting method to deal with the distillation errors and mean prediction problems from the consistency distillation. Experiments on translation benchmarks demonstrate the effectiveness of the proposed method.

strengths:
1) The paper is well-written and it clearly explains the proposed method.
2) The visualizations of component’s cumulative contributions on the toy dataset in Fig. (3) help appreciate the role of each part.
3) Experiments on both toy and highdimensional datasets demonstrate the effectiveness of IBCD.

weaknesses:
1) Missing comparison of the results for bidirectional translation.
2) Missing comparison of computation cost with the existing methods to show the efficiency of the proposed method.
3) The results in Tab. 3 show the model which added DMCD loss, cycle loss and adaptive DMCD degrades the performance in terms of PSNR and SSIM compared to the method using IBCD only.
4) The zero in the first row of Eq. (6) might be \chi_A\cap\chi_B.

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722985978

thread_type:
official_review

<thread_object>
creator: reviewer_ugkh

<content>
summary:
Diffusion models are widely used for image translation. This paper identifies limitations in existing approaches: slow inference, need for paired data, and one-way translation constraints. It introduces Implicit Bridge Consistency Distillation (IBCD) to address these issues. IBCD extends consistency distillation with a diffusion implicit bridge model. The paper proposes two improvements: Distribution Matching for Consistency Distillation (DMCD) and a distillation-difficulty adaptive weighting method. Experimental results show IBCD achieves state-of-the-art performance in one-step generation for bidirectional translation.

strengths:
Pro:

- The sampling speed in image-to-image translation is a critical problem in this area.
- The paper combines various techniques, including DDIB and consistency models.

weaknesses:
Con:

- The main concern is that the method seems too incremental, appearing to be merely a combination of DDIB and consistency models.
- In Table 3, the FID improvement from adding Cycle and DMCD is marginal. Is the author aware of what a 0.1 FID change means? If you repeat the experiment twice, the variance might be even larger. This becomes a significant issue when the FID is so high. Also, most baselines in Table 2 show the variance of FID, while the author didn't. As you can see, the variance of other methods is quite large, further undermining the ablation study in Table 3.
- With only a single step, the stochasticity is significantly reduced. The authors should include several other related metrics that highlight diversity, such as the Inception Score. Additionally, more failure cases should be provided for better understanding of the method's limitations.

questions:
as above
</content>
</thread_object>

</thread>
<thread>
creation date:
1732260718147

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank the reviewers for their insightful comments and suggestions. We are pleased that the reviewers found our paper well-written, theoretically grounded, and clear, particularly in the validation of the toy experiment. We have carefully addressed all the points raised and made corresponding revisions to the manuscript, with corrections and additions highlighted in blue.

**Key changes:**
- Added density-coverage metric to main experiement for a more comprehensive evaluation (Table 2)
- Update ablation studies (Table 3, Figure 6).
- Clarified the necessity of the auxiliary loss in vanilla IBCD (Section 3.2).
- Added quantitative comparison of real-world inference speeds (Appendix D.4, Table 5).
- Included quantitative comparison of bidirectional tasks (Appendix D.6, Table 6, Figures 12, 13).
- Added more failure cases (Figure 11).

We will respond to each of the reviewer's comments individually, providing detailed responses to the reviewer's concerns. We hope that the revised manuscript and responses address the reviewer's concerns and provide a valuable contribution to the field. If you have any further questions, please feel free to discuss them at any time.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a method for unpaired image-to-image translation. The key idea is to leverage consistency distillation on an implicit bridge model. Overall, the reviewers appreciate the task and the visualization of the paper. However, the reviewers have concerns over the contribution/novelty of the paper, and the significance of the empirical results. In particular, two reviewers expressed that the approach is indeed a combination and remained unconvinced after the discussion period.