Easing Training Process of Rectified Flow Models Via Lengthening Inter-Path Distance

Authors: Xu Shifeng, Yanzhu Liu, Adams Wai-Kin Kong
Primary area: generative models
Keywords: Rectified Flow Models, Training, Easing, Distance-Aware Noise-Sample Matching, DANSM

Abstract:
Recent research pinpoints that different diffusion methods and architectures 
trained on the same dataset produce similar results for the same input noise. 
This property suggests that they have some preferable noises for a given sample. 
By visualizing the noise-sample pairs of rectified flow models and stable diffusion models in two-dimensional spaces, 
we observe that the preferable paths, connecting preferable noises to the corresponding samples, 
are better organized with significant fewer crossings comparing with 
the random paths, connecting random noises to training samples. 
In high-dimensional space, paths rarely intersect. 
The path crossings in two-dimensional spaces indicate the shorter inter-path distance 
in the corresponding high-dimensional spaces. 
Inspired by this observation, we propose the Distance-Aware Noise-Sample Matching (DANSM) method 
to lengthen the inter-path distance for speeding up the model training. 
DANSM is derived from rectified flow models, which allow using a closed-form formula to calculate the inter-path distance. 
To further simplify the optimization, we derive the relationship between inter-path distance and path length, 
and use the latter in the optimization surrogate. 
DANSM is evaluated on both image and latent spaces by rectified flow models and diffusion models. 
The experimental results show that DANSM can significantly improve the training speed by 30\% $\sim$ 40\%
without sacrificing the generation quality.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730378913346

thread_type:
official_review

<thread_object>
creator: reviewer_ly7e

<content>
summary:
The paper reveals that different diffusion models trained on the same dataset tend to produce similar outputs when given the same input noise. This suggests the existence of "preferable noise-sample pairs" in the training process. The authors propose the Distance-Aware Noise-Sample Matching method to lengthen the inter-path distance for speeding up the training of diffusion-based models. The experiments show that the proposed method improves the training speed by about 30%~40%.

strengths:
The paper presents a framework that simplifies the optimization of inter-path distances to path length optimization, based on the observations of noise-sample pair relationships in diffusion models.

The approach leverages closed-form formulas derived from rectified flow models to enable efficient optimization without requiring architectural modifications, making it easy to integrate with existing methods.

weaknesses:
The authors heavily emphasize "consistent model reproducibility," which seems interesting but abruptly transitions to proposing a method for improving training speed. This makes the main idea of the paper quite unclear.

The presentation of the proposed method is vague, and Algorithm 1 cannot be effectively reproduced with the provided steps. I suggest adding mathematical formulations in Sections 3.3 and 4.2 to rigorously explain how the proposed model is optimized.

Regarding experiments, the paper lacks visual comparison results, which are crucial for method evaluation, especially when the FID quantitative metrics show minimal differences at the tiny scale discussed. I also recommend comparing with more methods and incorporating the proposed training speed improvement approach across more SD models. I believe the paper should present higher-resolution and larger-scale generation results. Given the computational benefits brought by this work's methodology, these tasks should be more computationally feasible.

questions:
Why is it necessary to learn latent-noise pairs that are well-aligned in the t-SNE space? This might suggest that straight flow matching is preferable, but the underlying reasoning is not clearly explained in the paper. Furthermore, I am confused about whether "consistent model reproducibility" is beneficial or detrimental for diffusion-based models. The fact that different model architectures can generate similar images from the same noise might indicate limited generation patterns due to the isotropic nature of the Gaussian noise.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730610566978

thread_type:
official_review

<thread_object>
creator: reviewer_zpez

<content>
summary:
This paper explores how different diffusion methods yield similar results with the same dataset, identifying "preferable noises" for samples. It introduces the Distance-Aware Noise-Sample Matching (DANSM) method to optimize training by increasing inter-path distances. DANSM significantly speeds up training by 30%–40% without losing quality, offering insights into enhancing diffusion model efficiency.

strengths:
1. The diagram is very clear, making it easy to understand how different diffusion methods yield similar results with the same dataset, identifying "preferable noises" for samples.

2. The proposed method significantly speeds up training by 30%–40% without losing quality, offering new insights into enhancing diffusion model efficiency.

weaknesses:
1. The FID calculations are measured with very few sampling steps, making it difficult to ensure the quality of the generated results. It would be beneficial to provide a comparison of different methods while maintaining image quality, ideally with visualizations, such as images decoded using stable diffusion.

2. The experiments were only validated on CIFAR and LSUN-BEDROOM datasets. Validation on a broader range of datasets, such as ImageNet and FFHQ, would provide more comprehensive insights. Additionally, including some actual generated images for visual comparison would be advantageous.

questions:
1. Include a comparison using longer sampling steps.

2. Conduct more extensive experiments on complex datasets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730672650530

thread_type:
official_review

<thread_object>
creator: reviewer_ovk9

<content>
summary:
This work proposes Distance-AwareNoise-Sample Matching (DANSM) to increase training speed in generative models without any loss (in fact gain) in performance. The proposed method is inspired by "the difference between random paths used in training and preferable paths from well-trained models".
The work is theoretically motivated and sound.
The work offers limited evaluations, however, they seem to be sufficient to make the case.

strengths:
The work is well written except for a few typos (for example, in line 198/199 "To **analysis**" is written instead of "To **analyse**", and in line 309/310 "have not been **evaluation** on RFM" is written instead of **evaluated**).

The idea is sound and is theoretically motivated, the results are in line with the hypothesis.
The implementation seems simple and straightforward. 

The claims made in the paper seem to be backed by empirical evaluations.

weaknesses:
I have 2 major points in this regard:
1. The plots are not well made, some of the lines in the plots, for example, Figure 6(b), Figure 7(b), and Figure (b) seem to be starting from arbitrary locations. It would help to invest more time and make better-looking plots without these artifacts, or if they are not artifacts then an explanation for the same would be very helpful.

2. The work lacks comparisons to "Immiscible Diffusion", while the paper does point out the key differences in lines 307-315, it would be interesting to see empirical evaluations in comparison to "Immiscible Diffusion" since the methods are closely related.

questions:
Q1- I would appreciate a better explanation for Figure 6 (b), the plot looks a bit unclear to me. Additionally, there are lines behind the legend curbing my ability to understand the plot completely. My essential question is that unlike Figure 7, in Figure 6 (b) I do not see any gains in training times when using the proposed method. Is this understanding of mine correct?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a novel method for optimizing diffusion models, demonstrating significant training speed improvements (30%-40%) without compromising quality, and provides theoretical support backed by empirical evaluations. The approach offers a practical solution for accelerating training in diffusion models while maintaining high performance. The use of closed-form formulas for optimizing inter-path distances is both efficient and easy to integrate into existing architectures. However, prior to the rebuttal, the reviewers pointed out that the presentation could be improved.

After the rebuttal, all reviewers recognized the contributions of the paper, with feedback generally leaning towards acceptance. The AC thoroughly reviewed the paper and rebuttal, agreeing with the consensus recommendation for acceptance based on the consistency of the feedback.