Immiscible Diffusion: Accelerating Diffusion Training with Noise Assignment
Authors: Yiheng Li
Heyang Jiang
Akio Kodaira
Masayoshi Tomizuka
Kurt Keutzer
Chenfeng Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Model
Training Efficiency
Primary area: diffusion_based_models
TL;DR: We propose Immiscible Diffusion to increase the training efficiency up to 3x with only one line of code by noise assignment.

Abstract: In this paper, we point out that suboptimal noise-data mapping leads to slow training of diffusion models. During diffusion training, current methods diffuse each image across the entire noise space, resulting in a mixture of all images at every point in the noise layer. We emphasize that this random mixture of noise-data mapping complicates the optimization of the denoising function in diffusion models. Drawing inspiration from the immiscibility phenomenon in physics, we propose *Immiscible Diffusion*, a simple and effective method to improve the random mixture of noise-data mapping. In physics, miscibility can vary according to various intermolecular forces. Thus, immiscibility means that the mixing of molecular sources is distinguishable. Inspired by this concept, we propose an assignment-then-diffusion training strategy to achieve *Immiscible Diffusion*. As one example, prior to diffusing the image data into noise, we assign diffusion target noise for the image data by minimizing the total image-noise pair distance in a mini-batch. The assignment functions analogously to external forces to expel the diffuse-able areas of images, thus mitigating the inherent difficulties in diffusion training. Our approach is remarkably simple, requiring only *one line of code* to restrict the diffuse-able area for each image while preserving the Gaussian distribution of noise. In this way, each image is preferably projected to nearby noise. To address the high complexity of the assignment algorithm, we employ a quantized assignment strategy, which significantly reduces the computational overhead to a negligible level (e.g. 22.8ms for a large batch size of 1024 on an A6000). Experiments demonstrate that our method can achieve up to 3x faster training for unconditional Consistency Models on the CIFAR dataset, as well as for DDIM and Stable Diffusion on CelebA and ImageNet dataset, and in class-conditional training and fine-tuning. In addition, we conducted a thorough analysis that sheds light on how it improves diffusion training speed while improving fidelity. The code is available at https://yhli123.github.io/immiscible-diffusion

Immiscible Diffusion: Accelerating Diffusion Training with Noise Assignment
Yiheng Li
Heyang Jiang 1.2+ Akio Kodaira' Chenfeng Xu' Tsinghua University
Masayoshi Tomizuka'
Kurt Keutzer' University of California, Berkeley
1 line of code:
2% % image-noise data point distance reduction
3x training efficiency e enhancement Nods Fllccis incondionel consilinnl geteralis tnw-aring
image-noise assignment* Fariucing tmage Nomalliwitn for Some Banedires
"Ony
One Line Code & Running Efficiently
Assignments Corresponding Images in Gaussian Noise S Space
Effectiveness Observed Both in FID and Image Comparison
noise_ immiscible Linear_sum satrr__u__ssssnnnee assignment (inage, noise)
Image Distribution
Unconditional Conditonal Fine-turing
Immiscible
6.7 ms* Execution Time *For Batch Sizes 256
Stable Diffusion
Noise Distribution
Stable
Diffusion
Figure 1: Immiscible Diffusion can use single line of code to efficiently achieve immiscibility by re-assigning batch of noise to images. This process results in only 2% reduction in distance post-assignment, leading to up to 3x increased training efficiency on top of the Consistency Model for CIFAR Dataset. Additionally, Immiscible Diffusion significantly enhances the image quality of Stable Diffusion for both unconditional and conditional generation tasks, and for both training from scratch and fine-tuning training tasks, ImageNet Dataset within the same number of training steps.
Abstract
this paper, we point out that suboptimal noise-data mapping leads to slow training of diffusion models. During diffusion training, current methods diffuse each image across the entire noise space, resulting in mixture of all images every point in the noise layer. We emphasize that this random mixture of noise- data mapping complicates the optimization of the denoising function in diffusion models. Drawing inspiration from the immiscibility phenomenon in physics, we propose Immiscible Diffusion, a simple and effective method to improve the random mixture of noise-data mapping. In physics, miscibility can vary according to various intermolecular forces. Thus, immiscibility means that the mixing of molecular sources is distinguishable. Inspired by this concept, propose training strategy to achieve Immiscible Diffusiio.. As example, prior diffusing the image data into noise, assign diffusion target
assigament-then-diiieee
"The work of this paper was done when Heyang was in internship at UC Berkeley. t Cooresponding Author: zuchenf eegeeereendbbrreeeyy .ew.eeeeeeyylle.leee.eee... edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

noise for the image data by minimizing the total image-noise pair distance in a mini- batch. The assignment functions analogously to external forces expel the diffuse- able areas of images, thus mitigating the inherent difficulties in diffusion training. Our approach is remarkably simple, requiring only one line of code to restrict the diffuse-able area for each image while preserving the Gaussian distribution of noise. In this way, each image is preferably projected to nearby noise. To address the high complexity of the assignment algorithm, we employ a quantized assignment strategy, which significantly reduces the computational overhead to negligible level (e.g. 22.8ms for large batch size of 1024 on A6000). Experiments demonstrate that our method can achieve up to 3x faster training for unconditional Consistency Models the CIFAR dataset, and Stable Diffusion on CelebA and ImageNet dataset, and in class-conditional training and fine-tuning. In addition, we conducted a thorough analysis that sheds light on how it improves diffusion training speed while improving fidelity. The code is available at https://yhie 123. github. io/iii//iii
well as for DDIM
1 Introduction
Diffusion models have made impressive progress in image generation by framing the process as a phase of denoising random Gaussian noise into the final image. Despite the advancements, training a diffusion model is resource intensive. For example, even in the primary image dataset CIFAR-10, the representative few-step diffusion model, Consistency Model [47], requires training for 10 days on 4 A6000 GPUs to reach a desired FID score of around 10. Similarly, with fewer model parameters. multiple-step diffusion model DDIM [44] still requires 24 hours on an A5000 GPU on the CIFAR-10 10 dataset. Although recent remarkable achievements in accelerating the inference of diffusion models [19, 33, 47, 28, 30, 31] have been accomplished, the inefficiency of diffusion training remains significant bottleneck, hindering the iterative development of vision generative Al.
Previous methods for improving diffusion training have focused on various strategies, such as balancing the impact of activation layers and neural weights [16]. modifying hyperparameters and design choices [46], and leveraging patchifying strategies [53] etc. Specifically, Karras et al. [16] modifies the activation magnitude, neural weight standardization, and group normalization, achieving significant acceleration in diffusion training. Besides, previous work [46] proposes a customized method for the Consistency Model to improve the performance and diffusion training. Our method is orthogonal to these previous methods. We got inspired by the Immiscible Diffusion in physics. As illustrated in Fig. 2 (a) left, miscible particles tightly jumble together after the diffusion process, making it difficult to separate them individually during the denoising phase. However, when the particles are rendered immiscible, they can still achieve a similar overall distribution while remaining clearly distinguishable (see Fig. 2 (a) right). This insight inspires our strategy for improving the disentanglement of diffused data.
We draw an analogy from the phenomenon of Immiscible Diffusion and relate the distribution of image data to the behavior of particles discussed above. In traditional diffusion processes, each image can be diffused to any point in the noise space, and conversely, each point in the noise space can be denoised to any source image, as illustrated in the left image of Fig. 2 (b). We hypothesize that the jumbled image-noise mapping creates a miscible diffusion effect and makes the optimization of the diffusion model difficult. Inspired by the Immiscible Diffusion, we are motivated to make the mixed diffusion phase distinguishable.
We propose one simple Immiscible Diffusion method. Note that we still sample Gaussian noise but perform a batch-wise assignment of noise to each image based on the distance between them during training. This approach ensures that each image is only diffused to surrounding areas while maintaining the overall Gaussian distribution of all noises. This technique was also used in flow matching based methods [36, 50] for optimizing the image-noise flow. Nevertheless, we find that the image-noise distance is reduced by only -2% after assignment, as provided in Part 4.3. This motivates us to ask which factors dominate the performance improvement? To investigate it, we propose another method that qualifies Immiscible Diffusion in Part 4.3. Experiments show that the training efficiency improvement is comparable to the function of flow optimization, demonstrating that immiscibility is the dominant factor.
2

---

Immiscible Diffusion in Physics
(b)
Immiscible Diffusion Image
Distribution
Distribution
Miscible
Immisclble
(Vanilla)
Figure 2: Physics illustration of Immiscible Diffussio. (a) depict the miscible and Immiscible Diffusion phenomenon in physics, while (b) demonstrate the image-noise pair relation in vanilla (miscible) and Immiscible Diffusion method.
However, technically, achieve immiscibility with the image data-noise assignment has an O(N9log complexity. This introduces significant overhead during training, especially for large-scale training with huge batch sizes and high-resolution images. To address this, we em- ploy a novel quantization method during assignment. We quantize the noise and image data into low-precision formats (e.g., 16-bit) during conducting the assignment algorithm. We highlight that this assignment operation only involves one line of code, and is performed only during the training phase without modifying the model architecture, the noise scheduler, the sampler, or the method of inference.
We conduct extensive experiments on three common modes: unconditional, conditional, and fine- tuning three diffusion baselines: Consistency Models, DDIM and Stable Diffusion and three datasets: CIFAR-10, CelebA and ImageNet datasets. Results show that our proposed method significantly improves the training efficiency in all experiments. Specifically, we achieve 3x training efficiency for the CIFAR- 10 dataset with immiscible unconditional Consistency Model compared to the original Consistency Model. Furthermore, we show that the FID is even lower with our method used, confirming the fidelity of our generated images. We also provide images generated from models trained with vanilla and immiscible models experiencing the same training steps, where we that those from immiscible models are much more complete and clearer, further proving the training efficiency enhancement resulted from the Immiscible Diffusion. Examples Fig. 1. Deeper analysis shows that our method, although with only one line of code and involving -2% image-noise datapoint distance changes, achieves all the benefits above in negligible running time.
shown the right of
To sum up, contributions are follows:
We clearly and specifically identify the miscibility issue in noisy diffusion steps, which leads to slow convergence of diffusion training.
To tackle the miscibility issue, we propose a simple and effective method, Immiscible Diffusion, a strategy that only requires one-line of code, to improve training efficiency for diffusion training.
Experiments demonstrate the effectiveness of our proposed method on several popular diffusion models across multiple datasets, and across unconditional, conditional, and fine-tun fine-tuning tasks. In addition, we conducted thorough analyses and ablation studies to elucidate how our method works and dominates the effectiveness.
Related Work
2.1 Diffusion Model with Efficient Inference
Diffusion models [48, 12, 39, 35] have been attracting huge attentions because of their high-fidelity image and video generation [11, 13, 13,34J, data-efficient perception [49, 32, 55], and even representation abilities robotics [3, 37, 1]. However, slow inference one of the key bottlenecks for diffusion models. To address this issue, various approaches have been proposed. For instance, techniques such as DDIM [44] have reduced the number of denoising steps from 1000 10, significantly speeding up the process. Furthermore, the introduction of Consistency Models [47] and LCM [33],

---

which utilize the properties of self-consisteney, enables denoising in as few as 1-4 steps, further enhancing the generation speed of diffusion models. Subsequently, the development of SD-turbo [42], which leverages GAN [8] loss for high-definition image generation in single step, has occurred. The Consistency Trajectory Models [18, 38] improve the generation quality of Consistency Models and accelerate research on efficient inference for diffusion models. Additionally, beyond reducing denoising steps, efforts to improve the inference efficiency of single function evaluation are being explored in various ways, including model quantization [26] and partitioning the generative components [25]. Moreover, StreamDiffussic inference at the pipeline level optimization. The improvement of the inference efficiency significantly pushes forward real applications based on diffusion models. Yet accelerating diffusion training is still
[19] streamlines denoising steps to achieve real-time
under-explored.
2.2 Diffusion Model with Efficient Training
Improving the training efficiency of diffusion models crucial. Various strategies have been proposed, including architectural modifications [16], approximating the diffusion phase with flow [28], and designing parameter choices [46] etc. Specifically, in [16], the authors discover that the magnitude of activation and the magnitude of neural weights significantly impact the training dynamics of diffusion models. They propose adjusting the activation magnitude and standardizing neural weights, as well as modifying the normalization layers to make diffusion training in a more smooth dynamic. Besides, Song el al. [46] aims to enhance the training efficiency of Consistency Models through customized design choices, significantly improving both training speed and fidelity. Furthermore, leveraging approximation strategies based on ODE assumptions [28] improves not only inference efficiency but also training efficiency since diffusion trajectories are prone to deterministic. Beyond improving diffusion training with either architecture adjustment or selection parameters, Wang er al. [53] introduce a novel patch strategy to control the ease of diffusion training, achieving both training and data efficiency. Gleichzeitig, Wang et al. [52] notices that the denoising of some noisy diffusion steps contains little information and is too easy to learn, so focusing more other steps would significantly improve training efficiency. Our method differs from previous works by clearly highlighting
under-explored problem: the miscibility problem of image data in the noise space, which plays a crucial role in training diffusion models. Our proposed Immiscible Diffusion is extremely simple yet significantly improves training efficiency.
2.3 Image-noise Optimal Transport in Generative Models
In ODE-based methods such as flow matching [27], straightening the flow with optimal transport (OT) has been used as a tool to improve the generation performance. Specifically, optimizing image-noise transport in batch [36, 50] has been found to be an effective way to improve performance. Training efficiency improvement was also observed [36, 50], and explained or posited with reduction of the variance of the training goal. However, the standard deviation reduction is only ~4% [36]. Moving forward, [24] pointed out the curvature problem in the ODE paths caused by the collapse of the reverse trajectories in the average direction. However, they replaced Gaussian noise sampling with VAE encoder-style structure to eliminate such curvatures, which destroys the strict Gaussian distribution in the noise space. Concurrent to our work, [17] applies batch-wise OT to diffusion models to achieve better FIDs, making posits in curvature reduction for the enhancement. Several methods were proposed to improve the speeds and effectiveness of OT, such as pre- pre-training with PF-ODE [54], using Schrodinger bridging [5], utilizing conditional Wasserstein distance [2] and generator-induced coupling in Consistency Models [14]. However, we are the first to emphasize that the dominant reason for training efficiency enhancement is the miscibility problem in noisy layers, which we prove in Part 4.3. Furthermore, most of previous methods are for unconditional flow matching methods only. Our work demonstrates the effectiveness in multiple diffusion models, datasets, conditional generation, and fine-tuning experiments.
3 Method
3.1 Physics Intuition
Diffusion models mimic the reverse thermodynamic diffusion phenomenon [43] to ease the denoising process. However, when the sources are miscible, as shown in the left of Fig. 2 (a), they end up

---

messily mixed. Predicting the reversal process from such random mixture encounters significant difficulties, and unfortunately, this a problem diffusion model always facing during denoising.
However, we notice that mixing can also be organized when sources are immiscible. Under that circumstance, the sources would take different continuous areas after diffusion, while the whole diffused area remains the same, as shown in the right image of Fig. 2 (a). Thereafter, the reversal process becomes smooth. Inspired by Immiscible Diffusion, we then introduce the diffusion models, with the aim of making the optimization easier and to achieve a higher training efficiency.
3.2 Immmiscible Diffusion Model
Similar to the physics phenomenon, we find that for diffusion models, any images are diffused to every corner of the noise space, which also means that each noise point can go back to any image. This would cause the denoising model to be confused on which image to go to, shown in the left of Fig. 2 (b).
Mimicing the immiscible phenomenon in physics, we hope design similar processes where each noise point is only matched to limited images, so as to avoid the confusion for the denoising model. However, the noise space must remain Gaussian help the sampling process. Therefore, we propose our first implementation way of Immiscible Diffusion, which assigns the batch of noises to the batch of images during training according the image-noise distance in their shared space. We minimize the total distance of the image-noise pairs in a batch during assignment, Here we use the L2 distance for assignment, which is ablated in Part A.1.3 in the Supplemental Materials. After assignment, the noise is still Gaussian, while each noise is assigned to nearer images like what happens in the immiscibility phenomenon, which significantly eases the difficulties for the denoising. Fig. 2 (b) right illustrates an extreme example of the Immiscible Diffusion, where the noise corresponding to each image is relatively separated.
For implementation, all we need to do is to perform a linear assignment [21] between the batch of images and noises according their distances. This can be achieved in only one line of code using Seipy [51]. The algorithm is shown below:
Algorithm Batch-wise Image-Noise Assignment
1: Input: Image batch Tb, random noise batch Arand,b, sampled diffusion steps to and diffusion schedule a
2: assign aasign_mat+e +
saarrr_ssmssssssmee Ft,b + 20,, + vI at, Rrand, [asssign_maat]
Drand,b))
4: Output: Diffused image batch 24.h
While linear assignment qualifies Immiscible Diffusion, Immiscible Diffusion can be achieved with multiple paths. In part 4.3, we ablate the linear assignment by letting it remain immiscible while disqualifying optimal transport, proving that immiscibility plays the dominant role in performance improvement.
3.3 Mathematical Illustration
In this section, we mathematically elucidate the denoising difficulty for traditional diffusion models based on DDIM [44, 12] and how our proposed Immiscible Diffusion reduces such difficulties.
In DDIM, we know for any image data-point 2o, when it is diffused to the last diffusion step T, i.e. t T. the image is sufficiently wiped out and nearly only gaussian noise is remaining, therefore,
<0r 2o) = (Nr:: 0, I) ze p(rr), where Ir (No. c) = v aaraa + VI- o4c, c~ C~, 1),
(1)
Where refers to the Utilizing Bayes' Rules and Equation 1, we can find that for a specific zy!
p( xT) q(0T ra) p(zo)
s p(xo)
p(2r)
which indicates that the distributions of the corresponding images for any noise data-point are the same as the distribution of all images.
5

---

(a)
DDIM
(b)
Immiscible DDIM
Sampled Noise
Image Output
Sampled Noise
Image Output
Predicted Noise
Predicted Noise
r=$-1
r=5-2
r=S-3
7=5-4
/=5
7=5-1
T=S-2
r=
r=S44
Denoised Image
Denoised Image
1=5-1
r=S-2
r=S-3
7=5-
T=
S=S-1
r=S-2
r=S-3
T=S4
Figure 3: Feature analysis of vanilla (miscible) and immiscible DDIM. Referring to [45], r = S represents the layer denoising from the pure noise. We show that while the sampled noises similar, the denoised image of immiscible DDIM significantly outperforms that of the traditional one, generating an overall reasonable image. The reason behind this traditional methods cannot successfully predict noises at noisy layers.
The simplified training objective DDIM [44, 12] is the added noise e t) at each diffusion step and at the point 2z. However, we find that for specific point ør in the noise space at diffusion step T.
e(ær,T) = aTo + bx = Σ (axo + bxT )p(o (T) = Σ zop(zo xT) + bxg Σ p(zo xT) 20
150
200
a Σ zop(zo) + buT = ago + bxx
20
where a
and VI- are constants, and fais an average of images in the dataset. When the number of images is large enough, Zo contains little meaningful information.
As shown in Fig. 3 (a), we study the predicted noises of different denoising layers in DDIM model with total inference steps of 20, to illustrate our discovery. Here we refer to [45], specifying the sampling step
the diffusing S the layer that denoises the pure noise, which is equal step t = T. We can see that the predicted noise for DDIM at T = S (equal to t = T in diffusing) does not provide much useful information, while denoising with this "noise" does not provide any distinguishable image, which all support our hypothesis for the difficulty in denoising when - in sampling (or z diffusing) is large. Similar observations are also shown the concurrent work [52]. However, in our Immiscible Diffusion, while for each batch, we still have
p(xT) = N (xr; 0, T);
for each specific data point dr or To. the conditional noise distribution does not follow the Gaussian distribution because the batch-wise noise assignment
p(zr mo) * ((1;; 0,
(5)
Instead of the Gaussian distribution, we assume that the predicted noise with noise assignment has distribution described as follows
p(rr .r) far Fo. bs. ...
.. Warr; 0, 0)
where / is function denoting the influence of assignment on the conditional distribution of and by is the training batch size. Apparently, according the definition of linear assignment problem
[2 1]. f decreases when 27 2o increases its norm, specifically the L2 norm in default setting.

---

Therefore, from Equation 2 and 6, we have
p TT) = (po.))))
(7)
which means that for a specific noise data-point, the possibility of denoising it to the nearby image data-point would be higher than far-away image.
For the noise prediction task, we sec that
(xr, T) = Σ (azo + berr) xT)
fØr To.... .xop((o)) + ber
(8)
= ano (xT - Iu....) bxT
where zp. (xT ro, ro....) the weighted average of zi with weights image data-points closer to the noisy data-point 27 itself. Therefore, the noise predicted would lead to the average of nearby image data-points, which makes more senses than pointing a constant. Indeed, in Fig. we see that even for the pure noise layer, immiscible DDIM can predict the noise effectively pointing to the shape of the horse image, and the prediction in one step by subtracting the predicted noise shows the outline of the horse correctly.
3.4 Accelerating Assignment in Immiscible Diffusion
The assignment problem has been studied extensively for decades [6, 22, 7]. In this paper, we use the Hungarian algorithm [22] as our main assignment method. However, Hungarian matching has high complexity with ( which drastically slow down the training especially when we have high-dimensional image data (e.g., even using the mini image data 32 x 32 x 33 3072). To mitigate this issue, we make novel use of quantization for image data and noise, that is, we quantize the fp32 image and noise data fp16 to carry out the assignment, while maintaining the same precision input to diffusion models. This trick significantly reduces the overhead to negligible level.
To efficiently perform Immiscible Diffusion when rumning on multiple GPUs, we assign the image- noise distance matrix computation to each process, and then gather them to execute the assignment. This is particularly important high resolutions and large batch sizes are frequently required in applications.
Experiments
4.1 Experiment Settings
To elaborate the performance of Immiscible Diffusion, we utilize the proposed method on Consistency Models [47], DDIM [45] and Stable Diffusion [41], and using CIFAR-10 [20], CelebA [29], tiny, random picked 10% and the full ImageNet [4] datasets due to the limitation of computation resource. The training hyperparameters are shown in Tab. 1. Unspecified hyperparameters are taken the same as those in their baseline methods' original papers. For evaluations, we compare the results generated by our Immiscible Diffusion method and baseline using both the quantitative evaluation metric FID [10] and qualitative assessments.
Note that for Consistency Models, we use the single-step generation consistency training. For DDIM. we add no noise during the sampling and use linear scheduling for picking sampling steps. For Stable Diffusion, we directly use the implementation from Diffusers Huggingface team [40]. For fine-tuning, we use Stable Diffusion v1.4 [40] as the pre-trained model.
4.2 Training Efficiency Improvement with Linear Assignment
Unconditional generation with Consistency Models: Fig. 4, we show the FIDs of images generated with baseline and immiscible Consistency Models trained with different training steps

---

Table 1: Experiment setting.
Model
Consistency Model
Consustency Model
Consistency Model
DDIM
Stable Diflusoon
Uncondiiccal
Sluble Diltusion Cluss-oociitoonl
Stable Difluion
Fine-ttniig ImageNet
Dataset
CIEAR-10
CelebA
Tiny ImageNei
ImageNett
Full ImageNet 2048 250 256
Botch
1024
2048
Resolution
32 x 32
64 x 64
64 64
32 32
256 296
206 256
Devicss
x .650000
8.x .6000
16 x A800
X 45000
4xx 46000
8 A800
4 X A6000
(8)
CIFAR
[b]
CeleBA
[c]
tmny-ImageNet
80
Coneistancy Model
Condietancy Model
Condistancy Hadel inmiaiibbe Consistancy Model
innrcctis Consistancc Model
inmiacctbl Conniitarcy Model
24
g.20
3X
16
42
30
200 300
400 Thairing Stepp
500
000
40 -00
100
100
120
Training Stops
Training Steps
Figure 4: Evaluation of baseline and immiscible Consistency Models on (a) CIFAR-10, (b) CelebA, and (c) tiny-ImageNet dataset. We illustrate the FID of two models with different training steps. Clearly, immiscible Consistency Models have much higher efficiency than the vanilla ones.
the CIFAR- 10 dataset, the CelebA dataset and the tiny ImageNet dataset, respectively. We observe that the immiscible Consistency Model trains much faster than the baseline Consistency Model, and converges to a significantly lower FID on all these datasets. We also show the images generated by immiscible and baseline Consistency Models trained for 100k steps in Fig. 9 in the Supplemental Materials, where we find that the images generated by the Immiscible Consistency Model are much more complete and realistic. Tab. 3 in the Supplemental Materials further presents the training steps necessary to achieve specific reasonable FID thresholds. We find that the immiscible Consistency Model significantly improves the training efficiency by around 3x, proving the effectiveness of Immiscible Diffusion in training accelerations.
In the main experiment, we observe that our method on top of the Consistency Model is effective across the datasets varying from different data sizes and resolutions. Indeed, the Consistency Model is a few-step diffusion model, and our proposed Immiscible Diffusion especially works on improving the denoising effect when the noise level high, as shown in Fig. 3. The improvement of the training efficiency on such few-step diffusion model further validates our findings.
One characteristic of the Consistency Model is that it approximates the SDE-diffusion model with the ODE approximation. Thus, the original image-noise mapping is highly jumbled together since it highly possible that closed image data points are diffused to distant noise points. Our Immiscible Diffusion improves this issue by adjusting the trajectories of image-noiss mapping and making them more distinguishable.
(a)
(b)
(c)
Inndicbble COM
5.5
130
Suaing
203
250
500
Samwy (1)
244
104
180
Sear
200
5 20 Steps
Steps
100 Steps
Figure 5: Evaluation of baseline and Immiscible DDIM on CIFAR-10 dataset with different inference steps S. We find that Immiscible DDIM ourperform the baseline more significantly when the number of inference steps is smaller.
More baselines: To show the generalization of Immiscible Diffusson for more baselines, we further conduct experiments on two baselines: DDIM [45] and Stable Diffusion [39] on the CIFAR- 10 and the randomly picked 10% ImageNet dataset, respectively. As shown in Fig. 5, and detailed in Tab. 4 in Supplemental Materials, we find that our immiseible DDIM significantly improves the training speed and the FID compared to those of the baseline DDIM on the CIFAR- 10 dataset, and the improvement
8

---

is more significant when the sampling step is lower. This demonstrates the effectiveness of our proposed method works beyond the Consistency Model and can be generalized to more few-step denoising models. We also provide discussion in Part 1.1.4, showing that the effectiveness of Immiscible Diffusion can persist wide range of batch sizes. To further evaluate generalizability on the popular baseline, Stable Diffusion [39], we also conduct unconditional generation experiments on the ImageNet dataset. We observe that immiscible Stable Diffusion and baseline Stable Diffusion achieve similar FID without significant gap. yet our immiscible Stable Diffusion is able to generate much higher quality images from a subjective human judgement. For example, Fig. 14 in the Supplemental Materials shows that our proposed method generates significantly clearer images compared to the baseline. More visualization without any therry-picking can be seen in Fig. 15 in the supplementary materials. We indicate that even though FID is the primary metric and is remarkably successful, the metric is known to sometimes disagree with human judgement [23]].
Class-conditional Generation: We extend Immiscible Diffusion to class-condiiional generations on ImageNet dataset with Stable Diffusion [41], to explore the performance of Immiscible Diffusion in conditional generations. Results are shown in Fig. 6 (a), where we observe that in 20k training steps, the FID for immiscible class-conditional Stable Diffusion is 16.43, which is 1.49 lower than our Stable Diffusion baseline. We further confirm such improvements on CMMD [15], where the immiscible and vanilla models get 1.385 and 1.436 respectively. Additional evaluation on CLIPScore [9] shows that both the immiscible and the baseline models generate images with CLIPScores of 28.55, with standard deviation of 0.01 and 0.02 respectively, indicating that Immiscible Diffusion does not hurt the image- image-prompt correspondence in complicated ImageNet dataset. Qualitative comparisons in Fig. 16 in Supplemental Materials further prove such performance enhancements, which augment the effectiveness of Immiscible Diffusions into more commonly-used conditional generations.
Our Immiscible Diffusion can also be applied to enhance the fine-tuning process where numerous applications fall in. We fine-tune the stable diffusion v1.4 model [41] on ImageNet dataset, finding that immiscible fine-tuning achieves an FID of 10.28 compared to 11.45 for vanilla fine-tuning with 5k training steps. Detailed results are shown in Fig. 6. This further broadens the application of Immiscible Diffusion.
tine-tuning:
Statle DiMhadion
Stable Difiuston Invetacilee Statie Dellasion
inniacble Stable Diffuaion
11.3
13
30 12 14
16
28
20
10
Tratstna (t)
Figure 6: Evaluation of baseline and immiscible class-conditional Stable Diffusion ImageNet dataset, using 20 inference steps. (a) FID of two models trained from scratch (b) FID of two models fine-tuned on Stable Diffusion vl.4.
4.3 Discussion
To further understand the proposed Immiscible Diffusion method, we delve into several key questions to ablate our approach:
How much does image-noise distance reduce in the assignment? Tab. 2 shows the reduction in distance after the image-noise assignment. We find that the L2 distance only reduces by about 2%, with slight increase observed at higher batch sizes. However, shown in Fig, 3, the assignment is sufficient to effectively activate denoising at high noise levels, significantly boosting training efficiency, even though the distance change is low. We attribute the low distance reduction rate after the assignment to the extremely high dimensionality (3072 for each image of the CIFAR-10 dataset) of the image and noise space.
How much time does image-noise assignment cost? In Tab. 2, we indicate that our assignment method does not introduce significant extra overhead due to our utilization of quantized assignment in our practical implementation. Even for a large batch size per GPU of 1024, our algorithm only brings in an additional 22.8 ms, demonstrating the potential of utilization for future applications.

---

Immiscible and OT: who dominates the training efficiency enhancement? Our Immiscible Diffusion claims enhance training efficiency by improving miscibility noisy diffusion steps. However, the method we take towards Immiscible Diffusion, i.e. linear assignment between image and noise, also serve as roughly approximate OT between image and noise, which might intuitively benefit the diffusion through straightening the diffusion paths [36, 50]. However, the previous section shows that the image-noise distance is only reduced by -2%, motivating us to ask if OT is really the dominant factor?
To answer this question, we ablate these two factors: OT and immiscibility. We design a non-Ol Immiscible Diffusion experiment which keeps the immiscible property while not involving the OT. This is achieved by assigning images to the flipped noise whose all dimensions are reversed, while using the original noise during diffusion. In such a way, the image-noise pair no longer follows OT, but still qualifies the Immiscible Diffusion i.e. images are still assigned to limited area. Interestingly, we observe that the OT Immiscible Diffusion can still accelerate and enhance the diffusion training, which is nearly comparable to the OT Immiscible Diffusion in final stages, as shown in in Fig. 7. Considering that the non-OT version introduce miscibility in middle diffusion layers, which we posit for its difference to OT version, we conclude that Immiscible Diffusion is dominant in enhancing the diffusion model's performance, compared to the benefits from OT.
Vanilla DCIM
Rox CT Inmracible DOIN immiscibie DOIM
Table 2: Image-noise data-point L2 distance reduction after the assignment for minimizing it and the time cost for the assignment.
Batch Size
128
256
512
1024
DDist.
-1.3%%
-2.16%
-2.32%
-2.44%
9.0
160 75
335 17% 150
200 225 250
Assignment
5,4
6.7
8.8
22.8
Figure 7: Ablation of OT in Immiscible Diffusion. FIDs of OT and non-OT Immiscible Diffusion indicates that it is the Immiscible Diffusion rather than OT that dominate the performance enhancement.
Time (ms)
Conclusion, Limitations, and Future Work
Inspired by the immiscibility phenomenon in physics, we introduce Immiscible Diffusion, method to improve image-noise mapping to accelerate diffusion training. Specifically, Immiscible Diffusion is an asiigmment-then-difuue
strategy. One way of it is to minimize the image-noise pair distance within mini-batch batch so that each image is diffused to nearby noise areas. This simple approach requires only one line of code and includes a crantized-assignmenn strategy to reduce computational overhead.
Experiments show Immiscible Diffusion approach speeds up Consistency Model's training by approximately 3x on the CIFAR-10 dataset, 1.3x on the CelebA dataset, and 1.2x on the tiny- ImageNet dataset, well as in conditional generation and fine-tuning Stable Diffusion. Thus, we show that Immiscible Diffusion can generalize to across datasets, baselines and tasks. Further analysis is provided to explain how this works.
Limitation. The assignment strategy is one straightforward way for Immiscible Diffusion, but not necessarily optimal. Due to the limited computational resources, experiments mainly conducted on small-scale datasets, so we lack the validation on larger-scale datasets such as LAION. In future work, we will improve the assignment strategy to cater to practical utilization of conditional generation such as accelerating the general text-to-image or text-to-video diffusion training.
Broader impact. With the increased use of diffusion models for image and video generation, the training of diffusion models certain to become an increasing portion of data center workloads. Moreover, training time is a significant bottleneck in model development. Our proposed method significantly improves the efficiency of diffusion model training. We believe that our method has the potential to accelerate progress and reduce the cost of development in this field.
10

---

References
[1] Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprini arXiv:2310 2300:.0699 2023.
[2] Jannis Chemseddine, Paul Hagemann, Gabriele Steidl, and Christian Wald. Conditional wasser- stein distances with applications in bayesian ot flow matching, 2024.
[3] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint 2XXiv:2303.04133 2023.
[4] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. Iece, 2009.
[5] Wei Deng, Weijian Luo, Yixin Tan, Marin Biloš, Yu Chen, Yuriy Nevmyvaka, and Ricky T. Q. Chen. Variational schrödinger diffusion models, 2024.
[6] Lawrence W Dowdy and Derrell V Foster. Comparative models of the file assignment problem.
ACM Computing Surveys (CSUR), 14(2):287-313, 1982.
[7] Paul C Gilmore. Optimal and suboptimal algorithms for the quadratic assignment problem. Journal of the society for industrial and applied mathematics, 10(2):305-313, 1962.
[8] Ian Goodfellow, Jean Pouget-Abate Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[9] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, 2022.
[10]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium, 2018
[11] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02335
2022. [12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in nearul information processing systems, 33:6840-6851, 2020.
[13] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8644 2022.
[14] Thibaut Issenhuth, Ludovic Dos Santos, Jean-Yves Franceschi, and Alain Rakotomamonjy. Improving consistency models with generator- induced coupling, 2024.
[15] Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation, 2024.
[16] Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. arXiv preprint arXiv:2312.02666, 2023.
[17] Daegyu Kim, Jooyoung Choi, Chaehun Shin, Uiwon Hwang, and Sungroh Yoon. Improving diffusion-basee generative models via approximated optimal transport, 2024.
[18] Dongjun Kim, Chich Hsin Lai, Wei-Hsiang Liao. Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279. 2023.
[19] Akio Kodaira. Chenfeng Xu, Toshiki Hazama, Takanori Yoshimoto, Kohei Ohno, Shogo Mitsuhori, Soichi Sugano, Hanying Cho, Zhijian Liu, and Kurt Keutzer. Streamdiffusion: A pipeline-level solution for real-time interactive generation. arXiv preprint ar2/112312.12491, 2023.
[20] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.

---

[21] H. W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly. 2(1-2):83 97, 1955.
[22] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[23] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes fr) ''echet inception distance. arXiv preprint arXiv:2203. 060266 2022.
[24] Sangyun Lec, Beomsu Kim, and Jong Chul Ye. Minimizing trajectory curvature of ode-based generative models, 2023.
[25] Muyang Li, Ji Lin, Chenlin Meng, Stefano Ermon, Song Han, and Jun- Zhu. Efficient spatially sparse inference for conditional gans and diffusion models, 2023.
[26] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang. Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 17535-17545, October 2023.
[27] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023.
[28] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In The Twelfth International Conference on Learning Representations, 2023.
[29] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
[30] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[31] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022.
[32] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Prucessing Systems, 36, 2024.
[33] Simian Luo, Yigin Tan, Longbo Huang. Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023.
[34] OpenAL. https://openai.coo//sno 2024.
[35] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint 12.0::22/2.09748, 2022.
[36] Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Earich, Brandon Amos, Yaron Lipman, and Ricky T. Q. Chen. Multisample flow matching: Straightening flows with minibatch couplings, 2023.
[37] Aaditya Prasad, Kevin Lin, Jimmy Wu, Linqi Zhou, and Jeannette Bohg. Consistency policy: Accelerated visuomotor policies via consistency distillation, 2024.
[38] Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis, 2024.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
12

---

[42] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17442, 2023.
[43] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies, 2015.
[44] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[45] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[46] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint XrXiv:23 10.::2110..188 14189, 2023.
[47] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023.
[48] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 1.e11.1.4455 13456, 2020.
[49] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36:1363-1389, 2023.
[50] Alexander Tong, Kiliaan Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport, 2024.
[51] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cour- napeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Ker, Eric Larson, CJ Carey, Îlhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, Aditya Vijaykumar, Alessandro Pietro Bardelli, Alex Rothberg, Andreas Hilboll, Andreas Kloeckner, Anthony Scopatz, Antony Lee, Ariel Rokem, C. Nathan Woods, Chad Fulton, Charles Masson, Christian Häggström, Clark Fitzgerald, David A. Nicholson, David R. Hagen, Dmitrii V. Pasechnik, Emanuele Olivetti, Eric Martin, Eric Wieser, Fabrice Silva, Felix Lenders, Florian Wilhelm, G. Young, Gavin A. Price, Gert-Ludwig Ingold, Gregory F. Allen, Gregory R. Lee, Hervé Audren, Irvin Probst, Jörg P. Dietrich, Jacob Silterra, James T Webber, Janko Slavič, Joel Nothman, Johannes Buchner, Johannes Kulick, Johannes L. Schönherggrrrrrrre José Vinícius de Miranda Cardoso, Joscha Reimer, Joseph Harrington, Juan Luis Cano Rodríguez, Juan Nunez- Iglesias, Justin Kuczynski, Kevin Tritz, Martin Thoma, Matthew Newville, Matthias Kümmerer, Maximilian Bolingbroke, Michael Tartre, Mikhail Pak, Nathaniel J. Smith, Nikolai Nowaczyk, Nikolay Shebanov, Oleksandr Pavlyk, Per A. Brodtkorb, Perry Lee, Robert T. McGibbon, Roman Feldbauer, Sam Lewis. Sam Tygier, Scott Sievert, Sebastiano Vigna, Stefan Peterson, Surhud More, Tadeusz Pudlik, Takuya Oshima. Thomas J. Pingel, Thomas Robitaille, Thomas Spura, Thouis R. Jones, Tim Cera, Tim Leslie, Tiziano Zito, Tom Krauss, Utkarsh Upadhyay, Yaroslav O. Halchenko, and Yoshiki Vázquez-Bazza, Seipy 1.0: fundamental algorithms for scientific computing in python. Nature Methods, 17(3):261-22) 27222, February 2020.
[52] Kai Wang, Yukun Zhou, Mingjia Shi, Zhihang Yuan, Yuzhang Shang, Xiaojiang Peng, Hanwang Zhang, and Yang You. closer look at time steps worthy of triple speed-up up for diffusion model training, 2024.
[53] Zhendong Wang. Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang. Weizhu Chen, and Mingyuan Zhou. Patch diffusion: Faster and more data-efficient training of diffusion models. arXiv preprint arXiv:2304. /2526, 2023.
[54] Siyu Xing. Jie Cao, Huaibo Huang, Xiao Yu Zhang, and Ran He. Exploring straighter trajecto- ries of flow matching with diffusion guidance, 2023.
[55] Chenfeng Xu, Huan Ling, Sanja Fidler, and Or Litany. 3difftection: 3d object detection with geometry-awar diffusion features, 2023.
13

---

A
Supplemental Materials
A.1
Additional Experiment Results
A.1.1
Quantitative Training Efficiency Improvements for Immiscible Consistency Model
Table 3: Immiscible Diffusion boosts training efficiency for Consistency Model on CIFAR- 10 dataset.
FID threshold
12.00
11.00
10.00
Training Steps (k) for Baseline Consistency Model
290
450
>800
Training Steps (k) for Immiscible Consistency Model
110
140
190
A.1.2
Quantitative FID Improvements for Immiscible DDIM with Different Inference Steps.
Table 4: FID improvements of Immiscible DDIM with different inference steps
Inference Steps
1000
500
100
50
20
FID with baseline DDIM
3.82
3.91
5.2
6.63
10.03
FID with Immiscible DDIM
3.67
3.74
4.32
5.14
7.46
AFID
-0.15
-0.17
-0.88
-1.49
-2.57
A.1.3 Ablation on the Distance Measurement Methods in Noise Assignment.
We use the L2 norm for experiments. However, note that the L2 norm may face more challenges in distance evaluation in nigh-dimensional spaces compared to the L1 norm. Therefore, compare the performance of immiscible DDIMs using assignments based the L1 and L2 norms. The results, as illustrated in Tab. 5, show that using the 1.2 norm provides better performance than the [I1 norm.
Table 5: FID of using L1 or L2 norm for noise assignment in immiscible DDIM on CIFAR- 10.
Training Steps (k)
70.2
93.6
117.0
140.4
163.8
DDIM
6.30
5.56
4.86
4.34
4.12
Immiscible DDIM using L2 Norm
5.28
4.56
4.13
3.81
3.70
Immiscible DDIM using L1 Norm
5.34
4.66
4.16
3.87
3.82
A.1.4 Ablation on the Batch Size on Immiscible DDIM.
The effectiveness image-nois assignment can intuitively rely on the batch size. Therefore, we perform comparison to see the effectiveness of Immiscible Diffusion on DDIM across selected range of batch sizes, whose result is shown in Fig. 8. We observe that while larger batch sizes consistently accelerate the training as expected, its training efficiency enhancement is not large as that from Immiscible Diffusion. Immiscible Diffusions continously improve the training efficiency and the performance in the whole selected range of batch sizes.
14

---

Vanilla deim-120
Immiscible ddim-122
Vanilla deim-256
Immiscible ddim-256
ddim-512
Immiscible ddim-5512
8-
FID
50
100
150
200
250
300
350
Steps(k)
Figure 8: Effectiveness of Immiscible DDIM selected range of batch sizes.
A.2
Qualitative Evaluations of Immiscible Diffusion
A.2.1 Generated images from immiscible and baseline Consistency Models trained on CIFAR-10 (Top) and CelebA (Down) with the same training steps.
Immiscible Consistency model
Consistency model
PCI
Immiscible Consistency model
Consistency model
Figure 9: Qualitative comparison for Immiscible and baseline Consistency Model. We show images generated with the two models trained for 100k steps respectively. Compared to baseline method, immiscible models capture more details and more features of objects.
15

---

A.2.2 Generated images from immiscible and baseline Consistency Models trained on CIFAR-10 Dataset for 100k steps without cherry-picking.
Figure 10: Generated images from baseline Consistency Models trained on CIFAR-10 Dataset for 100k steps without cherry-picking.
16

---

Figure 11: Generated images from immiscible Consistency Models trained on CIFAR-I0 Dataset for 100k steps without cherry-picking.
17

---

A.2.3 Generated images from immiscible and baseline Consistency Models trained on CelebA Dataset for 100k steps without therry-picking.
Figure 12: Generated images from baseline Consistency Models trained on CelebA Dataset for 100k steps without cherry-pickee picking.
18

---

Figure 13: Generated images from Immiscible Consistency Models trained on CelebA Dataset for 100k steps without cherry-picking.
19

---

A.2.4 Generated images from immiscible and baseline stable diffusion models trained unconditionally on 10% % ImageNet for 70k steps.
Immiscible Stable Diffusion
Stable Diffusion
Immiscible Stable Diffusion
Stable Diffusion
Figure 14: Images generated by immiscible and baseline Stable Diffusion trained unconditionally on ImageNet for 70k steps. We see that the Immiscible Stable Diffusion presents more reasonable modal and catch more general features and details.
20

---

A.2.5 Generated images from immiscible and baseline stable diffusion models trained unconditionally on 10% % ImageNet Dataset for 70k steps without eeerry-ickiing.
Immiscible Stable Diffusion
Stable Diffusion
Immiscible Stable Diffusion
Stable Diffusion
Immiscible Stable Diffusion
Stable Diffusion
Figure 15: Generated images from immiscible and baseline stable diffusion models trained uncondi- tionally on 10% ImageNet Dataset for 70k steps without cherry-pickinng
21

---

A.2.6 Generated images from immiscible and baseline stable diffusion models trained conditionally on ImageNet Dataset for 20k steps.
Immiscible
S ttable Diffusion
ttable Diffusion
Figure 16: Generated images from immiscible and baseline stable diffusion models trained condition- ally on ImageNet Dataset for 20k steps.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: Please refer our abstract and conclusion. We propose an extremely simple method inspired by physical phenomenon. With just one line of code, our method accelerate the diffusion training by large margin.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Our current assignment method is quite straightforward.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification: In our method part, we thoroughly explain how our immiscible diffusion work with mathematical illustration.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We illustrate all the experiment setting and we will publish the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our method is just one line of code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/guides/ Stweees/DooeSummmiiiiiioe
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss cttppsccccpbccccccceececcccccccc htteggudddeeeeoeSummiiieeeee /Cooeeeooiissseeeeeee
cessionPoocyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Besides the Immiscible Diffusion part, all training and inference are same to baseline.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We use commonly used FID.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Most our experiments are conducted on A6000 GPUs and some on A800 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https pettpa:/nneuripei htcc/public/Bthcccieeeeeee
Answer: [Yes]
Justification: We did.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See conclusion part
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not have.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
We do not use.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hco....c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve this.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "0u8gM9e9zI", "reviewer": "Reviewer_h6mk", "summary": "This paper shows that the current diffusion training strategy diffuses each image into the entire noise space, making it difficult to optimize the model and thus slow to converge. Inspired by the fact that miscibility can be changed according to various intermolecular forces in physics, this paper proposes Immiscible Diffusion, a simple and effective method that accelerates diffusion training by pre-assigning noises in each mini-batch before standard training, which can be implemented in just one line code. This assignment operation can be accelerated by fp16/fp8, which greatly improves training efficiency while avoiding algorithm complexity and is applicable to various baselines, including Consistency models, DDIM and Stable Diffusion.", "questions": "Please refer to the Weaknesses", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This paper points out that existing diffusion training will map each image to all points in the Gaussian distribution, which means that each point can be denoised to any source image, leading to inherent difficulties in diffusion training. This is a new perspective on diffusion training acceleration, and it sounds reasonable.\n2. The proposed assignment strategy is simple and effective, and one line of code achieves a significant performance improvement, without much computation complexity.\n3. Immiscible Diffusion improves diffusion training speed while improving the FID metric, and it is orthogonal to existing diffusion training acceleration methods.\n4. Clear and concise presentation, writing and diagrams.", "weaknesses": "1. The noise assignment strategy proposed in this paper essentially introduces a priori assumption into the diffusion training process: different images should correspond to different points in the Gaussian distribution. This prior reduces the difficulty of diffusion training, but also brings concerns about practicality and generalization:\n    - Practicality: The datasets for unconditioned generation used in this paper are all based on image classification (CIFAR/CelebA/ImagenNet), so these data distributions themselves already meet the above prior assumptions (each class is a kind of image distribution). However, for image datasets with more complex distributions, such as LAION, whether the proposed assignment strategy can still be used is an uncertain question, and this paper also lacks relevant experiments. You can demonstrate that this is not a problem by training unconditional generation experiments on a subset of LAION.\n    - Generalization: Since images with the same data distribution are assigned to a specific noise interval during training, this may cause the model to lose the ability that denoising each point in the noise space into any source image during inference, thereby damaging the generalization of the generated images. However, the paper lacks experiments to evaluate the generalization of generated images.\n\n2. Considering that all the unconditional generation results in the paper are trained on classification datasets, is the distribution of noise assigned by the proposed method strongly correlated with the category? Intuitively, after the proposed noise assignment, images of the same category should have closer noise in current batch.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are really excited to hear that you agree with our strengths listed above, and we sincerely hope that the proposed perspective of image-noise matching can be realized and further discovered by our diffusion society. We hope to address your concerns below:\n\n**W1-Practicality**\n\nWe thank you for the concern on whether immiscible diffusion would work on datasets without simple classification distributions, and we highly appreciate your suggestion on training on LAION. We train conditional Stable Diffusion model on a subset of LAION (laion-10k) with a batch size of 512. We see that after 25k steps, the FID for immiscible and vanilla diffusion is 124.55 and 146.69 respectively, showing that immiscible diffusion can work on the training efficiency of datasets with complex distributions. Note that the training steps are limited by the time and computational resources.\n\n**W1-Generalization**\n\nWe can’t agree more to explore on this concern, which we have been seriously considering after submission. A typical example and an important application here is conditional image generation, where noise is randomly sampled and paired with the text prompt. We performed class-conditional image generation on Stable Diffusion with ImageNet dataset. Our results in Fig. R2 showed that immiscible diffusion outperformed the vanilla one, specifically the FID for immiscible and vanilla diffusion is 20.90 and 22.44 respectively, and the training steps necessary to go below the FID <23 is 12.5k for vanilla and 7.5k for immiscible diffusion. This effectively suggests that the generalization problem is not significant for current immiscible diffusion methods, which is not surprising as the distance between image and noise shrinks only ~2% after the assignment. We hope this experiment can help to improve our discussion on generalization. \n\n**W2 Image Correlation according to Classes**\n\nWe appreciate your interesting questions, but the answer is no. The images are more correlated according to its own structure rather than the category. In our response to Reviewer y1v3 - W3 - (1), we perform an experiment showing that when enough perturbation is added onto a noise, the image would change to another image in another class, which proves our claim, as shown in Fig. R7. Furthermore, we had an experiment in W1-Practicality on LAION-10k, which nearly has no classifications and immiscible diffusion preliminarily works. This also proves that classification is not a prerequisite for immiscible diffusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u1NPoLQ538", "reviewer": "Reviewer_nEkA", "summary": "The authors propose an approach to mitigate the random correspondence of noise-data mapping in vanilla diffusion models. They first assign target noise by minimizing the total image-noise pair distance within a mini-batch, and then diffuse the data into noise. Experimental results seem to demonstrate the potential of this approach. However, some of the arguments are mathematically sloppy and not well testified.", "questions": "1. Is the proposed method, which does in-batch reassignment, sensitive to batch size?\n\n2. Is the proposed algorithm applied to all $t_b$? In theory, all the arguments that authors make may only hold for large timesteps. Thus, the proposed method should work effectively even if it is applied to large diffusion time. It will be nice to see experimental results related to this.\n\n3. Does the proposed method work effectively with fine-tuning? \n\n4. Even though authors propose to quantize to avoid computation bottleneck, by how much the batch-wise image-noise assignment increases the runtime of training?", "limitations": "Yes, authors have discussed potential limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The approach sounds novel and interesting. Also the implementation seems simple.", "weaknesses": "1. The way of presentation is not reader friendly. I suggest to review DDIM and define its notations first, explain your motivation from DDIM, and show your algorithm/solution.  \n\n2. No strong evidence showing your claim that vanilla diffusion models are miscible. Any toy examples to support it?\n\n3. The argument from L155-L161 is sloppy and hard to understand the argument. For instance, Eq. (2) holds in what sense (density? certain divergence?). Indeed, I believe it will not hold for most diffusion models.\n\n4. More justification and rationale is needed to the assumption Eq. (7). Do authors suppose that should hold for all $t$? If so, I do not think it is a correct assumption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your acknowledgement on the potential of our method. Besides, we also hope to emphasize that our method is one implementation to address the miscibility problem, into which we hope to inspire more work to work on. Below we will address your concerns:\n\n**W1 - Mathematical Proof**\n\nWe re-write the part 3.3 below:\n\nIn DDPM, we know for any image data-point $x_0$, when it comes to the last diffusion step $T$, i.e. $t = T$, the image is sufficiently wiped out and nearly only Gaussian noise remains. Therefore,\n$$\n    q(x_T \\mid x_0) \\approx \\mathcal{N}(x_T; 0, I) \\approx p(x_T),  where \\quad x_T(x_0,\\epsilon) = \\sqrt{\\overline{\\alpha_t}} x_0 + \\sqrt{1 - \\overline{\\alpha_t}} \\epsilon, \\quad \\epsilon \\sim \\mathcal{N}(0, 1), \\[1\\]\n$$\nUtilizing Bayes' Rules and Equation 1, we can find that for a specific $x_T$:\n$$\n\\quad p(x_0 \\mid x_T) = \\frac{q(x_T \\mid x_0) \\cdot p(x_0)}{p(x_T)} \\approx p(x_0) \\quad, \\[2\\]\n$$\nwhich indicates that the distributions of the corresponding images for any noise data-point are the same as the distribution of all images.\n\nThe simplified training objective in DDPM is the added noise $\\epsilon(x_t, t)$. However, we find that for a specific point $x_T$ in the noise space at diffusion step $T$,\n\n$$\n\\epsilon(x_T, T) = a x_0 + b x_T = \\sum_{x_0} (a x_0 + b x_T) p(x_0 \\mid x_T) = a \\sum_{x_0} x_0 p(x_0 \\mid x_T) + b x_T \\sum_{x_0} p(x_0 \\mid x_T) \\\\\n= a \\sum_{x_0} x_0 p(x_0) + b x_T = a \\bar{x_0} + b x_T  \\[3\\]\n$$\nwhere $a = -\\frac{\\sqrt{\\overline{\\alpha_t}}}{\\sqrt{1-\\overline{\\alpha_t}}}$ and $b = \\frac{1}{\\sqrt{1-\\overline{\\alpha_t}}}$ are constants, and $\\bar{x_0}$ is an average of images in the dataset. When the number of images is large enough, $\\bar{x_0}$ contains little meaningful information.\n\nHowever, in our Immiscible Diffusion, while we still have\n$$\np(x_T) = \\mathcal{N}(x_T; 0, I), \\[4\\]\n$$\n\nfor each specific data point $x_T$ or $x_0$, the conditional noise distribution does not follow the Gaussian distribution because of the batch-wise noise assignment\n$$\np(x_T \\mid x_0) \\neq \\mathcal{N}(x_T; 0, I). \\[5\\]\n$$\nInstead of the Gaussian distribution, we assume that the predicted noise with noise assignment has a distribution:\n$$\np(x_T \\mid x_0) = f(x_T - x_0, batch size, ...) \\mathcal{N}(x_T; 0, I), \\[6\\]\n$$\nwhere $f$ is a function denoting the influence of assignment on the conditional distribution of $x_T$. According to the definition of the linear assignment problem, $f$ decreases when $x_T - x_0$ increases its norm, say L2 norm as our default setting.\n\nTherefore, from Eqn. 2 and 6, we have\n$$\n\\quad p(x_0 \\mid x_T) = f(x_T - x_0, ...) p(x_0), \\[7\\]\n$$\nwhich means that for a specific noise data-point, the possibility of denoising it to the nearby image data-point would be higher than to a far-away image.\n\nFor the noise prediction task, we see that\n$$\n\\epsilon(x_T, T) = \\sum_{x_0} (a x_0 + b x_T) p(x_0 \\mid x_T) = a \\sum_{x_0} f(x_T - x_0, ...) x_0 p(x_0) + b x_T = a \\overline{x_0 f(x_T - x_0, ...)} + b x_T  \\[8\\]\n$$\nwhere $\\overline{x_0 f(x_T - x_0, ...)}$ is the weighted average of $x_0$ with more weights on image data-points closer to the noisy data-point $x_t$. Therefore, the predicted noise leads to the average of nearby image data-points, which makes more sense than pointing to a constant...\n\n**W2 - Evidence on Miscibility**\n\nThe evidence is in Fig. 3(a), which shows the predicted noise in each step, and the image generated solely with the predicted noise for this step, both from a trained DDIM. In this 20 steps sampling, T=20 is the step denoising from pure noise while T=1 is the step outputting the image. We see that the predicted noise at T=20 looks messy, and the image generated with this noise has little meaningful info. These support that the model can not determine the image to be denoised to, which supports the miscibility.\nMathematically, in vanilla diffusion models, the noise added to each image during training is N(0, I). Therefore, each image would be projected to the noise space with the same possibility distribution N(0, I), which is miscible.\n\n**W3&4 - Typo on Eqn. 2 and Eqn. 7**\n\nWe appreciate your comment. We change the notation t to T, which means the last diffusion step, so that Eqn. 2 would approx. hold considering the new Eqn. 1.\n\n**Q1 - Discussion on Batch Size (BS)**\nTo see its impact, we vary the BS on DDIM with CIFAR with 50 sampling steps. Results in Fig. R9 show that larger BS enjoys better FID, but the influence of immiscibility significantly outweighs that of BS, and such influence does not vary significantly across selected BS.\n\n**Q2 - Applying Assignment to Which Steps**\n\nYes! We further apply immiscible diffusion on large diffusion steps only, on conditional generation with Stable Diffusion (SD) and ImageNet, BS = 2048 and training steps = 20k. We compare: 1) Vanilla diffusion; 2) Immiscible diffusion on all steps; 3) Immiscible diffusion on t > 50% total diffusion steps;4) Immiscible diffusion on t > 75% total diffusion steps. The FID results are 1) 22.44; 2) 20.90; 3) 21.23; 4) 21.51. We find that denoising at large steps indeed improves the performance comparing to vanilla diffusion. This matches findings in Fig.3 that our method improves large diffusion steps. Assigning the image-noise across more steps can further improve the performance, which saves us the efforts to finetune the hyperparameters of steps.\n\n**Q3 - Finetuning**\n\nYes. We finetune conditional Stable Diffusion with pre-trained stable-diffusion-v1.4 model on ImageNet dataset with a batch size of 512. We see that after 2.5k steps, the FID for immiscible and vanilla stable diffusion is 11.10 and 12.19 respectively, supporting that our method works effectively for fine-tuning.\n\n**Q4 - Runtime for Assignment**\n\nWe refer to Tab. 3 for the detailed time necessary for the assignment with different batch sizes. In a typical situation (DDIM, batch size=256), each training step is \\~460ms, where the 6.7ms assignment time is truly negligible (~1.5% overhead).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HFfbydreQ3", "reviewer": "Reviewer_y1v3", "summary": "This paper is motivated by the miscible phenomenon of physics and transfers it into the diffusion training process, which is very interesting. The author proposes a noise-assigned strategy based on distance, which is simple but powerful for faster diffusion model training. The experiment and the theoretical analysis validate the core idea of the paper and show good results.", "questions": "See the weakness.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The author provides a faster diffusion model training strategy while maintaining the generation quality.\n2. The motivation is novel and interesting with good writing.\n3. The experiments show the robustness and university of the proposed method and the efficiency of the projected module.", "weaknesses": "1. In my view, grouping the proposed noise assignment method is similar to fixing and memorizing the relationship with noise and corresponding image, limiting the generation diversity of the diffusion model. However, the experiment shows that the FID could further be reduced, Hoping that the author can explain this phenomenon.\n2. As shown in the experiment, the noise distance change is only 2%, why such a minimal change could bring remarkable training speed up? The author should do a theory analysis in depth. \n3. To validate my concern, please provide two experiments i) providing the result generated by a noise and its minimal change variant (only 10 pixel value changes, etc). ii) providing the result generated by a certain noise with different seeds.\n4. I’m curious about the scaling law phenomenon. As shown in stable diffusion, training the diffusion model with large image data could bring surprising topological ability. However, the way the author trains the diffusion model, the noise seems to connect with certain images (same class, etc.) which will limit the generation diversity for more data. I know this experiment requires more computational resources, the author could only provide the theory analysis of the scaling law ability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your acknowledgement of our work. Besides, we also hope to emphasize that our method is one simple implementation to address the miscibility problem demonstrated to be important, into which we hope to inspire more work to address. We hope this work can benefit the diffusion model society beyond the implementation itself. We hope to address your concerns below from our understanding:\n\n**W1 - Diversity**\n\nWe sincerely thank you for the comment. Balancing the model fitting for FID and diversity is truly hard. When there are no image-noise relationships, as shown in Fig. 3(a), the denoising of noisy layers can not perform meaningful denoising functions due to miscibility so the model is hard to be optimized to fit the data; however when the relation comes too tight like “fixing and memorizing”, there can logically be diversity problems as you raised.\n\nOur method balances between these two extremes: we perform image-noise assignment in a mini-batch. In this way, we avoid significant miscibility and make noisy denoising layers effective (as shown in Fig. 3(b)) by encouraging the diffusion of each image to the surrounding area. From the other hand, considering the low batch size (~256 for DDIM in CIFAR) and the very high image dimension (32 \\* 32 \\* 3=3072 for CIFAR), such encouragement is very weak. Tab. 3 shows that the distance between image and noise only decreases ~2% after assignment (the stddev of it is >10%). Therefore, the diffuseable area for each image is still very flexible and the diversity can be kept. \n\nOur image generation experiments further confirm these claims. Firstly, as suggested by your 3rd point, we show in Fig. R7 that immiscible diffusion does not significantly affect the diversity. Furthermore, we employ immiscible diffusion to the conditional generation problem with Stable Diffusion on ImageNet. Results in Fig. R2 shows that image quality as well as the FID are improved, which further supports that our immiscible diffusion balances the miscibility and the diversity well.\n\nAs a conclusion, we believe that immiscible diffusion takes the balance between miscibility and diversity, thus providing faster training without significantly sacrificing the diversity.\n\n**W2 - Theory for Performance Enhancement**\n\nWe really appreciate your question. Immiscible diffusion works by making images less miscible in the noise space, which is proved in Fig. 3: where we see that the denoiser learns which image to reconstruct after immiscible training. Our further theoretical experiment in Fig. R5 shows that such immiscibility is mainly achieved by pushing away images far from each other in the noise space, as the fitted line only has a significant positive absolute value in the far right of the image. In this way, we achieve more immiscible diffusion while avoiding major disturbance to the diversity. The image-noise distance change of ~2% is to prove how minor our disturbance is, which supports that immiscible diffusion would not significantly affect the diversity as discussed in the last question.\n\n**W3 - 2 Proposed Experiments**\n\nWe are so grateful for your offering of experiment designs, which inspire us a lot. We perform both sampling experiments with the setting of Unconditional DDIM + CIFAR10 + Batch Size 256 + 187k Training Steps + 50 Sampling Steps:\n\ni) We generate images with a fixed noise, altered on 10/96/320 dimensions (10 pixel = 30 dimensions due to 3 channels), which are shown in Fig. R7. We found that altering 10 dimensions would not significantly alter the image for both the immiscible and the vanilla DDIM. With 96 dimensions altered, some minor changes are observed for both models. And when we altered 320 dimensions (~10%), both models show class alternations in generated images. In conclusion, we don’t observe a significant diversity difference between the vanilla and the immiscible DDIMs. \n\nii) The results can be found in Fig. R8 We observe that with the same noise, the global seed does not seem to have an impact on the generated image. We think that is caused by little randomness of the DDIM sampling process. For DDPM, diverse classes of images are generated, as randomness in DDPM can also come from the sampling path, in addition to the initial noise.\n\n**W4 - Scaling Law**\nThanks for the comment. We studied the scaling law from theoretical analysis. For theoretical analysis, as illustrated above, our method almost does not change the assumption that $P(X_T)$ should be distributed in Gaussian while we push the image-noise distribution away from each other. This means our method still preserves the manifold of diffusion training while making the optimization on the high-noise level easier (see Fig. 3 of our main paper). In that case, our method should not hurt the scaling-law facts for Diffusion Models. We also conducted experiments on the whole ImageNet dataset and found that our immiscible-Stable Diffusion can improve the performance of the vanilla Stable Diffusion model. This further demonstrates that our method can work on large-scale datasets (see general response).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gcKqWcWAxe", "reviewer": "Reviewer_Da24", "summary": "During training of Diffusion Models, images are corrupted with standard Gaussian noise and the Diffusion Model is tasked to denoise the corrupted images. The Gaussian noise used during training is often sampled independently of the clean image.\n\nThis submission presents the following \"Immiscible Diffusion\" method: within each mini-batch, we should reassign the pairs of images and noise samples, so that each image is corrupted with a ‘closer’ noise sample.\n\nThe submission claims \"Immiscible Diffusion\" is faster to train to reach the same FID threshold, with quantitative evaluation, and \"can catch more details and more feature of objects\" with some cherry-picked qualitative results.", "questions": "Here are my suggestions for this submission:\n\n**Q1/** Discussion and comparison of related works. By not discussing the related works, it is not possible for me to see the novelty in this submission, given that similar methods have been presented in existing literature. (see also **W2**)\n\n**Q2/** The presentation quality (figures, clarity) and mathematical explanations should be improved (see also  **W1**, **W3** and **W4**).\n\n**Q3/** All qualitative and quantitative results need to be shown clearly, without mistake or cherry picking (see **W4**).", "limitations": "**L1/** The authors apply some matching within each mini-batch. So there is now a dependence on batch size, it is not sure the method would work with other batch sizes. For instance on GPU with little memory and batch size of 1, the method actually doesn’t change training at all compared to normal diffusion.\n\n**L2/** Some theoretical results on diffusion may not hold anymore, e.g. that the ideal predicted denoising direction corresponds to the score (gradient of log likelihood) of the distribution of noisy data.\n\n**L3/** The method only works for unconditional generation, which is a severe limitation. (EDIT: the rebuttal contains 1 experiment on conditional generation, but discussion is missing (how come noising is immiscible but denoising is not [noise samples can still lead to any class]?) and arguments like “the average of images does not contain much meaningful information” do not hold anymore.)\n\n**L4/** Like any generative model, this work could be unethically used. It is crucial for researchers to consider designing safeguards to ensure the model can ignore inappropriate requests (eg, training or fine-tuning your Immiscible Diffusion model on inappropriate images). Can you provide a small discussion on existing safeguards and how follow-up work can address potential misuse of your method?\n\n**L5/** The authors did not communicate the details (incl. license and terms of use) of the datasets or the models that they used via structured templates. The checklist incorrectly answered “NA” for “12. Licenses for existing assets”", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "**S1/** The metaphor of “immiscible fluids” is creative.\n\n**S2/** The described method is simple.\n\n**S3/** L489: The authors say they will publish their code (although the paper claims it is only one line)", "weaknesses": "The main weaknesses and reasons to reject the paper are the following:\n\n**W1/** The notations and mathematical derivations are imprecise, unnecessarily complex, and potentially incorrect.\n\n**W2/** Similar methods have already been presented in previous literature. The submission does not discuss these existing works, nor compare with them.\n\n**W3/** The presentation, including the figures quality and writing, needs to be improved.\n\n**W4/** The presented results are not convincing.\n\n**More details:**\n\n**W1/ The notations and mathematical derivations are imprecise, unnecessarily complex, and potentially incorrect.**\n\n* The Equations in the Section 3.3 are very hard to understand, and I am not sure they are necessary as they make understanding more difficult. The section could simply be rephrased as follows without the need to introduce complex and potentially incorrect equations: When the noise samples and data points are sampled independently, the optimal (in MSE sense) denoising prediction from a completely noisy observation is the mean of the data distribution. If we additionally know that data points are only corrupted with \"close\" noise, then the optimal denoising prediction from a completely noisy observation is, intuitively, the average of only \"close\" data points, which can allow to converge faster to a data point at generation time.\n* The notation \"When $t → \\infty $, $p(x_t|x_0) = N(0, I)$\" in Equation 1 is not precise (equality? convergence?).\n* ​​L155 states that Equation 2 indicates \"the distributions of the denoised images for any noise data-point are the same, which is equal to the distribution of the image data\", but Equation 2 does not contain any \"denoised images\", only clean images $x_0$ and their corrupted versions $x_t$.\n* In Equation 3, the formulas for $a$ and for $b$ are inverted. Also the \"minus\" sign should be in front of $a$ and not $b$. The equation further does not really make sense according to the paper’s notations, since \"p(x_0|x_t) = p(x_0)\" only “when $t → \\infty$”. It is not mathematically correct to replace only a few terms of an equation by their limit, but keeping dependencies in $t$ for other terms. $a$ and $b$ are said to be constant, but their formula clearly shows they depend on $t$ (or are $a$ and $b$ the limits “when $t → \\infty$”). \n* L158 states that the average of a large number of images is a solid color. This is incorrect, see for instance Figure 1 in **[1]**.\n* In Equation 5, the notation ${\\{ \\}}_{\\textnormal{batch}}$ is not introduced and it is not clear. It seems to contradict equation 6.\n* Equation 7 is not justified. Why would $f$ only depend on the difference between $x_t$ and $x_0$?\n\n**References:**\n\n**[1]** Torralba, Antonio, and Aude Oliva. \"Statistics of natural image categories.\" Network: computation in neural systems 14.3 (2003): 391.\n\n**W2/ Similar methods have already been presented in previous literature. The submission does not discuss these existing works, nor compare with them.**\n\n* The notions of “coupling” and “optimal transport” between data points and noise samples have already been presented in the previous literature. See for instance **[2]** and **[3]**.\n* To understand the significance of this submission, it would be necessary to discuss these similar works and highlight the differences if any, and compare with such methods if possible. \n* As such, I do not see any novelty in this work (If I miss any, the authors fail to highlight the difference by discussing related work).\n\n**References:**\n\n**[2]** Tong, Alexander, et al. \"Improving and generalizing flow-based generative models with minibatch optimal transport.\" arXiv Feb 2023 + TMLR + ICML Workshop \n\n**[3]** Pooladian, Aram-Alexandre, et al. \"Multisample flow matching: Straightening flows with minibatch couplings.\" arXiv April 2023 + ICML \n\n\n**W3/ The presentation, including the figures quality and writing, needs to be improved.**\n\n* The parallel with “Immiscible Fluids” does not bring a better understanding of the method, no physics equations are given. What should we understand from Figure 2a? Even if this is schematic, where would the noise and data distribution be in this image?\n* The quality of Figure 3 is very bad. The images for predicted noise are completely saturated.\n* The authors fail to show any ablation on the “quantization method” they claim. It is not clear if this is really necessary and the choices of fp8 and fp16 appear rather arbitrary.\n* Many sentences do not make sense to me\n    * L163: “T=20 means the layer of pure noise”\n    * L160: what are “rich image classes”?\n    * L162: “the predicted noises of different layers”\n* Many typos:\n    * L89 “its” for diffusion models\n    * text written in math font in equations\n    * L422: “our method accelerate”\n    * L471: “our immsicible diffusion work” (2 typos)\n    * Caption Figure 5: “can catch more details and more feature”\n* Several incoherences:\n    * L93: DDIM is 10 steps. <->  the authors experiment with DDIM but never take less than 20 steps.\n    * linear_sum_assignment from Scipy (algorithm 1) <-> Hungarian matching (text)\n    * Some generated images in Figures 8 and 9 are completely black.\n\n**W4/ The presented results are not convincing.**\n\n* All qualitative results presented in the main submission are cherry picked. The only non cherry picked results are shown in the appendix and the generated images, and the qualitative results seem very similar to me (no significant qualitative improvement).\n* The quantitative results are difficult to trust. There seems to be incoherences between the results:\n    * The “-0.98” in the third row of Table 5 is not coherent with the two first rows.\n    * The plots have different scales and are not clear (why does the plot for imagenet in Figure 4 starts at 20k steps and not 0?)\n    * The quantitative results mix consistency models and diffusion models in an unorganized way (eg, the columns of table 1 alternate between them), making it difficult to understand the results.\n* It is not clear why some datasets use the diffusion approach and some use the consistency approach. Did the authors experiment with both for each dataset and only report the best results?\n* The datasets and evaluation settings are not clearly presented. How many images are there in each dataset? Was the dataset split into training, validation and test sets? Are the reported FID scores computed on training, validation or test set? The number of sampling steps is not reported for all experiments (eg table 2). Only part of the quantitative results are shown (thus potentially cherry picked too, eg not quantitative results for Stable Diffusion)\n\n----\n**Changes following the rebuttal and authors-reviewers discussion:**\n\n**Reduced soundness from 2 to 1**: The scope of the paper has been significantly expanded with the rebuttal (conditional generation, flipped dimension OT, decreased noise levels). Some arguments in the paper do not hold anymore (eg, average of all images). Revised mathematical proof remains imprecise and incorrect.\n\n**Reduced contribution from 2 to 1**: Discussions following the rebuttal made me dive more deeply in the Conditional Flow Matching paper (https://arxiv.org/abs/2302.00482). Most, if not all, findings of the reviewed submission seem to be already known and analyzed more deeply in the Conditional Flow Matching paper, which additionally uses Flow Matching formalism (generalization of diffusion to more prior distributions than Gaussian Noise) and contains sound proofs. I do not see any new meaningful contribution for the Neurips community in the reviewed submission.\n\n**W1** The authors did not address this concern correctly. To expand on the fact math assumption (Equation 7) is incorrect, suppose the target distribution is $P(x_0=0.1) = P(x_0=-0.1) = 0.5$. As $t \\rightarrow +\\infty$, it is clear that, if the batch size is large,  $p(x_t|x_0=0.1)$ is approximately $2 \\times N(x_t; 0, 1)$ if $x_t>0$ and approximately  0 if $x_t <0$. This is clearly not in the form of Equation 7. The ratio also does not decrease with the norm of $x_t – x_0$, as $p(x_t=-0.1|x_0=0.1) \\approx 0$ while $p(x_t=1.1|x_0=0.1) \\approx 0.4$.\n\n**W2** The authors did not address this concern correctly. They included new experiments to claim their method does not fit into “Conditional Flow Matching”, but it seems that it still does (Conditional Flow Matching also seems to allow to choose the coupling via the joint distribution $q(x_0, x_1)$).\n\n**W3** The authors did not address this concern. The quality of Figure 3 is extremely low (all images are either extremely blurry or extremely saturated. It is possible the authors forgot something (dividing by 255?) when saving the images of the predicted noise, and that they upscaled the other images with interpolation or something). The authors did not comment on the incoherences/typos/sentences that need clarifications. The parallel with physics appears decorative without supporting physics equations. \n\n**W4** is not addressed correctly. Qualitative results show marginal improvement on cherry picked results only. Quantitative results are presented weirdly (eg, axes of the plot) with incoherences.\n\n**W5** The new experiments introduced in the rebuttal significantly extend beyond the original scope of the paper (e.g., introducing OT with flipped dimensions, decreasing noise levels, and including conditional generation). Arguments made in the paper do not hold in this expanded scope. For instance, the paper's claim that the initial denoising direction points to the average of all images, and is thus not meaningful. This is not true in the conditional generation case, as the initial denoising direction points to the average of images of the desired class. Additionally, the authors seem to say that in immiscible diffusion, any noise sample can still lead to any class (so, is it miscible?), which needs to be discussed/explained/investigated more.\n\n**Reduced overall rating from 3 to 2** Due to the (important) remaining concerns, the extent of the changes proposed in the rebuttal and discussion (would need additional reviews after changes are made), as well as the conflicts of the proposed changes with arguments contained in the original submission.\n\n**Increased confidence from 3 to 5** I participated in the discussion and dived deeper into the CFM paper (which I was initially not very familiar with). I am certain about the need to reject the paper in its current state, with the possibility for the authors to address these issues in a future submission.\n\n ----", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for some of the constructive feedback on our work. We would like to present following updates:\n\nFor W1, we have revised our mathematical proof part in rebuttal to reviewer-neka - W1, taking into consideration your constructive suggestions.\n\nFor W1, we compute the average of the images on the ImageNet dataset (1M images), the result is shown in Figure R1, demonstrating that the average of images does not contain much meaningful information, as we stated in the revised mathematical proof.\n\nFor W2, we have included an additional discussion on our difference to OT-family methods in the global response for your reference.\n\nFor L1, we have included a discussion of the influence of the batch size in rebuttal to review-neka-Q1 for your reference.\n\nFor L3, we have provided the performance of conditional generation of immiscible diffusion in the global response for your reference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and ACs,

We are grateful for your time and effort in reviewing our work. We are glad to see that our work is recognized as reasonable (R-h6mk), novel and interesting (R-nEKA, R-y1v3), simple and effective (R-h6mk, R-nEKA, R-y1v3). We are also encouraged to hear that our experiments are acknowledged to show robustness, effectiveness and efficiency (R-y1v3) and our writing is clean and concise (R-h6mk, R-y1v3). We sincerely appreciate the insightful comments and we address the concerns as below:

**Conditional Generation** We perform a class-conditional image generation with Stable Diffusion (SD) on ImageNet-1k with a batch size of 2048. To cater to text-to-image generation, we use class name as the text prompt for SD. Results in Fig. R2 shows that the FID for immiscible class-conditional SD is 20.90, which is 1.53 lower than that of SD baseline. Qualitative comparisons further prove such enhancements, which augment the effectiveness of immiscible diffusions into commonly-used conditional image generation. 

**Comparison to OT-CFM and related works** Thanks for pointing it out and we indeed miss this discussion in our submission. We will include them in our final version. Our motivation is different from OT-CFM [1], Multisample flow matching [2] and Approximated-OT [3],  and we target diffusion-based methods while most of these works are for flow-matching based methods. We point out that it is a coincidence that we share a similar algorithmic spirit but we can achieve our motivation -- immiscibility from an orthogonal way. The discussion is presented below: 

Theoretical Difference

OT-CFM is designed to “yielding straighter flows that can be integrated accurately in fewer neural network evaluations”[1], while immiscible diffusion aims to keep images immiscible in the noise space. OT focuses more on the straightness of the diffusion path, while immiscible diffusion focuses on the immiscibility of diffusion destinations.
Therefore, minimizing image-noise distance in a mini-batch is only one way to achieve immiscibility. As long as the image’s miscibility is limited, it falls in our motivation. Of course, image-noise OT is one way without adding crossings in the diffusion path.

Additional Experiments Splitting Two Proposals.

To validate the effectiveness of immiscible diffusion alone, we design a controlled variable experiment which does not qualify OT. We still use assignment, but compared to linear assignment calculating distances between images and noises, we flip the dimension of the noise for performing the calculation and assignment. For example, say that the image and the noise are 3072D:  OT-qualified assignment would calculate distance between corresponding dimensions, i.e.

Dist. = sigma(||x0_i - noise_i||2)

Our experiment changes this distance goal into:

Dist.’ = sigma(||x0_i - noise_(3072-i)||2)

By doing this, we still limit the miscibility. However, this does not qualify OT because the real distance is not optimized. In fact, the real distance reduced only <1% in this way.
Interestingly, when we perform the vanilla OT assignment and flip assignment, on DDIM with 50 sampling steps, batch size of 256, no additional image normalization, they show similar performances after initial steps, as shown in Fig. R3, with OT one performs slightly better. This proves that the immiscible effect itself can help improve the diffusion performance. For their performance difference, note that the flip assignment causes crossings in diffusion paths. As shown in Fig. R4, if we have 4 images x0 to x3 and 4 noises n0 to n3, when we flip x and y axis, a path crossing shown in red circle would happen, which adversely affects the FID.

[1] Tong et al. "Improving and generalizing flow-based generative models with minibatch optimal transport."

[2] Pooladian et al. "Multisample flow matching: Straightening flows with minibatch couplings."

[3] Kim et al. “Improving Diffusion-Based Generative Models via Approximated Optimal Transport”

Domination of Immiscible Effect to OT in Noise Assignment

We further hope to understand either immiscible effect or the OT dominates the performance enhancements in image-noise assignment. So we do some stats: As shown in Tab. 3, the distance reduction after assignment is only 2%. We further calculated the stddev of distances in a batch, which is ~10%. Therefore, the distance reduction is so little that, as suggested by R-y1v3, is hard to believe to become the reason for performance enhancement. On the other hand, we calculated the relation between the distance between images and their corresponding noise points, shown in Fig. R5. We see that after assignment, those images far away are assigned to noise points far away, which constitutes the concept of immiscibility. Our results in Fig. 3 also  show that the vanilla diffusion model suffers from the miscible problem while immiscible diffusion significantly reduces it. In conclusion, we believe immiscible diffusion dominates the performance enhancement in image-noise assignment.

Immiscible Diffusion with Image Projection

We notice that simple image projection, i.e. multiple each image by a factor like 2.0 or 4.0, can also perfectly achieve this goal as long as the factor is not too large to wipe out the image in diffusion.

During training, We multiply the images in DDIM, whose original stddev is 0.5, by factors of 2 and 4 to let its stddev be 1.0 and 2.0. Note that all images are centralized so there is no impact on their means. In Fig. R6, we note that after multiplications, the FID significantly improves, which suggests the effectiveness of immiscible diffusion from another angle. Adding assignment can generally further increase the training speed, but the effect would be weakened when the factor is large, as the immiscible problem is significantly solved by the factoring. This offers another way of immiscible diffusion, which will be added in the final version.

Meta-review: The paper introduces "Immiscible Diffusion," a method to accelerate diffusion model training by refining noise-data mapping. Inspired by the concept of immiscibility in physics, the method assigns specific noise to images to simplify the optimization process, leading to faster training with maintained image quality. The reviewers agree that this approach requires only a minor code change to achieve significant speed improvements, and it speeds up training while maintaining high-quality image generation. On the other hand, the reviewers also raised the concerns that the paper still does not provide very strong comparisons with existing methods and the experimental validation is not robust enough. After carefully reviewing the paper, reviewer discussions, and rebuttals, the AC recommends accepting the paper.
Decision: Accept (poster)