Immiscible Diffusion: Accelerating Diffusion Training with Noise Assignment

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:48:33

Abstract: In this paper, we point out that suboptimal noise-data mapping leads to slow training of diffusion models. During diffusion training, current methods diffuse each image across the entire noise space, resulting in a mixture of all images at every point in the noise layer. We emphasize that this random mixture of noise-data mapping complicates the optimization of the denoising function in diffusion models. Drawing inspiration from the immiscibility phenomenon in physics, we propose *Immiscible Diffusion*, a simple and effective method to improve the random mixture of noise-data mapping. In physics, miscibility can vary according to various intermolecular forces. Thus, immiscibility means that the mixing of molecular sources is distinguishable. Inspired by this concept, we propose an assignment-then-diffusion training strategy to achieve *Immiscible Diffusion*. As one example, prior to diffusing the image data into noise, we assign diffusion target noise for the image data by minimizing the total image-noise pair distance in a mini-batch. The assignment functions analogously to external forces to expel the diffuse-able areas of images, thus mitigating the inherent difficulties in diffusion training. Our approach is remarkably simple, requiring only *one line of code* to restrict the diffuse-able area for each image while preserving the Gaussian distribution of noise. In this way, each image is preferably projected to nearby noise. To address the high complexity of the assignment algorithm, we employ a quantized assignment strategy, which significantly reduces the computational overhead to a negligible level (e.g. 22.8ms for a large batch size of 1024 on an A6000). Experiments demonstrate that our method can achieve up to 3x faster training for unconditional Consistency Models on the CIFAR dataset, as well as for DDIM and Stable Diffusion on CelebA and ImageNet dataset, and in class-conditional training and fine-tuning. In addition, we conducted a thorough analysis that sheds light on how it improves diffusion training speed while improving fidelity. The code is available at https://yhli123.github.io/immiscible-diffusion

Main paper: 
 introduction: 
 Diffusion models have made impressive progress in image generation by framing the process as a phase of denoising random Gaussian noise into the final image. Despite the advancements, training a diffusion model is resource intensive. For example, even in the primary image dataset CIFAR-10, the representative few-step diffusion model, Consistency Model [47] , requires training for 10 days on 4 A6000 GPUs to reach a desired FID score of around 10. Similarly, with fewer model parameters, multiple-step diffusion model DDIM [44] still requires 24 hours on an A5000 GPU on the CIFAR-10 dataset. Although recent remarkable achievements in accelerating the inference of diffusion models [19, 33, 47, 28, 30, 31] have been accomplished, the inefficiency of diffusion training remains a significant bottleneck, hindering the iterative development of vision generative AI.Previous methods for improving diffusion training have focused on various strategies, such as balancing the impact of activation layers and neural weights [16] , modifying hyperparameters and design choices [46] , and leveraging patchifying strategies [53] etc. Specifically, Karras et al. [16] modifies the activation magnitude, neural weight standardization, and group normalization, achieving significant acceleration in diffusion training. Besides, previous work [46] proposes a customized method for the Consistency Model to improve the performance and diffusion training. Our method is orthogonal to these previous methods. We got inspired by the Immiscible Diffusion in physics. As illustrated in Fig. 2 (a) left, miscible particles tightly jumble together after the diffusion process, making it difficult to separate them individually during the denoising phase. However, when the particles are rendered immiscible, they can still achieve a similar overall distribution while remaining clearly distinguishable (see Fig. 2 (a) right). This insight inspires our strategy for improving the disentanglement of diffused data.We draw an analogy from the phenomenon of Immiscible Diffusion and relate the distribution of image data to the behavior of particles discussed above. In traditional diffusion processes, each image can be diffused to any point in the noise space, and conversely, each point in the noise space can be denoised to any source image, as illustrated in the left image of Fig. 2 (b) . We hypothesize that the jumbled image-noise mapping creates a miscible diffusion effect and makes the optimization of the diffusion model difficult. Inspired by the Immiscible Diffusion, we are motivated to make the mixed diffusion phase distinguishable.We propose one simple Immiscible Diffusion method. Note that we still sample Gaussian noise but perform a batch-wise assignment of noise to each image based on the distance between them during training. This approach ensures that each image is only diffused to surrounding areas while maintaining the overall Gaussian distribution of all noises. This technique was also used in flow matching-based methods [36, 50] for optimizing the image-noise flow. Nevertheless, we find that the image-noise distance is reduced by only ~2% after assignment, as provided in Part 4.3. This motivates us to ask which factors dominate the performance improvement? To investigate it, we propose another method that qualifies Immiscible Diffusion in Part 4.3. Experiments show that the training efficiency improvement is comparable to the function of flow optimization, demonstrating that immiscibility is the dominant factor. However, technically, to achieve immiscibility with the image data-noise assignment has an O(N 2 logN )-O(N 3 ) complexity. This introduces significant overhead during training, especially for large-scale training with huge batch sizes and high-resolution images. To address this, we employ a novel quantization method during assignment. We quantize the noise and image data into low-precision formats (e.g., 16-bit) during conducting the assignment algorithm. We highlight that this assignment operation only involves one line of code, and is performed only during the training phase without modifying the model architecture, the noise scheduler, the sampler, or the method of inference.We conduct extensive experiments on three common modes: unconditional, conditional, and finetuning on three diffusion baselines: Consistency Models, DDIM and Stable Diffusion and three datasets: CIFAR-10, CelebA and ImageNet datasets. Results show that our proposed method significantly improves the training efficiency in all experiments. Specifically, we achieve 3x training efficiency for the CIFAR-10 dataset with immiscible unconditional Consistency Model compared to the original Consistency Model. Furthermore, we show that the FID is even lower with our method used, confirming the fidelity of our generated images. We also provide images generated from models trained with vanilla and immiscible models experiencing the same training steps, where we see that those from immiscible models are much more complete and clearer, further proving the training efficiency enhancement resulted from the Immiscible Diffusion. Examples are shown on the right of Fig. 1 . Deeper analysis shows that our method, although with only one line of code and involving ~2% image-noise datapoint distance changes, achieves all the benefits above in negligible running time.To sum up, our contributions are as follows:• We clearly and specifically identify the miscibility issue in noisy diffusion steps, which leads to slow convergence of diffusion training. • To tackle the miscibility issue, we propose a simple and effective method, Immiscible Diffusion, a strategy that can only requires one-line of code, to improve training efficiency for diffusion training. • Experiments demonstrate the effectiveness of our proposed method on several popular diffusion models across multiple datasets, and across unconditional, conditional, and fine-tuning tasks. In addition, we conducted thorough analyses and ablation studies to elucidate how our method works and dominates the effectiveness.2 Related Work 
diffusion model with efficient inference: 
 Diffusion models [48, 12, 39, 35] have been attracting huge attentions because of their high-fidelity image and video generation [11, 13, 34] , data-efficient perception [49, 32, 55] , and even representation abilities for robotics [3, 37, 1] . However, slow inference is one of the key bottlenecks for diffusion models. To address this issue, various approaches have been proposed. For instance, techniques such as DDIM [44] have reduced the number of denoising steps from 1000 to 10, significantly speeding up the process. Furthermore, the introduction of Consistency Models [47] and LCM [33] ,which utilize the properties of self-consistency, enables denoising in as few as 1-4 steps, further enhancing the generation speed of diffusion models. Subsequently, the development of SD-turbo [42] , which leverages GAN [8] loss for high-definition image generation in a single step, has occurred. The Consistency Trajectory Models [18, 38] improve the generation quality of Consistency Models and accelerate research on efficient inference for diffusion models. Additionally, beyond reducing denoising steps, efforts to improve the inference efficiency of single function evaluation are being explored in various ways, including model quantization [26] and partitioning the generative components [25] . Moreover, StreamDiffusion [19] streamlines denoising steps to achieve real-time inference at the pipeline level optimization. The improvement of the inference efficiency significantly pushes forward real applications based on diffusion models. Yet accelerating diffusion training is still under-explored.Improving the training efficiency of diffusion models is crucial. Various strategies have been proposed, including architectural modifications [16] , approximating the diffusion phase with flow [28] , and designing parameter choices [46] etc. Specifically, in [16] , the authors discover that the magnitude of activation and the magnitude of neural weights significantly impact the training dynamics of diffusion models. They propose adjusting the activation magnitude and standardizing neural weights, as well as modifying the normalization layers to make diffusion training in a more smooth dynamic. Besides, Song et al. [46] aims to enhance the training efficiency of Consistency Models through customized design choices, significantly improving both training speed and fidelity. Furthermore, leveraging approximation strategies based on ODE assumptions [28] improves not only inference efficiency but also training efficiency since diffusion trajectories are prone to deterministic. Beyond improving diffusion training with either architecture adjustment or selection parameters, Wang et al. [53] introduce a novel patch strategy to control the ease of diffusion training, achieving both training and data efficiency. Gleichzeitig, Wang et al. [52] notices that the denoising of some noisy diffusion steps contains little information and is too easy to learn, so focusing more on other steps would significantly improve training efficiency. Our method differs from previous works by clearly highlighting an under-explored problem: the miscibility problem of image data in the noise space, which plays a crucial role in training diffusion models. Our proposed Immiscible Diffusion is extremely simple yet significantly improves training efficiency.In ODE-based methods such as flow matching [27] , straightening the flow with optimal transport (OT) has been used as a tool to improve the generation performance. Specifically, optimizing image-noise transport in a batch [36, 50] has been found to be an effective way to improve performance. Training efficiency improvement was also observed [36, 50] , and explained or posited with reduction of the variance of the training goal. However, the standard deviation reduction is only ~4% in [36] . Moving forward, [24] pointed out the curvature problem in the ODE paths caused by the collapse of the reverse trajectories in the average direction. However, they replaced Gaussian noise sampling with a VAE encoder-style structure to eliminate such curvatures, which destroys the strict Gaussian distribution in the noise space. Concurrent to our work, [17] applies batch-wise OT to diffusion models to achieve better FIDs, making posits in curvature reduction for the enhancement. Several methods were proposed to improve the speeds and effectiveness of OT, such as pre-training with PF-ODE [54] , using Schrodinger bridging [5] , utilizing conditional Wasserstein distance [2] and generator-induced coupling in Consistency Models [14] . However, we are the first to emphasize that the dominant reason for training efficiency enhancement is the miscibility problem in noisy layers, which we prove in Part 4.3. Furthermore, most of previous methods are for unconditional flow matching methods only. Our work demonstrates the effectiveness in multiple diffusion models, datasets, conditional generation, and fine-tuning experiments.3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Our current assignment method is quite straightforward.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
physics intuition: 
 Diffusion models mimic the reverse thermodynamic diffusion phenomenon [43] to ease the denoising process. However, when the sources are miscible, as shown in the left of Fig. 2 (a), they end up messily mixed. Predicting the reversal process from such a random mixture encounters significant difficulties, and unfortunately, this is a problem diffusion model always facing during denoising.However, we notice that mixing can also be organized when sources are immiscible. Under that circumstance, the sources would take different continuous areas after diffusion, while the whole diffused area remains the same, as shown in the right image of Fig. 2 (a). Thereafter, the reversal process becomes smooth. Inspired by Immiscible Diffusion, we then introduce it to the diffusion models, with the aim of making the optimization easier and to achieve a higher training efficiency.Similar to the physics phenomenon, we find that for diffusion models, any images are diffused to every corner of the noise space, which also means that each noise point can go back to any image. This would cause the denoising model to be confused on which image to go to, as shown in the left of Fig. 2 (b) .Mimicing the immiscible phenomenon in physics, we hope to design similar processes where each noise point is only matched to limited images, so as to avoid the confusion for the denoising model. However, the noise space must remain Gaussian to help the sampling process. Therefore, we propose our first implementation way of Immiscible Diffusion, which assigns the batch of noises to the batch of images during training according to the image-noise distance in their shared space. We minimize the total distance of the image-noise pairs in a batch during assignment. Here we use the L2 distance for assignment, which is ablated in Part A.1.3 in the Supplemental Materials. After assignment, the noise is still Gaussian, while each noise is assigned to nearer images like what happens in the immiscibility phenomenon, which significantly eases the difficulties for the denoising. Fig. 2 (b ) right illustrates an extreme example of the Immiscible Diffusion, where the noise corresponding to each image is relatively separated.For implementation, all we need to do is to perform a linear assignment [21] between the batch of images and noises according to their distances. This can be achieved in only one line of code using Scipy [51] . The algorithm is shown below:Algorithm 1 Batch-wise Image-Noise Assignment 1: Input: Image batch x b , random noise batch n rand,b , sampled diffusion steps t b and diffusion schedule α 2: assign_mat ← scipy.optimize.linear_sum_assignment(dist(x b , n rand,b ))3: x t,b ← √ α t b x b + √ 1 -α t b • n rand,b [assign_mat] 4: Output: Diffused image batch x t,bWhile linear assignment qualifies Immiscible Diffusion, Immiscible Diffusion can be achieved with multiple paths. In part 4.3, we ablate the linear assignment by letting it remain immiscible while disqualifying optimal transport, proving that immiscibility plays the dominant role in performance improvement.In this section, we mathematically elucidate the denoising difficulty for traditional diffusion models based on DDIM [44, 12] and how our proposed Immiscible Diffusion reduces such difficulties.In DDIM, we know for any image data-point x 0 , when it is diffused to the last diffusion step T, i.e. t = T , the image is sufficiently wiped out and nearly only gaussian noise is remaining, therefore,EQUATIONWhere q refers to the Utilizing Bayes' Rules and Equation 1, we can find that for a specific x T :EQUATIONwhich indicates that the distributions of the corresponding images for any noise data-point are the same as the distribution of all images. Referring to [45] , τ = S represents the layer denoising from the pure noise. We show that while the two sampled noises are similar, the denoised image of immiscible DDIM significantly outperforms that of the traditional one, generating an overall reasonable image. The reason behind this is traditional methods cannot successfully predict noises at noisy layers.The simplified training objective in DDIM [44, 12] is the added noise ϵ(x t , t) at each diffusion step t and at the point x t . However, we find that for a specific point x T in the noise space at diffusion step T ,ϵ(x T , T ) = ax 0 + bx T = x0 (ax 0 + bx T )p(x 0 | x T ) = a x0 x 0 p(x 0 | x T ) + bx T x0 p(x 0 | x T ) = a x0 x 0 p(x 0 ) + bx T = a x0 + bx t (3) where a = - √ αt √ 1-αt and b = 1 √1-αt are constants, and x0 is an average of images in the dataset. When the number of images is large enough, x0 contains little meaningful information.As shown in Fig. 3 (a), we study the predicted noises of different denoising layers in a DDIM model with total inference steps of 20, to illustrate our discovery. Here we refer to [45] , specifying the sampling step τ = S as the layer that denoises the pure noise, which is equal to the diffusing step t = T . We can see that the predicted noise for DDIM at τ = S (equal to t = T in diffusing) does not provide much useful information, while denoising with this "noise" does not provide any distinguishable image, which all support our hypothesis for the difficulty in denoising when τ in sampling (or t in diffusing) is large. Similar observations are also shown in the concurrent work [52] . However, in our Immiscible Diffusion, while for each batch, we still haveEQUATIONfor each specific data point x T or x 0 , the conditional noise distribution does not follow the Gaussian distribution because of the batch-wise noise assignment p(x T | x 0 ) ̸ = N (x T ; 0, I).(5) Instead of the Gaussian distribution, we assume that the predicted noise with noise assignment has a distribution described as followsEQUATIONwhere f is a function denoting the influence of assignment on the conditional distribution of x T , and bs is the training batch size. Apparently, according to the definition of linear assignment problem [21] , f decreases when x T -x 0 increases its norm, specifically the L2 norm as in our default setting. Therefore, from Equation 2 and 6, we haveEQUATIONwhich means that for a specific noise data-point, the possibility of denoising it to the nearby image data-point would be higher than to a far-away image.For the noise prediction task, we see thatEQUATIONwhere x 0 f (x T -x 0 , ...) is the weighted average of x 0 with more weights on image data-points closer to the noisy data-point x T itself. Therefore, the noise predicted would lead to the average of nearby image data-points, which makes more senses than pointing to a constant. Indeed, in Fig. 3 , we see that even for the pure noise layer, immiscible DDIM can predict the noise effectively pointing to the shape of the horse image, and the prediction in one step by subtracting the predicted noise shows the outline of the horse correctly.The assignment problem has been studied extensively for decades [6, 22, 7] . In this paper, we use the Hungarian algorithm [22] as our main assignment method. However, Hungarian matching has high complexity with O(N 3 ), which drastically slow down the training especially when we have high-dimensional image data (e.g., even using the mini image data 32 × 32 × 3 = 3072). To mitigate this issue, we make a novel use of quantization for image data and noise, that is, we quantize the f p32 image and noise data to f p16 to carry out the assignment, while maintaining the same precision input to diffusion models. This trick significantly reduces the overhead to a negligible level.To efficiently perform Immiscible Diffusion when running on multiple GPUs, we assign the imagenoise distance matrix computation to each process, and then gather them to execute the assignment. This is particularly important as high resolutions and large batch sizes are frequently required in applications.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Our method is just one line of code.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiment settings: 
 To elaborate the performance of Immiscible Diffusion, we utilize the proposed method on Consistency Models [47] , DDIM [45] and Stable Diffusion [41] , and using CIFAR-10 [20] , CelebA [29] , tiny, random picked 10% and the full ImageNet [4] datasets due to the limitation of computation resource.The training hyperparameters are shown in Tab. 1. Unspecified hyperparameters are taken the same as those in their baseline methods' original papers. For evaluations, we compare the results generated by our Immiscible Diffusion method and the baseline using both the quantitative evaluation metric FID [10] and qualitative assessments.Note that for Consistency Models, we use the single-step generation consistency training. For DDIM, we add no noise during the sampling and use linear scheduling for picking sampling steps. For Stable Diffusion, we directly use the implementation from Diffusers of Huggingface team [40] . For fine-tuning, we use Stable Diffusion v1.4 [40] as the pre-trained model.Unconditional generation with Consistency Models: In Fig. In the main experiment, we observe that our method on top of the Consistency Model is effective across the datasets varying from different data sizes and resolutions. Indeed, the Consistency Model is a few-step diffusion model, and our proposed Immiscible Diffusion especially works on improving the denoising effect when the noise level is high, as shown in Fig. 3 . The improvement of the training efficiency on such a few-step diffusion model further validates our findings.One characteristic of the Consistency Model is that it approximates the SDE-diffusion model with the ODE approximation. Thus, the original image-noise mapping is highly jumbled together since it is highly possible that closed image data points are diffused to distant noise points. Our Immiscible Diffusion improves this issue by adjusting the trajectories of image-noise mapping and making them more distinguishable. More baselines: To show the generalization of Immiscible Diffusion for more baselines, we further conduct experiments on two baselines: DDIM [45] and Stable Diffusion [39] on the CIFAR-10 and the randomly picked 10% ImageNet dataset, respectively. As shown in Fig. 5 , and detailed in Tab. 4 in Supplemental Materials, we find that our immiscible DDIM significantly improves the training speed and the FID compared to those of the baseline DDIM on the CIFAR-10 dataset, and the improvement is more significant when the sampling step is lower. This demonstrates the effectiveness of our proposed method works beyond the Consistency Model and can be generalized to more few-step denoising models. We also provide a discussion in Part A.1.4, showing that the effectiveness of Immiscible Diffusion can persist in a wide range of batch sizes. To further evaluate generalizability on the popular baseline, Stable Diffusion [39] , we also conduct unconditional generation experiments on the ImageNet dataset. We observe that immiscible Stable Diffusion and baseline Stable Diffusion achieve similar FID without significant gap, yet our immiscible Stable Diffusion is able to generate much higher quality images from a subjective human judgement. For example, Fig. 14 in the Supplemental Materials shows that our proposed method generates significantly clearer images compared to the baseline. More visualization without any cherry-picking can be seen in Fig. 15 in the supplementary materials. We indicate that even though FID is the primary metric and is remarkably successful, the metric is known to sometimes disagree with human judgement [23] .ImageNet dataset with Stable Diffusion [41] , to explore the performance of Immiscible Diffusion in conditional generations. Results are shown in Fig. 6 (a), where we observe that in 20k training steps, the FID for immiscible class-conditional Stable Diffusion is 16.43, which is 1.49 lower than our Stable Diffusion baseline. We further confirm such improvements on CMMD [15] , where the immiscible and vanilla models get 1.385 and 1.436 respectively. Additional evaluation on CLIPScore [9] shows that both the immiscible and the baseline models generate images with CLIPScores of 28.55, with a standard deviation of 0.01 and 0.02 respectively, indicating that Immiscible Diffusion does not hurt the image-prompt correspondence in complicated ImageNet dataset. Qualitative comparisons in Fig. 16 in Supplemental Materials further prove such performance enhancements, which augment the effectiveness of Immiscible Diffusions into more commonly-used conditional generations.Fine-tuning: Our Immiscible Diffusion can also be applied to enhance the fine-tuning process where numerous applications fall in. We fine-tune the stable diffusion v1.4 model [41] on ImageNet dataset, finding that immiscible fine-tuning achieves an FID of 10.28 compared to 11.45 for vanilla fine-tuning with 5k training steps. Detailed results are shown in Fig. 6 . This further broadens the application of Immiscible Diffusion. To further understand the proposed Immiscible Diffusion method, we delve into several key questions to ablate our approach:How much does image-noise distance reduce in the assignment? Tab. 2 shows the reduction in distance after the image-noise assignment. We find that the L2 distance only reduces by about 2%, with a slight increase observed at higher batch sizes. However, as shown in Fig. 3 , the assignment is sufficient to effectively activate denoising at high noise levels, significantly boosting training efficiency, even though the distance change is low. We attribute the low distance reduction rate after the assignment to the extremely high dimensionality (3072 for each image of the CIFAR-10 dataset) of the image and noise space.How much time does image-noise assignment cost? In Tab. 2, we indicate that our assignment method does not introduce significant extra overhead due to our utilization of quantized assignment in our practical implementation. Even for a large batch size per GPU of 1024, our algorithm only brings in an additional 22.8 ms, demonstrating the potential of utilization for future applications.Immiscible and OT: who dominates the training efficiency enhancement? Our Immiscible Diffusion claims to enhance training efficiency by improving miscibility in noisy diffusion steps. However, the method we take towards Immiscible Diffusion, i.e. linear assignment between image and noise, also serve as a roughly approximate OT between image and noise, which might intuitively benefit the diffusion through straightening the diffusion paths [36, 50] . However, the previous section shows that the image-noise distance is only reduced by ~2%, motivating us to ask if OT is really the dominant factor?To answer this question, we ablate these two factors: OT and immiscibility. We design a non-OT Immiscible Diffusion experiment which keeps the immiscible property while not involving the OT. This is achieved by assigning images to the flipped noise whose all dimensions are reversed, while using the original noise during diffusion. In such a way, the image-noise pair no longer follows OT, but still qualifies the Immiscible Diffusion -i.e. images are still assigned to a limited area. Interestingly, we observe that the non-OT Immiscible Diffusion can still accelerate and enhance the diffusion training, which is nearly comparable to the OT Immiscible Diffusion in final stages, as shown in in Fig. 7 . Considering that the non-OT version introduce miscibility in middle diffusion layers, which we posit for its difference to OT version, we conclude that Immiscible Diffusion is dominant in enhancing the diffusion model's performance, compared to the benefits from OT.  
conclusion, limitations, and future work: 
 Inspired by the immiscibility phenomenon in physics, we introduce Immiscible Diffusion, a method to improve image-noise mapping to accelerate diffusion training. Specifically, Immiscible Diffusion is an assignment-then-diffusion strategy. One way of it is to minimize the image-noise pair distance within a mini-batch so that each image is diffused to nearby noise areas. This simple approach requires only one line of code and includes a quantized-assignment strategy to reduce computational overhead.Experiments show our Immiscible Diffusion approach speeds up Consistency Model's training by approximately 3x on the CIFAR-10 dataset, 1.3x on the CelebA dataset, and 1.2x on the tiny-ImageNet dataset, as well as in conditional generation and fine-tuning on Stable Diffusion. Thus, we show that Immiscible Diffusion can generalize to across datasets, baselines and tasks. Further analysis is provided to explain how this works.Limitation. The assignment strategy is one straightforward way for Immiscible Diffusion, but not necessarily optimal. Due to the limited computational resources, our experiments are mainly conducted on small-scale datasets, so we lack the validation on larger-scale datasets such as LAION.In future work, we will improve the assignment strategy to cater to practical utilization of conditional generation such as accelerating the general text-to-image or text-to-video diffusion training.Broader impact. With the increased use of diffusion models for image and video generation, the training of diffusion models is certain to become an increasing portion of data center workloads. Moreover, training time is a significant bottleneck in model development. Our proposed method significantly improves the efficiency of diffusion model training. We believe that our method has the potential to accelerate progress and reduce the cost of development in this field. We use the L2 norm for our experiments. However, we note that the L2 norm may face more challenges in distance evaluation in high-dimensional spaces compared to the L1 norm. Therefore, we compare the performance of immiscible DDIMs using assignments based on the L1 and L2 norms. The results, as illustrated in Tab. 5, show that using the L2 norm provides better performance than the L1 norm. The effectiveness of image-noise assignment can intuitively rely on the batch size. Therefore, we perform a comparison to see the effectiveness of Immiscible Diffusion on DDIM across a selected range of batch sizes, whose result is shown in Fig. 8 . We observe that while larger batch sizes consistently accelerate the training as expected, its training efficiency enhancement is not as large as that from Immiscible Diffusion. Immiscible Diffusions continously improve the training efficiency and the performance in the whole selected range of batch sizes. A.2.4 Generated images from immiscible and baseline stable diffusion models trained unconditionally on 10% ImageNet for 70k steps. A.2.5 Generated images from immiscible and baseline stable diffusion models trained unconditionally on 10% ImageNet Dataset for 70k steps without cherry-picking. Justification: Please refer to our abstract and conclusion. We propose an extremely simple method inspired by physical phenomenon. With just one line of code, our method accelerate the diffusion training by a large margin. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Besides the Immiscible Diffusion part, all training and inference are same to baseline.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We use commonly used FID.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: Most of our experiments are conducted on A6000 GPUs and some on A800 GPUs.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: See conclusion part Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We do not have. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: We do not use. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
: 
 noise for the image data by minimizing the total image-noise pair distance in a minibatch. The assignment functions analogously to external forces to expel the diffuseable areas of images, thus mitigating the inherent difficulties in diffusion training. Our approach is remarkably simple, requiring only one line of code to restrict the diffuse-able area for each image while preserving the Gaussian distribution of noise. In this way, each image is preferably projected to nearby noise. To address the high complexity of the assignment algorithm, we employ a quantized assignment strategy, which significantly reduces the computational overhead to a negligible level (e.g. 22 .8ms for a large batch size of 1024 on an A6000). Experiments demonstrate that our method can achieve up to 3x faster training for unconditional Consistency Models on the CIFAR dataset, as well as for DDIM and Stable Diffusion on CelebA and ImageNet dataset, and in class-conditional training and fine-tuning. In addition, we conducted a thorough analysis that sheds light on how it improves diffusion training speed while improving fidelity. The code is available at https://yhli123.github.io/immiscible-diffusion 
 
 Appendix: • The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We did.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper introduces "Immiscible Diffusion," a method to accelerate diffusion model training by refining noise-data mapping. Inspired by the concept of immiscibility in physics, the method assigns specific noise to images to simplify the optimization process, leading to faster training with maintained image quality. The reviewers agree that this approach requires only a minor code change to achieve significant speed improvements, and it speeds up training while maintaining high-quality image generation. On the other hand, the reviewers also raised the concerns that the paper still does not provide very strong comparisons with existing methods and the experimental validation is not robust enough. After carefully reviewing the paper, reviewer discussions, and rebuttals, the AC recommends accepting the paper.

Reviews: review: paper_summary: paper_summary: This paper shows that the current diffusion training strategy diffuses each image into the entire noise space, making it difficult to optimize the model and thus slow to converge. Inspired by the fact that miscibility can be changed according to various intermolecular forces in physics, this paper proposes Immiscible Diffusion, a simple and effective method that accelerates diffusion training by pre-assigning noises in each mini-batch before standard training, which can be implemented in just one line code. This assignment operation can be accelerated by fp16/fp8, which greatly improves training efficiency while avoiding algorithm complexity and is applicable to various baselines, including Consistency models, DDIM and Stable Diffusion.
strength_weakness: strength_weakness: 1. The noise assignment strategy proposed in this paper essentially introduces a priori assumption into the diffusion training process: different images should correspond to different points in the Gaussian distribution. This prior reduces the difficulty of diffusion training, but also brings concerns about practicality and generalization:
    - Practicality: The datasets for unconditioned generation used in this paper are all based on image classification (CIFAR/CelebA/ImagenNet), so these data distributions themselves already meet the above prior assumptions (each class is a kind of image distribution). However, for image datasets with more complex distributions, such as LAION, whether the proposed assignment strategy can still be used is an uncertain question, and this paper also lacks relevant experiments. You can demonstrate that this is not a problem by training unconditional generation experiments on a subset of LAION.
    - Generalization: Since images with the same data distribution are assigned to a specific noise interval during training, this may cause the model to lose the ability that denoising each point in the noise space into any source image during inference, thereby damaging the generalization of the generated images. However, the paper lacks experiments to evaluate the generalization of generated images.

2. Considering that all the unconditional generation results in the paper are trained on classification datasets, is the distribution of noise assigned by the proposed method strongly correlated with the category? Intuitively, after the proposed noise assignment, images of the same category should have closer noise in current batch.
questions: questions: Please refer to the Weaknesses
limitations: limitations: NA
paper_summary: paper_summary: The authors propose an approach to mitigate the random correspondence of noise-data mapping in vanilla diffusion models. They first assign target noise by minimizing the total image-noise pair distance within a mini-batch, and then diffuse the data into noise. Experimental results seem to demonstrate the potential of this approach. However, some of the arguments are mathematically sloppy and not well testified.
strength_weakness: strength_weakness: 1. The way of presentation is not reader friendly. I suggest to review DDIM and define its notations first, explain your motivation from DDIM, and show your algorithm/solution.  

2. No strong evidence showing your claim that vanilla diffusion models are miscible. Any toy examples to support it?

3. The argument from L155-L161 is sloppy and hard to understand the argument. For instance, Eq. (2) holds in what sense (density? certain divergence?). Indeed, I believe it will not hold for most diffusion models.

4. More justification and rationale is needed to the assumption Eq. (7). Do authors suppose that should hold for all $t$? If so, I do not think it is a correct assumption.
questions: questions: 1. Is the proposed method, which does in-batch reassignment, sensitive to batch size?

2. Is the proposed algorithm applied to all $t_b$? In theory, all the arguments that authors make may only hold for large timesteps. Thus, the proposed method should work effectively even if it is applied to large diffusion time. It will be nice to see experimental results related to this.

3. Does the proposed method work effectively with fine-tuning? 

4. Even though authors propose to quantize to avoid computation bottleneck, by how much the batch-wise image-noise assignment increases the runtime of training?
limitations: limitations: Yes, authors have discussed potential limitations.
paper_summary: paper_summary: This paper is motivated by the miscible phenomenon of physics and transfers it into the diffusion training process, which is very interesting. The author proposes a noise-assigned strategy based on distance, which is simple but powerful for faster diffusion model training. The experiment and the theoretical analysis validate the core idea of the paper and show good results.
strength_weakness: strength_weakness: 1. In my view, grouping the proposed noise assignment method is similar to fixing and memorizing the relationship with noise and corresponding image, limiting the generation diversity of the diffusion model. However, the experiment shows that the FID could further be reduced, Hoping that the author can explain this phenomenon.
2. As shown in the experiment, the noise distance change is only 2%, why such a minimal change could bring remarkable training speed up? The author should do a theory analysis in depth. 
3. To validate my concern, please provide two experiments i) providing the result generated by a noise and its minimal change variant (only 10 pixel value changes, etc). ii) providing the result generated by a certain noise with different seeds.
4. I’m curious about the scaling law phenomenon. As shown in stable diffusion, training the diffusion model with large image data could bring surprising topological ability. However, the way the author trains the diffusion model, the noise seems to connect with certain images (same class, etc.) which will limit the generation diversity for more data. I know this experiment requires more computational resources, the author could only provide the theory analysis of the scaling law ability of the proposed method.
questions: questions: See the weakness.
limitations: limitations: Yes.
paper_summary: paper_summary: During training of Diffusion Models, images are corrupted with standard Gaussian noise and the Diffusion Model is tasked to denoise the corrupted images. The Gaussian noise used during training is often sampled independently of the clean image.

This submission presents the following "Immiscible Diffusion" method: within each mini-batch, we should reassign the pairs of images and noise samples, so that each image is corrupted with a ‘closer’ noise sample.

The submission claims "Immiscible Diffusion" is faster to train to reach the same FID threshold, with quantitative evaluation, and "can catch more details and more feature of objects" with some cherry-picked qualitative results.
strength_weakness: strength_weakness: The main weaknesses and reasons to reject the paper are the following:

**W1/** The notations and mathematical derivations are imprecise, unnecessarily complex, and potentially incorrect.

**W2/** Similar methods have already been presented in previous literature. The submission does not discuss these existing works, nor compare with them.

**W3/** The presentation, including the figures quality and writing, needs to be improved.

**W4/** The presented results are not convincing.

**More details:**

**W1/ The notations and mathematical derivations are imprecise, unnecessarily complex, and potentially incorrect.**

* The Equations in the Section 3.3 are very hard to understand, and I am not sure they are necessary as they make understanding more difficult. The section could simply be rephrased as follows without the need to introduce complex and potentially incorrect equations: When the noise samples and data points are sampled independently, the optimal (in MSE sense) denoising prediction from a completely noisy observation is the mean of the data distribution. If we additionally know that data points are only corrupted with "close" noise, then the optimal denoising prediction from a completely noisy observation is, intuitively, the average of only "close" data points, which can allow to converge faster to a data point at generation time.
* The notation "When $t → \infty $, $p(x_t|x_0) = N(0, I)$" in Equation 1 is not precise (equality? convergence?).
* ​​L155 states that Equation 2 indicates "the distributions of the denoised images for any noise data-point are the same, which is equal to the distribution of the image data", but Equation 2 does not contain any "denoised images", only clean images $x_0$ and their corrupted versions $x_t$.
* In Equation 3, the formulas for $a$ and for $b$ are inverted. Also the "minus" sign should be in front of $a$ and not $b$. The equation further does not really make sense according to the paper’s notations, since "p(x_0|x_t) = p(x_0)" only “when $t → \infty$”. It is not mathematically correct to replace only a few terms of an equation by their limit, but keeping dependencies in $t$ for other terms. $a$ and $b$ are said to be constant, but their formula clearly shows they depend on $t$ (or are $a$ and $b$ the limits “when $t → \infty$”). 
* L158 states that the average of a large number of images is a solid color. This is incorrect, see for instance Figure 1 in **[1]**.
* In Equation 5, the notation ${\{ \}}_{\textnormal{batch}}$ is not introduced and it is not clear. It seems to contradict equation 6.
* Equation 7 is not justified. Why would $f$ only depend on the difference between $x_t$ and $x_0$?

**References:**

**[1]** Torralba, Antonio, and Aude Oliva. "Statistics of natural image categories." Network: computation in neural systems 14.3 (2003): 391.

**W2/ Similar methods have already been presented in previous literature. The submission does not discuss these existing works, nor compare with them.**

* The notions of “coupling” and “optimal transport” between data points and noise samples have already been presented in the previous literature. See for instance **[2]** and **[3]**.
* To understand the significance of this submission, it would be necessary to discuss these similar works and highlight the differences if any, and compare with such methods if possible. 
* As such, I do not see any novelty in this work (If I miss any, the authors fail to highlight the difference by discussing related work).

**References:**

**[2]** Tong, Alexander, et al. "Improving and generalizing flow-based generative models with minibatch optimal transport." arXiv Feb 2023 + TMLR + ICML Workshop 

**[3]** Pooladian, Aram-Alexandre, et al. "Multisample flow matching: Straightening flows with minibatch couplings." arXiv April 2023 + ICML 


**W3/ The presentation, including the figures quality and writing, needs to be improved.**

* The parallel with “Immiscible Fluids” does not bring a better understanding of the method, no physics equations are given. What should we understand from Figure 2a? Even if this is schematic, where would the noise and data distribution be in this image?
* The quality of Figure 3 is very bad. The images for predicted noise are completely saturated.
* The authors fail to show any ablation on the “quantization method” they claim. It is not clear if this is really necessary and the choices of fp8 and fp16 appear rather arbitrary.
* Many sentences do not make sense to me
    * L163: “T=20 means the layer of pure noise”
    * L160: what are “rich image classes”?
    * L162: “the predicted noises of different layers”
* Many typos:
    * L89 “its” for diffusion models
    * text written in math font in equations
    * L422: “our method accelerate”
    * L471: “our immsicible diffusion work” (2 typos)
    * Caption Figure 5: “can catch more details and more feature”
* Several incoherences:
    * L93: DDIM is 10 steps. <->  the authors experiment with DDIM but never take less than 20 steps.
    * linear_sum_assignment from Scipy (algorithm 1) <-> Hungarian matching (text)
    * Some generated images in Figures 8 and 9 are completely black.

**W4/ The presented results are not convincing.**

* All qualitative results presented in the main submission are cherry picked. The only non cherry picked results are shown in the appendix and the generated images, and the qualitative results seem very similar to me (no significant qualitative improvement).
* The quantitative results are difficult to trust. There seems to be incoherences between the results:
    * The “-0.98” in the third row of Table 5 is not coherent with the two first rows.
    * The plots have different scales and are not clear (why does the plot for imagenet in Figure 4 starts at 20k steps and not 0?)
    * The quantitative results mix consistency models and diffusion models in an unorganized way (eg, the columns of table 1 alternate between them), making it difficult to understand the results.
* It is not clear why some datasets use the diffusion approach and some use the consistency approach. Did the authors experiment with both for each dataset and only report the best results?
* The datasets and evaluation settings are not clearly presented. How many images are there in each dataset? Was the dataset split into training, validation and test sets? Are the reported FID scores computed on training, validation or test set? The number of sampling steps is not reported for all experiments (eg table 2). Only part of the quantitative results are shown (thus potentially cherry picked too, eg not quantitative results for Stable Diffusion)

----
**Changes following the rebuttal and authors-reviewers discussion:**

**Reduced soundness from 2 to 1**: The scope of the paper has been significantly expanded with the rebuttal (conditional generation, flipped dimension OT, decreased noise levels). Some arguments in the paper do not hold anymore (eg, average of all images). Revised mathematical proof remains imprecise and incorrect.

**Reduced contribution from 2 to 1**: Discussions following the rebuttal made me dive more deeply in the Conditional Flow Matching paper (https://arxiv.org/abs/2302.00482). Most, if not all, findings of the reviewed submission seem to be already known and analyzed more deeply in the Conditional Flow Matching paper, which additionally uses Flow Matching formalism (generalization of diffusion to more prior distributions than Gaussian Noise) and contains sound proofs. I do not see any new meaningful contribution for the Neurips community in the reviewed submission.

**W1** The authors did not address this concern correctly. To expand on the fact math assumption (Equation 7) is incorrect, suppose the target distribution is $P(x_0=0.1) = P(x_0=-0.1) = 0.5$. As $t \rightarrow +\infty$, it is clear that, if the batch size is large,  $p(x_t|x_0=0.1)$ is approximately $2 \times N(x_t; 0, 1)$ if $x_t>0$ and approximately  0 if $x_t <0$. This is clearly not in the form of Equation 7. The ratio also does not decrease with the norm of $x_t – x_0$, as $p(x_t=-0.1|x_0=0.1) \approx 0$ while $p(x_t=1.1|x_0=0.1) \approx 0.4$.

**W2** The authors did not address this concern correctly. They included new experiments to claim their method does not fit into “Conditional Flow Matching”, but it seems that it still does (Conditional Flow Matching also seems to allow to choose the coupling via the joint distribution $q(x_0, x_1)$).

**W3** The authors did not address this concern. The quality of Figure 3 is extremely low (all images are either extremely blurry or extremely saturated. It is possible the authors forgot something (dividing by 255?) when saving the images of the predicted noise, and that they upscaled the other images with interpolation or something). The authors did not comment on the incoherences/typos/sentences that need clarifications. The parallel with physics appears decorative without supporting physics equations. 

**W4** is not addressed correctly. Qualitative results show marginal improvement on cherry picked results only. Quantitative results are presented weirdly (eg, axes of the plot) with incoherences.

**W5** The new experiments introduced in the rebuttal significantly extend beyond the original scope of the paper (e.g., introducing OT with flipped dimensions, decreasing noise levels, and including conditional generation). Arguments made in the paper do not hold in this expanded scope. For instance, the paper's claim that the initial denoising direction points to the average of all images, and is thus not meaningful. This is not true in the conditional generation case, as the initial denoising direction points to the average of images of the desired class. Additionally, the authors seem to say that in immiscible diffusion, any noise sample can still lead to any class (so, is it miscible?), which needs to be discussed/explained/investigated more.

**Reduced overall rating from 3 to 2** Due to the (important) remaining concerns, the extent of the changes proposed in the rebuttal and discussion (would need additional reviews after changes are made), as well as the conflicts of the proposed changes with arguments contained in the original submission.

**Increased confidence from 3 to 5** I participated in the discussion and dived deeper into the CFM paper (which I was initially not very familiar with). I am certain about the need to reject the paper in its current state, with the possibility for the authors to address these issues in a future submission.

 ----
questions: questions: Here are my suggestions for this submission:

**Q1/** Discussion and comparison of related works. By not discussing the related works, it is not possible for me to see the novelty in this submission, given that similar methods have been presented in existing literature. (see also **W2**)

**Q2/** The presentation quality (figures, clarity) and mathematical explanations should be improved (see also  **W1**, **W3** and **W4**).

**Q3/** All qualitative and quantitative results need to be shown clearly, without mistake or cherry picking (see **W4**).
limitations: limitations: **L1/** The authors apply some matching within each mini-batch. So there is now a dependence on batch size, it is not sure the method would work with other batch sizes. For instance on GPU with little memory and batch size of 1, the method actually doesn’t change training at all compared to normal diffusion.

**L2/** Some theoretical results on diffusion may not hold anymore, e.g. that the ideal predicted denoising direction corresponds to the score (gradient of log likelihood) of the distribution of noisy data.

**L3/** The method only works for unconditional generation, which is a severe limitation. (EDIT: the rebuttal contains 1 experiment on conditional generation, but discussion is missing (how come noising is immiscible but denoising is not [noise samples can still lead to any class]?) and arguments like “the average of images does not contain much meaningful information” do not hold anymore.)

**L4/** Like any generative model, this work could be unethically used. It is crucial for researchers to consider designing safeguards to ensure the model can ignore inappropriate requests (eg, training or fine-tuning your Immiscible Diffusion model on inappropriate images). Can you provide a small discussion on existing safeguards and how follow-up work can address potential misuse of your method?

**L5/** The authors did not communicate the details (incl. license and terms of use) of the datasets or the models that they used via structured templates. The checklist incorrectly answered “NA” for “12. Licenses for existing assets”

Comments: title: Response to Reviewer Da24's Comments on Our Rebuttal
Response to Reviewer h6mk's Comments on Our Rebuttal - 2
Thanks for the Response
Response to Reviewer y1v3's Comments on Our Rebuttal
Response to Reviewer h6mk's Comments on Our Rebuttal
Thanks for the replies
Rebuttal to Reviewer Da24's comment on global responses
Answer to authors’ individual rebuttal
Answer to authors’ global rebuttal
comment: **Batch Size = 1 Problem**

The idea of the **alternative assignment**, assigning the image-noise pairs over a set larger than the batch seems interesting. It could also be extended to the cases of larger batch sizes, as it seems it allows to increase “immiscibility” without overly increasing gpu memory.

**Gradient accumulation** seems unrelated to the batch size = 1 problem (in which image-noise assignments are the same as normal diffusion)

**Fig. R9 Starting Point Problem**

I see, thanks for the clarifications about that all evaluations start at the same epoch but the plot uses training steps. However, the end of the curves are not aligned even within each batch size. Also, in these specific figures in the references, all curves start at the same x-axis value. I think the choice of the x-axis to be training steps rather than epochs or number of examples processed is confusing, since it tends to flatten and pushed to the right the curves for low batch-size.


**Influence of Batch Size**

“has little importance” and “influence is relatively minor” are synonyms to me. Given that batch size = 1 does not provide any improvement compared to basic diffusion, batch size has to have an important influence. Analyzing the tradeoff between improvements provided by your method and the batch size (GPU memory, cost of the GPU) could be useful for user to choose the batch size for training with the proposed method.

**1k / 1M**

Completely agree with you on 1k classes, 1M images. Thanks for the clarification.

**meaningful information at first denoising step**

I already gave comments on that. Also, this does not hold for conditional generation as I already pointed out.

**Mathematical Proofs**

**Adding three dots** like this does not provide anything rigourous. If I am allowed to put anything in place of these three dots, then f can really be anything, and it does not prove that the first denoising step is more meaningful. 

**limit t -> + inf:** The authors did not justify why it is ok (nor needed) in that case to apply the limit t -> + inf (a->1, b->0) to only certain terms of the equations. Simply saying that “the optimal (in MSE sense) denoising prediction when x_T is independent of x_0 is to predict the mean of the data distribution”, giving equations/proof for that, and proving that the first denoising step is more meaningful with the proposed coupling would be much clearer in my opinion.
**1.1.**

I do not deny that the FID plot is improved in the reported results for this experiment, but FID alone might not be sufficient for evaluation. 

I do not agree that cherry picking “helps the audience understand better how our method improves the baseline method”, as it shows improvements better than one can expect in average. 

Figures 7 to 11: I do not agree with “there might be a few cases that do not show significant visual improvement”. From what I can see, most cases do not show **significant** visual differences. 

**1.2. / 1.3.**

This would need to be explored in more details. I find it a bit weird that it is possible to claim that noising is immiscible (images are only noised locally) but that the model is able to do miscible denoising (model is still able to denoise any noise sample to any class). I suggest to look at the simple case I suggested with bright / dark images, and expand these explanations

**2.1. / 2.2**

I-CFM uses independent coupling, meaning the samples from prior (noise x_1) and target (data x_0) distributions are sampled independently, which does not qualify as “immiscible”.
It seems that the CFM formulation allows to select the coupling by a joint distribution q(z) = q(x_0, x_1) which can be arbitrary (not necessarily independent or OT: for instance SB-CFM, EB-CFM). The CFM paper (Arxiv 2023.02 / ICML 2023 workshop / TMLR 2024) also showed better metrics for non-OT and non-independent couplings (training time Table D1, Wasserstein distance Figure D6, although, yes, no FID are reported for that case)

**2.1 / 2.2**

I disagree that “high-level motivation is different and novel”. Furthermore, it seems that the CFM paper also includes “objective variance” analysis (D.1), which seems to be a way to quantify immiscibility.

**2.3**

Thanks for the clarification. The update rule in the diffusion reverse process is designed such that the distributions of x_t in the reverse process and in the forward process match for every t. This implicitly assumes x_T is indistinguishable from noise during training. Decreasing the maximal noise level can hurt this assumption.
Thank the reviewer for your reading and additional writing. We would like to offer some clarifications to the reviewer’s reply. For those related to the global response, we have provided a reply on the global response thread.

**Batch Size = 1 Problem**

We found that a training with batch size of 1 is hard to work for training any diffusion models *from scratch*. For examples, DDPM [Ho et al., “Denoising Diffusion Probabilistic Models”] uses batch sizes of 128/64 for training CIFAR-10/CelebA-HQ respectively, Stable Diffusion uses a batch size of 1200 for ImageNet [Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models”]. We kindly point out that even though we have to deal with GPU memory issues, we normally use gradient accumulation instead of barely using that batch size of 1. More importantly, we assume we have to deal with batch size with 1 for some extreme cases, an alternative way to our current assignment implementation is to generate a few noise points at the same time, assigning noises to images first and then perform diffusion on each image and its corresponding noise.

**Fig. R9 Starting Point Problem**

Firstly, we hope to kindly clarify that all reported curves in Fig. R9 start at the *same epochs*, with training step difference caused by batch size difference. For FIDs “from training step 0”, we hope to friendly address that at training step 0, the model is randomly initialized and contains no learned information. Therefore, per common practice, we empirically choose a start epoch for reporting FIDs when it reaches a reasonably low level, to save computational resources as well as to make plots readable. This is a common practice in many works (See Fig. 5 in Lipman et al., “Flow Matching for Generative Modeling” / Fig. 3 in Tong et al. "Improving and generalizing flow-based generative models with minibatch optimal transport."). Furthermore, it is quite clear in Fig. R9 that FIDs before the starting step would not change our claims about the influence of batch size on immiscible diffusion.

**Influence of Batch Size**

We respectfully disagree with the reviewer Da24 in the claim “The authors … seem to claim that the batch size has little importance”. In our response to reviewer neka, we wrote that “Results in Fig. R9 shows that larger BS enjoys better FID, but the influence of immiscibility significantly outweighs that of BS, and such influence does not vary significantly across selected BS.” We clearly state that batch sizes *have an influence* on diffusion performance, and our experiment shows that *comparing to the influence of immiscible diffusion*, the influence of BS is relatively minor, and *the influence of immiscibility* does not vary *significantly*, in *selected* BS (128-512 as shown in Fig. R9). We feel that our claim is significantly different from the reviewer’s comments, so we hope to provide a clarification to avoid misunderstanding.

**For the Average of the Dataset**

We kindly point out that in the word 1k in ImageNet-1k means 1k *classes*, and the name ImageNet-1k is the name of a popular version of ImageNet. We also kindly refer the reviewer to our revised math proof (https://openreview.net/forum?id=kK23oMGe9g&noteId=cgCNb4SgFk), where our claim that “When the number of images is large enough, $\overline{x_0}$ contains little meaningful information”, which corresponds to what Fig. R1 says. We also observe very blur average images similar to Fig. R1 for the average of images in each class of ImageNet.

**For Mathematical Proofs**

We believe that mathematical narrative is not “unnecessary” for the rigor of our proposed immiscible diffusion, and for readers outside the area to understand what we are doing as math can serve as the common language. We kindly propose that whether using the limit on $a$ and $b$ in Eqn. 3 and 8 doesn’t influence our main claim, so we do not explicitly discuss this in the proof. Also we hope to kindly point out that in the updated math proof, our $f$ is defined by $f(x_t-x_0, batch size, ...)$, so it *does not* depend only on the distance between $x_t$ and $x_0$.
We sincerely appreciate the reviewer's acknowledgement on our response, and we are grateful for all the time and efforts provided by the reviewer. We will implement the reviewer's interesting suggestion in our final version.
Thanks to the author for his efforts and reply. After reading the author's rebuttal carefully, I decided to increase my score to Weak Accept, but I hope the author can explain in detail the impact of the change on generalization in the subsequent version.

Specifically, you can generate 100 images for each class, then test the image features corresponding to these 100 images, and measure the distance between these image features to determine whether these images are diverse. If the generated images are diverse, it means that there are differences between these image features, and the distance between them is not very close, and there will be no mode collapse phenomenon similar to that in GAN (the features of all images are very close and collapse to a point in the feature space), and I believe t-SNE or the mentioned paper can better help you verify this problem.
We sincerely appreciate the reviewer's acknowledgement on our response, and we are grateful for all the time and efforts provided by the reviewer.
We thank the reviewer for reading and considering our responses. We hope to provide a few further clarifications on the problems addressed by the reviewer:

**1. About the generalizability**

Under the suggestion of reviewer h6mk, we have added additional metrics other than FID on images class-conditionally generated by immiscible SD trained on ImageNet. We provide CLIP-Score and CMMD for evaluating the image-prompt correlation and the image quality, respectively. We evaluate these metrics on 50k images generated from conditional Stable Diffusion models trained for 20k steps, which corresponds to the settings reported in Fig. R2.


The CLIP-Score for baseline and the immiscible model is both 28.55, with stddev of 0.02 and 0.01 respectively. We measured it for 3 times as the scores are so close, and our results further validate that there are no differences concerning the CLIP-Score, indicating that the image-prompt correlation is not  damaged by immiscible diffusion.


For CMMD, the value for baseline and immiscible model is 1.436 and 1.385 respectively (the lower the better). This further confirms that our immiscible diffusion outperforms the vanilla class-conditional SD.


**2. About Comparison to OT**


In our Global Response (https://openreview.net/forum?id=kK23oMGe9g&noteId=8Wj7beyL9K), we have included a thorough discussion on our difference to OT-CFM, including theoretical analysis and additional experiments. We kindly can not agree that significant similarities exist between I-CFM and us, and make our comments in the last part of our latest comment (https://openreview.net/forum?id=kK23oMGe9g&noteId=rYMAwGEnzS) We hope the reviewer can consider our opinions.

**3. About DDIM on different seeds**

We thank the reviewer for the question, but we hope to kindly address that in the upper part of Fig. R8, just like what the reviewer believes, we indeed observe no differences between the images generated either by vanilla or by immiscible DDIMs with the same noise and different random seeds. In the bottom part, we additionally provide images generated from vanilla and immiscible *DDPMs*, with the same *initial* noise and different seeds. As random noises are added during each sampling step of DDPM, which are not hold the same and can be influenced by the random seeds, we see diverse images generated either with vanilla or with immiscible DDPM.
We are sincerely grateful for the reviewer's time and efforts on reviewing our work! We hope to provide more information to clarify our findings on the reviewer's concerns:

**W1 - Practicality**

1. Larger Complex Image Dataset

We thank the reviewer for raising the concern. We are actively seeking to address this concern via training on large-scale LAION dataset. However, we found that the full LAION dataset has been removed due to the legal policy so we can not download it yet. We can only take the LAION-10K dataset which is legally available on Huggingface "wtcherr/LAION10K". To try our best to do reviewer h6mk a favor, we conduct experiments on the LAION-10K dataset, the results on LAION 10K show that we improve the baseline by 22.14 FID, demonstrating the effectiveness of our method. Despite the relatively small scale, we believe it still effectively shows the sign of life for our method on complex conditional generation.

2. Comparison Experiments on LAION-10k and on 10k Images in ImageNet

We thank the reviewer for proposing the experiment. We perform an additional experiment on 10k ImageNet data + 25k training steps with vanilla and immiscible conditional SD. The results are nearly the same as the experiments on LAION-10k dataset: the FIDs for vanilla SD on ImageNet (10k images) and LAION are 148.90 and 146.69 respectively, and those for immiscible SD are 128.66 and 124.55 respectively. The FID improvements are 20.24 and 22.14 respectively. The improvement is even a little bit larger on LAION-10k than on ImageNet (10k images), supporting that immiscible diffusion does not lose practicality in complex datasets.

**W1 - Generalization**

We understand the concern on the FID, and hope to present some additional evaluations to address your concerns:

For Performance:

We appreciate the suggestions of providing CLIP-Score and CMMD as other evaluation metrics for evaluating the image-prompt correlation and the image quality, respectively. We evaluate these metrics on 50k images generated from conditional Stable Diffusion models trained for 20k steps, which corresponds to the settings reported in Fig. R2.

The CLIP-Score for baseline and the immiscible model is both 28.55, with standard deviation of 0.02 and 0.01 respectively. We measured it for 3 times as the scores are so close, and our results further validate that there are no differences concerning the CLIP-Score, indicating that the image-prompt correlation is not damaged by immiscible diffusion.

For CMMD, the value for baseline and immiscible model is 1.436 and 1.385 respectively (the lower the better). This also demonstrates that our immiscible diffusion outperforms the vanilla class-conditional SD.

For generalization:

We sincerely thank the reviewer for proposing the additional experiment to prove the diversity of images under the same class-condition. We carefully read the paper provided, and we actively try to design experiments to show the image diversity with representation learning methods. However, the previous representation learning is studied on top of the feedforward models, and we find it unclear how to investigate the visual representation for a multi-step diffusion models. Nevertheless, we have submitted new proofs in Fig. R10, which contains images generated with the same prompt on our immiscible class-conditional SD without cherry picking, to the ACs with anonymous links. It clearly shows that the images are diverse in each class. We hope the release of those additional images can address the reviewer’s concerns.

**W2 - Image Correlation according to Classes**

We thank the reviewer for acknowledging our response, and are sincerely grateful for the reviewer’s efforts in reviewing our work.
I appreciate the reviewers' answer and it addresses some of my  concern. The generalization ability and the motivation compared with OT and FM is definitely the most important problem of this paper. The visualization in Fig R8 confuses me as even DDIM will generate different images for different seed and I disagree with the results, and I agree with Reviewer h6mk that FID is not the all to evaluate the generalization and the experiment he/she metioned should be added but actually not. For the concern of Reviewer Da24, which is mainly about the theory, I am not the expert of this and I will remain this problem for AC. I will reduce my score to boardline accept (In current evaluation benchmark, it is truly hard to judge the generalization ability so I believe that this method has some contribution to the diffusion model training).
I appreciate the reviewers' clarification and have decided to increase my score.
Reviewer `nEkA`:

It is only one day away, please provide your feedback to the authors' rebuttal.

AC
We thank reviewer Da24 for sharing valuable time with us and great efforts for reviewing and commenting on our rebuttals! Our further comments are attached below:

1. Conditional generation:

1) We emphasize that our quantitative results have already shown that immiscible diffusion improves 1.53 FID of conditional generation with SD (https://openreview.net/forum?id=kK23oMGe9g&noteId=8Wj7beyL9K). The qualitative results in Fig R2 are cherry-picked but given the quantitative improvement over vallina baseline, the visualization is used for supporting quantitative results and helps the audience understand better how our method improves the baseline method. Moreover, we refer the reviewer Da24 to check Fig 7 in our submission, the unconditional generation results on ImageNet are not cherry-picked. While there might be a few cases that do not show significant visual improvement, many examples like the 1st row 4th column, 2nd row 5th column, 3rd row 5th column etc. are apparently better than the baseline. Due to the fact that we have only 1 page for images, we were not able to put non-cherry-picked results during the rebuttal. We have now packed tens of classes of non cherry-picking images in an anonymous link that has been sent to the ACs for approval to show.

2) Thanks for the question from the reviewer Da24. We do not observe this phenomenon in our experiments. Our method does not hurt the conditional generation and results in the conjectures from the reviewer Da24. We randomly checked 4 classes, each with 5 images without cherry-picking, seeing that 1) these images correspond to their prompt class; 2) in each class, the images enjoy a sufficient diversity. We have concluded these results as Fig. R10, which has been sent to the ACs for approval to show. In the anonymous link we also include all ~50 images in each class checked for the reviewers’ information.

3) Thank the reviewer Da24 for the comment. In our experiments, immiscible conditional SD is indeed capable of performing better than vanilla one. Therefore, we still believe that immiscible diffusion works better in conditional generation cases. While the reviewer Da24 claims that “it is a good thing to first predict a direction point to the average of images of the class”, we kindly cannot find it excludes the possibility that our method is better. 

2. Comparison to OT-CFM

We sincerely appreciate the reviewer Da24 brings up flow-matching based methods during discussion. We emphasize that immiscible diffusion is a method upon diffusion models, and OT-CFM is built upon the framework of flow-matching method. Beyond it, we respond to the concern from Da24 as below.

1) We respectfully disagree that we are similar to coupling in I-CFM. To our knowledge, the random coupling in I-CFM in [1] does not qualify as an immiscible diffusion; also, our additional flip assignment design cannot be found in paper [1]. OT-CFM [1] makes the image-noise closer but our immiscible diffusion highlights they should be separated to each other, the flip assignment experiments demonstrate the effectiveness and the differences of our method. We believe this high-level motivation is different and novel.

2) We politely point out that we don’t see the effect claimed by the reviewer in paper [1]: We see that in the paper [1] Fig. 3 left, it clearly shows that only with OT the training efficiency be improved, the OT contributes to the majority of FID improvement. Furthermore, this is a work based on FM, which seems to be out of our scope in diffusion methods.

3) We very kindly suggest the reviewer noting that we are using DDIM, not SD, in this experiment.
Thanks for your rebuttal, here is my reply to your rebuttal:

**W1-Practicality**

Your reply partially addresses my concerns about the effectiveness of this method on complex datasets (rather than simple classification distributions). However, the training data for this experiment is only 10k, and I think the data size should be at least about 10% ImageNet (the dataset used for conditional generation in the paper) to prove its effectiveness.

In addition, could you provide the results of how much the improvement in FID on ImageNet is compared to the improvement on FID on LAION under the same training setting (10k data + 25k training steps)? It would be interesting to explore how much performance differences your method has on training sets with different data distributions.

**W1-Generalization**

The authors may have misunderstood my concerns. FID is good, but it is not an accurate measure of the quality and generalization of generated images. Lower FID does not mean your generative model is better in performance and generalization

* For performance, you can also report CLIP-Score [1] and CMMD [2] for conditional generation (class name as the text prompt).
* Regarding generalization, I am worried that after training with your method, the model will sample pictures with the same pattern for different random initialization noises, thereby damaging the diversity of generated images. To prove this, you can refer to this representation learning method [3] to see whether the features corresponding to the generated images of the same category are evenly distributed on the feature plane, and calculate their quantized values ​​to determine whether there is enough diversity between the generated images.

[1] Learning Transferable Visual Models From Natural Language Supervision, ICML 2021

[2] Rethinking FID: Towards a Better Evaluation Metric for Image Generation, CVPR 2024

[3] Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, ICML 2020

**W2-Image Correlation according to Classes**

I am satisfied with the answers and have no further questions or doubts.

**About Rating**

I tend to keep my score as is so far and will adjust the final score based on further explanations from the authors.
Thank you for your partial responses on my review. I have read the other reviews and the rebuttals (global one + individual ones). 

Unfortunately, the authors did not adress most of my concerns, many questions/weaknesses remain unanswered. 

Nevertheless, to answer to the authors: 
- **Mathematical proof in answer to nEkA W1**. This proof is still difficult to understand and unnecessary. When a denoising model is given pure noise as input, there is no doubt (classical derivations by setting gradient = 0) that the optimal denoising prediction (in MSE sense) is to predict the mean of the data distribution. In the case of immiscibility, similar derivations immediately show that the optimal denoising direction is to predicted a weighted mean of the data distribution, where the weights are the probabilities of each data-noise pair. Furthermore, to get equation 1, the authors assume that $a = \sqrt(1-\bar{\alpha_t})$ approaches 1, and $b = \sqrt(\bar{\alpha_t})$ approaches 0. It is unclear why the authors can use these limits for equation 1, but not in equations 3 and 8.
- **Mathematical proof in answer to nEkA W1**. I have doubts regarding the assumption that $f$ only depends on the difference between $x_t$ and $x_0$. This assumption seems incorrect to me. 
- **average of the images on the ImageNet dataset**. It can be seen on Figure R1 that the average is not a constant, as the center looks browner. Also, the caption of Figure R1 is inconsistent with the text of the rebuttal (1k or 1M?). Furthermore, this assumption that the average is constant is destroyed when the authors experiment with conditional image generation (see the third bullet point in https://openreview.net/forum?id=kK23oMGe9g&noteId=eeBmFmG8tn ).
- **W2**: See my answer in https://openreview.net/forum?id=kK23oMGe9g&noteId=eeBmFmG8tn
- **L1**: The provided plot (Figure R9) is very weird, it is not clear why curves do not start from the same height (same initial FID before stating training) and from training step 0. The authors did not verify/comment the special case batch size = 1, and seem to claim that the batch size has little importance ("such influence does not vary significantly across selected BS" in rebuttal to review-neka-Q1)
- **L3**: See my answer in https://openreview.net/forum?id=kK23oMGe9g&noteId=eeBmFmG8tn
Thank you for your rebuttal. Here is my answer to your global rebuttal:

**Conditional Generation**

The authors experimented on conditional generation by training a Stable Diffusion model to generate ImageNet images conditionally on their class. The authors show the FID score decreases faster with their method, which is encouraging. 

However, I have concerns related to this experiment:
- The authors only show 5 images (potentially cherry-picked) as qualitative examples. This concern was already raised in the initial reviews (Related to reviewer Da24 W4.1 Cherry-picking)
- It is possible that the model ignores the class name. The results shown in Figure R2 do not show that the generated images have the correct class. Furthermore, consider this thought experiment where you would have only two classes, “bright image” and “dark image”. With the proposed data-noise assignment pair, bright images will only be diffused to the brighter noise samples and dark images will only be diffused to the darker noise samples. Then the model will likely lose its ability to transform a “dark initial noise” (randomly sampled at inference time) into a “bright image” when conditioned on the class “dark image”.  The results shown in the rebuttal do not address this concern. This concern was already raised in the initial reviews (reviewer h6mk W1-Generalization)
- The main argument mentioned by the authors to support their method is that the first denoising direction predicted by the model (at timestep T) points to the average of the data distribution, which the authors say “does not provide any meaningful information” (L161). By “immiscibility” approach, the first denoising direction now points to the “average of nearby data-points” (L181). While this argument may hold for unconditional generation, it does not make sense anymore as (without immiscibly) it is a good thing to first predict a direction point to the average of images of the class.  This average of images of the class is informative (see https://web.mit.edu/torralba/www/ne3302.pdf Figure 1, see also first column of Figure 5 of https://arxiv.org/pdf/2305.08891 ). Predicting this average in the first denoising step allows to guide the generation towards images of the desired class. 

**Comparison to OT-CFM and related works**

The argument given by authors in the rebuttal are not convincing:

- To differentiate from OT-CFM, the authors now propose to redefine immiscibility as “immiscibility of diffusion destinations without necessarily minimizing image-noise distance in a mini-batch”, in other words: coupling (CFM) but no optimal transport (OT-CFM). However, this new definition still makes the proposed method very similar to CFM. The explanation in **Theoretical Difference** only explains the difference between OT-CFM and normal CFM. This concern was already mentioned in the initial reviews (Da24, W2.1 Coupling).
- **Additional Experiments Splitting Two Proposals/Domination of Immiscible Effect to OT in Noise Assignment**. The paper [1] (referenced in the rebuttal) already showed that CFM (new definition of “immiscibility”) can have better FID that standard FM. 
- **Immiscible Diffusion with Image Projection**: Multiplying all pixels by a factor of 2 or 4 in the training preprocessing is equivalent to reducing all noise levels by this factor. Some studies (https://arxiv.org/abs/2305.08891, https://www.crosslabs.org/blog/diffusion-with-offset-noise, https://arxiv.org/abs/2309.15842 ) showed however that the noise level used in Stable Diffusion (v1 and v2) are already too low, preventing the model to generate non-greyish images. The paper https://arxiv.org/pdf/2305.08891 showed (table 3) that increasing the noise levels fixes this issue and improve the FID score. I am very surprised that the authors here (Fig R6) find that decreasing all noise levels by a factor of 2 or 4 also improves the FID. Some discussion would be needed here, at least by showing qualitative examples.