Physics Informed Distillation for Diffusion Models
Authors: Joshua Tian Jin Tee
Kang Zhang
Hee Suk Yoon
Dhananjaya Nagaraja Gowda
Chanwoo Kim
Chang D. Yoo
Venue: Accepted by TMLR ()

Abstract: Diffusion models have recently emerged as a potent tool in generative modeling. However, their inherent iterative nature often results in sluggish image generation due to the requirement for multiple model evaluations. Recent progress has unveiled the intrinsic link between diffusion models and Probability Flow Ordinary Differential Equations (ODEs), thus enabling us to conceptualize diffusion models as ODE systems. Simultaneously, Physics Informed Neural Networks (PINNs) have substantiated their effectiveness in solving intricate differential equations through implicit modeling of their solutions. Building upon these foundational insights, we introduce Physics Informed Distillation (PID), which employs a student model to represent the solution of the ODE system corresponding to the teacher diffusion model, akin to the principles employed in PINNs. Through experiments on CIFAR 10 and ImageNet 64x64, we observe that PID achieves performance comparable to recent distillation methods. Notably, it demonstrates predictable trends concerning method-specific hyperparameters and eliminates the need for synthetic dataset generation during the distillation process. Both of which contribute to its easy-to-use nature as a distillation approach for Diffusion Models.

Published in Transactions on Machine Learning Research (06/2024)
Physics Informed Distillation for Diffusion Models
Joshua Tian Jin Tec* & Kang Zhang*
(joshualeeejjzzanggkka @aaist.aaaca.s.ccc kc.
School of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST)
Hee Suk Yoon
hskyoonn@aaist.cm kc.
School Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST)
Dhananjaya Nagaraja Gowda
d. d.goudda@samsung.coom
Samsung Research
Chanwoo Kim
chanwcom@korea..coo
Department of Artificial Intelligence Korea University
Chang D. Yoo
ed yoo@kaist.occ oc. kr
School of Electrical Engineering
Korea Advanced Institute of Science ond Technology (KAIST)
Reviewed on OpenReview: https: openreview. net/ forum? id= rd=aUuus69996
Abstract
Diffusion models have recently emerged as polent tool in generative modeling. However, their inherent iterative nature often results in sluggish image generation due to the require- ment for multiple model evaluations. Recent progress has unveiled the intrinsic link be- tween diffusion models and Probability Flow Ordinary Differential Equations (ODEs), thus enabling us to conceptualize diffusion models as ODE systems. Simultaneously, Physics Informed Neural Networks (PINNs) have substantiated their effectiveness in solving in- tricate different equations through implicit modeling of their solutions. Building upon these foundational insights, ploys student model to represent the solution of the ODE system corresponding to the teacher diffusion model, akin to the principles employed in PINNs. Through experiments on CIFAR 10 and ImageNet 64x64, we observe that PID achieves performance compara- ble to recent distillation methods. Notably, demonstrates predictable trends concerning method-speeific hyperparamotors and climinates the need for synthotic datasot generation during the distiilation process. Both of which contribute to its easy-to-use nature dis- tillation approach for Diffusion Models. Our code and pre-trained checkpoint are publicly available at:
introduce Physics Informed Distillation (PID), which em-
.ggtt
1
Introduction
Diffusion models -Scks-Dik et al. 2015 Song et al. 2021b
Ho et al.
2020 have demonstrated re-
markable performance in various lasks, including image synthesis DDhriiwal & Nichol 2021 Nichol et al.
2022
Ramesh et al.] 2022 Saharia et al. 2022a semantic segmentation Baranchuk et al. 2022 Wolleb
et
2022 Kirillov et al. 2023), and image restoration (Saharia al.] 2022b Whang al. 2022
et al.
2022 Niu al. 2023). With more stable training process, it has achieved better generation results
These authors contributed equally this work listed alphabetical order. Corresponding Author.

---

Published in Transactions on Machine Learning Research (06/2024)
that outperform other generative models, such GAN [Goodfellow et al. 2020), VAE (Kingma & Welling, 2013), and normalizing flows (Kingma & Dhariwal 2018 The success of diffusion models can mainly be attributed to their iterative sampling process which progressively removes noise from a randomly sampled Gaussian noise. However, this iterative refinement process comes with the huge drawback of low sampling speed, which strongly limits its real-time applications (Salimans & Ho 2022 Song et al. 2023
Recently, Song et al. (2021b) and Karras al. (2022) have proposed viewing diffusson models from a continuous time perspective. In this view, the forward process that takes the distribution of images to the Gaussian distribution can be viewed diffusion models learn the associated backward SDE through score matching. Interestingly, Song et al. (2021b) demonstrate that diffusion models can also be used to model probability flow ODE system that is equivalent in distribution to the marginal distributions the SDE. In addition, Physics Informed Neural Networks (PINNs) have proven effective in solving complex differential equations (Raissi et al. 2019 Cuomo 0t al. 2022 by learning the underlying dynamics and relationships encoded in the equations.
stochastic differential equation (SDE). On the other hand,
propose distillation method for diffusion models called Physics Informed Distillation (PID), method that takes PINNs-like approach to distill single-step diffusion model. Our method trains a model to predict the trajectory at any point in time given the initial condition relying solely on the ODE system. During training, we view the teacher diffussio model as an ODE system to be solved by the student model a physics-informed fashion. In this framework, the student model approximates the ODE trajectories, as illustrated in Figure | without explicitly observing the images in the trajectory. In detail, our contributions can be summarized follows:
Building upon these developments,
We propose and analyze Physics Informed Distillation (PID), knowledge distillation technique heavily inspired by PINNs that enables single-step image generation, providing theoretical bounds for the method.
Through experiments on CLEAR-10 and ImageNet 64x64, we showcase our approaches' effectiveness in generating high-quality images with only a single forward pass.
We demonstrate that similar to PINNs where the performance improvements saturate at sufficiently large number of collocation points, our approach with high enough discretization number performs best, showcasing its potential as a knowledge distillation approach that does not need additional tuning of method specific hyperparameters.
DATA
ODE trajectories
NOISE
X
2
PID trajectories
Figure 1: An overview of the proposed method, which involves training a model xz(2,)) to approximato the true trajectory x(z..)
2

---

Published in Transactions on Machine Learning Research (06/2024)
2
Related Works
The remarkable performance of diffusson models in various image generation tasks has garnered considerable attention. Nevertheless, the slow sampling speed associated with these models remains significant draw- back. Consequently, researchers have pursued two primary avenues of exploration to tackle this challenge: training-free free and training-bast based methods.
Training-free methods. Several training-free free methods have been proposed in the field of diffusion model research to expedite the generation of high-quality images. Notably, Lu et al. (2022) rntrodee DPM- Solver, a high-order solver for diffusion ODEs. This method significantly accelerates the sampling process
of diffusion probabilistic models by analytically computing the linear part of the solution. With only ten steps, DPM-Solver achieves decent performance compared to the usual requirement of hundreds to thousands of function evaluations. Another noteworthy approach is DETS (Zhang & Chen 2023), which exploits the semilineer structure of the empirical probability flow ODE, as in DPM-Solver, to enhance sampling efficiency, These methods demonstrate the surprising ability to significantly reduce the number of model evaluations while remaining training-free. On another note, Karras et al. (2022) propose enhancements to the SDE and probability flow ODE systems by implementing higher order solvers. Jolicocr-Martincau et al. (2021) also presents higher order solvers for SDE systems, aimed diminishing the number of functional evaluations required in a training free fashion. However, it is important to note that they only mitigate the core issue of the iterative sampling nature of diffusion models without climinating it entirely. In contrast to these training- free methods, training- based approaches have been proposed to perform single-step inference while aiming to maintain the performance of the teacher diffusion model. Our paper focuses on this particular field of study.
Training-based methods. In the domain of diffusion model distillation, training- based strategies can be broadly classified into two primary groups: those relying on synthetic data and those trained with original data. Among the former, noteworthy methods such as Knowledge Distillation (Luhman & Luhman 2021), DSNO (Zheng et al. 2023), and Rectified Flow (Liu et al.) 2023) have recently exhibited their efficacy in distilling single-step student models. However, a notable drawback inherent in these approaches is the computationally costly nature of generating such synthetic data, which presents scalability challenges, particularly for larger models. In response, methods using only the original dataset such as [Saliaan & Ho (2022) (Progressive Distillation) and [Song et al. (2023) (Consistency Models) have emerged as solutions to this issue. Progressive Distillation adopts a multi-stage distillation approach, while Consistency Models employ self-teacher model, reminiscent of self-supervised learning techniques.
Another notable mention is the recent data-free distillation approach BOOT (Gu et al. 2023), which solely relies on a teacher model without requiring synthetic data, similar to our proposed approach. However, there are fundamental differences between our work and BOOT. While our focus lics on solving the original probability flow ODE system, BOOT concentrates on Signal ODEs. In lackling the original ODE, we address challenges such as Lipschitz explosion problems near the origin, which can cause instability during training. We mitigate these issues through our paramctrization method outlined in Appendix A.44 On the contrary, BOOT's focus on Signal ODEs alleviates this particular challenge but introduces a different issue: the inability to satisfy boundary conditions through hard constraints. Consequently, starting from the distinct ODE systems addressed by BOOT (Gu et al. 2023) and our approach, our methods diverge due to the unique difficulties associated with each ODE system..
Another relevant work is the recent CTM (Kim et al. 2024). It's important to note that CTM shares a similar perspective with CT (Song et al.) 2023), focusing on distilling the paths traced by numerical solvers. On the other hand, we approach training from PINNs (Physics- (PhyssissInformeed Neural Networks) perspective, aiming to minimize the residual loss associated with the ODE system during training. It's worth highlighting that the similaritics between CTM and PID are primarily observed in the Euler method duc to its reliance on first- order gradient approximation. However, when considering second-order numerical gradient approximations, as presented in Table 3 our method demonstrates effectiveness despite its departure from CTM's training paradigm due to its basis the PINN paradigm. Consequently, CTM (Kim et al.] (2244) and PID can be seen as methods advocating from distinct perspectives, with their equivalence being tangential primarily in the first-order setting.
3

---

Published in Transactions on Machine Learning Research (06/2024)
3 Preliminaries
3.1 Diffusion Models
Physics Informed Knowledge Distillation is heavily based on the theory of continuous time diffusion models by Song et al. (2021b). These models describe the dynamics of a diffusion process through a stochastic differential equation:
dx f(x, t)dt + g())
(1)
where & E [0, is the standard Brownian motion (Uhlenbeck & Ornstein, 1930 Wang & Uhlenbeck
1945) (a.k.a Wiener process), f(---) and g(-) denote the drift and diffusion coefficients, respectively. The
distribution denoted as px()) with the initial distribution po(x) corresponding to the data distribution,
Pdata
As proven in Song et al. 20011), this diffusion process has corresponding probability flow ODE with the same marginal distributions px) of the form:
1 Vx log (x) de, dx -(x
(2)
where Vx Px denotes the score function. Diffusion models learn generate images through score matching (Song et al.) (021b), approximating the score function as logggp sx,,tt) with sx(x, t) being the score model paramcterized by ø. As such, diffusion models learn to model the probability flow ODE system of the data while relying on ilerative finite solvers to approximale the modeled ODE.
3.2 Physics Informed Neural Networks (PINNs)
PINNs are scientific machine-learning technique that solve any arbitrary known differential equa- 2019 Cuomo et al., 2022). They rely heavily the universal approximation theo- Hornik et al. (989) of neural networks to model the solution of the known differential equation (Cuomo
tion
Raissi et al.
rem
et al.
2022). To better explain the learning scheme of PINNs, let us first consider an ODE system of the
form:
dx dt u(xx(),
(3)
x(T) = z0.
where t E [0,T", u(:,-) is an arbitrary continuous function and 2o is arbitrary initial condition at the time T. To solve this ODE system, Physics Informed Neural Networks (PINNs) can be divided into two distinct approaches: soft conditioning approach (Cuomo et al., 2022), in which boundary conditions are sampled and trained, and a hard conditioning approach, (Lagaris et al. 1998 Cuomo et al. 2022) where such conditions are automatically satisfied through the utilization skip connections. For the sake simplicity, we exclusively focus on the latter casier approach, where the PINNs output reprosented as follows:
xo(t) = Cakip (t)20 + Cout (t))) where Cskip (T) = 1 and Cout (T) = 0,
(4)
here X, denotes neural network parametrized by 0 and the functions, Cakip and Cout, chosen such that the boundary condition is always satisfied. Following this, PINNs learn by reducing the residual loss denoted
(t) - u(x)(t))
dt
Through this, they can model physical phenomena represented by such ODE systems. Inspired by PINNs ability to solve complex ODE systems Lagaris et. al., 1998 Cuomo et al. 2022), ; we use a PINNs-like approach to perform Physics Informed Distillation. This distillation approach uses the residual loss in PINNs solve the probability flow ODE system modeled by diffusion models. Through this distillation, the student trajectory function in Figure can perform fast single-step inference by querying the end points of the trajectory.

---

Published in Transactions on Machine Learning Research (06/2024)
4 Physics Informed Knowledge Distillation
4.1 Trajectory Functions
In this section, we begin by introducing the concept of trajectory functions, representing solutions to a given arbitrary ODE system. Subsequently, we demonstrate how these trajectory functions can be learned through the application of a simple PINNs loss, resulting in a single-step sampler. It's noteworthy that a recent work, BOOT, also employs a PINNs-like approach to distill trajectory functions. However, their primary focus is on distilling the Signal ODE, while our emphasis lies in the direct distillation of the probability flow ODE. Later sections will delve into specific modifications, highlighting challenges encountered and successful adaptations made to effectively distill the probability flow ODE.
From this point onwards, we adopt the configuration introduced in EDM (Krrras et al. 2022 The config- uration utilizes the empirical probability flow ODE on the interval & E [e, 7) of the following form:
dx
dt (x, t)
(6)
(x,
where x(T) ~ N(O, T < = 0.002, 80 and Dx 1) is the teacher diffusion model. Under the assumption that Diffusion Models are Lipschitz continuous on the given time interval, this ODE system exhibits a unique solution (Grant 1999). As such, we can conceptualize each noise as having its uniquely associated trajectory function, illustrated in Figure [ We denote this trajectory function as x(z, t), where z represents the noisc's initial condition at time T. Our objective is to train a model (o t) that prcciscly captures the actual trajectories x(z, L) for any arbitrary noise, z. To achieve this, we adopt a PINNs-like approach to learn the trajectory functions x(2, t). Specifically, we can minimize the residual loss, similar to the methodology employed in PINNs.
Similar to in PINNs, we require our solution function, x (x)),,, to satisfy the boundary condition xœ(z, T) In PINNs literature, the two common approaches to achieve this is either by using soft condition (Cuomo et al. 2022 in which boundary conditions are sampled and trained or through strict condition (agaris
1998 where these conditions are arbitrarily satisfied using skip connections. Since the use of skip ct al.
connections is also common in diffusion training (Karras et al. 2022), and parametrize our model as:
we take inspiration from both fields
xz (tkip(t) z + Cout (t) Xo (Cin (T) z, Cnoise (t)), t where Cakip (t) T Cout (t)
-
1
and Groise (t) Int
Cin (T) v0.5² +
T
Here, X. denotes the neural network to be trained (insight. into the choice of skip connection functions are provided in Appendix A.4 Using the vanilla PINNs loss on the probability flow ODE, the loss is given as:
LPINNS Ei,z [d] (ccxo(e,))) Xe(z, tx) - -o (xo (z, ts), tx) t.).&e) ]
(8)
dd,
dl,
4
where d(...) is any arbitrary distance metric (e.g. L2 and LPIPS). In diffusion models, the Lipschitz constant of the ODE systems they model tend to explode to infinity near the origin (Vang et al., 2002).. For instance, in Equation 6 it can be seen that the ODE system explodes as t approaches O. Consequently, training a solver in vanilla physics informed fashion may yield suboptimal performance, it involves training the
gradients of our student model to correspond with exploding value. To alleviate this, we shift the variables as:
LPINNs = =Ei, (dl [x(xx(2, (xx(2z,t)) (x) dxx(2,44) (2) Do (xo(2, (xx(2,t4,,t)) (.).
(9)
dt;
Through this, we can obtain a stable training loss that does not explode when time values near the origin are sampled. By performing this straightforward variable manipulation, we can interpret the residual loss
5

---

Published in Transactions on Machine Learning Research (06/2024)
the process of learning to match the composite student model, parametrized by 8, the left portion of the distance metric with the output of the teacher diffusion model on the right portion of the distance metric. Since the output of the teacher diffussin model on the right is also dependent on the student model, we can also view this from a sclf-supervised perspective. gradients flowing from the teacher model (Grill al., 2020 Caron et al., 2021: Chen al. 2020). In line with this, we also stop the gradients flowing from the teacher diffusion model during training.
this perspective, it often conventional to stop the
4.2 Numerical Differentiation
In what follows, we will briefly overview the challenges associated with automatic differentiation for gradient computation. Then, we propose the adoption of numerical differentiation, a technique prevalent in standard PINNs training and also employed in the BOOT framework.
The residual loss in PINNs requires computing gradients of the trajectory function with respect to its inputs, which can be computationally expensive using forward-mode backpropagation and may not be readily available in certain packages. Furthermore, studies such as Chiu et al. (2022) have demonstrated that training PINNs using automatic differentiation can lead to convergence to unphysical solutions. To address these challenges, we employ a different approach by relying on a straightforward numerical differentiation method approximate the gradients. Specifically, we utilize first-order upwind numerical approximation, given as:
dx, dt
(0 xo - At)
(10)
At
num
By employing this numerical differentiation scheme, can efficiently estimate the gradients needed for the training process. This approach offers simpler alternative to costly forward-mode backpropagation and mitigates the issues related to convergence to unphysical solutions observed in automatic different bateen training of PINNs.
Algorithm Physics Informed Distillation Training
Input: Trained teacher model D&, PID model x8. LPIPS loss d(...), learning rate 1) discretization number N.
1: 0 + 0+ Initialize student from teacher
repeat
U10, 10, N| / Sample time index i.idd from a uniform distribution
4:
z~ 1~,, I // Sample data
5:
dx
+ (xo (2) (z, 4 (411))((t, (++1) // Numerical gradient approximation
di
Xteacher t- sg (D& (z, tz), t.)) // Get teacher model output
LPID < Xteacher: Xy Xo (z, ()) 44 de // Loss calculation according to Equation
8:
0+ VoLpD // Update weights
until model converged
4.3 Physics Informed Distillation
In this section, consolidate the insights gathered from the preceding sections to present Physics Informed Distillation (PID) as a distillation approach for single-step sampling. In addition, we conduct theoretical analysis of our proposed PID loss, establishing bounds on the generation error of the distilled PID student model.
Replacing the exact gradients with the proposed numerical differentiation and setting At 4 -141, we obtain the Physics Informed Distillation loss:
LPID = Fiz [a (xo(2. 4:) - Xo(z, t/) xo(z, ts+2)
(x((((x (2)44)4)))
(11)
to t.41

---

Published in Transactions on Machine Learning Research (06/2024)
Table 1: FID and IS table comparisons for various sampler based and distillation based methods on CIFAR-
10. The asterisk (*) denotes methods that require the generation of synthetic dataset. The neural function evaluations (NFE), FID score and IS values reported where obtained from the respective papers.
METHOD
Distance Metric
NFE
FID
IS
(4)
(4)
(1)
EDM Karras et al.
2022
36
2.04
EDM
Euler Solver ver
Karras et al. 2022)
250
2.10
DDPM Ho et al. (020
1000
3.17
9.46
DDIM Song ct al.
2021a
10
13.36
50
4.67
OPM-Solver-FRi Lu et al. 2022
10
4.70
3-DEIS (Zhang & Chen
2023
10
4.17
Teacher DDPM
Progressive Distillation Salimans & Ho
2022)
L2
1
8.34
8.69
DSNO* (Zheng et al. 2023
LPIPS
1
3.78
Teacher Rectilied Flow
2- R-Recifieed Flow (+ distill)* Liu et al.
2023
LPIPS
1
4.85
9.01
Teacher EDM
Consistency Model (Song et al.
2023
LPIPS
1
3.55
9.48
BOOT (uu et al. 2023
LPIPS
1
4.38
Diff- lnffrustttct (Luo et al. 2023
L2
1
4.12
9.89
Equilibrium Models (Geng ct al. 2023
L1
1
6.91
9.16
PID (Ours)
LPIPS
1
3.92
9.13
where sg(-) denotes the stop gradient operation. For the time-space discretization scheme, we follow the same schome as in EDM (Karras al.) 2022). The discretization error addod in such scheme is bounded as shown in the Lemma [ and we provide the proof in Appendix A.2
Lemma 1. Assuming tx(x,t) is Lipchitz continnous with respect to x, if Lrm = 0, ||x0(2.)) - x(2, t)ll O(At), where At = max [+++1 -t).
An intriguing aspect of this theorem lies in its connection to Euler solvers and first-order numerical gradient approximations. Specifically, when our model achieves loss of 0, the approximate trajectories traced by the PID-trained model, denoted as xz(2,t), effectively mirrors those obtained using a simple Eulcr solver em- ploying an equivalent number steps, denoted as N. Moreover, as indicated Lemmn [ the discretization error well controlled and diminishes with an increasing number of discretization steps, N. This observation suggests that by employing a higher number of discretization steps, we can effectively minimize the error associated with the discretization process, achieving improved performance.
In practice, we replace the traditional distance metric with LPIPS, motivated by the successes of Rectified Flow Distillation (Liu et al.) 2023) and Consistency Models (ong et al. 2023) using this metric. Despite not being a proper distance metric, LPIPS empirically produces the best results as it is less sensitive to pixel differences, allowing the model to focus its capacity on generating high-fidelity images without wasting capacity on imperceptible differences.
By employing the PID training scheme, the resulting model effectively match the trajectory function x(z,t) within the specified error bound mentioned earlier. Single-step inference be performed by simply querying the value of the approximate trajectory function (2(2,)) f) at the endpoint c, x,(2, c). With this, we observe that by treating the Leacher diffusion model as an ODE system, we can theoretically train a student model to learn the trajectory function up to a certain discretization error bound without data and perform fast single-step inference. Algorithm provides the pscudo-code describing our Physics Informed Distillation training process. After training, single-step generation can be achieved by querying the learned trajectory functions at the origin, denoted (2(2, )min).

---

Published in Transactions on Machine Learning Research (06/2024)
Table 2: FID table comparisons for various distillation based methods on ImageNet 64x64. The asterisk (*)
denotes methods that require the generation of a synthetic dataset.
Method
Distance Metric
NFE (L)
FID (1)
ADM Dhariwal & Nichol
2021
250
2.07
EDM Karras et al.
2022
79
2.44
EDM + Euler Solver Karras et al.
2022
250
2.41
BigGAN-deep (Brock al. 2019
4.06
Teacher DDPM
Progressive Distillation Salimans & Ho 2022
L2
1
15.39
DSNO* (Zheng et al. 2023
L11
7.83 1
Teacher EDM
Consistency Model (Song et al. 2023
LPIPS
1
6.20
BOOT (Gu et al.) 2023
LPIPS
1
12.3
Diff-Instruct (Luo ct al. 2023
L2
1
4.24
PID (Ours)
LPIPS
1
9.49
5 Results
In this section, we empirically validate our theoretical findings through various experiments on CIFAR- 10 (Krizhevsky & Hinton 2009) and ImageNet 64x64 (Deng al. 2009). The results compared according to Frechet Inception Distance (FID) (Heusel et al. 2017b) and Inception Score (TS) (Salimans et al. 2016
All experiments for PID were initialized with the EDM teacher model, and all the competing methods were also initialized with their respective teacher diffusion model weights as noted in Table | and Table 2 In addition, unless stated otherwise, a discretization of 250 and LPIPS metric was used during training. More information on the training details can be seen in Appendix A.1
We quantitatively compare the sample quality of our PID for diffusson models with other training-Tree and training- based methods for diffusion models, including DSNO [Zeng et al. 2023). Rectified Flow (Liu et al.
2023)
PD (Salimans & Ho 2022). CD (Song et al. (023). BOOT Gu et al. 2023 and Diff-Instruce (Luo et al. 2023 In addition to the baseline EDM (Karras et al. 2022) model, we make comparisons with other campler-based fast generative models, such as DDIM (Song et al. (0221), DPM-solver (Lu et al. 2022) and
DEIS Zhang & Chen 2023). In Table I we show our results on CIFAR 10 dataset. In this, PID maintains competitive result in both FID and IS with the most recent single-step generation methods, achicving an FID of 3.92 and IS of 9313, while outperforming few. In particular, we outperform Diff-Instruct, Rectified Flow, PD, Equilibrium Models and BOOT (Gu et al. 2023 by decent margin CIFAR-10. On the other hand, maintain competitive performance with DSNO and CD, only losing to it by small margin.
Table presents the results obtained from experiments conducted on ImageNet. 64x64. Analyzing Table 2 we observe that our method surpassos PD (Salimans & Ho 2022). achieving an FID of 9.49. Nevertheless, echoing our observations from the CIFAR-10 experiments, our approach lags behind DSNO (Zheng et al. 2023) and CD (Song al., 2023), which achieves lower FID of 7.83 and 6.20 respectively. Despite this, it worth emphasizing that method does not entail the additional costs associated with generating expensive synthetic data, which is characteristic feature of DSNO. This cost-effective aspect represents notable advantage of our approach in the context of ImageNet 64x64 experiments despite its poorer performance.
In Figure 2 we compare qualitatively between the images generated from the teacher EDM model and the student model. From this, we observe that in general, the student model aligns with the teacher model images, generating similar images for the same noise seed. Additional samples from Imagenet and CIFAR-10 are provided in Appendix A.6
8

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Random samples from EDM
(b) Random samples from PID
Figure 2: Conditional image generation comparison on ImageNet 64x64 for the same seed with the same class label 'Siberian husky'. Left panel: random samples generated by teacher model EDM. Right panel: gencrated by student PID model.
Table 3: FID lable comparisons for different order numerical differentiation on CIFAR 10.
Method
FID (1)
PTD (Ist Order)
3.92
PID (2nd order Central Difference)
3.68
6 Ablation on PID Design Choices and its properties
6.1 Comparing Higher Order Numerical Differentiation
In this section, investigate the impact of higher-order numerical differentiation the distillation per- formance of PID. Specifically, compare the outcomes of employing 1st order numerical differentiation with those obtained using the 2nd order Central Difference Method. While many higher-order approaches typically necessitate more than model evaluations, leading to increased computational costs for numerical gradient computation, the Central Difference method, despite being a second-order numerical differentiation technique, only requires 2 model evaluations, maintaining the same computational cost as the 1st order ap- proach. As indicated in Table 8 the 2nd order approach exhibits slightly superior performance compared to 1st order numerical differentiation. This aligns with observations from standard PINNs training, where higher-order numerical differentiation tends to yield solutions that closely align with the actual ODE system.
6.2 Automatic Differentiation vs Numerical Differentiation
In this section, empirically evaluate the importance of Numerical Differentiation in the PID training scheme. The experiment presented in Figure 3 was conducted on CIFAR 10, employing the same experi- mental settings as in the main results, albeit with the substitution of numerical diffcrentiation for automatic differentiation. From Figure 3, it can be observed that automatic differentiation in PID training leads to convergence towards unrealistic images characterized by high image contrast, resulting in poor FID scores. This observation aligns with findings in Chii et al. (2022) where PINNs dcaling with systems of differential equations, numerical differentiation often outperforms automatic differentiation, even in regions with large number of collocation points. Intuitively, numerical differentiation can be conceptualized utilizing local

---

Published in Transactions on Machine Learning Research (06/2024)
Automatic Difenentiaioon
Numerical
300
250
200
150
100
Training iterations 10000)
(a) Automatic Differentiation
(b) Numerical Differentiation
(c) FID comparison
Figure 3: Comparison between automatic differentiation and numerical differentiation on CIFAR-10 dataset.
points to stabilize the PINN loss, thereby preventing convergence to solutions too divergent from the ground truth ODE solutions.
6.3 Random Initialization
In this section, we examine the impact random initialization on the performance of PID. The experiment detailed Figure | was carried out using the same experimental settings as the main results on CIFAR 10, employing randomly initialized student weights. From Figure 4 is evident that random initialization leads PID to converge to a suboptimal local point, resulting in higher FID compared initializations with pre-trained weights. Hence, validating the use of initializing the student model with the pre- pre-trained teacher diffusion model instcad of random initialization.
Init from Teacher
14
10w
Training Iterations 10000)
(a) Random initialization
(b) Initialize from teacher
FID comparison
Figure 4: Comparison between student model weights random initialized and initialized with teacher model
weights on CIFAR-10 dataset.
6.4 Stop Gradient
In this section, we investigale the impact of stop gradient in the PID training scheme. The experiment outlined in Figure 5 was carried out under the same experimental settings as the main results on CIEAR 10, albeit without the use of stop gradient. From Figure 5 we obscrve a degradation in performance when backpropagating through the teacher diffusion model during PID training. We hypothesize that the removal of stop gradient which updates the student while backpropagation through the teacher diffusion
10

---

Published in Transactions on Machine Learning Research (06/2024)
Stop-Gradient
W Stop-Gradient
14
2 12
10
Training Iterations 10000)
(a) Without stop-gradient
(b) With stop-gradient
(c) FID comparison
Figure 5: The impact of stop gradient in the PID training on CIFAR-10 dataset.
model behaves akin to an adversarial attack on the teacher model. This results generated samples with low residual PINN loss but high levels of distortion in the images.
6.5 L2 vs LPIPS comparison
To thoroughly investigate the influence of different distance metrics on model performance, we conducted experiments on CIFAR-10 using the L2 and LPIPS metrics. These experiments were carried out following the same experimental setup described in the main results. Analyzing the results depicted in Figure
observe similar trend to previous studies utilizing LPIPS metric (Song et al. 2023 Liu et al., 2023 wherein a slower convergence rate with L2 compared to LPIPS was observed. This difference between L2 and LPIPS persists even up till convergence with the L2 metric obtaining an FID of 5.85 and the LPIPS metric achieving an FID of 3.92. Consequently, this reinforces the validity and suitability of the LPIPS metric in our training approach.
6.6 Comparing Discretization Numbers
To properly understand the effect of discretization number, N, on our method, the experiment on CIFAR-10 was repeated with different discretization values. The experiments conducted in this section were performed with the same hyperparamcter sctup as in the main results except with changing discrctization number. The discretization values investigated here were {35, 80, 120, 200, 250, 500, 1000).
In Figure can observe that the performance of our single step image generation model steadily increases with incrcasing discrotization. This behaviour aligns well with the thcoretical expectations according to Lemma I where the discretization error decreases with higher discretization. Additionally, this behaviour is also common to PINNs (Raissi et al., 2019) where increasing the collocation of points on a trajectory improves model performance. This stable and predictable trend with respect to discretization justifies our choice of setting our discretization number 250 where the performance has plateaued, achieving good performance despite not tuning any metheolllog-sspecifie hyperparameters.
Additionally, higher discretization number has no effect on training time seen in Figure f as it not only converges faster to a better FID value in the same number of iterations. Consequently, PID can be trained with an arbitrarily high discretization number to achieve optimal performance. This is in stark contrast to methods such as CD Song et al. (2023) whore inercasing or decrcasing discretization numbers away from optimal discretization number results performance degradation resulting in the need to search this optimal discretization number that differs from dataset to dataset.
11

---

Published in Transactions on Machine Learning Research (06/2024)
20
20
35
LPIPS, 250
18
N= 80
18
L2, N 250
16
120
16
N= 200
14
N 250
14
N 500
12
12
N 1000
10
10
25
Training iterations 10000)
Training iterations (x 10000)
Figure 6: Training curve with different discretizations number CIFAR-10.
Figure Training curve for different distance met- rics, L.2 and LPIPS, CIFAR-10.
7 Computational Costs comparisons
7.1 Training Efficiency Comparison
Due to the incorporation of numerical differentiation in our approach, we regrettably require two model evaluations, in contrast to the single model evaluations employed by other methods. In this section, we provide a detailed comparison of the training time elliciency of our method with recent works, CD (Song et al. 2023) and PD (Salimans & Ho. 2022). The analysis presented in Table Sublegrreeen reveals an intriguing finding: despite the necessity of two functional evaluations, our approach incurs only a 35% higher training cost compared to doubling, as might be expected. This discrepancy arises from the fact that, while our method uses only a single teacher model evaluation, recent methods instead requires two iterative teacher model evaluations, thus incurring additional training expenses. It is worth emphasizing that despite the additional training cost per iteration attributed to our approach, the consistent and stable trend we observe concerning the discretization number allows us to train our model without the need to fine-tune any methodology-spede hyperparameters. This, in turn, significantly reduces the cost associated with hyperparameter optimization.
Table 4: Training time comparisons betwoen PID and recent works on CIFAR 10.
Method PD CD DSNO
Training Time (seconds per 10 iterations)
5.15
5.21
7.21
PID (ours)
7.13
7.2 Full Training Time comparisons with Recent Methods
In Table we present comparison of the full training time across several recent distillation methods for diffusion models. The comparisons were conducted on ImageNet 64x64, following the training settings outlined in the respective papers. As observed in the CIFAR-10 time comparisons in Table g our method is slower than CD due to the need for 2 functional evaluation ionu for numerical differentiation.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Table 5: Training time comparison for each method, measured on 64 NVIDIA A100 GPUs. The asterisk("* denotes the time taken obtain the synthetic dataset used for distillation.
Method
Training Iteration
Total Training Time (hours)
PD
550000
104
CM
600000
144
PID
600000
197
DSNO
480000
24*
8 Conclusion
In this paper, we introduce Physics Informed Distillation (PID), a method designed to train a single-step diffusion model that draws significant inspiration from Physics Informed Neural Networks (PINNs). Through combination of empirical evaluations and theoretical underpiunings, we have demonstrated the robustness and competitiveness of our method in comparison to the majority of existing techniques. While falls slightly behind DSNO and CD, it distinguishes itself by eschewing the need for costly synthetic data generation or meticulous tuning of methodology-speee performance with constant methodology-speiile hyperparameters.
hyperparameters. Instead, our approach achieves competitive
9 Acknowledgement
This work was supported by Institule of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea MoVer (No 2022002--00,,,84, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy Ethics), and Institute for Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government(MSI1) (No. 2021-0-01381, Development of Causal AT through Video Understanding and Reinforcement Learning, and Its Applications to Real Environments).
References
Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficiett semantic seginntattin with diffusion models. In International Conference on Learning Representations, 2022. URL https://openreviest wwt.:/ooruriii--ssss22
Andrew Brock, Jelf Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL httpp:://penreevier het/fforum?dd=B1leeej0f9
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging propertics in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning. arXie preprint arXiv:2003.0;20 2020.
Pao-Hsiung Chin, Jian Cheng Wong, Chinchun Ooi, My Ha Dao, and Yew-Soon Ong. Can-pinn: A fast physics-inff informed neural network based coupled-automatic numerical differentiation method. Computer Methods in Applied Mechanics and Engineering, 2022.
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics-informed neural networks: where we are and what's next. Journal of Scientific Compuling, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEFE conference computer vision and pattern recognition, 2009.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 2021.
Zhengyang Geng, Ashwini Pokle, and Zico Kolter. One-step diffusion distillation via deep equilibrium models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openreview //openzeview.nnt//orrmmm (otppa/oissssnettooooouuddddtddd wtt.://ooormm..ddoobkaaaaaaaaaa
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 2020.
Christopher Grant. Lecture 4: Picard-lindelöf theorem. http://www. http://www..att.... heattth.yy.uuuuu/eee httpp/.dd//ggaan//uureeess
m634/f99/lec4 pdf) 1999.
Jean-Bastien
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchalskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 1121221--2288 2020.
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Joshua M Susskind. Boot: Data-free distillation of denoising diffusion models with bootstrapping. Tn JCML. 2023 Workshop on Structured Probabilistic Inference {10} Generative Modeling, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017a.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rulc converge to local nash equilibrium. Advances in neural infor- malion processing syslems, 2017b.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in newral 6851, 2020.
information processing systems,
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Newral networks, 1989.
Alexia Jolicoeur- Martineau, Ke Li, Rémi Piché- Taillefer, Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with scoro-based models. arXiv preprint arXiv:2105. 14080, 2021.
Tero Karras, Milka Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based gen- erative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Newral Information Processing Systems, 2022. URL https:://pennreviie nett//ooruumi.dckFFTTTTMM
Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ODE trajectory of diffusion. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevier. nww. wwwpteeuuwmniieeeemm (et
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. Advances in neural information processing systems, 2018.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4015-4026, October 2023.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical
Report 0, University of Toronto, Toronto, Ontario, 2009. URL https://wwww cs. toronto........ www.ttoo.ooooo..rr...... feamming-eatture=20000-1--
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 1998.
Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 2022.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations, 2020. URL https::/ http://pww.ppoooeoooooooo./////r
Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023.
URL https://openrrevier wttp://rww..etttoom/m//a//ytaaaa
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (cds.), Advonces in Neural Information Processing Systems, 2022.
URL htttps://ppenreviie wwtp:/oouummii--uuaaaa.. http:/arwreeeeeeeeffooommmimaaaa
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint or arXiv:2101.09358 2021.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang, Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. In seventh Conference on Neural Information Processing Systems, 2023. URL https:://operrevier MLIs51Rq4w
iid=
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mogrew, Ilya Sutskever, and Mark Chen, GLIDE: Towards photorealistic image generation and editing with text- guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learving Research, pp. 16784 16804. PMLR, 17 23 Jul 2022. URL https://proceedingg mlr. wwwpppeeeeeieeeeeee mii//iiiiee wwwp/////nionneeeeee wowhhholaaa...c html
Axi Niu, Kang Zhang, Trung X Pham, Jinqiu Sun, Yu Zhu, In So Kweon, and Yanning Zhang. Cdpmsr: Con- ditional diffusion probabilistic models for single image super-resolution. arXiv preprint XrXiv:2302.12899 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, ZZeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural informalion processing systems, 32, 2019.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 2019.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arxiv:2204.000 066222 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 2022a.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. TPAMI, 2022b.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference Learning Represenlations, 2022. http://ww//pppeeeeeeee.///e/aa/a
Tim Salimans, Tan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.
arXiv preprint Xr:i:: 1556, 2014.
Jascha Sohl-Dickstein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusson implicit models. 2021a. URL https: aepparrriiiv..eet//oooouussstiso
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben
Poole. Score-based generative modeling through stochastic differential equations. 2021b. URL https:
Yang Song, Prafulla. Dhariwal, Mark Chen, and 1lya Sutskever. Consistency models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
George E Uhlenbeck and Leonard S Ornstein. On the theory of the brownian motion. Physical review, 1930.
Ming Chen Wang and George Eugene Uhlenbeck. On the theory of the brownian motion ii Reviews of modern physics, 1945.
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR, 2022.
Julia Wolleb, Robin Sandkühler, Florentin Bioder, Philippe Valmaggia, and Philippe C Cattin. Diffusion models for implicit image segmentation ensembles. In International Conference on Medical Imaging with Deep Learning, 2022.
Zhantao Yang, Ruili Feng, Han Zhang, Yujun Shen, Kai Zhu, Lianghua Huang, Yifei Zhang, Yu Liu, Deli Zhao, Jingren Zhou, et al. Eliminating lipschitz singularitics in diffusion models. arXiv preprint 2rXiv 0ete22366.122... 11251, 2023.
Qinshong Zhang and Yongxin Chen. Fast sampling diffusion models with exponential intogrator. In The Eleventh International Conference on Leurning Representations, 2023. URL httpps:/ net/
Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In International Conference on Machine Learning, pp. 42390- 42402. PMLR, 2023.
16

---

Published in Transactions on Machine Learning Research (06/2024)
A Appendix
A.1 Additional Experiment details
Model Architectures All pre-trained models utilized our experiments were obtained from the EDM framework (Karras et al., 2022). Specifically, for the CIFAR-10 dataset, we employed the NCSN++ model architecture as described by (Song al. 2021b). For experiments conducted on the ImageNet 64x64 dataset, we followed the architecture detailed by Dhariwal & Nichol 2021). For small student distillation experiment
CIFAR-10, the same archilecture as the teacher model in (Song et al. 2021b) was used with the number of channels reduced from 128 to 64.
Evaluation Metrics For Frechet inception distance (FID, lower is better) (Housel et al.) 2017a and In- ception Score (IS, higher is better) (Salimans et al. 2016), 50,000 generated images were compared against their respective ground truth datasets. Three different seeds were employed, and the best result was selected since FID values typically exhibit approximately 2% variance between measurements (Karras al., 2022).
pretrained EDM model (Krraa et al, 2022) as our teacher model. Unless stated otherwise, all the student models X, were initialized with the same weight as the pretrained EDM teacher model Dø- In the experiment involving the smaller student model, the student model was randomly initialized. Rectified Adam optimizer (Liu et al. 2020) was used for distillation with weight decay [Karras et al. 2022), we use the EMA weights of student model for inference. The EMA docay value for both CIFAR-10 and ImageNet is same, 0.99995. Additional details on training hyperparameters are shown All the experiments are run with Py'Torch (Paszke et al. 2019) on NVIDIA A100 GPU.
Training Details For both CIFAR-10 and ImageNet,
and a constant learning rate throughout the training iteration. Following
in Table 6
Unless explicitly stated otherwise, we utilized LPIPS as the default distance metric for training. When utilizing LPIPS as the distance metric d(::), the input images were rescaled to 244x244 using bilinear upsampling before being fed into the VGG model (Simonyan & Zisserman 2014).
Table 6: Hyperparameters used for the training runs.
Hyperparameter
CIFAR-10
ImagetNet 64x64
Number of GPUs
8xA100
32xA100
Batch size
512
2048
Gradient clipping
V
Mixed-preeisson (FP16)
V
Learning rate x1
Dropout probability
0%
0%
EMA student model
0.99995
0.99995
A.2 Proof for Lemma 1
Proof.
When LPID loss goes to O, since the distance metric has the property,
I = + d(x, y) = 0.
(12)
We have the following equality Vz N(0, I2 E [0, 1..,, N 2].
[z, t.) t; X (z 1,) (z l++1)
(x tt),
tit1
At,xo(z,t,) txxe(z, tx) + txx t(41) = At, (2, tz),tt where At, = [t+t1 tyl
(13)
(o (441) = =x((2, 12) - At; (xo(2((,44) 4))44)
t
Since this equality holds Vz NO, I 10. N 2], the trajectory model x2(z,tt) will have equivalent trajectories that obtained through a euler solver. As such, it will have the same discretization
17

---

Published in Transactions on Machine Learning Research (06/2024)
error bound, O(At).
A.3 PID with Model Compression
Table 7: FID table comparisons for small student model and large student model on CIFAR 10.
Model
Parameters
FID (4)
Teacher (EDM)
55.7M
2.04
Student (PID)
55.7M
3.93
Student (PID) 13.9M
8.29
As is common in Knowledge Distillation literature, this field often involves model compression where a teacher model used to produce a student model with comparable performance. In line with this paradigm, we train our model with the same hyperparameters as in the CIFAR-10 main results with less than a quarter of the model parameters. The small student architecture was constructed by reducing all the channels the teacher models by half. More details on the small student architecture are provided in Appendix A..] From Table 7 we observe that despite the student model's significantly smaller size, it is still able to generate decent images, achieving an FID score of 8.29. Despite its drop in performance in contrast to the bigger student models, this result showcases the promising potential of Knowledge Distillation approaches in model compression and not. just NFE reduction.
A.4 Insight on Parametrization Choice for Physics Informed Distillation
To satisfy the boundary condition, we require a parametrization choice such that:
xo(2, 1) Cskop (1) + Caue(1) Xo (Cin(T) 2, Cnoiss (4)),
(14)
where Cout (T) O.
A straightforward selection for this would be linear function, such as T - 1. However, for cases where T of the forward diffusion process is large, as in the case of EDM (Karras et al. 2022), such a choice would amplify the model outputs which may cause poor performance. As such, we choose:
T -
Cout (t)
(15)
T
to ensure that the model is multiplied by factor no larger than 1. For the skip connection function, Cout (t), let's begin by examining the solution for the provided Probability Flow ODE system:
X₁ 2 + /' X - Do(xx, 1')
t' dc'
(16)
Given that the constant function, Cskip 1, already meets its boundary condition at T, it might appear to be the obvious choice for the skip function. However, when considering Equation 16 and Equation 14 it can be seen that the model, Xo, when it solves the ODE system is such that:
1
X (x( dt
Xo(2,1)
(t)
T
t
(17)
T
Xx. Dx(xv, t') dt'.
At time = 0,
X 0)
10)
(
X
Do(xu,t') di.
t'
(18)
Xg
2.
18

---

Published in Transactions on Machine Learning Research (06/2024)
Given the contrasting magnitudes of Xu within the range [-1, 1] and which has significantly larger values due to its high variance, aligning our model, Xo(z, 0), with xo at = 0 emerges as superior choice. More specifically:
(z, xo
0 =2 +
X
Dz(Xx, 1) de'
(19)
TT
t'
Thus, for any arbitrary time, i, we desire our model be expressed as:
T T-t
["
X
Do(xx,, 1') de'
Xo(z,1)= = x
(20)
By plugging in the above formula into Equation 14 considering the choice of Cout well as the solution of the probability flow ODE in Equation 16
obtain the given skip connection:
Cakip (t) T
(21)
For the choice of functions, Gu (t) and Cnoise (t), we opted to use the same functions utilized in the teacher EDM model. Additionally, the time value of the in function, Cin (T), set T, given that the input consists of noise, representing the distribution corresponding to time T.
A.5 Trajectory comparisons
In Figure we present a comparison of the trajectories obtained from both the teacher EDM model and the student PID model. Notably, for the same noise seed, we observe that the trajectories represented by (z,) in the student model align remarkably well with those of the teacher model, with only minor aberrations. This demonstration underscores the ability of our model to predict all points along the trajectory in a continuous manner and not only the origin.
19

---

Published in Transactions on Machine Learning Research (06/2024)
cesssssSS
Figure 8: Trajectory comparisons on ImageNet with EDM teacher (top) and PID student (bottom).
A.6 Additional Random Samples
In this section, provide additional samples from PID model for ImageNet 64x64 and CIFAR-10 10. The images are obtained by employing the same class for ImageNet 64x64 and applying the same noise seed to both the teacher EDM and student PID models.
A.6.1 CIFAR-10
(a) Random samples from EDM
(b) Random samples from PID
(c) Random samples from PID (small student)
Figure 9: Unconditional image generation comparison on CIFAR-10 for the same seed. Left panel: random samples generated by EDM teacher model. Middle panel: generated by PID student model. Right panel: generated by small PID student model.
20

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Random samples from EDM (FID=2.04)
(b) Random samples from PID (FID=3.92)
21

---

Published in Transactions on Machine Learning Research (06/2024)
(c) Random samples from PID small student (FID=8.29)
Figure 10: Unconditional image generation comparison on CIFAR-10 for the same seed.
22

---

Published in Transactions on Machine Learning Research (06/2024)
A.6.2
ImageNet 64x64
Random samples from EDM on ImageNet (FID=244)
LUNVOD
(b) Random samples from PID on ImageNet (FID=9.49)
Figure 11: Conditional image generation comparison on ImageNet 64 x64 for the same seed.
23


Reviews:
[{"review_id": "bM2pjCh8i8", "reviewer": "Reviewer_9HBp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The key idea of the paper to use PINNs to model the ODE solution trajectories is well motivated and very naturally suited to diffusion models.\n\n- Experimental results on CIFAR-10 and ImageNet-64x64 show that high quality image samples can be generated with the proposed physics informed distillation.\n- \nThe paper is well written and easy to follow.\n\n**Weaknesses**\n\nI do not have any major comments on the paper’s weaknesses. A few minor comments are provided below.\n\n- I am not entirely sure why the paper uses numerical differentiation over exact gradients that can be obtained using autodiff. The paper cites Chiu et al. to suggest that automatic differentiation can result in unphysical solutions. However, this appears to be the case only when there is a lack of enough grid points. Is that scenario relevant for the experimental setup used in the paper? Could the authors please clarify this?\n\n- The authors mention a distillation approach, BOOT, in the related work that is similar to their approach but do not explain on how it differs from the proposed work. A better discussion on this would help better place this paper in the context of literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary of the work**\n\nDiffusion models are powerful ways of sampling from complex data distributions and provide superior performance over GANs and VAEs. However, since they require an iterative process to produce clean images from initial noise samples, their sampling speed can be slow.\n\nThis paper addresses this problem by using a distillation approach inspired from Physics Inspired Neural Networks (PINNs). The backward process of a diffusion model can be expressed in the form of a probability flow ODE. Instead of using an iterative solution, the paper uses a PINN to learn the solution trajectory of the ODE thereby resulting in a single step solution.\n\nThe effectiveness of their approach in generating image samples is shown with experiments on CIFAR-10 and ImageNet-64x64 datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VLUmvvg1pS", "reviewer": "Reviewer_vn5G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The primary strength of this work is the straightforwardness and naturalness of the proposed method. The authors observe a clear connection between PINN and diffusion distillation and show that standard ideas from each domain can be combined into an effective distillation method. I found this connection interesting and believe it could inspire future works in a similar direction.\n* The paper was well-written and easy to understand. It provided suitable context for readers without much background in PINNs (such as myself).\n* The experimental results demonstrate that the method can achieve reasonably strong single-step distillation results compared to relevant prior works.\n\n**Weaknesses**\n* The recent work BOOT explores similar ideas, as discussed by the authors. However, in my opinion, the formulation presented in the present work is somewhat clearer.\n* The method relies on LPIPS to provide a significant boost to image quality. In this case, the method is not entirely self-contained because it implicitly relies on the perceptual dataset used to train LPIPS. While this design choice is shared by provide works, it does complicate comparisons with other works. I suggest notating which distillation works use LPIPS and which do not use LPIPS when presented experimental results such as Table 1 and 2.\n* Several of the practical considerations complicate the otherwise clear method. In particular, the numeric 1st order gradients could be a bottleneck that prevent the method from achieve higher quality results. The use of stop-gradient seems somewhat arbitrary.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a Physics-Informed method for distilling a pretrained teacher diffusion model into a student diffusion model that can generate samples in a single step while maintaining sample quality that is close to the teacher model. The method is based on a synergy between Physics-Informed Neural Networks (PINN) and the ODE formulation that describes the sampling paths of diffusion models. Since PINNs are naturally suited for learning the evolution of arbitrary ODEs, the work observes that one can learn the entire diffusion ODE given an initial condition during training time and try to guess the final ODE state given an initial noise sample during test-time, thereby yielding single-step generation. Several practical considerations (residual formulation to deal with initial conditions, numeric 1st order gradients, re-arranged loss with stop gradient added to one term) are introduced. Experimental results show the method is competitive with distillation methods like Consistency Models, without requiring any samples from the teacher diffusion model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I only have minor change requests at the moment. I suggest that the authors annotate which methods use LPIPS and which do not when presenting their experimental results. I also would appreciate more discussion/interpretation of the use of stop-gradient.", "broader_impact_concerns": "Broader impacts are not discussed, although this does not affect my evaluation of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KB81Yc6Dyo", "reviewer": "Reviewer_chFg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nPID tackles a **relevant and challenging problem**. Reducing the inference cost of diffusion models is an important line of work with direct practical implications. It also helps researchers design more cost-efficient models, as for consistency models.\n\nThe paper is overall **clear, well-written and easy to read**. PID benefits from **relevant design choices**: the adaptation of PINNs to this problem is natural, and supported by some well-motivated changes to the original setting. This results in a **simple method**, which is commendable as it makes it easily applicable.\n\nThe experimental results demonstrate the **competitiveness of the approach**, as precisely claimed in the paper. While PID is not state-of-the-art, I believe it can still be interesting for the community via its simplicity, potential applicability, and potential follow-ups.\n\n## Weaknesses\n\n### Supporting Claims on Temporal Efficiency\n\nBesides the competitive results, one of the experiments' claim is that it has an advantageous training efficiency. This claim is **insufficiently supported**, for four reasons.\n1. The paper insists that methods such as DSNO are expensive because they necessitate synthetic data. This should be quantified as the paper does not provide any element of comparison.\n2. The only provided comparison deals with iteration speed (w.r.t. PD and CD). While this is useful information (provided that batch size is the same), this does not provide the full picture, i.e. full training time.\n3. I do not understand why the model needs less hyperparameter tuning (claimed in Sections 1 and 6.4). The choice of discretization level has an impact on the performance, and could have an impact on the required training time.\n4. In Section A.1, it is stated that all student models were initialized with the weights of the pretrained models. To ensure fair comparison, it should be clear that this is the case for all applicable models.\n\n### Related Work\n\nThe paper does not, or fails to, contextualize its contribution w.r.t. to two closely related methods. This should be fixed so that the contributions of the paper are clarified.\n\nThe first one is **BOOT** (Gu et al., 2023). It is cited in the paper but only scarcely discussed. It is said to be similar to PID but on a different ODE. This requires further explanations: is the ODE choice the only difference between both methods? is there any specificity/advantage of PID?\n\nThe second one is **CTM** (Kim et al., 2024). I know that this paper and this submission are almost concurrent so I will not judge the novelty of the submission on this point, but I believe this deserves some discussion and possibly a comparison in the experiments. Indeed, to my understanding, although presented in a different fashion, CTM encompasses PID (same principle of learning the solution, possibility to rely on a single ODE discretization time step, similar loss) and also generalizes it to intermediate initializations and multi-step generation.\n\nLess importantly, I would suggest the authors to include Jolicoeur-Martineau et al. (2021) and Karras et al. (2022) in the \"Training-free methods\" paragraph of their related work section, as they leveraged a second-order solver to reduce NFE of diffusion models.\n\nJolicoeur-Martineau et al. Gotta Go Fast When Generating Data with Score-Based Models. arXiv, 2021.\\\nKim et al. Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion. ICLR 2024. [public since October 1st, 2023]\n\n### Reproducibility\n\nThe paper's reproducibility could be improved.\n- There is no source code and no promise to release it.\n- Algorithm 1 does not specify the batching strategy: how are time indices samples in a batch (all the same or i.i.d.)?\n- The appendix contains no specification of hardware or software.\n\n### Justification of Numerical Differentiation\n\nThe use of numerical instead of automatic differentiation **should be motivated by empirical evidence**. I understand that Chiu et al. (2022) showed that automatic differentiation can lead to bad solutions for PINNs, but this concerns, to my understanding, the case of few sampling points. It remains to be shown that this problem transfers to the setting of the diffusion ODE, and an experimental result would show the advantage of the chosen approach.\n\n### Straightforward Theory\n\nI understand the motivation for a theory supporting the proposed approach, but in this case **it is not an important contribution** and does not deserve a full theorem.\n\nIn a nutshell, the result states that if PID perfectly imitates the Euler-approximated diffusion ODE (loss equals to 0), then the approximation error of PID is the same as Euler. This is straightforward and does not provide much insight. This result should be downgraded to a remark / trivial lemma supporting the soundness of the approach.\n\n### Minor Issues\n\n- I have reservations regarding the qualification of the method as \"physics-informed\". I understand the inspiration from PINNs, but there is no physics here.\n- $D_{\\phi}$ is not the teacher diffusion model, but the teacher denoiser network (p. 4).\n- Notations in Section A.2 differ from the main paper (in particular $\\mathbf{x_0}$).\n- Quality issues:\n  - the norm in Eq. (5) should be adjusted to the height of its argument;\n  - the second mention of BOOT (in Section 4.1) should be referenced to improve text clarity;\n  - \"discritization\" should be \"discretization\" (p. 9);\n  - references should be checked to cite the most recent version papers (e.g. Consistency models which was published at ICML 2023);\n  - in Section A.2, $x'\\_t$ should be $x\\_{t'}$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to distill a (multistep by nature) diffusion model into a single-step network (PID). This network takes as input the initial noise and the diffusion time, and imitates the whole probability flow ODE trajectory. Taking inspiration in the literature of Physics-Informed Neural Networks (PINNs), this network is trained so that its numerical temporal first variation approaches the score estimated by the teacher model at any time. Inference is achieved by querying the network with an initial noise and the final diffusion time.\n\nSeveral adjustments to the usual PINNs framework are proposed to alleviate issues arising from the probability flow ODE's particularities, in particular by changing the PINN loss and the parameterization of the student network. The generation performance and temporal efficiency of the proposed method are then assessed w.r.t. competitive methods and via ablation studies.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe that most of the issues I described above can be addressed in a revision.\n\n**Essential changes.**\n- Improved empirical evidence supporting the claims of temporal efficiency.\n- Completed discussion of related work.\n\n**Desirable changes.**\n- Improved reproducibility (code, more details in appendix, etc.).\n- Empirical justification of numerical differentiation.\n- Less focus on the straightforward theory.\n\n**Quality changes.**\nCf. description above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FIkmt8aSpU", "reviewer": "Reviewer_hbhX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method exhibits strong similarities to CD with minor modifications. Both focus on constructing a feedforward ODE solver whose dynamics align with a pre-trained diffusion model. However, while CD relies on ground truth samples from p(x_t) and training data p(x_0), PID acquires samples from the student network itself. Intriguingly, PID outperforms CD.\n\nI conjecture that the method's robustness might be linked to initialization—student network initialization heavily influences distillation success. In some ways, PID resembles forward KL divergence, where the optimized distribution (the student network) must generate samples for loss evaluation. Conversely, CD's behavior is closer to reverse KL with its sampling process. Improper student initialization could lead to local minima if initial samples significantly deviate from the teacher model's training distribution, where the teacher model lacks strong generalization.\n\nI'd appreciate the authors' insights on these points. Additionally, while inspired by the EDM work, the claim of 'Physics Informed Network' seems somewhat exaggerated as the core contribution lies in the denoiser network parameterization.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose Physics Informed Distillation (PID), a novel approach for distilling a single-step image generator from pre-trained diffusion models. This method offers the distinct advantage of eliminating the need for training images or sampled clean images, consequently reducing training costs. Empirical experiments validate the performance of PID, demonstrating its superiority over existing baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above. Would like to hear the authors' response.", "broader_impact_concerns": "No ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is strong with good numerical and theoretical evidence of the validity of the methods. The authors have answered most of the concerns of the reviewers. While it is not a state-of-the-art method in terms of distillation, this paper suggests that further experimental improvements could bridge the gap between the physics inspired PINN methods and consistency models. However, summarizing some of the points broudhgt by Reviewer chFg we would like the authors to update their manuscript on the following points.

* The details of training time comparison with DSNO should be included in the paper: GPU hours, training iterations time, bug in the public code.

* The remark on the reliance of CD in discretization parameters and the initialization of baselines should be in the paper (cf. Sections 1.3 and 1.4 of this response).

* A clean and easy to reproduce code as well as model checkpoints must be included in the final submission following the authors' promise.

* Following the downgrade of the theoretical part to a simple lemma, the theoretical contribution in the introduction (2nd item p. 2) should be reformulated and integrated into the first item. 

* The choice of LPIPS instead of a distance metric in Section 4.3 should follow the lemma. It should be stated explicitly that it is an implementation choice, as the lemma does not cover LPIPS (which is not a distance metric contrary to the statement in Section 4.1 - to be corrected).

* All these changes should be smoothly integrated into the flow of the paper. Most of the changes to the paper, especially the additional ablation studies, were simply appended to the paper, whereas a smoother integration in the different sections would improve the writing. In particular, additional ablation studies should not be subsubsections of Section 6.5, and some of them might be better put together with other subsections (Section 6.5.1 with Section 6.1 or 6.2, Section 6.5.4 with Section 6.4).
Decision: Accept (poster)