Optical Diffusion Models for Image Generation
Authors: Ilker Oguz
Niyazi Ulas Dinc
Mustafa Yildirim
Junjie Ke
Innfarn Yoo
QIFEI WANG
Feng Yang
Christophe Moser
Demetri Psaltis
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion based model
image generation
optical computing
efficient computing
Primary area: diffusion_based_models
TL;DR: Light propagation can be programmed to perform denoising diffusion efficiently by transmitting through learned modulation layers.

Abstract: Diffusion models generate new samples by progressively decreasing the noise from the initially provided random distribution. This inference procedure generally utilizes a trained neural network numerous times to obtain the final output, creating significant latency and energy consumption on digital electronic hardware such as GPUs. In this study, we demonstrate that the propagation of a light beam through a transparent medium can be programmed to implement a denoising diffusion model on image samples. This framework projects noisy image patterns through passive diffractive optical layers, which collectively only transmit the predicted noise term in the image. The optical transparent layers, which are trained with an online training approach, backpropagating the error to the analytical model of the system, are passive and kept the same across different steps of denoising.  Hence this method enables high-speed image generation with minimal power consumption, benefiting from the bandwidth and energy efficiency of optical information processing.

Optical Diffusion Models for Image Generation
Ilker Oguz'
Niyazi Ulas Dine' Mustafa Yildirim' Junjie Ke² Innfarn Yoo?
Qifei Wang² Feng Yang Christophe Moser
Demetri Psaltis
École Polytechnique Fédérale de Lausanne 2 Goooge Research "Google
filker. oguz niyazi dinc, mustafa. yataa.iidla wildirim, christophe moser, demetri. d pttperi...paatissspp...... (junjiek, innfarn, qfwang, fengyang)@googgle com
Abstract
Diffusion models generate new samples by progressively decreasing the noise from the initially provided random distribution. This inference procedure generally utilizes a trained neural network numerous times to obtain the final output, creating significant latency and energy consumption digital electronic hardware such GPUs. In this study, we demonstrate that the propagation of a light beam through a semi-transparent medium can be programmed to implement denoising diffusion model on image samples. This framework projects noisy image patterns through passive diffractive optical layers, which collectively only transmit the predicted noise term in the image. The optical transparent layers, which are trained with an online training approach, backpropagating the eтгог the analytical model of the system, are passive and kept the same across different steps of denoising. Hence this method enables high-speed image generation with minimal power consumption, benefiting from the bandwidth and energy efficiency of optical information processing.
Introduction
Diffusion models create new samples that resemble their training sets by gradually undoing the diffusion process, which requires the learned reverse process to be applied numerous times ①. While this method demonstrated unprecedented capabilities by producing highly realistic samples 2 it is also highly time-consuming and expensive in terms of energy consumption and computing resources since large number of steps are required for generating each sample (2). This prolonged processing time not only limits accessibility but also contributes to a significant environmental footprint.
Currently, generating new samples with diffusion models relies electronic, general-pune purpose computing hardware such as GPUs or TPUs. However, due to the repetitive nature of the reversal process required in this task, deploying specialized hardware instead of general- purpose significantly enhance the efficiency of sampling. For instance, the use of ASIC's in cryptocurrency mining for hashing algorithms has demonstrated substantial improvements in computational speed and energy efficiency [9]. However, both GPUs and ASICs, among other electronic digital computers, face the same challenges like heat dissipation, energy consumption, and the diminishing returns of Moore's Law, as transistors shrink and encounter quantum effects and physical limits that hinder further gains 10] Therefore, exploring alternative computing modalities, such optical computing- which offers high bandwidth and low loss-is increasingly important [IT]. Optical computing has already shown promise in various applications, including high-speed data transmission and real-time signal processing, Optical computing addresses the inefficiencies of electronic hardware by leveraging the inherent parallelism that light allows for the simultaneous processing of multiple data channels, significantly speeding up the computational process. Several optical neural networks have been
general-purpose
could
'Equal advising.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

reported to perform complex calculations at reduced latency and energy consumption compared to traditional electronic systems [2]
In this study, we demonstrate that optical wave propagation can be programmed to act as computing engine specifically designed for implementing denoising diffusion models. As light passes through specially engineered transparent layers, features related to the original distribution are filtered out without any additional power consumption or computing latency, as depicted in Figure [ This is due to the passive nature of the transparent layers, which are minimally absorptive and do not require active components or external power function. These layers are designed manipulate the light solely through their physical structure, which allows for the noise prediction to exit the system efficiently. Near zero energy consumption of passive optical components reduces the overall power requirements, making the system more energy-efficient.
Through iterative noise prediction and removal, the Optical Denoising Unit (ODU) can generate new images using a minimal number of these passive optical modulation layers. Since these layers do need power or active control, they do introduce any latency or energy overhead. Constrained only by optoelectronic input and readout hardware, this approach has the potential to significantly reduce the computational time and energy consumption of diffusion models, specifically performing inference in more sustainable and scalable ways.
Optical Denoising Unit
Denoising NN
Optical Detector
9
3
Digital Electronics
Optical Modulator
Programmable
Passive Optics
Conventional Diffusion Model Sampling
Optical Diffusion Model Sampling
Figure 1: Comparison between conventional and proposed methods of image generation based on diffusion models. The conventional method runs on digital electronics based computing units such as GPUs TPUs. The proposed method utilizes an optical denoising unit that formed by passive optical layers. The image to be denoised is sent to the system with a modulator and the output is read with detector.
The main contributions this study are:
The propagation of light through multiple modulation layers is programmed to perform denoising diffusion image generation by predicting and transmitting the noise term in the input images. The system uses only single modulation plane and multiple reflections.
A time-aware denoising policy is specifically designed for analog optical computing hard- ware. This policy facilitates the use of passive building blocks to achieve multi-step comput- ing at low power, translating the time-embedding in digital Denoising Diffusion Probabilistic Models (DDPMs) into optical hardware.
An online learning algorithm introduced for training ODUs real-life scenarios, where alignment and calibration errors exist. The algorithm tracks and alleviates experimental discrepancies with constant updates to a digital twin (DT) during training time.
2 Related Work
Diffusion models have become popular, with their superior image generation performance compared to Generative Adversarial Networks (GANs) [3] High-resolution, guided diffusion process is

---

curretly widely utilized for on on-demand image generation 6 7 14]. One of the main concerns with these highly capable models is the significant time taken for generating new samples, which can exceed 10 seconds for each high-resolution image [6]. Different methods have been proposed to alleviate this condition and improve the efficiency of diffusion models. While Latent Diffusion Models [6] work on a lower dimensional representation of images to decrease the computational load, Denoising Diffusion Implicit Models [15] introduce a deterministic and non- non-Markovian sampling process to reduce the number of required steps. Similarly, FastDPM [16,, uses domain-specific conditional information for faster sampling with diffusion models. Another approach is to distill multiple denoising steps single with teacher- student setting [17]. As these methods aim to decrease iterative denoising steps required for sampling through algorithmic innovations, the potential improvements obtained by exploiting the repetitive nature of these models on the computing hardware side remain to be seen.
Optical processors have shown substantial energy efficiency improvements, particularly with larger model sizes, potentially outperforming current digital systems [8]] They can be implemented through various architectures, each leveraging different aspects of optical technology perform computations. Free-space optical networks use spatial light modulators or fixed modulation layers for performing matrix multiplications and convolutions as light propagates [19H21], making them highly efficient for image processing tasks. These applications include super-resolution [22), noise removal [23), and implementation of convolutional neural network layers (24). which are shown to
competitively perform different downstream tasks such as segmentation 25 On the other hand, photonic integrated circuits utilize optical components like Mach-Zehnde interferometers, microring resonators, and waveguides on a single chip. enabling compact vector-matrix multipliers [26, 27 Together, these developments highlight the transformative potential of optical computing in enhancing the performance and efficiency of computationally intensive tasks.
Considering the significant computational demands of denoising tasks, there is a clear need for specialized hardware to scale these operations effectively. Despite the advancements in optics, deep learning, and optical image processing, the realization of an optical diffusion denoiser remains gap in current research. Bridging this gap could leverage the synergy between these fields to develop highly efficient and scalable solutions for denoising diffusion models.
In addition to their wide range of advantages, optical computing systems also have disadvantages related to the energy cost of modulation and detection of light, its limited programmability. and experimental precision. Calculating the gradients of the experimental loss through the optical wave propagation model allows for close match between optical experiments and computational models [9]. Similarly, a pre-trained neural network-basee emulator of a physical system can also be used for the same purpose (28). Moreover, it is crucial to perform as many computations as possible with the data while the optical domain to avoid energy and time expenditure of optoelectronic devices.
3 Description of the Study
3.1 Denoising Diffusion Models
DDPMs progressively corrupt data with Gaussian noise in forward process and subsequently learn to reverse this corruption through a denoising process. This way they can generate new data samples that closely resemble the training data distribution. The forward diffusion process involves the sequential corruption of a data sample To q(ro) through the addition of Gaussian noise over T timesteps. At each timestep I, the data sample rx-1 1 is perturbed to produce rt. Tt = VI Brr.-1 1 + VBee where 3e E (0,) 1) is a variance schedule that determines the amount of noise added and e, ~ No00, I) is standard Gaussian noise. This process transforms the original data into nearly pure noise by timestep T.
The reverse denoising process in DDPMs aims reconstruct the original data from highly noisy sample. Starting from completely Gaussian noise rr ~ I), the sample is iteratively denoised by removing the prediction of e, in the image, co (re, t), which is provided by a trained neural network:
Tt-1
(r+ B,eo(re,t)),
(I)
The training objective of the neural network can be simplified to minimize the mean squared error (MSE) between the true noise e and the predicted noise ea (r f) C Erro. [ller ce ce(re.
3

---

where t is uniformly sampled from {1..... , T). Finally, to generate new data samples, the model starts with sample rr NO, I) and applies the learned reverse transitions iteratively.
3.2
Propagation of Modulated Light Beams
Original
Image, Pt-1
Combined Input Pattern
Trained Modulation Patterns
Predicted Noise
l ())
Lx(x,)
Lz(x,))
Lx(x,))
Ztayer
x h
h
U2 U3
Uy y) ea(r) ||0, (x,
Inserted Noise, Er
Un (U,-r(x,y)> h(x, (x,y)
Figure 2: The main operation principle of ODU. Consequent modulation and free space propagation events can be represented with multiplication and convolution operations. When the input beam (1 y), which is patterned with noisy input images, is introduced to the ODU, the output intensity pattern ( y) corresponds to the trained optical system's prediction of the noise component in the input pattern, ca (rr))
In this study, denoising framework is presented by combining the modulation of a light beam with consequent transparent reflective patterns and its propagation in free space (environments such as vacuum or air, where the refractive index of light is approximately 1), as shown in Figure This process can be explained by the Fresnel diffraction theory since the features on the layers not only larger than the optical wavelengths but only sufficiently smaller than the distance between different modulation layers [29] According to this formalism, the electromagnetic field after propagating distance 2 in free space, Uxx y, 2), can be calculated from its distribution at = 0 by convolution with "the impulse response of free space". h(2, y):
U1r,y,2 = (zx 0) * h(x, where h(x,y) = iAz
eukz
jk 2z
(x² + +X²)
12]
(2)
Here, A denotes the wavenumber of the field, and À is the wavelength. In other words, the field's value the plane of = 2o, at a given location (æ; y). is the weighted sum of the values at z = O. where the weight of each location is determined by the response function. Being complex numbers, all of these weights have the same magnitude but their phase depends on the location. In the frequency domain, the transfer function of free space becomes
H (Jx,fy) exp j2m V (ASx)² (ASv)²
-
This indicates that for spatial frequencies larger than 1/2, the magnitude of the transfer function decays to zero exponentially. Hence, only features that are larger than the wavelength of the light can propagate to the far field. Morcover, frequency domain expression of diffraction, Eqn. 3 allows for also the efficient digital simulation of the propagation of light in free space with the utilization of Fast Fourier Transforms (FFTs) in a parallelized manner. Later on, we will benefit from this fact for GPU accelerated training of the diffractive modulation layers.
The proposed method applies trainable weights to the light beam at consequent planes with thin modulation layers. The interaction between layers and light can be represented wise multiplication between the incident field and the layer, which followed by the propagation of the field in free space until the next layer,

---

(z. [Un-1 v) * h(x, (x,)))).))
where (2. y) is the field distribution right before reaching the modulation layer and (x, y) is the complex modulation coefficient distribution of the trained modulation layers or the weights of the optical diffusion model. L(x, y) = [xx, e) can be only a real number(0(2; y) 0), just phase modulation (I, x)] = 1) or an arbitrary complex number depending on the implemented modulation principle. In this paper, we demonstrate our approach with phase-only liquid crystal spatial light modulator (SLM), which can set o(x, y) to any value in the range of [0, 2 electronically, and (xr, y) everywhere.
3.3 Training of Optical Modulation Layers
As descrihed in section 3.2 propagation of light can be analytically explained in a succinet manner for the scale considered in this study (Stager >> dpinet A). This allows for defining some free variables in this representation, such as refractive index distribution L(z, y) or input wavefront distribution Uo (ar, y). and optimizing these variables for minimizing a cost function. The gradients of these variables can be found with either manual calculation [30] or automatic differentiation packages 20). In this study, our first goal to find optimal modulation layers, or refractive index distributions, such that after the light beam encoded with the noisy images propagates through them only the predicted noise term reaches the detector, as shown in Figure 2 Moreover, the denoising network should be aware of the given timestep in the diffusion process while predicting the noise, eo (2 t), so that it would have a priori information about the variance of the noise term. Since noise level awareness is crucial aspect of successful sample generation, most of the current implementations of diffusion models utilize time-embeddingg layers to modify activations of the neural network across different layers depending on the diffusion time step. Instead, the proposed method divides the diffusion timeline consisting of T timesteps into M subsets, and for each subset of time frames {Sm m== trains separate set of modulation layers {L Jn=1 each containing N layers. Then, for E Sm. the noise prediction.ce., (x) becomes only a function of x. In this scheme, the training objective for each time step is
c. = Exu.c [ile c (x
(5)
where total loss is the sum over all ranges:
M
C
C..
(6)
i= SE
This decoupling of denoising at different timesteps by removing time-embedding layers also elimi- nates the necessity for digital computations to modifications at different layers. By circumventing this problem we perform denoising all-optically. Moreover, a fixed optical modulation pattern performs denoising at multiple consequent timesteps. For instance, we later demonstrate that for T = 1000, M = 10 creates optimal results. So, a single layer set can process 100 timesteps and the entire sampling workflow can be operated with only 10 fixed parallel devices, or with only 10 updates to the SLM.
After defining the forward calculation of the system with the analytical explanation of light prop- agation and the loss function as the mean square error of noise prediction (Eqn. (6). the trainable parameters of the system {L J (I, 3) are optimized by automatic differentiation B1
4 Results
4.1 All-Optical Denoising based Image Generation
Following the same experimental settings with the initial DDPM study 22 we set T = 1000 and values to be in the linear range between 3 10-4 to Br = 0.02. The results Figure are reported with the beam propagation model (Eqn. 3) of the optical system designed to have
300 x 300 pixels per layer and four modulation layers. The number of layer sets (M) is 10. Several
intermediate results alongside final outputs at T = 1000 are reported in Figure 3 for 3 classes of the
5

---

Fashion-
Quick, Dans!
MINIST
f
/
2
2
250
400
2.00
390
175
1.5
250
125
0
250
500
750
1000
250
500
750
1000
Denoising Steps (T-
Denoising Steps (T
Figure 3: Images generated by the Optical Diffusion Model at different timesteps and when trained with various datasets. The generated images and their corresponding Inception and FID scores are calculated between timesteps T 10 to T 950 are acquired after training with the MNIST digits dataset. Final outputs at time 1000, acquired from ODUs trained for the MNIST digits samples have FID 206.6, for Fashion MNIST, FID 227.7 and for Quick, Draw!, FID 131.4
MNIST digits (32), Fashion-MNIST 33 and the clock category of the Quick, Draw! datasets (34). Furthermore, the evaluation of image generation quality metrics, Inception Score (IS) and Fréchet Inception Distance (FID), which are detailed in Appendix A.6 across different generation timesteps captures the improved realism of images with the optical diffusion procedure.
4.2 Effects of Optical Model's Dimensionality the Image Denoising and Generation Performance
3x
2.9 10-1
FC Conv QDU
Conv ODU
x.4 10-1
2.5 10
MSE Score
10
FID Score
10
1.5 x 10
20 Output Resolution
14
20
28
Resolution
Figure 4: Scaling of the denoising capabilities (left) and generation performance (right) of Optical Diffusion, and pure digital convolutional U-Net and fully connected networks with the output image resolution.
This section provides further analysis with different output dimensions and parameter counts along with the comparisons with purely digital implementations quantify Optical Diffusion's 's scalability to

---

large-scale diffusion problems. The first investigation is into the performance with higher resolution datasets; two digital architectures of similar performance with the ODU, one being fully connected and the other convolutional U-Net [5 are trained for the same tasks with the ODU, generating images with the MNIST digits dataset at different resolutions. Their architecture is detailed in Appendix Table The results shown in Figure |
indicate that ODU consistently outperforms the two digital neural networks, and all three scale in a similar manner both in terms of denoising and generation performances when the generated image dimension is increased while the model sizes are kept constant.
Nuso Prod. MSE
Nolse Prod. MBE
Nokse Prod, MSE
0.6
0.35
FR)
340
HD
300
Noise Pred MSE
0.5
Noise Pred. MSE
0.30
Noie MSE
0.110
0.4-
280
0.25
280
300
2
0.105
260
280
0.3
260
260
240
0.15
0.095
240
0.1
0.10
220
0.090
220
100
200
300
3
3
6
10
16
33
Layer Resolution [pixels]
Layer Count
Timestap Sets
0.50
0.400
KD
2.6
0.45
2.50
0.40
2.25
2.2
Inception Score
0.350
2.4
0.40
2.00
2.0
22
0.30
0.35
+
1.75
1.50
Inception
0.30
1.6
0.275
0.300
2.0
0.25
1.25
0.25
0.225
0.20
1.00
0.20
0.200
1.5
100
200
300
2
30
10
Layer Resolution [ptnels]
Layer Count
Timestep Sets (M)
Figure 5: The dependency of denoising performance (MSE) and generation quality scores(FID,, KID and Inception score), on the hyperparameters of the ODUs (number of pixels of optical modulation layers, number of modulation layers and number of denoising layer sets (M)).
Secondly, the scaling of Optical Diffusion's 's performance with respect to the number of total parame- ters is probed through three hyperparameters: layer resolution, layer count, and timestep sets. The effects are tracked with different metrics, MSE for denoising, FID, IS and Kernel Inception Distance (KID) for the generation quality. In Figure we observe that, as in digital neural networks, there is a clear tendency to perform better with larger number of trainable parameters when layer resolution and layer count are increased.
On the other hand, having a larger number of denoising layer/timestep sets improves the results only until they reach a certain level. Afterward, increasing the number of sets is detrimental as shown in Figure As the total number of training steps is fixed in this experiment, increasing the number of timestep sets decreases the training sample count per layer set, hence potentially deteriorating the performance after a particular threshold, which is found to be M =1=10. 10.
Through the aggregation of data points acquired with different layer resolutions and counts in Figure 5 the relationship between the total trainable parameter count of ODUs and their image generation performance is depicted in Figure 6 This relationship remarkably follows the same widely accepted, power-law trend of digital generative models [36]. Most significantly, when the optical implementation fitted to a power-law equation, the exponential of the power law (-0.15) is approximately the same as the reported value (-0.16) for singe-scale image generation networks in [36]. This fit parameter gives the slope of the line in the logarithmic plot, indicating how fast the generation performance scales with the number of parameters, in this case showing that ODU improves its performance at similar speed with large-scale digital image generation networks while its parameter count is increased. The single outlier in this trend is the case where there is only single modulation layer, which does not benefit from the multiple optical modulations aspect of the proposed architecture.
7

---

3
1-layer
32x
3
RID Score
10
2.4 10
Power-Law: 2083x-9.3
ODU Score
10a
Parameter Counts
Figure 6: The relationship between the total number of parameters in an ODU and its generation performance in terms of FID scores.
4.3 Higher Experimental Fidelity with the Online Learning Algorithm
To address the challenge of training an optical system with imperfect calibration, as faced in many other analog computing paradigms, we propose an online learning algorithm that updates and leverages DT during training. During inference, the DT does not incur any additional overhead. The DT 0 again utilizes Fresnel diffraction based model light propagation, as surrogate to compute gradients and guide the optimization of the system's 's trainable parameters. However, matching the DT's parameters (Oaligumen)). for instance, input beam angle, precise locations of the layers, and their angles, perfectly the experimental conditions of the physical system is challenging task. Therefore, during each iteration training, the output of the experiment (fe and DT (S dea Baigueen) is compared and the DT's parameters are updated accordingly, as shown in Algorithm
Algorithm Online Learning Algorithm
Initialize physical system fense with parameters Ouyers
Initialize DT f0 :0 with parameters Bluyers altggmment
while not converged do
Forward Pass:
Input data into the physical system sormo
Obtain physical system output y fureen
Compute error E = loss(y,,. (urget))
Backward Pass:
ay Bawer
Compute Jacobian of DT at x,
Compute gradients Vonger = JJ
0E
Update physical system parameters Owayrss + Otayers V Puyer
DT Refinement:
Obtain DT output
Compute MSE between DT and physical system outputs L = MSE(Y F.)
Compute gradients V a0gmme
Update DT alignment parameters fllggment end while
aaiggmment
ax V aagmon
In parallel, the DT is also employed to compute the gradients of the trainable parameters of the
By experiment, with respect to the output( ōbhaas These gradients are then utilized to update the physical system's parameters through backpropagation, informed by the enor obtained from the physical system. Concurrently, the DT refined using the latest inputs and outputs from the physical system to better approximate its behavior, despite the initial parameter mismatches. This iterative process of

---

forward and backward passes, coupled with the continuous refinement of the DT, enables the physical system progressively improve performance and align more closely with the desired outcomes.
Digital Twin
Optical Inference
Experiment Error Backprop.
Digital Twin Refinement
Input
Experiment
Output
Offline Train, MSE
2.0
Exp. Backprop.
1.0
Online MSE
1.5
Dischee
Offiine Exp. MSE
MSE
Offline Discrepancy
Exp. Backprop, Discrepancy
Online Discrepancy
0.4
0.5
0.0
0.0
10
20
30
40
Epoch
Figure 7: The upper block illustrates the online training scheme. The forward pass is calculated with experiment (blue), while the gradients of the prediction error are backpropagated using DT of the experiment (green) and updating the physical trainable parameters. The difference between the outputs of the experimental setup and the DT continuously refines the DT (red). The lower graph block compares offline and online training methods. Offline training relies on pre-traine pre-trained DT for both the forward and backward passes, with the experimental performance of this method indicated the star. Experimental backpropagation executes the physical forward pass but does not incorporate DT refinement.
In Figure 7 scenario. this scenario, we define two different optical models, while actually both of them are simulations of the optical wave propagation for an exact insight into the algorithm, we designate the first one as the "optical experiment" by configuring with the calibration angles obtained from the physical experiment. These four angles account for the slight misalignment of the experiment and define the input angle the beam to the cavity and the angle between the mirror and the SLM, in x and axes, all being in the range of few milliradians and their measurement details being provided in Appendix I The second model, considered as the DT, initialized with their calibration angles 20% higher. We used 3 different algorithms, offline, experimental error backpropagatton 28), and the proposed online training schemes. During training, MSE (training loss), and the diserepancy between the DT and the experiment's 's outputs are tracked. The discrepancy, D, is inversely related to cross-correlation, C, of the experimental and the DT's normalized outputs, Oezp and O respectively, D =1 C, where C C E. E, Oesp (x, y 0) y)-
we explore the efficiency of the proposed algorithm by modeling a possible experiment
Offline training improves the loss function when evaluated with the DT, but when evaluated in the experimental setting, it has higher MSE, as shown with in Figure 7 When the online training method is used, the DT is aligned with the actual experiment swiftly and the experimental loss approximates the MSE of perfect calibration case, the results of this approach with the optical experiment are also provided in Appendix Figure TO, Backpropagating the experimental loss, MSE does not decrease significantly. However, for smaller misalignment, this method was also demonstrated converge. This experiment implements multiple modulation layers single device,

---

with single phase-oni only SLM and a mirror in parallel to resource-efficienly prototype the proposed computing method, shown in Figure 7 and detailed in Appendix A.3
5
Conclusion
In this study, we introduced an optical diffusion denoising framework for image generation, utilizing time-aware denoising strategy that enables optical low-power realization. By exploiting light propagation through transparent media, ODU effectively reduces noise in images, with much smaller energy budget compared to electronics since optical wave propagation has very small intrinsic loss while acquiring comparable, or better quality. This is especially interesting because diffusion models are currently one of the most costly generative artificial intelligence models due to their repetitive denoising process, with correspondingly large environmental impact (37]
The integration of time-aware policy enables Optical Diffusion to adjust light modulation dynam- ically according to different stages of the denoising process, thus improving image quality with a minimal number of changes to the modulation layers or parallel optical processing units. Looking ahead, the incorporation of larger modulation layers with more parameters and the exploration of nonlinear optical effects could enhance the functionality of the system. Scaling analyses also show evidence for the ODU to improve its performance at the same rate as digital models while increasing its size. These potential improvements suggest promising directions for future research in expanding the range of applications for this technique.
The proposed method can utilize off-the-shelf consumer electronics such digital micromirror devices that can be found in portable projectors for input modulation and CMOS cameras for recording output prediction. The online learning algorithm accounts for variations between these devices and closes the gap between the analytical and experimental realization of the ODU. On the other hand, as analyzed in detail in Appendix A.4 these devices have the potential of implementing denoising steps on the order of microsecond latencies while consuming few Watts only. With the utilization of high-speed light modulation technologies, million frumes-per-seconee nevell denoising can be achieved again with Watt level energy consumption, while still utilizing the proposed approach for predicting noise in provided images 38
Acknowledgments and Disclosure of Funding
This work was supported by Google Research under grant number 901381. We would like to thank Research Computing Platform EPFL and Google Cloud Services for providing access to compute units.
References
[1] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics," Proceedings of the 32nd International Conference on Machine Learning. PMLR, Jun. 2015, pp. 2256-2265. iSSN: 1938-7228.
[Online]. Available: httpss/mooeeeeinns.mmrrrrrssss//
15.tmll
[2] J. Ho, A. Jain, and P. Abbeel, "Denoising Diffusion Probabilistic Models," " in Advances Neural Information Processing Systems. vol. 33. Curran Associates, Inc.. 2020. pp. 6840-6851, [Online]. Available: Abbtaaaabb-btttt......m
Ac5befec35544tt0466777
[3] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score- Generative Modeling through Stochastic Differential Equations." Oct. 2020. [Online]. Available: https://openreview.. nwt/foorummddd=PII1111111 httprr/sssstttataannagng-.-.P.. 20News&uum mediun=emailkuum
[4] A. Q. Nichol and P. Dhariwal, "Improved Denoising Diffusion Probabilistic Models," in Proceedings of the 38th International Conference on Machine Learning. PMLR, Jul. 2021, pp. 8162-8171, iSSN: 2640-3498. [Online]. Available: https:/ponceednng....ppess///
nichol2 ha.h
10

---

[5] A. Jalal, M. Arvinte. G. Daras, E. Price, A. Dimakis, and J. Tamir, "Robust Compressed Sensing MRI with Deep Generative Priors," " Nov. 2021. [Online]. Available:
[6] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-Resolution Image Synthesis with Latent Diffusion Models," " in 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA, USA: IEEE, Jun. 2022, pp. 10674-10685, [Online]] Available: https///iceepplor.eeee.o//// icee
[7] C. Saharii,, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, R. Gontijo-Lopes, B. K. Ayan, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi, "Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding," " Oct. 2022. [Online].
Available: http:://pppneevee..e///ommmmmimo
Yk-
[8] C. Saharia, Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, "Image Super-Resolution via Iterative Refinement," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 4713-4726, Apr. 2023, conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence. [Online]. Available:
[9] Küfeoğlu and M. Özkuran, "Bitcoin mining: A global review of energy and power demand," Energy Research & Social Science, vol. 58, p. 101273, Dec. 2019. [Online]. Available:
[10] C. Leiserson, N. C. Thompson, J. S. Emer, B. Kuszmaul, B. W. Lampson, Sanchez, and T. B. Schardl, "There's plenty of room at the Top: What will drive computer performance after Moore's law?" Science, vol. 368, no. 6495, p. eaam9744, Jun. 2020, publisher: American Association for the Advancement of Science. [Online]. Available: http://www..ciincc..rr/////////o htwp://o...../cccccc.aaaaaaa 944494
[11] G. Wetzstein, A. Ozcan, S. Gigan, S. Fan, D. Englund, M. Soljačić, C. Denz, D. A. B. Miller, and D. Psaltis, "Inference in artificial intelligence with deep optics and photonics," Nature 2020 588:7836, vol. 588, no. 7836, pp. 39-47, Dec. 2020, 101 citations (Semantic Scholar/DOI) [2022-01-233 -13] Publisher: Nature Publishing Group. [Online]. Available:
http://wwww.natuu..com//aalllll/
[12] P. L. McMahon, "The physics of optical computing," Nature Reviews Physics, vol. 5, no. 12, pp. 717-734, Dec. 2023, publisher: Nature Publishing Group. [Online]. Available:
https://www. www.://ntu.....m wwweeee.ccccaaaaaaeeeeeeeeeeeeee
[13] P. Dhariwal and A. Q. Nichol, "Diffusion Models Beat GANs on Image Synthesis," Nov. 2021. [Online]. Available: hntps///pparreeee.eee//oomm http://wwwwwweee/ooooo/oeeecaaaa AWucAvaavva
[14] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, "Hierarchical Text- Conditional Image Generation with CLIP Latents," Apr. 2022. [Online]. Available: https://ariv.orr//bs//0....0
[15] Song, C. Meng, and S. Ermon, "Denoising Diffusion Implicit Models," Oct. 2020. [Online]. Available:
[16] Z. Kong and W. Ping, "On Fast Sampling of Diffusion Probabilistic Models," Jun. 2021,
arXiv:2106.00132 [cs]. [Online]. Available: http://arav..o//////006....11111
[17] T. Salimans and Ho, "Progressive Distillation for Fast Sampling of Diffusion Models," Jun. 2022, arXiv:2202.0051 12 [cs, stat]. [Online]. Available: http://awww..or//////2000..00000
[18] M. G. Anderson, S.-Y. Ma. T. Wang, L. G. Wright, and L. MeMahon, "Optical Transformers." Feb. 2023, arXiv:2302. 10360 [physics]. [Online]. Available: http://arrivv.rg//ab/2333.
[19] T. Zhou, X. Lin, J. Wu, Y. Chen, H. Xie, Y. Li. J. Fan, H. Wu, L. Fang, and Q. Dai, "Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit," Nature Photonics, vol. 15, no. 5, pp. 367-373, May 2021, number: 5 Publisher: Nature Publishing Group. [Online]. Available: https:/wwwwnatturr www//aticcls/s//411/11 156.eee//ss/16661-. 021-00796
[20] X. Lin, Y. Rivenson, N. Yardimei, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, "All-optical machine learning using diffractive deep neural networks." Science, vol. 361, no. 6406, pp. 1004 1008, Sep. 2018, 523 citations (Semantic Scholar/DOI) [2022-01-23] Publisher: American Science_eprint:: 1804.08711.
Association for the Advancement of Science _eprint:

---

[21] J. Hu, D. Mengu, D. C. Tzarouchis, B. Edwards, N. Engheta, and A. Ozcan, "Diffractive optical computing in free space." Nature Communications, vol. 15, no. 1, p. 1525, Feb. 2024, publisher: Nature Publishing Group. [Online]. Available:
[22] 1511, D. Mengu, Y. Zhao, A. Tabassum, Li, Luo, M. Jarrahi, and A. Ozcan, "Super-resolution image display using diffractive decoders," Science Advances, vol. 8, no. 48, p. eadd3433, Dec. 2022, publisher: American Association for the Advancement of Science. [Online]. Available: https://www..sieene.rrg///ooooot http:////.....1.00..ooooa.addaad
[23] Isıl, T. Gan, F. O. Ardic, K. Mentesoglu, J. Digani, H. Karaca, H. Chen, J. Li, D. Mengu, M. Jarrahi, K. Akşit, and A. Ozean, " "All-optical image denoising using diffractive visual processor," Light: Science & Applications, vol. 13, no. 1, p. 43, Feb. 2024, publisher: Nature Publishing Group. [Online]. Available: hutps://www.naturee http://wwwwuuueoeooooooooo/////0
[24] Chang, V. Sitzmann, X. Dun, W. Heidrich, and G. Wetzstein, "Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification,"" Scientific Reports, vol. 8, no. 1, p. 12324, Aug. 2018, number: Publisher: Nature Publishing Group. [Online]. Available: http:://www.atttrrrccoootttttttt
[25] K. Wei, X. Li, J. Froech, P. Chakravarthula, J. Whitehead, E. Tseng, A. Majumdar, and F. Heide, "Spatially varying nanophotonic neural networks," arXiv preprint arXiv:2308.03400 2023.
[26] Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, "Deep learning with coherent nanophotonic circuits," Nature Photonics, vol. 11, no. 7, pp. 441-446, Jun. 2017, 925 citations (Semantic Scholar/DOI) [2022-01-23] Publisher: Nature Publishing Group _eprint: 1610.02365.
[27] A. N. Tait, T. Ferreira de Lima, M. A. Nahmias, H. B. Miller, H.-T. Peng, B. J. Shastri, and P. R. Prucnal, "Silicon Photonic Modulator Neuron," " Physical Review Applied, vol. 11, 6. p. 064043, Jun. 2019, publisher: American Physical Society, [Online]. Available: 1wt.skeevpppled.111.66600
http::/link.aps.. www.pppp.oo////......oo http://www.....oooooooooo//////a
[28] L. G. Wright, T. Onodera, M. M. Stein, T. Wang, D. T. Schachter, Z. Hu, and P. L. McMahon, "Deep physical neural networks trained with backpropagation." Nature, vol. 601, no. 7894, pp. 549-555, 555, Jan. 2022, number: 7894 Publisher: Nature Publishing Group. [Online]. Available:
https:/wwww.naturcc
http://www..nttuttcooo//////ss/s wtt.c/oootaaaccccccce 158--0241160111111 021--022--44
[29] J. Goodman, Introduction Fourier Optics, 4th ed. New York: W. H. Freeman, May 2017.
[30] U. S. Kamilov, I. N. Papadopoulos, M. H. Shoreh, A. Goy, C. Vonesch, M. Unser, and D. Psaltis, "Learning approach to optical tomography," Optica, vol. 2, no. 6, pp. 517-522, Jun. 2015, publisher: Optica Publishing Group. [Online]. Available:
[31] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. Vito,,, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, "Automatic differentiation in PyTorch," Oct. 2017. [Online]. Available:
[32] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-basee learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
[33] H. Xiao, K. Rasul, and R. Vollgraf. (2017) Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms.
[34]
Oct. 2024, original-date: 2017-05-09T11:22:322 [Online]. Available: http:///shlbbboooooooporrarraaat
[35] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in Medical image computing and computer-assisted interventionMICCAA 2015: 1 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer, 2015, pp. 234-241.
[36] T. Henighan, Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhari- wal, S. Gray et al., "Scaling laws for autoregressive generative modeling," arXiv preprint arXiv:2010. 14701, 2020.
[37] S. Luccioni, Y. Jernite, and E. Strubell, "Power hungry processing: Watts driving the cost of ai deployment?" in Proceedings of the 2024 ACM Conference on Fairness, Accountability, and
12

---

Transparency, ser. FAceT '24. New York, NY. USA: Association for Computing Machinery, 2024, p. 85-99. [Online]. Available: httpss//dooorg/10001444/66600000
[38] C. L. Panuski, I. Christen, M. Minkov, C. J. Brabec, S. Trajtenberg-Mills, A. D. Griffiths, J. J. D. McKendry, G. L. Leake, D. J. Coleman, C. Tran, J. St Louis, J. Mucci, C. Horvath, S. F. Preble, M. D. Dawson, M. J. Strain, M. I. Fanto, and D. R. Englund, "A full sfgree-of-freedonn spatiotemporal light modulator," Nature Photonics, vol. 16, no. 12. pp. 834-842, Dec. 2022, publisher: Nature Publishing Group. [Online]. Available:
J. N.
13

---

A
Appendix
A.1 Differentiable Modelling of Light Propagation in ODUs
To benefit from parallelized and optimized FFT algorithm and automatic differentiation, the wave propagation in the proposed system is modeled in PyTorch environment with a Split-Step Fourier formalism which is derived from Eqn. 2 The diffraction step of the propagation is calculated with nonparaxial diffraction kernel [ in Fourier domain and effects such as reflection or modulation of the light beam with layer parameters applied in the spatial domain. Such that the electric field after propagating a distance As becomes:
(4+(1)+13) k 14 18
E( +()) -
In addition to the parameters of layers L (ar, y) (or simply Olayers); the spatial term R(x, y) can include the angle changes of the beam. For instance, if the beam is not perpendicular to the SLM or the mirror, the reflection creates change in the angle of the beam, Aa (Oa,, Oty). Then, on the SLM plane Rm, (xr, v) Lm (Ir u)e - jk(2 sin a, tysini ay), where Rm (2, y) is the compound spatial term, Lm (z. y) is the modulation parameters layer m, and e =jk(zz o + tysn is the operator that changes the direction of the wave propagation vector. Similarly to the SLM, also the angle of the mirror determines the propagation direction of the beam, which can be included in the model as Rmimoor (Ir y) e jkk( sin o. +y sin To calibrate the model of the experiment with the actual experiment, we define three trainable alignment parameters, Zgap (distance between the mirror and the SLM), mamer (twice the angle between the mirror and the SLM), and AOteam (twice the angle between the input beam and the SLM). This group of trainable model parameters is called aaiignment and as its constituents appear the forward model within differentiable functions, the suto-differentiation algorithm [2] can calculate their derivatives with respect to the error between the predicted camera images and the acquired ones. aatigument initially pre-trained with experiments placing square shaped phase differences randomly on the SLM. During the online training procedure, is further trained with the data from denoising experiments.
A.2 Details of Scaling Studies
A.2.1 Scaling with Output Image Resolution
Table 1: Properties of Optical, Convolutional U-Net and Fully Connected Denoising Networks, the same training settings with the main text are used in all experiments with the MNIST digits dataset.
Architecture
Parameters
FLOPS/Step
Energy/Image [J]
Images/s
Fully Connected
19.6 M
39.0 M
1.74
41.3
Convolutional U-Net
220 K
3.11 M Not Applicable
5.37
13.9
ODU
3.6 M
0.23
23.0
We investigated the change in the performance of Optical Diffusion when the image resolution changed while the model size was kept the same. This scaling behavior is also compared with well-established digital neural network architectures under the same diffusion settings on the MNIST digits dataset. The energy consumption and speed of the ODU in Appendix Table are indicated for the simple laboratory implementation where the efficiency is not optimized, while the digital benchmarks are run an Nvidia L4 GPU, one of the of-te-of-theant devices available today.
Fully Connected Denoising Neural Network. This fully connected architecture consists of 4 fully connected layers with 1200 neurons, SiLU nonlinearity, and one me--imensional batch normalization layers following each fully connected layer. The outputs of the first and third layers are summed with time-embedde embeeddnn representations. Inputs are interpolated to 76 x 76 and flattened to vectors of 5776 elements, while the last layer outputs a same-sized vector which is again reshaped to 76 x 76 and scaled to the target resolution. During inference time, this network generated a batch of 64 images in 1,55s, on an NVIDIA L44 GPU utilized 100% at 72 W. This amounts to 41.3 images/s at 1.74 J/image.
14

---

Convolutional U-Net Denoising Neural Network. This U-Net architecture 3 has 2 downsampling,
I bottleneck, and upsampling blocks, featuring total of 32 convolutional layers with 3 x kernels and SiL.U nonlinearity. Every block also includes time embedding and batch normalization. Similarly, inputs are interpolated 76 x 76 pixels and the same-sized outputs are scaled to the target resolution. During inference time, this network generated a batch of 64 images in 4.60 on an NVIDIA L4 GPU utilized 100% at 72 W. This amounts to 13.9 images/s 5.37 J/image.
ODU. The ODUs consist of 10 sets of 4 optical layers, each with 300 x 300 modulation parameters, as detailed in Section[4.1 At an image rate of 23 kfps and total energy consumption of 5.3 between the DMD and the camera, the generation can be operated at 23.0 images/s at 0.23 J/image. In this scenario, the SLM is assumed to be passive device due to the very small number of updates.
In addition to the comparison in Figure using the MNIST digits dataset up to a resolution of 28 x 28, studied further the generation quality by training with the AFHQ dataset's cat class 4 40 40 resolution. The results in Appendix Figure[cconiineg the same successful scaling trend with the ODU overperforming the digital networks. Even though this relatively more complex problem necessitates larger and more capable denoising networks, similarly with the smaller scale experiments, Optical Diffusion obtained the best FID.
T 300
7 730
1= 1000
Fully Connected MSE: 0.632
Original Data
FID: 386
20
U-Net
MSE: 0.085
FID: 324
20
ODU MSE: 0.218 FID: 290
Figure 8: Comparison of image generation performances on the AFHQ datasct's cat class 4 at 40-by-40 resolution.
A.3 Details of the Experimental System and Online Learning
In the ODU, the trainable parameters are the pixel values on the modulation layers and are digitally adjusted using computer model, as outlined in Appendix [A.I] After optimizing these parameters the computer, they are implemented across various layers in the experimental setup, utilizing the SLM. The beam generated by a continuous-wan dye laser (M-Squared Solstis 2000) À 850 nm reaches the phase-only SLM (Meadowlark HSP 1920-500-1200 after reflecting from a digital micromirror the
device (DMD), which can be used for spatial amplitude modulation. After 4 reflections modulating area SI,, the beam imaged onto a CMOS camera (FI.R BFS--U3-04S2M-CS) An 11.6 mm mirror, placed 17.1 mm from the SLM display, captures four reflections. To direct the input beam toward the SLM, 4F imaging was employed, transferring the beam from the DMD, which serves programmable aperture ensuring the beam's 's alignment with the first modulation layer of the SLM. We assigned 260 x 260 pixel patches for the modulation layers on the SLM, which has pixel pitch of 9.2 μm. After the fourth reflection, the beam is imaged with another 4F imaging system onto CMOS camera that records the output intensity as a 130 x 130 pixel image, where the camera's 's pixel pitch 3.45 jun. The examples of input patterns to the optical system, the corresponding output intensities at the camera plane and the resulting noise pattern predictions are provided in Appendix Figure The noise predictions are obtained by the downsampling and normalization of the pixel values the output intensity recordings.
15

---

Input Pattern
Output
Intensity
Predicted Noise
Figure 9: Some input patterns, output intensities at the camera plane, and the corresponding noise prediction for Fashion MNIST. The intensities the camera plane are converted to noise predictions by downsampling and normalization.
SLM
Camera
Mirro
SLM
Input
1.00
Online
.... Exp. Backprop MSE
0.95
Online
2.00
.... Exp. Backprop Decorr.
0,,00
198
0.85
196
0.75
Epochs
Figure 10: Schematic representation and photograph of the experimental system. Denoising error (MSE) and the decorrelation between the experimental system and the DT are plotted for online and only experimental backpropagation based trainings of the experimental system.
A.4 Scalability and Efficiency Outlook on Optical Diffusion Models
In this section, we investigate the potential improvement in the proposed method's performance by using the same type of commercially available hardware and optimally scaling images onto the DMD and the camera. The widespread availability of newer optoelectronic technologies would enhance calculated accuracies further.
16

---

Considering fixed and passive modulation layers, the main energy and time consumption of the optical system stems from the imput optical modulator, DMD and output array detector, CMOS camera. The DMD unit, Texas Instruments' DIP9500, can display 23, 148 patterns per second at resolution of 1920 1080, with an electrical consumption of 4.5 W board level, including data transfer (i.e. looping back from the detector). When representing 8-bit images with the aggregation of 256 binary pixels as a superpixel, this device is capable of displaying 8-bit images with 24 nJ px. On the detection side, the CMOS camera (FLIR BFS-U3-04S2M-CS) can obtain 8-bit, 0.4 MP images at 522 fiss rate with 3 W power consumption, acquiring images at nJ px. If we consider 1000 timesteps with 20 x 20 pixels images in the rest of this study, the consumption due optoelectric conversion and transfer would amount to 15 mJ /image.
Another potential energy expense item could be the light intensity required to provide a sufficient signal-to-noise ratio (SNR) at the detector. The SNR, especially in the context of a digital device like a detector or an ADC (analog-to-dis to-digital converter), is typically calculated as SN R(dB) = 6.02x where n represents the number of bits. Considering the noise-limited scenario, the
required number of photons can be calculated using Ninpur Drocittiiin kyes Tieercoor
SNR2
where 7mmodulation layers
accounts for portion of light scattered out in passive layers and Tocteettr accounts for the conversion efficiency of the detector. Assuming 90% transmittance for each modulation layer and 50% efficiency for the detector, and using an 850 nm wavelength to calculate the energy of each photon: E X the required energy for light is only a few picojoules, which is negligible compared to the energy consumption of optoelectronic devices.
With the optimal configuration of the ODU, where each pixel on the analytical model is precisely mapped one-to-one the optoelectronic devices, the same set of hardware can generate images with a consumption of approximately 15 15m /image which is significantly lower than the 1.7 required by conventional methods (see Appendix A.2. making the ODU more than 100 times more energy-efficient.
A.5 Training of Modulation Layers
As shown in 3 we utilized 3 different datasets to demonstrate the proposed approach.
First digits from MNIST-digits
First classes from Pashion-MNIST
20000 "Clock" images from Quick, Draw! dataset
After being downsampled to 20 20, the images are used for 250 epochs to train the ODUs, using Adam optimizer with learning rate of 0.006, which took ~ 10 hours on an A100 GPU.
A.6 Definition of Performance Metrics
Mean Squared Error (MSE)
MSE
Σ 13 (1)²
(8)
1=1
where yi the true values and are the predicted values. Fréchet Inception Distance (FID)
FID (z. g) 1| Mgll3 Tr(E, Eg -((EE,)))))
(9)
where Ez,) and (Ng: Eg) are the mean and covariance of the feature vectors of the real and generated data, respectively.
Kernel Inception Distance (KID)
KID(,,)) n(nn 1) ifj
1
Σ ((f(=:) m(m 1) it
Σ
k(f(a)) 2
Σ k(f(z))
(10)
17

---

where k is polynomial kernel and is the Inception network function that extracts features.
Inception Score
IS(G) exp (Exx-p, [D [D<x((())))]((()]]
(11)
where p(y|x) the conditional label distribution given generated image and p(y) is the marginal distribution over all generated images.
A.7 Code Availability
The source code is available hbtzee///oogggghhuooooogggffaaau
Appendix References
[1] M. Feit and J. Fleck, "Beam nonparaxiality, filament formation, and beam breakup in the self-focusing of optical beams," JOSA B, vol. 5, no. 3, pp. 633-640, 1988.
[2] Paszke, S. Gross, Chintala, G. Chanan, Yang, DeVito, Lin, Desmaison,
L. Antiga, and A. Lerer, "Automatic differentiation in PyTorch," Oct. 2017. [Online]. Available:
[3] O. Ronneberger, P. Fischer, and T. Brox. "U-net: Convolutional networks for biomedical image segmentation," in Medical image computing and computer-assisted interenntio-MMCCO 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18. Springer, 2015, pp. 231-241.
[4] Y. Choi, Y. Uh, Yoo, and J.-W. Ha, "Stargan v2: Diverse image synthesis for multiple domains," " in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2020,
pp. 8188-8197.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Section 3
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
There is no new theoretical formulation.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
The access link is provided Appendix A.7
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
lc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppulli//uuuii wwwtc//gguie///eeeeceoocc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix A.5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix A.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::://neuripses httpp//www.....uuueeeeeeee http://...ccooooccttttccctttteea
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
22

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There is release of new data model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.reeewthhhe....c... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
23

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "eLQGYEN9rz", "reviewer": "Reviewer_hbsN", "summary": "The authors present a hardware-based implementation of a denoising diffusion model.\nInstead of using a neural network, the authors propose to use an optical setup to perform the denoising steps during image generation.\nThe system is trained using a digital twin simulating the optical setup.\nThe authors evaluate their method on different datasets and report the relevant quality metrics.", "questions": "Can the authors discuss how their denoiser implementation differs from the one in [23]?\nCould the could other optical denoisers be utilised as well?", "limitations": "I feel the fact that the denoiser is fully linear is quite limiting.\nHow could non-linear elements be included into the setup in the future?\nWould the increase execution time?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Building an efficient hardware-based diffusion model can be dramatically impactful as generative AI becomes more and more popular and finds more and more applications.\n\n* The potential speed up compared to conventional networks mentioned by the authors would be a game changer.\n\n* I believe the solution the authors came up with to enable training using the digital twin is quite elegant and makes a lot of sense.", "weaknesses": "* There are now baselines in the experimental evaluation. Obviously, this is only a proof of concept and we cannot expect competitive numbers. Still there may be insightful baselines to compare the system against. E.g. How does the performance compare to the simulated digital twin? It would also be great to show qualitative results for comparison.\nAnother baseline could be simple neural network with the same runtime?\n\n* The presented implementation uses purely optical elements that are limited to linear operations. \nEven though the authors hint that this may be fixed in future work, I think this is an important caveat.\nAfter all, the power of the conventionally used artificial neural networks heavily relies on the non-linearities which are used. \n\n* I am missing a qualitative evaluation of the denoising task itself. Can the authors include images directly showing inputs and outputs of the optical system, instead of the results of the diffusion process.\n\n* I am not completely sure NeurIPS is the correct venue for this type of publication which is very much focussed on the exerimental hard ware aspect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their overall positive appreciation of our manuscript. We start with point-by-point replies to the mentioned weaknesses (W1-4), questions (Q), and limitations (L) respectively. We will incorporate the additional results we present here and revisions in the main text.\n\n**W1.** As can be now seen in the common rebuttal Reb. Table 1, we looked into both fully connected and convolutional architectures to compare with our performance. We have seen that depending on the output image resolution we have similar scaling with these pure digital networks as depicted in Reb. Figure 1. Moreover, Reb. Figure 3 shows a scaling of model parameters with task accuracy, indicating a similar trend by following the well-established power law in digital neural networks [10.48550/arXiv.2010.14701] with an on-par slope. \n\n**W2.** We thank the reviewer for allowing us to clarify this point. In the proposed framework, several factors create the nonlinear transformation of the input information. One of these effects is the inherently nonlinear nature of the intensity detection. While the programmable parameters and free-space propagation are acting on the wave propagation by affecting both phase and magnitude, with complex numbers in mathematical terms, the detection only records real numbers by applying an absolute square operation, as indicated in Fig. 2. Note that programmable parameters are introduced with phase-encoding, which also creates a nonlinear relation between those and the detected output intensity. The similar performance with nonlinear neural networks, as shown in Reb. Fig. 1-3, are obtained thanks to these factors introducing nonlinearity in the system. We discuss the possible additional nonlinearities in our reply to the comment in the Limitations section (please see below).\n\n**W3.** We thank the reviewer for this feedback. We agree that images corresponding to the output intensities of the ODU would help the readers to follow the method more easily. We included these images in Reb. Fig. 4 and will be included in the main text after acceptance. The camera plane images contain more pixels. They are matched with data size by downsampling and normalization, where these steps are part of the forward pass and error backpropagation on the digital twin to match it with the experiment. The reviewer can notice the effect of downsampling on the camera plane by emerging high-intensity grid matching the dataset pixelation.\n\n**W4.** While we propose new hardware for denoising diffusion-based image generation, since this study aims to create a different type of machine learning infrastructure, we hoped that it fits the description given in the NeurIPS’s Call for Papers: “Infrastructure (e.g., libraries, improved implementation, and scalability, distributed solutions)...Machine learning is a rapidly evolving field, and so we welcome interdisciplinary submissions that do not fit neatly into existing categories.”, and wanted to build an interdisciplinary bridge so that the development of novel hardware can utilize the feedback from the machine learning community.\nMoreover, programming the hardware for performing the image generation task required the development of a novel algorithm for efficient and reliable operation. Our main contributions in this direction are the removal of time embedding steps for multi-step denoising, and the online learning algorithm to train with non-ideal experimental systems.\nThese algorithmic tools are not limited to specific hardware but can be beneficial to use in virtually any analog computing architecture especially where updating weights is the expensive procedure.\n\n**Q.** Both implementations denoise images physically with multiple modulation layers in free space, while they differ in terms of different aspects such as their physical architectures, electromagnetic spectra, denoising methods, and noise types. Ref. 23 utilizes a single set of modulation layers to filter out the salt and pepper type of noise in images and predict images with lower noise. In contrast, our work is a multi-pass architecture where the layers do change with respect to the timestep and the prediction is the Gaussian noise term in input images, not the denoised image. Hence we can do image generation, which is not possible for ref. 23.\n\nOther differences are that ref. 23 works with terahertz frequency level electromagnetic waves, and 3D printed, fixed modulation layers where the wavelength and feature sizes are on the order of millimeters, while this study uses the multiple reflections of an optical light beam off a single reconfigurable spatial light modulator, where the wavelengths and features are on the micrometer scale. This allows for compatibility with mature optical technologies for an efficient prototype and a compact form factor.\n\n\n**L.** While our system benefits from multiple nonlinear effects and shows similar scaling trends with digital nonlinear networks, it is also realizable to add more nonlinear interactions through the addition of different optical mechanisms to the system. \nThe addition of a saturable absorber [1] or optically limiting film [2] would benefit from optical nonlinearities without increasing the energy budget substantially. Moreover, these physical effects can have lifetimes as low as picoseconds, so they would not increase the latency of the system either.\n\n[1] Zubyuk, Varvara V., et al. \"Low-power absorption saturation in semiconductor metasurfaces.\" Acs Photonics 6.11 (2019): 2797-2806.\n[2] Vivien, L., et al. \"Carbon nanotubes for optical limiting.\" Carbon 40.10 (2002): 1789-1797.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "35T01jfNnV", "reviewer": "Reviewer_Kx6v", "summary": "Summary: This paper has trained free-space diffractive optical neural networks as an implementation of diffusion model for image denoising application. On-chip learning and hybrid training have been used to improve the noise and variation robustness.", "questions": "1.\tThe novelty is limited. No new hardware system is proposed. Diffusion model uses the standard algorithm. On-chip learning and error calibration are also previously proposed methods in the literature. Most of the method section is standard DDPM, diffusion, on-chip learning algorithms. No new contributions.\n\n2.\tWhat are the major advantages of free-space optical diffusion model compared to other accelerators or other diffusion neural network models? Rigorous area/speed/throughput/energy efficiency evaluation is missing. Thorough comparison to other diffusion models is missing.\n\n3.\tOnly toy examples are shown on MNIST, Fashion MNIST. High-resolution image denoising benchmarks are required.\n\n4.\tThe free-space phase mask is not reconfigurable. SLM also has low-resolution or slow reprogramming speed. The flexibility and speed/efficiency have major concerns.", "limitations": "1.\tThe novelty is limited. No new hardware system is proposed. Diffusion model uses the standard algorithm. On-chip learning and error calibration are also previously proposed methods in the literature. Most of the method section is standard DDPM, diffusion, on-chip learning algorithms. No new contributions.\n\n2.\tWhat are the major advantages of free-space optical diffusion model compared to other accelerators or other diffusion neural network models? Rigorous area/speed/throughput/energy efficiency evaluation is missing. Thorough comparison to other diffusion models is missing.\n\n3.\tOnly toy examples are shown on MNIST, Fashion MNIST. High-resolution image denoising benchmarks are required.\n\n4.\tThe free-space phase mask is not reconfigurable. SLM also has low-resolution or slow reprogramming speed. The flexibility and speed/efficiency have major concerns.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The application of optical computing hardware is interesting. Robustness has been considered in the training procedure with online learning for error calibration.", "weaknesses": "1.\tThe novelty is limited. No new hardware system is proposed. Diffusion model uses the standard algorithm. On-chip learning and error calibration are also previously proposed methods in the literature. Most of the method section is standard DDPM, diffusion, on-chip learning algorithms. No new contributions.\n2.\tWhat are the major advantages of free-space optical diffusion model compared to other accelerators? Rigorous area/speed/throughput/energy efficiency evaluation is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and efforts. We reply to the comments under four main sections:\n\n**Limitations - 1** \n\nThis paper shows diffusion-based image generation with the advantages of the optical modality for the first time. We use an architecture that incorporates off-the-shelf hardware in an innovative way while maintaining near-future deployment practicality and the reproducibility of the study. \n\nMoreover, the algorithmic novelty of the study is two-fold:\n\n1-   The proposed time-aware denoising algorithm allows image generation with minimal changes to the set of passive optical weights, this results in a substantial decrease in the latency and energy consumption of the process.\n\n2-  To the best of our knowledge, we propose and show the effectiveness of the online learning algorithm for the first time, differing from the existing literature by continuously updating the physical alignment parameters of the system during training. This alleviates the model drift issue significantly, where a fixed digital model of a physical system becomes insufficient with the change in the operation point of the system over the training. We experimentally show that online training achieves convergence with a physical system, which is too challenging with existing methods.\n\n**Limitations - 2** \n\nOur study demonstrates that optical wave propagation can be a new modality for realizing diffusion models with the proposed algorithms. This is crucial because as opposed to electronics, optics has an intrinsic near-zero loss and parallelization capability. We also hope that this would motivate the community to realize machine learning algorithms with different modalities and address the environmental impact of AI models.\n\nEven though this initial study mainly aims to be a proof-of-concept with non-optimized hardware, we discuss the energy consumption in Appendix 4, indicating that the proposed method can enhance the energy efficiency of diffusion models significantly. Furthermore, to address the need for a deeper analysis you mentioned, now we provide a more extensive comparison in the rebuttal phase between our approach and two digital models of comparable accuracies in the shared rebuttal section. You can refer to Reb. Table 1 for the details of the comparison. We show with this new set of measurements in Reb. Fig. 1 that ODU scales in the same way as digital implementations in terms of the output image complexity and can perform comparably with models of MFLOPs of compute. Reb. Fig. 3, plotted with the re-organization of the information in Fig. 4, shows the scaling of the ODU’s performance with the number of model parameters. Remarkably, the scaling constant matches closely with the power-law scaling constant of large-scale image generation models from the literature [1]. Even though ODU has a completely different architecture, the similar scaling trend shows its potential for tackling high-resolution real-life scenarios. We plan to include these additional findings in the article.\n\n**Limitations - 3** \n\n In this study, we aimed to show the realizability of performing diffusion-based image generation with optics, while working with a number of parameters that can be realized with easily accessible devices, so that the findings could be reproduced by different teams. The scaling studies in Fig.3, Reb. Fig. 1 and 3 provide evidence that the method can scale similarly to digital models, which we know are capable of high-resolution tasks when they reach sufficient complexities. \n\nAnother new piece of evidence in this regard is the comparison of ODU and two other digital models in a more complex and higher-resolution dataset of cat images with 40-by-40 pixels from the AFHQ dataset. Even though none of the models has enough complexity to create realistic images, as shown in Reb. Fig. 2, ODU still obtains a better generation and respective FID score than digital models with this dataset. This result further strengthens the expectation that the proposed method will perform competitively in high resolution with a larger number of parameters. \n\n\n**Limitations - 4** \n\n When the model is deployed with passive phase masks for inference, the phase masks will not be reconfigurable but provide a significant power efficiency advantage. Please check the common rebuttal for detailed speed/power/efficiency/scaling results, augmented and represented. The use of SLM is for cases that require reconfigurability: training and modalities with interchangeable inference tasks. We do not agree on the resolution being low. Typical pixel pitch is 8 microns and there are models with lower pixel pitch employing >10 million pixels in a single cm-scale device which can be cascaded for larger networks. Regarding speed, there are 1 kHz models commercially available with ongoing research for faster models. The reviewer can reach out to the mentioned models and specifications on various vendors' catalogs available online (e.g. Holoeye).  We use one set of masks for 100 passes, meaning that the SLM pattern does not need to change for 100 passes. This is one of the novelties of our algorithm. When 1 kHz is taken as the refresh rate, it would mean the SLM can be fed with the data as high as 100 kHz, which would mean completing 1000 passes in 10 milliseconds. \n\n\n[1] Henighan, Tom, et al. \"Scaling laws for autoregressive generative modeling.\" arXiv:2010.14701 (2020).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B3WgvQQgOK", "reviewer": "Reviewer_5uf9", "summary": "This paper proposes an opto-electronic realization of a toy diffusion model for image generation. The system consists of a DMD for optical image projection, SLMs for phase modulation, and a CMOS photosensor for signal intensity recording. These physical components, together with a digital processor (for noise addition), work collaboratively in an iterative manner to implement a multi-step diffusion process (i.e., iterative denoising and noise synthesis). To account for the imperfection of the physical setup (e.g., systematic errors stemming from the fabrication and misalignment), it proposes an online learning scheme to optimize the real physical parameters (of SLMs). It demonstrates that such a system can be used to generate simple images (in toy datasets) in an energy-efficient way in comparison to the dominant electronic approach.", "questions": "See above.\n\n**Post-rebuttal:**  \nThe authors have carefully addressed my prior concerns and several follow-up questions. \nOverall, this work is novel to me in terms of both the new application (an optical realization of the diffusion model) and the unique online training algorithm that bridges the gap between simulation and actual experimentation.\nTherefore, I vote for acceptance of this paper.", "limitations": "See above", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Innovative demonstration of an opto-electronic implementation of a toy diffusion model \n\n- The proposed online learning scheme sounds effective for bridging the gap between digital simulation and real experiments.", "weaknesses": "Though the overall idea (the opto-electronic diffusion model realization and the online training algorithm to alleviate the calibration issues) sounds interesting and reasonable for me, the paper itself seems to be written in a hurry, where the content is not well polished and its clarity can be further improved. \n\n**Online training algorithm**\n\nI'm particularly interested in the online learning and the experimental system parts, but some technical details are unclear or missing in the paper. In L216-223, the authors stated, \"The digital twin is calibrated to approximate an experiment, and a 20% misalignment exists between its calibration and the actual experiment’s alignment, which is simulated by another diffraction model,\"  but how to calibrate the digital twin to approximate the experiment is not mentioned. What 20% misalignment suggests is also vague for me; do you mean the MSE between the simulation and the experimental results on average? Also, I don't understand the meaning of \"simulated by another diffraction model.\" Please further elaborate on this point to improve the reproducibility of the proposed method. \n\nIn Algorithm 1, it seems like the authors maintain two sets of phase variables (\\thete_layer and \\theta_alignment) during the training. \\theta_layers will be used to configure the SLM in the real experiment, while \\theta_alignment is used in the digital twin to account for the imperfection of the system calibration. If so, I would like to see the final difference between the SLM's phase profile and the one used in the digital model. This will be helpful to understand the misalignment  between the physical system and its digital twin.\n\nBesides, the online learning algorithm without Digital Twin Refinement is exactly the hardware-in-the-loop approach invented in computer-generated holography [A], which should be mentioned in the related work. \n\nI am not familiar with the experimental error backpropagation algorithm compared in the paper. Could the authors provide additional discussion to clarify the differences between the proposed algorithm and this method?\n\nFinally, did you first pretrain the digital model in simulation and then finetune it with the online learning algorithm, or train the model with online learning from scratch? Since online learning requires the hardware-in-the-loop scheme, I doubt its implementation efficiency in practice. \n\n[A] Peng, Yifan, et al. \"Neural holography with camera-in-the-loop training.\" ACM Transactions on Graphics (TOG) 39.6 (2020): 1-14.\n\n**Experimental setup**\n\nRegarding the experimental system setup in the supplementary material, Figure 6 only provides an incomplete illustration of the system, so it's a bit unclear how the authors realized the experimental setup in reality. Specifically, how many SLMs did you use in the experiment? From Figure 6, it seems four independent SLMs are used to implement four optical diffraction layers, and their weights are dynamically reconfigured to support time-aware iterative denoising. \n\nIt's therefore necessary to discuss the Time Multiplexing technique used in this experiment.\n\n**Limitations incurred by opto-electronic conversion**\n\nOne of the major limitations and bottlenecks of such an opto-electronic system is not discussed, i.e., the repeated analog-to-digital and digital-to-analog conversions, which are especially energy-consuming since many conversion cycles are needed in the iterative inference process.\n\n**Benchmark performance**\n\nFinally, it would be informative to understand the performance of this optical diffusion model by situating it in the landscape of pure digital diffusion models. For example, what is the pure digital model that is comparable to the proposed model in terms of generation quality (FID, etc.)? I understand that at this stage, the competing digital model would be quite simple and naive (e.g., just a simple MLP with small numbers of parameters). However, providing such information would shed light on the path to further improve the performance of the optical neural network. \n\n**Minor suggestions:**\n\n- Please add a spacing between citation bracket and the prior word\n\n- L175, \" (==autodiff citation==).\", the citation TODO is not addressed. \n\n- L180, \"300x300\", please replace the letter \"x\" by the multiplication operator \"\\times\". \n\n- L211, \"1 − CorrelationCoefficient\" looks a bit ugly. please also clarify how to compute CorrelationCoefficient\n\n- It is preferable to have self-contained captions for each figure so that readers won't need to consult the main paper's text in order to understand the figure. \n\n- A recent paper regarding the optical neural network [B] might be worthy of being mentioned in the paper. \n\n[B] Wei, Kaixuan, et al. \"Spatially varying nanophotonic neural networks.\" arXiv preprint arXiv:2308.03407 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the in-depth analysis of our study, and numerous crucial suggestions for improving the quality of the manuscript significantly. We hope to address the points raised with the following responses and the common author rebuttal.\n\n**Online training algorithm**\n\n**Re:** \"...Also, I don't understand the meaning of \"simulated by another diffraction model.\" ...\"\n\nWe will clarify the experiment in Fig. 5 as follows with an addition to the main text and the appendix:\n\n\"In Fig. 5, we explore the efficiency of the proposed algorithm by modeling a possible experiment scenario. In this scenario, we define two different optical models, while actually both of them are simulations of the optical wave propagation for an exact insight into the algorithm, we designate the first one as the “optical experiment” by configuring it with the calibration angles obtained from the physical experiment. These four angles account for the slight misalignment of the experiment and define the input angle of the beam to the cavity and the angle between the mirror and the SLM, in x and y axes, all being in the range of a few milliradians and their measurement details being provided in A.7. The second model, considered as the digital twin, is initialized with their calibration angles 20% higher.\"\n\n**Re:** \"... I would like to see the final difference between the SLM's phase profile and the one used in the digital model...\"\n\nThanks for pointing out this distinction to be clarified. The digital twin and the physical system share the same layer parameters, these are denoted by $\\theta_{layer}$ and they indicate the phase value for each pixel of the layers. On the other hand,  $\\theta_{alignment}$ is the expected alignment parameters of the physical system, including the beam and mirror angles, and layer distances.\n\n**Re:** \"...the experimental error backpropagation algorithm compared in the paper....\"\n\nThe main difference between the Neural Holography approach and the one in [27] is that in [27] backpropagation of the experimental loss is realized over a pre-trained neural-network-based emulator of the physical system instead of the wave propagation-based physical model. Thanks for the suggestion, we will rewrite starting from L97: “Calculating the gradients of the experimental loss through the optical wave propagation model allows for high-quality computer generated holograms[A]. Similarly, a pre-trained neural network-based emulator of a physical system can also be used for the same purpose [27].” \n\n**Re:** \"...did you first pretrain the digital model in simulation and then finetune it with the online learning algorithm ...\"\n\nIn our implementation, $\\theta_{alignment}$ was pre-trained while $\\theta_{layer} $was trained from scratch in the optical experiment mainly because the experiment can create outputs faster than its digital model. Still, as mentioned, for deploying this framework at scale it is not practical to train models from scratch for each hardware instance. Fortunately, we expect that with some fine-tuning epochs, the pre-trained models should be deployed to different hardware with small differences. The original model can be trained with a vaccination approach completely digitally, by slightly changing optical alignment,  so that it is resilient to small mismatches with some performance penalty [1]. Then on hardware fine-tuning can further improve its performance.\n\n**Experimental setup**\n\nWe apologize for the confusion and we will re-work on the presentation. In the schematic of Figure 6, we intended to show that we use one SLM device shown by the gray outer box. Within this single device, we show four modulation layers by placing them in sub-regions of the single device. Using a mirror positioned across, light bounces back and forth and gets modulated by the layers (sub-regions) successively. On the right-hand side of Figure 6, we show a photograph of the experimental system where the single SLM display is indicated as well as the highlighted light path and the mirror position. Since we used a single device, we did not need to have time multiplexing. But it would be an interesting direction that we will look into for the future when many devices are used together to scale up the system.\n\n**Limitations incurred by opto-electronic conversion**\n\nWe have provided the power comparisons in the common rebuttal showing a significant advantage for the ODU and we will modify the manuscript accordingly. Our calculations incorporated board-level energy consumptions meaning that opto-electronic transformations including analog to digital conversions and data transfer in the loops are included. Since these repeated conversions and data transfers are happening in kHz scale, the power consumption does not increase dramatically as the dissipated power depends on the frequency. We agree with the reviewer on the fact that these calculations would be quite different if we want to operate on the GHz scale. While kHz-scale looping already yields higher performance as presented by our results, we acknowledge that one major future research direction is efficient conversions and data transfer in the GHz scale.\n\n\n**Benchmark performance**\n\nWe thank you for the suggestion. Correspondingly, we compared the generation performances with two pure digital models, a convolutional U-Net architecture and a fully connected network for different output resolutions of Fashion-MNIST dataset, and a more complex AFHQ dataset. The results show that ODU performs and scales in the same manner as large-scale digital models while benefiting from the energy efficiency of optics, showing promise for real-life tasks. We refer you to the author rebuttal for their details.\n\n**Minor suggestions**\n\nThank you for the suggestions, they will be included and we believe they will improve the paper’s quality and readability.\n\n[1]: Mengu, Deniz, et al. \"Misalignment resilient diffractive optical networks.\" Nanophotonics 9.13 (2020): 4207-4219.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vJ0U6hGp3G", "reviewer": "Reviewer_heiE", "summary": "This work presents an image generation framework using denoising diffusion models implemented though optical computing. The method uses passive diffractive optical layers that are trained to manipulate light propagation through the system — peforming effective denoising. Using the physical properties of light, the framework allows to rapid image processing with significantly reduced power consumption. The authors demonstrate the effectiveness of their framework using MNIST, Quick Draw and Fashion MNIST.", "questions": "I noticed a typo in line 175: ==autodiff citation ==. \n\nHow well does this framework scale with resolution and complexity of the images?\n\nThe motivation of the work is to address the environmental footprint from classical computing hardware. The comparison of the method with respect to GPU hardware is presented in the appendix. I recommend you discuss that in the main text — it would strengthen your work.", "limitations": "The authors don’t discuss explicitly the limitations of their work. Although in the Check List the authors refer to section 4.2 — however, it is not clear what the authors define as limitations.\n\n\n\n\n\n\n**Post-rebuttal**:\nAll questions have been carefully addressed by the authors. I believe this work presents a novel approach while addressing the environmental impact of traditional computing methods. I modify my score: 7.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and clear. The study addresses the environmental impact of traditional computing methods for image processing — the presented framework is 400 times more energy-efficient than conventional GPU approaches.", "weaknesses": "The paper doesn’t discuss in detail the scalability of the proposed models, especially in the context when deployed in challenging real-world scenarios. \n\nFurther comparison with respect to available GPU-based denoising frameworks would be adequate. The work could benefit from a broader comparison with existing approaches, particularly in terms of image quality and convergence time or processing speed under different scenarios. \n\nThe maintenance of these systems in practical, long-term deployments is not discussed. For example, liquid crystal displays (e.g., SLMs) are temperature-sensitive — how this affects their performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and attention to our work. We would like to provide some replies to the issues raised, and they will be incorporated into the main text in the next revision.\n\n**Scalability of the Proposed Model and Comparison with GPU-based Denoising Frameworks**\n\nThank you for highlighting this potential for improvement, we agree with the importance of the scaling of the framework with the resolution and complexity of the images and accordingly, we looked into the scalability of our approach from different aspects in the author's rebuttal part and compared the performance of the ODU with digital models in terms of energy and image quality. \n\nFor different output image resolutions, while models are kept fixed, we demonstrate ODU can constantly generate higher quality images than the MFLOP-scale convolutional and fully connected digital networks with a smaller energy budget (Rebuttal Figure 1). This trend persisted even when a more challenging task is benchmarked as shown in Reb. Fig. 2.\n\nSimilarly, when the task is fixed and the number of model parameters is changed, ODU has nearly the same scaling trend as large-scale digital image generation networks. The slope of the trend shown in Reb. Fig. 3, which is plotted with the data from the main text's Fig. 3, is approximately the same as the power-law scaling trend of generative digital models in [1].\n\n**Practical Aspects of the Hardware**\n\nWe thank the reviewer for bringing up the important practical deployment aspect of the proposed method. In general, we acknowledge that analog hardware is more challenging to operate in terms of stability and degradation with time and these challenges are tackled, usually with feedback, only if the advantages obtained weigh significantly more. We believe our demonstration is in this category. Moreover, Liquid Crystal on Silicon (LCoS) SLM is one of the most mature and reliable opto-electronic devices, widely used in networking infrastructure with solutions provided by companies such as Coherent, Lumentum, Huawei, etc. For example, we invite the reviewer to check the press release by Coherent ultrareliable Datacenter Lightwave Cross-Connect for datacenter optical circuit switches.\nMoreover, note that LCoS SLMs are also used in telecommunications in undersea applications, where the maintenance is very, if not impossible, restricted. Hence, we are confident in the feasibility of industrial-grade deployment while acknowledging additional challenges common for all analog computing hardware.   \n\n**Limitations of the Proposed Work**\n\n-As the framework reaches the best performance for a given configuration of hardware, such as angles of the input beam and the mirror, when a trained model is transferred to new hardware with some potential alignment differences, this would require a fine-tuning of the model before deployment.  This requirement should be further analyzed in the following studies quantitatively. However, considering that once it is trained, the deployed model can be utilized for years, this occasional fine-tuning should have a negligible overall effect. \n\n- We demonstrated this proof-of-principle study with easily available, off-the-shelf opto-electronic equipment such as cameras and light modulators. Even though they are ideal for initial studies, their speed and energy efficiency are not optimized. Consequently, the speed and energy consumption of the proposed framework is also limited by their performance. Similarly, reaching higher accuracies requires larger models with ODU as in other modalities, and currently, the size of the optical model is limited by the modest number of pixels on light modulators. Due to these reasons, reaching competitive performances and addressing real-world problems efficiently with optics will be possible with the development of specialized hardware for this purpose.\n\n[1] Henighan, Tom, et al. \"Scaling laws for autoregressive generative modeling.\" arXiv preprint arXiv:2010.14701 (2020).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their time and efforts in providing their insightful comments. They recognized our work as an “innovative demonstration of an opto-electronic implementation of a toy diffusion model (5uf9)”, agreed with its potential impact; “efficient hardware-based diffusion model can be dramatically impactful (hbsN)”,” addresses the environmental impact of traditional computing(heiE)” and found the proposed solution useful, “training using the digital twin is quite elegant (hbsN),  proposed online learning scheme sounds effective for bridging the gap (5uf9)”.

We would like to start by briefly reiterating our findings and motivation. This study demonstrated for the first time how programmed optical wave propagation can be used as a computing engine for image generation. The main potential benefit of this new method is the sampling of diffusion models with a much smaller energy budget compared to electronics since optical wave propagation has a very small intrinsic loss while acquiring comparable, or better quality (Rebuttal (Reb.) Fig. 1 and 2).  This is especially interesting because diffusion models are currently one of the most costly generative AI models due to their repetitive denoising process, with a correspondingly large environmental impact. The generation of a single image can emit more than a kilogram of $CO_2$ [1].

Moreover, the scaling behavior of the optical implementation follows the same trend as the digital. The scaling of the proposed method was a common comment in the majority of the reviews and we address it by introducing additional data (Reb. Fig. 1 and 3).

To enable this proof-of-principle we developed a novel method,  the time-aware denoising algorithm, which allowed image generation with minimal changes to the set of passive optical weights, with a substantial decrease in the latency and energy consumption of the process. In addition, the proposed online learning algorithm achieved convergence showing the promise of the framework for real-life scenarios. Thanks to these flexible algorithms, this method can be used to implement image generation with different physical hardware.

**Scaling Image Resolution and Comparison with Digital Neural Networks**

Receiving the overlapping comments of the reviewers about this topic made it obvious that the study should provide further details on the scaling of the proposed system both in terms of parameter counts and output dimensions along with the comparisons with purely digital implementations. As detailed in the attached Reb. Table 1, two digital architectures, one being fully connected and the other convolutional U-Net [2], are trained for the same image generation tasks with the Optical Diffusion Unit (ODU). Here we would like to emphasize that the energy consumption and speed of the ODU are indicated for the simple laboratory implementation where the efficiency is not optimized, hence the energy efficiency can be significantly improved. For the digital implementations, we performed the benchmark on an Nvidia L4 GPU, which is one of the most energy-efficient devices available today. Due to the changes in the digital hardware and models, we will update the corresponding appendix section. The results shown in Reb. Fig. 1 indicate that ODU outperforms the two digital neural networks, and all three scale in a similar manner both in terms of denoising and generation performances when the generated image dimension is increased while the model sizes are kept constant.
To extend the analysis on the output image size to a higher resolution and real-life images, we trained the same three networks on cat images with 40-by-40 pixels from the AFHQ dataset [3]. As shown in Reb. Fig. 2, while this problem required more capable models for realistic images, ODU kept acquiring better FID scores from digital neural networks at this scale.

**Scaling of Model Parameter Counts**

In addition to the scaling with respect to the image resolution, by combining and replotting the first two tiles of Fig. 4 of the main paper, Reb. Fig. 3 illustrates the same widely accepted power-law trend of the performance versus parameter count of digital networks in ODU as well [4]. Most significantly, when the optical implementation is fitted to a power-law equation, the exponential of the power law (-0.15) is approximately the same as the reported value (-0.16) for large-scale image generation networks in [4]. This fit parameter gives the slope of the line in the logarithmic plot, indicating how fast the generation performance scales with the number of parameters, in this case showing that ODU improves its performance at a similar speed with large-scale image generation networks while its parameter count is increased. Finally, we remark that the single outlier in this trend is the case where there is only a single modulation layer, which does not benefit from multiple optical modulations in the proposed architecture.

[1] Luccioni, Sasha, Yacine Jernite, and Emma Strubell. "Power hungry processing: Watts driving the cost of AI deployment?." The 2024 ACM Conference on Fairness, Accountability, and Transparency. 2024.

[2] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer International Publishing, 2015.

[3]  Choi, Yunjey, et al. "Stargan v2: Diverse image synthesis for multiple domains." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020. 

[4] Henighan, Tom, et al. "Scaling laws for autoregressive generative modeling." arXiv preprint arXiv:2010.14701 (2020).

Meta-review: The paper proposes a novel approach for implementing denoising diffusion models using optical computing, leveraging passive diffractive optical layers to perform efficient denoising with minimal energy consumption. The method addresses the high energy demands of traditional diffusion models on electronic hardware by utilizing light propagation through trained modulation layers, offering a promising alternative for energy-efficient image generation.
The paper received mixed opinions.
After going through all the reviews and discussions, we borderline accept the paper. While it shows promise, it would benefit from further development in scalability, practical deployment, and detailed comparisons with existing methods. The potential impact on energy-efficient AI is significant, and with additional refinements, this work could make a valuable contribution to the field. Lastly, we encourage the authors to polish their paper significantly.
Decision: Accept (poster)