Physics-Constrained Comprehensive Optical Neural  Networks
Authors: Yanbing Liu
Jianwei Qin
Yan Liu
Xi Yue
Xun Liu
Guoqing Wang
Tianyu Li
Fangwei Ye
Wei Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Optical Neural Networks;Physical Neural Networks;Error Compensation
Primary area: machine_vision
TL;DR: Our work enhances AI-driven free space optical neural networks by using a physics-constrained learning framework to correct systematic errors, significantly boosting image recognition accuracy.

Abstract: With the advantages of low latency, low power consumption, and high parallelism, optical neural networks (ONN) offer a promising solution for time-sensitive and resource-limited artificial intelligence applications. However, the performance of the ONN model is often diminished by the gap between the ideal simulated system and the actual physical system. To bridge the gap, this work conducts extensive experiments to investigate systematic errors in the optical physical system within the context of image classification tasks. Through our investigation, two quantifiable errors—light source instability and exposure time mismatches—significantly impact the prediction performance of ONN. To address these systematic errors, a physics-constrained ONN learning framework is constructed, including a well designed loss function to mitigate the effect of light fluctuations, a CCD adjustment strategy to alleviate the effects of exposure time mismatches and a ’physics-prior based’ error compensation network to manage other systematic errors, ensuring consistent light intensity across experimental results and simulations. In our experiments, the proposed method achieved a test classification accuracy of 96.5% on the MNIST dataset, a substantial improvement over the 61.6% achieved with the original ONN. For the more challenging QuickDraw16 and Fashion MNIST datasets, experimental accuracy improved from 63.0% to 85.7% and from 56.2% to 77.5%, respectively. Moreover, the comparison results further demonstrate the effectiveness of the proposed physics-constrained ONN learning framework over state-of-the-art ONN approaches. This lays the groundwork for more robust and precise optical computing applications.

Physics-Constrained Comprehensive Optical Neural Networks
Yanbing Liu ybliu@bupt.edi edu cn
Jianwei Qin qinju3esjtu. edu, cn
Yan Liu?
liu-yanosjtu edu. cn
Xi Yue! yuexi@bupt edu. on
Xun Liu Lauxun__aby@163.c com
Guoqing Wang' @gwangg420thommill.com
Tianyu Li cosmos, c yu@homailll com
Fangwei Year fangweiye@sjtu. edu.
Wei L wei wei_1i_bj@163.cm com
Beijing University of Posts and Telecommunications, Beijing.100876.40 100876,China
2 Shanghai Jiao Tong University. Shanghai 200240,, China
3 Beijing Institute of Space Mechanics and Electricity, Beijing, 100094,Chins
4 University of Electronic Science and Technology of China, Chengdu, 611731, China
Abstract
With the advantages of low latency, low power consumption, and high parallelism, optical neural networks (ONN) offer promising solution for time-sensitive and resource-limited artificial intelligence applications. However, the performance of the ONN model is often diminished by the gap between the ideal simulated system and the actual physical system. To bridge the gap, this work conducts extensive experiments to investigate systematic errors in the optical physical system within the context of image classification tasks. Through our investigation, two quantifi- able errors- source instability and exposure time mismatches--igniieeee significantly impact the prediction performance of ONN. To address these systematic errors, a physics-constraieed ONN learning framework constructed, including a well- designed loss function to mitigate the effect of light fluctuations, CCD adjustment strategy to alleviate the effects exposure time mismatches and a 'physics-prior- based' eтror compensation network to manage other systematic errors, ensuring consistent light intensity across experimental results and simulations. our ex- periments, the proposed method achieved test classification accuracy of 96.5% on the MNIST dataset, substantial improvement over the 61.6% achieved with the original ONN. For the more challenging QuickDraw 16 and Fashion MNIST datasets, experimental accuracy improved from 63.0% to 85.7% and from 56.2% 77.5%, respectively. Moreover, the comparison results further demonstrate the effectiveness of the proposed physics-constraing ONN learning framework over state-of-the-a -ftt---r-----ttttt ONN approaches. This lays the groundwork for more robust and precise optical computing applications.
Introduction
In recent years, optical neural networks (ONNs) have garnered significant research attention for infer- ence tasks such object detection and object classification [
4 attributed their advantages
"Equal contribution Corresponniig Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

of low energy consumption, high transmission speed, and large information capacity [5 6 77. To establish ONNs, they are typically simulated as Deep Neural Networks(DNNs)) devices and trained using the backpropagation algorithm 11 12 13 after which the trained model parameters are deployed to physical ONN systems (as shown in Fig Ia)). Theoretically, ONNs can maintain prediction performance comparable to that of their simulated electronic counterparts. How- ever, during experimental implementation, errors are inevitably introduced, unexpectedly reducing their prediction accuracy [4 15
TO] on electronic
Some measurable errors in the optical system, such as light field perturbations caused by the scattering of impurity particles in the environment and optical distortions due to lens aberrations. can be mitigated by explicitly modeling them and integrating these error models into electronic training Process[[6] [7] (as shown in Fig [b)), while other unmeasurable errors, such as ambient light effects, laser instability, and crosstalk between light fields, are difficult to correct through physical modeling, posing significant challenges in compensating for discrepancies between simulations and experiments [88 19 20
(E)
splko
Gap
Hdd mark
pirzette
Tallay tried przsate
@ealster fadtin
Uigpplyy pazearer
dytnllyy trkkel
GTreaki for specils Vale
Figure 1: (a) Experimental schematic diagram of the ONN and corresponding simulated DNN on the electronic device. (b) Error compensation method based on the physical error model (c) Hybrid training method with pure digital DNN with error compensation functionality (d) Error compensation method with an ideal physical model and a digital error compensation DNN
The introduction of "hardware- inrree-ii--ie in-the-loop" training techniques 18, 22 23
has opened up new possibilities for tackling the aforementioned challenges. Unlike in silico training. this hybrid training method incorporates the actual ONN physical response in each update loop, to mitigate the impact shown in Fig CJc). This approach allows the simulated electronic neural
of system errors network to capture the physical dynamic changes the input light field along its propagation path, including the propagation process of the light field itself, the influences of various eтror sources and disturbances encountered. However, the simulated electronic neural network, especially for the primarily linear ONNs, might be too simplistic to learn the complex input-output light field mapping relationship, including light propagation, various errors, and the coupling between light fields.
Therefore, eтror compensation neural networks proposed to better simulate the physical optical system (25). as illustrated in Fig d). The addition of the error compensation neural network in the hybrid training scheme is expected to bridge the gap between simulation and physical systems. However, due to the large optimization space and the lack of physical constraints in the simulation system, the training process can be slow and may even diverge.
which integrate physical information into the network for optimization, we investigate systematic errors in the optical physical system first and then propose a physics-constraineed ONN learning framework for image recognition tasks. In our approach, critical physical information is quantitatively integrated
Inspired by the concept of Physics-Infie Informed Neural Networks (PINNs) 26 27

---

into a "physics-prinee preeos-proo-bureee prior-based' error compensation network 30 31. This narrows the search space and reduces the complexity of the required DNNs. our experiments, the error compensation network converges rapidly and effectively minimizes the disparity between simulation results and actual observations, which significantly enhances the image recognition accuracy of experimental ONNs, leading to state- ofte-of-the-art (SOTA) performance on several datasets.
The main contributions of this work are follows:
(1) To describe the transmission equations of complex optical systems, we combine quantifiable physical information with machine learning. Given known ideal physical models and key parameters, we use minimal data and lightweight neural networks, leveraging both physics-drivee driven and data-driven approaches for rapid and precise modeling of complex optical systems.
We) informed the network of two significant quantitative errors- laser source instability and camera exposure mismatch-aa as physical prior information, greatly enhancing convergence efficiency.
Un)r multiple physical constraints, our network can focus more efficiently on learning other unmeasurable system errors beyond the two aforementioned quantitative errors, such as crosstalk between beams, device imperfections, and alignment errors. Consequently, achieves state-of-the-art results across multiple datasets.
Related Work
Training hase on the ideal optical 4f system The 4f-ONN is constructed with two Fourier lenses and Spatial light modulator (SLM) in the focal plane, modulating the frequency spectrum of the light field, which allows for the automatic realization of optical convolution operations and makes
well-suited for optical neural networks (32
Taking the optical 4f system as example, in the absence of errors, its ideal transformation process adheres to the Fresnel diffraction [35] (3) integral under ideal conditions[[S] The loss between the simulated output optical field image and the target optical field is backpropagated through gradient descent, affecting the frequency domain phase distribution of the 4f system. Through multiple training iterations on the electronic device, an optimized spectral phase distribution is obtained. This phase distribution is then loaded onto the focal plane of the 4f system via the SLM, enabling image classification tasks.
[3 3 34
Fitting errors based on the physical model. 16 aim to model the primary disturbances and errors in optical systems, integrating the disturbance model appropriately into the ideal simulated physical transformation model as an accurate description of the optical system. While this approach enhances the robustness of the model against certain interferences, it only considers one or a few disturbances existing in the physical system, thus unable to account for all disturbances, unmeasurable quantities, and their coupling and crosstalk within the model. Therefore, its description of the physical system model is not yet precise enough.
aims to describe real optical systems through autonomous learning using ONNs. By feeding input and output signals obtained from experiments into the ONNs, the ONNs learn the functionality of the optical system in a data-driven manner. Ultimately, this enables the network accurately reflect the functionality of the actual optical system. However, learning the input-output relationships in experiments might be overly complex for optical neural networks, particularly for simple linear optical neural networks.
ONNs auto-learning 24 18
integrates ideal physical models with deep neural networks (DNNs) in optical system modeling to compensate system errors and accelerate convergence speed. However, the lack of physical parameters and physics-constrained loss functions requires the network to learn this information from more data during training. Consequently, it necessitates the design of more complex DNN architectures and the acquisition of additional experimental data to support the training process.In In contrast, through the integration of physical information into the simulated model, our method able to achieve high prediction performance using a relatively simple error compensation network.
Error compensation network
3

---

3 Method
3.1 Introducing quantifiable physical information into optical systems
Introducing quantifiable physical information into optical systems is a key strategy for enhancing the accuracy and efficiency of simulations in image recognition applications. The relationship between the input and output signals of the optical system in experiments can be represented by a simplified model, depicted in Fig 2a):
g(u) f(u) + (u) + Afi(() ( +... n7(u)
(I)
where fu)) denotes the ideal transformation process. Afer indicates deviations caused by imperfec- tions in optical devices, A fjn represents deviations due to laser jitter, and 7j encompasses deviations caused by unmeasurable quantities in the system.
Physical
(a)
(b)
WW
output sigmal
Input signal
f(a) information plosical trandformation
W
Doep rocra petivakk DNN(()
yf
fio
Transformation equalions for physical sy systems g(x) = fo (00-
Qsantiathvv Partcalladematis
Mfz(o) Afse(t)mmc
< (fg x
Compuler modeling and training
Schematic diagram illustrating the integration of physical information reduce experimen- tal errors in the precise simulation physical systems.
Figure 2:
Accurate simulation of the actual optical signal transmission process necessitates not only simulating the ideal transformation of the light path but also compensating for system errors, illustrated in Fig 2(b). Here, fu) symbolizes the ideal transformation used the simulation, and DNN is employed as the neural network for error compensation, with fáv and A f representing quantifiable system errors. Methods such as the Finite Difference Time Domain 38 39 40) (FDTD) for solving the Helmholtz equation the paraxial Schrödinger equation using the split-step Fourier transform 11thod[41] 42 43) are utilized to effectively simulate the ideal transformation f'u)) of the input light field. The primary role of the DNN is to learn from the difference between the output of f()) and the experimental output, focusing on experimental errors beyond f(u) and their coupling effects-a complex task. To reduce the complexity of the DNN and ensure the convergence of our model, quantifiable physical information, such as the range of laser jitter and the grayscale value range of images received by cameras, is incorporated into the DNN. Through carefully designed loss functions 44 [55 [66] and by adjusting the overall data bias, quantifiable physical information integrated into the DNN model, thereby reducing its complexity and significantly increasing convergence speed.
3.2 Architecture of optical neural networks and measurement of quantifiable experimental errors
We employ the method described above to precisely simulate the light propagation process within an magseclassifiiatton optical neural network, where the experimental errors contribute to reduction in classification accuracy. The experimental architecture is illustrated in Fig a), we established an experimental framework for optical neural network with error compensation capabilities. Detailed descriptions of the experimental setup can be found in the supplementary materials. Using the Fresnel derive the ideal transformation process within this optical system (i.e., the function f in Equation This setup allows for precise control and manipulation of the light field, facilitating detailed investigations into the dynamics of image processing within optical neural networks.
diffraction integral,
Eout
iXfi22 [t(z, v) * c00 w/)]
Uaph
(2)

---

f = |oor P2, [max]]
(3)
where *** denotes the convolution operation, Si, f2 are focal length o1 and L2, respectively. (x, y) is the input image in DMD, loading in the form of binary transmission function. 6(x', , represents for the phase distribution on SLM and deph is a spherical wave phase factor. Due to the inherent limitations of CCD which can only detect light intensity, and given that the maximum intensity that the CCD can read is faxx the output light field must transform as dictated by Equation 3 After transforming the optical field of the input image using the optical 4f system and SLM, the output optical field is obtained. Taking the MNIST dataset as an example, the output optical field is divided into 10 equally sized regions for recognition and classification. The light intensity [ 1, 1,2, ..... 10) in each region is measured to form the intensity sequence [T, J2...., /o] The region with the highest light intensity corresponds the classification result.
In this optical system, beyond the ideal transformation process f, some quantifiable physical quan- tities can be utilized to simplify the complexity of the error compensation network. Two primary quantifiable experimental errors include the instability of laser intensity and the consistency of light intensity between the output image and the simulated image. To quantitatively analyze and mitigate these errors, we measured the instability of laser intensity and the impact of camera exposure time on the experimental output image.
3.2.1 Quantitative compensation of laser intensity instability
For the instability of laser intensity A Aja. we used a CCD to measure the overall grayscale values within a fixed area to assess the stability of the laser. Over 700 minutes, the variation of pixel values changed randomly, as depicted in FigB]b, To incorporate errors caused by intensity instability into our error compensation network, A well-designed loss function is used to increase the gap between the maximum and second maximum intensity values within the classification area. The designed loss function is shown below:
N
Loss = ReLu([Wau - -Imax 12ndmoal } + Σ (
(4)
Here, the Gap denotes the light intensity gap, N denotes the total number of classes, y represents a one-hot encoded vector that indicates the true class labels, with y/ being the i-th element of the vector y. The term ý corresponds to the network's 's output probabilities, which are typically derived through the application of a softmax function, with is representing the probability that the model assigns to the likelihood that the sample pertains to class i. In the experiment, the gap is configured to substantially exceed the range of instability variations, thereby mitigating the reduction in experimental accuracy attributable to laser instability. However, the value of WGap cannot be set too high, as this would reduce the network's 's fitting ability, leading to decreased classification accuracy. Here, we have experimentally measured the impact of different WGap values on experimental accuracy (as shown in the inset of Fig b)). When WGap is set to 10, the experimental accuracy is optimal. In this setting, the network still maintains good fitting ability, and the instability of laser intensity is also compensated. This approach ensures that despite fluctuations in laser output, the performance of the optical neural network remains robust, providing reliable and precise experimental outcomes.
3.2.2 Quantitative compensation of exposure time mismatches
For the noise caused by optical devices A experimentally. the primary and measurable error source comes from the intensity of light received by the CCD. The distribution of pixel values read by the CCD corresponds to the distribution of light intensity it receives. However, inherent biases in the CCD itself result in discrepancies between simulated and experimental pixel values, leading to reduced experimental accuracy. The CCD's pixel values depend on the exposure time and the intensity of the emitted laser light. To eliminate this error, we fixed the output laser intensity and adjusted the exposure time of CCD to measure the gap between the simulated and experimental CCD pixel values. For each exposure time, we loaded 1000 images from the MNIST dataset onto the DMD and calculated the difference in pixel values across classification regions, as shown in Fig (c). In the ideal case, the error is completely eliminated (67 = 0, as indicated by the red bars). In actual experiments, we first coarsely adjust the exposure time to bring the average 57 close to zero (green and blue bars in Fig 3 ()), and then finely adjust the exposure time to minimize the variance of 81
5

---

Physicll implement
Intenstyyypleee valuc)
1.40
1.30
LANER
Lemi
Lem2
PIS
100
90
1000
Mmlil
Ideal physical trans form + neural network
DNN (D:n)
800
Count
600
8
ideal of transform (Pretrained::))
400
200
0
51(pixel value)
Figure 3: (a) Schematic of an image classification optical neural network with an error-compensating DNN incorporating quantitative physical information. (b) The random fluctuations of output light intensity, measured in three classification regions over 700 minutes. Inset: the experimental accuracy function of light intensity gap Wap (c) The difference between simulated and experimental CCD reading values under various exposure times.
(orange bars in Fe). This method effectively eliminates errors caused by the CCD and improves classification accuracy.
3.3 Training process of physics-prior-basee error compensation network
In addition to the quantifiable system errors 48 previously discussed, several types of errors within optical systems are unmeasurable and can categorized into fixed and coupling errors. Fixed errors arise from experimental deviations or mismatches, including the misalignment of the spatial light modulator, rotation of the input image, and diffraction effects due lens apertures. Coupling errors, on the other hand, originate from the interaction between the light field and environmental impurities. As the phase map in the SLM is adjusted, while the light field accordingly altered, the fixed errors remain unchanged, whereas the coupling errors vary. These experimental errors can affect the output light field of the 4f system, causing to deviate from the output image in a perfect scenario, thereby reducing the recognition accuracy of ONN.
To mitigate the impact of such errors, DNN network is used model various environmental noises and experimental errors, aligning the ideal output image of the 4f system with the actual output image. The specific error compensation process is below:
Preetrrainn Based on the Fresnel diffraction integral, train the simulational phase distribution mo of SLM in the ONN without considering any experimental errors and environmental noises.
Error compensation: Load the phase map mi:-1 onto the SLM to obtain the actual output image. Train the enror compensation DNN ng to minimize the difference between the ideal and actual output image.
Rerrannn::: Load DNN to compensate for the errors and retrain a new phase map my of ONN to minimize the well-designed loss function (eq 4h in simulation. These networks, ONN and DNN, are connected by residual connections.
(iv)lteration:
The coupling errors altered when the phase map changes from m m4- To compensate for that, repeat steps (ii) and (iii, iterating until the experimental classification accuracy maximized.
Through such an iteration process, the error compensation network can model effectively the en- vironmental noises and experimental errors, significantly improving the classification accuracy.

---

Incorporating quantifiable physical data, such as laser intensity fluctuations and camera exposure effects, directly into the network's architecture enables to adapt more robustly the inherent variability of optical systems.For more detailed information on the training process, please refer to the supplementary materials.
Result
4.1 Dataset
The MNIST (Modified National Institute of Standards and Technology) 500 51 dataset consists of 70,000 28x28 pixel grayscale images of handwritten digits, widely used benchmark for training and testing image processing systems in machine learning and computer vision. The QuickDraw 16 dataset, subset of Google's Quick, Draw! project includes 16 categories of hand-drawn images that mimic natural variations in handwritten and sketched drawings, serving valuable resource for image classification and recognition tasks. The Fashion MNIST dataset[52], provided by Zalando, contains 70,000 28x28 pixel grayscale images categorized into 10 different fashion items, such as T-shirts, trousers, shoes, and bags. Designed as a direct replacement for the traditional MNIST dataset, Fashion MNIST is extensively used for rescarch and evaluation in image classification tasks. During training, we used the entire training set. To evaluate accuracy, we randomly selected 1,000 images from the test sets of each of the three datasets to assess experimental accuracy.
4.2 Error compensation network without quantifiable physical information
In comparison to our approach of incorporating quantifiable physical information into the error compensation network, we also conducted an experiment where DNN was trained to model the system errors without introducing any quantifiable physical data. As illustrated in Fig 4a)), the network trained only on the ideal transformation process of the light field achieved an experimental accuracy of only 61.6%. This lower accuracy is due to several factors, including the instability of the laser A fit. mismatches in exposure time f and various unmeasurable errors 17, which resulted in significant deviations between the output light field images in experiments and simulations, as shown Fig 4 a2-a33.. The accuracy image recognition depends on the distribution of light intensity across ten regions within the output image; thus, the experimental accuracy was significantly reduced.
Without Physical Information
Without Plasical Information
Compensation
With Compensation DNN
122
180
(al)
Esperimect
Compensation DNN
1224
(a3)
(13)
Alghtal
Figure 4: The simulation and experiment result with and without compensation DNN MNIST dataset, maintaining without quantifiable physical information. (a1,b1) The simulational light intensity in ten classification regions depicted the inset. (a2.b2)The experimental light intensity in classification regions is depicted in the inset. (a3,b3) Histogram of light intensity difference 81 between simulation and experiment. (44, Exxpermnatti confusion matrix.
After incorporating an error compensation network without quantifiable physical information, the discrepancies light intensity distribution between experiments and simulations were mitigated, as shown in Fig (b4), boosting the experimental accuracy to 93.5%. As indicated in Fig 4 (3), the role of the error compensation DNN at this point was twofold: to modulate the overall light intensity and to adjust the local distribution of light intensity, thereby narrowing the gap between

---

simulation and experiment. However, as observed in Fig the modulation resulted in a small difference between the highest and second- hrihed- light intensities, reducing the robustness of the neural network.
4.3 Error compensation network with quantifiable physical information
By integrating quantifiable physical information, effectively compensated for system caused by laser instability A fja and mismatches in exposure time, represented as A fdev- Consequently, the DNN now primarily focuses on compensating for unmeasurable errors 1). which challenging to predict and correct. This approach has significantly improved the convergence speed and accuracy of the error compensation network. For the MNIST dataset, we mitigated the error fav by adjusting the exposure of the CCD, thus ensuring that the overall light intensity between the simulation and experiment was consistent. This adjustment allowed the DNN to focus solely on modulating the local distribution of light intensity, thereby simplifying its complexity. As shown in Fig (a4),, even without an error compensation network, the classification accuracy increased to 83.7% due to the consistency of the overall light intensity distribution.
inherently more
Physical Information Compensation DNN
With Physical Information
With Compervation
Simollation
Experiment
Strmntation
Experiment
Compensation DNN
Cluss
Clun
(x3)
(h-3
(4
4tpbdrvdta
Figure 5: The simulation and experiment result with and without compensation DNN on MNIST dataset, maintaining with quantifiable physical information. (al.bl) The simulational light intensity in ten classification regions depicted the inset. (a2,b2)The experimental light intensity in ten classification regions is depicted in the inset. (a3,b3) Histogram of light intensity difference 57 between simulation and experiment. (44.b4))xperimentt confusion matrix.
Additionally, well-designed loss function was used to address the instability of laser intensity, Afn. ensuring that the difference in light intensity between the maximum and the second maximum values within the classification regions was significantly greater than the range of fluctuations in laser intensity. After training, this error compensation network, enhanced with quantifiable physical information, increased the classification accuracy of the MNIST dataset to 96.5%.
4.4 Further experiment in different datasets
For the more complex Quickdraw 16 dataset and FMNIST dataset, applying the same method of adjusting the CCD exposure to ensure consistency in overall light intensity, and using well-designed loss function to address the instability of laser intensity significantly enhanced the recognition accuracy. The classification accuracy for the Quickdraw 16 dataset increased from 63.0% to 85.7% through these adjustments. Similarly, the classification accuracy for the FMNIST dataset improved from 56.2% to 77.3% through these adjustments. The analysis the convergence speed of the error compensation network can be found in the supplementary materials, Our approach can also handle more challenging datasets like CIFAR-10. To do this, we simply replace the dataset loaded onto the DMD with CIFAR-10. The initial classification accuracy with CIFAR- 10 is 30%, which improves to 57% after optimization using our method. However, since current research[25] 377 on spatial 2D light-based ONNs predominantly uses the datasets mentioned in the paper, we do not present a detailed and specific demonstration of the results of CIFAR- 10 in this paper due to the lack of comparative benchmarks.

---

Without Physical Information
With Physical Information
Without Compensation DNN
With Compensation DNN
Quickdrra
Simulation
Quickdraw
Simulation
(c))
FLAIIIUOOOO Cle
Compensation
Expersment (a2)
DNN
Einger clunent
PANIST
Simelatiio (b2)
PSNIST
Simulation
Experiment
Experiment (d.3)
Figure 6: (al) Experimental confusion matrix in Quickdraw 16 dataset without physical information and compensation DNN. (b1) Experimental confusion matrix in FMNIST dataset without physical information and compensation DNN. (c1) Expeexrrimttt confusion matrix in Quickdraw 16 dataset with physical information and compensation DNN. (d1 Experimental confusion matrix in FMNIST dataset with physical information and compensation JaN.(a2.b2.c2.dl The simulational light intensity in classification regions. (a3,b3,c3,d3) The experimental light intensity in classification regions.
4.5 Accuracy comparison with other error compensation network
We have compared our work with other studies and found that our experimental classification accuracy on the MNIST dataset aligns closely with the results from Tsinghua University (25] and those published in Nature magazine; on the QuickDraw16 16 dataset, we achieved an experimental classification accuracy 85%. These results indicate that, unlike models that do not incorporate physical information, our approach of quantitatively introducing physical information has enabled our optical-DNN to be fed with known measurable experimental perturbations and errors. This allows the DNN that follows the ideal model to learn more about unknown perturbations, errors, and imperfections in optical devices, and their interdependencies. Consequently, compared to neural networks that do not incorporate physical information quantitatively, optical-DNN(optica- Deep Neural Network) able to more accurately represent real experimental systems, enhancing the consistency between simulation and experimental results, and further improving experimental accuracy. Additionally, our eтror compensation network is very lightweight, with approximately 5,000 learnable parameters(For more details, please refer to the supplementary materials). Therefore, the introduction of quantitative physical information into neural networks is crucial, especially when dealing with complex datasets that require highly precise simulations of actual physical processes.
This work Qualitative
This work Quantifiable
Hybrid CNN
DAT 25
PAT
Architecture
Directly deployed
Directly deployed
Directly deployed
Directly deployed
optimizc
optimize
optimize
optimize
MNIST EMNIST
24.9% 8.4%
92.4% 5%
24.9%
61.4%
61.6%
93.5%
83.7%
96.5%
77.3%
56.2%
77.5%
Quickdraw16
Table The accuracy comparison between other error compensation network architecture in various datasets.
5 Conclusion
In this study, we explore the mapping relationship between input and output physical ONN system, which can be modeled as the sum of ideal transformation process, measurable errors (laser-

---

related deviations, CCD-related deviations) and other unmeasurable enrors. To reduce the disparity between the ideal simulated ONN model and the real physical ONN model, a physics-constrained ONN learning framework is constructed for image classification tasks. Specifically, we introduce well-designed loss function to mitigate the laser-related deviations and CCD adjustment strategy to reduce the CCD-related deviations. In addition, a "physies-piir--aseee prior-based' error compensation network is proposed to manage other unmeasurable errors. The effectiveness of our approach demonstrated through extensive experiments on the MNIST, Fashion MNIST and QuickDraw 16 datasets.
Although we aim to reduce the gap between the ideal simulated ONN model and the real physical ONN model, it is important to note that our research primarily focuses on the image classification task. Specifically, the intensity gap loss function we propose may not be applicable to other Al tasks, However, we believe that the quantifiable error analysis and processing methods could serve as valuable references for other applications, as the two quantifiable errors-light source instability and exposure time mismatches-ant are inevitable in current optical systems. For future work, we plan to develop a more generalizable approach address systematic errors in ONN systems and investigate its application in other Al tasks, such image restoration and image segmentation.
Acknowledgments and Disclosure of Funding
This work was supported by the Shanghai Outstanding Academic Leaders Plan (No. (DXII4402000) and the Scientific and Technological Innovation Funds of Shanghai Jiao Tong University. Additionally, it received partial support from the National Natural Science Foundation of China under grant U23B2011.
References
[1] Xiangyan Meng, Guojie Zhang, Nuannuan Shi, Guangyi Li, José Azaña, José Capmany, Jianping Yao, Yichen Shen, Wei Li, Ninghua Zhu, et al. Compact optical convolution processing unit based on multimode interference. Nature Communications, 14(1):3000, 2023.
[2] Ziyu Gu, Yesheng Gao, and Xingzhao Liu. Optronic convolutional neural networks of multi- layers with different functions executed in optics for image classification. Optics Express, 29(4):5877-5889, 2021.
[3] Yanbing Liu, Shaochong Liu, Tao Li, Tianyu Li, Wei Li, Guoqing Wang, Xun Liu, Wei Yang, and Yuan' an Liu. Towards constructing doe-based practical optical neural system for ship recognition in remote sensing images. Signal Processing, page 109488, 2024.
[4] Xing Lin, Yair Rivenson, Nezih Yardimci, Muhammed Veli, Yi Luo, Mona Jarrahi, and Aydogan Ozcan. All-optical machine learning using diffractive deep neural networks. Science, 16406 2018
[5] Tianyu Wang, Yha-Yuan Ma, Logan G Wright, Tatsuhiro Onodera, Brian Richard, and Peter McMahon. An optical neural network using less than I photon per multiplication. Nature Communications,, 13(1):123, 2022.
[6] HH Zhu, Jun Zou, Hengyi Zhang, YZ Shi, SB Luo, N Wang, H Cai, LX Wan, Bo Wang, XD Jiang, et al. Space-efficient optical computing with an integrated chip diffractive neural network. Nature communications, 13(1):1044, 2022.
[7] Ryan Hamerly, Liane Bernstein, Alexander Sludds, Marin Soljačić, and Dirk Englund. Large- scale optical neural networks based on photoelectric multiplication. Physical Review X. 92):021032, 2019.
[8] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521((753)::436 444, 2015.
[9] Andreas Kamilaris and Francese X Prenafeta-Bolda Deep learning in agriculture: A survey, Computers and electronics in agriculture, 147:70-90, 2018
[10] Yanming Guo, Yu Liu, Ard Oerlemans, Songyang Lao, Song Wu, and Michael $ Lew. Deep learning for visual understanding: A review. Nenrocomputing, 187:27-48. 2016.
10

---

[11] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by pack-propagatin errors. nature, 323(6088):533-556 1986.
[12] Ali Momeni, Babak Rahmani, Matthieu Malléjac, Philipp Del Hougne, and Romain Fleury. Backpropagation-Fi training of deep physical neural networks. Science, (82(6676):1227-111 2023.
[13] Sunil Pai, Zhanghao Sun, Tyler W Hughes, Taewon Park, Ben Bartlett, Ian AD Williamson, Momchil Minkov, Maziyar Milanizadeh, Nathnael Abebe, Francesco Morichetti, et al. Ex- perimentally realized situ backpropagation for deep learning photonic neural networks. Science, 380(6643):398 4044, 2023.
[14] Julie Chang. Vincent Sitzmann, Xiong Dun, Wolfgang Heidrich, and Gordon Wetzstein. Hybrid optical-clectronic convolutional neural networks with optimized diffractive optics for image classification. Scientific reports, 8(1):1-10, 2018.
[15] Mario Miscuglio, Zibo Hu, Shurui Li, Jonathan K George, Roberto Capanna, Hamed Dalir, Philippe M Bardet, Puneet Gupta, and Volker J Sorger. Massively parallel amplitude-only fourier neural network. Optica, 7(12):1812-1819) 2020.
[16] Deniz Mengu, Yifan Zhao, Nezih T Yardimei, Yair Rivenson, Mona Jarrahi, and Aydogan Ozcan. Misalignment resilient diffractive optical networks. Nanophotonics, 9(13):227--42) 4219, 2020.
[17] Hoda Sadeghzadeh and Somayyeh Koohi. inntlatiiooiiviiaant optical neural network for image classification. Scientific Reports, 12(1):17232, 2022.
[18] Hans-Christian Ruiz Euler, Marcus N Boon, Jochem T Wildeboer, Bram van de Ven, Tao Chen, Hajo Broersma, Peter A Bobbert, and Wilfred G van der Wiel. A deep-learning approach to realizing functionality in nanoelectronic devices. Nature nanotechnology, 15(12):992-998, 2020.
[19] Yichen Shen, Nicholas C Harris, Scott Skirlo, Mihika Prabhu, Tom Bachr-Jones, Michael Hochberg, Xin Sun, Shijie Zhao, Hugo Larochelle, Dirk Englund, et al. Deep learning with coherent nanophotonic circuits. Nature photonics, 11(7):41-446, 2017.
[20] Julian Bueno, Sheler Maktoobi, Luc Froehly, Ingo Fischer, Maxime Jacquot, Laurent Larger, and Daniel Brunner. Reinforcement learning in a large-scale photonic recurrent neural network. Optica, 5(6):756-760, 2018.
[21] Yuchi Huo, Hujun Bao, Yifan Peng, Chen Gao, Wei Hua, Qing Yang. Haifeng Li, Rui Wang, and Sung-Ear Eui Yoon. Optical neural network via loose neuron array and functional learning. Nature Communications, 14(1):2535, 2023.
[22] Darcy Bullock, Brian Johnson, Richard Wells, Michael Kyte, and Zhen Li. Hardware-in- the-loop simulation. Transportation Research Part C: Emerging Technologies, 12(1):73-89, 2004.
[23] Sebastian Schmitt, Johann Klähn, Guillaume Bellec, Andreas Grübl, Maurice Guettler, Andreas Hartel, Stephan Hartmann, Dan Husmann, Kai Husmann, Sebastian Jeltsch, et al. Neuromorphic hardware the loop: Training a deep spiking network on the brainscales wafer-scale system. In 2017 international joint conference on neural networks (IJCNN, pages 2227-2234 IEEE, 2017.
[24] James Spall, Xianxin Guo, and Alexander Lvovsky. Hybrid training of optical neural networks. Optica, 9(7):803-811. 2022.
(25) Ziyang Zheng, Zhengyang Duan, Hang Chen, Rui Yang, Sheng Gao, Haiou Zhang, Hongkai Xiong, and Xing Lin. Dual adaptive training of photonic neural networks. Nature Machine Intelligence, 1110): 2023.
[26] Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Kariiadaiis. Physics-informed neural networks (pinns) for fluid mechanies: review. Acta Mechanica Sinica, 37212):177-117388 2021.

---

[27] Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics- informedd neural networks: Where we are and what's next. Journal of Scientific Computing, 92(3):88, 2022.
[28] Zhiping Mao, Ameya D Jagtap, and George Em Karniadakis. Physics-informed neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering, 360: 1 12789, 2020.
[29] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics- informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
[30] Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelios Koutsourelakis, and Paris Perdikaris. Physics- constrained deep learning for high-dimensional surrogate modeling and uncertainty quantifica- tion without labeled data. Journal of Computational Physics, 394:56 111556-81,22 81, 2019.
[31] Mo Deng, Shuai Li, Zhengyun Zhang. Iksung Kang, Nicholas Fang, and George Barbastathis. On the interplay between physical and content priors in deep learning for computational imaging. Optics Express, 28(16):24152224 100, 2020.
[32] Sudhir Cherakulappurath. Georges Boudebs, and André Monteil. 4f coherent imager system and its application to nonlinear optical measurements. JOSA B, 21(2)2273- 2004.
[33] Nien-An Chang and Nicholas George. Speckle in the 4f optical system. Applied optics, -104):A11-AA0,, 2008
[34] Hoda Sadeghzadeh and Somayyeh Koohi. High-speed multi-layer convolutional neural network based on free-space optics. IEEE Photonics Journal, 14(4):1-12, 2022.
[35] CJR Sheppard and M Hrynevych. Diffraction by a circular aperture: a generalization of fresnel diffraction theory. JOSA A, (22)2274-281, 1992.
[36] Yusuf Z Umul. Equivalent functions for the fresnel integral. Optics express, 13(21):8469-848) 2005.
[37] Logan G Wright, Tatsuhiro Onodera, Martin M Stein, Tianyu Wang, Darren T Schachter, Zoey Hu, and Peter L. McMahon. Deep physical neural networks trained with backpropagation. Nature, 601(7894):549-5555 2022.
[38] Karl S Kunz and Raymond J Luebbers. The finite difference time domain method for electro- magnetics. CRC press, 1993.
[39] Allen Taflove, Susan C Hagness, and Melinda Piket-May. Computational electromagneties: the finite-difference time-domain method. The Electrical Engineering Handbook, 3662-6670): 2005.
[40] Allen Taflove. Review of the formulation and applications of the finite-difference time-domain method for numerical modeling of electromagnetic wave interactions with arbitrary structures. Wave Motion, 1066):547-582, 1988.
[41] Simone Gaiarin, Francesco Da Ros, Rasmus T Jones, and Darko Zibar. End-to-end optimization of coherent optical communications over the split-step fourier method guided by the nonlinear fourier transform theory. Journal of Lightwave Technology, 39(2):418 428, 2020.
[42] Oleg V Sinkin, Ronald Holzlöhner, John Zweck, and Curtis R Menyuk. Optimization of the split-step fourier method in modeling optical-fiber communications systems. Journal of lightwave technology, 21(1):61, 2003.
[43] GM Muslu and HA Erbay. A split-step fourier method for the complex modified korteweg-de vries equation. Computers & Mathematics with Applications, 45(1-3):503-5144 2003.
[44] Qi Wang, Yue Ma, Kun Zhao, and Yingjie Tian, A comprehensive survey of loss functions in machine learning. Annals of Data Science, pages 1-26, 2020.
12

---

[45] Lorenzo Rosasco, Ernesto De Vito, Andrea Caponnetto, Michele Piana, and Alessandro Verri. Are loss functions all the same? Neural computation. 16(5):1063-1076 2004.
[46]
Katarzyna Janocha and Wojciech Marian Czarnecki. On loss functions for deep neural networks in classification. arXiv preprint arXiv: (XXv:/702..5659, 2017.
[47] Youfang Cao, Anna Terebus, and Jie Liang. State space truncation with quantified errors for accurate solutions to discrete chemical master equation. Bulletin of mathematical biology, 78:617-661, 2016.
[48] Rasool Khadem, Clement C Yeh, Mohammad Madeghi-Tehhann Michael R Bax, Jeremy Johnson, Jacqueline Nemny Welch, Eric P Wilkinson, and Ramin Shahidi. Comparative tracking error analysis of five different optical tracking systems. Computer Aided Surgery, 5(2):98-107, 2000.
[49] Haitao Sun and Jochen Autschbach. Influence of the delocalization error and applicability of optimal functional tuning density functional calculations of nonlinear optical properties of organic donor-acceptor chromophores. ChemPhysChem 14(11):2450-246) 2013.
[50] Alejandro Baldominos, Yago Saez, and Pedro Isasi. A survey of handwritten character recogni- tion with mnist and emnist. Applied Sciences, (15):3116, 2019.
[51] Norman Mu and Justin Gilmer. Mnist-c: A robustness benchmark for computer vision. arXiv preprint arXiv: 996.0023777 2019.
[52] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 178.077777 2017.
13

---

NeurIPS paper checklis
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
Supplemental material and Section4
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuiu wwwiii/gguie///eeeeeeoeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section4 Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Supplemental material
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpprs////uripss http///urrs.cccppuii/i/e htt.://.......blccctttcccccttted
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Section4
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee wwworeewwthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "Y3aQ89qYnX", "reviewer": "Reviewer_8n42", "summary": "This work presents a physics-constrained framework to improve optical neural networks for image classification. The study identifies and reducing two key errors: light source instability and exposure time mismatches.The authors introduce a physics-constrained ONN learning framework that includes a specialized loss function to reduce light fluctuations, a CCD adjustment strategy to handle exposure time mismatches, and a physics-prior-based error compensation network to manage other systematic errors. This approach significantly enhances classification accuracy on various datasets, achieving 96.5% on MNIST, 85.7% on QuickDraw16, and 77.5% on Fashion MNIST, compared to lower baseline performances.", "questions": "How well does your framework generalize to datasets with different characteristics (e.g., noise levels, resolutions, and complexity)?\n\nHow did you determine the parameters for the loss function?", "limitations": "The authors have identified key limitations, particularly their focus on image classification tasks and the specific applicability of their intensity gap loss function. I think autohors should also discuss how the framework might perform under real-world conditions in this work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality: The paper introduces a novel approach by implementing a physics-constrained learning framework to enhance the performance of optical neural networks (ONNs). This framework specifically targets physical errors like light source instability and exposure time mismatches, providing unique solutions to these problems. \nQuality: The authors identify and quantify key sources of error in physical ONNs and offer practical solutions to mitigate these errors. The substantial improvements in classification accuracy across several datasets underscore the robustness and effectiveness of the framework. \nClarity:The methodology is explained in detail, with clear descriptions of the error sources, proposed solutions, and the experimental setup. Diagrams and equations clear illustrate the points.\nSignificance: By narrowing the gap between simulated and physical ONNs, the research provides a way to more accurate and reliable optical computing applications.", "weaknesses": "Providing an analysis of the computational costs associated with the proposed framework, including comparisons of training and inference times.\nIncluding a more in-depth analysis of how the proposed methods differ fundamentally from other approaches would clarify their unique advantages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your thorough review and the positive feedback on our work. As you mentioned, the core aim of our study is to narrow the gap between simulated and physical ONNs and to provide a pathway for more accurate and reliable optical computing applications. Your description is a very precise summary of our paper. \n\n1: (1) Providing an analysis of the computational costs associated with the proposed framework, including comparisons of training and inference times. (2) Including a more in-depth analysis of how the proposed methods differ fundamentally from other approaches would clarify their unique advantages.\n\nRegarding (1), during the training of the ONN, we train based on the transmission equations of light in a 4f system, achieving end-to-end mapping through the adjustment of the system's tunable parameters. The DNN used is a lightweight fully connected structure, which converges quickly. The only limitation on training efficiency is the acquisition of experimental images. To address this, we developed a synchronized trigger control unit to ensure that image refresh rates in the DMD (Digital Micromirror Device) are synchronized with camera image capture. The DMD refresh rate is typically very high, capable of refreshing an image every 10 microseconds, but industrial cameras often struggle to match this speed. Consequently, training and forward inference efficiency depend on the camera frame rate; using a higher frame rate camera can improve training efficiency and accelerate inference times.\n\nRegarding (2), unlike other methods, we are the first to incorporate quantitative physical information into network training. By integrating this quantitative physical data into the model training, we are able to reduce model complexity of error compensation networks and enhance convergence efficiency as discussed in Convergence Rate section of supplemental material.\n\n2：（1）How well does your framework generalize to datasets with different characteristics (e.g., noise levels, resolutions, and complexity)?（2）How did you determine the parameters for the loss function?\n\nOur experiments were conducted on three datasets with different characteristics: MNIST, Fashion-MNIST, and QuickDraw16. Additionally, we also processed the CIFAR-100 dataset. The features extracted for classification from these datasets and the number of categories differ significantly, and there is substantial variation in the brightness and contrast among the datasets. Additionally, the optical paths used in practice are highly sensitive to variations in brightness and contrast. For the MNIST and CIFAR-100 datasets, our method improved accuracy from 61.6% and 30% to 96.5% and 57%, respectively. Ablation studies across these datasets demonstrate that our method consistently aligns simulation and experimental results and significantly enhances experimental accuracy for datasets with varying features. Regarding the noise levels in the datasets, due to time constraints, we plan to incorporate noise into the data in upcoming experiments to further assess the generalization capability of our method.\n\nThe parameters for the loss function are determined based on physical prior information. For example, due to factors such as laser jitter, camera dark current, and other irregular noises, the grayscale values of the same region in an image are not constant but fluctuate over time. We record the range of grayscale value fluctuations and use this information to design the relevant parameters in the loss function.\n\n3：Limitations：The authors have identified key limitations, particularly their focus on image classification tasks and the specific applicability of their intensity gap loss function. I think authors should also discuss how the framework might perform under real-world conditions in this work.\n\nThe issues you raised are highly practical. As shown in the Experimental Setup section of the supplementary materials, the actual optical layout of the 4f system indeed presents challenges such as large volume, numerous optical components, and difficulty in optical alignment. However, our proposed method addresses common errors in optical systems and ensures consistency between experimental and simulation results, significantly improving the final experimental accuracy even in error-prone 4f systems. Therefore, we believe that our method can be effectively applied to single-element systems, such as metasurface-based ONNs, which are more suited for real-world conditions.\n\nWe sincerely thank you for your valuable feedback and thoughtful comments. Your insights have greatly contributed to improving the quality of our manuscript. We appreciate the time and effort you have dedicated to reviewing our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dtuyFfeDr2", "reviewer": "Reviewer_M382", "summary": "This paper proposes to use two analytical error terms to model laser jitter noise and camera exposure time error, and use a DNN-based error compensation network to correct unknown noises for higher accuracy after deployment.", "questions": "Weaknesses:\n1.\tNovelty is quite limited. Using physical or analytical error modeling in training for error compensation is common practice. Using a neural network to fit unknown errors to emulate the response of real hardware is also widely used in analog ONNs. What novel techniques are proposed here, especially for NeurIPS, which is the main ML contribution? Claiming “analytical noise modeling in optimization” as physics-informed neural network will not make it a novel contribution.\n\n2.\tThe parallel residual DNN compensation network on digital hardware significantly reduces the system throughput and efficiency. What are the main advantages of such a system compared to analog AI hardware or digital accelerators?\n\n3.\tThe experiments are performed in very small examples. For ONNs submitted to NeurIPS, CIFAR100 or TinyImageNet are required.\n\n4.\tIt does not compare the efficiency and robustness with extensive literature on robustness-driven hardware/software co-design methods or on-chip training, hybrid training methods. It only compares to direct deployment.", "limitations": "N/A", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "It has thorough ablation study on the effectiveness of the error compensation terms and DNN compensation model and have experimental results to validate the effectiveness.", "weaknesses": "Weaknesses:\n1.\tNovelty is quite limited. Using physical or analytical error modeling in training for error compensation is common practice. Using a neural network to fit unknown errors to emulate the response of real hardware is also widely used in analog ONNs. What novel techniques are proposed here, especially for NeurIPS, which is the main ML contribution? Claiming “analytical noise modeling in optimization” as physics-informed neural network will not make it a novel contribution.\n\n2.\tThe parallel residual DNN compensation network on digital hardware significantly reduces the system throughput and efficiency. What are the main advantages of such a system compared to analog AI hardware or digital accelerators?\n\n3.\tThe experiments are performed in very small examples. For ONNs submitted to NeurIPS, CIFAR100 or TinyImageNet are required.\n\n4.\tIt does not compare the efficiency and robustness with extensive literature on robustness-driven hardware/software co-design methods or on-chip training, hybrid training methods. It only compares to direct deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "I am very grateful to your comments for the manuscript. Your questions were answered below.\n\n1: Novelty is quite limited. (1) Using physical or analytical error modeling in training for error compensation is common practice. (2) Using a neural network to fit unknown errors to emulate the response of real hardware is also widely used in analog ONNs. \n\nThe purpose of this paper is not to emulate the response of actual hardware for building simulation-based ONN models. Instead, we aim to use deep neural networks to ensure that the actual optical path accurately replicates the results of the simulation model, ultimately achieving consistency between simulation and experimental results.\n\nRegarding point (1), while error compensation training based on physical error model analysis is widely used in Physics-Informed Neural Networks (PINNs), this work is the first work to conduct large-scale physical experimental analysis of light source instability and exposure time mismatches in optical neural networks (ONNs). These issues are common in optical setups and significantly impact the final experimental accuracy. We have proposed corresponding solutions to address the issue. As Reviewer 8n42 mentioned, our proposed strategies have significantly narrowed the gap between simulated and physical ONNs. Our research thus provides a pathway to more accurate and reliable optical computing applications. Moreover, the physical experimental results we have documented can serve as a valuable reference for other researchers engaged in innovations in optical networks.\n\nRegarding point (2), unmeasurable errors in optical systems are categorized into fixed and coupling errors (analyzed in Section 3.3 of the main text). While fixed errors can be addressed using DNNs, our approach updates optical system parameters each epoch, causing changes in coupling errors. We synchronize DNN updates with these parameters, allowing the DNN to correct both fixed and coupling errors more effectively. This synchronization has significantly improved experimental ONN performance. As noted by Reviewer W9o4, our method achieves state-of-the-art results on the MNIST dataset. Our method represents a significant innovation, offering unique advantages over existing approaches and demonstrating its potential for laying the foundation for the practical development of ONNs.\n\n2: The parallel residual DNN compensation network on digital hardware significantly reduces the system throughput and efficiency. What are the main advantages of such a system compared to analog AI hardware or digital accelerators?\n\nAs noted, our approach uses residual connections between the ideal ONN model and the DNN to ensure consistency between the simulation model and the actual optical path. The DNN's ~5,000 parameters, compared to the ONN's 360,000(as detailed in the supplementary materials under Model Architecture), minimally impact system efficiency while improving consistency between experimental and simulation results.\n\nAdditionally, the DNN affects only the training complexity, not the inference process. In practical engineering applications, we are more concerned with the inference process rather than the training process. During inference, ONNs offer advantages such as low power consumption, high parallelism, and strong resistance to interference compared to analog AI hardware or digital accelerators. These advantages are highlighted in the papers “Deep Physical Neural Networks Trained with Backpropagation” published in Nature and so on.\n\n3: The experiments are performed in very small examples. For ONNs submitted to NeurIPS, CIFAR100 or TinyImageNet are required.\n\nFor the CIFAR-100 dataset, our system was adapted by replacing the DMD images with CIFAR-100 images, maintaining the training process as described. For CIFAR-100, we used only one channel of grayscale images for training and forward experiments, achieving an improvement in classification accuracy from an initial 30% to 57%, further demonstrating the effectiveness of our method. Due to a lack of comparative results, this was not included in the main text. Current ONNs using 2D spatial structured light primarily utilize the datasets mentioned in our paper, as noted in the aforementioned articles and \"Hybrid Training of Optical Neural Networks\" in Optica.\n\n4: It does not compare the efficiency and robustness with extensive literature on robustness-driven hardware/software co-design methods or on-chip training, hybrid training methods. It only compares to direct deployment.\n\nTable 1 compares our classification performance with methods from \"Dual Adaptive Training of Photonic Neural Networks\" in Nature Machine Intelligence and \"Deep Physical Neural Networks Trained with Backpropagation\" in Nature. Section B of the supplementary materials details the convergence rate, showing that our method achieves convergence within 5 epochs despite the lower precision of initial experimental deployment. These results underscore our advantages in both accuracy and training efficiency.\n\nIn Table 1, PAT and DAT are hybrid training methods. Our results show that our method outperforms these in both training efficiency and final experimental accuracy. On-chip learning, which involves backpropagating optical signals through hardware, introduces significant experimental errors and is thus highly challenging. Most researchers use optical forward inference combined with loss function computation and backpropagation in computer systems, as seen in the papers \"Deep Physical Neural Networks Trained with Backpropagation\" in Nature and \"Backpropagation-Free Training of Deep Physical Neural Networks\" in Science. Therefore, we did not compare our method with on-chip learning approaches.\n\nFinally, we sincerely thank you for your valuable feedback on our paper. We will address the issues you raised through further in-depth research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5s5toaqmx9", "reviewer": "Reviewer_W9o4", "summary": "In this paper, the author proposed the use of a 4f optical system to implement alternative neural networks due to the advantages of reduced latency and lower power consumption. The main challenges identified are optical misalignments caused by dispersion and imperfections in optical fiber fabrications.", "questions": "1. One drawback of the 4f approach is that it requires three elements (two lenses and a spatial filter), resulting in a bulky optical system with a greater propensity for misalignments compared to single-element optical systems. Therefore, The author might need to explain why use the 4f system, and how does it compare with meta surface approaches?\n\n2. It is also unclear how the author handles the nonlinearity issue. The author only mentioned that they use a linear ONN, but dropping nonlinearity from NNs is non-trivial. It might be better to explain this gap.", "limitations": "1. The experiments were conducted only on the MNIST level dataset. The 4f system might not perform as well on more challenging datasets, such as CIFAR or ImageNet. While I don't think the author needs to conduct these additional experiments, some discussions would be great.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Motivation:** The motivation is clear, since errors from optical systems are unavoidable.\n2. **Experimental Improvement:** The experiments showed a clear improvement, achieving approximately a 20% enhancement and presenting state-of-the-art ONN approaches on the MNIST dataset.", "weaknesses": "1. **Limited Experimentation:** The experiments were conducted only on MNIST or similar, similar challenging tasks, which can be effectively addressed even with basic approaches. The author might need to validate their approach on more complex datasets. A simple single-layer fully connected (FC) network could handle MNIST, so the robustness of the proposed method on more challenging tasks should be explored.\n2. **Broader Impact:** The author should emphasize the benefits of optical neural networks (ONNs) for a broader audience, such as comparing power consumption more comprehensively.\n3. **Previous Work:** Some important previous work, such as from the \"meta surface ONN\" field, might be missing. Notable references include or discussed:\n   - [1] Photonic Advantage of Optical Encoders\n   - [2] Compressed Meta-Optical Encoder for Image Classification\n   - [3] The Physics of Optical Computing\n   - [4] Optical convolutional neural network with atomic nonlinearity", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and for acknowledging the motivation and improvements in our study. Our work addresses errors in practical optical neural networks, enhances experimental performance, and aims to advance reliable optical computing. We will reply your questions and suggestions below.\n\n1: Limited Experimentation: The experiments were conducted only on MNIST or similar, similar challenging tasks, which can be effectively addressed even with basic approaches. The author might need to validate their approach on more complex datasets. A simple single-layer fully connected (FC) network could handle MNIST, so the robustness of the proposed method on more challenging tasks should be explored.\n\nThank you for your question about the robustness of our method. Our approach can handle more challenging datasets like CIFAR-100. To do this, we simply replace the dataset loaded onto the DMD with CIFAR-100 images. The initial classification accuracy with CIFAR-100 is 30%, which improves to 57% after optimization using our method. However, since current research (\"Dual Adaptive Training of Photonic Neural Networks\" published in Nature Machine Intelligence, and \"Deep Physical Neural Networks Trained with Backpropagation\" published in Nature.) on spatial 2D light-based ONNs predominantly uses the datasets mentioned in the paper, we did not include the CIFAR-100 classification results in the main text due to the lack of comparative benchmarks.\nMoreover, the core contribution of this paper is addressing the inconsistency between simulation and actual optical neural networks, rather than just developing a high-performing ONN. For instance, experimental accuracy for MNIST was 61.6% without our method, but improved to 96.5% with it. Similarly, accuracies for CIFAR-100 and Quickdraw16 increased from 30% and 63% to 57% and 85.7%, respectively. These results validate the effectiveness and robustness of our method in correcting experimental errors.\n\n2：One drawback of the 4f approach is that it requires three elements (two lenses and a spatial filter), resulting in a bulky optical system with a greater propensity for misalignments compared to single-element optical systems. Therefore, the author might need to explain why use the 4f system, and how does it compare with meta surface approaches?\n\nThank you for pointing out the drawback of the 4f approach. We agree that it is more prone to misalignments than single-element systems, and its bulkiness complicates error correction. However, since the errors analyzed are common to optical systems, our method can also be effectively applied to single-element systems, where misalignment errors are easier to manage.\nWe agree that the meta surface approach is better for spatially structured light computing due to its industrial potential and benefits for miniaturizing ONNs. Our other team is already working on meta surface-based ONNs. We used the 4F system here because it's a classic setup that allows us to validate our method in the laboratory, but our approach is also applicable to meta surface-based ONNs.\n\n3：It is also unclear how the author handles the nonlinearity issue. The author only mentioned that they use a linear ONN, but dropping nonlinearity from NNs is non-trivial. It might be better to explain this gap.\n\nYour comments on nonlinearity are crucial. Our ONN includes a nonlinear activation function at the camera's receiving surface, performing a squared transformation on the light field matrix to output its squared magnitude. This operation is recognized as nonlinear in the literature, such as \"Optronic Convolutional Neural Networks of Multi-Layers with Different Functions Executed in Optics for Image Classification\" in Optics Express. In both our simulations and optical experiments, we incorporated this nonlinear operation. However, due to noise, this nonlinearity did not improve experimental accuracy. Nonetheless, our method can also ensure consistency between simulation and experimental results, ultimately enhancing accuracy.\n\n4. Broader Impact: The author should emphasize the benefits of optical neural networks (ONNs) for a broader audience, such as comparing power consumption more comprehensively. Previous Work: Some important previous work, such as from the \"meta surface ONN\" field, might be missing. Notable references include or discussed.\n\nThank you for your suggestion. The advantages of optical neural networks regarding energy consumption and processing speed are mentioned in the Introduction and discussed in the referenced articles. We will add further discussion to the manuscript to comprehensively cover these advantages. \n“ONNs offer significant advantages over traditional electronic neural networks, including high-speed computations and low latency due to massive parallel processing with light. This efficiency reduces power consumption, handles complex tasks effectively, and is scalable for compact systems. Additionally, light transmission minimizes electrical interference and heat dissipation, enhancing performance and reliability.”\n\n5. Limitations:\nThe experiments were conducted only on the MNIST level dataset. The 4f system might not perform as well on more challenging datasets, such as CIFAR or ImageNet. While I don't think the author needs to conduct these additional experiments, some discussions would be great.\n\nThank you for your concern. This issue is mentioned in our reply to Question 1, Our work can also handle more challenging datasets as well, such as the CIFAR-100 dataset, which indicates that the method proposed in the paper exhibits a degree of generalization capability.\n\nFinally, we would like to express our gratitude for your valuable feedback. In our future work, we aim to contribute to the practical implementation of metasurfaces for optical neural networks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1uJYiKVNYW", "reviewer": "Reviewer_fa6W", "summary": "The authors present a novel approach on using optical neural networks by integrating physical constrains into the training process. This work is motivated by the issues that ONN models present, the difference between the ideal, simulated system and the real, physical implementation — which significantly degrades the model performance. \n\nThe authors present a physics-constrained ONN that models and compensates for quantifiable physical errors (e.g., laser instability and exposure times mismatches in the camera). The authors conduct several classifications experiments on the framework, using MNIST, QuickDraw16 and Fashion MNIST; the results demonstrate the framework effectiveness with significant enhancement on the classification accuracy (e.g., from 61.6%, not using these corrections, to 93.5% when enabling the error compensation.", "questions": "1. the integration of physical constraints might introduce additional complexity in the training — which could potentially limit its use in more resource-constrained situations. How to the authors address that?\n\n2. How does the model adapt to previously unseen physical environments? I am thinking, for example, as an implementation in autonomous vehicles or robotic systems.", "limitations": "The authors briefly mention the limitations of this study, focusing specifically on the demonstration only for classification tasks. However, questions such as the application of this method on dynamic, unpredictable environments, remains unclear.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors present a learning method for ONNs which integrates quantifiable physical errors into the training process of the ONN. By doing this, the authors enhance the model capabilities in image classification, demonstrating high accuracy. This approach builds upon existing work in the field but delivers significantly advances on the classification accuracies for different datasets. This work improves the ONNs’ state of the art, enabling a more precise and robust optical computing applications. \n\nThe paper is well written and structured, and the level of detail is adequate for a broader audience.", "weaknesses": "1. The generalization of the method to broader applications, not only image classification, and its scalability remains unclear. \n\n\n2. The performance depends on the accuracy of the physical models used to represent the system. This could limit the applicability of the framework in environments/situations where the physical modeling is challenging or impractical. \n\n\n3. The authors discuss measurable physical errors — however, less insight is provided into the impact of unpredictable errors, which could be of significant importance when applying the framework to dynamic environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and valuable suggestions for improving the quality of our manuscript. As you rightly pointed out, our work is indeed motivated by the challenges presented by ONN models, specifically the differences between the ideal, simulated systems and their real, physical implementations. Furthermore, as you noted, our ultimate goal is to achieve more precise and robust optical computing applications. We have carefully considered your comments, and below are our responses to your feedback.\n1：The generalization of the method to broader applications, not only image classification, and its scalability remains unclear.\n\nThanks for your insightful question. As we mentioned in our conclusion. Although our loss design is tailored for classification tasks, the focus of our work is on compensating for various known and unknown errors in the physical system to achieve consistency between experimental and simulation results. Therefore, the framework of our method can still be applied to other tasks. We will explore the scalability of our approach to different tasks in future research.\n\n2：The performance depends on the accuracy of the physical models used to represent the system. This could limit the applicability of the framework in environments/situations where the physical modeling is challenging or impractical.\n\nYour question is very practical. Our work has been validated for accuracy in the relatively simple 4f physical system. To achieve consistent results with more complex and challenging physical models, we would need to optimize and adjust the DNN architecture to adapt to these more complex systems, as well as integrate more quantitative physical information extracted from the physical system into the network training.\n\n3：The authors discuss measurable physical errors — however, less insight is provided into the impact of unpredictable errors, which could be of significant importance when applying the framework to dynamic environments.\n\nThank you for your valuable feedback. In our next steps, we will consider testing in dynamic environments.\nIn addition to static and fixed unpredictable errors in experiments, some other errors can vary with changes in system parameters and often interact with each other. Modeling and analyzing their formation process is quite challenging. However, there are certain characteristic disturbances these errors cause in the experimental output images. Therefore, we chose to use DNNs to learn the impact of unpredictable errors by comparing ideal simulation images with actual experimental outputs, ultimately reducing their effect on experimental accuracy. For dynamic environments, we may need to further explore new DNN architectures and training methods based on the findings of this work.\n\n4：the integration of physical constraints might introduce additional complexity in the training — which could potentially limit its use in more resource-constrained situations. How to the authors address that?\n\nThe method proposed does add training burdens. However, it provides computational efficiency advantages compared to other error correction methods (as detailed in the \"Convergence Rate and Model Architecture\" section of the supplementary materials. By incorporating quantitative physical information, our DNN remains simple with fewer parameters, enhancing efficiency. Our focus is on model deployment and application; once trained, the model improves experimental results without altering the physical system. The inference process does not require additional computational resources due to the physical constraints introduced during training.\n\n5：How does the model adapt to previously unseen physical environments? I am thinking, for example, as an implementation in autonomous vehicles or robotic systems.\n\nYou’ve raised an excellent point that aligns with our ultimate goal. In automotive and robotics applications, issues such as slight changes in the relative positions of optical components, vibrations, and significant variations in laser intensity may arise. Since our current optical experiments are conducted in a controlled laboratory environment, further research is needed to address unpredictable errors in dynamic systems. We greatly appreciate your practical insight, as it will guide our future research and provide direction for the practical implementation of ONNs. We will continue to work towards this goal in future research, aiming to enhance ONN performance in previously unseen physical environments.\n\n6. Limitations: The authors briefly mention the limitations of this study, focusing specifically on the demonstration only for classification tasks. However, questions such as the application of this method on dynamic, unpredictable environments, remains unclear.\n\nWe will add the following paragraph to the paper to discuss the limitations mentioned by the reviewer:\n“Our method was developed and tested in a laboratory environment, where most errors are static and some are predictable. By incorporating predictable errors as physical priors into network training along with a relatively lightweight DNN architecture, we achieved consistency between simulation and experimental results. However, in dynamic and unpredictable environments, our proposed method would require more complex DNN architectures and new training approaches. We will work towards this direction in future research to develop ONNs that can operate stably and accurately under dynamic and unpredictable conditions.”\n\nWe sincerely appreciate the valuable feedback and insightful comments you have provided. Your suggestions have been instrumental in refining our work and enhancing the quality of our manuscript. Thank you for your thoughtful review and for contributing to the improvement of our research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their thoughtful comments. We appreciate their findings that our work focuses on narrowing the gap between simulated and physical optical neural networks(8n42) by addressing the inevitable errors encountered during the physical realization of these networks(W9o4). Consequently, our approach has significantly enhanced the performance of experimental optical neural networks(W9o4，8n42，fa6W), achieving state-of-the-art classification accuracy across MNIST dataset(W9o4). This advancement not only demonstrates the effectiveness of our method but also provides a pathway to more accurate and reliable optical computing applications(8n42，fa6W). Large-scale physical ablation experiments also demonstrate the efficacy of our proposed method in compensating for system errors and enhancing the consistency between experimental and simulated results.(M382,8n42). We will incorporate all suggestions. We sincerely hope this will help reviewers finalize their evaluations.

M382,W9o4. Contributions of our paper.Our work is the first work to conduct large-scale physical experimental analysis of light source instability and exposure time mismatches in optical neural networks (ONNs). These issues are common in optical setups and significantly impact the final experimental accuracy. Moreover, the core contribution of this paper is not merely the development of a high-performing optical neural network, but rather the resolution of the widespread issue of inconsistency between simulation systems and actual optical neural networks. We conducted extensive ablation experiments in a 4f system with numerous sources of error, which demonstrate both the efficiency and accuracy of our approach.

M382,W9o4,8n42. Datasets used in our paper. Our work can indeed handle more challenging datasets as well, such as the CIFAR-100 dataset. To do this, we simply replace the MNIST or Fashion-MNIST dataset loaded onto the DMD with CIFAR-100 images. Due to the lack of comparative results with other studies, we did not include this result in the main text. Additionally, for current ONNs based on 2D spatial structured light, researchers（The aforementioned articles, along with the paper 'Hybrid Training of Optical Neural Networks' published in Optica） primarily use the datasets mentioned in our paper. Our experiments were conducted on three datasets with different characteristics, and ablation studies across these datasets demonstrate that our method consistently aligns simulation and experimental results and significantly enhances experimental accuracy for datasets with varying features.

M382, W9o4. The benefits of optical neural networks. Optical neural networks (ONNs) offer several compelling advantages over traditional electronic neural networks. They can perform massive parallel processing with light, leading to extremely high-speed computations and low latency compared to analog AI hardware or digital accelerators.

fa6W,M382. The training efficiency of our approach. Unlike other methods, we are the first to incorporate quantitative physical information into network training. By integrating this quantitative physical data into the model training, we are able to reduce model complexity and enhance convergence efficiency. Hence, compared to the 360,000 parameters of the ONN, the DNN's approximately 5,000 trainable parameters (as detailed in the supplementary materials under Model Architecture) have minimal impact on training efficiency while significantly improving the consistency between experimental and simulation results. Moreover, the addition of DNN only affects the complexity and computational burden of the training process, without impacting the inference process of the physical ONN. In practical engineering applications, we are more concerned with the inference process rather than the training process.

8n42,fa6W. The performance of our work in practical environments. As shown in the Experimental Setup section of the supplementary materials, the actual optical layout of the 4f system indeed presents challenges such as large volume, numerous optical components, and difficulty in optical alignment. However, our proposed method addresses common errors in optical systems and ensures consistency between experimental and simulation results, significantly improving the final experimental accuracy even in error-prone 4f systems. Therefore, I believe that our method can be effectively applied to single-element systems, such as meta surface-based ONNs, and perform well under real-world conditions. However, in dynamic and unpredictable environments, our proposed method would require more complex DNN architectures and new training approaches. We will work towards this direction in future research to develop ONNs that can operate stably and accurately under dynamic and unpredictable conditions.

M382, W9o4, fa6W. Some other questions. (i) Table 1 in the main text presents a comparative analysis of our classification performance against the hybrid training methods, which highlights our advantages in both accuracy. (ii) The bulkiness of 4f’s structure indeed makes it more challenging to address system errors. Given that the types of errors analyzed in this work are common to optical systems, we believe our method can be effectively applied to single-element optical systems as well, where misalignment errors are easier to handle. And we also agree that the meta surface approach is a better choice for realizing spatially structured light computing due to its significant potential for industrial applications and its role in miniaturizing and lightening ONNs. We have another team that has just started working on meta surface-based ONNs. (iii) We will include a section in the paper discussing the impact of unpredictable errors on experimental results. (iv) We will polish the writing in the revised manuscript.

Meta-review: All reviewers agree on the originality of the problem addressed in the paper and the quality of the experimental validations. 

Reviewers M382, W9o4, fa6W ask for more challenging data experiments and authors provide further results for CIFAR10 and show that their technique can improve in that scenario as well.

Reviewer M382 is not satisfied with the novelty of the proposed DNN-based error calibration method and considers this to be a straightforward engineering solution. Authors reply to the reviewer by showing how their mathematical modeling of noise in the training objective can boost the performance on top of the DNN-based approach.
Decision: Accept (poster)