Neural Tangent Knowledge Distillation for Optical Convolutional Networks
Authors: Jinlin Xiang
Minho Choi
Yubo Zhang
Zhihao Zhou
Arka Majumdar
Eli Shlizerman
Venue: NeurIPS 2025 poster ()
Keywords: Optical Neural Networks
Knowledge Distillation
Primary area: applications
TL;DR: A task-agnostic and hardware-agnostic pipeline for Optical Neural Networks based on Neural Tangent Kernel Distillation.

Abstract: Hybrid Optical Neural Networks (ONNs, typically consisting of an optical frontend and a digital backend) offer an energy-efficient alternative to fully digital deep networks for real-time, power-constrained systems. However, their adoption is limited by two main challenges: the accuracy gap compared to large-scale networks during training, and discrepancies between simulated and fabricated systems that further degrade accuracy. While previous work has proposed end-to-end optimizations for specific datasets (e.g., MNIST) and optical systems, these approaches typically lack generalization across tasks and hardware designs. To address these limitations, we propose a task-agnostic and hardware-agnostic pipeline that supports image classification and segmentation across diverse optical systems. To assist optical system design before training, we design the metasurface layout based on fabrication constraints. For training, we introduce Neural Tangent Knowledge Distillation (NTKD), which aligns optical models with electronic teacher networks, thereby narrowing the accuracy gap. After fabrication, NTKD also guides fine-tuning of the digital backend to compensate for implementation errors. Experiments on multiple datasets (e.g., MNIST, CIFAR, Carvana Masking) and hardware configurations show that our pipeline consistently improves ONN performance and enables practical deployment in both pre-fabrication simulations and physical implementations.

Neural Tangent Knowledge Distillation for Optical Convolutional Networks
Jinlin Xiang*
Minho Choi
Yubo Zhang!
Zhihao Zhou
Arka Majumdar"
Eli Shlizerman' Ttll
Abstract
Hybrid Optical Neural Networks (ONNs, typically consisting of an optical frontend and a digital backend) offer energy-efficiee alternative to fully digital deep net- works for real-time, power-constrained systems. However, their adoption is limited by two main challenges: the accuracy gap compared to large-scale networks during training, and discrepancies between simulated and fabricated systems that further degrade accuracy. While previous work has proposed end-to-end optimizations for specific datasets (e.g., MNIST) and optical systems, these approaches typically lack generalization across tasks and hardware designs. To address these limitations, we propose a task-agnostic and aannosie-agnostic pipeline that supports image classification and segmentation across diverse optical systems. To assist optical system design before training, we design the metasurface layout based on fabrica- tion constraints. For training, we introduce Neural Tangent Knowledge Distillation (NTKD), which aligns optical models with electronic teacher networks, thereby narrowing the accuracy gap. After fabrication, NTKD also guides fine-tuning of the digital backend to compensate for implementation errors. Experiments on multiple datasets (e.g., MNIST, CIFAR, Carvana Image Masking Dataset) and hardware configurations show that our pipeline consistently improves ONN performance and enables practical deployment in both pre-fabrication simulations and physical implementations.
1 Introduction
Optical Neural Networks (ONNs) offer promising approach to achieve efficient computation and energy use compared to digital implementations such as Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), making them well-suited for cnouurc-constrrinele real-time physical systems (T. For example, ONNs have been proposed for power-limited applications (illustrated in Figure a), including satellites (2). unmanned aerial vehicles (3), smart home devices autonomous
driving systems (5), wearable electronics [6), and medical devices (7).
Among different ONN implementations, hybrid optical-electronic architectures are practical options under current hardware constraints (8). In such systems, the optical frontend accelcrates computation at the speed of light, while the digital backend refines predictions to improve robustness (9]. The
These authors contributed equally to this work.
DDepartment of Applied Mathematics, University of Washington, Seattle, WA 98195, USA. 2 DDepartment of Electrical and Computer Engineering, University of Washington, Seattle, WA 98195, USA. Depaatmeen of Physics, University of Washington, Seattle, WA 98195, USA. 'Department of Electrical Engineering, Ulsan National Institute of Science and Technology. Ulsan 44919.
South Korea
Corresponding authors:
@hlizee@uw.edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

ONN
Training:
Optical Frontend
8
2) ONN System Design & Knowledge Transfer
Camera
Backend
1) User Inputs
3) Optical Fabrication
4) Error Compensation
Inference:
Training
Frozen
Real-world Object
Optical Frontend
Backend
a) ONN Applications
b) ONN Deployment Pipeline
Overview of potential applications for ONNs and our proposed deployment pipeline. (a) ONNs for real-time decision-making in power-constrained scenarios. (b) Our proposed pipeline includes user-driven design, knowledge transfer training, fabrication, and error compensation.
Figure I:
optical frontend generally consists of a single linear layer (as shown Figure Ta), since: (1) im- plementing nonlinear activation functions in physical optics remains extremely challenging due to material and device limitations; and (2) without nonlincarity, multiple linear transformations can be mathematically compressed into single linear transformation. Moreover, from a theoretical stand- point, the universal approximation property could still be satisfied by shallow networks, suggesting that hybrid ONNs retain sufficient expressive power for a wide range of tasks when appropriately optimized 12]
Despite their promises, ONNs remain difficult to design and train due to both architectural limita- tions and fabrication-related challenges. First, existing ONN architectures are typically significantly simpler than modern deep CNNs or ViTs. These simplifications cannot be directly obtained through pruning or quantization [3 Second, physical fabrication and experimental deployment in- evitably introduce various sources of noise, such as optical misalignment, material variability, and measurement noise, further degrading performance [16]. While some end-to-end optimization strategies have been proposed to address these challenges, they typically designed for a specific dataset (e.g., MNIST) and tailored to a particular optical system, rather than providing a generalized solution (as also summarized in Related Works). In contrast, we aim to develop task-agnostic and agnnssce-agnostic pipeline that can generalize across different datasets and optical hardware setups.
To address these challenges, knowledge transfer, particularly Knowledge Distillation (KD), offers promising solution by transferrigg knowledge from pre-trained digital networks to optical models [17] 18]. Moreover, recent work shows that successful KD implicitly leads to student-teacher Neural Tangent Kernel (NTK) similarity, where NTK captures how the network's predictions change with respect to small changes in its parameters [19]. As we show here, utilizing NTK for matching is particularly effective for ONNs, as the NTK provides a linear approximation of network behavior, naturally aligning with the linear operations performed by optical systems.
Thus, we propose Neural Tangent Knowledge Distillation (NTKD) pipeline that generalizes across different optical network designs and datasets to support multiple tasks such as classification and segmentation (also shown in Figure (b). The pipeline starts with specifying the task, the dataset, and the optical structure. Then, NTKD optimization transfers knowledge from digital teacher models to hybrid ONNs by matching their NTKs, effectively transferring the relational structure between classes rather than just matching final predictions. Furthermore, the pipeline compensates for errors introduced during fabrication and experimental deployment by aligning the student's 's and teacher's NTKs through small fraction (e.g., 10%) of real experimental data.
In summary, our contributions are as follows:
We introduce a Neural Tangent Knowledge Distillation (NTKD) pipeline that supports diverse tasks and optical structures, addressing the challenges of shallow architectures and physical imperfections.
We experimentally validate our pipeline with different ONN implementations on both classification and segmentation tasks, demonstrating its effecctivnnes through both simulations and fabrications.

---

Table I: Summary of previous ONN works categorized by task type (classification, segmentation) and implementation level-either simulation only (denoted Sim) or with physical fabrication and experimental validation (denoted as Fab).
Classification (Sim)
Classitication (Fab)
Segmentation (Sim)
Segmentation (Fab)
ONN Capability
Works
Monochromatic
2018-2025:
16 20
30
X
X
2023-2025: 16
28,
29
39
V
X
X
Polychromatic
EXtr (2025)
X
X
Ours (NTKD)
We leverage NTK analysis to estimate the achievable accuracy of given hybrid ONNs, providing theoretical guidance on their design and optimization.
Related Works
ONN Tasks and Implementations: Table categorizes ONN applications into two tasks (classi- fication and segmentation) and two optical implementations (monochromatic and polychromatic systems). Most previous work focused monochromatic ONNs for MNIST image classification, including fully optical systems that performed linear transformations (20 21), physically nonlinear
ONNs that used atomic vapors or intensifiers (4), and hybrid architectures that combined an optical frontend with a digital backend (75 J. Previous polychromatic ONNs for classification were limited small datasets such as CIFAR- 10, as ONN architectures faced challenges in scaling to complex benchmarks [28|29) 29]. Segmentation tasks are still in the early stages, with previous study based only on simulation S]. Our work considers both classification and segmentation tasks. In our pipeline, image reconstruction is implicitly incorporated by encouraging the optical frontend output align with the simulated result, as the physical output deviates from simulation and requires correction.
Transfer learning for ONNs: Transfer learning, particularly Knowledge Distillation (KD), offers promising solution for transferring knowledge from pre-trained digital networks to optical models [17] 4T). KD minimizes the divergence between compact student model's predictions and those of a pre-trained teacher model, thereby encouraging the student to actively mimic the teacher's s behavior [17] Beyond conventional KD, recent studies have shown NTK-based approaches to understand knowledge transfer (43). For example, theoretical insights into KD transfer risk and data efficiency in wide networks have been established through NTK analysis [44]. Subsequent work demonstrated that successful knowledge distillation implicitly led to student- NTK aligmment [19], and NTK similarity was further applied to quantify task affinities in multi-task learning (45 In contrast, our work targets physically constrained ONNs, and introduces an explicit NTK matching strategy to directly guide the distillation process from digital teacher to an optical student.
Compensation Strategies for Practical ONNs: Fabrication imperfections and system noise in phys- ical ONNs often lead to significant performance drops compared to simulations. Some approaches used deep learning to model the system directly in a data-driven manner (48), including ONN auto- learning 49 50). where ONNs were trained to fit experimental input-output mappings. Other methods introduced physical information via hardware-in -eeee-in-tt-oooot training. For example, physics- conssiss-oostrainedd frameworks embedded fabrication aware models and losses to better align learning with optical behavior [16]. Moreover, some approaches avoided simulation entirely by randomly fabricating optical kernels and training digital backend to adapt to the fixed frontend structure (51555). This simplified fabrication but placed the learning burden entirely on the backend. is also not clear such random surfaces perform better than an ordinary lens. In contrast, our work identifies sources of physical errors and introduces an NTK alignment strategy for effective compensation.
Methods
3.1
Optical Frontend Design
At the initialization of the pipeline, user inputs are required to define the optical system (also shown in Figure 2 l). Specifically, the user specifies (1) the physical size of the optical frontend (c.g., the
3

---

8 Mota-optics Dimension
Training Frozen
User
Task
Train Batch
Networks
10% Train Batch
Teacher Networks
(1) NTK
Match
NTK
System
Optical Design NTK Estimation
ONN (Simulation)
Updates
ONN (abrication)
Noisy Results
Backend
(1) Optical Frontend Design
(2) Knowledge Transfer Training
(3) Error Compensation
Figure 2: Overview of the pipeline. It consists of three steps: (1) Optical Frontend Design based on user-specified inputs, (2) Knowledge Transfer Training using Neural Tangent Kernel (NTK) matching, and (3) Error Compensation for fabricated optical frontends.
number of meta-optic kernels), (2) the target dataset for the task, and (3) the desired network structure, such as the number of layers and channels. Prior works have demonstrated that optical convolution can physically realized using either 4f system or a PSF-based free-space propagation system
[1]30] 30 38 56]. In this work, practically implement PSF-based metasurface design due to its advantages in compactness, alignment robustness, and ease of fabrication 30).
We consider metasurface of size (h, w), onto which we aim to place Nieenels square optical kernels, each of size (in mm), with a minimum edge-to-edge spacing d to satisfy fabrication constraints. To compute the maximum number of kernels that can be placed while preserving symmetry, we define
Optical Frontend Layout:
W d
h d
Coolss
k d
Trows
k: +
Piernels = Noools X Thows
Performance Estimation: Once the physical layout of the ONN is determined, we aim to estimate its expected performance without empirical training. We adopt the Neural Tangent Kernel (NTK) framework, which captures the training dynamics of infinitely wide neural networks under gradient descent. In particular, we introduce reference network that shares the same architecture as the designed ONN (e.g., number of layers and connectivity) but has infinite width at each layer. Under this assumption, the predictions of the reference network correspond NTK regression (8). Let the reference network if [a; 0) be a neural network parameterized by 0, which maps an input æ to an output fz; 8). The NTK is defined
O x')
(2)
where Vof 0) is the Jacobian of the network output with respect to its parameters. train y. trin Maiin be the training data and {a!( (es T==l
Let
the be the test data. We compute
liel
train.train
(1
tmain
train
E
X Ximiie
Oretttrann
(e mmin) E X
(3)
and use kernel regression to predict outputs on the test set
lest 0) rest,train (Orain.rrain + A1) y
trin
where À is regularization parameter, which is selected via grid search on validation set.
The NTK- based performance estimation serves as diagnostic tool to evaluate whether the specified ONN architecture is expressive enough for the given task. While this estimation is not used for training or loss computation, provides an early signal to guide architectural decisions and allows users to iteratively refine the optical design before full training and fabrication. For example, if the estimated test aceuracy is much lower than the expected performance, may suggest a mismatch between the ONN's capacity (c.g., depth) and task complexity.

---

3.2 Knowledge Transfer Training
After the user specifies the ONN architecture, we train the system for the target task. We define supervised learning problem with input-output pairs (z.y), where represents the input samples and y denotes the corresponding ground -ruthttrthhhhh labels. The network parameters (0), including the optical frontend and the digital backend, are initialized and optimized jointly (shown in Figure 2)..
End-to-end loss: The first loss term consider is standard end-to-end supervision loss, which directly minimizes the discrepancy between the network's predictions and the ground-truth labels. Formally, optimize the following objective
LEEF B(x [e(Sow(<;; 0),
(5)
where -(:-)
-) standard loss function such as cross-cntropy for classification tasks, foNw (z; 0) denotes the output of the network with parameters 0, and D represents the training dataset.
Neural Tangent Knowledge Distillation (NTKD) Loss: addition to the standard end-to-end loss, we introduce a knowledge transfer loss based on Neural Tangent Kernel (NTK), Specifically, we assume access pretrained teacher network, such as LeNet for MNIST, AlexNet for CIFAR- 10, or U-Net for image segmentation tasks.
Given minibatch of input samples {ara (=1 Vitua we compute the Jacobian matrices of both the teacher network (J (eacher) and student ONN (JONN) with respect to their parameters (Qeacher BNNN
O fvecher (2i)
a foNN (x:)
Juescher
JONN =
(6)
Ueaccher
δBONN
i=1
1=1
The Jacobians Juacher E Tiuca x Phesber X Pidass and JONN E R. x X Kiaass may differ in width depend- ing the number of trainable parameters in each network. Here, Preacher and PONN denote the number of parameters in the teacher and ONN, respectively. Their corresponding NTK matrices,
Oreacher =
Jeeecher leacherr
OONN = JONN-
are both of size Thatch Yibarch- We define the NTKD loss by minimizing the discrepancy between the NTK matrices of the teacher network and the ONN (e.g.. MSE)
LNTKD
E (xx))"""h>>> [ (Oreacher OONN)]
(8)
Then, minimize weighted sum of two losses, controlled by hyperparameters and
min (aLEZE + +LLTKDD)
(9)
#
3.3 Error Compensation
The physical fabrication uses the optimized simulation parameters obtained through the process described in Section 3.2 The fabrication fixes the optical frontend, and only the digital backend remains tunable. Due unavoidable fabrication and experimental errors, discrepancies arise between the designed and realized optical system. Given input image and convolution kernel kt the ideal output y = a The fabricated optical convolution output with noise at location (i,
Ruse
Ui.j = 033
Oi 0i+m-1jtnn- (kmm,n + 6m,,) + Gij-
(10)
TIE m
Here, the scaling factors (image brightness) and 3 (image- kernel misalignment) can be experi- mentally calibrated to match the designed system, while the sensor noise is primarily determined by the imaging device characteristics. We further quantify the impact of fabrication noise ō (with proof provided in the Supplementary). In particular, we show that perturbations in the NTK caused by kernel fabrication entors (6.z) scale
A ~ o |
(11)
where A@ONN denotes the NTK perturbation, and m is the number of kernels (i.e., the network width). This result indicates that networks with more kernels inherently more robust to fabrication
5

---

Side View
Side View
UpCorm
525rom
450mm 532 650mom
FineMet
(a) Compressed Meta
(b) Polychromatic Meta
(c) Extreme Meta
(d) Polychromatic Meta (Segmentation)
OLED display
Color camera
Metasurface array
Figure 3: Optical systems. (a) Compressed Meta ONN on MNIST [37]: (b) Polychromatic Meta
ONN on CIFAR-10 (29): (c) EXtremeMETA for segmentation with dual optical frontends (5); (d)
Customized Polychromatic Meta ONN for segmentation (Ours); (e) Optical measurement setup: (f)
Fabricated PSF-engineered meta-optics; (g) Scanning electron microscopy image of the meta-optics.
Table 2: Teacher and student network details.
Student Architecture
Teacher Model
Teacher Accuracy
Teacher Size
Compressed Meta (0.05M)
LeNet
99.1%
~0.18M
Polychromatic Meta (1.62M)
AlexNet
85.4%
233.29M
Polychromatic Meta (1.06M)
U-Net
95.4%
196.97M
noise. Indeed, if m > |0]], the impact of fabrication noise on the prediction can be considered negligible. If 8 is interpreted m - 0o, the NTK (9) remains constant during training. implying AO - (57].
gradient descent step, Eq. 11 consistent with NTK theory: as
To correct these errors, we re-apply minimization in Eq. 9.. but restrict optimization to the unfrozen backend parameters (shown in Figure 33.. The teacher network takes the raw input images and computes its NTK matrix over a batch of samples, as defined in Eq. 7 The student network receives feature maps from the fixed optical frontend, which processes the same batch of input images and produces an NTK matrix of size Thatch Thatth.
Results
4.1 Implementation Details, Pre-trained Teachers, Datasets and Evaluation Metrics
Implementation Details: Figure demonstrates four different optical systems used in the exper- iments. For monochromatic image classification, we conducted experiments on the Compressed Meta ONN architecture (37) using the MNIST dataset [59]. This system consists of single optical frontend with 8 kernels (7 x 7) and a compact digital backend composed of two fully connected layers. For polychromatic image classification,
evaluated the Polychromatic Meta ONN (2 on the CIFAR-15 dataset [60]. This model consists of a single optical frontend with 16 kernels (7 x 7) and digital backend consisting of three fully connected layers.
For image segmentation, we performed experiments on both the Extreme Meta ONN (5) and modified version of the Polychromatic Meta ONN [29], using Kaggle's Carvana Image Masking dataset. EXtremeMETA consists of two parallel polychromatic optical frontends, followed by a dual-path digital backend composed of CoarseNet for global feature extraction and FineNet for detail enhancement, with their outputs fused to produce the final segmentation. Our customized polychromatic segmentation system extends the Polychromatic Meta ONN by incorporating single optical frontend with 56 kernels (8, 16, and 32 kernels to capture hierarchical depth representations, each of size x 3) and backend composed of upconvolutional layers to support dense prediction tasks. For a fair comparison between the two systems, we matched the number of optical kernels. Optical implementation details are in the Supplementary.
In experiments, we manipulate polychromatic- red, green, and blue-point point spread functions (PSFs) of the meta-optics using gradient descent algorithm. Physical shapes and dimensions of the PSF-

---

(a) MINIST, No Transfer
(b) MNIST, NTKD
(c) MNIST, No Transfer
(d) MNIST, NTKD
10-
50
-20
00 63
35
35 04
SNV Clm
CAT1
(e) CIFAR-10. 10, No Transfer
(1) CIFAR-10, 10, NTKD
(s) CIFAR-10, No Transfer
(h) CIFAR-10, NTKD
25-3333 25
LOME 3mm
23
-20
-30
-20
-10
Dim
Figure 4: Simulation: t-SNE and confusion matrices in MNIST (a-d) and CIFAR-I0 (e-h).
engineered meta-optics are shown in Figure Since the meta-optics designed with PSFs that function as convolutional kernels, optical convolution occurs naturally during image capture. Our experimental setup is straightforward: we replace conventional imaging lens with PSF-engineered meta-optics. The meta-optics are carefully positioned and aligned with the color camera, enabling the capture of PSFs and convolved images using laser pointer and OLED display, respectively. A schematic representation and a photograph of the setup are provided in Figure 3
Computing the NTK explicitly via Jacobian-Jacobics products is memory memory-intensive intensive and infeasible at scale. To address this, we adopt the approximation strategy of NAP-SAP (61-63), which estimates the NTK trace rather than constructing the full matrix. We use batch size of 128 for MNIST and CIFAR-10/100, batch size of 64 for ImageNet- 100, and batch size for segmentation tasks.
The MNIST and CIFAR-10 datasets each consist of 50, 000 training images and 10, 000 testing images. The Carvana dataset, originally introduced in Kaggle's Carvana Image Masking Challenge, contains 5,088 high-resolution 1920 x 1280 car images. We adopt LeNet (99.1 % accuracy on MNIST), AlexNet (84.5% on CIFAR-10) and a full U-Net (95.4% mloU on Segmentation) as teacher models for their respective tasks. Table summarizes the teacher and student networks.
Pre-trained teachers and Datasets:
For classification tasks, we ran each experiment five times with different random seeds and reported the mean and standard deviation (std) of the classification accuracy. For segmentation tasks, we similarly conducted five independent runs and reported the mean Intersection over Union (mloU) along with the standard deviation.
Evaluation Metrics:
4.2 Main Results
Simulation Results: Table 3 summarizes the accuracy of different ONN training strategies in classification and segmentation tasks. For monochromatic classification, NTKD achieved 97.3% accuracy and outperformed both KD-based transfer (95.9%) and the non-transfer baseline (91.4%). Similar trends were observed in the more challenging polychromatic classification setting, where NTKD achieved 75.6%, surpassing KD (72.5%) and baseline (56.4%). For segmentation tasks, we evaluated models on both the Extreme Meta and our proposed Polychromatic Meta datasets. NTKD
Table 3: Performance comparison of ONN methods across different tasks and training strategies.
Methods
Monochromatic Classiication (%)
Polychromatic Classification (S)
Polychromatic Segmentation (mloUU
Compressed Meu (2025) 377
Polychromaiic Meta (2025)
Funene Meta (2025) 5] Polychromatic Misa (Qurs)
Simulation, No Transfer Sinulation, KD)
91.4 :: 0.8%
56.4 +: 1.9%
68.3 =: 0.5%
74.3 2 0.4%
95.9 + 0.6%
72,5 + 2.1%
75.3 + 0.2%
86.7 + 0.4%
Sinulation, NTKD
0.2%
0.4%
7

---

(a) MINIST, No compensation
(b) MNIST, NTKD
(c) MNIST, No compensation
(d) MNIST, NTKD
Glass
to
N-12
Lm
-30
-10-
-20-
-20
30
-20
-10
1:NE Dim
1-SNE
3km
(e) CIFAR-10, No compensation (f) CIFAR-10, NTKD
(s)
CIFAR-10, No compensatoon (h) CIFAR-10, NTKD
Claps
2-
25
54
23-23333
-30-
-10-
6-33
6-21
-20
-20
-20
-10
10
20
-20
-10
10
L-SNE
Jlm
Figure 5: Fabrication: t-SNE and confusion matrices on MNIST (a-d) and CIFAR-10 (c-h).
consistently achieved higher mIoU scores across both optical systems, surpassing KD-based transfer and end-to-end training without transfer (75.3% and 86.7%, respectively).
These results indicate that training ONNs end-to-end without transfer often leads suboptimal performance. Incorporating knowledge transfer through KD or NTKD improves learning efficacy and overall segmentation quality. In particular, the NTKD approach outperformed KD in different tasks, demonstrating its ability to guide representation learning in optical neural networks.
Figure M demonstrates knowledge transfer strategies on MNIST and CIFAR-10 representations and classification performance. Compared to the no-transfer baseline, these strategies improve class separability in t-distributed Stochastic Neighbor Embedding (t-SNE) visualizations and reduce noise in confusion matrices. NTKD transfer shows improved clustering and accuracy in experiments.
Fabrication and Compensation Results: Table 4 summarizes the impact of error compensation strategies on ONN performance in classification and segmentation tasks. Due to unavoidable fabrica- tion and experimental errors, optical frontends suffer significant accuracy drops, especially in the polychromatic setting. Without compensation, the monochromatic system exhibits an 8. % drop, and the more fabrication-sensitive polychromatic system shows 28.3% drop on CIFAR-10 and a 41.8% reduction mloU on the Carvana segmentation task. This performance gap highlights the increased challenge of fabricating RGB-sensitive kernels in polychromatic ONNs compared to monochromatic ONNs. Our results demonstrate that knowledge transfer methods are able to assist with denoising that gap. NTKD compensation yields higher accuracy in both cases (95.1% for monochromatic, 74.9% for polychromatic and 8 81.2% for image segmentation task), outperforming end-to-end deep learning compensation, and validating its effectiveness in robust corrections for fabrication.
Figure 5 compares the t-SNE visualizations and confusion matrices of MNIST and CIFAR-10 representations under different compensation strategies. Without compensation (Figures 5 a, g). both optical systems exhibit class overlap in the feature space and reduced classification accuracy, primarily due to fabrication errors and optical misalignments. The NTKD correction (Figures 5 d, f, h) compensates for these errors and improves the clustering structure and classification accuracy, demonstrating robustness and generalization across datasets and optical systems.
Table 4: Evaluation of ONN error compensation methods on fabricated optical systems across both classification and segmentation tasks.
Method
Monochromatic Classification (6) Compressed (2025) 37
Polychromatic Clussificution (%) Polychromatic (2025)
Polychromatie Polychromatic Meta (Ouns)
(mloU)
No compensation (baseline)
89.2%
47.3%
49.7%
Brree compensation. (End-te-Ean (ndd-teEEn))
93.2 1 0.1%
70.4 + 2.1%
62.7 4: 0.9%
Erroe compensation (NTKD)
95.1 1: 0,1
71.9 1: 1.3%
81.2 0.6%
8

---

Table 5: Random PSF kernel design across different tasks.
Monochromatic Classification (%)
Polychromatic Classification (%)
Polychromatic Segmentation (mloU)
kernels
96.12%
36.73%
49.3%
500 kernels
96.81%
49.32%
64.1%
1000 kernels
97.24%
56.23%
69.9%
oc kemels
97.72%
67.33%
72.1%
4.3 Discussion and Ablation Study
The complexity of the backend plays a critical role in the performance of optical systems, particularly in hybrid optical-digital architectures. When strong digital backend is employed, it can effectively denoise and recover accurate outputs, even when the optical frontend introduces significant noise. However, a strong backend not only diminishes the contribution of the optical frontend but also significantly increases power the core motivation for adopting optical computing in resource- cesource-constrained environments. In such scenarios, digital networks (e.g., ViT or U-Net) are often more practical and effective choice than hybrid ONNs with disproportionately strong backends. For practical deployment, we need to carefully balance the computational load between optics and computational backend and find a tradeoff between acceptable energy consumption/ latency and acceptable accuracy, which will be an application dependent trade-off.
Backend Complexity of ONNs:
Random vs. Designed Parameters: Another possible direction for designing and training ONNs is to use randomly initialized optical parameters while training only the digital backends. This approach aims to avoid the need for extensive simulation and frontends. We conducted experiments using a single optical convolutional layer and a lightweight backend-consse
optimization of the optical
of single fully connected layer for classification, or single upsampling layer for segmentation. As shown in Table increasing the number of random kernels consistently improves performance across tasks. For example, in polychromatic classification, accuracy improves from 36. 73% (8 kernels) to 56.23% (1000 kernels), and further to 67.33% in the NTK regime, which approximates an infinite number of random kernels. Similarly, in polychromatic segmentation, mloU rises from 49.3% to 69.9% and reaches 72.1% % under NTK estimation. These results demonstrate that while increasing the number of random PSFs improves performance, they still underperform our approach (designed kernels with knowledge transfer).
Scalability of ONNs: Scaling current ONNs remains challenging, as most designs rely on shallow structures with limited linear computational capacity. Implementing nonlinear operations in ONNs is especially difficult due to physical constraints, such as the limited pixel size. These hardware limitations make it hard for ONNs to support deep and expressive architectures like those used in digital networks. We observed that using different kernels to simulate multiple layers of digital network leads to better performance, compared to simply compressing a deep CNN into a single-layer ONN.
Table 6: Impact of Teacher Complexity on NTK Distillation Performance for Classification and Segmentation Tasks.
Dataset
Task Classification
Teacher
Student
Teacher Accuracy
Student Accuracy (with without NTKD)
ImageNet- 100
ResNet- 18
Polychramatic Meta
78.43%
46.32% / 33.45%
ImageNet- 100
Classification
ResNet-50
Polychromatic Meta
88. 32%
47. 86% / 33.45%
COCO-Stuff 10k
Segmentation
U-Net
Polychromatic Meta
61.89 898
41.43% / 35.03% 42.73% / 35.03%
COCO-Stuff 10k
Segmentation
ResU-Net
Polychromatic Meta
69.23%
Table 6 examines the impact of stronger teachers on the student ONN under optical physical limita- tions, with additional experiments conducted using ResNet variants and more complex datasets such as ImageNet-100 100 and COCO-Stulf 10k. For the classification task, we employed both ResNet- 18 and ResNet-50 to train a Polychromatic Meta student network. While both teacher models improved performance through NTK distillation, the gain from ResNet-18 to ResNet-50 was marginal with only an increase of 1.54%. Similarly, for segmentation. we used U-Net and ResU-Net as teacher networks to train a Polychromatic Meta-optical student on COCO-Stuff 10k for binary foreground- senground-bbckgrounng background segmentation. In this setting, the foreground includes all semantic object classes, and the background consists of non-object regions. Again, while both stronger teacher models provided improvements, the performance gain from more complex teacher was limited.

---

These results indicate that bottleneck in performance is primarily due to the physical modeling limitations of current ONN hardware. rather than the complexity of the teacher model. In the case of ONN technology being improved in the future, as one would expect, in such case, a stronger teacher network can provide additional gains performance through the NTKD approach that will transfer knowledge from stronger teachers to enhanced students. In summary, scalable and expressive ONNs will ultimately rely on physical advances enabling deep and nonlinear optical computations.
Fabrication Analysis: Several factors may help explain the discrepancy between the designed and measured kernels. First, the local periodic approximation, which simplified the metasurface optics design by assuming the scatterers were arranged periodically, neglected the coupling between adjacent dissimilar scatterers and could introduce phase errors. Second, unavoidable fabrication errors further contributed to the observed discrepancy. Third, the pre-designed kernel needed to be properly matched to the sensor's pixel array; any misalignment between the metasurface opties and the camera could also lead to deformation of the measured kernels. Regarding the polychromatic versus waaeeenggt kernels, metasurfaces inherently suffered from strong chromatic aberrations, universal characteristic of diffractive optics. While we co-optimized the kernel across multiple wavelengths during the design process to ensure consistent behavior, the performance remained limited by the intrinsic material properties.
operations (MACs) and power consumption of hybrid ONNs using our polychromatic ONN as an example (details in the Supplementary). The total number of MAC operations in the simulated digital network is approximately 239 MMACs (while the full U-Net requires 65.9 GMACs and Efficient U-Net reaches 1.37 GMACs), which is reduced to 65 MMACs after incorporating optical frontends (64). The total energy consumption includes both image capture and digital computation. The full U-Net (pre-trained teacher) consumes 2.03J for computation and 2.36 mJ for image capture, totaling 2.04 per image. The compact digital network requires 7.37 mJ for computation and 2.36 mJ for image capture, totaling 9.73 mJ per image. In contrast, our hybrid ONN consumes 3.82 mJ for image capture and 2.01 mJ for backend processing, totaling 5.83 rep-eeeersnnniig over a 40% reduction in system-level energy consumption compared to the simulated digital network, and over x00 energy compression compared to the pre-trained teacher U-Net.
MACs and Power Consumption: We estimated the
5 Conclusion
We propose a comprehensive NTKD pipeline that addresses multiple tasks and multiple optical sys- tems in ONN design, training, and compensation. By incorporating knowledge transfer, particularly NTK-based knowledge distillation, our framework consistently improves the accuracy of different optical systems across both classification and segmentation tasks. Based on extensive experiments, we observe that the current ONN performance is primarily limited by the shallow and linear nature of existing optical architectures. Future advances in deeper, nonlinear ONNs may help narrow the performance gap between optical and electronic neural networks, improving scalability and accuracy.
Acknowledgement
The research is supported by the National Science Foundation support from HDR Institute: Accelerated AI Algorithms for Data-Driven Discovery (A3D3) National Science Foundation grant PHY-2117997 (JX,ES). Part of this work was conducted at the Washington Nanofabrication Facility/ Molecular Analysis Facility, a National Nanotechnology Coordinated Infrastructure (NNCI) site at the University of Washington with partial support from the National Science Foundation via awards NNCI-1542101 and NNCI-2025489, The authors also acknowledge the partial support by the Departments of Electrical Computer Engineering, Applied Mathematics and Physics at the University of Washington. The authors are also thankful to the eScience Center at the University of Washington.
WI--BRAID-2223499 and partial
References
[1] Guibin Zhao, Pengfei Li, Zhibo Zhang, Fusen Guo, Xueting Huang, Wei Xu, Jinyin Wang, and Jianlong Chen. Towards sar automatic target recognition: Multi-category sar image classification
10

---

based on light weight vision transformer. In 2024 21st Anmual International Conference on Privacy, Security and Trust (PST), pages 1-6. IEEE, 2024.
[2] Tingzhao Fu, Jianfa Zhang. Run Sun, Yuyao Huang, Wei Xu, Sigang Yang. Zhihong Zhu, and Hongwei Chen. Optical neural networks: progress and challenges. Light: Science & Applications, 13(1):263, 2024.
[3] Yingzhang Wu, Wenbo Li, Jie Zhang, Banghei Tang, Jinlin Xiang, Shen Li, and Gang Guo. Driver's ' hand-foot coordination and global-regional brain functional connectivity under fatigue: Via graph theory and explainable artificial intelligence. IEEE Transactions on Intelligent Vehicles, 32)3339333508, 2023.
[4] Adith Boloor, Weikai Lin, Tianrui Ma, Yu Feng, Yuhao Zhu, and Xuan Zhang. Privateeye: In-sensor privacy preservation through optical feature separation. In 2025 IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pages 2357-2367. IEEE, 2025.
[5] Quan Liu, Brandon T Swartz, Ivan Kravchenko, Jason G Valentine, and Yuankai Huo. Ex- trememeta: High-speed lightweight image segmentation model by remodeling multi-channel metamaterial imagers. Journal of Imaging Science and Technology, pages 1-10, 2025.
[6] Baiheng Zhao, Junwei Cheng, Bo Wu, Dingshan Gao, Hailong Zhou, and Jianji Dong. Inte- grated photonic convolution acceleration core for wearable devices. Opto-Electio Electronic Science, 2(12):230017-1, 2023.
[7] Jinlin Xiang, Hillol Sarker, Bozhao Qi, Ruisu Zhang. Roger Trullo, Salvatore Badalamenti, Maria Wiekowski, Annie Kruger, Etienne Pochet, Qi Tang, et al. Endoscopic scoring and localization in unconstrained clinical trial videos. In 2025 IEEE/CVF Winter Conference on Applicaatoons of Computer Vision (WACV), pages 4015. IEEE, 2025.
[8] Runqin Xu, Pin Lv, Fanjiang Xu, and Yishi Shi. A survey of approaches for implementing optical neural networks. Optics & Laser Technology, 136: 136:106787,, 2021.
[9] Shane Colburn, Yi Chu, Eli Shilzerman, and Arka Majumdar. Optical frontend for a convolu- tional neural network. Applied optics, 11912) 2019.
[10] Zhiyuan Lu, Huan Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: view from the width. In NeurIPS, 2017.
[1]] Ding-Xuan Zhou, Universality of deep convolutional neural networks. Applied and Computa- tional Harmonic Analysis, 48(2):787-794, 2020.
[12] Changshuo Bao, Qianxiao Li, and Haizhao Yang. Approximation analysis of convolutional neural networks from a feature extraction view. Applied and Computational Harmonic Analysis, 54:47-84, 2021.
[13] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[14] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 2016.
[15] Mariia Seleznova, Dana Weitzner, Raja Giryes, Gitta Kutyniok, and Hung Husu Chou. Neural (tangent kernel) collapse. Advances in Neural Information Processing Systems, 36: 16240-16270. 2023.
[16] Yanbing Liu, Jianwei Qin, Yan Liu, Xi Yue, Xun Liu, Guoqing Wang, Tianyu Li, Ye Ye, and Wei Li. coysics-constrainng comprehensive optical neural networks. Advances in Neural Information Processing Systems, 000922066992262 2024.
[17] Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. Distilling causal effect of data in class-incremental learning. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pages 3957-3966, 2021.
[18] Jinlin Xiang, Shane Colburn, Arka Majumdar, and Eli Shlizerman. Knowledge distillation circumvents nonlinearity for optical convolutional neural networks. Applied Optics, (1)22217 2183, 2022.
[19] Hrayr Harutyunyan, Ankit Singh Rawat, Aditya Krishna Menon, Seungyeon Kim, and San- jiv Kumar. Supervision complexity and its role in knowledge distillation. arXiv preprint 201.42230..1224, 2023.
11

---

[20] Xing Lin, Yair Rivenson, Nezih T. Yardimci, Muhammed Veli, Yi Luo. Mona Jarrahi, and Aydogan Ozcan, All-optical machine learning using diffractive deep neural networks. Science, 361 (6446)): 1004-1088, 2018.
[21] Deniz Mengu, Yi I Luo, Yair Rivenson, and Aydogan Ozcan. Analysis of diffractive optical neural networks and their integration with electronic neural networks. IEEE Journal of Selected Topics in Quantum Electronics, 26(1), 2020.
[22] Mingwei Yang, Elizabeth Robertson, Luisa Esguerra, Kurt Busch, and Janik Wolters. Optical convolutional neural network with atomic nonlinearity. Optics Express, 31(10):16451-1645) 2023.
[23] Albert Ryou, James Whitehead, Maksym Zhelyeznyakov, Paul Anderson, Cem Keskin, Michal Bajesy, and Arka Majumdar. Free-space optical neural network based on thermal atomic nonlinearity. Photonics Research, (24):B128-BB) -134, 2021.
[24] Tianyu Wang, Mandar M Sohoni, Logan Wright, Martin M Stein, Shi- Yuan Ma, Tatsuhiro Onodera, Maxwell G Anderson, and Peter L McMahon. Image sensing with multilayer nonlinear optical neural networks. Nature Photonics, 17(5):408-411 2023.
[25] Hanyu Zheng, Quan Liu, You Zhou, Ivan Kravchenko, Yuankai Huo, and Jason Valentine. Meta-optic accelerators for object classifiers. Science Advances, 8300):ca 2022.
[26] Tiankuang Zhou, Xing Lin, Jiamin Wu, Yitong Chen, Hao Xie, Yipeng Li, Jingtao Fan, Huaqiang Wu, Lu Fang, and Qionghai Dai. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nature Photonics, 15(5):367-373) 2021.
[27] Luocheng Huang, Quentin AA Tanguy, Johannes E Fröch, Saswata Mukherjee, Karl F Böhringer, and Arka Majumdar. Photonic advantage of optical encoders, Nanophotonics, 191-1199, 2024.
[28] Zhiwei Xue, Tiankuang Zhou, Zhihao Xu, Shaoliang Yu, Qionghai Dai, and Lu Fang. Fully forward mode training for optical neural networks. Nature, 632(8024) 2024.
[29] Minho Choi, Jinlin Xiang, Anna Wirth-Singh, Seung-HWan Baek, Eli Shlizerman, and Arka Majumdar. Transferable polychromatic optical encoder for neural networks. Nature Communi- cations, 16(1):5623, 2025.
[30] Julie Chang. Vincent Sitzmann, Xiong Dun, Wolfgang Heidrich, and Gordon Wetzstein. Hybrid optical-clectronic convolutional neural networks with optimized diffractive optics for image classification. Scientific reports, 8(1):1-10, 2018
[31] Carlos Mauricio Villegas Burgos, Tianqi Yang. Yuhao Zhu, and A Nickolas Vamivakas. De- sign framework for metasurface optics-based convolutional neural networks. Applied Optics, 60(15):4356-4365 2021.
[32] Zibo Hu, Shurui Li. Russell LT Schwartz, Maria GGoggone,,, Mario Miscuglio, Puneet Gupta, and Volker Sorger. High-throughput multichannel parallelized diffraction convolutional neural network accelerator. Laser & Photonics Reviews, 16( 16(12:22000213 2022.
[33] Cong He, Dan Zhao, Fei Fan, Hongqiang Zhou, Xin adn Yao Li, Junjie Li, Fei Dong, Yin-Xiao Miao, Yongtian Wang, and Lingling Huang. Pluggable multitask diffractive neural networks based on cascaded metasurfaces. Opto-Electronic Advonces, 7(23005), 2024.
[34] Kaixuan Wei, Xiao Li, Johannes Froech, Praneeth Chakravarthula, James Whitchead, Ethan Tseng, Arka Majumdar, and Felix Heide. Spatially varying nanophotonic neural networks. Science Advances, 10(45)eead00391, 2024.
[35] Hanyu Zheng. Quan Liu, Ivan I Kravchenko, Xiaomeng Zhang, Yuankai Huo, and Jason G Valentine, Multichannel meta-imagers for accelerating machine vision. Nature nanotechnology. 19(4):77--77, 2024.
[36] Cong He, Dan Zhao, Fei Fan, Hongqiang Zhou, Xin Li, Yao Li, Junjie Li, Fei Dong, Yin-Xiao Miao, Yongtian Wang, et al. Pluggable multitask diffractive neural networks based on cascaded metasurfaces. Opto-Electron. Adv, 7(2):230005, 2024.
[37] Anna Wirth-Singh, Jinlin Xiang, Minho Choi, Johannes E Fröch, Luocheng Huang, Shane Colburn, Eli Shlizerman, and Arka Majumdar. Compressed meta-optical encoder for image classification. Advanced Photonics Nexus, 422):0260009026009, 2025.
12

---

[38] Anna Wirth-Singh, Jinlin Xiang, Minho Choi, Johannes Fröch, Luocheng Huang. Eli Shlizer- man, and Arka Majumdar. Compressed meta-optical encoder for image classification. In CLEO: Fundamental Science, pages FF1J-1. Optica Publishing Group, 2024.
[39] Yuchi Huo, Hujun Bao, Yifan Peng, Chen Gao, Wei Hua, Qing Yang, Haifeng Li, Rui Wang, and Sung-Fui Yoon. Optical neural network via loose neuron array and functional learning. Nature Communications, 14(1):2533 2023.
[40] Md Sadman Sakib Rahman and Aydogan Ozcan. Time-lapse image classification using a diffractive neural network. Advanced Intelligent Systems. 5(5):2200387, 2023.
[41] Chenyu You, Jinlin Xiang, Kun Su. Xiaoran Zhang, Siyuan Dong. John Onofrey. Lawrence Staib, and James $ Duncan. Incremental learning meets transfer learning: Application to multi-site prostate mri segmentation. In International Workshop on Distributed, Collaborative, and Federated Learning, pages 3-16. Springer, 2022.
[42] Jinlin Xiang, Bozhao Qi, Marc Cerou, Wei Zhao, and Qi Tang. Dn-ode: Data-driven neural-ode modeling for breast cancer tumor dynamics and progression-fiee survivals. Computers in Biology and Medicine, 180: 180:76 2024.
[43] Jachoon Lee, Lechao Xiao, Samuel S Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Finite versus infinite neural networks: An empirical study. In Advances in Neural Information Processing Systems, volume 33, pages 15156-15172 2020.
[44] Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. Advances in Neural Information Processing Systems, 33:20823-20833, 2020.
[45] Yoann Morello, Emilie Grégoire, and Sam Verboven. Exploring task affinities through ntk alignment and early training dynamics in multi-task learning. In NeurlPS 2024 Workshop on Mathematics of Modern Machine Learning, 2024.
[46] Jinlin Xiang and Eli Shlizerman. Tkil: tangent kernel approach for class balanced incremental learning. Xiviv preprint (.XXv:2226608499 2022.
[47] Jinlin Xiang and Eli Shlizerman. Tkil: Tangent kernel optimization for class balanced incremen- tal learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop, pages 3529-3539, 2023.
[48] Ziyang Zheng, Zhengyang Duan, Hang Chen, Rui Yang. Sheng Gao, Haiou Zhang, Hongkai Xiong, and Xing Lin. Dual adaptive training of photonic neural networks. Nature Machine Intelligence, 5(10): 1190::1119-111111 129, 2023.
[49] James Spall, Xianxin Guo, and Alexander Lvovsky. Hybrid training of optical neural networks. Optica, 9(7):803-811, 2022.
[50] Mans-Christiian Ruiz Euler, Mareus N Boon, Jochem T Wildeboer, Bram van de Ven, Tao Chen, Hajo Broersma, Peter A Bobbert, and Wilfred G van der Wiel. A deep-learning approach to realizing functionality in nanoelectronic devices. Nature nanotechnology. 15(12):992-998, 2020.
[51] Yubo Zhang, Rui Chen, Minho Choi, Johannes E Fröch, and Arka Majumdar. General image compression using random-psf metasurfaces and computational back-end. In 2024 Conference on Lasers and Electro-Optics (CLEO), pages 1-3. IEEE, 2024.
[52) Mengran Zhao, Shitao Zhu, Die Li, Thomas Fromenteze, Mohsen Khalily, Xiaoming Chen, Vincent Fusco, and Okan Yurduseven. Frequency-diverse bunching metasurface antenna for microwave computational imaging. IEEE Transactions on Antennas and Propagation, 2024.
[53] Anders Pors, Fei Ding, Yiting Chen, Ilya P Radko, and Sergey Bozhevolnyi. Random-phass metasurfaces at optical wavelengths. Scientific reports, 6(1):28448, 2016.
[54] Matthieu Dupré, Liyi Hsu, and Boubacar Kanté. On the design of random metasurface based devices. Scientific reports, 8(1):7162. 2018
[55] Parker R Wray and Harry A Atwater. Light-matter interactions in films of randomly distributed unidirectionally scattering dielectric nanoparticles. ACS Photonics, 78)):1055-2144 2020.
[56] Laj Cutrona, Leith, Palermo, and I. Porcello. Optical data processing and filtering systems. IRE Transactions on Information Theory, 6(3):386-400, 2003.
13

---

[57] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[58] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov, and Ruosong Wang. On exact computation with infinitely wide neural net. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Process- ing Systems, volume 32. Curran Associates, Inc., 2019.
[59] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11))227--233 1998.
[60] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. Technical report.
[61] Yite Wang, Dawei Li, and Ruoyu Sun. Ntk-sap: Improving neural network pruning by aligning training dynamics. arXiv preprint arXiv:2304. 028400 2023.
[62] Ryan Vogt. Yang Zheng, and Eli Shlizerman. Lyapunov-guides representation of recurrent neural network performance. Neural Computing and Applications, 36(34):21211-2122) 2024.
[63] Caleb Zheng and Eli Shlizerman. Hyperpruning: Efficient search through pruned variants of recurrent neural networks leveraging lyapunov spectrum. arXiv preprint arXiv:2.566.7997. 2025.
[64] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ArXiv, 0bs//1955111466, 2019.
14

---

1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract and introduction clearly reflect the paper's scope and contribu-
tions, stating the proposed NTKD pipeline and supporting claims with both theory and experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: This paper acknowledges limitations such as the shallow and linear nature of current ONNs, the difficulty of implementing nonlinearities and scaling up, and fabrication- induced issues in the Discussion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification:
The Supplementary provides the assumptions and a complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The Supplementary includes optical implementation and setup details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The experiments use open-source benchmark datasets, and additional imple- mentation details are provided in the Supplementary Material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/guides/ htaalees//ooeeSSmmimiiionoffeee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppllligggddpeeeeeeeeeee Cooee//dddeeebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The implementation details provided in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Experiments are repeated with multiple random seeds, and results are reported with mean and standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide estimates of MACs and system-level power consumption.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This work complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work does not have a significant societal impact, and it is foundational research.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: ] We correctly credited all the assets used in the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce any datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasettcoodemmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This research did not involve any crowdsourcing human subject studies.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve human subjects or user studies and therefore does not require IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
20

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The core method development in this research does involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standaadd components.
Please refer to our LLM policy for what should or should not be described.
21


Reviews:
[{"review_id": "qCGJozKu61", "reviewer": "Reviewer_MQJm", "summary": "The paper proposes a Neural Tangent Kernel based knowledge distillation process to address the accuracy gap between Optical Neural Networks (ONNs) and large scale networks, and also the discrepancies between simulated and fabricated ONN performance. The paper considers ONN architectures that includes an optical frontend to perform early stage linear convolution and an electronic backend to perform fully connected non-linear operations. Once the optical frontend design is done, the ONN is trained in simulation with end-to-end loss and NTK based loss with pretrained and frozen teacher networks. After ONN fabrication, the digital backend is retrained with NTK loss to compensate for the differences in simulation and fabrication. Experiments are done using MNIST and CIFAR10 for classification, and Carvana for segmentation.", "questions": "1. How would the proposed NTK based approach scale for large models?\n2. What’s the size of n_{batch} in the experiments and how is it determined?\n3. How are the two loss terms balanced in Eq. 9 and how sensitive is this choice?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n-\n1. The proposed NTK-based distillation approach is general enough for different ONN architectures and vision tasks.\n2.The paper shows performance improvements on Compressed Meta, Polychromatic Meta, and Extreme Meta ONNs (in simulation) for both classification and segmentation.\n3. Experimental evaluations are conducted in both simulated (pre-fabrication) and fabricated (post-fabrication) settings.\n\nWeaknesses\n-\n1. The dataset and teacher networks used are not SOTA anymore and relatively simple. While these are used widely in ONN literature, using ResNet variants with ImageNet and COCO type datasets would’ve made the paper stronger.\n\n2. NTK loss requires computing product of Jacobians which is computationally expensive and will not scale for large networks. Also, what’s the size of n_{batch} in the experiments and how is it determined?\n\n3. The paper doesn’t include any discussion on alpha and beta in the Eq. 9. How are the two loss terms balanced?", "was_revised": true, "final_justification": "The paper proposes an effective approach for knowledge distillation for ONN and conducted experiments for different ONNs, network  architectures, and datasets. Scalability is a concern but the authors showed benefits even with an approximation startegy.", "rebuttal": "We appreciate the reviewer acknowledging the generalizability of our NTK-based distillation pipeline across diverse ONN architectures and vision tasks, as well as its contribution to advancing current ONNs’ performance in both simulated (pre-fabrication) and fabricated (post-fabrication) stages.\nBelow, we provide clarifications in response to the reviewer’s questions.\n\n## 1. Clarification on Dataset and Teacher Network Selection (W1)\nWe appreciate the reviewer’s feedback and agree that our current experiments are based on relatively simple datasets (e.g., MNIST, CIFAR-10) and teacher networks (e.g., AlexNet). \n\nThis choice reflects the current state of the field, particularly for **SOTA ONN architectures** such as Polychromatic Meta (Nature Communications 2025) [1], where the primary bottleneck lies in the complexity of physical implementation. Most fabricated ONNs are limited to shallow and linear architectures due to challenges in optical alignment, lack of physical nonlinear activation, and maintaining signal fidelity. Given these optical constraints, our goal is not to match the performance of large-scale digital networks, but to improve ONN accuracy under realistic hardware limitations. Our pipeline is designed to enhance ONN performance in such settings and help narrow the gap between ONNs and digital networks. In summary, while the datasets in our experiments are considered basic in vision tasks, they remain state of the art benchmark datasets in the ONN field.\n\n**To address the reviewer’s concern and to examine the impact of stronger teachers on the student ONN according to optical physical limitations that we describe above and in the paper, we also conducted additional experiments using ResNet variants and more complex datasets, including ImageNet-100 and COCO-Stuff 10k.** For the classification task (as we stated in the main paper, line 277), we employed both ResNet-18 and ResNet-50 to train a Polychromatic Meta student network. While both teacher models improved performance through NTK distillation, the gain from ResNet-18 to ResNet-50 was marginal, with only a 1.54% increase.\n\nSimilarly, for the segmentation task, we used U-Net and ResU-Net as teacher networks to train a Polychromatic Meta-optical student on COCO-Stuff 10k for binary foreground-background segmentation. In this setting, the foreground includes all semantic object classes, and the background consists of non-object regions. Again, while both stronger teacher models provided improvements, the performance gain from a more complex teacher was limited. \n\nThese results indicate that the performance bottleneck is primarily due to the physical modeling limitations of current ONN hardware, rather than the complexity of the teacher model. We do expect the ONN technology to improve in the future, where a stronger teacher network can provide added benefit. In such cases the NTKD approach should be applicable for knowledge transfer to enhanced students. We will add the results below to the revised version of our paper and include the discussion above in the discussion section.\n\n**Table 1: Impact of Teacher Complexity on NTK Distillation Performance for Classification and Segmentation Tasks**\n\n| Dataset         | Task          | Teacher     | Student             | Teacher Accuracy | Student Accuracy (with / without NTKD) |\n|----------------|---------------|-------------|----------------------|------------------|----------------------------------------|\n| ImageNet-100   | Classification| ResNet-18   | Polychromatic Meta  | 78.43%           | 46.32% / 33.45%                         |\n| ImageNet-100   | Classification| ResNet-50   | Polychromatic Meta  | 88.32%           | 47.86% / 33.45%                         |\n| COCO-Stuff 10k | Segmentation  | U-Net       | Polychromatic Meta  | 61.89%           | 41.43% / 35.03%                         |\n| COCO-Stuff 10k | Segmentation  | Res U-Net   | Polychromatic Meta  | 69.23%           | 42.73% / 35.03%                         |\n|||||||\n\n## 2. Clarification on NTK calculation and batch size selection (W2, Q1-2)\n\nExplicit computation of the NTK via Jacobian-Jacobian products is indeed memory-intensive and could become impractical for large-scale settings, especially when scaling to deeper networks or larger datasets such as ImageNet or COCO.\n\nTo make NTK computationally feasible across different network architectures and scales, we follow the approximation strategy proposed in NTK-SAP (ICLR 2023) [2]. This method has been shown to be effective in computing NTK for ResNet-18 and ResNet-50, further supporting its scalability to deep architectures. Specifically, this method estimates the trace of the NTK rather than constructing the full NTK matrix, defined as\n\n$\\|\\| \\Theta \\|\\|_{tr} = \\|\\| \\nabla _\\theta f(X; \\theta) \\|\\|^2_F$\n\nwhere $\\Theta$ denotes the Neural Tangent Kernel (NTK), $f(X; \\theta)$ represents the neural network output for an input batch and parameters $ \\theta $, and $ \\|\\| \\cdot \\|\\|^2_F $ denotes the squared Frobenius norm.\n\nComputing the Frobenius norm of the Jacobian matrix is both memory- and computation-intensive, especially for deep networks and large input batches. To alleviate this, NTK-SAP proposes a finite-difference approximation of the trace, which avoids explicit Jacobian construction. Following this approach, we approximate the NTK trace by computing the expectation\n\n$\n\\|\\|\\nabla_\\theta f(X; \\theta)\\|\\|^2_F \\approx \\frac{1}{\\epsilon} \\cdot \\mathbb{E}_{\\Delta\\theta \\sim \\mathcal{N}(0, \\epsilon I)} \\left[\\|\\| f(X; \\theta) - f(X; \\theta + \\Delta\\theta)\\|\\|^2_2\\right]\n$\n\nwhere we set $\\epsilon$=0.01, following the choice in the original NTK-SAP paper. The computational load is comparable to a single forward pass in evaluation mode. While it is an approximation, this strategy has been validated in prior work (NTK-SAP, ICLR 2023) to capture essential NTK trends and ensure stable training, making it a practical choice for scaling NTK-based methods to deeper networks and realistic ONN applications [3-4].\n\n**Batch Size**: We follow the batch size strategy from NTK-SAP on NVIDIA V100 GPUs, which uses batch sizes up to 256 for most datasets with ResNet-18 and ResNet-50. In our experiments, we observed that larger batch sizes lead to slightly higher loss, so we tested batch sizes of 64, 128, and 256 across different datasets. For U-Net segmentation with input size 384 by 216, we start with a batch size of 8 and also test smaller sizes such as 4 and 2. While the performance may slightly vary, the overall trends remain consistent, with accuracy fluctuations within ±2%.\n\n**To address the reviewer’s concern, we have added the following clarification to the main paper**:\n\n_\"Explicitly computing the NTK via Jacobian–Jacobian products is memory-intensive and impractical for large-scale settings. Therefore, we follow the approximation strategy proposed in NTK-SAP (ICLR 2023), which estimates the trace of the NTK rather than constructing the full NTK matrix.\"_\n\n_\"In terms of batch size, we use 128 for MNIST, CIFAR-10/100, 64 for ImageNet-100 experiments, and 8 for image segmentation tasks.\"_\n\n## 3. Clarification on balancing loss terms (W3, Q3)\n\nWe thank the reviewer for pointing this out. We experimentally observed that the two loss terms are on a similar scale, so we balance them using a scalar weight alpha, selected through hyperparameter search over {0.1, 0.3, 0.5, 0.7, 0.9} on a validation set. We will clarify this in the revised manuscript. For other datasets beyond those we tested, users can tune their own hyperparameters or design a custom decay schedule if needed.\n\n**To address the reviewer’s concern, we have added the following clarification to the main paper**:\n\n_\"We experimentally observed that the two loss terms are on a similar scale, so we balance them using a scalar weight alpha, selected through hyperparameter search over {0.1, 0.3, 0.5, 0.7, 0.9} on a validation set.\"_\n\n[1] Choi, Minho, et al. \"Transferable polychromatic optical encoder for neural networks.\" Nature Communications 16.1 (2025): 5623\n\n\n[2] Wang, Y., Li, D., & Sun, R. (2023). NTK-SAP: Improving neural network pruning by aligning training dynamics. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR).\n\n[3] Vogt, R., Zheng, Y., & Shlizerman, E. (2024). Lyapunov-guided representation of recurrent neural network performance. Neural Computing and Applications, 36(34), 21211-21226.\n\n[4] Zheng, C., & Shlizerman, E. (2025). Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum. arXiv preprint arXiv:2506.07975."}, {"review_id": "MKnJu0q8UB", "reviewer": "Reviewer_zR5Y", "summary": "This paper introduces a neural tangent kernel (NTK)-based knowledge distillation (NTKD) method to enhance the performance of hybrid opto-electronic neural networks (ONNs). Through extensive simulations and real-world experiments in image classification and segmentation, the authors demonstrate that NTKD significantly improves model accuracy while compensating for fabrication errors via digital-backend fine-tuning.", "questions": "Please refer to [Cons]", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## [Pros]\nThe major novelty of this paper lies at the proposed NTKD loss defined in (8), for which the authors demonstrate via comprehensive experiments the advantages over existing baselines. The effectiveness is evaluated under various optical neural network settings (monochromatic, polychromatic, different architectures, datasets) in both simulation and real-world prototype experiments, which is in general convincing. \n\n## [Cons]\n\n### Major comments:\nAs a seemingly generic knowledge distillation method, the authors should discuss why this is tailored for optical neural networks (ONNs). NTK, as a (somewhat impractical) theoretical tool is seldom used in deep learning practice as the large computation efforts are entailed to evaluate the NTK matrices. The use of NTKD loss function should be highly expensive, thus hindering its applicability in deep neural networks. It's therefore unclear how did authors apply this loss function to relative heavy deep neural network (i.e., U-Net as the teacher network mentioned for image segmentation)\nThe dimension specification of Jacobians matrices in L149 seems to be wrong. The number of classes is ignored there, IMO, the Jacobians matrix dimension should be $n_{batch} \\times p*c$ where c is the number of classes. Such a linear dependency in number of classes also imposes larger computation burden for handling more complex image classification tasks (such as ImageNet classification)\n\n### Minor comments:\nIn Table 2, the results of the teacher networks should be added as references. The number of digital parameters of the teacher and student network should also be provided for comprehensive evaluation of the proposed method. The paragraph \"Parameter-Space Compressibility\" in Section 4.3 somewhat deviates from central theme of this paper, thus would be better defered to supplementary material. Similarly, more discussions of the ONNs in supplementary could be moved in main paper. \nIn abstract, the authors state \"To assist optical system design before training, we estimate achievable model accuracy based on user-specified constraints such as physical size and the dataset\", however, there is no insightful discussion in the paper how the NTK can be used to guide ONN design before training. \n\n### Justification\nThrough the proposed NTKD method, this paper advances a recent line of research to bridge the gap between ONN and electronic networks. The proposed method looks sound and the experimental results are promising. However, the computation cost and the scalability are not well exposed, which are likely the key limitations of the proposed approach. As a result, I hold a borderline opinion at this moment, and would adjust my position in the rebuttal stage.", "was_revised": true, "final_justification": "I am satisfied with the authors' response. I thus raise my score and vote for acceptance of this work.", "rebuttal": "We appreciate the reviewer’s thoughtful feedback and the recognition of our NTK-based loss and the comprehensive experimental validation across diverse ONN architectures and tasks. While the NTK loss is an important component, our main contribution lies in proposing a holistic pipeline for ONNs, encompassing training, fabrication, and post-fabrication fine-tuning. Motivated by the challenges of bridging simulated and fabricated ONNs, we demonstrate that NTK-based distillation provides a general and effective solution. Our pipeline is validated across multiple ONN designs, including empirical optical implementations, to systematically improve ONN performance under hardware constraints and narrow the gap to digital networks.\n\n## 1. Clarification on NTKD Applicability and NTK Computation Feasibility (Major Concern)\n> “As a seemingly generic knowledge distillation method, the authors should discuss why this is tailored for optical neural networks (ONNs).”\n### **Clarification on NTKD’s Suitability for ONNs:**\nWe appreciate the reviewer’s concern and would like to clarify why our NTK-based distillation is particularly suited for optical neural networks (ONNs). Most SOTA ONNs use optical frontends that perform fixed and linear operations, such as diffraction or Fourier transforms. NTK provides a linear approximation of how the network behaves under small parameter changes, which aligns well with the linear nature of our optical systems. This makes NTK-based matching a natural fit for transferring knowledge from digital teacher models to ONNs.\n\n### **Clarification on NTK calculation:**\nExplicit computation of the NTK via Jacobian-Jacobian products is indeed memory-intensive and could become impractical for large-scale settings, especially when scaling to deeper networks or larger datasets such as ImageNet or COCO. We also thank the reviewer for pointing out the omission of the number of classes in the Jacobian dimension specification (L149). We will revise the text.\n\nTo make NTK computationally feasible across different network architectures and scales, we follow the approximation strategy proposed in NTK-SAP (ICLR 2023) [2]. This method has been shown to be effective in computing NTK for ResNet-18 and ResNet-50, further supporting its scalability to deep architectures. Specifically, this method estimates the trace of the NTK rather than constructing the full NTK matrix, defined as\n\n$\\|\\| \\Theta \\|\\|_{tr} = \\|\\| \\nabla _\\theta f(X; \\theta) \\|\\|^2_F$\n\nwhere $\\Theta$ denotes the Neural Tangent Kernel (NTK), $f(X; \\theta)$ represents the neural network output for an input batch and parameters $ \\theta $, and $ \\|\\| \\cdot \\|\\|^2_F $ denotes the squared Frobenius norm.\n\n$\n\\|\\|\\nabla_\\theta f(X; \\theta)\\|\\|^2_F \\approx \\frac{1}{\\epsilon} \\cdot \\mathbb{E}_{\\Delta\\theta \\sim \\mathcal{N}(0, \\epsilon I)} \\left[\\|\\| f(X; \\theta) - f(X; \\theta + \\Delta\\theta)\\|\\|^2_2\\right]\n$\n\nwhere we set $\\epsilon$=0.01, following the choice in the original NTK-SAP paper. The computational load is comparable to a single forward pass in evaluation mode. While it is an approximation, this strategy has been validated in prior work (NTK-SAP, ICLR 2023) to capture essential NTK trends and ensure stable training, making it a practical choice for scaling NTK-based methods to deeper networks and realistic ONN applications [2-3].\n\n**To address the reviewer’s concern, we have added the following clarification to the main paper**:\n\n_\"Explicitly computing the NTK via Jacobian–Jacobian products is memory-intensive and impractical for large-scale settings. Therefore, we follow the approximation strategy proposed in NTK-SAP (ICLR 2023), which estimates the trace of the NTK rather than constructing the full NTK matrix.\"_\n\n### **NTK Distillation Load Analysis in Image Segmentation (U-Net):**\nIn our segmentation task, the input is a 3-channel image of size H×W (shape: [B, 3, H, W]), and the model outputs a predicted binary mask with shape [B, 1, H, W]. Following the definition in NTK-SAP, the Jacobian matrix has a shape [B × H × W, number_of_parameters]. We adopt this reshaping convention (as described in Section 4, Equations 4–6 of NTK-SAP) to calculate the NTK trace for image segmentation. \n\n## 2.Clarification on Minor Comments\nWe thank the reviewer for the helpful suggestions and agree with all the points raised.\n> \"In Table 2, the results of the teacher networks should be added as references. The number of digital parameters of the teacher and student network should also be provided for comprehensive evaluation of the proposed method. \"\n\n- The teacher model results are already included in Lines 203–207; we will also add this information to Table 2 for clarity.\n**Table: Comparison of NTKD Performance Across Different Student Architectures and Teacher Models**\n\n| Student Architecture            | Teacher Model        | Teacher Accuracy | Teacher Size  |\n|--------------------------------|----------------------|------------------|---------------|\n| Compressed Meta (0.05M)        | LeNet                | 99.1%            | ~0.18M        |\n| Polychromatic Meta (1.62M)     | AlexNet              | 85.4%            | 233.29M       |\n| Polychromatic Meta (1.06M)     | U-Net                | 95.4%            | 196.97M       |\n|||||\n\n> \"The paragraph \"Parameter-Space Compressibility\" in Section 4.3 somewhat deviates from central theme of this paper, thus would be better defered to supplementary material. Similarly, more discussions of the ONNs in supplementary could be moved in main paper. \"\n\n- The paragraph on “Parameter-Space Compressibility” will be moved to the supplementary material. We will also add discussion from the supplementary material as suggested.\n\n>\"In abstract, the authors state \"To assist optical system design before training, we estimate achievable model accuracy based on user-specified constraints such as physical size and the dataset\", however, there is no insightful discussion in the paper how the NTK can be used to guide ONN design before training.\"\n\n- We thank the reviewer for pointing out the need for further clarification of the abstract. While the NTK provides a theoretical estimate of model capacity and can be used to predict performance trends before training (e.g., Eq.4 in main paper: $f(x^{\\text{test}};\\theta) = \\Theta_{\\text{test,train}} \\left( \\Theta_{\\text{train,train}} + \\lambda I \\right)^{-1} y^{\\text{train}}$), we agree that its utility for guiding ONN design is less practical in real-world scenarios, particularly when applied to large-scale datasets. \n\n**We will revise the abstract to more accurately reflect our contribution:**\n\n_\"To assist optical system design before training, we design the metasurface layout based on fabrication constraints.\"_\n\n[1] Wang, Y., Li, D., & Sun, R. (2023). NTK-SAP: Improving neural network pruning by aligning training dynamics. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR).\n\n[2] Vogt, R., Zheng, Y., & Shlizerman, E. (2024). Lyapunov-guided representation of recurrent neural network performance. Neural Computing and Applications, 36(34), 21211-21226.\n\n[3] Zheng, C., & Shlizerman, E. (2025). Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum. arXiv preprint arXiv:2506.07975."}, {"review_id": "sg2HmpQ9au", "reviewer": "Reviewer_KgM7", "summary": "This work introduces a unified framework for ONNs that integrates the Neural Tangent Knowledge Distillation strategy to address key limitations in current ONN research: 1) the accuracy gap between ONNs and large-scale digital neural networks during training; 2) the performance degradation caused by discrepancies between simulation and physical fabrication, stemming from optical misalignments, material imperfections, and sensor noise. The extensive experiments show that the proposed pipeline works for both image classification and segmentation tasks.", "questions": "Please check my questions in the weaknesses part.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nS1. The motivation is clear and the introduction of NTKD seems clever.\n\nS2. The method demonstrates good performance on segmentation tasks, which are particularly challenging for optical neural networks due to their complexity and dense prediction requirements.\n\nS3. The proposed method is validated on both simulated and fabricated systems, bridging simulation-reality gaps.\n\nWeaknesses:\n\nW1. The NTK-based estimation in this work appears to be demonstrated only for PSF-based implementations, and it remains unclear how it generalizes to other optical systems like 4f architectures. This raises some concern regarding the hardware-agnostic claim of the proposed pipeline.\n\nW2. From line 110, the authors state that “The optical convolution is realized either through a 4f system or via point spread function (PSF)-based free-space propagation.” However, the paper does not present any experimental results involving a 4f system. This omission undermines the claim of the pipeline being hardware-agnostic.\n\nW3. The experiments for classification tasks are limited to MNIST and CIFAR-10, both of which are relatively simple datasets. To more convincingly demonstrate the generalizability and robustness of the proposed approach, it would be beneficial to evaluate on more complex datasets, such as CIFAR-100 or ImageNet subsets.\n\nW4. The criteria for selecting teacher networks are not clearly explained. The authors mention in line 145 that they “assume access to a pretrained teacher network, such as LeNet for MNIST, AlexNet for CIFAR-10, or U-Net for image segmentation tasks.” However, it is unclear why more powerful or modern teacher networks were not considered, which might further improve distillation effectiveness and the student ONN’s performance.", "was_revised": true, "final_justification": "The authors have adequately addressed my concerns regarding the dataset and teacher network selection. Their clarifications were clear and convincing. I believe the proposed approach has strong practical value for ONN applications. Based on the rebuttal and improvements, I have decided to raise my rating.", "rebuttal": "We sincerely thank the reviewer for their feedback and recognizing the motivation, novelty of our NTKD approach, and its effectiveness in both classification and segmentation tasks, as well as its potential to bridge the simulation-to-fabrication gap in ONNs.\n\n## 1. Clarification on Optical System Generality (W1–2)\n\nWe appreciate the reviewer’s concern regarding the inclusion of 4f optical systems, which may seem to challenge the “hardware-agnostic” claim of our pipeline. We would like to clarify that this concern likely stems from a conflation of theoretical feasibility and practical implementation scope. **To clarify this, we address the issue in two parts: (1) theoretical equivalence of optical convolutions; and (2) practical motivation for implementing PSF-based ONNs.**\n\n### 1) **Theoretical equivalence of optical convolutions**\n\nBoth the 4f optical system and PSF-engineered metasurface follow the convolutional relationship as the multiplications in the frequency space are identical to the convolutional operations in the real space [1]. Once convolutional kernels are trained digitally, the convolution operation can be physically implemented by either a 4f optical system or a PSF-based metasurface. Therefore, we state that “the optical convolution is realized either through a 4f system or via PSF-based free-space propagation.”, which is supported by prior studies [1–4]. \n\nSpecifically, the 4f system can realize convolution operation by performing a 2D Fourier transform of the input image using the first lens by Fraunhofer diffraction, applying a spatial frequency filter in the Fourier plane, and then using the second lens to perform an inverse Fourier transform [1]. In our context, once a convolutional kernel is trained by NTKD, it can be converted into an optical filter design by mapping its frequency-domain representation onto the spatial light modulator or mask placed at the Fourier plane.\n\n### **2) Practical Implementation Choice**\nWhile we acknowledge that our experimental implementation focuses on the PSF-based system, this decision is guided purely by practicality rather than technical limitation. Previous works have identified several challenges in scaling 4f systems for real-world deployment:\n- **Bulk and alignment sensitivity**: A 4f system typically requires two lenses and a spatial filter, increasing the physical size and introducing greater sensitivity to misalignments [5].\n\n- **Parallelization bottlenecks**: In 4f systems, the filtering optics must be confined to a compact region at the Fourier (focal) plane, which restricts the system’s ability to perform parallel processing. This architectural constraint makes it challenging to efficiently handle multi-channel inputs (e.g., RGB). In contrast, metasurfaces can modulate the entire optical wavefront in a single chip, providing a more compact, scalable, and multiple-channel solution [5-6].\n\nIn summary, our physical prototype adopts the PSF-based metasurface design to reflect a more realistic and deployable architecture, while maintaining theoretical compatibility with 4f architectures. **To address the reviewer’s concern, we will also adjust our claim in the original paper:**\n\n_\"Prior works have demonstrated that optical convolution can be physically realized using either a 4f system or a PSF-based free-space propagation system [1-4]. \"_\n\n_\"In this work, we practically implement a PSF-based metasurface design due to its advantages in compactness, alignment robustness, and ease of fabrication [5-6].\"_\n\n## 2. Clarification on Dataset and Teacher Network Selection (W3–4)\nWe appreciate the reviewer’s feedback and agree that our current experiments are based on relatively simple datasets (e.g., MNIST, CIFAR-10) and teacher networks (e.g., AlexNet). \n\nThis choice reflects the current state of the field, particularly for **SOTA ONN architectures** such as Polychromatic Meta (Nature Communications 2025) [1], where the primary bottleneck lies in the complexity of physical implementation. Most fabricated ONNs are limited to shallow and linear architectures due to challenges in optical alignment, lack of physical nonlinear activation, and maintaining signal fidelity. Given these optical constraints, our goal is not to match the performance of large-scale digital networks, but to improve ONN accuracy under realistic hardware limitations. Our pipeline is designed to enhance ONN performance in such settings and help narrow the gap between ONNs and digital networks. In summary, while the datasets in our experiments are considered basic in vision tasks, they remain state of the art benchmark datasets in the ONN field.\n\n**To address the reviewer’s concern and to examine the impact of stronger teachers on the student ONN according to optical physical limitations that we describe above and in the paper, we also conducted additional experiments using ResNet variants and more complex datasets, including ImageNet-100 and CIFAR-100.** For the classification task (as we stated in the main paper, line 277), we employed both ResNet-18 and ResNet-50 to train a Polychromatic Meta student network. While both teacher models improved performance through NTK distillation, the gain from ResNet-18 to ResNet-50 was marginal, with only a 1.54% increase.\n\nThese results indicate that the performance bottleneck is primarily due to the physical modeling limitations of current ONN hardware, rather than the complexity of the teacher model. We do expect the ONN technology to improve in the future, where a stronger teacher network can provide added benefit. In such cases the NTKD approach should be applicable for knowledge transfer to enhanced students. We will add the results below to the revised version of our paper and include the discussion above in the discussion section.\n\n**Table 1: Impact of Different Teacher Models and NTKD on Student Performance**\n\n| Dataset        | Teacher     | Student             | Student Accuracy (with / without NTKD) |\n|----------------|-------------|---------------------|----------------------------------------|\n| ImageNet-100   | ResNet-18   | Polychromatic Meta  | 46.32% / 33.45%                        |\n| ImageNet-100   | ResNet-50   | Polychromatic Meta  | 47.86% / 33.45%                        |\n| CIFAR-100      | ResNet-18   | Polychromatic Meta  | 49.32% / 43.37%                        |\n| CIFAR-100      | ResNet-50   | Polychromatic Meta  | 49.54% / 43.37%                        |\n|||||\n\n[1] Chang, Julie, et al. \"Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification.\" Scientific reports 8.1 (2018): 12324.\n\n[2] Cutrona, Laj, et al. \"Optical data processing and filtering systems.\" IRE Transactions on Information Theory 6.3 (2003): 386-400.\n[3] Burgos, Carlos Mauricio Villegas, et al. \"Design framework for metasurface optics-based convolutional neural networks.\" Applied Optics 60.15 (2021): 4356-4365. \n\n[4] Xiang, Jinlin, et al. \"Knowledge distillation circumvents nonlinearity for optical convolutional neural networks.\" Applied Optics 61.9 (2022): 2173-2183.\n\n[5] Wirth-Singh, Anna, et al. \"Compressed meta-optical encoder for image classification.\" Advanced Photonics Nexus 4.2 (2025): 026009-026009.\n\n[6] Choi, Minho, et al. \"Transferable polychromatic optical encoder for neural networks.\" Nature Communications 16.1 (2025): 5623."}]

Author rebuttal:

Meta-review: Reviewers agree that this is a significant step forward in the field. The rebuttal and post-rebuttal discussion clarified initial questions. Congratulations, this submission is accepted.
Decision: Accept (poster)