L2ight: Enabling On-Chip Learning for Optical Neural Networks via Efficient in-situ Subspace Optimization
Authors: Jiaqi Gu
Hanqing Zhu
Chenghao Feng
Zixuan Jiang
Ray Chen
David Z. Pan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Optical Neural Networks
On-Chip Learning
Efficient Training
Scalability
Hardware-Software Co-Design

Abstract: Silicon-photonics-based optical neural network (ONN) is a promising hardware platform that could represent a paradigm shift in efficient AI with its CMOS-compatibility, flexibility, ultra-low execution latency, and high energy efficiency. In-situ training on the online programmable photonic chips is appealing but still encounters challenging issues in on-chip implementability, scalability, and efficiency. In this work, we propose a closed-loop ONN on-chip learning framework L2ight to enable scalable ONN mapping and efficient in-situ learning. L2ight adopts a three-stage learning flow that first calibrates the complicated photonic circuit states under challenging physical constraints, then performs photonic core mapping via combined analytical solving and zeroth-order optimization. A subspace learning procedure with multi-level sparsity is integrated into L2ight to enable in-situ gradient evaluation and fast adaptation, unleashing the power of optics for real on-chip intelligence. Extensive experiments demonstrate our proposed L2ight outperforms prior ONN training protocols with 3-order-of-magnitude higher scalability and over 30x better efficiency, when benchmarked on various models and learning tasks. This synergistic framework is the first scalable on-chip learning solution that pushes this emerging field from intractable to scalable and further to efficient for next-generation self-learnable photonic neural chips. From a co-design perspective, L2ight also provides essential insights for hardware-restricted unitary subspace optimization and efficient sparse training. We open-source our framework at the link.

L'ight: Enabling On-Chip Learning for Optical Neural Networks via Efficient in-situ Subspace Optimization
Jiaqi Gu, Hanqing Zhu, Chenghao Feng, Zixuan Jiang, Ray T. Chen, David Z. Pan ECE Department, University of Texas Austin
fiqgu, (.ccuu.engchenghaoo9999.... @ edu,,s.edu, {chen, dpan @ @ee.utexxs.eeee
Abstract
pilicon-photonics-baaee optical neural network (ONN) is a promising hardware platform that could represent a paradigm shift in efficient Al with its CMOS- compatibility, flexibility, ultra-low execution latency, and high energy efficiency. In-situ training the online programmable photonic chips is appealing but still en- counters challenging issues in on-chip implementability. scalability, and efficiency. In this work, we propose closed-loop ONN on-chip learning framework L'ight to enable scalable ONN mapping and efficient in-situ learning. L L'tght adopts a three-stage learning flow that first calibrates the complicated photonic circuit states under challenging physical constraints, then performs photonic core mapping via combined analytical solving and zeroth-order optimization. A subspace learning procedure with multi-level sparsity is integrated into L L'ight to enable in-situ gra- dient evaluation and fast adaptation, unleashing the power of optics for real on-chip intelligence. Extensive experiments demonstrate our proposed L L'ghht outperforms prior ONN training protocols with 3-order-of-maage of-manituude higher scalability and over 30x better efficiency, when benchmarked on various models and learning tasks. This synergistic framework the first scalable on-chip learning solution that pushes this emerging field from intractable to scalable and further to efficient for next-generation self-learnable photonic neural chips. From co-design perspective, L'ight also provides essential insights for hardware-restricte unitary subspace optimization and efficient sparse training. We open-source our framework at link.
Introduction
The escalating scales of deep learning models and datasets have brought increased demand for computing capacities in electronic processors. Stringent performance and efficiency constraints in practical applications raise surging need to develop more efficient computing solutions. As promising substitute for conventional electronics, optical neural networks (ONNs) have attracted extensive research interests owing their sub-nanosecond latency and autojoule/muliillyacccea accuuullae operation (MAC) energy efficiency [42, 6, 51, 41, 54, 14, 15], shown in Figure 1(a).
However, robustness and trainability still critical issues for photonic Al engines [58, 22, 60). Due to the analog computing nature of ONNs, the photonic DNN model inevitably suffer from performance degradation errors, non-ideal device controls, and undesired circuit noises, shown in Figure ((b). Though non-ideal effects can be simulated and considered during software training [58, 22] improve noise tolerance. the variation simulation is physically inaccurate (especially with unknown process variations) and prohibitively expensive, shown in Figure 1(c).
complete malfunction [58, 60] with the existence of manufacturing
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Recently, on on-device train- ing has become an appeal- ing trend towards adapt- able and self-learning ONNs. However, training on photonic neural chips is non-trivial and much less explored than on con- ventional platforms. Prior work [57, 25, 21, 18] only demonstrated small proto- types, and their scalability and efficiency limited. To push the limits of DNNs in optics, we propose an efficient three-stage learning framework L L'ight that consists of variation-agnonn aggnostic identity calibration, alternate projection-based parallel mapping, and multi-level sparse subspace learning. The main contributions of this work are four-fold,
307
Optical
Accurary Accuraa (%)
90
Notso-sin sim
70
Ruatime
109
50
CEPAR: 10
10-1
109
A5IC
GPU
CLFAILUD
CIPU
30
Orig
+Q +CT
-DV
+PB
10
10a
109
Variations
Mutrix Size N
(a)
(b)
(c)
Figure 1: Comprehensive motivations. (a) Computational efficiency superior- ity ONNs [42]. (b) Noise sensitivity of ONNs (Q: 8-bit quantization, CT: crosstalk, DV: device variation, PB: phase bias). (c) Runtime of noise-free matrix multiplication vs, w/ noise simulation (Q+CT+DV).
rather
Scalability, For the first time, an ONN learning protocol can scale up million-level parameters under practical circuit non-ideality, over 3-order-of-magnitude more scalable than prior arts.
Efficiency. We explore multi-level sparsity in in-situ gradient evaluation to trim down unnecessary on-chip training energy and runtime cost.
Learnability. By trading redundant representability, our restricted subspace optimization can provide ONNs with enough adaptability for on-device self-learning and task transfer.
Robustness. Various practical device noises and process variations are considered in situ to facilitate noise-resilient photonic Al engines.
To our best knowledge, this the first framework that supports on-chip training on million- parameter ONNs, over 1000 more scalable and 30x more efficient than prior art. We open-source a PyTorch-centric [38] ONN library torchonn and release our on-chip training framework at link.
Related Work
Optical Neural Network and Training Methods. One of recent ONN architectures adopts singular value decomposition (SVD) to implement matrix multiplication [42], i.c., y = W x = UVV** x. Cas- caded 2-by-2 optical devices, i.e., Mach-Zehnder interferometers (MZIs), are used to construct unitary matrices as the product of series of 2-dimensional unitary rotators, (n) D I [T [1;=1 be found in Appendix A. Beyond offline training [22], ONN
Re((or)) (dtz).
A detailed introduction to ONNs on-chip training methods are proposed to offload the process back onto photonics [25, 21, 18), shown
in Table 1. Brute-force device tuning (BFT) [42, 59] and evolutionary algorithms [57] are applied to search MZI settings. An adjoint variable method (AVM) [25] is proposed to directly evaluate gradients using in-situ light field monitoring. Stochastic zeroth-order optimization (ZOO) [21, 18] is later applied to improve the training efficiency. However, prior methods are hard to scale to larger ONNs either due algorithmic inefficiency or unrealistic hardware complexity.
Table Scalability comparison with prior ONN on-chip training protocols in terms of #Params they can handle, used algorithm, resolution requirement (Req.), and circuit observability requirement. Coh. 1/0 short for coherent input/output [33, 56). ZO, FO mean zeroth- and first-order methods.
BFT [42]
PSO [57]
AVM [25]
FLOPS [21]
MixedTIm [4]
L L'ight
iPaaams Algorithm
100
100
100
1000
-500
~00 M
Z0
ZO
FO
ZO
Z0
ZO+FO
Resolution Req.
Medium
High
Medium
High
Med
Medium
Observability Req.
Coh. I/O
Coh. 1/O
Coh. Per device monitor
Coh. 1/O
Coh.
Coh. I/O
Efficient NN Training Methods. Extensive work has been devoted to accelerating DNN training, among which an important branch is sparse backpropagation. Previous methods mainly focus on approximating matrix multiplication by sparsifying the pre-activation gradients [44], forward and feedback matrices [1. 39), and input feature maps [37]. Quantization to the pre-activation gradients is

---

adopted in [52] to induce sparsity by trading off quantization steps and performance. Other methods also exist, e.g., distributed and low-precision training [2. 3. 26]. However, they are not readily applicable to analog photonic engines, thus not in the scope of our discussion.
Subspace Neural Networks.
Subspace neural networks are special DNN models with restricted parameter space but demonstrate comparable representability to classical NNs. Sparse NNs [23, 50], low-rank NNs [9, 28, 45J, structured NNs [11, 30, 48], Fourier-domain NNs [19, 35, 34], and general frequency-domain NNs [20] were introduced to trim down the redundancy in DNNs by restricting the NN structure, matrix rank, numerical resolution, etc. this work, we deeply explore the trade-off between ONN learnability, trainability, and efficiency in the restricted unitary subspace.
Challenges of ONN On-Chip Training. As unique hardware- optimization problem, ONN in-situ learning encounters fundamental challenges causing scalability issues in prior methods:
Lack of full-observability for in-situ light field. Tracking physical optical field on every waveguide in U and V* is not scalable or practical when ONNs scale up. Per device light field monitoring and calibration [17, 25] involves intractable hardware complexity. In practice, only E can be precisely monitored and efficiently tuned.
Limited input/output observability. In photonic tensor cores, for efficiency consideration, only the final output signals after UEV' can be coherently detected. Intermediate signals of a single unitary projection can not be easily read out without extra hardware support.
Inaccessible gradients for most control variables. Due to the above two limitations, it is challenging to obtain true derivatives w.r.t. the MZI rotation phases in U and V [42, 21, 18], casting fundamental in-situ optimization difficulty as ONN scales up.
To enable in-situ self leffleerning for ONNs, the proposed synergistic framework L'ight provides scalable, efficient, and cthipimmleementable solution that overcomes those hardware restrictions.
3 Synergistic ONN On-Chip Learning Framework L L'ight
In this section, we give formal description of the ONN on-chip training problem and detailed demonstration of our proposed three-stage learning flow L L'ight, shown in Figure. 2.
Manufoctured ONN
Pretrained?
No
Yes
(3) Subspace Learning
[1] Identity Calibration
(2) Parallel Mapping
Forward
Initialize @ q
Map Zeroth-Order Opt.
Data Sampling Feature Sampling
In situ Backward
Calibrate U. to I: Zeroth-Order Opt.
Map V": Zeroth-Order Opt.
Feedback Sampling
Record Best ""
Map Σ: Optimal Singular-value
Update Stochastic First-Order Opt.
Projection
No restort diverge
converge?
Target Ace?
No
Target Acc?
Yes
Yes
Done
Figure 2: Proposed three-stage ONN on-chip learning flow L' l'ight.
3.1 Understanding the ONN On-Chip Learning Problem
The ONN that supports on-chip learning shown in Figure 3, constructed by local storage, control units, interconnects, and photonic tensor cores with coherent I/O [33, 56) and wavelength-division multiplexing (WDM) [55, 46) for parallel processing. The target is to optimize MZI phases & directly on chip under variations. Formally the hardwarre learning problem is,
4 argmin (((QQ))())
Dern),
s.t. ( = W pq pq { -----1---11 p=0.==0
Wp = Upg(
( pq
S pq) E
Vo ( k
Upg( (b)) = T
R Rpas' (D
(D = D
R R (o
pg
pq
j=
Epy (.)) = max
Σ diag
(... cos Si....... :..,,
b, ~ (0. r T~ N~((
3

---

L'ight
Intorconneatts
GLB
XX XX - XX
XX
XX - XX
PTC
Cohement WEM Impuly
XX
w
XX
XX
XX
XX
U XX XX
Caherent WDM Inpul' Readoout
XX
XXL
GLB
LCU
GLB
LCU
Readout
XX
1X
XX
DXC
XX
XXX
PTC
PTC
Figure 3: ONN architecture. PTC: photonic tensor core, GLB: global buffer, LCU: local control unit, EO:
cloctrical-to-optical conversion.
The linear projection in an ONN adopts blocking matrix multiplication, where the M x N weight matrix partitioned into Px Q blocks of size x k. During the optimization of 4. we jointly consider control resolution limits 2(.) [22, 42], device process variations r [22, 21, 18], thermal crosstalk among adjacent devices Ω [21, 60), and unknown phase bias due to manufacturing error 4 for in-situ vobustness-awart training. A detailed non-ideality analysis is in Appendix A.3. For practicality, robustness, and convergence consideration, we select k=9, which is explained Appendix F.
3.2 Identity Calibration (IC): Variation-Agnostic Circuit State Preparation
After manufacturing, unknown process variations in waveguides make the initial state of PTCs unpre- dictable [47, 60]. A primary task is to prepare U and V be identity matrices. However, the calibra- tion problem, i.c., minge vv Ep [IlU the observability and controllability constraints on U and V The closest auxiliary problem that we can solve is the one with absolute operations on unitaries, i.e., minge ,v Ep.q (IllU pg (
( L pq/ - I 2 + V (Φ pq 112). is not solvable given
pg))
4II + V/ ( (g)) 112). We denote those two mean square errors as MSE and MSEV. We rewrite it as surrogate minimization of Lı that can lead to the same solution,
pg
(MSBC MSB"
30D
fii
min
Upp "
Epg Vi 4 pq
E 11. (2)
Pi2
1
The optimal solution for this auxiliary prob- lem is U V = I, where I is not guaran- teed to be an identity matrix but a more general sign-flippigg matrix with arbitrary and unob- servable sign flips on the same columns in U and rows in V*, shown in Figure 4(a). We adopt zeroth-order optimization (ZOO) on Th and TV to calibrate U and V to approach i,
== 7
1'=
PTC Forword (3)
(a)
(b)
Figure (a) Identity calibration with sign flip. (b) Dif-
ferent 70 optimizers on identity calibration. (ZGD: 70
gradient descent with momentum, 7CD: 70 coordinate de- scent, ZTP: ZO three-point. B is best solution recording
shown in Figure 4(b). We show the converged solution of Eq. (2) with unobservable sign flips and suboptimality only has marginal impacts on the following training procedure in later sections.
3.3 Parallel Mapping (PM): Alternate Projection-based Model Deployment
The target is to map the pre-trained weights W onto photonic MZI meshes fidelity. We formulate the parallel mapping a batched k x k-block- wise regression problem,
( with high
ZGD
ZCD
2GD
ZCD
Norm. Distance
10P
ZCD-B
Acqurca Sa (50
100
2TP
ZCD-B
drop by D8P
60
80
2.5%
boost
min Σ
W WPp
W
10-1
byt DSP
10-2
10
20
30
40
60
10
20
30
40
50
60
As analyzed before, aw and aw
a4 aav are too expensive compute in situ. We propose a parallel mapping flow with alternate zeroth-order optimiza- tion on th and yy After conver- gence, we will perform analytical op- timal singular-vall value projection (OSP) to minimize the regression error given fixed singular vectors.
PTC Forward (k)
#PTC Forward
(a)
(b)
Figure 5: ZTP and ZCD-B perform the best in parallel mapping. The optimal singular-value projection leads to significant error drop and accuracy jump.)

---

We show why OSP gives the optimal solution under sign flips and how to perform it on the PTC. Claim 1. Optimal singular-value projection (OSP): the optimal singular value problem, i.e, Eopt argming |IUEV* WI,, WI can be analytically solved on-chip with arbitrary and unknown sign flip.
Proof.
Dopt diag(U d (W""))-) (aag(("wV)) - -iaag (ovg(("""'"'"""
(4)
OSP can be directly achieved using the limited operation set, i.c., {U, U" V, V. supported by the reciprocal PTC itself. Specifically, we configure V = and Σ = I, and shine in a coherent WDM light beam that carries W from right ports. Since the coherent photonic circuit is reciprocal [33], read W on the left ports. Then we configure U I and Σ I, and shine in its adjoint field from left, i..., WUU. We can directly read out the projected optimal diagonal on the right because the sign flips in the unitary matrices naturally cancel out on the diagonal.
Figure 5 compares different ZO optimizers this task. Coordinate-wer optimizers (ZCD [31] and ZTP [13]) outperform the gradient-based ZGD [16] with higher accuracy and convergence speed. This procedure is highly parallel and efficient since the mapping involves stochasticity and only happens locally within each PTC. We can also observe that OSP effectively reduces the normalized matrix
distance () W
W /
W 12)
and boosts the accuracy by 2-5% almost for free.
3.4 Subspace Learning: Hardware-Award Multi-Level Sparse Training
Besides mapping from an offinee-tained model, L'ight also supports in-situ self-learning fully on chip. We name this feature as subspace learning. To make L'ight hardware-awand we trade expensive full-space trainability for efficient subspace gradient evaluation, i.c.. 85 with the general frequency-done this learning stage involves stochasticity, turns out to be the efficiency bottleneck, especially the backward pass. Hence, we explore multi-level sparsity for efficient in-situ gradient approximation.
ac which coincides ONNs [19, 20] and subspace NN design concept [43]. Since
3.4.1 In-situ Subspace Gradient Acquisition via Reciprocity in Optics
The conventional way to compute first-order gra- ac dients w.r.t. Σ is E diag(U* SW V). How-
Forword
V
ac = 0c x requires arbitrary matrix mul- ever, aw By tiplication, which is not implementable by weight- stationary PTCs. Hence, we remap it as,
XXL
poOT mooood
lyv
Oc OΣ
Oc Øgr (iu'
Oc Zy
(IV' (5)
(r
ayg
(7%)
Oy:
XX
XX
xx
By shining in coherent WDM beams carrying the inputs and upstream gradients forward and back- ward through the reciprocal PTCs, respectively, as shown in Figure 6, the weight gradients can be efficiently obtained with lightweight element- wise multiplication O which can be offloaded to electrical units. i naturally cancels out by the Hadamard product with no impacts on gradient fidelity.
r. 0c
XX
500
8%
Kioodd
Backword
Figure 6: In-situ subspace gradient acquisition.
3.4.2 Multi-Level Sparse Subspace Learning
Inspired by sparse backpropagation methods [44, 37, 49, 36, 39], propose multi-level sparse subspace learning to cut down both energy cost and total time steps in on-chip gradient evaluation.
Balanced Feedback Sampling. To improve the efficiency of the error feedback process, i.e., WT ec
0g
as shown in Figure 7, we sample the feedback matrix W E R NxM with structured sparse mask Pw = cw (Sw @ 1) generated by the Kronecker product between a boolean mask Sw {0, 1 with sparsity aw and an all-ones matrix 1, where the scaling factor ew is set ow Tr(ST for unbiased estimation, proven in Appendix D. The efficiency benefits come from two aspects: (1) the structurally masked PTCs are entirely idle, directly saving energy, and (2) the product accumulation depth/step is reduced by factor of aw, effectively trimming time steps.
PQ
5

---

Ang Angular Sto
2.4
boople + 1008
Avg. Auguinr Stn
0,9
htopk (ony
0.6)
Aok. Noem, Distanse
Ave. Augulae Sim.
0.9
Ang. Angular Sim
1.0
AVE Norma Dunatee
acpeii cars
0.8
0.9
accr190 92
accn
0.7
0.7
0.7
0.8
0.0
0.6
CONVI
CONV2
0.5
120
150
CONVI
CONV2
CONVS
0.
#-5ini- Butch (K)
CONV Layer
* 4.linn Butch (K)
CONY Layer
(a)
(b)
(c)
(d)
Figure 8: Average gradient angular similarity with different feedback sparsity (a) and three normalization methods (b). none, exp, and var represents no, expectation- maintinndaittiieed, and pariance-mainnainned normalization.
Average gradient angular similarity with spatial and column sampling (c) and three normalization methods (d).
However, two major downsides exist on traditional uniform and layer-wise topk sampling [39].
First, backward path, multiple feedback sampling opera- tors will be cascaded, such that inportance-unaware uni orm sampling can lead to an exponentially large variance [37]. Second, topk sampling is overly greedy and tends to break the load balance as the feedback latency can be bottlenecked by the longest partial product accumulation path, shown in Figure 7. To tackle this, we propose balanced top-K K sam- pling (btopk) to draw Sw from a guided distribution that locally prefers blocks with large Frobenius norm, which can be efficiently evaluated by |Wplll'= Tr[[[ppll)) It strikes a balance between gradient variance and bias by fine-grained row-wise top-K sampling and eliminates load-imin load-imbalance by guaranteeing the same sparsity for different rows of W Ep (Q.)).
Feedback Sampling
dx
dy
of
Backward
W
Idle PTC -less energy/ time
Balanced
Row-wise
Sampling
Imbalanced
Layer-wise Sampling
Le.. Σp (L,:) = E, Sw (2.:) =
Figure 7: Balanced v.s. imbalanced feedhack matrix sampling.
Figure 8(a), 8(b) shows the gradient approximation fidelity in terms average angular similarity [5] and normalized matrix distance. Our staples weight gradients align well with the true gradients. With the unbiased (exp) normalization factor aw, btopk shows the best gradient angular similarity and inference accuracy compared with others.
Information-Per Preservvnn Column Sampling. Input feature sparsification can also effectively cut down the gradient evaluation cost [39, 37], especially for costly CONV layers. However, with tradi- tional spatial sampling (SS) [39, 37], the input feature map x barely maintains its sparsity regularity after being transformed to flattened patches X via im2col if the kernel size is larger than 1, shown in Figure 9. Hence, we propose a novel column sampling (CS) as better solution. We sample X using
mask (0, 1) H'W" with uniform sparsity ac, which is shared across batches with negligible overhead. This leads to both information preser- vation and efficiency improvement. First, in Fig- ure 9, a pixel appears in multiple columns, such that partial information column sampling. Second, this highly-structuue column dropping directly translates less PTC forward energy and fewer partial gradient accu- mulation steps. In contrast, with spatial mask Sş and spatial sparsity (rg, the masked pixel will be completely dropped with poor regularity after im2col, at the cost of large variance due to infor- mation loss and almost no runtime improvement on this dense linear projection engines. Note that for CONVIXI x 1. CS turns out to be equivalent SS, which can simultaneously save memory and runtime. Figures 8(c), 8(d) show that our proposed CS can obtain better gradient approximation fidelity than prior SS. Different normalization has small effects on model accuracy since feature sampling only happens locally within each layer, without any variance cascade effect. Note that simultancous scaling by aw and ac tends to generate overly-
(1) Spetial Sompling
Pixel Information
y Runtime Reduction
C
H
be maintained after
B Con
BI'W"
(2) Column Sampling
0c
Partial Info Remained
Less Runtime
yy
a.
ow
Figure 9: Spatial and column sampling for CONV.

---

confident gradient approximation, which empirically leads to harmful gradient variance. Hence, we will adopt ac=1 in all experiments.
Data Sampling. After parallel mapping, the ONN is initialized fairly close to the target pre-trained model. It is reasonable and intuitive calibrate it with representative calibration set instead of the entire training set. Inspired by the mini-batch dropping (SMD) technique [49], we integrate this SMD technique into our framework further explore data-level sparsity. Within one training epoch, randomly skip each iteration with probability ap, directly translating to training time reduction.
3.5 Complexity Analysis of Three Stages in L'ight
We assume the total step in IC, PM, and SL is Ti, T2, and T3, respectively. The ONN has L layers, each including an N x N weight matrix partitioned into multiple k x & blocks.
Identity Calibration and Parallel Mapping. Each block optimizes k(k: All LN2 /k² blocks are optimized in parallel. The total step is 2 1)2, for IC and 2L 1)22/k + for PM. The total PTC call is around 2 or 2LN for IC and PM, respectively.
phases using 7O0.
Subspace Learning. We assume the feature map size H W with batch size of B. The detailed complexity analysis is given in Appendix G. The total step is approximately T3INBHWA
According to training profiler, IC and PM in total is 3f -frno-ntud cheaper than the SL stage, since the batched parallel regression is deterministic and data-independent.
4 Results
4.1 Experiment Setup
Datasets. We evaluate L'ight Vowel [10], MNIST [29], FashionMNIST [53], CIFAR-10, CIFAR- 100 [27], and Tinylmagenet [7]. On CIFAR-10/100 and Tinylmagenet, we adopt random crop, flip. color jittering for augmentation.
Models. All models are implemented with our open-source PyTorch-centric ONN library torchonn. We evaluate on a customized MLP (8- 16-16-4) [18] on Vowel, CNN-S SOONVKK3S-CCNNGKKS2 and VGG- 8 [8]/ResNet-188 [24] on CIFAR-IO/ION CNN-L/FashionMNIST is used for ablation studies. VGG-8/ResNet-18 on CIFAR-10/100 are used for accuracy and efficiency comparison. Training details can be found in Appendix E.
FC10) [18] on MNIST, CNN-L ([CONV64K3) FC11)) on FashionMNIST,
Efficiency Evaluation. We assume fully parallel 9-blocking matrix multiplication in photonic tensor cores and sequential partial product accumulation in electronics. All experiments and perfor- mance measurements are based on software simulation with various noise modeling. Our simulator counts the total number of PTC calls as the normalized energy indicator and the longest accumulation path as the normalized latency/runtime indicator. Details of profiling can found in Appendix G.
4.2 Main Results
Scalability Comparison with Prior ONN Learning Protocols. Figure 10 compares L L'ight with two SOTA ONN on-chip training proto- cols, FLOPS [21] and MixedTrn [18]. For ZO methods. i.e., FLOPS and MixedTm, we count the energy and latency of for- ward PTC query in Appendix G. Prior protocols can only handle toy models (~22,000 params) given their algorithmic in- efficiency and instability, while our L'ight shows >1 1,000x higher scalability to han- dle large ONNs (~10 M) on challenging
Pretrain
FL.0PS
MixedTrn
igght
100
80
Accuracy
peed?
40
G Kenergy
20
V/MLP (<-500)
M/CKN (>-1500)
F/CNN (w/B0k)
C16/VCGS c100/Rca1
(~4MM)
(~10M)
ONN Scale (Dataset /Model)
Figure 10: Compare scalability with prior protocols [21, 18).
tasks with comparable accuracy to full-space pre-trained models. Though MixedTrn achieves com-
http:///hhhub..ooooccanau/utooop cifar
7

---

parable accuracy to L' L'ight on small benchmarks, we are still 1.7x faster and 6.9x more energy efficient.
The superiority of ightt provides three important insights: (1) decoupling Z00 from stochasticity and partitioning a large-scale regression problem into a batch of sub-tasks can greatly mitigate the curse of dimensionality both in convergence and efficiency. (2) mapping before learning can fully leverage the pre-trained model to reduce the learning cost. Prior methods have to learn from a severely corrupted solution under variations, while L'ight recovers most accuracy via mapping, leaving a very light workload for subspace learning. (3) Restricted subspace learning provides adequate degree of freedom for training from scratch and task transfer. Also, its compatibility with firstfist-ooo first-order methods significantly boosts the trainability and breaks the scalability barrier for ONN training. We now validate the above insights by extensive experiments.
Training Efficiency Comparison with Prior Sparse Training Methods. In Figure 11, we show accuracy and efficiency comparison of 1) baseline Liight-SL. (BS), 2) L'i with spatial sampling (RAD), 3) L'ight-SL with weight and spatial sampling (SWAT-U), and 4) Leight-SL with all three introduced sampling methods (feedback, column, and data sampling), and 5) our proposed full flow with IC, PM, and sparse SL (L²ight). To clarify, LLighhtSSL performs subspace learning on-chip from scratch without using pre-trained weights, while L igtt includes the full flow, i.e., pre-training mapping, and on-chip training. When we perform subspace learning from scratch, our proposed multi-leve sampling strategies outperform previous RAD and SWAT U by ~3x in hardware cost with comparable accuracy. Though RAD can save the forward peak memory, it leaves the most expensive backward pass unoptimized, which does not fully exploit the sparsity in ONN training. SWAT-U tries to save forward cost by simultaneously sparsifying the forward and feedback weights with shared masks/patterns. However, in our experiment, the forward sparsification considerably degrades the performance, which dilates the efficiency benefits from it. Parallel mapping can fully leverage the pre-trained weights and help our full three-stage flow L L'gght achieve the best accuracy with much faster convergence, leading to over 30x higher energy efficiency and fewer time steps.
M.W.
100
300
400
JID3 HES
"Energy scales by B.1
1c0
221
250
1.0
100
RAD
SHAT
Light-
SHAT
SWAT
Light
VSG Koww
VEG VEG-REFFARR.. IEA 100
AB M00
Barkwward Weigh:
Baadkaard Irput
Accurasy
Fonward
B Barkwass Weight
@ Barkward Ieput
Are Aery
Figure 11: Accuracy and hardware efficiency comparison on VGG-8 (left) and ResNet- (right).
Note that the energy efficiency and latency improvement is not just on the photonic part but systematic performance boost. Our three-level sampling methods directly skip the pruned block, which means the corresponding cost of memory transaction, computation, control, and communication are removed together. Therefore, the sampling sparsity can be directly translated the energy/latency improvement ratio regardless of whether the electrical part dominates the total cost.
4.3
Ablation Studies and Discussion
4.3.1
Multi-Level Sparsity in Efficient Training
Feedback Sparsity. To investigate the impact of feedback sampling strategies, we visualize the gradient approximation fidelity and accuracy curves in Figure 12(a). uniform sampling shows varying performance under different sparsity values due to large gradient variances. topk shows worse performance after sufficient steps due to its biased gradient estimation from overly greedy block selection. In contrast, our proposed load- baad-balancing btopk strikes a balance between variance and bias via block-wise sampling and also leads to less runtime it forces load balance among massively parallel PTCs. In Table 2, feedback sampling saves 50-60% time steps on the most costly error feedback V VzL, leading to 1.5-1.8x overall time step reduction with minimum accuracy drop.
Feature Sparsity. Figure 12(b) compares the accuracy and weight gradient computation time steps on two feature sampling techniques. Though spatial sampling (as) can save peak storage by dropping a subset of activations during the forward pass, shows no gradient computation step reduction. Our hardware-Fe fariedlyy column sampling (cs) directly leads to energy and runtime reduction due to its
8

---

93
02
Large
Accuracy (%)
92
Aceurecy (%)
Accursey (3))
.9
91
btopk
Rantime Reductton
Poak Mentory
Redtaction
90
CNN-L FashionMNIST
90
wniform
8
colurn
88
ow
PC
look
spetial
0.0
89
1.99
8g
0.5
Balance
Load
10
60
120
#Stop in 0C/0z (1E12)
#Skep in 0C/02 (1E16)
# #Mini-Batch (K)
(a)
(b)
(c)
Figure 12: Accuracy v.s. weight gradient computation steps with three feedback sampling strategies (a)
and different feature sampling techniques (b). Accuracy (93.02%) from full-space trained model (green).
CNN-L/FahhouMMNISSI is used for (a) and (b). Compare different data sampling sparsity (c).
structured sparsity. In Table 2, when column sampling is further added, we observe ~50% PTC energy saving on weight gradient computation VyL at the cost of ~1% accuracy drop.
Data Sparsity. In the data level, we also demonstrate how SMD with sparsity aD impacts the training efficiency in Figure 12(c). With the best selected ow and ac, data sparsity directly reduces training time by skipping iterations [49]. The data sampling selects a uniform subset of the training set to represent the original data distribution, leading to less data processing with comparable generalization in the extracted features. Another explanation is that the variance increased by partial replacement serves as a regularization mechanism improve the model performance [49]. For relatively easy tasks, aggressive sparsity (ox==0.8 is sweet point, while for larger datasets shown Table 2, a medium sparsity (0.5) can be a good setting to balance both the training cost and accuracy. With all three sampling methods integrated, our L L'ight-SL shows competitive accuracy and ~33 x higher efficiency than RAD and SWAT-U, More advanced dataset sampling methods are left for future exploration.
4.3.2 Learnability of Restricted Subspace ONNs
Impacts Calibration//apping Quality. Table shows that with IC and PM, the full L'ight flow achieves the highest accuracy with 32-35x efficiency boost over baselines. We further evaluate the impact of different mapping accuracy and the calibration quality subspace learning in Figure 13. First, parallel mapping or pre-training is not a must. Our subspace learning supports first-order optimization on-chip from random initialization. Second, the optimality on subspace bases influences the final accuracy as it determines the upper bound of accuracy that can be recovered by subspace learning. With roughly optimized space bases, i.e., U, V*. subspace learning can efficiently train basis coefficients, i.e., Σ, achieving 5-6% higher accuracy and 9.9x less energy and steps compared with random unitaries (train from scratch). Third, subspace optimization shows low sensitivity on mapping quality and is able to compensate for the suboptimality in singular vectors within a reasonable range.
Energy (#Step) Itatio
1.0
00.03
0.8
Sub. Learn Aoc
energy step
0.6
0.4
9.9x
0.2
-d- Aco-NT
85
0.0
83
from scratch
60
70
80
90
Mapped Acc (%)
Figure 13: Impact of mapping accuracy (VGG-8 CIFAR-10 with aw=ac=0.6, ap=0..). acc-NI is the curve with non-ideal I.
Even with 60% mapped accuracy,
Table 2: Compare sampling strategies on CIFAR-10 in terms of accuracy, activation size reduction, energy. and time step. Forward, weight gradient, and error feedback are denoted C. VvL, and V.C. L SSL learning from scratch, and L L'ight (IC-PM-SL) is the full flow with pre-trained weights and non-ideal i.
Acces (%)
Act((()
V+
Noem, PTC Energy
V.C
Toxal (Ratio)
Vr.
Norm. #Step
V.C
Total (Ratio)
L'ight-Sl. SL. (Baseline) VGG+8
86.062001
8.58
17.16
8.34
54.08 (1.00)
$2.64
5.49
92.02
150.14
-Feelbuck Sampling (o1pm))66)
86.4 26.412005
(1.17)
Column Sampling (mk-=0.6)
85.58
8.58
7.16
(1.72)
32.64
4.67
73.07 (1.78)
Data Sampling (0.500.5)
24.45
4.29
3.58
1.69
9.56 (3.56)
16.32
2.34
17.89
36.54 (3.56)
RAD [37] ((x558885
83 83.68
11.78
8.38
17.16
8.34
34,08 (1.00)
32.64
5.49
92.02
130,14 (1.00
SWAT-U (1,, 0g)
2329
8.31
6.01
(1.17)
(13.66
L.igat UCC .PM (SL)
90.20.00s
0.43
0.36
0.17
0.96 (35.64)
1.63
0.23
1.79
3.65 (35.64)
T."gght-S.. (Baseline) R8
144.49
310.33 (1.000
27.23
1,478.84
1,969.48 (1.00)
Feechack Sampling (05)
31.35amm
72.24
144.49
48.13
(1.17
463.40
27.23
747:22
1,237.85 (1.59)
Columa Sampling (0.5
90.02
4.47
72.24
72.49
48.13
192.86 (1.61)
463.40
15.68
747.21
1,226.30
Data Sampling (0.)00.5)
89.07 oas
4.17
36.13
36.26
21.07
96,16 (3.23)
231.76
7.84
373.71
613.31 (3.21)
[37] [0.9-
89.442011
143.72
309.58
463.53
16.03
1,478.84
1,969000
SWAT-U [39] (aw =0.3, 0.5)
89.21
25.89
50.57
143.64
65.52
259.73
(1.19)
358.40
26.56
1.17.96
1.802.00
L" 1igb
(CC-PM--L))
93.9
4,47
3.61
3.62
2.41
9,64 (32,000
23.16
0.78
37.34
(3,19

---

singular value optimization has enough capability to recover the accuracy to ~90 Fourth. our subspace learning is robust to gradient noises caused by non-ideal (MSEU MMSE x0.013), which shows that ghtgh can tolerate reasonable suboptimality the calibration and mapping stages.
In-situ Transferability in the Restricted Subspace. Another important question to answer is the
transferability of subspace learning.
After mapping, we fix the inherited
(acra.) Res (scra.)
C-10 (sera.)
C-100 (sera.)
VGGS (tron.)----
Rca18 (tron.)
C-10 (tran.)
C-100 (tran.)
unitaries and adapt different tasks
CIFAR-100
CIFAR-10
TinyTnagenet
CIEAR 10/100
90
by only training the singular val- ues. Figure 14 shows that the inher- ited bases span good design space with enough transferability. The in-
Pretiamn
Accuracy (%)
3.5
80
T.5
Accuracy
1.9%
70
70
Pretinin
1.3%
60
380
situ subspace transfer learning shows 1-2% higher final accuracy. Also,
60
Mapped
50
Mapped
ResNet-18
50
40
it uses 3-5x fewer steps to ob- tain the same accuracy as training from scratch. Hence, our proposed Liight finds highly trainable de- sign point while the learnability is still mostly maintained.
0
30 60 90 120
150
0
30
60
90 120 150
# #Mini-Batch (K)
#Mini-Batch (K)
(a)
(b)
Figure 14: (a) Transfer VGG8/Res18 from CIFAR-100 to CIFAR-
10. (b) Transfer Res18 from Tiny to CIFAR 10 and 100.
Conclusion
In this work, we propose the first scalable and efficient on-chip learning framework L'ight for emerging optical neural networks. Our proposed three-stage flow synergistically enables on-chip self-learning via automatic circuit state calibration, parallel model mapping, and efficient subspace learning. To further improve the learning efficiency, explore multi-level sparsity, including balanced feedback sampling, informatiin--peesrvving column feature sampling, and runtime-reduced data sampling. Extensive ablation studies and comparison experiments show Lforder-of-magnnuee scalability improvement over prior on-chip training protocols and 30x efficiency boost compared with previous sparse training methods. We open-source Py PyTorch-centric ONN library torchonn, based on which we release our on-chip learning framework L' L'ight at link. In the future, we will go beyond current software simulation and experimentally validate the effectiveness of L L''ghtt real photonic neural chips.
The authors acknowledge the Multidisciplinary University Research Initiative (MURI) program through the Air Force Office of Scientific Research (AFOSR), contract No. FA 9550-17-1-0077 monitored by Dr. Gernot S. Pomrenke.
Acknowledgments
10

---

References
[1] Menachem Adelman and Mark Silberstein. Faster neural network training with approximate tensor operations, arXiv preprint arXiv: 0000088,,.00,, 2018.
[2] Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In Proc. NeurIPS, 2018.
[3] Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signSGD: Compressed optimisation for non-conves problems. In Proc. ICML,, volume 80, pages 560-569, 2018.
[4] Adel Bibi, EI Houcine Bergon, Ozan Sener, Bernard Ghanem, and Peter Richtarik. A stochastic derivative- free optimization method with importance sampling: Theory and learning control. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[5] Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Ergjardo-Cespedess Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Universal sentence encoder. arXiv preprint 1rXiv:1803.
2018.
[6] Q. Cheng, J. Kwon, M. Glick, M. Bahadori, L. P. Carloni, and K. Bergman. Silicon Photonics Codesign for Deep Learning. Proceedings of the IEEE, 2020.
[7] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. CVPR. pages 248-255, 2009.
[8] Lei Deng, Peng Jiao, Jing Pei. Zhenzhi Wu, and Guoqi Li. GXNOR-Net Training deep neural net- works with terary weights and activations without full precisiin memory under unified discretization framework. Neural Networks, 2018.
[9] Emily Denton, Wojeicch Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation In Proc. NIPS, 2014.
[10] D.H. Deterding, Speaker normalisation for automatic speech recognition. PhD thesis, University of Cambridge, 1989.
[11] Caiwen Ding, Siyu Liao, Yanzhi Wang, Zhe Li, Ning Liu, et al. CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-Cireulant Weight Matrices. In Proc. MICRO, pages 395-408, 2017.
[12] Nicolas Dupuis, Benjamin G. Lee, et al. Design and Fabrication of Low-Insertion-Loss and Low Crosstalk Broadband 2x2 Mach-Zehnder Silicon Photonic Switches. JLT, 2015.
[13] El Houcine Bergou and Eduard Gorbunov and Peter Richtárik. Stochastic Three Points Method for Unconstrained Smooth Minimization. SIAM Journal on Optimization, 2020.
[14] Johannes Feldmann, Nathan Youngblood, Maxim Karpov, Helge Gehring, Xuan Li, Maik Stappers, Manuel Le Gallo, Xin Fu, Anton Lukashchuk, Arslan Raja, Junqiu Liu, David Wright, Abu Sebastian, Tobias Kippenberg. Wolfram Pernice, and Harish Bhaskaran. Parallel convolutional processing using an integrated photonic tensor core. Nature, 2021.
[15] Chenghao Feng, Jiaqi Gu, Hanqing Zhu, Zhoufeng Ying, Zheng Zhao, David Z. Pan, and Ray T. Chen. Sil- icon photonic subspace neural chip for hardware-cfficient deep learning. arXiv preprint 1rX.::2111.06705, 2021.
[16] Saeed. Ghadimi and Guanghui. Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 2013.
[17] Stefano Grillanda. Marco Carminati, Francesco Morichetti, et al. Non-invasive monitoring and control in silicon photonics using CMOS integrated electronics. Opticu, 2014.
[18] Jiaqi Gu, Chenghao Feng. Zheng Zhao, Zhoufeng Ying, Ray T Chen, and David Z. Pan. Efficient on-chip learning optical neural networks through power-aware sparse zeroth-order optimization. In Proc, AAAI. 2021.
[19] Jiaqi Gu, Zheng Zhao, Chenghao Feng, et al. Towards area-efficient optical neural networks: an FFT-based architecture. In Proc. ASPDAC, 2020.
[20] Jiaqi Gu. Zheng Zhao, Chenghao Feng, et al. Towards Haurdware-ffficieen Optical Neural Networks: Beyond FFT Architecture via Joint Learaaility. IEEE TCAD, 2020.
11

---

[21] Jiaqi Gu, Zheng Zhao, Chenghao Feng, Wuxi Li. Ray T. Chen, and David Z. Pan. FLOPS: Efficient On- Chip Learning for Optical Neural Networks Through Stochastic Onder Optimization. In Pruc. DAC, 2020.
[22] Jiaqi Gu, Zheng Zhao, Chenghao Feng. Hanqing Zhu, Ray Chen, David Z. Pan. ROQ: A noise-aware quantization scheme towards robust optical neural networks with low-bit controls. In Proc. DATE, 2020.
[23] Song Han, Huizi Mao, and William Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In Proc. ICLR, 2016.
[24] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR, pages 770-778, 2016.
[25] Tyler W. Hughes, Momchil Minkov, Yu Shi, and Shanhui Fan. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica, 2018.
[26] Xianyan Jia. Shutao Song. Wei He, Yangzihao Wang, Haidong Rong. Feihu Zhou, al. Highly Scalable Deep Learning Training System with Precis-ne Training ImageNet Four Minutes. arXiv preprint arXiv:1807.11200 2018.
[27] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[28] Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Vietor Lem. Speeding-up convolutional neural networks using fine-tuned cp-decompositiinn. In Proc. ICLR, 2015.
[29] Y. LeCun. The MNIST database of handwritten digits. http: / //annn lecun. com/ /e////dddmmis/eeeeeeeeeeeeeeee/e /mmp/s/////////////// 1998.
[30] Zhe Li, Shuo Wang, Caiwen Ding, et al. Efficient recurrent neural networks using structured matrices in fpgas. In ICLR Workshop, 2018.
[31] Xiangru Lian, Huan Zhang, Cho-Jui Hsieh, Yijun Huang, and Ji Liu. A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order. In Proc. NeurIPS, 2016.
[32] Maziyar Milanizadeh, Douglas Aguiar, Andrea Melloni, and Francesco Morichetti. Canceling thermal cross-talk effects in photonic integrated circuits. Light. Technol., 2019.
[33] David A.B. Miller, Analyzing and generating multimode optical fields using self-configuring networks. Optica, 2020.
[34] Mario Miscuglio, Zibo Hu, Shurui Li, Jonathan K. George, Roberto Capanna, Hamed Dalir, Philippe M. Bardet, Puncet Gupta, and Volker J. Sorger. Massively parallel amplitude-only fourier neural network. Optico, 7(12):1812-1819, Dec 2020.
[35] Mario Miscuglio, Zibo Hu, Shurui Li, Jiaqi Gu, et al. Million-channel parallelism Fourier-optic convolu- tional filter and neural network processor. Proc. CLEO, 2020.
[36] Arild Nokland. Direct Feedback Alignment Provides Learning Deep Neural Networks. In Proc. NIPS, 2016.
[37] Deniz Oktay, Nick McGreivy, Joshua Aduol, Alex Beatson, and Ryan P Adams. Randomized automatic diffcrentiation. In Proc. ICLR, 2021.
[38] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv: 1912.0173, 2019.
[39] Md Aamir Raihan and Tor M. Aamodt. Sparse weight activation training. In Proc. NeurIPS, 2020.
[40] M Reck, Zeilinger, HJ Bernstein, al. Experimental realization of any discrete unitary operator. Physical review letters, 1994.
[41] Bhavin Shastri, Alexander N. Tait, T. Ferreira de Lima, Wolfram H. P. Pernice, Harish Bhaskaran, C. Wright, and Paul R. Prucnal. Photonics for artificial intelligence and neuromorphic computing. Nature Photonics, 2021.
[42] Yichen Shen, Nicholas C. Harris, Scott Skirlo. et al. Deep learning with coherent nanophotonic circuits. Nature Photonics, 2017.
12

---

[43] Mengying Sun, Inci M. Baytas, Liang Zhan. Zhangyang Wang, and Jiayu Zhou. Subspace network: Deep multi-task censored regression for modeling neurodegenerrative diseases. In Proc. KDD, page 2259-2268, 2018.
[44] Xu Sun, Xuancheng Ren, Shuming Ma, and Houfeng Wang. meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting, Proc. ICML, 2017.
[45] Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, and Weinan E. Convolutional neural networks with low-rank regularization. In Proc. ICLR, 2016.
[46] D. T. H. Tan, A. Grieco, and Y. Faimman. Towards 100 channel dense wavelength division multiplexing with 100ghz spacing on silicon. Opt. Express, 2014.
[47] Erman Timurdogan, Zhan Su, Christopher V. Poulton, et al. AIM Process Design Kit (AIMPDKv2.0) Silicon Photonics Passive and Active Component Libraries on a 300mm Wafer. Optical Fiber Commu- nication Conference, 2018.
[48] Yitu Wang, Fan Chen, Linghao Song, C.-J. Richard Shi, Hai Helen Li, and Yiran Chen. ReBoc: Accelerat- ing Block-Circulant Neural Networks in ReRAM. In Proc. DATE, pages 1472-1477, 2020.
[49] Yue Wang. Ziyu Jiang. Xiaohan Chen, Pengfei Xu. Yang Zhao, Yingyan Lin, and Zhangyang Wang. 22-Train: Training State-of-the-art CNNs with Over 80% Less Energy. In Proc. NeurlPS, 2019.
[50] Wei Wen, Chunpeng Wu, Yandan Wang. Yiran Chen, and Hai Li, Learning structured sparsity in deep neural networks. In Proc. NIPS, 2016.
[51] Gordon Wetzstein, Aydogan Ozcan, Sylvain Gigan, Shanhui Fan, Dirk Englund. Marin Soljačić, Cornelia Denz, David A. B. Miller, and Demetri Psaltis. Inference in artificial intelligence with deep optics and photonics. Nature, 2020.
[52] Simon Wiedemann, Temesgen Mehari, Kevin Kepp, and Wojciech Samek. Dithered backprop: A sparse and quantized backpropagation algorithm for more efficient deep neural network training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, 2020.
[53] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: Novel Image Dataset for Benchmarking Machine Learning Algorithms. Arxiv, 2017.
[54] Xingyuan Xu, Mengxi Tan. Bill Corcoran, Jiayang Wu, Andreas Boes, Thach G. Nguyen, Sai T. Chu, Brent E. Little, Damien G. Hicks, Roberto Morandotti, Arnan Mitchell, and David J. Moss. 11 TOPS photonic convolutional accelerator for optical neural networks. Nature, 2021.
[55] J. Yu and X. Zhou, Ultra hitra-high-capacci dwdm transmission system for 100g and beyond. IEEE Communi- cations Magazine, 2010.
[56] H. Zhang, M. Gu, X. D. Jiang, J. Thompson, H. Cai,, S. Paesani, R. Santagati, A. Laing, Y. Zhang, M. H. Yung, Z. Shi, K. Muhammad, G. Q. Lo, X. S. Luo, B. Dong, D. L. Kwong. L. Kwek, and A. Q. Liu. An optical neural chip for implementing complex- valued neural network. Nature Communications, 2021.
[57] Tian Zhang, Jia Wang, Yihang Dan, Yuxiang Lanqiu. Jian Dai, Xu Han, Xiaojuan Sun, and Kun Xu. Efficient training and design of photonic neural network through neuroevolution. Optics Expresx, 2019.
[58] Zheng Zhao, Jiaqi Gu, Zhoufeng Ying, et al. Design technology for scalable and robust photonic integrated circuits. In Proc. ICCAD, 2019.
[59] Hailong Zhou, Yuhe Zhao, Gaoxiang Xu, Xu Wang, Zhipeng Tan, Jianji Dong, and Xinliang Zhang. Chip-Scale Optical Matrix Computation for PageRank Algorithm. JSTQE, 2020.
[60] Ving Zhu, Grace Li Zhang, Bing Li, et al. Countering Variations and Thermal Effects for Accurate Optical Neural Networks. In Proc. ICCAD, 2020.
13


Reviews:
[{"review_id": "jZbwIMp11S", "reviewer": "Reviewer_f3Zb", "summary": "Optical Neural Networks (ONN) with their attojoule per Multiply Accumulate energy efficiency and sub-nanosecond latency are becoming useful implementation hardware for large scale deep learning models and datasets. Despite their efficiency and speed, ONNs suffer significant loss in accuracy due to manufacturing defects/non-ideal device controls and circuit noises. Simulating these non-idealities during software training is not scalable. \n\nTraining ONN on-chip is a hard problem that is exacerbated due to above factors. This paper tackles the problem by proposing a three-stage learning framework L2ight. The three stages are variation-agnostic calibration, alternate-projection based parallel mapping and multi-level sparse subspace learning. \n\nL2ight enables scaling up on-chip ONN training, for the first time to models with 10M params (1000x scalable than baseline). This scalability is built on top of an efficient training algorithm utilizing multi-level sparsity and a restricted subspace optimization step which enables sufficient adaptability for on-device learning and task transfer. L2ight also achieves 30x higher efficiency than prior art. Lastly, the subspace learning step is agnostic to device/process variation and mapping noise enabling robust implementation of deep neural networks (DNN).   \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a series of novel strategies at different stages of implementing a pre-trained DNN onto an ONN hardware followed by training for same/different task as pretrained. Optimal Singular Value Projection and batch break down of the mapping problem helps achieve fast and high accuracy mapping of pretrained/from scratch weights on to the ONN.  In-situ subspace gradient acquisition exploits reciprocity in optics for minimizing the cost of expensive gradient calculation. Ablation studies demonstrate the robustness of subspace learning step against device/mapping stage errors. Further, unique sampling strategies such as balanced topk and column sampling further help lower the number of hardware operations without cascading gradient variance across layers. Lastly, the paper also utilizes mini batch dropping (SMD) to improve training efficiency. \n\nSection 4.1, paper describes the usage of number of PTC calls as the normalized energy indicator. Similarly, longest accumulation path is assumed to be the normalized latency/runtime indicator. Authors present a detailed breakdown of the energy/latency calculation in appendix based on the parameters of a given convolution layer. While these metrics are surely key factors impacting the energy consumption and latency of a DNN running on the ONN. It’s not clear that when accounting for other electrical components highlighted in Figure 3 (e.g., EO/GLB/Interconnects) if these factors will dominate the ONN energy and latency. Specifically, as highlighted in Section 3.4.1, part of gradient calculation (element-wise multiplication) is outsourced to electrical control units, is it obvious that the energy/latency cost of this outsourcing is negligible? It would be appreciated if the authors can justify this. \n\nTable 2 shows a comparison of normalized PTC Energy and the normalized number of Steps, the ratios for both metrics are tightly aligned with a couple of exceptions, this goes back to my argument that while the proposed techniques help reduce the number of steps (PTC passes) it’s unclear if a hardware ONN system will receive energy efficiency benefits of the same order (30x). Similar observation holds true for Figure 13 with identical energy/step value variation. Consider consolidating this reduction of steps and normalized energy into a single parameter, e.g., no. of PTC operations, for which considerable reduction is demonstrated. \n\nSection 4.2 and Figure 10 discuss a comparison of L2ight with prior work and demonstrates its superiority. A few clarifications about this data will be helpful, are the reported accuracy numbers the best achievable accuracy without any restrictions on number of training steps? Stats like the number of training steps/wall clock time for the different data points on Figure 10 would be helpful.\n\n Additionally, I would appreciate if the authors could clarify the following question. Where does the 1000x improvement in scalability come from? Is it simply from the batching of the optimization problem or is it due to efficient gradient calculation as described in Section 3.4.1? Also is there something unique about the L2ight flow that allows the demonstrated batching, or the batching could be applied to MixedTrn prior work to get similar scalability improvements? \n\nThe paper is well organized, and the detailed figures provide useful insights. \n\nTypos and minor errors: \n1.\tLine 36, on-device training “has become/is becoming” an appealing trend towards …\n2.\tLine 60, 30x more efficient than prior “art”.\n3.\tLine 66, pi notation should have i=2 as starting and n as the max value. The same inversion is used in later equations representing U(n) in the paper and appendix. \n4.\tFigure 2: Stage 3 should have an accuracy check before finishing, it might be performed in block 3 (Update Σ), consider listing it out separately. \n5.\tFigure 9: Column sampling figure, the arrows start from x12 instead of x22 as in spatial sampling.\n6.\tFigure 14b: Legend appears wrong as it lists dataset name instead of DNN name. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not include any actual hardware measurements, it would be helpful if the authors specify it at some point as a limitation. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "gSri8IAsGI", "reviewer": "Reviewer_EGjz", "summary": "The paper proposes a calibration and training method for optical neural networks, and more precisely suited for Mach Zhender interferometers arrays. The method is claimed to be scalable, and based on easily accessible optical quantities on chip. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is overall well organized, and compared with the state of the art. The method is well described, and the performances are well backed through simulations of a 9x9 photonic tensor core. The results are evaluated in terms of efficiency, memory and number of step reduction. \n\nThe significance is potentially high, optical implementations of ONN using MZI remains highly dependent on the efficient tuning of the interferometer to a given transformation and/or to direct training, and subject to many sources of imperfection. All these aspects are taken into account and the algorithm appears to give a satisfactory answer to all these potential pitfalls at once. The method is also quite interesting, based on injecting the circuit from both sides iteratively to benefit from its time reversal symmetry.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nWhat is not fully clear is the scaling. The paper claims up to 10M parameters, but remains restricted to 9x9 MZI arrays, if I understand well. This means that only relatively small unitaries are performed. I would really have liked to see a study of the scaling with respect to the size of the array, which is a major issue in future ONN. Another issue that could have been detailled is the WDM : how does the algorithms deals with non-idealities of the circuit with respect to multiple wavelength travelling through the same circuit and sensing a slightly different matrix. \n\nPotential negative social impact: None, as this is low level hardware training. ", "review_text": "", "extra_scores": {"ethical_concerns": "None, as this is low level hardware training. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "2CZk3-fg3_X", "reviewer": "Reviewer_jcqb", "summary": "This paper introduces L2ight, a framework for mapping a pre-trained machine learning model to an optical neural network implemented with MZIs. The framework also supports on-chip learning: this is done by fine-tuning the implemented weights to compensate for manufacturing imperfections in the chip, or to perform transfer learning from one domain to another.  \nTo do so, L2ight first performs an identity calibration step, then map the pre-trained weights, and finally uses subspace learning to enable efficient model fine-tuning to a specific task or to a specific chip. The paper provides extensive empirical evidence motivating this approach, focusing on compute vision applications. It is able to deal with architectures orders of magnitudes larger than the current state-of-the-art.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*For ease of answering, I have annotated my various points with O.1/Q.1/etc...*\n\nThis is overall a solid paper, introducing a framework significantly improving the state-of-the-art, and with well thought experiments. The paper is very dense, but the authors have made a noticeable effort to make it clear. \n\nOne significant flaw however sticks out: the lack of discussion of the limitations of the framework proposed. This discussion is not optional, and is an integral part of a good scientific study. Currently, it is completely missing, even in a very short form. This leads to some part of the paper being borderline misleading: no actual training from scratch is ever performed in the experiments, and none of the experiments are on actual hardware.\n\n**Because of the absence of a discussion around limitations of the method, this paper is marginally below the acceptance treshold (5)**. However, if such a discussion of limitations is added during the rebuttal, I would be willing to increase my review to an accept or a clear accept (7/8). \n\n### Originality\n\nThis paper builds upon a body of work on optical neural network, and more specifically on the question of mapping a pre-trained network to the weights implemented on the chip by MZIs. It is well placed in the literature, and makes a clear, novel contribution. \n\n### Quality\n\nThe simulated experiments provided are very thorough and all concrete claims are well supported. \n\nHowever, limitations of the study are not outlined at all. \n**Q.1**: One significant issue is that all experiments are limited to simulations, and the scheme is never implemented on a real optical chip. In itself, this is acceptable: this is cutting-edge hardware, which may be difficult to source and use. However, this is never explicitly mentioned in the paper. This is a significant limitation that needs to be discussed, in particular as inaccurate simulations could potentially worsen results (this is mentioned by the authors for other frameworks, l33).  \n\n**Q.2**: A small issue is that the various methods introduced in section 3 lacks any indication of scaling. It's not clear at which scale (size of matrices) each experiment is performed, and it's not clear how the compute/time cost scales with the size of the matrix. Explicitly introducing an O(n/log(n)/n^2) quantization of the cost of the different steps would be interesting, even if the scaling is only evaluated empirically. This could be added to the supplementary.\n\n### Clarity\n\nDespite being quite dense, the writing is always clear, and the authors have made a welcome effort of using bold/italics to highlight and structure information. The structure is easy to follow and provides a good narrative. \n\nHowever, once again, the lack of discussion around the limitations of the proposed method hurt the paper.\n**C.1**: Some of the sentences in the paper reads more like a marketing copy than a scientific paper. \"unleashing the power of optics for real on-chip intelligence\" (l12), \"pushes this emerging field from intractable to scalable and further to efficient for next-generation self-learnable photonic neural chips\" (l17) are two such significant claims made in the abstract, which are never backed in the paper. This is worsened by the absence of any discussion of the limitations of the approach, which is unacceptable for a conference paper. \n\n**C.2**: I feel like the concept of \"on-chip learning\" needs to be clarified. Although a person having ordinary skills in the art of ONNs may understand what is meant, this is not the case for the more general machine learning oriented audience of NeurIPS. In particular, \"on-chip learning\" brings expectations that: a) learning (backward and optimization) operations are actually performed by the chip; b) neural networks are learned from scratch, with randomly initialized weights. Here, what the authors propose is more akin to fine-tuning: a pre-trained model is fine-tuned from its trained configuration to better fit the imperfections of the chip, and potentially to transfer to another domain (section 4.3.2). Currently, none of the experiments provided prove that learning can occur from scratch. This should be clearly indicated. \n\nThe following two points are small recommendations: they have not influenced my scoring of the paper. \n**C.3**: The paper is very dense, and so are the figures. Because they are quite small, they can be really challenging to parse sometimes. I feel like they sometimes contain unnecessary information (such as the DaDiannao ASIC in Figure 1.a), which is never mentioned/discussed in the main text). However, the authors have clearly made an effort to make the figures more readable with annotations, so I do not hold this point strongly against them. \n\n**C.4**: The paper could use a final proof-reading pass. There are a number of typos, such as  \"*the* optical neural networks (ONNs)\" (l25), \"Recently, on-device training *becomes*\" (l36), and repeated equations l117 and l119. Once again, this is fairly minor and I do not account for it in my current score, expecting that this will be fixed in the rebuttal phase.\n\n### Significance\n\nL2ight is a significant improvement over the state-of-the-art, building upon and improving previous works clearly. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As I mentioned above, the authors currently does not discuss at all the limitations of their work. There needs to be a section or a discussion in the conclusion doing so, in particular regarding: (1) transfer to real hardware; (2) use of pre-trained networks only, with no demonstrated ability to train from scratch. This could also be done by more clearly framing the motivations for this work initially. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a calibration and training method for optical neural networks, a possible alternative to the curent implementation in silico. The main topic of the paper is therefore on hardware, and optical implementation of machine learning.  The presented approach here is suited for Mach Zhender interferometers arrays. The framework supports on-chip learning by fine-tuning the implemented weights to compensate for manufacturing imperfections in the chip.

The consensus among reviewer is that this was a very original paper, discussing new direction on the hardware implementation of machine learning algorithm that could be important in the future. The paper was found to be well organised,  and discuss well the state of the art. The new proposed method was found to be clearly described, and the performances to be well backed through simulations. Expert reviewers in both optics and ML assed that the work was interesting and promising.

In the initial round of reviewal: the lack of discussion of the limitations of the framework proposed was pointed out. After rebuttal and discussion between the authors and the reviewers, this was however addressed to the satisfaction of the reviewer.

In conclusion, the proposed approach, L2ight, was thus found to be a significant improvement over the state-of-the-art, building upon and improving previous works clearly. Given the importance of the topic of hardware implementation of machine learning, the area chair thus recommend acceptance to Neurips.




Decision: Accept (Poster)