Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces L2ight, a framework for mapping a pre-trained machine learning model to an optical neural network implemented with MZIs. The framework also supports on-chip learning: this is done by fine-tuning the implemented weights to compensate for manufacturing imperfections in the chip, or to perform transfer learning from one domain to another.  
To do so, L2ight first performs an identity calibration step, then map the pre-trained weights, and finally uses subspace learning to enable efficient model fine-tuning to a specific task or to a specific chip. The paper provides extensive empirical evidence motivating this approach, focusing on compute vision applications. It is able to deal with architectures orders of magnitudes larger than the current state-of-the-art.
main_review: *For ease of answering, I have annotated my various points with O.1/Q.1/etc...*

This is overall a solid paper, introducing a framework significantly improving the state-of-the-art, and with well thought experiments. The paper is very dense, but the authors have made a noticeable effort to make it clear. 

One significant flaw however sticks out: the lack of discussion of the limitations of the framework proposed. This discussion is not optional, and is an integral part of a good scientific study. Currently, it is completely missing, even in a very short form. This leads to some part of the paper being borderline misleading: no actual training from scratch is ever performed in the experiments, and none of the experiments are on actual hardware.

**Because of the absence of a discussion around limitations of the method, this paper is marginally below the acceptance treshold (5)**. However, if such a discussion of limitations is added during the rebuttal, I would be willing to increase my review to an accept or a clear accept (7/8). 

### Originality

This paper builds upon a body of work on optical neural network, and more specifically on the question of mapping a pre-trained network to the weights implemented on the chip by MZIs. It is well placed in the literature, and makes a clear, novel contribution. 

### Quality

The simulated experiments provided are very thorough and all concrete claims are well supported. 

However, limitations of the study are not outlined at all. 
**Q.1**: One significant issue is that all experiments are limited to simulations, and the scheme is never implemented on a real optical chip. In itself, this is acceptable: this is cutting-edge hardware, which may be difficult to source and use. However, this is never explicitly mentioned in the paper. This is a significant limitation that needs to be discussed, in particular as inaccurate simulations could potentially worsen results (this is mentioned by the authors for other frameworks, l33).  

**Q.2**: A small issue is that the various methods introduced in section 3 lacks any indication of scaling. It's not clear at which scale (size of matrices) each experiment is performed, and it's not clear how the compute/time cost scales with the size of the matrix. Explicitly introducing an O(n/log(n)/n^2) quantization of the cost of the different steps would be interesting, even if the scaling is only evaluated empirically. This could be added to the supplementary.

### Clarity

Despite being quite dense, the writing is always clear, and the authors have made a welcome effort of using bold/italics to highlight and structure information. The structure is easy to follow and provides a good narrative. 

However, once again, the lack of discussion around the limitations of the proposed method hurt the paper.
**C.1**: Some of the sentences in the paper reads more like a marketing copy than a scientific paper. "unleashing the power of optics for real on-chip intelligence" (l12), "pushes this emerging field from intractable to scalable and further to efficient for next-generation self-learnable photonic neural chips" (l17) are two such significant claims made in the abstract, which are never backed in the paper. This is worsened by the absence of any discussion of the limitations of the approach, which is unacceptable for a conference paper. 

**C.2**: I feel like the concept of "on-chip learning" needs to be clarified. Although a person having ordinary skills in the art of ONNs may understand what is meant, this is not the case for the more general machine learning oriented audience of NeurIPS. In particular, "on-chip learning" brings expectations that: a) learning (backward and optimization) operations are actually performed by the chip; b) neural networks are learned from scratch, with randomly initialized weights. Here, what the authors propose is more akin to fine-tuning: a pre-trained model is fine-tuned from its trained configuration to better fit the imperfections of the chip, and potentially to transfer to another domain (section 4.3.2). Currently, none of the experiments provided prove that learning can occur from scratch. This should be clearly indicated. 

The following two points are small recommendations: they have not influenced my scoring of the paper. 
**C.3**: The paper is very dense, and so are the figures. Because they are quite small, they can be really challenging to parse sometimes. I feel like they sometimes contain unnecessary information (such as the DaDiannao ASIC in Figure 1.a), which is never mentioned/discussed in the main text). However, the authors have clearly made an effort to make the figures more readable with annotations, so I do not hold this point strongly against them. 

**C.4**: The paper could use a final proof-reading pass. There are a number of typos, such as  "*the* optical neural networks (ONNs)" (l25), "Recently, on-device training *becomes*" (l36), and repeated equations l117 and l119. Once again, this is fairly minor and I do not account for it in my current score, expecting that this will be fixed in the rebuttal phase.

### Significance

L2ight is a significant improvement over the state-of-the-art, building upon and improving previous works clearly. 

limitations_and_societal_impact: As I mentioned above, the authors currently does not discuss at all the limitations of their work. There needs to be a section or a discussion in the conclusion doing so, in particular regarding: (1) transfer to real hardware; (2) use of pre-trained networks only, with no demonstrated ability to train from scratch. This could also be done by more clearly framing the motivations for this work initially.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes a calibration and training method for optical neural networks, and more precisely suited for Mach Zhender interferometers arrays. The method is claimed to be scalable, and based on easily accessible optical quantities on chip. 

main_review: The paper is overall well organized, and compared with the state of the art. The method is well described, and the performances are well backed through simulations of a 9x9 photonic tensor core. The results are evaluated in terms of efficiency, memory and number of step reduction. 

The significance is potentially high, optical implementations of ONN using MZI remains highly dependent on the efficient tuning of the interferometer to a given transformation and/or to direct training, and subject to many sources of imperfection. All these aspects are taken into account and the algorithm appears to give a satisfactory answer to all these potential pitfalls at once. The method is also quite interesting, based on injecting the circuit from both sides iteratively to benefit from its time reversal symmetry.

limitations_and_societal_impact: 
What is not fully clear is the scaling. The paper claims up to 10M parameters, but remains restricted to 9x9 MZI arrays, if I understand well. This means that only relatively small unitaries are performed. I would really have liked to see a study of the scaling with respect to the size of the array, which is a major issue in future ONN. Another issue that could have been detailled is the WDM : how does the algorithms deals with non-idealities of the circuit with respect to multiple wavelength travelling through the same circuit and sensing a slightly different matrix. 

Potential negative social impact: None, as this is low level hardware training.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Optical Neural Networks (ONN) with their attojoule per Multiply Accumulate energy efficiency and sub-nanosecond latency are becoming useful implementation hardware for large scale deep learning models and datasets. Despite their efficiency and speed, ONNs suffer significant loss in accuracy due to manufacturing defects/non-ideal device controls and circuit noises. Simulating these non-idealities during software training is not scalable. 

Training ONN on-chip is a hard problem that is exacerbated due to above factors. This paper tackles the problem by proposing a three-stage learning framework L2ight. The three stages are variation-agnostic calibration, alternate-projection based parallel mapping and multi-level sparse subspace learning. 

L2ight enables scaling up on-chip ONN training, for the first time to models with 10M params (1000x scalable than baseline). This scalability is built on top of an efficient training algorithm utilizing multi-level sparsity and a restricted subspace optimization step which enables sufficient adaptability for on-device learning and task transfer. L2ight also achieves 30x higher efficiency than prior art. Lastly, the subspace learning step is agnostic to device/process variation and mapping noise enabling robust implementation of deep neural networks (DNN).   

main_review: The paper presents a series of novel strategies at different stages of implementing a pre-trained DNN onto an ONN hardware followed by training for same/different task as pretrained. Optimal Singular Value Projection and batch break down of the mapping problem helps achieve fast and high accuracy mapping of pretrained/from scratch weights on to the ONN.  In-situ subspace gradient acquisition exploits reciprocity in optics for minimizing the cost of expensive gradient calculation. Ablation studies demonstrate the robustness of subspace learning step against device/mapping stage errors. Further, unique sampling strategies such as balanced topk and column sampling further help lower the number of hardware operations without cascading gradient variance across layers. Lastly, the paper also utilizes mini batch dropping (SMD) to improve training efficiency. 

Section 4.1, paper describes the usage of number of PTC calls as the normalized energy indicator. Similarly, longest accumulation path is assumed to be the normalized latency/runtime indicator. Authors present a detailed breakdown of the energy/latency calculation in appendix based on the parameters of a given convolution layer. While these metrics are surely key factors impacting the energy consumption and latency of a DNN running on the ONN. It’s not clear that when accounting for other electrical components highlighted in Figure 3 (e.g., EO/GLB/Interconnects) if these factors will dominate the ONN energy and latency. Specifically, as highlighted in Section 3.4.1, part of gradient calculation (element-wise multiplication) is outsourced to electrical control units, is it obvious that the energy/latency cost of this outsourcing is negligible? It would be appreciated if the authors can justify this. 

Table 2 shows a comparison of normalized PTC Energy and the normalized number of Steps, the ratios for both metrics are tightly aligned with a couple of exceptions, this goes back to my argument that while the proposed techniques help reduce the number of steps (PTC passes) it’s unclear if a hardware ONN system will receive energy efficiency benefits of the same order (30x). Similar observation holds true for Figure 13 with identical energy/step value variation. Consider consolidating this reduction of steps and normalized energy into a single parameter, e.g., no. of PTC operations, for which considerable reduction is demonstrated. 

Section 4.2 and Figure 10 discuss a comparison of L2ight with prior work and demonstrates its superiority. A few clarifications about this data will be helpful, are the reported accuracy numbers the best achievable accuracy without any restrictions on number of training steps? Stats like the number of training steps/wall clock time for the different data points on Figure 10 would be helpful.

 Additionally, I would appreciate if the authors could clarify the following question. Where does the 1000x improvement in scalability come from? Is it simply from the batching of the optimization problem or is it due to efficient gradient calculation as described in Section 3.4.1? Also is there something unique about the L2ight flow that allows the demonstrated batching, or the batching could be applied to MixedTrn prior work to get similar scalability improvements? 

The paper is well organized, and the detailed figures provide useful insights. 

Typos and minor errors: 
1.	Line 36, on-device training “has become/is becoming” an appealing trend towards …
2.	Line 60, 30x more efficient than prior “art”.
3.	Line 66, pi notation should have i=2 as starting and n as the max value. The same inversion is used in later equations representing U(n) in the paper and appendix. 
4.	Figure 2: Stage 3 should have an accuracy check before finishing, it might be performed in block 3 (Update Σ), consider listing it out separately. 
5.	Figure 9: Column sampling figure, the arrows start from x12 instead of x22 as in spatial sampling.
6.	Figure 14b: Legend appears wrong as it lists dataset name instead of DNN name. 

limitations_and_societal_impact: The paper does not include any actual hardware measurements, it would be helpful if the authors specify it at some point as a limitation.
review_title: null

Meta-review: The paper proposes a calibration and training method for optical neural networks, a possible alternative to the curent implementation in silico. The main topic of the paper is therefore on hardware, and optical implementation of machine learning.  The presented approach here is suited for Mach Zhender interferometers arrays. The framework supports on-chip learning by fine-tuning the implemented weights to compensate for manufacturing imperfections in the chip.

The consensus among reviewer is that this was a very original paper, discussing new direction on the hardware implementation of machine learning algorithm that could be important in the future. The paper was found to be well organised,  and discuss well the state of the art. The new proposed method was found to be clearly described, and the performances to be well backed through simulations. Expert reviewers in both optics and ML assed that the work was interesting and promising.

In the initial round of reviewal: the lack of discussion of the limitations of the framework proposed was pointed out. After rebuttal and discussion between the authors and the reviewers, this was however addressed to the satisfaction of the reviewer.

In conclusion, the proposed approach, L2ight, was thus found to be a significant improvement over the state-of-the-art, building upon and improving previous works clearly. Given the importance of the topic of hardware implementation of machine learning, the area chair thus recommend acceptance to Neurips.




Decision: Accept (Poster)