Integrated Photonic Lattice Filter for Accelerating Deep Convolutional Networks

Venue: NeurIPS_cc_2024_Workshop_MLNCP
Published: 2024-09-11 21:51:55

Abstract: The increasing computational demands of machine learning models have driven interest in developing unconventional computing hardware to improve speed and energy efficiency. In this work, we introduce an integrated photonic chip designed to perform convolution operations in deep convolutional neural networks. The convolutional kernel is implemented in the optical circuit, which functions as a two-port lattice filter, by modifying the optical signal paths through phase shifters. Using a simulated model of the optical chip that implements the convolutional layers, we evaluate the performance of a deep convolutional network trained on the CIFAR-10 dataset. We also examine the impact of hardware limitations, such as system noise and quantization, on the model performance.

Main paper: 
 introduction: 
 Convolutional neural networks (CNNs) are fundamental deep learning architectures widely used in computer vision tasks such as image classification, object detection, and video analysis [1, 2] . Although "state-of-the-art" CNN architectures continue to evolve, convolutions remain the primary operation executed by these models. As digital computers near their fundamental energy and speed limitations, there is a significant need for novel hardware capable of supporting the growing computational requirements of deep learning models.Integrated photonics is a promising platform for implementing machine learning systems due to its potential for high throughput and low power consumption [3] . Neural network inference and training have both been demonstrated using photonic integrated chips [4, 5, 6] . Photonic circuits for implementing small-scale CNNs have also been proposed and experimentally validated [7, 8] .Several methods for encoding neural network weights in optical systems have been proposed, including Mach-Zehnder interformeters (MZIs) [9] , microring resonators [10] , and phase-change materials [11] .In this paper, we design and simulate an integrated photonic circuit capable of executing convolution operations for application in deep CNNs. We present a method for implementing 2D multi-channel convolutions using the optical circuit and model the performance of the optical hardware. We simulate a deep CNN based on the VGG-11 model, with convolutional layers implemented using the optical circuit, and train the model on the CIFAR-10 dataset [12, 13] . Finally, we evaluate the impact of phase shifter and measurement noise on the model accuracy, as well as the effect of quantization due to conversion between the analog and digital domains. The optical circuit implements an FIR filter to perform convolution operations. The convolutional kernel is determined by the impulse response of the system, which can be tuned using thermo-optic phase shifters. 
integrated photonic circuit: 
 In this section, we introduce the silicon photonic chip, shown in Fig. 1 , designed to perform convolution operations. The optical circuit implements a finite impulse response (FIR) filter using optical delay lines and tunable MZIs [14] . This design enables the convolution of a time-discrete input signal with a finite convolutional kernel at each time step, separated by an interval of ∆t:EQUATIONHere, x k and z k are the input and output signals at the k-th time step, respectively; h n is the n-th element of the convolutional kernel, and g(•) is the measurement function applied by the photodetection setup. The input optical signals are coherent, which enables both phase and amplitude modulation.The complex-valued kernel elements {h 1 , . . . , h N } are determined by the impulse response of the optical system and can be modified using thermo-optic phase shifters. Depending on the photodetection setup, the output measurement can yield either the complex-valued field or the optical intensity.At each time step, the next input signal is fed into the optical circuit and the output is measured, which is equivalent to shifting the convolutional kernel across the input by one position. This approach allows for the reuse of input signals already propagating through the optical circuit. Thus, N multiply-accumulate operations are performed at each time step for a convolutional kernel consisting of N elements.The optical circuit is composed of a sequence of convolutional units, each containing a delay line and tunable MZI, as illustrated in Fig. 2 . The MZI can be modeled as a two-port device containing two 3 dB directional couplers connected by two waveguides. The relative phase between the two paths can be adjusted using the phase shifter φ, which determines the amount of light directed to the top and bottom output ports. The top output port of the MZI is linked to a phase shifter θ, while the bottom port is connected to a delay line (a folded waveguide) that introduces a propagation delay of ∆t.Using an input signal with frequency ω and treating the optical circuit as lossless, the scattering matrix of the c-th convolutional unit isEQUATIONwhich simplifies toEQUATIONThe impulse response of the system depends on the scattering matrix of each convolutional unit.Specifically, the n-th element of the convolutional kernel, h n , is determined by the superposition of optical paths with a total propagation delay of n∆t resulting from the delay lines. For a circuit composed of N convolutional units, this is given byEQUATIONwhere p represents a path from the set of all optical paths P n∆t with total propagation delay n∆t, andS (c)ip,jp is the (i p , j p ) entry of the scattering matrix defined in Eq. (3).For example, the convolutional kernel elements for a circuit consisting of three convolutional units are defined ash 1 = S (3) 2,1 • S (2) 1,1 • S (1) 1,2 , h 2 = S (3) 2,2 • S (2) 2,1 • S (1) 1,2 + S (3) 2,1 • S (2) 1,2 • S (1) 2,2 , h 3 = S (3) 2,2 • S (2) 2,2 • S (1) 2,2 .(5)An illustration of the optical path required to implement a simple kernel is shown in Fig. 3 .The previously described approach can be generalized to perform 2D multi-channel convolution operations using the optical circuit. In this method, each row of an M × N filter is treated as an independent 1D filter applied sequentially across the image in a series of 1D convolutions. Convolutions across several input channels are executed by processing each channel individually and then summing the outputs at each spatial pixel.In 2D multi-channel convolutions, the kernel is represented as a 4D tensor H, where H i,l,j,k is the kernel element corresponding to the i-th output channel, l-th input channel, j-th spatial row, and k-th spatial column. The 3D input and output tensors are denoted as X and Z, respectively, where Z is the output from convolving H across X:EQUATIONover all valid tensor index values of i, j, and k [15] . As in Eq. (1), g(•) denotes the measurement function applied by the photodetection setup. Similar to the 1D case, the values for H are determined by the impulse response of the optical circuit. The optical path, illustrated in red, required to implement the simple kernel h 1 = 1, h 2 = 0, and h 3 = 0 in a circuit consisting of three convolutional units. In this case, the signal is only delayed by one time step by routing through the final delay line in the third convolutional unit. 
results: 
 Optical Convolutional Layer We simulated a 2D convolutional layer using PyTorch that models the optical chip using Eqs. ( 4) and ( 6) [16] . During training, instead of optimizing the convolutional kernel H of each layer directly, we treat the set of phase shifters {φ 1 , . . . , φ N , θ 1 , . . . , θ N } as learnable parameters. This approach accurately models the optical circuit and accounts for constraints imposed by the hardware.Our model accounts for imperfections in the optical system by injecting noise and quantizing values where necessary. We implement the following features in our simulation:• Phase shifter noise: Gaussian noise with standard deviation σ φ,θ is added to the phase shifter values during each forward pass.• • Input Signal Quantization: The input signal is quantized to N b bits by the digital-toanalog converter.We implemented a deep CNN model in PyTorch using the previously discussed optical convolutional layers. The model, shown in Fig. 4 , is based on the VGG-11 architecture and contains 18.4 million parameters [13] . The Tanh activation function is used in the model to ensure the output values are within a known range. We employed the Adam optimizer with a learning rate that followed a cosine decay schedule from 0.01 over 200 epochs. The model was trained with an augmented CIFAR-10 dataset, using random cropping and horizontal flipping.We trained several CNNs using varying levels of noise added to the phase shifters. This enabled evaluation under different hardware conditions, leading to insights regarding the hardware specifications required to achieve the desired performance. We evaluated the CNNs on the test dataset using quantization levels N b of 8, 6, and 4 bits. The training and validation curves during training are shown in Fig. 5 , and the test results under different quantization conditions are given in Table 1 .Table 1 : Test accuracy (%) achieved by models trained on CIFAR-10 dataset with quantization to N b bits. σ φ,θ /2π N b = 8 N b = 6 N b = 4 2 -σ ϕ,θ /2π 2 -7 2 -6 2 -5 2 -4 2 -3 
discussion: 
 Integrated photonic chips offer several advantages for information processing tasks, including the ability for real-time processing of high-speed data [17] . One of the primary advantages of our proposed photonic circuit is the ability to reuse the input signal over multiple convolution operations.This feature can enhance the system's overall energy efficiency and throughput speed, as a circuit with N convolutional units can perform up to N multiply-accumulate operations at each time step while only feeding in a single input value.The simulation results demonstrate that the photonic circuit is capable of successfully implementing convolutional layers in deep CNNs. As expected, the performance of the model is dependent on the amount of noise present in the system, as well as the level of quantization. The model performance generally improves with reduced system noise and increased bit precision. The CNN trained with the lowest noise (σ φ,θ /2π = 2 -7 ) achieved the highest test accuracy of 89.14% using N b = 8 bits. This performance is similar to the accuracy attained by a model we trained using the same architecture with standard convolutional layers (i.e., floating-point arithmetic), which achieved a test accuracy of 89.65% without quantization.A crucial consideration for the design of analog neural networks is the impact of noise in the system.Several mitigation strategies have been proposed depending on the source of the noise [18] . Additionally, the inclusion of noise during the training process has shown to improve system robustness and reduce the simulation-to-reality gap [19] . As shown in Table 1 , the model accuracy generally decreases as the phase shifter noise increases. However, at low precision of N b = 4 bits, the best performance is achieved by the CNN trained with σ φ,θ /2π = 2 -5 . In this case, we believe the models trained with less noise (2 -6 and 2 In future work, we aim to experimentally demonstrate the use of our photonic integrated chip for implementing a deep CNN. We will use our simulation to determine the optimized values for the phase-shifters, which will be applied to the optical circuit for model evaluation. The simulation can be further improved by incorporating experimental data from the photonic hardware during the training process. This approach, known as "hardware-in-the-loop training," has been shown to mitigate the impact of hardware imperfections on system performance [20] . The application of insitu training, where the training process is performed directly on the photonic chip rather than using a digital computer, can also be explored as a method to improve results. 
 
 Appendix:  



Reviews: review: title: title: This paper presents a silicon photonic circuit for 1D convolutions applied over sequential inputs in discrete-time. The circuit is simulated in software and its parameters are optimized for CNNs trained with pytorch. The effect of noise is studied. Unfortunately, performance estimations of the real hardware are missing.
main_review: main_review: The authors present a novel silicon photonic circuit for 1D convolution over input signals provided in discrete time steps. A photograph of the circuit is shown, but no details about a hardware implementation are given (like chip size etc.). The circuit uses Mach-Zehnder-Interferometers, which is a common approach in neuromorphic photonics. The novelty is the proposal of a convolution circuit unit which, when chained, allows for N-coefficient 1D convolutions. The circuit’s transfer function is theoretically described by means of scattering matrices, its overall functionally is also nicely described in text and illustrative figures.
Next, the convolution circuit and its parameters are integrated into pytorch, and a VGG-11 architecture is trained on CIFAR-10. By this approach, the circuit parameters are directly trained, and no conversion from normal convolutional layers is needed (avoiding any degradation after conversion).
Different noise levels of the phase shifters and different quantization levels of inputs and outputs are studied. The model with lowest noise and 8-bits even achieves a higher test accuracy than the full-precision floating-point reference CNN.
The main contributions of this work are the circuit concept and the methodology for training the CNN with such circuits.
More details about the hardware (technical details, circuit measurements or simulations) and the concept for integration of this circuit into a larger neuromorphic photonic chip with activation functions and multiple layers are missing but would help to understand general usability and effectiveness of the approach.

# quality
The authors present a new neuromorphic photonic circuit for 1D convolutions. This is a very nice concept, especially as it allows for direct training in frameworks such as pytorch. When feeding in a full sequence of inputs, this approach performs the equivalent number of MAC operations like the kernel size. Training results are shown for a standard CNN architecture on CIFAR-10. 

# clarity
The paper is well written in general.

Here are minor suggestions for improvement:
- Fig 1: please annotate delay line, MZI, and phase shifters in the chip photo. Also provide the circuit dimensions
- The capability of the circuit, that you can feed in an L-size input signal and process it in L timesteps to obtain an (L-K+1) size output signal (assuming K is the kernel size), could be better explained. It was at the beginning unclear to me, whether for the 2nd output value, we need to re-feed the input or not.

# originality
The authors provide a novel circuit architecture for 1D convolution. This is accompanied with a direct training of the circuit model parameters, which is expected to be superior to conversion of normally trained convolutions.
significance
The silicon photonic circuit offers a new way to implement 1 D convolutions. Whether this solution offers better performance (speed, energy, robustness to noise) than other solutions will only become clear when a real-chip has been produced and measured. Also, its integrability into a full neuromorphic photonic system has yet to be discussed.

# pros
- authors suggest a novel silicon photonic circuit for 1D convolution
- its functionality is well described in text and in equations
- the circuit’s function is replicated in pytorch and the circuit parameters (phase shifts \theta and \phi) are directly trained
- For a low noise-level the approach achieves accuracies very close to reference CNN with same architecture and full-precision floating point weights. 

# cons
- No concept is presented how this circuit shall be integrated into a larger neuromorphic photonic system, both in terms of structure and in terms of operation (Is the approach cascadable?)
- Suggested concept relies on D/A and A/D converters, which potentially consume a lot of power. Especially in case of 2D convolutions, the digital outputs of the circuit represent partial sums of the Conv2D which need to be accumulated. Hence, digital buffers are needed.
- Details about the silicon photonic circuit missing: size, technology, noise, energy estimation, lowest \Delta_t in practice
- Comparison to other work doing photonic convolutions is very brief. Remaining space of the paper could be used for a more detailed discussion.