# Optical Convolutional Neural Networks: Methodology and Advances (Invited)
## 2 Development of Optical Convolution Neural Network
### Optical CNN Based on Optical Diffraction  
Optical computing with free-space optical diffraction has been demonstrated over decades [60; 61]. The basic scheme is the Fourier transform property of the lens [62]. When coherent light passes through the lens, a two-dimensional (2D) Fourier transform is performed without power consumption, and the focal plane behind the lens (which is also called the Fourier plane) presents the result of the Fourier transform. Since the convolution operation in the spatial domain is able to transform into a multiplication operation in the Fourier domain following the convolution theorem [63], the convolution operation is implemented with a \"\(4F\) system\" by adding a specially designed mask to the Fourier plane of the first lens. As shown in Figure 2, the noted \"\(4F\) system\" is composed of two lenses placed coaxially with a distance of \(2F\), and the distance between the input plane and output plane is \(4F\). The Fourier transform of the input image \(g(x,y)\) is accomplished in the focal plane of the first lens  
\[G(u,v)=\mathcal{F}\{g(x,y)\}, \tag{2}\]  
where \(\mathcal{F}\) represents the Fourier transform and \(u,v\) are the coordinates in the frequency domain. In the Fourier plane, the specially designed mask \(M(u,v)\) introduces additional amplitude and phase modulation,  
\[G^{\prime}(u,v)=G(u,v)M(u,v). \tag{3}\]  
Then, the inverse Fourier transform is realized with the second lens, which can be expressed as  
\[ \begin{array}{ll}g_{o}(x^{\prime},y^{\prime})&=\mathcal{F}^{-1}\{G^{\prime}( u,v)\}\\ &=\mathcal{F}^{-1}\{G(u,v)M(u,v)\}\\ &=g(x,y)*m(x,y),\end{array}  \tag{4}\]  
where \(\mathcal{F}^{-1}\) is the inversive Fourier transform, \(m(x,y)\) is the inverse Fourier transform of \(M(u,v)\), and \(*\) represents the convolution operation. From Equation (4), the convolution operation of the input image is simultaneously achieved when passing through the \"\(4F\) system\", and then the feature maps are recorded by the camera sensor and converted into digital electrical signals to conduct nonlinear activation, pooling, and final recognition in the fully connected layer. The CNN on optical diffraction was proposed and experimentally demonstrated in 2018 [64], and the schematic of the hybrid optoelectronic CNN as well as the experimental setup is shown in Figure 3a. The optical convolution operation was accomplished using the diffractive optical element (DOE) as the phase mask placed in the Fourier plane of the \"\(4F\) system\". The optical feature image was converted into a digital electrical signal with a camera sensor, and the processes in terms of nonlinearity activation and full connection were still addressed in the electrical domain. The parallel convolution of multiple kernels was realized via the optimization design of the phase mask. The experimental accuracy reached 44% on the grayscale CIFAR-10 dataset, which improved by approximately 50% relative to the single digital fully connected layer (accuracy of 30%). Apart from the diffractive optical element, the optical metasurface as the phase mask offered another approach using multiple physical lays or wavelength-sensitive material to conduct multichannel free-space convolution [65; 66; 67; 68]. An amplitude-only weighting optical CNN in the Fourier plane was proposed in 2020 using a digital micromirror device (DMD) as a reconfigurable amplitude modulation device [69]. A typical schematic representation of a \"\(4F\) system\" on digital micromirror devices is shown in Figure 3b. Benefiting from the large-scale and high-speed properties of digital micromirror devices, classification tasks on two megapixel matrices at 10 kHz rates were performed. The classification accuracies were up to 98% on the MNIST database and 54% on the CIFAR-10 database. After that, multiple kernel parallel convolution operations based on the optical diffraction orders (Figure 3c) were realized by the same research group [70].