## 3 Deep Neural Networks Assisted Nanophotonics Design for Silicon Platform
### Training Discriminative Neural Networks as Forward Models
#### 3.1.1 Multi-Layer Perceptron  
The origin of MLP dates back to 1989 when [PERSON] et al. proposed a feed-forward network with a single hidden layer containing a finite number of neurons. Under mild assumptions on the activation function, this neural network could approximate almost any continuous function with enough training data samples [104]. We take [85] for example to illustrate the process of constructing and training an MLP to surrogate EM simulation of grating couplers. As shown in Figure 6, the geometric parameters (grating pitch, duty cycle, fill factor, fiber angle and polarization) are encoded as data feature \(\mathbf{x}=[x_{1},x_{2},x_{3},x_{4},x_{5}]\). Peak efficiencies and corresponding wavelengths for TE and TM modes are marked as data label \(\mathbf{y}=[y_{1},y_{2},y_{3},y_{4}]\). The MLP adopted by [PERSON] et al. consists of one input layer (5 neurons), three hidden layers (100, 50, 50 neurons) and one output layer (4 neurons), where the output value of each non-input neuron is determined by all the neurons in the last layer. Taking the first neuron in the third hidden layer \(x_{1}^{3}\) for example, the weighted summation of neurons in the second hidden layer is shown in Equation (15):  
\[s_{1}^{3}=\sum_{i=1}^{50}w_{i}^{3}x_{i}^{2}+b^{3}, \tag{15}\]  
where \(w_{i}^{3},\ b^{3}\) are the \(i\)th weight and bias in the third hidden layer, respectively. The weighted summation is activated to get the output value of neuron as shown in Equation (16):  
\[x_{1}^{3}=f\Big{(}s_{1}^{3}\Big{)}, \tag{16}\]  
where \(f(\cdot)\) is the activation function which normalizes the output of each neuron to a finite range [0, 1] or [\(-1\), 1]. Activation functions are supposed to be computationally efficient, for there may exist abundant neurons in a DNN [105]. In this example, Rectified Linear Units (ReLu) activation function is adopted, which is expressed as in Equation (17):  
\[f(x)=\max(0,x). \tag{17}\]  
Through this feed-forward propagation, the relationship between input data feature \(\mathbf{x}\) and output label \(\mathbf{y}\) is shown in Equation (18):  
\[\mathbf{y}=f\Big{(}\mathbf{W}^{4}\cdot\mathbf{x}^{3}+\mathbf{b}^{4}\Big{)}=f \Big{(}\mathbf{W}^{4}\cdot\Big{(}\mathbf{W}^{3}\cdot\mathbf{x}^{2}+\mathbf{b }^{3}\Big{)}+\mathbf{b}^{4}\Big{)}=\ldots=F(\mathbf{x};\mathbf{W},\mathbf{b}), \tag{18}\]  
Figure 6: Training of an MLP to predict the coupling efficiency of grating couplers. where \(\mathbf{W}:\left[\mathbf{W}^{1},\mathbf{W}^{2},\mathbf{W}^{3},\mathbf{W}^{4}\right]\) and \(\mathbf{b}:\left[\mathbf{b}^{1},\mathbf{b}^{2},\ \mathbf{b}^{3},\ \mathbf{b}^{4}\right]\) consists of weights and biases in all the layers. These weights and biases are determined by the training of MLP with known data samples, which is accomplished by backpropagation. Firstly, the loss function in the form of mean square error (MSE) is defined as in Equation (19):  
\[L\left(\mathbf{y}^{\mathbf{P}},\mathbf{y}\right)=\frac{1}{4}\sum_{i=1}^{4} \left(y_{i}^{p}-y_{i}\right)^{2}, \tag{19}\]  
where \(y_{i}^{p}\) is the predicted value of \(i\)th output neuron and \(y_{i}\) is the \(i\)th label of a data sample. Apart from MSE, the loss function can be expressed, e.g., in the form of cross entropy loss function [106]. The loss function \(L(\mathbf{y}^{\mathbf{P}},\mathbf{y})\) is optimized to its minimal by gradient descent, like stochastic gradient descent (SGD) or batch-size SGD. After the calculation of gradients for the \(t\)th sample, weights are updated as in Equation (20):  
\[\mathbf{W}^{(t+1)}=\mathbf{W}^{(t)}+\alpha\
abla_{\mathbf{w}^{(t)}}L\left(F \left(\mathbf{x}^{(t)};\mathbf{W}^{(t)}\right),\mathbf{y}^{(t)}\right), \tag{20}\]  
where \(\alpha\) is the learning rate to control the update speed. Bias \(\mathbf{b}\) is updated in a similar way. In the example of reference [85], 9190 data samples are generated via 2D FDTD and separated with a ratio of 85:15 for training and validation.