# A Review on AI for Smart Manufacturing: Deep Learning Challenges and Solutions
## 2 Deep Learning Overview
### Model Training  
During neural network training, the key objective is to minimize the so-called _loss function_ through an appropriate choice of the model parameters. Loss functions are usually defined according to the desired goals to achieve through the neural networks, such as the discrepancies between the predictions and labels for supervised classification, and to indicate how well the neural networks perform the tasks. Additionally, the loss functions can include the numerical regularization on the neural network parameters, such as weight sparsity. However, loss functions are usually non-convex and may hold many local minima, as depicted in Figure 3. Moreover, DNNs usually hold thousands to millions of parameters. To search for a globally minimized loss analytically is, therefore, virtually unsolvable. Hence, DNNs are usually trained in an iterative fashion based on gradient descent methods. During each iteration, the parameters are updated slightly towards the direction that  
Figure 2: (**a**) The neuron output \(y\) is the weighted sum of the elements in input vector \(x\) and bias \(b\) followed by a non-linear activation function \(\varphi(\cdot)\). Activation functions are normally _Sigmoid_, _Tanh_, or _ReLU_[38]. (**b**) A toy neural network with three layers, in which neuron outputs pass in the forward direction indicated by the blue arrows. \(\omega_{n}\) are the weights. the loss drops fastest, which is represented by the partial derivative with respect to each parameter, as given in (1). Vector \(\mathbf{p}\) holds the DNN parameters, and \(\mathbf{p}^{\prime}\) stands for the updated parameters after each iteration. \(\alpha\) is the _learning rate_ that adjusts the step size for each update, and \(\mathbf{L(p)}\) is the loss function, which depends on the parameters. \[\mathbf{p}^{\prime}=\mathbf{p}-\alpha\bigtriangledown\mathbf{L(p)} \tag{1}\]  
To compute the gradients, the data samples are firstly forward-passed to obtain the outputs, i.e., similar to inference, and then the loss values and gradients are computed. Due to the layer-wise structure of DNNs and the large number of parameters to optimize, the gradients are normally efficiently computed using the chain rule and back-propagation strategies. The gradients in the last layer can be directly computed since they are spatially closest to the outputs and loss functions, as illustrated in Figure 4. The gradients in the earlier layers are then derived in sequence layer-by-layer based on the gradients computed in later layers. The entire dataset is normally used for training a neural network. However, how to assign the data samples for each parameter update is a problem. One sample per update is burdensome and fluctuates the training processes. On the other hand, using the entire dataset at once for each update, i.e., _batch training_, can be highly computationally intensive. Hence, the training data are usually divided into _mini-batches_, i.e., a smaller portion of  
Figure 4: Graphical illustration of back-propagation with a three-layer fully connected neural network as an example. \(\mathbf{x}=[x_{1},x_{2}]\) are the inputs, and \(\mathbf{y}=[y_{1},y_{2}]\) are the outputs. \(\mathbf{\omega}=[\omega_{1},\omega_{2},\ldots]\) are the weights, and \(\mathbf{b}=[b_{1},b_{2},b_{3},b_{4}]\) are the biases. \(\mathbf{L(y)}\) is the loss function. \(\mathbf{z}=[z_{1},z_{2},z_{3},z_{4}]\) are the intermediate neuron inputs, for example \(z_{3}=\omega_{3}\cdot\sigma(z_{1})+\omega_{4}\cdot\sigma(z_{2})+b_{3}\). In addition, \(\sigma(\cdot)\) is the activation function. Normally, the gradients are calculated from the last to the first layer.