# A Review on AI for Smart Manufacturing: Deep Learning Challenges and Solutions
## 2 Deep Learning Overview
### Deep Neural Network Architectures  
According to the _universal approximation theorem_, a shallow neural network consisting of infinite neurons can approximate any continuous functions [39]. Therefore, shallow NNs are theoretically powerful enough to model any datasets. However, conventionally, their inputs are manually annotated features instead of correlated and redundant raw data for successful and efficient learning [40]. Features are pieces of attributes or encodings of raw data that are more outstanding for solving the learning tasks, e.g., the object edges in images are better features for human detection compared with raw pixel values. However, the learning tasks and data are becoming more and more complex. Manual feature annotation is hence less and less effective. Moreover, shallow networks are poor at generalization with raw complex data and slow to be trained [41]. Thankfully, DNNs can automatically learn hierarchical features that are generic and reusable among various data samples. Hence, DL is known for feature learning or representation learning in contrast to manual feature engineering. Thanks to the rapid improvements to the computational power in recent decades, the deeper and more complicated neural networks can be widely used. DNN architectures refer to the number of neurons, their connection topology as well as their intended learning tasks. Numerous DNN architectures have been invented in recent decades for more efficient and effective feature learning. The success of deep learning started from supervised learning, especially image processing tasks, with the invention of convolutional neural networks (CNNs) [42]. CNNs have convolutional layers (see Figure 5a), which are derived from the convolution operators in traditional signal processing. Convolution kernels are good at exploiting the relations between neighboring input signals and learning spatial-invariant features that might be distinguished data representations (salient features). They are also more computationally efficient than fully-connected layers and specifically used for image processing tasks, such as visual inspection [10]. In order to learn the features describing the relations between successive input signals, recurrent neural networks (RNN) were proposed, which are characterized by the use of RNN cells [43]. Unlike feedforward neural networks, in which the signals are propagated sequentially without any loops, i.e., revisiting previous data samples, RNN cells compute outputs based on current and previous inputs in order to learn temporal-invariant features, as shown in Figure 5b. At each time step, the new data samples combined with the outputs from the last time step are fed into the network. RNNs are usually used for time series analysis, such as in anomaly detection for quality assurance with continuous sensory measurements [44]. CNNs and RNNs are _layer-level architectures_ that can be combined with other layer-level operators, e.g., fully-connected layers. For example, multiple layers of RNNs or CNNs can be stacked to learn multi-level spatial- or temporary-invariant features, respectively [7]. DNN architectures are defined in such layer-wise stacking manner for different tasks and data modalities. In order to exploit the internal connections of the complex datasets, DNNs are also exploited for unsupervised learning due to their distinguished feature learning abilities. Auto-encoders (AEs) are a type of representative techniques for dimension reduction. As shown in Figure 6, an AE consists of two different DNNs, the encoder and the decoder. The encoder maps the inputs to a space of lower dimensions, the so-called _latent space_. The latent values describe the original data source more efficiently than the actual raw data produced and, hence, can be used as a general representation for all data samples. During training, the loss function is defined to encourage the decoder to recover latent space data back to the original inputs as well as possible.