# A Review on AI for Smart Manufacturing: Deep Learning Challenges and Solutions
## 2 Deep Learning Overview
### Deep Neural Network Architectures
Therefore, encoders are capable of capturing the key information that is sufficient for data reconstruction. However, AEs extract the features in an unstructured way since the latent values are automatically learned by the network and sparsely connected. [PERSON] et al. hence propose variational autoencoders (VAEs), assuming that the data samples follow underlying probabilistic distributions, i.e., Gaussian [45]. VAEs are trained to find the parameters of these Gaussian distributions that can be utilized to exploit the statistical relations between the datasets. Meanwhile, each sample from latent distributions can be reconstructed into meaningful plain data (see [45] for details). Samples from the latent distributions can be reconstructed into novel inputs, which are known as _generative models_. In addition to VAEs, generative adversarial networks (GAN) are another class of generative models. As shown in Figure 7, a GAN consists of two parts, a generator DNN and a discriminator DNN. The two parts are trained in an adversarial way: the generator is encouraged to produce fake samples that resemble real samples as much as possible, whereas the discriminator is encouraged to detect fake ones. The training stops when both are optimized. The generator will finally be capable of imitating the statistical distribution of the real data and create new realistic samples. Both encoder/decoder and generator/discriminator can be CNNs, RNNs, or hybrid. Figure 5: (a) A toy CNN example. Only neighboring neurons are connected to the neuron in the next layer (highlighted in red), whereas all neurons in the previous layer are connected to the next layer neuron in fully-connected layers (highlighted in green). (b) A simplified example of an RNN cell. The network inputs are the combination of history outputs (indicated using arrows) and the new input samples (highlighted in red). Parts of the output neurons are network outputs (highlighted in green). Figure 6: Basic topology of an AE. The latent values contain the key information for generating the original inputs. Apart from the above-mentioned network architectures, the _Attention_ mechanism is one of the greatest breakthroughs in the machine learning society, which was firstly proposed for machine translation [46]. Attention allows the neural models to automatically pay more attention to certain parts of inputs that are more relevant to the task outputs. The attention mechanism has been widely applied in industrial applications to improve performance. [PERSON] et al. utilize GRU (an RNN variant) with an attention mechanism for sales forecasting in [47]. [PERSON] et al. designed CNN with attention to machinery prognostics in [48]. We summarize the above-introduced network architectures in Table 2. There are tens of other network architectures proposed, such as deep belief networks or neural Turing machines. A more comprehensive introduction to DNN architectures is available in _The Neural Network Zoo_[49]. Moreover, a successful industrial deep learning application is far more than the right model selection. After our extensive study on the current research and industrial applications of AI in smart manufacturing, we have identified three dominating challenges: data quality, data secrecy and AI safety. These challenges, along with potential solutions will be discussed in the remainder of the paper.