# Moving Deep Learning to the Edge
## 3 Deep Learning
### Deep Neural Network Models  
The design space of deep neural networks is very large and mostly empirically explored. Many networks with a particular number of layers, kernels, and activation functions, among other variables, have been proposed in the last few years with a focus on CNNs. In the following, the structure and complexity of well-known network models are described to establish a relationship between accuracy and computing requirements. One of the first deep neural networks, LeNet, was proposed in [36] with a CNN for hand-digit classification. LeNet-5, the most accurate version, with two convolutional layers, two fully connected layers, and a softmax output layer. The input of LeNet-5 is a \(32\times 32\) grayscale image. The first convolutional layer processes the input image with six kernels with a window of size five and a stride of one. The output of the first layer is therefore six maps of size \(28\times 28\) (there is no padding). Then, an average pooling layer is applied with a size of \(2\times 2\) and a stride of two generating six output maps with a size \(14\times 14\). A second convolutional layer is applied with 16 kernels with a window size of \(5\times 5\) producing 16 maps of size \(10\times 10\). A second identical average pooling is applied to these maps generating 16 maps of size \(5\times 5\). Two fully connected layers are then applied: the first with 120 filters and the second with 84. Finally, a softmax layer is used to classify ten possible output digits. With a total of 60K parameters and 682K operations, LeNet-5 achieved a classification accuracy above 99%, higher than the accuracy of any other previous machine learning algorithm. Larger convolutional neural networks for image classification were proposed a decade latter with AlexNet [42]. While LeNet-5 was designed for small images of size \(32\times 32\), AlexNet was designed for image classification of images of size \(227\times 227\). The network has five convolutional layers, three fully connected, and three max pooling layers. The first convolutional layer applies 96 kernels of size \(11\times 11\times 3\) to the input image with a stride of four. The first max pooling is applied to the output of this first convolutional layer. The second convolutional layer applies 256 kernels of size \(5\times 5\times 48\) followed by another overlapping max pooling with size \(3\times 3\) and a stride of two. The next three convolutional layers apply 384, 384, and 256 kernels of size \(3\times 3\times 256\), \(3\times 3\times 192\), and \(3\times 3\times 192\), respectively. A final max pooling is applied to the last convolutional layer. The first two fully connected layers have 4096 neurons each, and the last is a softmax layer that generates the probability distributions over 1000 classes. A large proportion of the computing operations are spent on the convolutional layers, while most of the parameters are in the fully connected layers. The number of weights and operations rose to 61 million and 1448 million operations. AlexNet design and tuning is en empirical method without sustained design guidance. In 2013, a multilayer deconvolutional neural network was proposed, known as ZfNet [43]. The model consisted of a deconvolution process to monitor the activity of neurons. When applied to AlexNet, this process permitted identifying active and inactive neurons. From this analysis, the convolutional layers were modified to improve the accuracy. ZfNet permitted fine-tuning AlexNet to achieve a top five error rate of 11.7%. AlexNet was followed by deeper networks as a way to improve accuracy. VGG-16 [44] is a CNN with 16 layers. Three of them are fully connected. The first two layers apply 64 filters of size \(3\times 3\) followed by max pooling. The next two convolutional layers apply 128 filters with the same window size followed by max pooling. The next three convolutional layers apply 256 filters of size \(3\times 3\) and \(1\times 1\) followed by max pooling.