# A Survey of Convolutional Neural Networks on Edge with Reconfigurable Computing
## 2 Convolutional Neural Networks
### Recent Models of CNNs  
Many different CNN models have been proposed with different number and type of layers, that is, with different memory requirements and computational complexity since the success of LeNet [12]. LeNet was the first convolutional neural network to achieve a high accuracy for hand digit classification and robustness to rotation, distortion, and scaling. LeNet has three convolutional layers, two fully connected, and a softmax classifier. The model uses the hyperbolic tangent for the activation function. The network has 60 K parameters and obtained an accuracy above 99% in the classification of digits represented with black and white pictures of size \(32\times 32\). The first large CNN model proposed to classify color images with a size of \(227\times 227\times 3\) was AlexNet ([PERSON], 2012). It has five convolutional layers, some followed by pooling layers and three fully connected layers in a total of 60 M weights, \(1000\times\) more than LeNet. The activation function adopted by AlexNet was ReLU that permitted the improvement of the convergence rate of learning and the problem of the vanishing gradient [13]. With over 700 million MACC operations per image, AlexNet won the 2012 ILSVRC (ImageNet Large-Scale Visual Recognition Challenge) with a top-5 error rate of 17.0% (top-5 is the error rate at which given an image the network does not include the correct label within its top five predictions) and a top-1 error rate of 37.5% with ImageNet. The trial-and-error method of designing CNN models led to the work of [14]. The authors proposed ZefNet, a CNN model designed from the analysis of the outputs of each layer. The process consisted of a deconvolution process that permitted to observe the activity of neurons of AlexNet and from this to establish correlations between features and input neurons. The idea was already used in Deep Belief Networks [15]. With the deconvolution method, the authors were able to fine tune AlexNet to achieve a top-5 error rate of 11.2%. AlexNet and ZefNet use filters with different sizes at different layers, which complicates software and hardware implementations. In 2014, VGG [16] was proposed as a CNN model driven by uniformity of layers. To compensate for any loss accuracy caused by this uniformity, the depth of the VGG model was increased. A total of six different architectures were tested and the one with best accuracy was VGG-16 with 16 layers using only \(3\times 3\) filters with a stride and pad of 1, and \(2\times 2\) pooling with a stride of 2. An important fact was also published with VGG, stating that two or three convolutional layers with \(3\times 3\) filters has the same results of a single layer with \(5\times 5\) or \(7\times 7\) filters respectively. The advantage of this serialization is the reduction of the number of parameters and the number of operations. VGG achieved a top-5 error rate of 7.3% with 138 million weights. A common feature of previous CNN models is the utilization of regular layers explained in the previous section. In 2014, GoogleNet or Inception-v1 [17] network model considered a different layer type: The inception layer. The inception layer runs several convolutions in parallel, namely three convolutions with different sizes (\(1\times 1\), \(3\times 3\), and \(5\times 5\)) and one \(3\times 3\) max pooling. To reduce the number of operations, the non unitary convolutions are preceded by unitary convolutions to reduce the number of weights (see Figure 4). The complete model has nine inception layers without any dense layers. An average pool is used in the last layer. With a total of 6.8 million weights, GoogleNet achieved a top-5 error rate of 6.7% for ImageNet. The original version of Inception-v1 was modified into inception-v2, inception-v3, and inception-v4 [18]. Following an idea of VGG, \(5\times 5\) convolutions were replaced by two serial \(3\times 3\) convolutions.