# Convolutional Neural Networks for Off-Line Writer Identification Based on Simple Graphemes
## 3 Convolutional Neural Network Models for Simple Grapheme Analysis  
The CNNs are capable of automatically extracting the characteristics of images [26], making them suitable for the study of images [27]. The CNN typical architecture is composed in the following way (illustrated in Figure 2):  
* Convolutional Layer: It is a set of convolutional filters which activate the characteristics of the image. * Layer of activation function: It is a non-linear activation function. * Subsampling Layer or pooling layer: It reduces the dimension of the feature banks at the output of the convolutional layer. * Fully connected Layer: It flattens the output of the previous layers by converting the output to 1D. * Softmax Layer: It gives the probabilities of each category as established in the database at the beginning to perform the classification. There are CNNs previously trained for image classification that have learned to extract characteristics and information from the images, thus using them as a starting point to learn a new task. Most of these CNNs were trained using the ImageNet database [28], which is used in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [29]. The three main uses of pre-trained CNNs are shown in Table 1. \begin{table}
\begin{tabular}{l l} \hline \hline
**Purpose** & **Description** \\ \hline Transfer Learning & Fine-Tune on new dataset \\ \hline Feature Extraction & Use of pre-trained network as a features extractor \\ \hline Classification & Apply pre-trained networks directly to classifications problems \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Applications of pre-trained CNNs [17]. Figure 2: Basic architecture of a Convolutional Neural Network (CNN). However, because the original and rectified graphemes are very different from the images included in the Imagenet database, the graphemes cannot be classified directly using the pretrained CNNs. Consequently, a learning transfer process invariably takes place. This process consists of properly adjusting and training the previously trained CNN with the new images. The idea is usually to adjust the CNN output layers keeping the rest of the network unchanged and taking the pre-trained weights. Figure 3 illustrates a simplified diagram of the learning transfer process with pre-trained CNNs. This paper adopts CNN models widely known in the literature:  
* AlexNet [19]: was one of the first deep networks in and a significant step in the development of CNNs. It is composed of 5 convolutional layers followed by 3 fully connected layers. * VGG [20] versions VGG-16 and VGG-19, Developed by the Visual Geometry Group (VGG) of the University of Oxford, it is an AlexNet enhancement by replacinglarge kernel-sized filters with multiple 3 \(\times\) 3 kernel size filters one after another, increasing network depth and thus being able to learn more complex features. * ResNet (ResNet-18) [21], is an innovation over previous architecture, solving many of the problems of deep networks. It uses residual blocks of direct access connections, managing to reduce the number of parameters to be trained, with a good compromise between performance, structural complexity and training time. Table 2 shows the general characteristics of these networks: depth, size of the network, number of parameters and dimension of the input image. Figure 4 shows the architecture of the AlexNet, VGG-16, VGG-19 and ResNet-18 networks. The description of the elements that form the blocks of this figure is as follows:  
* Conv: The size of the convolutional filters. * @: The number of filters to apply. * s: The stride of the filter over the image. * ReLU: The activation function at the output of the convolutional filters
* MaxPool: The subsampling operation with the filter dimension.