## 3 Materials and Methods for Emojis Classification
### Experiments Settings
#### 3.5.2 Deep Classifiers  
We trained three deep classifiers using transfer learning to emoji classification. To such an extent, this section provides an overview of the deep classifiers and the training setup information with different parameters.  
_AlexNet [14]_ is a pre-trained Convolutional Neural Network (CNN) trained with the ImageNet [36] dataset of 1000 object categories. This CNN consists of eight layers, of which the first five are convolutional layers and the last three are fully connected layers. AlexNet requires an input image of [227 227 3]. The output of the last layer, i.e., the softmax layer, produces a distribution over the given categories. AlexNet uses Rectified Linear Units (ReLU).  
_Inceptionv3 [18]_ is a 48-layer deep CNN architecture that assists in image analysis and object recognition. It is pre-trained on the ImageNet [36] dataset requiring an input image of size [299 299 3]. It employs factorized convolutions, as it reduces the number of parameters involved in network training. A 3 \(\times\) 3 convolutional layer is replaced by a 1 \(\times\) 3 convolution followed by a 3 \(\times\) 1 convolution. It is structured in three main blocks: the basic convolution block, the inception, and the classification block.  
_Googlenet [15]_ is a pre-trained 22-layer deep CNN architecture requiring an input image of size [224 224 3]. It is comprised of nine inception modules and contains two max-pooling layers between some inception modules to downsample the input. A dropout layer before the linear layer reduces eventual overfitting.  
_MobileNetV2 [17]_ is a CNN architecture based on inverted residual structure. It is a pre-trained model on the ImageNet [36] dataset requiring an input image of size [224 224 3]. It uses inverted residual blocks with bottlenecking features to bear a lower parameter count and optimize performance on mobile devices.  
_SqueezeNet [16]_ architecture consists of _squeeze_ and _expand_ layers. A squeeze convolutional layer has only a 1 \(\times\) 1 filter. Data are fed into an expand layer, which contains a mixture of 1 \(\times\) 1 and 3 \(\times\) 3 convolutional filters. Between all the squeeze and expand layers, the ReLU activation function is applied, and dropout layers are added to reduce overfitting. The network is already trained on the ImageNet [36] dataset requiring an input image of size [227 227 3].  
Deep Classifiers adopted for emoji recognition using transfer learning can be found in research by [37]. To fine-tune deep neural networks for emoji recognition, we used three different training optimizers, i.e., Adaptive Moment Estimation (adam)[38], Stochastic Gradient Descent with Momentum (sgdm) [33], and Root Mean Square Propagation (rmsprop) [39]. Adam is an extension of stochastic gradient descent that has a small memory footprint and requires only first-order gradients, while sgdm uses stochastic gradient descent with momentum, i.e., a moving average of the gradients is utilized to update the weights. Finally, rmsprop uses an adaptive learning rate rather than specifying it as a hyperparameter. The learning rate is an important hyper-parameter that controls how quickly weights are updated in response to estimated errors, therefore controlling both the time and resources required to train a neural network. Finding an optimal learning rate is usually a tricky and time-consuming task: excessively large learning rates can lead to fast but unstable training and a small value usually results in a long training period and can even become stuck before completing correctly. We trained each deep model by varying the learning rate {0.01, 0.001, 0.0001, 0.00001} with a batch size of 32.