# Visual Sentiment Analysis Using Deep Learning Models with Social Media Data
## 4 Methodology
### ResNet50V2 Architecture  
The ResNet (Residual Networks (ResNet) deep neural networks were successfully used to solve image classification problems (ILSVRC 2015), and they outperformed the other deep neural architectures to get first prize in the competition. There are many ResNet architectures available that come up with different numbers of deep layers, such as ResNet-18, ResNet-50, ResNet-101, ResNet-151, and so on. The number that follows ResNet indicates the number of layers available in that network architecture. The residual network's purpose is to solve the problem of vanishing gradients that occurs during the training of a deep neural network with several layers. When the error is backpropagated and the gradient is computed during the training process, the gradient value decreases dramatically once the early layers are reached. The weight updation becomes very modest, to the point where the original weights are nearly identical to the updated weights gained from backpropagation. This adds the network's input to the output of the convolutional block with weight layers through a connection known as an \"identity connection\" or \"skip connection\". The concept of identity/skip connection in the residual network is shown in Figure 3.  
Figure 3: Residual Network with identity connection. Figure 2: The VGG-19 architecture. where  
x is the input of the residual block with weight layers I and II;  
G(x) is the output of the residual block;  
F(x) is the output of the stacked convolutional or weight layers I and II. The output function is represented as  
\[\text{y}=\text{F}\big{(}\text{x},\ \{w_{j}\}\big{)}+\text{x} \tag{2}\]  
where  
F(x, \(\{w_{j}\}\)) denotes the residual block;  
\(w_{j}\) represents the weight layers (\(w_{1}\) and \(w_{2}\)). The dimensions of the input \"x\" and the output \"F (x)\" of the stacked convolutional layers I and II should be the same before adding them. If the dimensions are not the same, zeros can be added, that is, zero padding can be done. The inclusion of \"identity connection\" helps in overcoming the problem of vanishing gradients by providing a short path for the gradient. The following table (Table 2) summarizes the different layers present in the basic Resnet50V2 model. The ResNet50V2 model accepts 224 \(\times\) 224 \(\times\) 3 pixel input images. On input images, it first performs the convolution (7 \(\times\) 7) and max-pooling (3 \(\times\) 3) operations. There are four stages in this model, as well as some residual and identity blocks. For instance, in Stage I, the network provides three residual blocks (each with three layers) that conduct convolution operations with kernel sizes of 64 and 128. The size (width and height) of the input images are cut in half as it progresses through the phases, and the width of the image channel is doubled. The key difference between the ResNetV2 and ResNetV1 versions is that the former one does the convolution operation later, that is, after performing the batch normalization and ReLU activation. The ResNetV1 performs batch normalization and ReLU activation after convolution. We modified the classification part with additional layers, such as a global average pooling layer, a dropout layer (0.5), a dense layer (512 units) with an L2 regularizer, and a final dense layer (2 units) for positive and negative classes. We fine-tuned the network by freezing the layers that were up to Stage III of the network (conv3_block4_out) and unfreezing the remaining layers that were trained on the Crowdflower image sentiment dataset. The architecture of the fine-tuned ResNet50 network architecture used in our work is depicted in Figure 4.