## 2 Materials and Methods
### Transfer Learning Using ResNet50, VGG16 and DenseNet169  
In this paper, transfer learning (TL) using the architectures of the three CNNs models, ResNet50, VGG16 and DenseNet169, was applied to the diabetic retinopathy images. In TL, learned features from one task are applied to a different task without having to learn from scratch. This is commonly used when building CNN models since the process of training from scratch requires a lot of computational resources, large datasets and a lot of time [31]. CNN models consist of multiple layers, namely: the convolution layer, pooling layer and fully connected layer. CNN models employ multiple perceptrons to evaluate picture inputs and eventually extract different patterns from the images to output to the fully connected layer. Our CNN models extracted representative patterns to form the feature maps. A 3 \(\times\) 3 kernel was passed over the input matrix of the diabetic retinopathy image, as illustrated in Figure 4.  
Figure 4: Convolution layer.  
Figure 3: Augmented APTOS dataset.  
The classification function, which is the output of the fully connected layer, plays an important role in the process, whereby the different patterns of the five stages of diabetic retinopathy, learnt by the feature extraction layers, are used to perform the multiclass classification.  
The VGG16 model, a CNN architecture pre-trained on the ImageNet dataset, was adopted for the development of our diabetic retinopathy application as it has been fully tested in a similar domain, achieving good performance [32; 33]. VGG16 consists of 13 convolutional layers and 3 fully connected layers. There are 5 blocks each containing 2 or 3 convolution layers and ending with a max-pooling layer, as illustrated by Figure 5. A fixed-size image of dimensions (224, 224, 3) is the input to the VGG16 model.  
ResNet50, another popular CNN architecture, consists of 50 layers organized in so-called residual blocks [9]. It is known for its skip connection approach, which eventually solves the vanishing gradient problem. ResNet50 contains 48 convolution layers along with 1 MaxPool and 1 AveragePool layer. This was desired in our diabetic retinopathy application as it allows the later layers to learn lesser semantic information that was captured in the early layers. A 3 \(\times\) 3 filter was used to perform the spatial convolution, which was eventually reduced using the max-pooling method. Figure 6 illustrates the ResNet 50 model with the 48 convolution layers and the 16 skip connections.  
The third model that was considered was the DenseNet169 model [34]. Compared to the ResNet50 model, it has more layers. However, it contains a similar block to skip connections called the dense block. With the increase in the number of layers, it gives the model the opportunity to learn more distinctive features. In fact, the architecture consists of four dense blocks with varying numbers of layers as illustrated in Figure 7. Our design for this model consisted of the 2D average pooling, which is in the original architecture, where a dropout layer set to 0.5 was added.  
Figure 5: Architecture of the VGG16 model.  
Figure 6: Architecture of ResNet50 model.