## 6 Discussion
### The Optimizers Used  
The main task of the optimizer during network training is to update the weights to reduce the value of the loss function. The optimizer can have a huge impact on the convergence of the training process, especially for transfer learning, as pointed out by [PERSON] et al. [56] and [PERSON] et al. [62]. Four optimizers were mainly reported by the authors, namely SGD for studies [52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65] SGD with momentum for studies [56; 58; 59], Adam for studies [56; 67; 70], and RMSProp in [68]. The stochastic gradient descent optimizer (SGD) allows for a faster training process than the traditional gradient descent because it only considers, at each iteration, a subset of the training set. Thus, it generally achieves faster iterations in trade for a (slightly) lower convergence rate. SGD with momentum (SGDM) can be used instead of SGD: by adding the momentum and thus determining the next update of the weights based on a linear combination of the gradient and the previous update, it prevents the training process to show oscillatory behavior. This should result in faster and accurate convergence. The RMSProp optimizer, a member of the adaptive gradient group, was introduced to overcome the problem of determining the initial value of the learning rate, which is now learned during the training  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Architecture** & **Count** \\ \hline InceptionV3 & 9 \\ AlexNet & 7 \\ VGG16 & 6 \\ VGG19 & 3 \\ VGG-S & 2 \\ Xception & 2 \\ DenseNet & 2 \\ ResNet & 5 \\ GoogleNet & 4 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Architectures analysis.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Dataset** & **Count** \\ \hline Kaggle & 9 \\ Messidor & 3 \\ Private & 6 \\ e-ophtha & 1 \\ DR1 & 1 \\ STARE & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Datasets analysis.  
process. The Adam optimizer was introduced to combine the benefits from both the SGDM and the RMSProp optimizer.  
The LR chosen by the authors was very low to avoid losing the original weights of the layers, and it ranged from \(1\times 10^{-\times}\) to \(1\times 10^{-0}\). [PERSON] et al. [59] used LR = \(1\times 10^{-0}\) for AlexNet and VGG16, as well as LR=\(1\times 10^{-\times}\) for InceptionV3. [PERSON] et al. [65] used LR = \(5\times 10^{-0}\). [PERSON] et al. [56] used LR = \(1\times 10^{-0}\). [PERSON] et al. [68] used LR = \(2\times 10^{-0}\). [PERSON] et al. [62] used LR = \(2\times 10^{-0}\). [PERSON] et al. [64] used LR = \(0.01\). [PERSON] et al. [58] used LR = \(1\times 10^{-0}\). [PERSON] et al. [63] used LR = [0.1-0.0001]. [PERSON] et al. [70] used LR = \(1\times 10^{-0}\). Not all the authors reported the optimizer that they used or the LR used.  
The choice of the optimizer and the learning rate can play a vital role in network performance and the convergence time, especially when using transfer learning. Optimizers like SGD and SGDM can take a longer time to reach convergence, while the RMSProp optimizer can take a shorter time but might not reach the same performance of SGD and SGDM. The Adam optimizer can reach the performance of SGD and SGDM while taking a shorter time, like RMSProp. The learning rate is very important as well because the choice of a high learning rate can completely change the pretrained weights, thus deteriorating the performance of the network. On the other hand, with a low learning rate value, the network weights will be adjusted to the new dataset without completely change the original weights. A summary of the optimizers that were found in the reviewed papers is shown in Table 6.