## 4 Implementation
### Hyperparameters  
For training our models on different datasets, we used various hyperparameters to optimize the performance. These hyperparameters included the Input Size, Batch Size, Epochs, Learning Rate, Early Stopping, Learning Rate Scheduler, Dropout Rate, and L2 Regularization. The specifics of these parameters for different datasets are outlined in Table 4.  
In the CK+ dataset, the original images were re-sized to (224, 224, 3). We applied a dropout rate of 0.5 for regularization purposes. Additionally, we implemented early stopping, which halts the training process if the validation accuracy does not improve for five consecutive epochs to prevent overfitting. We used the same techniques for data augmentation and resizing on the KDEF dataset. We also used early stopping and a learning rate scheduler. If the validation accuracy did not improve for three consecutive epochs, the learning rate was multiplied by 0.5 to decrease it. If the accuracy did not increase for five consecutive epochs, the training process was stopped. The dropout rate for this dataset was set to 0.1, and l2 kernel regularization with a value of 0.01 was applied for optimal performance. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Hyper Parameters** & **CK+** & **KDEF** & **FER2013** \\ \hline \multirow{3}{*}{Input Size} & **(224,224,3)** & **(224,224,3)** & (224,224,3) \\  & (48,48,3) & (48,48,3) & (48,48,3) \\  & (144,144,3) & (144,144,3) & **(144,144,3)** \\ \hline \multirow{3}{*}{Batch Size} & 16, **32**,64 & 16, **32**,64 & 16, 32,**64** \\ \hline \multirow{3}{*}{Epochs} & 300 & 300 & 30 \\ \cline{2-4}  & Learning Rate & 0.01,0.001,**0.0001** & 0.01,0.001,**0.0001** & 0.01,0.001,**0.0001** \\ \hline \multirow{3}{*}{Early Stop} & \begin{tabular}{c} Monitor Validation Accuracy \\ (Patience = 5) \\ \end{tabular} & \begin{tabular}{c} Monitor Validation Accuracy \\ (Patience = 5) \\ \end{tabular} & \begin{tabular}{c} Monitor Validation Accuracy \\ (Patience = 3, Factor = 0.5) \\ \end{tabular} \\ \hline \multirow{2}{*}{Learning Rate Scheduler} & \multirow{2}{*}{-} & \
\begin{tabular}{c} Monitor Validation Accuracy \\ (Patience = 3, Factor = 0.5) \\ \end{tabular} &
\begin{tabular}{c} Cosine Annealing (T\_max = 30, initial\_tr = 0.0001) \\ \end{tabular} \\ \hline \multirow{2}{*}{Dropout Rate} & 0.1,0.3,**0.5** & **0.1**,0.3,0.5 & **0.1**,0.3,0.5 \\ \cline{2-4}  & - & **0.01**,0.1,0.2 & 0.01,**0.1**,0.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Hyperparameters for the models on different datasets. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Layer (Type)** & **Output Shape** & **Trainable Parameters** & **Non-Trainable Parameters** \\ \hline \multirow{3}{*}{Global AveragePooling 2D} & **VGG19 Base Model \({}^{*}\)** & 10,585,152 & 9,439,232 \\  & (None, 512) & 0 & 0 \\ Dropout & (None, 512) & 0 & 0 \\ (Dense + ReLU) & (None, 1024) & 525,312 & 0 \\ Dropout & (None, 1024) & 0 & 0 \\ (Dense + Softmax) & (None, 7) & 7175 & 0 \\ \hline \multirow{2}{*}{**Trainable Parameters**} & \multirow{2}{*}{9,971,719} & 0 \\  & & 0 & 10,585,152 \\ \hline \multirow{2}{*}{**Total Parameters**} & \multirow{2}{*}{20,556,871} \\ \end{tabular} \({}^{*}\)The layers of the VGG19 model, excluding the last five, were frozen.  
\end{table} 
Table 3: The proposed modified VGG19. For the FER2013 dataset, we conducted extensive experimentation with various learning strategies and optimizers. Through our analysis, we discovered that employing cosine annealing with the adam optimizer yielded the most accurate results. During training, _initial_lr_ set the starting learning rate, while _T_max_defined the duration of a cycle of the cosine annealing schedule. The learning rate decreased gradually from the initial value to a minimum over T_max epochs, following a cosine curve pattern. Additionally, we explored different image sizes and were surprised to find that the transfer learning models performed exceptionally well when the images were resized to (144,144,3).