## 4 Proposed Solution
### Main Characteristics of the Multi-Agent Gradient Descent Algorithm (MAGD)
MAGD provides 20% improvement ratio over Adam, and 50% over GD. The results of neural network training indicate that even when testing error is higher, MAGD often outperforms SBGD, Adam, and GD, further validating its effectiveness. ### Training Duration Evaluation

This paragraph evaluates the execution time of the proposed algorithm compared to other methods as shown in Table 21 that reports the mean number of convergence iterations for each algorithm to train the neural networks. One can observe that SBGD consumes the shortest time for neural network training compared to the other algorithms. This behavior contrasts with its performance on mathematical functions. The reason for SBGD's faster training lies in its rapid convergence, as shown in Table 22. However, this rapid convergence results in lower performance scores and shorter execution times, primarily due to the numerous hyperparameters that require tuning to control the algorithm's convergence behavior. Among the algorithms, FMA requires the longest execution time. However, the extreme time overhead of FMA is not greatly correlated with its performance. MAGD consumes the longest time among the remaining algorithms (SBGD, Adam, and GD), but this extended execution time is correlated with its superior performance scores. Despite this observation, statistical inference suggests the differences in execution time between MAGD and GD or Adam algorithms are not statistically significant, as elaborated in the next analysis and discussion. \begin{table}
\begin{tabular}{c c} \hline \hline
**Comparison** & \(\mathbf{P_{super}}\) \\ \hline MAGD \(\succ\) Adam & 20\% \\ MAGD \(\succ\) GD & 50\% \\ \hline \hline \end{tabular}
\end{table} 
Table 20: Comparison of MAGD, Adam, and GD in terms of the superiority ratio in regression tasks \(P_{super}\). \begin{table}
\begin{tabular}{c c} \hline \hline
**Comparison** & \(\mathbf{P_{super}}\) \\ \hline MAGD \(\succ\) Adam & 50\% \\ \hline \hline \end{tabular}
\end{table} 
Table 19: Comparison of MAGD and Adam in terms of the superiority ratio in classification tasks \(P_{super}\). **Statistical Analysis of Training Duration:** To derive insights from the statistical results of the different algorithms, statistical features such as the mean, standard deviation (std), median, and the 90% Highest Density Interval (HDI) are presented in Table 23. We opted for the 90% HDI instead of the 95% HDI due to the presence of an outlier in the Adult dataset, where all algorithms exhibit large deviations from their distributions. This outlier impacts the accuracy of the highest density interval estimation when using a 95% interval. Additionally, a \(t\)-test was conducted to evaluate the significance of differences between the algorithms (\(p\)-value). The \(t\)-test results, presented in Table 24. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Comparison** & **MAGD** & **FMA** & **SBGD** & **Adam** & **GD** \\ \hline mean \(\pm\) std & \(10.6\)\(\pm\) 34.5 & \(52.8\)\(\pm\) 172.6 & \(2.5\)\(\pm\) 8.7 & \(6.0\)\(\pm\) 19.2 & \(6.0\)\(\pm\) 19.0 \\ median & \(1.9\) & \(7.9\) & \(0.2\) & \(0.9\) & \(0.8\) \\
90\% HDI & \([1.1,11.6]\) & \([6.4,57.7]\) & \([0.1,3.0]\) & \([0.7,7.2]\) & \([0.7,8.1]\) \\ \hline \hline \end{tabular}
\end{table} 
Table 23: Statistical comparison between the execution duration of different algorithms for neural network training.