## 4 Proposed Solution
### Main Characteristics of the Multi-Agent Gradient Descent Algorithm (MAGD)
One possible explanation is that SBGD has not demonstrated superior performance in high-dimensional problems, as noted in its original study [15]; the mathematical functions tested here were limited to two dimensions, while neural network cost functions have a larger number of dimensions. Additionally, SBGD's performance is sensitive to its numerous hyperparameters, which can cause premature convergence and limit solution quality. Our experiments also used a relatively small number of agents (10) for SBGD due to computational constraints, which could affect its efficacy. In Table 16, we again omit FMA and refer to the best results among MAGD, SBGD, Adam, and GD with *. The results show consistent patterns between test error and training error. SBGD has the least error in only 1 dataset out of 20, GD in 2 datasets, while MAGD and Adam achieve the best values, with MAGD slightly underperforming Adam. However, this underperformance is not significant, as MAGD has the best values in 8 datasets out of 20, while Adam has the best values in 11 datasets out of 20. Nevertheless, since MAGD performs best in terms of training error, it still holds great promise for optimization problems where local minima arise. Additionally, a lower mean square error does not always correlate with worse classification or regression accuracy. These metrics are represented in Tables 17 and 18. Table 17 compares the test accuracy of different algorithms on classification datasets, while Table 18 compares the \(R^{2}\) scores for different algorithms on various regression datasets. The same configuration applies here as for the training and testing errors, where we omit FMA and refer to the best score among MAGD, SBGD, Adam, and GD with *. Note that the maximum value is the best, in contrast to MSE. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset Name** & **MAGD** & **FMA** & **SBGD** & **Adam** & **GD** \\ \hline Iris & 0.1756 & **0.1723** & 0.5853 & **0.1723 * & 0.6774 \\ Wine & **0.0914 *** & **0.0914** & 0.3137 & **0.0914 * & 0.281 \\ Breast Cancer Wisconsin & 0.1443 & 0.1443 & 0.3590 & **0.1434 * & 0.3865 \\ Digits & **0.3250 *** & **0.3250** & 1.5970 & 0.3269 & 1.8109 \\ Pima Indians Diabetes & 0.5381 & 0.5378 & **0.5264 *** & 0.5340 & 0.5449 \\ Connectionist Bench & 0.8415 & 0.8075 & 1.3797 & **0.7793 * & 1.4082 \\ Glass Identification & 1.2559 & 1.1650 & 1.3674 & **1.1613 * & 1.4225 \\ Balance Scale & **0.3393 *** & **0.3393** & 0.4879 & 0.3603 & 0.7383 \\ Banknote Authentication & 0.1013 & **0.1007** & 0.3460 & 0.1009 * & 0.5542 \\ Adult & **0.3430 *** & **0.3430** & 2.5628 & **0.3430 * & 2.4413 \\ Diabetes & 0.5584 & 0.6359 & 0.5827 & 0.5825 & **0.4455 * \\ California Housing & 0.3338 & **0.3303** & 0.5903 & 0.3314 * & 0.4498 \\ Boston Housing & **0.1522 *** & 0.1533 & 0.4468 & 0.1644 & 0.2982 \\ Auto MPG & 0.1173 & **0.1133** & 0.1941 & 0.1165 * & 0.1926 \\ Concrete Compressive Strength & **0.2063 *** & 0.2070 & 0.4983 & 0.2086 & 0.4252 \\ Energy Efficiency & 0.0670 * & **0.0661** & 0.1706 & 0.0728 & 0.1229 \\ Yacht Hydrodynamics & 0.0705 & **0.0613** & 0.3533 & 0.0695 * & 0.3237 \\ Forest Fires & 3.8441 & 3.2475 & 0.5428 & 2.9355 & **0.4613 *** \\ Airfoil Self-Noise & **0.2906 *** & **0.2906** & 0.6447 & 0.3325 & 0.4701 \\ Concrete Slump Test & 0.0973 & **0.0756** & 0.3574 & 0.0829 * & 0.2542 \\ \hline \hline \end{tabular}
\end{table} 
Table 16: Testing error values obtained by MAGD, FMA, SBGD, Adam, and GD for each dataset. Bolded values indicate the best result. * denotes the best result among the algorithms excluding FMA, as FMA is not always the best. The results demonstrate the superiority of MAGD, particularly in classification tasks, where it achieves the highest test scores across all classification datasets. In regression tasks, while FMA slightly outperforms MAGD, FMA is excluded from our comparison. In most cases, MAGD's lower training error corresponds to superior generalization, with the exception of the \"Forest Fires\" dataset.