## 4 Proposed Solution
### Main Characteristics of the Multi-Agent Gradient Descent Algorithm (MAGD)
For clearer visualization, the trend was fitted using Random Sample Consensus (RANSAC), a linear regression technique that is strengthened by features designed to protect the model from being affected by outliers [49; 50] (p. 2). The red cross symbol represents outliers, defined as data points where the residual distance (predicted value \(-\) actual value) is greater than 1. We observe that larger testing errors for larger training errors are not typical, but rather exceptional. There are at most two outliers, if any, out of 20 experiments. Tables 15 and 16 compare the training costs achieved by the four algorithms in addition to FMA. The bold values indicate the least error among all algorithms. FMA achieves the least error in almost all datasets, but since we consider FMA as merely multiple executions of algorithms, it is not fair to judge its superiority here, as it consumes significantly longer time, as discussed in the next paragraph. Therefore, we will highlight the least error among the remaining algorithms (MAGD, SBGD, Adam, and GD) with * and use this to evaluate the superiority of these four algorithms. Looking at the training error values in Table 15, both MAGD and Adam demonstrate superior performance, with MAGD achieving slightly better solutions in most cases. Although this difference between Adam and MAGD is not statistically significant (a \(t\)-test is unnecessary for this comparison), practical applications may still value these small improvements. Minor increases in classification accuracy or \(R^{2}\) in machine learning can have a significant practical impact, particularly in fields like medical diagnostics [51] (p. 4) and [52]. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset Name** & **MAGD** & **FMA** & **SBGD** & **Adam** & **GD** \\ \hline Iris & 0.2114 & **0.2084** & 0.5458 & **0.2084 \({}^{*}\)** & 0.6506 \\ Wine & 0.0872 & **0.0869** & 0.3065 & **0.0869 \({}^{*}\)** & 0.3303 \\ Breast Cancer Wisconsin & **0.0688 \({}^{*}\)** & **0.0688** & 0.3058 & 0.0689 & 0.3588 \\ Digits & **0.2574 \({}^{*}\)** & **0.2574** & 1.5106 & 0.2599 & 1.7374 \\ Pima Indians Diabetes & 0.4480 \({}^{*}\) & **0.4470** & 0.5293 & 0.4510 & 0.5441 \\ Connectionist Bench & 0.0855 \({}^{*}\) & **0.0852** & 1.3007 & 0.0863 & 1.0639 \\ Glass Identification & **0.6890 \({}^{*}\)** & **0.6890** & 1.2473 & 0.7306 & 1.2543 \\ Balance Scale & **0.3320 \({}^{*}\)** & **0.3320** & 0.5087 & 0.3521 & 0.7724 \\ Banknote Authentication & 0.1027 & **0.1021** & 0.3473 & 0.1023 \({}^{*}\) & 0.5598 \\ Adult & **0.3333 \({}^{*}\)** & **0.3333** & 2.5611 & **0.3333 \({}^{*}\)** & 2.4446 \\ Diabetes & 0.4137 \({}^{*}\)** & **0.4022** & 0.6237 & 0.4194 & 0.5368 \\ California Housing & 0.3351 & **0.3328** & 0.5682 & 0.3335 \({}^{*}\) & 0.4378 \\ Boston Housing & 0.1267 & **0.1252** & 0.5145 & 0.1256 \({}^{*}\) & 0.3396 \\ Auto MPG & 0.1332 \({}^{*}\) & **0.1324** & 0.2308 & 0.1337 & 0.2324 \\ Concrete Compressive Strength & 0.1678 \({}^{*}\) & **0.1660** & 0.4276 & 0.1682 & 0.3844 \\ Energy Efficiency & 0.0667 \({}^{*}\) & **0.0651** & 0.1787 & 0.0753 & 0.1316 \\ Yacht Hydrodynamics & 0.0681 & **0.0596** & 0.4158 & 0.0672 \({}^{*}\) & 0.3596 \\ Forest Fires & 0.4871 \({}^{*}\) & **0.4801** & 1.5411 & 0.5106 & 1.4490 \\ Airfoil Self-Noise & **0.2983 \({}^{*}\)** & **0.2983** & 0.6478 & 0.3406 & 0.4724 \\ Concrete Slump Test & 0.0391 \({}^{*}\) & **0.0361** & 0.1948 & 0.0405 & 0.1354 \\ \hline \hline \end{tabular}
\end{table} 
Table 15: Training error values obtained by MAGD, FMA, SBGD, Adam, and GD for each dataset. Bolded values indicate the best result. \({}^{**}\) denotes the best result among the algorithms excluding FMA, as FMA is not always the best. Figure 27: Trend of testing error variance as a function of training error variance in neural network training tasks. GD and SBGD perform substantially worse in training cost optimization than Adam and MAGD, which raises the question of why SBGD, which performs well in low-dimensional mathematical function tests, falls short in neural network training.