## 4 Proposed Solution
### Main Characteristics of the Multi-Agent Gradient Descent Algorithm (MAGD)
\begin{table}
\begin{tabular}{c c c c} \hline
**Dataset Name** & **MAGD-500** & **Adam-500** & **Adam-750** & **Adam-1000** \\ \hline Diabetes & **0.41** & 0.38 & 0.36 & 0.36 \\ California Housing & **0.71** & **0.71** & **0.71** & **0.71** \\ Boston Housing & **0.89** & 0.88 & 0.88 & 0.88 \\ Auto MPG & 0.89 & 0.89 & **0.90** & **0.90** \\ Concrete Strength & **0.85** & **0.85** & **0.85** & **0.85** \\ Energy Efficiency & **0.96** & 0.95 & 0.95 & **0.96** \\ Yacht Hydrodynamics & **0.98** & **0.98** & **0.98** & **0.98** \\ Forest Fires & \(-\)23.66 & \(-\)**17.24** & \(-\)19.25 & \(-\)19.81 \\ Airfoil Self-Noise & **0.79** & 0.72 & 0.72 & 0.72 \\ Concrete Slump Test & 0.95 & 0.96 & **0.97** & **0.97** \\ \hline \end{tabular}
\end{table} 
Table 29: \(R^{2}\) scores of MAGD with 500 maximum iterations compared to Adam with 500, 750, and 1000 maximum iterations across various regression datasets. Bold values indicate the highest score achieved for each dataset. \begin{table}
\begin{tabular}{c c c} \hline
**Comparison** & \(ATP\) (\(MAGD\), \(Adam\)) & \(P_{super}\) (\(MAGD\)\(\succ\)\(Adam\)) \\ \hline MAGD-500 vs. Adam-500 & 1.1 & 20\% \\ MAGD-500 vs. Adam-750 & 0.34 & 10\% \\ MAGD-500 vs. Adam-1000 & \(-\)0.67 & 0\% \\ \hline \end{tabular}
\end{table} 
Table 31: Comparison of MAGD for fixed maximum iterations and Adam with varying maximum iterations using \(ATP\) (average time penalty) and \(P_{super}\) (performance superiority) in regression tasks. Tables 30 and 31 demonstrate that MAGD consistently maintains its superiority in classification tasks, even when Adam is allocated a longer time budget. Specifically, Table 30 shows that MAGD outperforms Adam in classification tasks, despite Adam receiving a longer execution time. Table 31 further reveals that when Adam is given 750 maximum iterations, resulting in a similar time consumption as MAGD with 500 iterations, MAGD still achieves a 10% improvement. Even when Adam is granted 1000 iterations and a longer execution time than MAGD, it performs only equivalently to MAGD, underscoring the role of multi-agent initialization in improving optimization results. ## 6 Conclusions

In this study, we introduced a novel multi-start optimization algorithm, Multi-Agent Gradient Descent (MAGD), motivated by the need to address challenges in machine learning where gradient-based algorithms often struggle with local optima. By adaptively pruning underperforming agents, MAGD achieves a balance between simplicity and computational efficiency. Theoretical analysis demonstrated that MAGD maintains computational efficiency comparable to single-agent gradient-based algorithms in scenarios where the dimensionality of the problem or the number of iterations significantly exceeds the number of agents. This scalability highlights MAGD's suitability for high-dimensional and extended optimization tasks. We evaluated MAGD on mathematical non-convex problems (20 benchmark 2D mathematical functions) and neural network training (10 regression and 10 classification datasets). MAGD consistently outperformed GD and Adam, even when Adam was given equivalent or longer time budgets, and demonstrated optimization performance comparable to Swarm-Based Gradient Descent (SBGD) and Fixed Multi-Agent Adam (FMA). However, MAGD showed significant advantages in execution time over both SBGD and FMA, while requiring minimal hyperparameter tuning and being easier to implement compared to SBGD. These results highlight MAGD's practicality and robustness. In neural network training experiments, where the number of iterations greatly exceeded the number of agents, MAGD exhibited execution times statistically comparable to GD and Adam. In contrast, for mathematical function experiments involving a high number of agents and low-dimensional problems, MAGD achieved significantly slower execution times than single-agent methods. These findings align with our theoretical analysis. Despite these promising results, MAGD has limitations.