## 4 Proposed Solution
### Main Characteristics of the Multi-Agent Gradient Descent Algorithm (MAGD)
It does not guarantee finding the global minimum, and although we theoretically demonstrated its effectiveness in high-dimensional optimization problems, this claim was not empirically verified. Our experiments were restricted to 2D mathematical functions and shallow neural networks with low to medium-dimensional datasets. In the future, we plan to test MAGD on high-dimensional mathematical problems and deep neural network training to validate its efficiency and scalability. Additionally, we aim to compare MAGD with a broader range of multi-start optimization variants to provide a more comprehensive evaluation. Further research should also focus on improving its global optimization capabilities and exploring its applicability to broader optimization tasks beyond machine learning. We believe MAGD's simplicity, reduced reliance on hyperparameters, and competitive performance make it a robust alternative to Adam in machine learning and deep learning applications. Often, well-designed, simple solutions prove to be the most effective or at least highly competitive. **Supplementary Materials:** The Colab notebook for implementing the algorithms for both mathematical functions and neural network training tasks is provided at the link below. This notebook includes the core classes and functionalities of the proposed algorithms, along with demonstrations for mathematical functions and neural network training. While the notebook does not include the full code for generating all experiments and results presented in this study, it provides sufficient resources to reproduce the experiments using the basic classes and the hyperparameter settings described earlier in this paper. [[https://colab.research.google.com/drive/1](https://colab.research.google.com/drive/1) IIcpEA11 NaG-ns6 oT1 CKOTdmojLXWht?usp=sharing]([https://colab.research.google.com/drive/1](https://colab.research.google.com/drive/1) IIcpEA11 NaG-ns6 oT1 CKOTdmojLXWht?usp=sharing), accessed on 26 January 2025. **Author Contributions:** Conceptualization, M.S. and M.R.B. ; methodology, M.S. and M.R.B. ; validation, M.S. and M.R.B. ; investigation, M.S. ; resources, M.S. ; writing--original draft preparation, M.S. and M.R.B. ; writing--review and editing, M.S. and M.R.B. ; visualization, all authors; supervision, M.R.B. ; project administration, M.R.B. All authors have read and agreed to the published version of the manuscript. **Funding:** This research received no external funding. **Institutional Review Board Statement:** Not applicable. **Informed Consent Statement:** Not applicable. **Data Availability Statement:** The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding author. **Acknowledgments:** We would like to express our gratitude to Innopolis University for their financial support in conducting this research. **Conflicts of Interest:** The authors declare no conflicts of interest. ## References

* [1] [PERSON]; [PERSON] _Convex Optimization_; Cambridge University Press: Cambridge, UK, 2004. * [2] [PERSON]; [PERSON]; [PERSON] Sustainability and Optimization: From Conceptual Fundamentals to Applications. _Sustainability_**2020**, _12_, 2027. [CrossRef]
* [3] [PERSON] An overview of gradient descent optimization algorithms. _arXiv_**2017**, arXiv:1609.04747. [CrossRef]
* [4] [PERSON] Gradient Descent based Optimization Algorithms for Deep Learning Models Training. _arXiv_**2019**, arXiv:1903.03614. [CrossRef]
* [5] [PERSON]; [PERSON] On the Linear Convergence of Descent Methods for Convex Essentially Smooth Minimization. _SIAM J. Control Optim._**1992**, _30_, 408-425. [CrossRef]
* [6] [PERSON] Optimization for deep learning: Theory and algorithms. _arXiv_**2019**, arXiv:1912.08957. [CrossRef]
* [7]_Numerical Optimization_; Springer Series in Operations Research and Financial Engineering; Springer: New York, NY, USA, 2006. [CrossRef]
* [8] [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training.