## 4 Proposed Solution
### Main Characteristics of the Multi-Agent Gradient Descent Algorithm (MAGD)
* Mean Squared Error (MSE): MSE measures the average of the squared differences between predicted and actual values. It quantifies the model's prediction error, with lower values indicating a better fit. \[\text{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}\] (24)
* Accuracy Score: Accuracy score is the ratio that measures the probability that the model correctly predicts the target variable. \[\text{Accuracy}=\frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}}=\frac{1}{n}\sum_{i=1}^{n}(y_{i}=\hat{y}_{i})\] (25)
* Coefficient of Determination (\(R^{2}\)): \(R^{2}\) indicates the capability of the regression model to explain the variance in the dependent variable that is predictable from the independent variables (features). A value closer to 1 suggests a better fit. The \(R^{2}\) value typically ranges from 0 to 1, but it can be less than 0 in some cases, indicating that the model poorly fits the data. \[R^{2}=1-\frac{\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}{\sum_{i=1}^{n}(y_{i}- \hat{y})^{2}}\] (26) where
* \(n\) is the total number of data points. * \(y_{i}\) is the true label for the \(i\)-th data point. * \(\hat{y}_{i}\) is the predicted label for the \(i\)-th data point. * \((y_{i}=\hat{y}_{i})\) is an indicator function that equals 1 if \(y_{i}=\hat{y}_{i}\) (i.e., the prediction is correct) and 0 otherwise. * \(\hat{y}\) is the mean of the true labels. For the accuracy score metric, we will consider the models' accuracy on the test datasets. However, for the MSE metric, we will compare algorithms using both training and testing datasets. While our primary focus is on developing an optimization algorithm for neural network training, where the MSE on the test data is more relevant, we also compare performance on the training data. This comparison highlights the algorithm's potential for broader applications, demonstrating its general-purpose capabilities beyond just the specific task of neural network optimization. Many researchers, such as [PERSON] et al. [48] (p. 8), argue that minimizing the training error too aggressively can lead to overfitting. While this is true in some cases, it is not always the expected outcome. In general, achieving a smaller training error often leads to better accuracy on both the training and testing sets. However, when there is a significant gap between training and testing performance, it can be a sign of issues such as inefficient data splitting, insufficient data, or other factors like outliers in the training set or overly complex models. Our results support this hypothesis. This risk can be mitigated using established techniques, including regularization, dropout, and data augmentation. In our experiments, we employed \(L2\) regularization, which reduces model complexity by modifying the training cost function to include a penalty on trainable weights. It is important to note that while this technique can help mitigate overfitting risks, it does not completely eliminate the problem. The formula for \(L2\) regularization is shown in Equation (27). \[L2\text{ Regularization}=\lambda\sum_{j=1}^{p}w_{j}^{2} \tag{27}\]

where

* \(\lambda\) is the regularization parameter (also known as the shrinkage or penalty term). * \(p\) is the number of features or weights. * \(w_{j}\) represents the \(j\)-th weight in the model. * \(\sum_{j=1}^{p}w_{j}^{2}\) is the sum of the squared weights, which penalizes large weights. Thus, the task is to minimize the cost function shown in Equation (28). \[\text{Cost}=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}+\lambda\sum_{j=1}^ {p}w_{j}^{2} \tag{28}\]

Our findings indicate that in most cases, more optimal training errors lead to better generalization accuracy. We observe an increasing linear trend for testing error versus training error, as shown in Figure 27. Figure 27 has five axes, each representing the variance of testing error versus training error.