# Nonsmooth Optimization-Based Hyperparameter-Free Neural Networks for Large-Scale Regression
## 5 LMBNNR Algorithm
The convergence rate of the LMBM, employed as an underlying solver in the proposed LMBNNR algorithm, has not been studied extensively, but, as a bundle method, it is at most linear (see, e.g., [62]), while the search direction can be computed within \(O(m)\) operations [3]. For comparison purposes, we mention that the time complexity of a BPN algorithm is typically \(O(n)\), where \(n\) is the number of training samples. A more specific analysis of the time complexity of LMBNNR remains a subject for future research._

### Results and Discussion

The results of our experiments are given in Tables 2 and 3, where we provide the RMSE values for the test set and the used CPU time in seconds. The results using the other three performance measures (MAE, \(R^{2}\), and \(r\)) are given as Supplementary Material for this paper. Since the algorithms are implemented in different programming languages and run on different platforms, the CPU times reported here are not directly comparable. Nevertheless, we can still compare the magnitudes of the computational times used. For the BPN algorithms, the results (including CPU times) are the average of results over ten runs, unless some of the runs lead to NaN loss function values. In that case, the results are the average of the successful runs. Note that obtaining a NaN loss function value is a natural property of SGD and related methods if the learning rate is too large. If there is \"NaN\" in the tables, then all ten runs lead to the NaN solution. In all separate runs, the maximum time limit is set to 1 h when \(n\times m<10^{6}\) (\(n\) and \(m\) are the number of samples and features, respectively) and 2 h otherwise. In tables, \"t-lim\" means that an algorithm gives no solution within the time limit. To evaluate the reliability of ASP, we force LMBNNR to solve ReLU-NNR problems up to 200 nodes. With LMBNNR, the results with smaller numbers of nodes are obtained as a side product. In tables, we report the results obtained with different numbers of nodes (the same numbers that we used for testing the other algorithms), the best solution with respect to the RMSE of the test set, and the results when applying ASP. The last two are denoted by \"Best\" and \"ASP\", respectively. As we report all the results with different model parameter combinations (i.e., results with different numbers of nodes or different combinations of batch sizes and epochs), we do not use separate validation sets in our experiments. Instead, we use the best results, namely the smallest RSME of the test set, obtained with any other NNR algorithm than LMBNNR, in our comparison. Naturally, in real-world predictions, we would not know which result is the best, and a separate validation set should be employed to tune the hyperparameters with NNR algorithms but LMBNNR. Note that this kind of experimental setting favors the other methods over the proposed one. In tables, the best results obtained with any other NNR algorithm than LMBNNR are presented using boldface font. To make the comparison even more challenging for the LMBNNR algorithm, we compare the result that it gives with ASP to the best result obtained with any other tested algorithm. We point out that the results using the other three performance measures support the conclusions drawn from RMSE. Nevertheless, MAE often indicates slightly better performance of the LMBNNR algorithm than RMSE (see the Supplementary Material). This is, in particular, due to the used loss function or the regularization term.