# Nonsmooth Optimization-Based Hyperparameter-Free Neural Networks for Large-Scale Regression
## 5 LMBNNR Algorithm
The source code is available at [[http://napsu.karmitsa.fi/lmbmnr](http://napsu.karmitsa.fi/lmbmnr)]([http://napsu.karmitsa.fi/lmbmnr](http://napsu.karmitsa.fi/lmbmnr)) (accessed on 10 September 2023). There are no tuneable parameters in the LMBNNR algorithm. The BPN algorithm is implemented using TensorFlow in Google Colab. We use the ReLU activation for the hidden layer, the linear activation for the output layer, and the MSE loss function. We use MSE since it usually worked better than MAE in our preliminary experiments with the BPN algorithm. This is probably due to the smoothness of MSE. Naturally, with the proposed LMBNNR algorithm, we do not have difficulties with the nonsmoothness as it applies the nonsmooth optimization solver LMBM. In the BPN algorithm, the Keras optimizer SGD with the default parameters and the following three different combinations of _batch size_ (the number of samples that will be propagated through the network) and _number of epochs_ (the number of complete passes through the training data) are used:

* Mini-Batch Gradient Descent (MBGD): batch size \(=32\) and number of epochs \(=1\). These are the default values for the BPN algorithm;
* Batch Gradient Descent (BGD): batch size \(=\) size of the training data and number of epochs \(=1\). These choices mimic the proposed LMBNNR algorithm;
* Stochastic Gradient Descent (SGD): batch size \(=1\) and number of epochs \(=100\) for data sets with less than 100,000 samples and number of epochs \(=10\) for larger data. We reduce the number of epochs in the latter case due to very long computational times and the fact that the larger number of epochs often leads to NaN loss function values. SGD aims to be as a stochastic version of the BPN algorithm as possible. The algorithms ELM and MONMLP are implemented in R using the packages \"elmN-NRcpp\" (ELM is available at [[https://CRAN.R-project.org/package=elmNNRcpp](https://CRAN.R-project.org/package=elmNNRcpp)]([https://CRAN.R-project.org/package=elmNNRcpp](https://CRAN.R-project.org/package=elmNNRcpp)) (accessed on 10 September 2023)) and \"mommlp\" (MONMLP is available at [[https://CRAN.R-project.org/package=monmlp](https://CRAN.R-project.org/package=monmlp)]([https://CRAN.R-project.org/package=monmlp](https://CRAN.R-project.org/package=monmlp)) (accessed on 10 September 2023)). The parameters are the default values provided in the given references, but the number of hidden layers with MONMLP was set to one. The tests with ELM and MONMLP were run on Windows, 2.6 GHz Intel(R) Core(TM) i7-9750H. We run ELM, MONMLP, and the BPN algorithms with the number of hidden nodes set to 2, 5, 10, 50, 100, 200, 500, and 5000. Moreover, since SGD is used as an optimizer in the BPN algorithms as a stochastic method, we run the BPN algorithms ten times for all problems and report the average. Note that LMBNNR needs to be applied to solve the ReLU-NNR problem only once for each data set. \begin{table}
\begin{tabular}{l r r r} \hline
**Data Set** & **No. of Samples** & **No. of Features** & **Reference** \\ \hline Combined cycle power plant & 9568 & 5 & [50; 51] \\ Airfoil self-noise & 1503 & 6 & [52] \\ Concrete compressive strength & 1030 & 9 & [53] \\ Physicochemical properties & & & \\ of protein tertiary structure & 45,730 & 10 & [52] \\ Boston housing data & 506 & 14 & [54] \\ SGEMM GPU kernel performance 1 & 241,600 & 15 & [55; 56] \\ Online news popularity & 39,644 & 59 & [57] \\ Residential building data set 1 & 372 & 108 & [58] \\ BlogFeedback 2 & 52,397 & 281 & [59] \\ ISOLET & 7797 & 618 & [52] \\ CIFAR-10 & 60,000 & 3073 & [60] \\ Greenhouse gas observing network & & 2921 & 5232 & [61] \\ \hline \end{tabular} 1

\end{table} 
Table 1: Brief description of data sets. **Remark 6**. : _At every iteration of the LMBNNR algorithm, we solve an optimization problem with the dimension \(H(m+2)+1\), where \(H\) is the number of hidden nodes and \(m\) is the number of features in training samples.