# Nonsmooth Optimization-Based Hyperparameter-Free Neural Networks for Large-Scale Regression
## 5 LMBNNR Algorithm
Compute \(\xi_{1}\in\partial f_{H}(u_{k}^{H})\);  Set \(k=1\), \(k=1\), \(d_{1}=-\xi_{1}\), \(\xi_{1}=\xi_{1}\), and \(\beta_{1}=0\); while\(\leq k_{\text{max}}\) and the termination condition \(\xi_{k}^{-}d_{k}+2\beta_{k}\leq\varepsilon\) is not metdo  Find step sizes \(t_{1}^{i}\) and \(t_{k}^{i}\) and the subgradient locality measure \(\beta_{k+1}\);  Set \(u_{k+1}^{H}=u_{k}^{H}+t_{k}^{i}d_{k}\) and \(u_{k+1}=u_{k}^{H}+t_{k}^{i}d_{k}\);  Evaluate \(f_{H}(u_{k+1}^{H})\) and \(\xi_{k+1}\in\partial f_{H}(u_{k+1})\);  Store the new correction vectors \(s_{k}=\eta_{k+1}-u_{k}^{H}\) and \(u_{k}=\xi_{k+1}-\xi_{k}\);  Set \(\hat{m}_{k}=\min(s_{k},\hat{m}_{k})\); if\(t_{1}^{i}>0\)then (Serious step)  Compute the search direction \(d_{k+1}=-D^{k}\xi_{k+1}\), where \(D^{k}\) is calculated using the L-BFGS update with \(\hat{m}_{k}\)  most recent correction vectors;  Set \(\hat{k}=k+1\) and \(\hat{\beta}_{k+1}=0\); else (Null step)  Determine multipliers \(\lambda_{k}^{1}\) satisfying \(\lambda_{k}^{1}\geq 0\) for all \(i\in\{1,2,3\}\), and \(\sum_{i=1}^{3}\lambda_{i}^{k}=1\) that minimize the function \[w^{1}=\left(\frac{1}{m^{\prime}},\frac{1}{m},\ldots,\frac{1}{m},0,1,0\right)^{ \top}\in\mathbb{R}^{m+3}.\]1 and compute the aggregate values \[w^{1}=\left(\frac{1}{m^{\prime}},\frac{1}{m},\ldots,\frac{1}{m},0,1,0\right)^{ \top}\in\mathbb{R}^{m+3}.\]2  Compute the search direction \(d_{k+1}=-D^{k}\xi_{k+1}\), where \(D^{k}\) is calculated using the L-SRI update with \(h_{k}\)  most recent correction vectors;  end while Set \(k=k+1\);  end while
```

**Algorithm 2**LMBM for ReLU-NNR problems

We now recall the convergence properties of the LMBM in the case of ReLU-NNR problems. Since the objective function \(f_{H}:\mathbb{R}^{H(m+2)+1}\rightarrow\mathbb{R}\) is locally Lipschitz continuous and upper semi-smooth (see e.g., [47]), and the level set \(\{\,w^{H}\in\mathbb{R}^{H(m+2)+1}\mid f_{H}(w^{H})\leq f_{H}(w^{H}_{1})\,\}\) is bounded for every starting weight \(w^{H}_{1}\in\mathbb{R}^{H(m+2)+1}\), all the assumptions needed for the global convergence of the original LMBM are satisfied. Therefore, the theorems on the convergence of the LMBM proved in [3, 4] can be modified for the ReLU-NNR problems as follows. Theorem 1: _If the LMBM terminates after a finite number of iterations, say at iteration \(k\), then the weight \(w^{H}_{k}\) is a stationary point of the ReLU-NNR problem (3)._

Theorem 2: _Every accumulation point \(\varpi^{H}\) of the sequence \(\{w^{H}_{k}\}\) generated by the LMBM is a stationary point of the ReLU-NNR problem (3)._

## 6 Numerical Experiments

Using some real-world data sets and performance measures, we evaluate the performance of the proposed LMBNNR algorithm. In addition, we compare it with three different widely used NNR algorithms whose implementations are freely available. That is, the backpropagation NNR (BPN) algorithm utilizing TensorFlow ([[https://www.tensorflow.org/](https://www.tensorflow.org/)]([https://www.tensorflow.org/](https://www.tensorflow.org/))(accessed on 10 September 2023)), Extreme Learning Machine (ELM) [48], and Monotone Multi-Layer Perceptron Neural Networks (MONMLP) [49]. ### Data Sets and Performance Measures

A brief description of data sets is given in Table 1 and the references therein. The data sets are divided randomly into training (80%) and test (20%) sets. To get comparable results, we use the same training and test sets for all the methods. We apply the following performance measures: root mean square error (RMSE), mean absolute error (MAE), coefficient of determination (\(R^{2}\)), and Pearson's correlation coefficient (\(r\)) (see the Supplementary Material for more details). ### Implementation of Algorithms

The proposed LMBNNR algorithm was implemented in Fortran 2003, and the computational experiments were carried out on an iMac (macOS Big Sur 11.6) with a 4.0 GHz Intel(R) Core(TM) i7 machine and 16 GB of RAM.