# Nonsmooth Optimization-Based Hyperparameter-Free Neural Networks for Large-Scale Regression
## 1 Introduction  
Regression models and methods are extensively utilized for prediction and approximation in various real-world scenarios. When data involve complex relationships between the response and explanatory variables, regression methods designed using _neural networks_ (NNs) have emerged as powerful alternatives to traditional regression methods [1]. Hence, we focus on _NNs for regression_ (NNR): we introduce a new approach for modeling and solving regression problems using fully-connected feedforward NNs with the _rectified linear unit_ (ReLU) activation function, the \(L_{1}\)-loss function, and the \(L_{1}\)-regularization. We call this problem the ReLU-NNR problem, and the method for solving it is the _limited memory bundle_ NNs (LMBNNR) algorithm.  
The ReLU-NNR problem is nonconvex and nonsmooth. Note that ReLU itself is nonsmooth. Thus, even if we used a smooth loss function and regularization, the underlying optimization problem would still be nonsmooth. Conventional global optimization methods, including those based on global random search, become time-consuming as this problem contains a large number of variables. In addition, the authors in [2] showed that it is impossible to give any guarantee that the global minimizer is found by a general global (quasi-)random search algorithm with reasonable accuracy when the dimension is large. On the other hand, local search methods are sensitive to the choice of starting points and, in general, end up at the closest local solutions, which may be significantly different from the global ones. To address the nonconvexity of the ReLU-NNR problem, we propose a constructive approach. More specifically, we construct initial weights by using the solution from the previous iteration. Such an approach allows us to find either global or deep localminimizers for the ReLU-NNR problem. To solve the underlying nonsmooth optimization problems, we apply a slightly modified version of the _limited memory bundle method_ (LMBM) developed by [PERSON] (nee [PERSON]) [3; 4]. We use this method since it is one of the few algorithms capable of handling large dimensions, nonconvexity, and nonsmoothness all at once. In addition, the LMBM has already proven itself in solving machine learning problems such as clustering [5], cardinality and clusterwise linear regression [6; 7], and missing value imputation [8].  
We consider NNs with only one hidden layer but the number of hidden nodes is determined automatically using a novel constructive approach and an _automated stopping procedure_ (ASP). More precisely, the number of nodes is determined incrementally by starting from one node. ASP bases the intelligent selection of initial weights on the iteratively self-updated regularization parameter. It is applied at each iteration of the constructive algorithm to stop training if there is no further improvement in the model. Thus, there are no tuneable hyperparameters in LMBNNR.  
To summarize, the LMBNNR algorithm has some remarkable features, including:  
* It takes full advantage of nonsmooth models and nonsmooth optimization in solving NNR problems (no need for smoothing etc.);
* It is hyperparameter-free due to the automated determination of the proper number of nodes;
* It is applicable to large-scale regression problems;
* It is an efficient and accurate predictive tool.  
The structure of the paper is as follows. An overview of the related work is given in Section 2, while Section 3 provides the theoretical background on nonsmooth optimization and NNR. The problem statement--the ReLU-NNR problem--is given in Section 4, and in Section 5, the LMBNNR algorithm together with ASP is introduced. In Section 6, we present the performance of the LMBNNR algorithm and compare it with some state-of-the-art NNR algorithms. Section 7 concludes the paper.