# A Piecewise Linear Regression Model Ensemble for Large-Scale Curve Fitting
## 3 Parameter Estimation from Small Datasets  
As already mentioned, the previous model is completely defined by the pairs of coordinates \((k_{j},h_{j})\) of the knots. Hence, there are three main groups of parameters to fix: the number of knots (\(K\)), that will determine the complexity of the resulting model; their x-coordinates \((k_{j})\), that will set the basis functions \(B_{j}\); and their y-coordinates \((h_{j})\).  
In the simplest case, let us assume that the number and the x-positions of the knots are fixed, and the only parameters that must be estimated are the \(h_{j}\). Given the scattered data, this problem can be directly solved by using OLS, or in the more general case, when dealing with weighted input data, by applying weighted least squares (WLS). In matrix terms,  
\[\mathbf{h}=\mathbf{A}^{-1}\mathbf{B}^{T}\mathbf{W}\mathbf{y}=(\mathbf{B}^{T} \mathbf{W}\mathbf{B})^{-1}\mathbf{B}^{T}\mathbf{W}\mathbf{y}, \tag{4}\]  
where \(\mathbf{h}=(h_{1},h_{2},...,h_{K})^{T}\), \(\mathbf{B}\) is an \(N\times K\) matrix containing all the basis functions evaluated in every x-coordinate of the dataset, \(\mathbf{y}=(y_{1},y_{2},...,y_{N})^{T}\), and \(\mathbf{W}\) is an \(N\times N\) diagonal matrix with the weights of all the points.  
Figure 1: Continuous piecewise linear regression model. **Top:** Scattered data (N = 150 points) and fitted piecewise linear regression model, composed of \(K=6\) knots, plus and minus pointwise 99% standard error bands. **Bottom:** Set of six linear basis functions corresponding to the top model. The broken vertical lines indicate the x-positions of the knots.  
Having estimated the parameters \(\mathbf{h}\) by using Equation (4), their standard error can be also estimated as  
\[\mathbf{se}(\mathbf{h})=\mathbf{A}^{-1}\mathbf{B}^{T}\mathbf{W}\sigma_{y}^{2} \mathbf{W}^{T}\mathbf{B}\mathbf{A}^{-1}\mathbf{y}, \tag{5}\]  
where \(\sigma_{y}^{2}\) is the variance of the output variable \(y\).  
In Figure 1, the confidence band around the piecewise linear model represents an estimated 99% pointwise standard error band, calculated in this way by applying Equation (5).  
In the second case, when the number of knots (\(K\)) is given but their x-positions are unknown, there is no straightforward technique to fix them optimally. However, there are several different approaches in the literature, from the simplest ones, such as placing them uniformly in x or in its quantiles [10], to more sophisticated methods, such as [11; 12].  
Finally, the most complex case is the one in which both the complexity (\(K\)) of the model and x-position of the knots (\(k_{j}\)) must be estimated. According to [4], this situation changes the regression problem from a straightforward linear problem, solved by standard least squares, to a combinatorially hard non-linear problem. In order to deal with it, greedy algorithms or two-stage processes are required.  
This last case is addressed in [9], where the LHM is presented. The LHM is a continuous piecewise linear model, given by Equation (3), in which all the parameters are automatically estimated by applying a stage-wise greedy approach. A new version of its learning algorithm has been developed in this paper to be used as the fundamental piece of the models shown in the following sections. This new version follows the same strategy and steps as the former one, adding some performance improvements, but the major change that has been introduced is the possibility of using weighted data. In the Appendix A, a brief summary of the LHM's learning algorithm is included, as well as some additional references.