# A Piecewise Linear Regression Model Ensemble for Large-Scale Curve Fitting
## 6 Empirical Results
### Performance of the Reference Algorithm  
Since the selection of the size of each sub-sample is determinant in the final result, an initial analysis of the performance of the standard LHM learning algorithm when varying the number of points was carried out. It is clear that for the same noise level, the more input data we use as the training set, the lower error we achieve. First, in order to illustrate the differences between the models obtained from different values of \(N\), Figure 8 shows LHMs obtained from a small and a big sample size. It can be seen that by increasing \(N\) from 1 k to 100 k, we can reach a great improvement in the accuracy of the model. Using a small SLS, we obtain a 'weak' model, whereas with \(N\) = 100 k both the accuracy and complexity (\(K\)) of the model increase. \begin{table}
\begin{tabular}{l l} \hline \hline
**Problem** & **True Function** \\ \hline PR1 & \(f(x)=x+2x^{2}\) \\ PR2 & \(f(x)=40/(1+e^{x})\) \\ PR3 & \(f(x)=10-sin((\tau\cdot x)/2)\) \\ PR4 & \(f(x)=10+x^{3}\cdot sin(\tau\cdot x)\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Problem definition for testing the proposed methods. Figure 8: LHMs obtained in one replication using two different sample sizes: Problems PR1 (**top**) and PR4 (**bottom**), with datasets of \(N\) = 1 k (**left**) and \(N\) = 100 k (**right**). The black curve represents the true underlying function. This increasing complexity of the LHM with \(N\) is systematic in those problems. According to Figure 9, \(K\) continuously grows with \(N\). Note that its growth ratio decreases with \(N\), i.e., larger increments in \(N\) are required to produce an effective increase in \(K\). Furthermore, for a given sample size, there is a small dispersion around the median value of \(K\), so it can be concluded that the learning algorithm of the LHM estimates the complexity in a robust way. Regarding the error of these models, Figure 10 shows their MSE over the different datasets. It can be seen that, as expected, the MSE(TTS) decreases with increasing \(N\), and both MSE(SLS) and MSE(STS) stabilise around the variance of the noise. Despite having more and more knots when \(N\) increases, these models are not over-fitted, being more and more accurate. Therefore, we can conclude that the reference algorithm does a reasonable job by adapting the estimated model's complexity to the underlying structure's complexity and the amount of available data. Finally, in order to achieve a deeper knowledge of the performance of the reference algorithm, the MSE can be broken down more finely. It is well known that the MSE over the TTS can be decomposed in two terms (see, e.g., [4]):  
\[\text{MSE(TTS)}=bias^{2}+variance. \tag{10}\]  
The first term is the \(bias^{2}\), and it measures how closely the average model (given the set of models estimated from different samples of the same size) matches the true function. The second term represents the variance of those models from the average one, and it measures the certainty of the estimated models for a given sample size. Figure 11 shows this decomposition of the MSE(TTS) of the LHMs when varying \(N\). It can be seen that in both problems, most of the error is due to the variance, especially when the SLS is small. Furthermore, the bias and variance decrease rapidly when \(N\) increases. Comparing the results obtained from the different problems, it can be observed that the  
Figure 9: Distribution of \(K\) when varying the sample size \(N\) (calculated from 30 replications), for problems PR1 (**top**) and PR4 (**bottom**). For each \(N\) a compact boxplot shows the distribution of \(K\). The red line connects their median values. simplicity of PR1 results in a much lower error than the one obtained in PR4. In PR4, for very small sample sizes, a high bias is obtained due to the fact that the algorithm is not able to recover the most complex part of the underlying structure. However, in PR1 the bias obtained even with small values of \(n\) is quite small compared with the total error.