## 2 Theory
### Prediction Models
#### 2.2.2 Averaging PLSR  
Assume that \(\mathbf{x}_{new}\) is a new observation to predict, and that \(\hat{y}_{new,A}\) is the prediction returned by the PLSR-averaging model having a number \(A\) of LVs (in practice, \(A\) will be larger than the dimensionality \(a\) selected in the usual PLSR, i.e., corresponding to the minimum prediction error rate). We define the averaging model prediction by:  
\[\hat{y}_{new,\;\text{avg}[A]}=w_{0}\hat{y}_{new,0}+w_{1}\hat{y}_{new,1}+\ldots+ w_{A}\hat{y}_{new,A} \tag{2}\]  
where \(w_{r}\) (\(r=0,\ldots,A\)) is the weight (bounded between 0 and 1) of the model with \(r\) LVs, with the constraint:  
\[\sum_{r=0}^{A}w_{r}=1.\]  
As indicated in Section 2.2.1, \(\hat{y}_{new,0}\) (\(r=0\) LV) is the simple mean of \(\mathbf{y}\). Vector \(\mathbf{w}=\{w_{0},w_{1},\ldots,w_{A}\}\) represents a pattern of weights. The shape of this pattern is specific to a given averaging method.  
Five PLSR-averaging methods (patterns \(\mathbf{w}\)) are considered in this paper. The first method (AVG) assumes simply a uniform importance for each model, i.e.,  
\[w_{r}=1/(A+1).\]  
The three next methods (referred to as AVG-CV, AVG-AIC, and AVG-BIC, respectively) are detailed in Section 2.2.3. All assume that weight \(w_{r}\) decreases when the performance of prediction of the model decreases. Assume that \(d_{r}\) is a prediction error rate estimated on the training data \(\{\mathbf{X},\mathbf{y}\}\) for the PLSR model with \(r\) LVs. In the three methods, pattern \(\mathbf{w}\) is computed from the rates \(\{d_{0},d_{1},\ldots,d_{A}\}\) given as input data in a bi-square weighting function [13]. Examples of bi-square function curves are given in Figure 1. The principle is as follows. Let us note \(\widetilde{d}_{r}\) the error rate normalized to an upper limit \(d_{\text{up}}\) that represents the value above which the model with \(r\) LVs is removed from the average displayed in Equation (2):  
* \(\widetilde{d}_{r}=d_{r}/d_{\text{up}}\) if \(d_{r}<d_{\text{up}}\)
* \(\widetilde{d}_{r}=1\) if \(d_{r}\geq d_{\text{up}}\) (this case implies a null weight). The bi-square weights are defined by \[\widetilde{w}_{r}=(1-\widetilde{d}_{r}^{2})^{2}.\] In practice, PLSR models with \(d_{r}<d_{\text{up}}\) have a decreasing weight \(\widetilde{w}_{r}\) when their error rate \(d_{r}\) increases. Models with \(d_{r}\geq d_{\text{up}}\) receive a null weight (\(\widetilde{w}_{r}=0\)). To compute the average in Equation (2), the final weights are normalized to sum to 1, i.e., \[w_{r}=\widetilde{w}_{r}/\sum_{r=0}^{A}\widetilde{w}_{r}.\] (3)  
The scalar \(d_{\text{up}}\) is defined by the quantile of order \(1-\alpha\) (where \(\alpha\) is a parameter to set between 0 and 1) of the set \(\mathbf{d}=\{d_{0},d_{1},\ldots,d_{A}\}\), i.e., the set of the error rates estimated for the \(A+1\) PLSR models initially considered in the average. In other words, the 100 \(\times\) a less performant models are removed from the average (Equation (2)). Increasing \(\alpha\) allows for selecting only the models with the better performances that, in the end, may be expected to return better predictions.  
Finally, the fifth method of computing \(\mathbf{w}\) (AVG-SHENK) was proposed by Shenk et al. [14] for their local (i.e., \(k\)-nearest-neighbors based method) PLSR algorithm referred to as \"LOCAL\". For each new observation \(\mathbf{x}_{\text{new}}\) to predict and the PLSR model with \(r\) LVs, the weight \(\widehat{w}_{r}\) is defined by the product between the root mean squared \(\mathbf{x}\)-residuals for \(\mathbf{x}\) (i.e., the residuals between \(\mathbf{x}_{\text{new}}\) and its projection to the model PLS space) and the norm of the \(\mathbf{b}\)-coefficients vector. As before, the final weights are normalized to sum to 1, such as in Equation (3).