## 3 Methodology
### Classical Normal Linear Regression Model
The sample mean of the resubstitution OLS residuals (14) will always be zero:  
\[\overline{e}=\frac{1}{n-p}\underset{i=1}{\overset{n}{\sum}}e_{i}=0 \tag{16}\]  
Utilizing (16), the sample standard deviation of the resubstitution OLS residuals (14) (aka standard error estimate) can be obtained by:  
\[\partial_{\varepsilon}=\sqrt{\frac{1}{n-p}\underset{i=1}{\overset{n}{\sum}}e_ {i}^{2}} \tag{17}\]  
According to (13), the point estimate \(\overset{\rightarrow}{b}=\left(b_{1},b_{2},\ldots,b_{p}\right)^{T}\) of the parameter vector \(\overset{\rightarrow}{\beta}\) depends on the vector of the observed values \(\overset{\rightarrow}{y}=(y_{1},y_{2},\ldots,y_{n})^{T}\). The latter contains \(n\) random variates of the dependent variable \(Y\) (the hydrodynamic efficiency) which is an r.v. It follows that each component, \(b_{i}\), of \(\overset{\rightarrow}{b}\) is an r.v. and the latter is a random vector. In that sense, the predicted value \(\hat{y}\) of the r.v. \(Y\) according to (14) is an estimate of the systematic component. Its randomness is determined by the random estimate \(\overset{\rightarrow}{b}\) of the parameter vector \(\overset{\rightarrow}{\beta}\), so the randomness of \(Y\) is determined both by the randomness of the stochastic component and by the randomness of the estimate of the systematic component. According to [42], the covariance matrix of \(\overset{\rightarrow}{b}\) is a \([p\times p]\) symmetric matrix:  
\[K_{\overset{\rightarrow}{b}}=\hat{\sigma}_{\varepsilon}^{2}\left(\mathbf{X}^{T} \mathbf{X}\right)^{-1} \tag{18}\]  
From (18), we can find the estimated standard error of the parameter estimates \(b_{i}\) as square root of the corresponding diagonal element of \(K_{\overset{\rightarrow}{b}}\):  
\[se\left(b_{i}\right)=\sqrt{K_{\overset{\rightarrow}{b}}^{-}[j,j]}\text{, for }j=1,2,\ldots,p \tag{19}\]  
Using the OLS residuals (15) the model performance resubstitution indicators _RMSE_, _MAE_, and \(R^{2}\) can be estimated by (1), (2), and (3). Typically, for the MLR model an adjusted coefficient of determination \(R_{adj}^{2}\) is used which improves the \(R^{2}\) by considering the degrees of freedom (that is the count of the parameters used in the model):  
\[\overrightarrow{b}=arg\underset{\overrightarrow{\beta}}{min}\left\{nRMSE^ {2}\!\left(\overrightarrow{\beta}\right)\right\}=arg\underset{ \overrightarrow{\beta}}{min}\left\{\sum\limits_{i=1}^{n}\left[y_{i}- \overrightarrow{y}_{i}\!\left(\overrightarrow{\beta}\right)\right]^{2}\right\} \tag{12}\]0The resubstitution measure \(R_{adj}^{2}\) is useful when it is positive, otherwise the developed MLR model is inadequate. One of the amazing properties of the MLR is that it can cheaply calculate the holdout predicted residual for the \(i\)th observation if the MLR model is built using all other observations in the dataset. Such a residual is also known as a leave-one-out (LOO) residual and can be denoted as \(e_{i,HO}\) according to Section 3.1:  
\[\overrightarrow{b}=arg\underset{\overrightarrow{\beta}}{min}\left\{nRMSE^ {2}\!\left(\overrightarrow{\beta}\right)\right\}=arg\underset{ \overrightarrow{\beta}}{min}\left\{\sum\limits_{i=1}^{n}\left[y_{i}- \overrightarrow{y}_{i}\!\left(\overrightarrow{\beta}\right)\right]^{2}\right\} \tag{12}\]1  
Here, the \(i\)th row of the design matrix (10) is denoted as \(row_{i}^{\to X}\), which is a \(p\)-dimensional row vector [43]. The LOO formula (21) is used \(n\) times (once for each record in the dataset), which corresponds to a test set with \(n_{HO}=n=126\). Then the holdout residuals can be substituted in (1), (2), and (3) to produce the three holdout measures of the MLR data set \(RMSE_{HO}\), \(MAE_{HO}\), and \(R_{HO}^{2}\). Each of the predicted residuals \(e_{i,HO}\) is obtained by a model \(MOD_{i}\), which is marginally worse than the MLR model (14) since \(MOD_{i}\) neglects the observation \(\left\langle\overrightarrow{x}_{i},y_{i}\right\rangle\) in its training set.