# Algorithms for Drug Sensitivity Prediction
## 4 Predictive Models
### Linear Regression Models  
In a predictive modeling setting, linear regression is one of the most commonly-applied models. Linear regression has been extensively studied and with it the statistical properties of the parameter estimation for linear models, which are easier to determine as compared to the non-linear scenario. To explain a linear model, consider \(x(i,j)\) and \(y(i)\) (\(i=1,\ldots,n;j=1,\ldots,M\)) to denote the training predictor features and output response samples, respectively. A linear regression model considers the generation of regression coefficients \(\beta_{1},\beta_{2},\ldots,\beta_{M}\), such that \(y(i)-\sum_{j=1}^{M}\beta_{j}x(i,j)=e_{i}\) is minimized. The minimization of the error term \(e_{i}\) is usually considered in the sum of squared error sense, i.e., minimize \(\sum_{i=1}^{n}\epsilon_{i}^{2}\). For linear regression, we generally incorporate a constant term \(\beta_{M+1}\) commonly known as the intercept, i.e., \(e_{i}=y(i)-\sum_{j=1}^{M}\beta_{j}x(i,j)-\beta_{M+1}\). In terms of matrix notation, we can write it as follows:  
\[\mathbf{Y}=\mathbf{X}\beta+\epsilon \tag{2}\]  
\[\mathbf{Y}= \begin{bmatrix}y(1)\\ y(2)\\ \cdot\\ y(n)\end{bmatrix} \quad\mathbf{X}= \begin{bmatrix}x(1,1)&x(1,2)&\cdots&x(1,M)&1 \\ x(2,1)&x(2,2)&\cdots&x(2,M)&1\\ \cdot&\cdot&\cdots&\cdot&\cdot\\ \cdot&\cdot&\cdots&\cdot&\cdot\\ x(n,1)&x(n,2)&\cdots&x(n,M)&1\end{bmatrix} \quad\beta= \begin{bmatrix}\beta_{1 }\\ \beta_{2}\\ \cdot\\ \cdot\\ \beta_{M+1}\end{bmatrix} \quad\epsilon= \begin{bmatrix}\epsilon_{1}\\ \epsilon_{2}\\ \cdot\\ \cdot\\ \epsilon_{n}\end{bmatrix} \]  
The vector \(\mathbf{Y}\) consists of the response or dependent variables; \(\mathbf{X}\) consists of the regressors, predictor or independent variables; vector \(\beta\) is the parameter vector that needs to be estimated; and \(\epsilon\) denotes the error or disturbance terms. Note that the development of the linear regression framework consists of a linear combination of parameters \(\beta_{i}\) and predictor variables. Transforming the predictor variables to higher orders can allow us to utilize the framework in modeling with nonlinear terms, which is known as polynomial regression [53]. Ordinary least squares is the simplest form of linear regression where our goal is to select \(\beta\), such that it minimizes \(\sum_{i=1}^{n}\epsilon_{i}^{2}=(\mathbf{Y}-\mathbf{X}\beta)^{T}(\mathbf{Y}- \mathbf{X}\beta)\). Ordinary least squares-based linear regression models are straightforward to understand and relatively simple to interpret in terms of feature weights and their positive or negative influences. However, it is not suitable to solve the large feature dimension problem with limited samples, and overfitting is a common problem. Some of the assumptions of ordinary least squares are not satisfied in drug sensitivity studies, such as output response being a linear summation of covariates, and thus, ordinary least squares is not the best approach in terms of predictive accuracy. Thus, ordinary least squares are suitable for scenarios where the number of features has already been reduced to a smaller set, and we are interested in an easily interpretable and robust model rather than a highly accurate one. A commonly-used modification to ordinary least squares is to add a regularization term in the cost function to avoid overfitting. Regularization usually consists of adding a penalty on the complexity of the model, which in regression is often included in the form of \(||\beta||_{1}\) or \(||\beta||_{2}\) norms added to the cost function \((\mathbf{Y}-\mathbf{X}\beta)^{T}(\mathbf{Y}-\mathbf{X}\beta)\).