# MFCC Selection by LASSO for Honey Bee Classification
## 2 Materials and Methods
### Feature Selection via LASSO Regression  
In this section, we introduce feature selection techniques: the LARS (Least Angle Regression) regularization algorithm and its modification LASSO (Least Absolute Shrinkage and Selection Operator). The regularization LARS [36] method differs from LASSO [37] regularization mainly in the way variables are selected for the model--with regularization in \(\ell 2\)-norm for LARS and in \(\ell 1\)-norm for the LASSO algorithm. In the case of LARS, variables are added to the model sequentially based on their correlation with the residuals. At each step, LARS selects the variable with the highest correlation, and then increases its coefficient evenly until the next variable reaches the same correlation. At this point, two variables are added simultaneously, and their coefficients are increased at the same rate. This process is repeated until all variables have been added or a specified limit on the number of variables is reached. The LARS regularization minimizes the residual sum of squares \(RSS\) (in \(\ell^{2}\)-norm) as follows:  
\[\hat{\beta}^{LARS}=\arg\min_{\beta}\Bigl{\{}||y-X\beta||_{2}^{2}\Bigr{\}}, \tag{4}\]  
where \(X=(X_{ij})_{i=1,\ldots,N}^{j=1,\ldots,p}\) is a matrix of signal features, with feature vectors \((X_{i1},X_{i2},\ldots,X_{ip})\) in the \(i\)th row for the right signal from the training set counting \(N\) recordings, and \(y\) is a vector of class labels \(y_{i}\in\{-1,1\}\). In our experiments, the labels are used for feature vectors representing worker bee and drone bee recordings, and the features \(X_{ij}\) are calculated as MFCC coefficients. The \(\beta\) is a \(p\)-dimensional vector of regression coefficients \(\beta_{j}\), that is, \(\beta=(\beta_{1},\beta_{2},\ldots,\beta_{p})\). The regression coefficients \(\beta\) are obtained due to minimization of the residual sum of squares \(RSS=y-X\beta\) in \(\ell^{2}\)-norm. In the case of LASSO regularization, however, variables are also added to the model sequentially but in a more rigorous manner. At each step, LASSO selects the variable that has the largest influence on the residuals, then increases its coefficient while decreasing the coefficients of the remaining variables. Variables that do not contribute to the model have \(\beta_{j}\) coefficients equal to 0. This process is repeated until all variables have been added to the model, or until certain constraints on the number of variables, or the sum of the absolute values of the coefficients, have been reached. The LASSO algorithm can be defined by the following expression:  
\[\hat{\beta}^{LASSO}=\arg\min_{\beta}\Bigl{\{}\|y-X\beta\|_{2}^{2}\Bigr{\}} \text{ subject to }\|\beta\|_{1}\leq t, \tag{5}\]  
where \(\|u\|_{p}=\left(\sum_{i=1}^{N}|u_{i}|^{p}\right)^{1/p}\) is the standard \(\ell^{p}\)-norm. The above expression can be written more compactly in the Lagrangian form  
\[\hat{\beta}^{LASSO}=\arg\min_{\beta}\biggl{\{}\frac{1}{N}\|y-X\beta\|_{2}^{2}+ \alpha\|\beta\|_{1}\biggr{\}}, \tag{6}\]  
where the exact relationship between the regularization parameters \(t\) and \(\alpha\) is data-dependent. Finally, the formula takes the following form:  
\[\hat{\beta}^{LASSO}=\arg\min_{\beta}\biggl{\{}||y-X\beta||_{2}^{2}+\alpha\sum _{j=1}^{p}|\beta_{j}|\biggr{\}}. \tag{7}\]The coefficients \(\hat{\beta}^{LASSO}\), obtained by the LASSO algorithm, for the sequentially decreasing values of the regularization parameter \(\alpha\) are presented in Figure 4. Each colorful line in Figure 4 represents the values of an individual coefficient \(\beta_{j}\) from the \(\hat{\beta}^{LASSO}\) vector for different \(\alpha\) values. Stopping the regularization procedure for a chosen \(\alpha\) value results in obtaining a set of non-zero coefficients \(\beta_{j}\). For \(\alpha=0\), we would obtain the LARS method, and all coefficients \(\beta_{j}\) would be non-zero--compare expressions in (4) and (7). Regularization by LASSO can be understood as a type of forward stepwise feature selection technique.