## 5 Discussion
### Comparison of Prediction Accuracy of Classical and Penalized Methods Using CV as Criterion  
We evaluated the prediction accuracy of classical variable selection methods (BE, BSS, and FS) and penalized regression methods (NNG, lasso, ALASSO, and RLASSO), using CV as the model selection criterion. BSS, BE, and FS performed quite similarly in most scenarios; however, in certain scenarios, such as large sample sizes with high correlation and high SNR, BE and FS had a slight edge (see Figure A5 in Appendix A). This difference can be attributed to the nature of BSS, which considers a much larger set of models compared to BE and FS. While this extensive search may seem advantageous, it can lead to overfitting and high variance in coefficient estimates, resulting in poor generalization [4; 43]. FS outperformed BE in high-correlation settings with low-to-moderate SNR. This advantage is likely due to the sequential nature of FS, which begins with a simple model and gradually adds variables. In contrast, BE starts with a more complex model and removes variables, which may be less effective when information is limited and the risk of overfitting is high. [PERSON] [46] demonstrated the advantages of BE over FS and concluded that BE is more appropriate than FS. This is not consistent with our findings, likely because their evaluation focused on variable selection rather than prediction performance. In addition, they did not consider a broader range of scenarios. Overall, the predictive accuracy of classical methods was inferior to those of penalized methods, especially in settings with small sample sizes, high correlation among covariates, and low SNR (i.e., SNR \(<\) 1), which aligns with previous findings [40]. In such scenarios, classical methods are prone to instability and high variance due to their discrete nature of either retaining or discarding variables in the model, unlike penalized methods, which apply continuous shrinkage to coefficients estimates [9; 21; 40]. Some researchers [41] have argued against the use of classical variable selection methods, citing their limitations in settings with a large number of potential predictors. However, our findings indicate that classical methods can outperform modern variable selection methods in both predictive accuracy and model simplicity under certain settings. This observation is supported by other studies [24; 40; 42], which have reported similar results under comparable conditions. To improve the prediction performance of classical methods, stacking and post-estimation shrinkage have been proposed [42; 61]. Instead of selecting a single best model for prediction, stacking combines linear predictors from various models with different numbers of covariates and estimates optimal weights for each linear predictor [10; 61]. While this approach can improve prediction, it sacrifices interpretability compared to selecting a single model [10]. Alternatively, post-estimation shrinkage applies shrinkage (estimated via CV) to the regression estimates of the best-selected model [22; 38; 42; 62]. Although stacking and post-estimation shrinkage both offer ways to enhance prediction accuracy, they are beyond the scope of this study. In settings with large sample sizes, low correlation, and high SNR, classical methods outperformed lasso, which is consistent with previous findings [24; 40]. In these scenarios, classical methods produced predictions comparable to those of NNG, RLASSO, and ALASSO. Lasso selected too many variables (Figure A6), which, combined with its tendency to over-shrink large effects, led to inferior prediction accuracy despite the availability of sufficient information [15; 24; 40]. These results are in agreement with findings from [24; 40; 63]. Among the penalized methods, lasso consistently outperformed NNG, ALASSO, and RLASSO in scenarios with small sample sizes and low SNR, as well as in high correlation.