Table 16: Benchmarking feature selection methods for MLP and FT-Transformer downstream models on datasets with random extra features. We report performance of models trained on features selected by different FS algorithms in terms of accuracy for classification and negative RMSE for regression problems. % refers to percent of extra features in the dataset: either  $50\%$  or  $75\%$  features are second-order. Bold font indicates the best numbers dataset-wise and lower rank indicates better overall result.
%	FS method	AL	CH	CO	EY	GE	HE	HI	HO	JA	MI	OT	YE	rank
50	XGBoost Baseline	0.923±0.0	-0.429±0.001	0.969±0.0	0.709±0.003	0.683±0.003	0.374±0.001	0.729±0.001	-0.541±0.002	0.726±0.001	-0.846±0.0	0.825±0.001	-0.781±0.0
50	No FS + MLP	0.941±0.001	-0.48±0.007	0.961±0.001	0.538±0.005	0.466±0.007	0.366±0.001	0.798±0.001	-0.622±0.007	0.703±0.001	-0.911±0.002	0.773±0.004	-0.801±0.001	8.08
50	Univariate + MLP	0.96±0.001	-0.447±0.004	0.97±0.0	0.575±0.01	0.515±0.005	0.379±0.002	0.811±0.001	-0.549±0.008	0.715±0.002	-0.891±0.001	0.808±0.001	-0.776±0.001	2.66
50	Lasso + MLP	0.949±0.001	-0.454±0.004	0.969±0.001	0.547±0.016	0.458±0.018	0.38±0.002	0.812±0.0	-0.599±0.003	0.715±0.003	-0.907±0.001	0.805±0.004	-0.787±0.001	5.91
50	IL Lasso + MLP	0.952±0.001	-0.451±0.003	0.969±0.001	0.564±0.019	0.474±0.015	0.375±0.003	0.811±0.001	-0.568±0.005	0.715±0.004	-0.897±0.0	0.796±0.008	-0.773±0.001	4.91
50	AGL + MLP	0.958±0.001	-0.512±0.064	0.969±0.0	0.578±0.017	0.473±0.018	0.386±0.003	0.81±0.001	-0.557±0.003	0.718±0.003	-0.898±0.001	0.799±0.005	-0.778±0.001	4.33
50	LassoNet + MLP	0.954±0.001	-0.445±0.004	0.969±0.001	0.552±0.006	0.495±0.01	0.385±0.002	0.811±0.003	-0.557±0.008	0.715±0.004	-0.907±0.001	0.783±0.003	-0.787±0.002	5.16
50	AM + MLP	0.953±0.001	-0.444±0.003	0.968±0.001	0.554±0.016	0.498±0.007	0.382±0.002	0.813±0.002	-0.566±0.012	0.722±0.002	-0.904±0.003	0.801±0.003	-0.777±0.002	3.83
50	RF + MLP	0.955±0.001	-0.453±0.003	0.969±0.0	0.589±0.011	0.594±0.023	0.386±0.002	0.814±0.001	-0.572±0.006	0.72±0.001	-0.904±0.0	0.806±0.002	-0.786±0.001	2.91
50	XGBoost + MLP	0.956±0.001	-0.444±0.002	0.969±0.0	0.59±0.015	0.502±0.013	0.385±0.002	0.812±0.0	-0.56±0.014	0.72±0.003	-0.893±0.001	0.805±0.003	-0.777±0.001	2.33
50	Deep Lasso + MLP	0.959±0.001	-0.443±0.002	0.968±0.001	0.573±0.007	0.485±0.006	0.383±0.002	0.814±0.001	-0.549±0.005	0.72±0.003	-0.894±0.001	0.802±0.002	-0.776±0.001	2.33
75	XGBoost Baseline	0.917±0.0	-0.414±0.003	0.968±0.0	0.74±0.002	0.691±0.003	0.376±0.001	0.731±0.001	-0.542±0.002	0.725±0.001	-0.849±0.0	0.825±0.001	-0.789±0.0
75	No FS + MLP	0.925±0.002	-0.527±0.006	0.955±0.001	0.502±0.007	0.417±0.005	0.348±0.002	0.778±0.002	-0.674±0.007	0.671±0.002	-0.917±0.001	0.749±0.003	-0.812±0.001	7.41
75	Univariate + MLP	0.96±0.001	-0.447±0.004	0.97±0.0	0.575±0.008	0.502±0.005	0.381±0.002	0.81±0.001	-0.549±0.008	0.713±0.001	-0.89±0.001	0.806±0.003	-0.776±0.0	2.50
75	Lasso + MLP	0.959±0.001	-0.454±0.002	0.967±0.0	0.543±0.013	0.491±0.009	0.381±0.002	0.811±0.001	-0.612±0.004	0.716±0.002	-0.907±0.001	0.802±0.003	-0.789±0.001	4.33
75	IL Lasso + MLP	0.957±0.002	-0.448±0.004	0.968±0.001	0.555±0.015	0.432±0.056	0.38±0.002	0.809±0.001	-0.572±0.013	0.717±0.001	-0.903±0.003	0.799±0.003	-0.775±0.002	4.41
75	AGL + MLP	0.954±0.002	-0.447±0.003	0.968±0.001	0.561±0.013	0.429±0.03	0.382±0.002	0.809±0.005	-0.571±0.012	0.719±0.003	-0.901±0.0	0.762±0.035	-0.777±0.001	4.33
75	LassoNet + MLP	0.958±0.001	-0.452±0.006	0.966±0.002	0.528±0.009	0.475±0.005	0.383±0.004	0.809±0.003	-0.555±0.014	0.705±0.004	-0.913±0.001	0.768±0.003	-0.794±0.002	4.75
75	RF + MLP	0.949±0.001	-0.453±0.003	0.968±0.001	0.584±0.008	0.61±0.008	0.386±0.002	0.814±0.001	-0.585±0.012	0.718±0.002	-0.902±0.0	0.808±0.003	-0.784±0.001	2.91
75	XGBoost + MLP	0.958±0.001	-0.451±0.003	0.969±0.0	0.576±0.007	0.583±0.015	0.382±0.002	0.81±0.001	-0.568±0.009	0.72±0.002	-0.892±0.0	0.804±0.002	-0.774±0.001	2.08
75	Deep Lasso + MLP	0.957±0.001	-0.446±0.004	0.969±0.0	0.569±0.013	0.479±0.007	0.387±0.001	0.814±0.001	-0.559±0.015	0.721±0.002	-0.893±0.01	0.8±0.002	-0.774±0.001	2.33
50	No FS + FT	0.959±0.002	-0.432±0.002	0.966±0.001	0.673±0.015	0.5±0.004	0.384±0.002	0.817±0.001	-0.577±0.005	0.73±0.002	-0.902±0.04	0.813±0.003	-0.792±0.002	6.58
50	Univariate + FT	0.963±0.001	-0.424±0.002	0.97±0.001	0.7±0.009	0.519±0.01	0.389±0.002	0.819±0.001	-0.554±0.003	0.733±0.003	-0.897±0.04	0.819±0.001	-0.789±0.001	2.83
50	Lasso + FT	0.952±0.003	-0.419±0.003	0.96±0.01	0.682±0.018	0.489±0.013	0.388±0.003	0.819±0.0	-0.594±0.011	0.728±0.02	-0.999±0.0	0.817±0.001	-0.998±0.0	6.33
50	IL Lasso + FT	0.963±0.001	-0.423±0.004	0.969±0.0	0.72±0.011	0.489±0.012	0.382±0.003	0.818±0.001	-0.577±0.008	0.732±0.003	-0.904±0.04	0.819±0.002	-0.791±0.003	5.16
50	AGL + FT	0.899±0.023	-0.42±0.003	0.969±0.01	0.701±0.01	0.48±0.016	0.393±0.002	0.822±0.001	-0.586±0.008	0.733±0.002	-0.915±0.006	0.814±0.002	-0.832±0.032	5.25
50	LassoNet + FT	0.963±0.001	-0.426±0.004	0.97±0.01	0.67±0.01	0.505±0.005	0.392±0.002	0.818±0.002	-0.559±0.005	0.733±0.002	-0.904±0.04	0.808±0.002	-0.791±0.001	4.83
50	AM + FT	0.962±0.001	-0.425±0.003	0.968±0.01	0.657±0.047	0.505±0.01	0.389±0.002	0.82±0.001	-0.554±0.012	0.735±0.001	-0.903±0.06	0.815±0.002	-0.789±0.001	3.75
50	RF + FT	0.963±0.001	-0.42±0.003	0.969±0.01	0.718±0.014	0.591±0.011	0.395±0.002	0.821±0.001	-0.558±0.007	0.737±0.001	-0.9±0.04	0.82±0.001	-0.791±0.001	1.75
50	XGBoost + FT	0.963±0.001	-0.42±0.002	0.969±0.01	0.725±0.012	0.572±0.012	0.392±0.001	0.82±0.001	-0.558±0.008	0.734±0.003	-0.898±0.05	0.82±0.002	-0.789±0.001	1.91
50	Deep Lasso + FT	0.962±0.001	-0.419±0.003	0.969±0.01	0.703±0.011	0.504±0.011	0.392±0.002	0.817±0.005	-0.56±0.012	0.733±0.002	-0.9±0.05	0.817±0.002	-0.788±0.002	3.66