Table 17: Benchmarking feature selection methods for MLP and FT-Transformer downstream models on datasets with second-order extra features. We report performance of models trained on features selected by different FS algorithms in terms of accuracy for classification and negative RMSE for regression problems. % refers to percent of extra features in the dataset: either  $50\%$  or  $75\%$  features are second-order. Bold font indicates the best numbers dataset-wise and lower rank indicates better overall result.
%	FS method	AL	CH	CO	EY	GE	HE	HI	HO	JA	MI	OT	YE	rank
50	XGBoost Baseline	0.854±0.0	-0.415±0.001	0.946±0.0	0.666±0.003	0.618±0.003	0.359±0.001	0.726±0.001	-0.559±0.002	0.723±0.001	-0.891±0.001	0.796±0.001	-0.797±0.0
50	No FS + MLP	0.946±0.001	-0.475±0.004	0.965±0.001	0.557±0.007	0.525±0.003	0.37±0.001	0.802±0.002	-0.607±0.005	0.703±0.003	-0.909±0.001	0.778±0.004	-0.797±0.001	8.00
50	Univariate + MLP	0.955±0.001	-0.451±0.004	0.966±0.001	0.556±0.007	0.514±0.005	0.346±0.003	0.81±0.001	-0.62±0.004	0.717±0.002	-0.92±0.0	0.795±0.001	-0.828±0.001	7.33
50	Lasso + MLP	0.955±0.001	-0.449±0.003	0.968±0.001	0.548±0.017	0.512±0.011	0.382±0.002	0.813±0.001	-0.602±0.002	0.713±0.003	-0.903±0.0	0.796±0.001	-0.795±0.001	5.42
50	IL Lasso + MLP	0.955±0.002	-0.447±0.002	0.968±0.001	0.566±0.01	0.515±0.018	0.382±0.002	0.812±0.001	-0.581±0.006	0.718±0.003	-0.902±0.003	0.795±0.002	-0.78±0.004	4.75
50	AGL + MLP	0.953±0.001	-0.45±0.005	0.968±0.001	0.588±0.015	0.538±0.009	0.386±0.001	0.813±0.001	-0.561±0.006	0.722±0.002	-0.902±0.001	0.796±0.002	-0.78±0.001	3.00
50	LassoNet + MLP	0.955±0.001	-0.452±0.005	0.969±0.001	0.57±0.016	0.556±0.009	0.382±0.002	0.811±0.002	-0.551±0.007	0.719±0.002	-0.905±0.001	0.795±0.004	-0.777±0.001	3.83
50	AM + MLP	0.955±0.001	-0.449±0.003	0.967±0.001	0.583±0.012	0.527±0.009	0.381±0.002	0.814±0.001	-0.555±0.012	0.722±0.001	-0.905±0.004	0.797±0.003	-0.78±0.002	3.58
50	RF + MLP	0.951±0.001	-0.453±0.003	0.967±0.001	0.574±0.007	0.568±0.01	0.383±0.001	0.81±0.002	-0.565±0.004	0.724±0.002	-0.904±0.0	0.788±0.002	-0.786±0.001	4.67
50	XGBoost + MLP	0.954±0.001	-0.454±0.003	0.969±0.0	0.583±0.011	0.51±0.009	0.385±0.003	0.815±0.002	-0.553±0.006	0.722±0.002	-0.892±0.001	0.803±0.002	-0.779±0.001	2.67
50	Deep Lasso + MLP	0.955±0.001	-0.447±0.003	0.968±0.0	0.577±0.011	0.525±0.015	0.388±0.002	0.815±0.001	-0.567±0.005	0.721±0.002	-0.895±0.001	0.801±0.002	-0.776±0.001	2.58
75	XGBoost Baseline	0.83±0.0	-0.425±0.001	0.929±0.0	0.625±0.003	0.58±0.002	0.353±0.001	0.721±0.002	-0.573±0.001	0.719±0.001	-0.894±0.001	0.795±0.001	-0.801±0.0
75	No FS + MLP	0.921±0.001	-0.516±0.004	0.956±0.001	0.518±0.01	0.503±0.005	0.356±0.002	0.788±0.001	-0.632±0.005	0.686±0.002	-0.913±0.001	0.762±0.003	-0.808±0.001	7.58
75	Univariate + MLP	0.955±0.0	-0.569±0.004	0.941±0.001	0.51±0.008	0.495±0.005	0.347±0.002	0.742±0.001	-0.62±0.004	0.686±0.003	-0.921±0.0	0.779±0.002	-0.838±0.001	7.50
75	Lasso + MLP	0.948±0.001	-0.454±0.005	0.963±0.001	0.565±0.01	0.49±0.021	0.373±0.003	0.81±0.001	-0.593±0.005	0.717±0.002	-0.903±0.001	0.795±0.002	-0.791±0.0
75	IL Lasso + MLP	0.955±0.001	-0.444±0.002	0.967±0.0	0.549±0.026	0.495±0.021	0.38±0.002	0.811±0.001	-0.576±0.007	0.715±0.008	-0.903±0.003	0.797±0.004	-0.779±0.001	3.25
75	AGL + MLP	0.928±0.015	-0.566±0.199	0.967±0.001	0.548±0.011	0.49±0.019	0.382±0.003	0.811±0.001	-0.574±0.005	0.714±0.002	-0.904±0.0	0.788±0.002	-0.78±0.001	4.92
75	LassoNet + MLP	0.947±0.001	-0.452±0.006	0.969±0.001	0.539±0.008	0.533±0.009	0.383±0.002	0.805±0.005	-0.572±0.012	0.708±0.004	-0.908±0.001	0.791±0.003	-0.785±0.002	4.33
75	RF + MLP	0.952±0.001	-0.45±0.003	0.963±0.001	0.547±0.011	0.533±0.007	0.372±0.003	0.805±0.002	-0.573±0.005	0.716±0.002	-0.903±0.0	0.765±0.003	-0.788±0.001	4.92
75	XGBoost + MLP	0.954±0.001	-0.515±0.048	0.968±0.001	0.571±0.009	0.53±0.01	0.381±0.002	0.811±0.001	-0.571±0.011	0.721±0.002	-0.895±0.001	0.8±0.003	-0.784±0.002	2.58
75	Deep Lasso + MLP	0.959±0.001	-0.441±0.002	0.968±0.0	0.554±0.015	0.517±0.006	0.386±0.002	0.813±0.002	-0.563±0.009	0.718±0.003	-0.898±0.001	0.804±0.003	-0.778±0.001	1.42
50	No FS + FT	0.96±0.001	-0.43±0.004	0.967±0.001	0.686±0.008	0.576±0.022	0.386±0.002	0.818±0.001	-0.574±0.007	0.731±0.002	-0.901±0.003	0.809±0.002	-0.793±0.001	6.08
50	Univariate + FT	0.963±0.001	-0.422±0.003	0.965±0.001	0.681±0.011	0.574±0.013	0.345±0.002	0.812±0.002	-0.628±0.007	0.733±0.001	-0.92±0.022	0.812±0.022	-0.826±0.0	6.17
50	Lasso + FT	0.952±0.004	-0.422±0.003	0.936±0.037	0.697±0.022	0.556±0.012	0.387±0.002	0.82±0.0	-0.586±0.006	0.732±0.003	-0.937±0.0	0.812±0.022	-0.915±0.001	6.42
50	IL Lasso + FT	0.962±0.001	-0.419±0.002	0.969±0.001	0.718±0.008	0.571±0.009	0.389±0.002	0.82±0.01	-0.57±0.004	0.731±0.001	-0.899±0.003	0.816±0.003	-0.795±0.004	3.50
50	AGL + FT	0.906±0.021	-0.426±0.003	0.969±0.001	0.697±0.018	0.591±0.009	0.392±0.002	0.82±0.01	-0.552±0.005	0.735±0.002	-0.914±0.005	0.816±0.003	-0.83±0.016	4.08
50	LassoNet + FT	0.962±0.001	-0.426±0.003	0.97±0.001	0.679±0.012	0.578±0.014	0.393±0.002	0.814±0.005	-0.572±0.011	0.736±0.002	-0.903±0.04	0.813±0.022	-0.79±0.01	3.92
50	AM + FT	0.962±0.001	-0.424±0.002	0.969±0.0	0.68±0.017	0.572±0.018	0.392±0.001	0.82±0.01	-0.549±0.007	0.734±0.001	-0.901±0.04	0.817±0.022	-0.79±0.02	3.42
50	RF + FT	0.962±0.001	-0.422±0.004	0.969±0.01	0.679±0.016	0.6±0.009	0.387±0.002	0.819±0.002	-0.557±0.005	0.735±0.002	-0.898±0.02	0.816±0.022	-0.79±0.02	3.42
50	XGBoost + FT	0.963±0.001	-0.422±0.002	0.97±0.0	0.706±0.011	0.564±0.009	0.392±0.001	0.821±0.001	-0.548±0.003	0.735±0.001	-0.897±0.03	0.816±0.001	-0.79±0.01	2.42
50	Deep Lasso + FT	0.961±0.001	-0.422±0.004	0.968±0.01	0.725±0.008	0.577±0.014	0.393±0.002	0.821±0.001	-0.561±0.013	0.736±0.002	-0.898±0.04	0.816±0.023	-0.788±0.01	2.67