Table 18: Benchmarking feature selection methods for MLP and FT-Transformer downstream models on datasets with second-order extra features. We report performance of models trained on features selected by different FS algorithms in terms of accuracy for classification and negative RMSE for regression problems. % refers to percent of extra features in the dataset: either  $50\%$  or  $75\%$  features are second-order. Bold font indicates the best numbers dataset-wise and lower rank indicates better overall result.
%	FS method	AL	CH	CO	EY	GE	HE	HI	HO	JA	MI	OT	YE	rank
50	XGBoost Baseline	0.923±0.0	-0.429±0.001	0.969±0.0	0.709±0.003	0.683±0.003	0.374±0.001	0.729±0.001	-0.541±0.002	0.726±0.001	-0.846±0.0	0.825±0.001	-0.781±0.0
50	No FS +MLP	0.96±0.001	-0.443±0.002	0.969±0.001	0.631±0.009	0.605±0.012	0.383±0.002	0.811±0.0	-0.549±0.006	0.719±0.002	-0.891±0.002	0.8±0.003	-0.786±0.001	4.50
50	Univariate + MLP	0.961±0.001	-0.439±0.002	0.959±0.0	0.584±0.013	0.582±0.01	0.357±0.001	0.817±0.001	-0.614±0.009	0.724±0.002	-0.902±0.0	0.798±0.002	-0.81±0.001	6.58
50	Lasso + MLP	0.955±0.001	-0.443±0.004	0.966±0.004	0.608±0.012	0.59±0.022	0.366±0.007	0.816±0.001	-0.564±0.004	0.724±0.002	-0.891±0.0	0.806±0.001	-0.783±0.0	5.33
50	IL Lasso + MLP	0.959±0.0	-0.445±0.005	0.969±0.001	0.634±0.014	0.571±0.036	0.38±0.003	0.815±0.007	-0.565±0.01	0.728±0.001	-0.89±0.0	0.808±0.003	-0.78±0.001	3.92
50	AGL + MLP	0.961±0.001	-0.443±0.008	0.953±0.052	0.637±0.015	0.594±0.019	0.383±0.002	0.807±0.007	-0.565±0.007	0.73±0.009	-0.89±0.001	0.806±0.003	-0.776±0.001	3.25
50	LassoNet + MLP	0.959±0.001	-0.442±0.004	0.969±0.0	0.641±0.011	0.611±0.006	0.379±0.001	0.816±0.002	-0.595±0.01	0.724±0.003	-0.893±0.0	0.797±0.002	-0.784±0.002	4.50
50	AM + MLP	0.961±0.001	-0.439±0.005	0.968±0.001	0.622±0.009	0.604±0.01	0.381±0.003	0.819±0.001	-0.566±0.033	0.73±0.002	-0.892±0.002	0.802±0.003	-0.778±0.002	3.50
50	RF + MLP	0.958±0.001	-0.437±0.004	0.969±0.0	0.639±0.008	0.619±0.007	0.37±0.002	0.818±0.001	-0.586±0.002	0.735±0.002	-0.89±0.0	0.801±0.002	-0.781±0.001	3.25
50	XGBoost + MLP	0.87±0.001	-0.438±0.002	0.97±0.001	0.635±0.009	0.604±0.007	0.373±0.004	0.818±0.001	-0.579±0.01	0.734±0.002	-0.891±0.001	0.805±0.002	-0.786±0.001	3.83
50	Deep Lasso + MLP	0.961±0.001	-0.441±0.007	0.969±0.001	0.648±0.009	0.6±0.01	0.384±0.001	0.815±0.001	-0.572±0.017	0.733±0.001	-0.89±0.001	0.805±0.003	-0.776±0.001	2.67
75	XGBoost Baseline	0.917±0.0	-0.414±0.003	0.968±0.0	0.74±0.002	0.691±0.003	0.376±0.001	0.731±0.001	-0.542±0.002	0.725±0.001	-0.849±0.0	0.825±0.001	-0.789±0.0
75	No FS + MLP	0.952±0.001	-0.451±0.003	0.969±0.0	0.63±0.006	0.598±0.006	0.388±0.001	0.808±0.001	-0.542±0.006	0.717±0.002	-0.9±0.001	0.792±0.003	-0.792±0.002	4.92
75	Univariate + MLP	0.96±0.001	-0.53±0.004	0.488±0.0	0.553±0.008	0.531±0.005	0.352±0.002	0.812±0.001	-0.608±0.004	0.72±0.002	-0.908±0.0	0.785±0.002	-0.82±0.001	7.25
75	Lasso + MLP	0.96±0.001	-0.434±0.002	0.968±0.001	0.612±0.023	0.519±0.039	0.363±0.005	0.82±0.001	-0.554±0.005	0.739±0.002	-0.894±0.001	0.807±0.001	-0.793±0.001	3.67
75	IL Lasso + MLP	0.96±0.001	-0.452±0.044	0.966±0.006	0.654±0.01	0.579±0.022	0.375±0.004	0.818±0.001	-0.549±0.007	0.741±0.001	-0.893±0.002	0.805±0.002	-0.782±0.001	3.33
75	AGL + MLP	0.958±0.001	-0.438±0.005	0.968±0.001	0.647±0.019	0.601±0.007	0.384±0.002	0.819±0.001	-0.545±0.007	0.736±0.002	-0.893±0.0	0.8±0.003	-0.781±0.001	2.67
75	LassoNet + MLP	0.958±0.001	-0.454±0.027	0.968±0.001	0.633±0.015	0.615±0.012	0.362±0.002	0.813±0.005	-0.569±0.011	0.726±0.002	-0.895±0.0	0.793±0.003	-0.786±0.002	4.83
75	RF + MLP	0.956±0.001	-0.445±0.005	0.968±0.001	0.627±0.012	0.566±0.006	0.339±0.002	0.819±0.0	-0.615±0.011	0.728±0.002	-0.892±0.0	0.794±0.002	-0.789±0.001	5.08
75	XGBoost + MLP	0.459±0.001	-0.495±0.054	0.968±0.0	0.627±0.009	0.555±0.012	0.358±0.007	0.821±0.001	-0.588±0.018	0.738±0.002	-0.892±0.001	0.803±0.003	-0.795±0.001	5.00
75	Deep Lasso + MLP	0.959±0.001	-0.448±0.007	0.969±0.001	0.647±0.018	0.582±0.015	0.378±0.001	0.821±0.001	-0.568±0.02	0.74±0.001	-0.89±0.0	0.805±0.002	-0.78±0.001	2.17
50	No FS + FT	0.962±0.001	-0.425±0.003	0.968±0.001	0.733±0.011	0.558±0.011	0.391±0.002	0.819±0.001	-0.552±0.008	0.732±0.002	-0.91±0.004	0.818±0.002	-0.79±0.001	4.17
50	Univariate + FT	0.963±0.001	-0.419±0.003	0.942±0.002	0.615±0.009	0.573±0.011	0.351±0.002	0.82±0.0	-0.613±0.005	0.728±0.001	-0.91±0.003	0.81±0.001	-0.819±0.001	6.67
50	Lasso + FT	0.962±0.002	-0.422±0.003	0.967±0.004	0.72±0.011	0.56±0.015	0.381±0.003	0.823±0.0	-0.561±0.011	0.733±0.002	-0.905±0.02	0.816±0.002	-0.874±0.001	5.25
50	IL Lasso + FT	0.964±0.001	-0.421±0.004	0.97±0.001	0.748±0.008	0.604±0.019	0.391±0.002	0.812±0.011	-0.557±0.008	0.736±0.002	-0.896±0.02	0.817±0.002	-0.795±0.004	2.83
50	AGL + FT	0.935±0.024	-0.421±0.008	0.95±0.063	0.747±0.008	0.54±0.012	0.388±0.002	0.821±0.0	-0.557±0.008	0.73±0.008	-0.907±0.04	0.821±0.002	-0.821±0.014	4.17
50	LassoNet + FT	0.963±0.001	-0.423±0.006	0.97±0.001	0.729±0.012	0.612±0.011	0.378±0.004	0.817±0.003	-0.592±0.005	0.733±0.001	-0.91±0.04	0.808±0.002	-0.792±0.001	5.00
50	AM + FT	0.963±0.001	-0.418±0.002	0.97±0.001	0.719±0.009	0.597±0.019	0.387±0.003	0.82±0.01	-0.579±0.024	0.736±0.002	-0.902±0.05	0.819±0.002	-0.791±0.02	3.83
50	RF + FT	0.963±0.001	-0.422±0.006	0.97±0.001	0.733±0.007	0.615±0.008	0.365±0.008	0.822±0.001	-0.62±0.013	0.738±0.001	-0.895±0.04	0.816±0.002	-0.793±0.02	3.67
50	XGBoost + FT	0.871±0.001	-0.424±0.003	0.962±0.001	0.737±0.014	0.594±0.013	0.379±0.002	0.822±0.01	-0.587±0.014	0.736±0.001	-0.896±0.02	0.812±0.003	-0.792±0.001	4.33
50	Deep Lasso + FT	0.963±0.001	-0.422±0.008	0.97±0.001	0.726±0.011	0.608±0.012	0.388±0.002	0.822±0.01	-0.558±0.005	0.738±0.001	-0.897±0.03	0.819±0.001	-0.789±0.001	2.42