Table 3: Benchmarking feature selection methods for MLP and FT-Transformer downstream models on datasets with second-order extra features. We report performance of models trained on features selected by different FS algorithms in terms of accuracy for classification and negative RMSE for regression problems. % refers to percent of extra features in the dataset: either  $50\%$  or  $75\%$  features are second-order. Bold font indicates the best numbers dataset-wise and lower rank indicates better overall result.
%	FS method	AL	CH	CO	EY	GE	HE	HI	HO	JA	MI	OT	YE	rank
50	No FS + MLP	0.960	-0.443	0.969	0.631	0.605	0.383	0.811	-0.549	0.719	-0.891	0.800	-0.786	4.50
50	Univariate + MLP	0.961	-0.439	0.959	0.584	0.582	0.357	0.817	-0.614	0.724	-0.902	0.798	-0.810	6.58
50	Lasso + MLP	0.955	-0.443	0.966	0.608	0.590	0.366	0.816	-0.564	0.724	-0.891	0.806	-0.783	5.33
50	1L Lasso + MLP	0.959	-0.445	0.969	0.634	0.571	0.380	0.815	-0.565	0.728	-0.89	0.808	-0.780	3.92
50	AGL + MLP	0.961	-0.443	0.953	0.637	0.594	0.383	0.807	-0.565	0.730	-0.89	0.806	-0.776	3.25
50	LassoNet + MLP	0.959	-0.442	0.969	0.641	0.611	0.379	0.816	-0.595	0.724	-0.893	0.797	-0.784	4.50
50	AM + MLP	0.961	-0.439	0.968	0.622	0.604	0.381	0.819	-0.566	0.730	-0.892	0.802	-0.778	3.50
50	RF + MLP	0.958	-0.437	0.969	0.639	0.619	0.370	0.818	-0.586	0.735	-0.89	0.801	-0.781	3.25
50	XGBoost + MLP	0.870	-0.438	0.97	0.635	0.604	0.373	0.818	-0.579	0.734	-0.891	0.805	-0.786	3.83
50	Deep Lasso + MLP	0.961	-0.441	0.969	0.648	0.600	0.384	0.815	-0.572	0.733	-0.89	0.805	-0.776	2.67
75	No FS + MLP	0.952	-0.451	0.969	0.630	0.598	0.388	0.808	-0.542	0.717	-0.900	0.792	-0.792	4.92
75	Univariate + MLP	0.960	-0.530	0.488	0.553	0.531	0.352	0.812	-0.608	0.720	-0.908	0.785	-0.820	7.25
75	Lasso + MLP	0.96	-0.434	0.968	0.612	0.519	0.363	0.820	-0.554	0.739	-0.894	0.807	-0.793	3.67
75	1L Lasso + MLP	0.96	-0.452	0.966	0.654	0.579	0.375	0.818	-0.549	0.741	-0.893	0.805	-0.782	3.33
75	AGL + MLP	0.958	-0.438	0.968	0.647	0.601	0.384	0.819	-0.545	0.736	-0.893	0.800	-0.781	2.67
75	LassoNet + MLP	0.958	-0.454	0.968	0.633	0.615	0.362	0.813	-0.569	0.726	-0.895	0.793	-0.786	4.83
75	RF + MLP	0.956	-0.445	0.968	0.627	0.566	0.339	0.819	-0.615	0.728	-0.892	0.794	-0.789	5.08
75	XGBoost + MLP	0.459	-0.495	0.968	0.627	0.555	0.358	0.821	-0.588	0.738	-0.892	0.803	-0.795	5.00
75	Deep Lasso + MLP	0.959	-0.448	0.969	0.647	0.582	0.378	0.821	-0.568	0.74	-0.89	0.805	-0.78	2.17
50	No FS + FT	0.962	-0.425	0.968	0.733	0.558	0.391	0.819	-0.552	0.732	-0.901	0.818	-0.790	4.08
50	Univariate + FT	0.963	-0.419	0.942	0.615	0.573	0.351	0.820	-0.613	0.728	-0.910	0.810	-0.819	6.25
50	Lasso + FT	0.962	-0.422	0.967	0.720	0.560	0.381	0.823	-0.561	0.733	-0.905	0.816	-0.874	5.08
50	1L Lasso + FT	0.964	-0.421	0.970	0.748	0.604	0.391	0.812	-0.557	0.736	-0.896	0.817	-0.795	2.83
50	AGL + FT	0.935	-0.421	0.950	0.747	0.540	0.388	0.821	-0.557	0.730	-0.907	0.821	-0.821	4.16
50	LassoNet + FT	0.963	-0.423	0.970	0.729	0.612	0.378	0.817	-0.592	0.733	-0.901	0.808	-0.792	5.00
50	AM + FT	0.963	-0.418	0.970	0.719	0.597	0.387	0.820	-0.579	0.736	-0.902	0.819	-0.791	3.83
50	RF + FT	0.963	-0.422	0.970	0.733	0.615	0.365	0.822	-0.600	0.738	-0.895	0.816	-0.793	3.66
50	XGBoost + FT	0.871	-0.424	0.962	0.737	0.594	0.379	0.822	-0.587	0.736	-0.896	0.812	-0.792	4.08
50	Deep Lasso + FT	0.963	-0.422	0.970	0.726	0.608	0.388	0.822	-0.558	0.738	-0.897	0.819	-0.789	2.41