SlowGuess/NeurIPS_2023_OCR huggingface dataset:
that in our setup the recall numbers are identical to the precision numbers, since we always "select" top-k features, where k is fixed. Therefore, the number of false positives is equal to the number of false negatives. We observe similar trends to results in the Table 1, where we evaluated algorithms with respect to the downstream performance. In particular, GBDT based approaches, univariate statistical test and Deep Lasso outperforming other considered feature selection methods.
F.3 Does feature selection help bridging the gap with GBDT models?
In Section 4, we demonstrated that neural networks are more sensitive to noise than GBDT models, particularly the MLP architecture. In this section, we analyze whether feature selection helps narrow the gap between neural networks and GBDT models in the presence of noisy features. In Tables 13, 15, 14 we present results for the baseline XGBoost models, neural networks trained without feature selection, and neural networks trained with Deep Lasso feature selection, based on our benchmark. We observe that with feature selection, MLP models perform comparably to the XGBoost models in second-order setup and significantly better in random and corrupted features setup, while transformer models consistently outperform GBDT models in terms of average rank.
Table 13: Benchmarking Neural Networks with feature selection against XGBoost models on datasets with random extra features.
%	FS method	AL	CH	CO	EY	GE	HE	HI	HO	JA	MI	OT	YE	rank
50	XGBoost Baseline	0.858	-0.416	0.951	0.665	0.613	0.360	0.728	-0.562	0.725	-0.885	0.801	-0.797	1.83
50	No FS + MLP	0.941	-0.480	0.961	0.538	0.466	0.366	0.798	-0.622	0.703	-0.911	0.773	-0.801	2.58
50	Deep Lasso + MLP	0.959	-0.443	0.968	0.573	0.485	0.383	0.814	-0.549	0.720	-0.894	0.802	-0.776	1.42
75	XGBoost Baseline	0.839	-0.42	0.941	0.626	0.583	0.349	0.726	-0.57	0.718	-0.894	0.796	-0.802	1.92
75	No FS + MLP	0.925	-0.527	0.955	0.502	0.417	0.348	0.778	-0.674	0.671	-0.917	0.749	-0.812	2.67
75	Deep Lasso + MLP	0.957	-0.446	0.969	0.569	0.479	0.387	0.814	-0.559	0.721	-0.893	0.8	-0.774	1.25
50	XGBoost Baseline	0.858	-0.416	0.951	0.665	0.613	0.360	0.728	-0.562	0.725	-0.885	0.801	-0.797	2.25
50	No FS + FT	0.959	-0.432	0.966	0.673	0.500	0.384	0.817	-0.577	0.730	-0.902	0.813	-0.792	2.08
50	Deep Lasso + FT	0.962	-0.419	0.969	0.703	0.504	0.392	0.817	-0.56	0.733	-0.900	0.817	-0.788	1.17
Table 14: Benchmarking Neural Networks with feature selection against XGBoost models on datasets with corrupted extra features.
%	FS method	AL	CH	CO	EY	GE	HE	HI	HO	JA	MI	OT	YE	rank
50	XGBoost Baseline	0.854	-0.415	0.946	0.666	0.618	0.359	0.726	-0.559	0.723	-0.891	0.796	-0.797	1.83
50	No FS + MLP	0.946	-0.475	0.965	0.557	0.525	0.370	0.802	-0.607	0.703	-0.909	0.778	-0.797	2.67
50	Deep Lasso + MLP	0.955	-0.447	0.968	0.577	0.525	0.388	0.815	-0.567	0.721	-0.895	0.801	-0.776	1.58
75	XGBoost Baseline	0.830	-0.425	0.929	0.625	0.58	0.353	0.721	-0.573	0.719	-0.894	0.795	-0.801	1.92
75	No FS + MLP	0.921	-0.516	0.956	0.518	0.503	0.356	0.788	-0.632	0.686	-0.913	0.762	-0.808	2.67
75	Deep Lasso + MLP	0.959	-0.441	0.968	0.554	0.517	0.386	0.813	-0.563	0.718	-0.898	0.804	-0.778	1.42
50	XGBoost Baseline	0.854	-0.415	0.946	0.666	0.618	0.359	0.726	-0.559	0.723	-0.891	0.796	-0.797	2.25
50	No FS + FT	0.960	-0.430	0.967	0.686	0.576	0.386	0.818	-0.574	0.731	-0.901	0.809	-0.793	2.17
50	Deep Lasso + FT	0.961	-0.422	0.968	0.725	0.577	0.393	0.821	-0.561	0.736	-0.898	0.809	-0.788	1.25
Table 15: Benchmarking Neural Networks with feature selection against XGBoost models on datasets with secondorder extra features.
%	FS method	AL	CH	CO	EY	GE	HE	HI	HO	JA	MI	OT	YE	rank
50	XGBoost Baseline	0.923	-0.429	0.969	0.709	0.683	0.374	0.729	-0.541	0.726	-0.846	0.825	-0.781	1.58
50	No FS + MLP	0.960	-0.443	0.969	0.631	0.605	0.383	0.811	-0.549	0.719	-0.891	0.800	-0.786	2.42
50	Deep Lasso + MLP	0.961	-0.441	0.969	0.648	0.600	0.384	0.815	-0.572	0.733	-0.890	0.805	-0.776	1.83
75	XGBoost Baseline	0.917	-0.414	0.968	0.74	0.691	0.376	0.731	-0.542	0.725	-0.849	0.825	-0.789	1.67
75	No FS + MLP	0.952	-0.451	0.969	0.630	0.598	0.388	0.808	-0.542	0.717	-0.900	0.792	-0.792	2.25
75	Deep Lasso + MLP	0.959	-0.448	0.969	0.647	0.582	0.378	0.821	-0.568	0.74	-0.890	0.805	-0.78	1.67
50	XGBoost Baseline	0.923	-0.429	0.969	0.709	0.683	0.374	0.729	-0.541	0.726	-0.846	0.825	-0.781	2.00
50	No FS + FT	0.962	-0.425	0.968	0.733	0.558	0.391	0.819	-0.552	0.732	-0.901	0.818	-0.790	2.25
50	Deep Lasso + FT	0.963	-0.422	0.97	0.726	0.608	0.388	0.822	-0.558	0.738	-0.897	0.819	-0.789	1.58