SlowGuess/NeurIPS_2023_OCR huggingface dataset:
the continuous feature selection tensor into an accurate binary selection, we adopt the straight-through estimator(STE) function [1] for continuous sparsification. The STE can be formulated as a customized function  $S(\cdot)$ , with its forward pass as a unit step function
$$
\mathcal {S} (x) = \left\{ \begin{array}{l l} 0, & x \leq 0 \\ 1, & x > 0 \end{array} , \right. \tag {10}
$$
and backward pass as  $\frac{d}{dx} S(x) = 1$ , meaning that it will directly pass the gradient backward. Therefore, we can mimic a discrete feature interaction selection while providing valid gradient information for the value-level selection parameters  $\mathbf{E}_v \& \theta_v$ , making the whole process trainable. Hence, Equation 9 is re-written as:
$$
\mathbf {A} _ {v} \left[ k _ {i}, k _ {j} \right] = S \left(f _ {\theta_ {v}} \left(\mathbf {E} _ {v} \left[ k _ {i}, : \right]\right) \times \boldsymbol {\Sigma} _ {v} \times f _ {\theta_ {v}} ^ {T} \left(\mathbf {E} _ {v} \left[ k _ {j}, : \right]\right)\right). \tag {11}
$$
Similar to Equation 11, we can obtain the field-level selection matrix  $\mathbf{A}_f$  using the following equation:
$$
\mathbf {A} _ {f} [ i, j ] = S \left(g _ {\theta_ {f}} \left(\mathbf {E} _ {f} [ i, : ]\right) \times \boldsymbol {\Sigma} _ {f} \times g _ {\theta_ {f}} ^ {T} \left(\mathbf {E} _ {f} [ j,: ]\right)\right), \tag {12}
$$
where  $\mathbf{E}_f\in \mathbb{R}^{n\times \hat{d}}$  is the field-level embedding table for feature interaction selection and  $g_{\theta_f}$  ..  $\mathbb{R}^{n\times \hat{d}}\to \mathbb{R}^{n\times d'}$  is another neural network parameterized by  $\theta_{f}$
After obtaining the value-level selection matrix  $\mathbf{A}_v$  and field-level selection matrix  $\mathbf{A}_f$ , we need to merge them and obtain the hybrid selection result  $\mathbf{A}$ . Inspired by DARTS [12] and its success in previous works [11, 15, 13], we relax the hybrid tensor  $\alpha$  into a continuous tensor  $\alpha_c \in \mathbb{R}^{n^t}$ , which can be trained jointly with other parameters via gradient descent. To ensure convergence, we apply the sigmoid function  $\sigma(\cdot)$  over  $\alpha_c$ . Hence, during training time, Equation 6 can be re-written as:
$$
\mathbf {A} = \sigma \left(\alpha_ {c}\right) \cdot \mathbf {A} _ {f} + \left(1 - \sigma \left(\alpha_ {c}\right)\right) \cdot \mathbf {A} _ {v}. \tag {13}
$$
Hence, the final search objective can be formulated as:
$$
\hat {\mathbf {W}} ^ {*} = \min  _ {\hat {\mathbf {W}}} \mathcal {L} \left(\mathcal {D} _ {\mathrm {t r a}} \mid \left\{\mathbf {W} ^ {*}, \hat {\mathbf {W}} \right\}\right), \text {s . t .} \mathbf {W} ^ {*} = \min  _ {\mathbf {W}} \mathcal {L} \left(\mathcal {D} _ {\mathrm {v a l}} \mid \left\{\mathbf {W}, \hat {\mathbf {W}} ^ {*} \right\}\right). \tag {14}
$$
Here  $\hat{\mathbf{W}} = \{\mathbf{E}_v,\theta_v,\mathbf{E}_f,\theta_f,\pmb {\alpha}\}$  denoting the parameter required in the interaction selection process, and  $\mathcal{D}_{\mathrm{val}}$  denotes the validation set. We summarize the overall process of our sparsification-based selection algorithm in Algorithm 1.
Algorithm 1 Sparsification-based Selection Algorithm
Require: training and validation set  $\mathcal{D}_{\mathrm{tra}}$  and  $\mathcal{D}_{\mathrm{val}}$    
Ensure: main model parameters W and feature interaction selection parameters  $\hat{\mathbf{W}}$  1: for epoch  $= 1,\dots ,T$  do 2: while epoch not finished do 3: Sample mini-batch  $\mathcal{B}_{\mathrm{val}}$  and  $\mathcal{B}_{\mathrm{tra}}$  from validation set  $\mathcal{D}_{\mathrm{tra}}$  and training set  $\mathcal{D}_{\mathrm{val}}$  4: Update selection parameter  $\hat{\mathbf{W}}$  using gradients  $\nabla_{\hat{\mathbf{W}}}\mathcal{L}(\mathcal{B}_{\mathrm{val}}|\{\mathbf{W},\hat{\mathbf{W}}\})$  5: Update model parameter W using gradients  $\nabla_{\mathbf{W}}\mathcal{L}(\mathcal{B}_{\mathrm{tra}}|\{\mathbf{W},\hat{\mathbf{W}}\})$  6: end while 7: end for
3.3.3 Retraining with Optimal Selection Result
During the re-training stage, the optimal hybrid tensor is determined as  $\alpha^{*} = \mathbb{1}_{\alpha_{c} > 0}$  following previous works [12, 15, 13]. Hence, we can obtain the optimal selection result
$$
\mathbf {A} ^ {*} \approx \alpha^ {*} \cdot S \left(g _ {\theta_ {f} ^ {*}} \left(\mathbf {E} _ {f} ^ {*}\right) \times \boldsymbol {\Sigma} _ {f} ^ {*} \times g _ {\theta_ {f} ^ {*}} ^ {T} \left(\mathbf {E} _ {f} ^ {*}\right)\right) + (1 - \alpha^ {*}) \cdot S \left(f _ {\theta_ {v} ^ {*}} \left(\mathbf {E} _ {v} ^ {*}\right) \times \boldsymbol {\Sigma} _ {v} ^ {*} \times f _ {\theta_ {v} ^ {*}} ^ {T} \left(\mathbf {E} _ {v} ^ {*}\right)\right). \tag {15}
$$
We freeze the selection parameters  $\hat{\mathbf{W}}$  and retrain the model from scratch following the existing works [12, 7]. This reason for introducing the retraining stage is to remove the influence of suboptimal selection results during the selection process over the model parameter. Also,
Datasets To validate the effectiveness of our proposed method OptFeature, we conduct experiments on three benchmark datasets (Criteo, Avazu $^3$ , and KDD12 $^4$ ), which are widely used in previous work on deep sparse networks for evaluation purposes [7, 11]. The dataset statistics and preprocessing details are described in the Appendix B.
<sup>3</sup>http://www.kaggle.com/c/avazu-ctr-prediction
<sup>4</sup>http://www.kddcup2012.org/c/kddcup2012-track2/data
Baseline Models We compare our OptFeature with previous works in deep sparse networks. We choose the methods that are widely used and open-sourced:
- Shallow network: Logistic Regression(LR) [22] and Factorization Machine(FM) [21].
- DSNs: FNN [27], DeepFM [8], DCN [24], IPNN [18].
- DSNs with feature interaction selection(DSNs with FIS): AutoFIS $^5$  [11] and PROFIT $^6$  [7].
For the shallow networks and deep sparse networks, we implement them following the commonly used library torchfm<sup>7</sup>. These baselines and our OptFeature are available here<sup>8</sup>. For AutoFIS and PROFIT, we re-use their original implementations, respectively.
Evaluation Metrics Following the previous works[8, 30], we adopt the commonly-used performance evaluation metrics for click-through rate prediction, AUC (Area Under ROC curve) and Log loss (also known as cross-entropy). As for efficiency evaluation, we also use two standard metrics, inference time and model size, which are widely adopted in previous works of deep sparse network [7, 11].
Parameter Setup To ensure the reproducibility of experimental results, here we further introduce the implementation setting in detail. We implement our methods using PyTorch. We adopt the Adam optimizer with a mini-batch size of 4096. We set the embedding sizes to 16 in all the models. We set the predictor as an MLP model with [1024, 512, 256] for all methods. All the hyper-parameters are tuned on the validation set with a learning rate from [1e-3, 3e-4, 1e-4, 3e-5, 1e-5] and weight decay from [1e-4, 3e-5, 1e-5, 3e-6, 1e-6]. We also tune the learning ratio for the feature interaction selection parameters from [1e-4, 3e-5, 1e-5, 3e-6, 1e-6] and while weight decay from [1e-4, 3e-5, 1e-5, 3e-6, 1e-6, 0]. The initialization parameters for the retraining stage are selected from the best-performed model parameters and randomly initialized ones.
Model Stability To make the results more reliable, we ran the repetitive experiments with different random seeds five times and reported the average value for each result.
Hardware Platform All experiments are conducted on a Linux server with one Nvidia-Tesla V100-PCIe-32GB GPU, 128GB main memory and 8 Intel(R) Xeon(R) Gold 6140 CPU cores.
4.2 Benchmark Comparison
Model Performance We present the model performance across three benchmark datasets in Table 1 and make the following observations. Firstly, OptFeature consistently outperforms other models on all three benchmark datasets, validating the effectiveness of our hybrid-grained selection space and selection algorithm. Secondly, DSNs generally yield better performances than shallow models, underscoring the importance of designing powerful sparse networks. Thirdly, DSNs with feature interaction selection tend to perform superiorly compared to those incorporating all possible interactions. This confirms the significance of conducting feature interaction selection within DSNs. Lastly, methods that decompose the selection space (e.g., OptFeature and PROFIT) consistently outperform AutoFIS, which directly navigates the original selection space.
Model Efficiency We also evaluate model efficiency and present the results in Figure 3. We can observe that OptFeature outperforms all other methods in terms of inference time, while its model size is comparable to other feature interaction selection methods. Both model accuracy and inference efficiency are critical factors when deploying DSNs. Our OptFeature not only achieves the best performance but also reduces inference time, indicating its practical value.
$^{5}$ https://github.com/zhuchenxv/AutoFIS
$^{6}$ https://github.com/NeurIPS-AutoDSN/NeurIPS-AutoDSN
<sup>7</sup>https://github.com/rixwew/pytorch-fm
$^{8}$ https://github.com/fuyuanlyu/OptFeature
Table 1: Overall Performance Comparison
Dataset	Criteo	Avazu	KDD12
Category	Model	AUC	Logloss	AUC	Logloss	AUC	Logloss
Shallow	LR	0.7882	0.4609	0.7563	0.3928	0.7411	0.1637
FM	0.8047	0.4464	0.7839	0.3783	0.7786	0.1566
DSNs	FNN	0.8101	0.4414	0.7891	0.3762	0.7947	0.1536
DeepFM	0.8097	0.4418	0.7896	0.3757	0.7969	0.1531
DCN	0.8096	0.4419	0.7887	0.3767	0.7955	0.1534
IPNN	0.8103	0.4413	0.7896	0.3741	0.7945	0.1537
DSNs with FIS	AutoFIS	0.8089	0.4428	0.7903	0.3749	0.7959	0.1533
PROFIT	0.8112	0.4406	0.7906	0.3756	0.7964	0.1533
OptFeature	0.8116	0.4402	0.7925*	0.3741*	0.7982*	0.1529*
Here * denotes statistically significant improvement (measured by a two-sided t-test with p-value  $< 0.05$ ) over the best baseline. The best and second best performed results are marked in bold and underline format
Figure 3: Efficiency comparison between OptFeature and DSN baselines. Model size excludes the embedding layer. Inference time refers to the average inference time per batch across validation set.
4.3 Investigating the Selection Process
In this section, we delve into the selection process of OptFeature. We introduce two variants of OptFeature for this purpose: (i) OptFeature-f, which carries out field-level interaction selection, and (ii) OptFeature-v, which conducts value-level interaction selection. Both methods adopt the same selection algorithm outlined in Section 3.3.2. We will compare OptFeature and its two variants to other DSNs interaction selection baselines regarding effectiveness and efficiency.
Table 2: Performance Comparison over Different Granularity.
Dataset	Criteo	Avazu	KDD12
Model	Granularity	AUC	Logloss	AUC	Logloss	AUC	Logloss
AutoFIS	Field	0.8089	0.4428	0.7903	0.3749	0.7959	0.1533
PROFIT	Field	0.8112	0.4406	0.7906	0.3756	0.7964	0.1533
OptFeature-f	Field	0.8115	0.4404	0.7920	0.3744	0.7978	0.1530
OptFeature-v	Value	0.8116	0.4403	0.7920	0.3742	0.7981	0.1529
OptFeature	Hybrid	0.8116	0.4402	0.7925	0.3741	0.7982	0.1529
Investigating the Selection Effectiveness We evaluate the effectiveness of various DSN feature interaction methods in terms of comparable performance. Several observations can be made from the results presented in Table 2. Firstly, OptFeature consistently delivers superior results on all three datasets. This confirms that the hybrid-grained selection space allows OptFeature to explore finer-grained while the selection algorithm effectively performs based on the decomposition. Secondly, the value-level selection method consistently surpasses field-level selection, suggesting that field-level selection might be too coarse-grained to leverage informative values within uninformative fields. Lastly, OptFeature-f outperforms both AutoFIS and PROFIT in terms of model performance.
The distinguishing factor among these three methods lies in the selection algorithm. AutoFIS directly optimizes the gating vector for each interaction field, rendering the space too large for efficient exploration. PROFIT, on the other hand, adopts a progressive selection algorithm, leading to sub-optimal interaction selection. This finding further emphasizes the superiority of our sparsification-based selection algorithm in selecting informative interactions.
Investigating the Selection Efficiency Next, we perform a comparison study on the efficiency of the selection methods. In Figure 4, we report the search cost, measured in GPU hours. The search cost reflects the GPU resource consumption when selecting interactions from scratch. It's important to note that the cost of re-training is excluded as our focus is on the selection process. We observe that the search cost of OptFeature is lower than other baselines on the Criteo and KDD12 datasets. OptFeature surpasses the other two variants as it typically converges in fewer epochs. These comparisons underscore the feasibility of our proposed method. However, on the Avazu dataset, OptFeature is slightly outperformed by its two variants, i.e., OptFeature-f and OptFeature-v, because this dataset usually converges in one epoch [28]. As a result,
all three methods converge within the same epoch. Furthermore, compared to AutoFIS and PROFIT, OptFeature-f incurs a lower cost with the same complexity of selection space. This observation further highlights the superiority of our selection algorithm in terms of convergence speed.
Figure 4: Comparisons between OptFeature and other DSNs interaction selection baselines on the search cost.
In this work, we introduce a hybrid-grained selection approach targeting both feature field and feature value level. To explore the vast search space that extends from field to value, we propose a decomposition method to reduce space complexity and make it trainable. We then developed a selection algorithm named OptFeature that simultaneously selects interactions from field and value perspectives. Experiments conducted on three real-world benchmark datasets validate the effectiveness and efficiency of our method. Further ablation studies regarding search efficiency and selection granularity illustrate the superiority of our proposed OptFeature.
Limitations Despite OptFeature demonstrating superior performance over other baselines on model performance and inference time, it requires a larger model size than certain DSNs [8, 27]. Additionally, it lacks online experiments to validate its effectiveness in more complex and dynamic scenarios.
[1] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013.
[2] Alexander M. Chebotarev and Alexander Evgen'evich Teretenkov. Singular value decomposition for the takagi factorization of symmetric matrices. Appl. Math. Comput., 234:380-384, 2014.
[3] Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. Learning to explain: An information-theoretic perspective on model interpretation. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, volume 80 of Proceedings of Machine Learning Research, pages 882-891, Stockholm, Sweden, 2018. PMLR.
[4] James Enouen and Yan Liu. Sparse interaction additive networks via feature interaction detection and sparse selection. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 13908-13920. Curran Associates, Inc., 2022.
A General Form of Selection Tensor Decomposition
In this section, we further extend the selection tensor decomposition in Section 3.3.1 from a special case, where  $t = 2$ , to a more general case, where  $2 \leq t \leq n$ . The interaction selection tensor  $\mathbf{A}_v^t$  for  $t$ -th order features is also semi-positive and symmetric. By extending the Takagi factorization [2], we have:
$$
\mathbf {A} _ {v} ^ {t} = \boldsymbol {\Sigma} _ {v} \times_ {1} \mathcal {U} \times_ {2} \mathcal {U} \times_ {3} \dots \times_ {t} \mathcal {U}, \tag {16}
$$
where  $\Sigma_v$  is a  $\underbrace{d' \times \cdots \times d'}_{\texttt{t times}}$  diagonal tensor,  $\times_t$  denotes the  $t$ -mode matrix multiplication [5],
$\mathcal{U} \in R^{m \times d'}$  and  $d' < m$ . Similar to the special case where  $t = 2$ , we adopt multi-mode tensor factorization [5] to replace  $\mathcal{U}$  as an output of a neural network, denoted as:
$$
\mathcal {U} \approx f _ {\hat {\theta}} \left(\mathbf {E} _ {v}\right), \tag {17}
$$
where  $f_{\hat{\theta}}: \mathbb{R}^{m \times \hat{d}} \to \mathbb{R}^{m \times d'}$ ,  $\hat{d} \ll d'$  is a neural network with parameter  $\hat{\theta}$  and  $\hat{\mathbf{E}} \in \mathbb{R}^{m \times \hat{d}}$  is an additional embedding table for generating feature interaction selection tensor. The element of architecture metric  $\mathbf{A}_v^t[k_{i_1}, \dots, k_{i_t}]$  can be calculated given the following equation:
$$
\mathbf {A} _ {v} ^ {t} \left[ k _ {i _ {1}}, \dots , k _ {i _ {t}} \right] = \boldsymbol {\Sigma} _ {v} \times_ {1} f _ {\theta_ {v}} \left(\mathbf {E} _ {v} \left[ k _ {i _ {1}}, : \right]\right) \times_ {2} \dots \times_ {t} f _ {\theta_ {v}} \left(\mathbf {E} _ {v} \left[ k _ {i _ {t}}, : \right]\right). \tag {18}
$$
The original value-grained selection tensor  $\mathbf{A}_v^t$  consists of  $\mathcal{O}(m^t)$  elements. The trainable elements is reduced to  $\mathcal{O}(md')$  after the Takagi factorization [2] and to  $\mathcal{O}(\hat{d}(m + d'))$  after the multi-mode tensor factorization [5].
B Dataset and Preprocessing
We conduct our experiments on two public real-world benchmark datasets. The statistics of all datasets are given in Table 3. We describe all these datasets and the pre-processing steps below.
Table 3: Dataset Statistics
Dataset	#samples	#field	#value	pos ratio
Criteo	4.6 × 107	39	6.8 × 106	0.2562
Avazu	4.0 × 107	24	4.4 × 106	0.1698
KDD12	1.5 × 108	11	6.0 × 106	0.0445
Industrial	3.0 × 108	134	2498	0.1220
Note: #samples refers to the total samples in the dataset, #field refers to the number of feature fields for original features, #value refers to the number of feature values for original features, pos ratio refers to the positive ratio.
Criteo dataset consists of ad click data over a week. It consists of 26 categorical feature fields and 13 numerical feature fields. Following the best practice [30], we discretize each numeric value  $x$  to  $\lfloor \log^2 (x) \rfloor$ , if  $x > 2$ ;  $x = 1$  otherwise. We replace infrequent categorical features with a default "OOV" (i.e. out-of-vocabulary) token, with min_count=2.
Avazu dataset contains 10 days of click logs. It has 24 fields with categorical features. Following the best practice [30], we remove the instance_id field and transform the timestamp field into three new fields: hour, weekday and is_wednesday. We replace infrequent categorical features with the "OOV" token, with min_count=2.
KDD12 dataset contains training instances derived from search session logs. It has 11 categorical fields, and the click field is the number of times the user clicks the ad. We replace infrequent features with an "OOV" token, with min_count=10.
Industrial dataset is a private large-scale commercial dataset. This dataset contains nearly 3.5 million samples with 134 feature fields and 2498 feature values. Please notice that this dataset has more feature fields and fewer feature values, which differs from the previous benchmarks with fewer feature fields and larger feature values. The results of this dataset are shown in Appendix C.5.
C.1 Feature Interaction Operation
In this section, we conduct an ablation study on the feature interaction operation, comparing the performance of the default setting, which uses the inner product, with the outer product operation. We evaluate these operations on OptFeature and its two variants: OptFeature-f and OptFeature-v. The results are summarized in Table 4.
Table 4: Performance Comparison over Different Feature Interaction Operation.
Dataset	Criteo	Avazu	KDD12
Category	Model	AUC	Logloss	AUC	Logloss	AUC	Logloss
inner product	OptFeature-f	0.8115	0.4404	0.7920	0.3744	0.7978	0.1530
OptFeature-v	0.8116	0.4403	0.7920	0.3742	0.7981	0.1529
OptFeature	0.8116	0.4402	0.7925	0.3741	0.7982	0.1529
outer product	OptFeature-f	0.8114	0.4404	0.7896	0.3760	0.7957	0.1535
OptFeature-v	0.8113	0.4405	0.7902	0.3752	0.7961	0.1533
OptFeature	0.8115	0.4403	0.7899	0.3753	0.7961	0.1533
From the table, we observe that the inner product operation outperforms the outer product operation. This performance gap is particularly significant on the Avazu and KDD12 datasets, while it is relatively insignificant on the Criteo dataset. The drop in performance with the outer product operation is likely due to the introduction of a significantly larger number of inputs into the final predictor. This makes it more challenging for the predictor to effectively balance the information from raw inputs and feature interactions.
C.2 Dimension Selection
In this section, we perform an ablation study on the feature interaction selection dimension  $\hat{d}$ . We compare the AUC performance with the corresponding dimension  $\hat{d}$  and present the results in Figure 5. From the figure, we can observe that as the dimension  $\hat{d}$  increases, the AUC performance remains relatively consistent over the Criteo dataset. This suggests that it is relatively easy to distinguish value-level selection on the Criteo dataset. However, on the Avazu and KDD12 datasets, the AUC performance improves as the selection dimension  $\hat{d}$  increases. This indicates that distinguishing informative values is comparatively more challenging on these two datasets.
Figure 5: Ablation over feature interaction selection dimension on OptFeature.
C.3 Higher-Order Feature Interactions
In this section, we investigate the influence of higher-order feature interactions over the final results on the KDD12 dataset. We compare
the default setting which only considers second-order interactions with two other settings: (i) only third-order interactions and (ii) both second and third-order interactions. We visualize the result in Figure 6.
From the figure, we can draw the following observations. First, only considering third-order interactions leads to the worst performance. This aligns with the common understanding that second-order interactions are typically considered the most informative in deep sparse prediction [15]. Second, for field-level selection, the performance improves when both second and third-order interactions are incorporated into the model. This finding is consistent with previous studies [11, 7], as the inclusion of additional interactions introduces extra information that enhances the performance. In contrast, for value-level selection, the performance tends to decrease when both second and third-order interactions are included. This could be attributed to the fact that value-level selection operates at a finer-grained level and might be more challenging to optimize directly. Finally, OptFeature constantly outperforms its two variants over all settings. This indicates the feasibility and effectiveness of hybrid-grained selection, which combines both field-level and value-level interactions.
C.4 Selection Visualization
(a) Criteo Dataset
(b) Avazu Dataset
Figure 7: Visualization of the Feature Interaction Selection Results.
(c) KDD12 Dataset
In this section, we present the visualization of the interaction selection results for OptFeature and its two variants in Figure 7. OptFeature-f performs a binary selection for each interaction field, allowing for easy visualization through a heatmap representation where one indicates keep and zero indicates drop. On the other hand, OptFeature-v and OptFeature involve value-level interaction selection. Hence, we visualize them by setting each element as the percentage of being selected over the training set. The detailed equation for calculating the value for the interaction field (i, j) is shown in Equation 19.
(a) AUC
(b) Logloss
Figure 6: Performance Comparison over Different Feature Interaction Orders.
$$
\mathbf {P} _ {(i, j)} = \frac {\# \text {S a m p l e s k e e p i n g i n t e r a c t i o n f i e l d (i , j)}}{\# \text {T r a i n i n g S a m p l e s}} \tag {19}
$$