SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fuyuan Lyu $^{1}$ , Xing Tang $^{2\dagger}$ , Dugang Liu $^{4}$ , Chen Ma $^{3}$ , Weihong Luo $^{2}$ , Liang Chen $^{2}$ , Xiuqiang He $^{2}$ , Xue Liu $^{1}$
<sup>1</sup>McGill University, <sup>2</sup>FiT, Tencent, <sup>3</sup>City University of Hong Kong,
<sup>4</sup>Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)
fuyuan.lyu@mail.mcgill.ca, xing.tang@hotmail.com,
dugang.1dg@gmail.com, chenma@cityu.edu.hk,
{lobbyluo,leocchen,xiuqianghe}@tencent.com,
xue.liu@cs.mcgill.ca
Deep sparse networks are widely investigated as a neural network architecture for prediction tasks with high-dimensional sparse features, with which feature interaction selection is a critical component. While previous methods primarily focus on how to search feature interaction in a coarse-grained space, less attention has been given to a finer granularity. In this work, we introduce a hybrid-grained feature interaction selection approach that targets both feature field and feature value for deep sparse networks. To explore such expansive space, we propose a decomposed space which is calculated on the fly. We then develop a selection algorithm called OptFeature, which efficiently selects the feature interaction from both the feature field and the feature value simultaneously. Results from experiments on three large real-world benchmark datasets demonstrate that OptFeature performs well in terms of accuracy and efficiency. Additional studies support the feasibility of our method. All source code are publicly available<sup>1</sup>.
Deep Sparse Networks (DSNs) are commonly utilized neural network architectures for prediction tasks, designed to handle sparse and high-dimensional categorical features as inputs. These networks find widespread application in real-world scenarios such as advertisement recommendation, fraud detection, and more. For instance, in the context of advertisement recommendation, the input often comprises high-dimensional features like user id and City, which significantly contribute to the final prediction.
As depicted in Figure 1(a), the general DSN architecture consists of three components. Firstly, the embedding layer transforms different feature values into dense embeddings. Following this, feature interaction layers create feature interactions [4] based on the embeddings of raw features, as illustrated in Figure 1(b). Finally, the predictor makes the final prediction based on the features and their interactions. A core challenge in making accurate DSNs predictions is effectively capturing suitable feature interactions among input features [8, 11, 16, 15].
Various methods have been proposed to address the issue of modelling feature interactions. Wide&Deep first models human-crafted feature interactions using linear regression [22]. To eliminate the need for human expertise, DeepFM [8] models all second-order feature interactions by utilizing
$^{1}$ https://github.com/fuyuanlyu/OptFeature
a factorization machine [21] and adopts a multi-layer perceptron (MLP) as a predictor. Further advancements have replaced the factorization machine with different operations, such as product operations [18], cross network [24, 25], or an MLP component [19]. However, these methods have inherent drawbacks as they directly model all possible feature interactions. This inevitably introduces noise into the models, increasing their complexity and potentially degrading performance.
Neural architecture search(NAS) [12] has been introduced as a powerful approach for feature interaction selection in DSNs for both efficiency and effectiveness [11, 7, 15]. AutoFIS [11] first propose to select feature interactions by adding an attention gate to each possible one. PROFIT [7] suggests progressively selecting from a distilled search space with fewer parameters. Alongside feature interaction selection, some works like AutoFeature [9] and OptInter [15] aim to search for the interaction operation. However, all these works focus on evaluating the whole feature field as a whole instead of individual feature value. To be more precise, the entire user id or City feature field would be retained or discarded as a unit instead of considering individual feature values like User x479bs or New Orleans. Considering only feature fields inevitably be coarse-grained, which may overlook informative values in uninformative fields, and vice versa.
In this work, we propose extending the selection granularity of feature interactions from the field to the value level. The extension of granularity would significantly increase the complexity of the entire selection space, leading to an increase in exploration time and memory usage. We manage this challenge by decomposing the selection space using tensor factorization and calculating the corresponding parameters on the fly. To further improve the selection efficiency, we introduce a hybrid-grained feature interaction selection space, which explicitly considers the relation between field-level and value-level. To perform feature interaction selection, we develop a sparsification-based selection algorithm named OptFeature(short for Optimizing Feature Interaction Selection), which efficiently selects the feature interaction concurrently from both feature fields and feature values. We conduct experiments over three large-scale real-world benchmarks and compare accuracy and efficiency with
state-of-the-art models. Empirical results demonstrate the superiority of our method on both dimensions. Additional studies support the feasibility of our method.
Figure 1: Illustration of deep sparse network and feature interaction layer
2.1 Deep Sparse Networks (DSNs)
It is commonly believed that feature interaction is the critical challenge towards accurate prediction in deep sparse networks [15]. Various models have been proposed to address the feature interaction layer. DeepFM [8] utilizes a factorization machine [21] to model all second-order feature interactions, followed by a multi-layer perceptron as the predictor. More sophisticated feature interaction layers have been proposed to replace the factorization machine, such as the inner product operation in IPNN [18], the cross-network in DCN [24, 25] or the MLP component in PIN [19].
With the advancement of neural architecture search [31, 14, 12] and continuous sparsification [1], various methods have been proposed to select the informative feature interactions [11, 7] and reduce computational costs. AutoFis [11] employs a sparse optimizer to select suitable feature interactions at the field level. PROFIT [7] formulates field-level feature interaction selection within a distilled search space. It employs a progressive search for efficient exploration. AutoIAS [26] takes one step
further to integrate the feature interaction selection as part of the search space and jointly conduct the search with other components like MLP architecture or embedding dimension. GAIN [13], on the other hand, focuses on the DCN [24]-like architectures and conducts the feature interaction selection jointly with the model training. However, all previous works conduct feature interaction selection on the field level. Our work builds on the existing approaches for modelling feature interactions in DSNs. We extend the selection granularity to the value level and propose a hybrid-grained selection approach.
Furthermore, there also exists some works such as OptInter [15], AutoFeature [9], and NAS-CTR [29] take a different angle and search for suitable operations (such as inner product, outer product or element-wise sum) to model each feature interaction properly. These works are perpendicular to our study.
2.2 Feature Interaction Selection in DSNs
To conduct feature interaction selection in DSNs, previous works [11, 7, 26, 13] borrow the ideas from neural architecture search [26, 11] and continuous sparsification [7, 13]. We briefly introduce these two techniques in the following. Previous works focusing on feature selection [3] are excluded from the discussion.
2.2.1 Neural Architecture Search
Neural architecture search (NAS) automatically identifies optimal architectures for specific tasks and datasets, eliminating the need for expert design [31, 12, 14, 20]. This approach has led to substantial advancements in various domains, including vision [31, 12, 14], natural language processing [23] and sparse representation learning [11, 7]. There are two key aspects to neural architecture search: search space and search algorithm. The search space comprises all potential architectures and is usually task-dependent. An effective search space should be neither too complex, leading to high search costs, nor too shallow, ensuring the inclusion of suitable architectures. The search algorithm explores the search space to find appropriate architectures. Search algorithms can generally be categorized into controller-based [31], evaluation-based [20], and gradient-based [14, 12] classes. Our work distinguishes itself from existing research by addressing the challenge of feature interaction selection for deep sparse networks, where the primary obstacles involve high search costs and expansive search spaces.
2.2.2 Continuous Sparsification
Continuous sparsification focuses on reducing a continuous value to a sparse form and is closely related to our method [1]. It is often used to prune less informative components or parameters from neural networks, which is intrinsically equivalent to solving an  $L_{0}$  normalization problem. This technique has been leveraged in various aspects of deep sparse networks (DSNs), notably in the embedding layer [17] and the feature interaction layer [11]. Our work employs continuous sparsification to discretize the selection of value-level feature interactions during the search process. By doing so, we can effectively manage the complexity of the search space and maintain a high level of efficiency in the search algorithm.
This section will first formulate the general DSNs in Section 3.1. Then it will elaborate on the feature interaction selection problem from field and value perspectives in Section 3.2. We further concrete the proposed OptFeature method, which conducts hybrid-grained feature interaction selection in Section 3.3.
3.1 DSNs Formulation
Feature Field and Value The input of DSNs comprises multiple feature fields, each containing a set of feature values. Hence, two forms of expression exist for each data sample, deriving from feature field and feature value perspectives. More precisely, a data sample  $\mathbf{x}$  consists of  $n$  feature fields can be written as:
field perspective:  $\mathbf{x} = [\mathbf{z}_1, \mathbf{z}_2, \dots, \mathbf{z}_n]$ ,
value perspective:  $\mathbf{x} = [x_{k_1}, x_{k_2}, \dots, x_{k_n}]$ . (1)
The notation  $\mathbf{z}_i = \{x_{k_i} \mid k_i \in [1, m_i], k_i \in N_+\}$  represents the set of feature values within the  $i$ -th field. Here  $m_i$  denotes the number of values in each field, and  $m = \sum_{i=1}^{n} m_i$  refers to that across all fields. An illustration figure is shown in Figure 1(c). We highlight this formulation as it is one of the core differences between our method, which focuses on both field and value perspectives, and previous works [11, 7], which focuses only on the field perspective. This allows us to capture more granular and informative interactions, potentially leading to better model performance.
Deep Sparse Networks As mentioned in Section 1, the DSN commonly consists of three major components: the embedding layer, the feature interaction layer and the predictor. The embedding layer usually employs an embedding table to convert  $\mathbf{z}_i$  from high-dimensional, sparse vectors into low-dimensional, dense embeddings. This can be formulated as:
$$
\mathbf {e} _ {i} = \mathbf {E} \times \mathbf {z} _ {i} = \mathbf {E} \times x _ {k _ {i}}, \tag {2}
$$
where  $\mathbf{E} \in \mathbb{R}^{m \times d}$  is the embedding table,  $d$  is the pre-defined embedding size. These embeddings are further stacked together as the embedding vector  $\mathbf{e} = [\mathbf{e}_1, \mathbf{e}_2, \dots, \mathbf{e}_n]$ , which also serves as the input for feature interaction layer.
The feature interaction layer further performs interaction operations over its input. The  $t$ -th order feature interaction can be generally represented as:
$$
\mathbf {v} ^ {t} = \mathcal {V} ^ {t} (\mathbf {e}), \tag {3}
$$
where  $\mathcal{V}(\cdot)$ , as the  $t$ -th order interaction operation,  $2 \leq t \leq n$ , can vary from a multiple layer perceptron [19] to cross layer [24]. In this paper, we select the inner product, which is recognized to be the most commonly used operation for feature interaction [18, 7], as our interaction operation. For  $t$ -th order feature interactions, cardinality  $|\mathbf{v}^t| = C_n^t$ . Note that the selection of  $t$  is usually task-dependent. For instance, in the click-through rate prediction,  $t = 2$  is considered sufficient [15].
In the prediction layer, both the embedding vector  $\mathbf{e}$  and feature interaction vectors  $\mathbf{v}^t$  are aggregated together to make the final prediction  $\phi_{\theta}(\mathbf{e}, \{\mathbf{v}^t \mid 2 \leq t \leq n\})$ , where  $\phi_{\theta}(\cdot)$  represents the prediction function parameterized on  $\theta$ . The mainstream prediction function is MLP [18, 19]. Finally, the loss function is calculated over each data sample and forms the training object:
$$
\mathcal {L} \left(\mathcal {D} _ {\mathrm {t r a}} \mid \mathbf {W}\right) = \frac {1}{\left| \mathcal {D} _ {\mathrm {t r a}} \right|} \sum_ {(\mathbf {x}, y) \in \mathcal {D} _ {\mathrm {t r a}}} \ell (y, \hat {y}) = \frac {1}{\left| \mathcal {D} _ {\mathrm {t r a}} \right|} \sum_ {(\mathbf {x}, y) \in \mathcal {D} _ {\mathrm {t r a}}} \ell \left(y, \phi_ {\theta} \left(\mathbf {E} \times \mathbf {x}, \left\{\mathbf {v} ^ {t} \mid 2 \leq t \leq n \right\}\right)\right), \tag {4}
$$
where  $\mathbf{W} = \{\mathbf{E},\theta \}$  denotes the model parameters,  $\mathcal{D}_{\mathrm{tra}}$  refers to the training set.
3.2 Feature Interaction Selection
Feature interaction selection is crucial in DSNs [11, 7]. In general DSNs, all feature interactions, whether informative or not, are included as inputs for the final predictor. This indiscriminate inclusion of all interactions inevitably introduces noise into the model, which can hinder its performance [11, 7]. Selecting informative feature interactions is called feature interaction selection. Proper feature interaction selection can significantly improve the accuracy and efficiency of DSNs by reducing the amount of noise and focusing on learning the most informative interactions [11].
For a  $t$ -th order feature interaction  $\mathbf{v}^t = \{\mathbf{v}_{(i_1,i_2,\dots ,i_t)}\}$ ,  $1\leq i_{t}\leq m_{i_{t}}$ , we aim to determine whether to keep each element or not. We take  $t = 2$  as an example to illustrate the problem. The selection can be formulated as learning an gate weight  $\mathbf{a}_{(i_1,i_2)}\in \{0,1\}$  corresponding to each  $\mathbf{v}_{(i_1,i_2)}$ . All  $\mathbf{a}_{(i_1,i_2)}$  compose the second-order feature interaction selection tensor  $\mathbf{A}^2$ , which is formulated as:
$$
\mathbf {A} ^ {2} = \left[ \begin{array}{c c c c c} I & \mathbf {a} _ {(1, 2)} & \mathbf {a} _ {(1, 3)} & \dots & \mathbf {a} _ {(1, n)} \\ \mathbf {a} _ {(1, 2)} ^ {T} & I & \mathbf {a} _ {(2, 3)} & \dots & \mathbf {a} _ {(2, n)} \\ \mathbf {a} _ {(1, 3)} ^ {T} & \mathbf {a} _ {(2, 3)} ^ {T} & I & \dots & \mathbf {a} _ {(3, n)} \\ \vdots & \vdots & \vdots & \ddots & \\ \mathbf {a} _ {(1, n)} ^ {T} & \mathbf {a} _ {(2, n)} ^ {T} & \mathbf {a} _ {(3, n)} ^ {T} & \dots & I \end{array} \right]. \tag {5}
$$
Here the tensor is symmetric due to the feature interaction being invariant to perturbation and the diagonal is composed of identical tensors. According to different perspectives in Equation 5, there are two forms for Equation 1: (i) when  $\mathbf{a}_{(i_1,i_2)}$  is a scalar,  $\mathbf{A}^2$  is a  $2 - D$  matrix which indicates the field level interaction selection  $\mathbf{A}_f^2\in \{0,1\}^{n^2}$ . (ii) when  $\mathbf{a}_{(i_1,i_2)}\in \{0,1\}^{m_{i_1}\times m_{i_2}}$  is a tensor,  $A^2$  is a  $2 - D$  matrix which indicates the value level interaction selection  $\mathbf{A}_v^2\in \{0,1\}^{m^2}$ . Therefore, feature interaction selection tensor  $\mathbf{A}^t$  of different order  $t$  compose the selection tensor set  $\mathbf{A} = \{\mathbf{A}^t\mid 2\leq t\leq n\}$ , which is the learning goal of the feature interaction selection problem.
Field-grained In previous methods [7, 11], the feature interaction selection is usually conducted at the field level. This can be formulated as learning the field-grained selection tensor set  $\mathbf{A} = \{\mathbf{A}_f^t\mid \mathbf{A}_f^t\in \{0,1\}^{n^t},2\leq t\leq n\}$ , where  $\mathbf{A}_f^t$  refers to the field-grained feature interaction selection tensor at  $t$ -th order. Finding a suitable  $\mathbf{A}_f^t$  is an NP-hard problem [7]. Some works have been devoted to investigating the search strategy. AutoFis [11] intuitively assigns a continuous tensor  $\mathbf{C}_f^t\in \mathbb{R}^{n^t}$  and adopts a sparse optimizer for continuous sparsification so that  $\mathbf{C}_f^t$  will approximate  $\mathbf{C}_f^t$  eventually. PROFIT [7], on the other hand, uses symmetric CP decomposition [10] to approximate  $\mathbf{A}_f^t$  result. This can be formulated as  $\hat{\mathbf{A}}_f^t\approx \sum_{r = 1}^{R}\underbrace{\boldsymbol{\beta}^r\circ\cdots\circ\boldsymbol{\beta}^r}_{\mathrm{t~times}}$ , where  $\circ$  denotes broadcast time operation,
$\beta^r \in \mathbb{R}^{1 \times n}$  refers to the decomposed vector and  $R \ll n$  is a pre-defined positive integer.
Value-grained To extend the feature interaction selection from field-grained to value-grained, we focus on the value-grained selection tensor set  $\mathbf{A} = \{\mathbf{A}_v^t\mid \mathbf{A}_v^t\in \{0,1\}^{m^t},2\leq t\leq n\}$ . Although this formulation is analogous to the field-level one, it is substantially more complex given  $m\gg n$ . This expansion in complexity poses a challenge, referred to as the size explosion problem, as the computational cost and memory usage for the value-level selection increases dramatically. So it is hard to utilize previous methods [11, 7] directly into solving the value-grained selection problem. We will discuss how we tackle this size explosion problem in Section 3.3.1 in detail.
Hybrid-grained Apart from learning the field and value-grained selection tensor  $\mathbf{A}_f$  and  $\mathbf{A}_v$ , the hybrid-grained selection also needs to learn a proper hybrid function, we formulate the hybrid function as a binary selection equation,
$$
\mathbf {A} = \left\{\mathbf {A} ^ {t} \mid 2 \leq t \leq n \right\}, \mathbf {A} ^ {t} = \boldsymbol {\alpha} ^ {t} \mathbf {A} _ {f} ^ {t} + (1 - \boldsymbol {\alpha} ^ {t}) \mathbf {A} _ {v} ^ {t} \tag {6}
$$
where tensor  $\alpha^t\in \{0,1\}^{n^t}$  denoting the hybrid choice. As the quality of the field and value-grained selection tensors will heavily influence the hybrid tensor  $\alpha^t$ , joint optimization of both selection tensors  $\mathbf{A}_f$  &  $\mathbf{A}_v$  and hybrid tensor  $\alpha$  is highly desired. We will discuss how we tackle this joint training problem in Section 3.3.2 in detail.
Previous field-grained and value-grained selection can be viewed as a special case for Equation 6. By setting the hybrid tensor  $\alpha$  in Equation 6 as an all-one tensor  $\mathbf{1}$ , the original hybrid-grained selection problem is reduced to a field-grained one. Similarly, the value-grained selection problem can also be obtained from Equation 6 by assigning the hybrid tensor  $\alpha$  as an all-zero tensor  $\mathbf{0}$ .
3.3 OptFeature
In this section, we formalize our proposed method OptFeature aiming to tackle the hybrid-grained feature interaction selection problem depicted in Section 3.2. Such a problem contains two critical issues:
- Size explosion problem: As the number of feature values  $m$  is significantly larger than the number of feature fields  $n$ , denoted as  $m \gg n$ , the value-grained selection tensor  $\mathbf{A}_v$  increase dramatically and is hard to be stored or explored. To provide a concrete example regarding the size explosion, we take the commonly used benchmark Criteo² as an example. This dataset contains 39 feature fields and  $6.8 \times 10^6$  feature values. Even if we only consider  $t = 2$  for the second-order feature interactions, the corresponding selection tensor increase from  $\| \mathbf{A}_f^2 \| = 39^2 = 1521$  to  $\| \mathbf{A}_v^2 \| =$
$^{2}$ https://www.kaggle.com/c/criteo-display-ad-challenge
$6.8 \times 10^{6^2} = 4.6 \times 10^{13}$ , making it impossible to be stored or explored using vanilla NAS approaches [12, 31, 20] or continuous sparsification methods [1, 6].
- Joint training problem: As the quality of the field and value-grained selection tensors  $\mathbf{A}_f^t$  &  $\mathbf{A}_v^t$  will heavily influence the hybrid tensor  $\alpha$ , jointly optimization of both selection tensors and hybrid tensor is required. How to efficiently optimize so many large binary tensors remains unsolved.
To address the two critical issues mentioned in Section 3.2 and above, we propose our method OptFeature with 1) the selection tensor decomposition to address the size explosion issue and 2) sparsification-based selection algorithm for the joint training problem.
3.3.1 Selection Tensor Decomposition
To efficiently explore a large space  $\mathbf{A}_v$ , we first decompose the feature interaction selection tensor. Without the loss of generality, we only focus on the second order, i.e.  $t = 2$ , value-grained feature interaction selection tensor  $\mathbf{A}_v^2$ . More details of the general cases  $t \geq 2$  are listed in the Appendix A. We omit the superscript for simplicity and use  $\mathbf{A}_v$  instead of  $\mathbf{A}_v^2$  in later sections. Given that the selection tensor is semi-positive and symmetric, we have the Takagi factorization [2] as:
$$
\mathbf {A} _ {v} \approx \mathcal {U} \times \boldsymbol {\Sigma} _ {v} \times \mathcal {U} ^ {T}. \tag {7}
$$
Here  $\pmb{\Sigma}_v$  is a  $d' \times d'$  diagonal tensor,  $\mathcal{U} \in R^{m \times d'}$  and  $d' < m$ . To further reduce the consumption of memory and reduce factorization error, we introduce the deep multi-mode tensor factorization [5] that replaces the  $\mathcal{U}$  as an output of one neural network, denoted as:
$$
\mathcal {U} \approx f _ {\theta_ {v}} \left(\mathbf {E} _ {v}\right), \tag {8}
$$
where  $f_{\theta_v} : \mathbb{R}^{m \times \hat{d}} \to \mathbb{R}^{m \times d'}$ ,  $\hat{d} \ll d'$  is a neural network with parameter  $\theta_v$  and  $\mathbf{E}_v \in \mathbb{R}^{m \times \hat{d}}$  is an additional embedding table for generating feature interaction selection tensor. Figure 2 shows a detailed illustration of our decomposition.
Notably, during the training and inference stage, we do not need to calculate the entire metric  $\mathbf{A}$  all in once. We only need to calculate a batch of data in each trial. The element of architecture metric  $\mathbf{A}_v[k_i,k_j]$  can be calculated given the following equation:
$$
\mathbf {A} _ {v} \left[ k _ {i}, k _ {j} \right] = f _ {\theta_ {v}} \left(\mathbf {E} _ {v} \left[ k _ {i}, : \right]\right) \times \boldsymbol {\Sigma} _ {v} \times f _ {\theta_ {v}} ^ {T} \left(\mathbf {E} _ {v} \left[ k _ {j}, : \right]\right). \tag {9}
$$
The original value-grained selection tensor  $\mathbf{A}_v$  consists of  $\mathcal{O}(m^2)$  elements. By introducing the Takagi factorization [2], the number of trainable elements is reduced to  $\mathcal{O}(md')$ . We further reduce that number to  $\mathcal{O}(\hat{d}(m + d'))$  through the multi-mode tensor factorization [5]. Hence, we can train the neural network over the value-level selection on modern hardware.
Figure 2: Illustration of the selection tensor decomposition
3.3.2 Sparsification-based Selection Algorithm
After decomposing the selection tensor, we still need to jointly conduct feature interaction selection and train the model parameters for an efficient search. To help convert