Progressive Feature Interaction Search for Deep Sparse Network

1 Introduction . The deep sparse network ( DSN ) [ 9 , 4 ] is a special type of deep networks , which targets learning from sparse and categorical features . Such a learning problem frequently appears in many industrial applications . Take the click-through rate ( CTR ) prediction as an example , features such as user id , user age , item id , and item category , are usually sparse and high-multidimensional [ 4 ] . In advertisement recommendation [ 27 ] , there are sparse features such as category , brand , etc . In fraud detection [ 23 ] , there are sparse features such as income level , region , etc . A general DSN framework ( see Figure 1 ) consists of three layers . First , the embedding layer transforms the raw sparse features into dense embeddings . Then the feature-interaction layer can construct cross-features of raw features ’ embeddings via feature-interaction operation . The cross-features are further fed into the output layer to calculate the prediction score . To achieve high prediction accuracy , researchers found that the crux of the deep sparse networks is to design complex featureinteraction layers [ 9 , 31 , 5 , 18 , 25 ] . For example , in Figure 1 , the second-order cross-feature , AB , is constructed by two raw features , A and B , via feature interaction . DeepFM [ 9 ] feed the two-order cross-features into multi-layer perceptions for implicitly generating high-order cross-features . Further advances replace the MLP in DeepFM with more complex neural networks , including deep cross network in xDeepFM [ 18 ] , outer product network in PNN [ 25 ] , self-attention network in AutoInt [ 31 ] , and logarithmic transformation 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . network in AFN [ 5 ] . Since these deep neural networks work as a black box , high-order cross-features are learned in an implicit manner . Neural architecture search ( NAS ) [ 8 , 39 ] , has become a powerful approach in replacing human experts in proposing both effective and lightweight models for various areas such as computer vision [ ? ] , natural language processing [ 29 ] , etc . More recently , there are some works that adopt neural architecture search for designing the feature-interaction layer in DSN . For example , SIF [ 40 ] searches simple first order interaction function between users and items ; AutoFIS [ 20 ] and AutoGroup [ 19 ] explicitly search for more powerful higher order cross-features . Although the searched featureinteraction layer is more lightweight to some extent , these methods are suffering from high search costs , resulting in limited prediction performance and application value . However , these works require extremely large search costs , limiting their application values on large datasets . In general , the human-designed DSNs are faced with the efficiency issue since they mainly pursue prediction accuracy . In fact , efficiency has a very high impact on the application value of machine learning models in real-world scenarios [ 3 , 33 , 28 ] . Considering that today ’ s machine learning systems are always deployed at diverse hardware platforms , inference time and model size are the two most significant factors . Specifically , the inference time can largely affect the time delay of the user services , which largely affects the application value . The model size can determine whether the model can be deployed at specific devices , which sets the constraint to the model ’ s application scenarios , especially for edge devices . Therefore , these complicated deep sparse networks are still limited due to their inferior efficiency . In this work , to address the above-mentioned efficiency-accuracy dilemma of state-of-the-art deep sparse networks , we propose to utilize NAS to design lightweight and effective models . However , this problem suffers from two challenges . exponentially with the order of cross-features and searching in this space is very hard and time-consuming . Second , in sparse prediction tasks , there exists the order-priority property , which is hard to handle . Specifically , the lower-order cross-features should be assigned higher priority compared with high-order ones . To tackle them , we propose a PROFIT method ( short for PROgressive Feature InTeraction Search ) . We first design a distilled search space , a low-rank approximation of the full search space , which can vastly reduce the difficulty of finding desired models . More precisely , the full search space , which uses a tensor to represent all possible architectures , is approximated by the composition of low-rank tensors . We then develop a progressive search algorithm that searches cross-features from lower orders to higher orders sequentially . When searching for the higher orders , the corresponding architecture parameters of lower orders are fixed , helping distinguish the various importance of different orders . We conduct experiments on three real-world benchmark datasets and compare both accuracy and efficiency with state-of-the-art models . Empirical results demonstrate that the model searched by our PROFIT can achieve high accuracy while keeping low computation cost and model size . Further studies verify the feasibility of each design , consisting of the search space and search algorithm . 2 Related Work . 2.1 Neural architecture search ( NAS ) . Neural architecture search ( NAS ) [ 42 , 1 , 39 ] replaces human experts in designing neural networks architectures , which has been demonstrated a promising approach in many research areas , e.g. , vision and natural language processing . There are two significant perspectives that determines its utility : search space and search algorithm . The search space is defined as the set of all possible architectures . It is task-specific and should be general enough to cover existing models but not too general to incur high search costs . The search algorithm attempts to find the desired architecture in a given space . A good search algorithm should well explore domain properties to find good architectures and achieve high search efficiency . More recently , some pioneer works [ 10 , 2 ] propose NAS methods for automatically designing lightweight models for computer vision tasks , which can reduce the inference time on edge devices . Compared with the existing works , our work approach the problem of searching for deep sparse networks , of which there exist critical challenges of large search space and high search cost . 2.2 Deep Sparse Networks ( DSNs ) . As mentioned in Section 1 , deep sparse networks ( DSNs ) serve as the state-of-the-art solution for the sparse prediction task . Its crux is to extract cross-features , which can have strong abilities in prediction , from raw input features . Here , for the sparse prediction task , we recap the evolution from earlier shallow models to DSNs . Factorization machine [ 27 ] is a pioneer work that assigned a linear matrix factorization operation to all possible second-order cross-features ( built with two raw features ) , which was extended by NFM [ 13 ] and AFM [ 35 ] . To overcome the shortcomings of these shallow feature-interaction operations , deep sparse networks [ 9 , 18 , 5 , 31 , 25 ] are proposed . In general , deep sparse networks adopt neural feature-interaction layers to extract cross-features , as illustrated in Figure 1 . Wide & Deep [ 4 ] and DeepFM [ 9 ] feed the low-order features into multi-layer perceptrons ( MLPs ) to implicitly reveal any order cross-features . More complex neural networks are proposed to serve as the feature-interaction layer , replacing MLPs , such as the compressed interaction network ( CIN ) in xDeepFM [ 18 ] and outer product network in PNN [ 25 ] . There are some recent works [ 31 , 5 ] approaching the cross-feature search of DSNs . Specifically , in AutoInt [ 31 ] , the feature-interaction layer is designed to stack multi-layer self-attention architectures ; AFN [ 5 ] used a logarithmic transformation layer to convert the order to a learnable parameter and then adopted a feature summation as the feature-interaction layer , which fuzzes the order of cross-features . However , there is no model that can achieve both promising efficiency and high accuracy . Thus , it is important to solve the efficiency-accuracy dilemma in DSNs . It is worth mentioning that there are some recent works [ 22 , 41 ] that adopt AutoML in feature embeddings , which is the input of DSNs , for reducing the parameter number of embeddings . 2.3 Deep Network Sparsification . Besides , it is worth mentioning there is another research direction closely related to this work , network sparsification [ 26 , 34 , 11 , 37 ] , which aims to prune the network components or parameters to reduce the network size while keeping a little performance decrease . Different from them , our work searches for the architecture in an interpretable and explicit manner : finding architectures that reserve important cross-features and remove useless ones ; for network pruning models , embedding parameters are dropped in an implicit manner — the pruning masks are always implicitly learned from data , does not provide any interpretability . In deep sparse prediction tasks , it is more significant to explicitly lighten the model at the level of cross-feature , while existing works of network sparsification or network pruning is at the level of latent dimension . 3 The Proposed Method . 3.1 The Search Problem . Existing works of DSNs [ 18 , 31 , 5 ] have shown that the feature-interaction layer is an important component . It can be defined as a network module of which the input is embeddings of raw features , and the output is interactions of embeddings . The interactions have extracted features ’ co-relations and can then be fed to the output layer for obtaining the final prediction results . A feature-interaction layer that extracts o-th order cross-features can be formulated as Layero= { fc1 fc2 · · · fco |co ∈ Co } , s.t . |Co|≤No , ( 1 ) of which c denotes a l-size vector of involved features ’ indexes of a o-th cross-feature ( co , c2 , ... , co are indexes of raw features ) , f denotes the embeddings of raw features ( i.e. , the output of embedding layer ) , and Co denotes the set of o-th order cross-features on which there is a sparsity constraint of No , ensuring both the quality of cross-features and the model efficiency . The feature interaction is perturbation invariant , i.e. , f1 f2 = f2 f1 . The element-wise product is the most used operation for feature interaction [ 18 , 31 , 5 ] . Let F = { f1 , f2 , · · · , fM } . Then , in short , the general objective of DSN can be formulated as follows , minF L ( φ ( F , { Co } ) ; D ) , ( 2 ) where L denotes the loss function and φ is the prediction function , of which a linear predictor or MLP can be used . The objective of the AutoDSN problem should be finding good sets of feature-interaction cells , { Co } , with o from 1 to O , formulated as follows . { Co } ∗ = min { Co } L ( φ ( F∗ , { Co } ) ; Dval ) , s.t . { F∗ = argminF L ( φ ( F , { Co } ) ; Dtra ) , |Co|≤ No , ∀o = 1 , 2 , ... , O , ( 3 ) where Dtra and Dval denote the training and validation datasets , respectively . For the AutoDSN problem , it is a challenging problem due to multiple aspects . First , the search space for C is relatively large , which burdens the search cost . Second , the desired sets are discrete , and there are also sparse constraints , making it an NP-hard problem . Third , the desired cross-features are closely co-related with each other , from lower orders to higher orders , which should be carefully considered .

Summary: In this paper, the authors study the problem of deep sparse prediction based on NAS. In order to reduce the search cost, they propose a distilled search space, which is a low-rank approximation of the full space. They also propose a progressive differentiable search to capture the order-priority in sparse prediction. Numerous experiments are conducted to demonstrate the effectiveness of the proposed method.