Progressive Feature Interaction Search for Deep Sparse Network

1 Introduction . The deep sparse network ( DSN ) [ 9 , 4 ] is a special type of deep networks , which targets learning from sparse and categorical features . Such a learning problem frequently appears in many industrial applications . Take the click-through rate ( CTR ) prediction as an example , features such as user id , user age , item id , and item category , are usually sparse and high-multidimensional [ 4 ] . In advertisement recommendation [ 27 ] , there are sparse features such as category , brand , etc . In fraud detection [ 23 ] , there are sparse features such as income level , region , etc . A general DSN framework ( see Figure 1 ) consists of three layers . First , the embedding layer transforms the raw sparse features into dense embeddings . Then the feature-interaction layer can construct cross-features of raw features ’ embeddings via feature-interaction operation . The cross-features are further fed into the output layer to calculate the prediction score . To achieve high prediction accuracy , researchers found that the crux of the deep sparse networks is to design complex featureinteraction layers [ 9 , 31 , 5 , 18 , 25 ] . For example , in Figure 1 , the second-order cross-feature , AB , is constructed by two raw features , A and B , via feature interaction . DeepFM [ 9 ] feed the two-order cross-features into multi-layer perceptions for implicitly generating high-order cross-features . Further advances replace the MLP in DeepFM with more complex neural networks , including deep cross network in xDeepFM [ 18 ] , outer product network in PNN [ 25 ] , self-attention network in AutoInt [ 31 ] , and logarithmic transformation 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . network in AFN [ 5 ] . Since these deep neural networks work as a black box , high-order cross-features are learned in an implicit manner . Neural architecture search ( NAS ) [ 8 , 39 ] , has become a powerful approach in replacing human experts in proposing both effective and lightweight models for various areas such as computer vision [ ? ] , natural language processing [ 29 ] , etc . More recently , there are some works that adopt neural architecture search for designing the feature-interaction layer in DSN . For example , SIF [ 40 ] searches simple first order interaction function between users and items ; AutoFIS [ 20 ] and AutoGroup [ 19 ] explicitly search for more powerful higher order cross-features . Although the searched featureinteraction layer is more lightweight to some extent , these methods are suffering from high search costs , resulting in limited prediction performance and application value . However , these works require extremely large search costs , limiting their application values on large datasets . In general , the human-designed DSNs are faced with the efficiency issue since they mainly pursue prediction accuracy . In fact , efficiency has a very high impact on the application value of machine learning models in real-world scenarios [ 3 , 33 , 28 ] . Considering that today ’ s machine learning systems are always deployed at diverse hardware platforms , inference time and model size are the two most significant factors . Specifically , the inference time can largely affect the time delay of the user services , which largely affects the application value . The model size can determine whether the model can be deployed at specific devices , which sets the constraint to the model ’ s application scenarios , especially for edge devices . Therefore , these complicated deep sparse networks are still limited due to their inferior efficiency . In this work , to address the above-mentioned efficiency-accuracy dilemma of state-of-the-art deep sparse networks , we propose to utilize NAS to design lightweight and effective models . However , this problem suffers from two challenges . exponentially with the order of cross-features and searching in this space is very hard and time-consuming . Second , in sparse prediction tasks , there exists the order-priority property , which is hard to handle . Specifically , the lower-order cross-features should be assigned higher priority compared with high-order ones . To tackle them , we propose a PROFIT method ( short for PROgressive Feature InTeraction Search ) . We first design a distilled search space , a low-rank approximation of the full search space , which can vastly reduce the difficulty of finding desired models . More precisely , the full search space , which uses a tensor to represent all possible architectures , is approximated by the composition of low-rank tensors . We then develop a progressive search algorithm that searches cross-features from lower orders to higher orders sequentially . When searching for the higher orders , the corresponding architecture parameters of lower orders are fixed , helping distinguish the various importance of different orders . We conduct experiments on three real-world benchmark datasets and compare both accuracy and efficiency with state-of-the-art models . Empirical results demonstrate that the model searched by our PROFIT can achieve high accuracy while keeping low computation cost and model size . Further studies verify the feasibility of each design , consisting of the search space and search algorithm . 2 Related Work . 2.1 Neural architecture search ( NAS ) . Neural architecture search ( NAS ) [ 42 , 1 , 39 ] replaces human experts in designing neural networks architectures , which has been demonstrated a promising approach in many research areas , e.g. , vision and natural language processing . There are two significant perspectives that determines its utility : search space and search algorithm . The search space is defined as the set of all possible architectures . It is task-specific and should be general enough to cover existing models but not too general to incur high search costs . The search algorithm attempts to find the desired architecture in a given space . A good search algorithm should well explore domain properties to find good architectures and achieve high search efficiency . More recently , some pioneer works [ 10 , 2 ] propose NAS methods for automatically designing lightweight models for computer vision tasks , which can reduce the inference time on edge devices . Compared with the existing works , our work approach the problem of searching for deep sparse networks , of which there exist critical challenges of large search space and high search cost . 2.2 Deep Sparse Networks ( DSNs ) . As mentioned in Section 1 , deep sparse networks ( DSNs ) serve as the state-of-the-art solution for the sparse prediction task . Its crux is to extract cross-features , which can have strong abilities in prediction , from raw input features . Here , for the sparse prediction task , we recap the evolution from earlier shallow models to DSNs . Factorization machine [ 27 ] is a pioneer work that assigned a linear matrix factorization operation to all possible second-order cross-features ( built with two raw features ) , which was extended by NFM [ 13 ] and AFM [ 35 ] . To overcome the shortcomings of these shallow feature-interaction operations , deep sparse networks [ 9 , 18 , 5 , 31 , 25 ] are proposed . In general , deep sparse networks adopt neural feature-interaction layers to extract cross-features , as illustrated in Figure 1 . Wide & Deep [ 4 ] and DeepFM [ 9 ] feed the low-order features into multi-layer perceptrons ( MLPs ) to implicitly reveal any order cross-features . More complex neural networks are proposed to serve as the feature-interaction layer , replacing MLPs , such as the compressed interaction network ( CIN ) in xDeepFM [ 18 ] and outer product network in PNN [ 25 ] . There are some recent works [ 31 , 5 ] approaching the cross-feature search of DSNs . Specifically , in AutoInt [ 31 ] , the feature-interaction layer is designed to stack multi-layer self-attention architectures ; AFN [ 5 ] used a logarithmic transformation layer to convert the order to a learnable parameter and then adopted a feature summation as the feature-interaction layer , which fuzzes the order of cross-features . However , there is no model that can achieve both promising efficiency and high accuracy . Thus , it is important to solve the efficiency-accuracy dilemma in DSNs . It is worth mentioning that there are some recent works [ 22 , 41 ] that adopt AutoML in feature embeddings , which is the input of DSNs , for reducing the parameter number of embeddings . 2.3 Deep Network Sparsification . Besides , it is worth mentioning there is another research direction closely related to this work , network sparsification [ 26 , 34 , 11 , 37 ] , which aims to prune the network components or parameters to reduce the network size while keeping a little performance decrease . Different from them , our work searches for the architecture in an interpretable and explicit manner : finding architectures that reserve important cross-features and remove useless ones ; for network pruning models , embedding parameters are dropped in an implicit manner — the pruning masks are always implicitly learned from data , does not provide any interpretability . In deep sparse prediction tasks , it is more significant to explicitly lighten the model at the level of cross-feature , while existing works of network sparsification or network pruning is at the level of latent dimension . 3 The Proposed Method . 3.1 The Search Problem . Existing works of DSNs [ 18 , 31 , 5 ] have shown that the feature-interaction layer is an important component . It can be defined as a network module of which the input is embeddings of raw features , and the output is interactions of embeddings . The interactions have extracted features ’ co-relations and can then be fed to the output layer for obtaining the final prediction results . A feature-interaction layer that extracts o-th order cross-features can be formulated as Layero= { fc1 fc2 · · · fco |co ∈ Co } , s.t . |Co|≤No , ( 1 ) of which c denotes a l-size vector of involved features ’ indexes of a o-th cross-feature ( co , c2 , ... , co are indexes of raw features ) , f denotes the embeddings of raw features ( i.e. , the output of embedding layer ) , and Co denotes the set of o-th order cross-features on which there is a sparsity constraint of No , ensuring both the quality of cross-features and the model efficiency . The feature interaction is perturbation invariant , i.e. , f1 f2 = f2 f1 . The element-wise product is the most used operation for feature interaction [ 18 , 31 , 5 ] . Let F = { f1 , f2 , · · · , fM } . Then , in short , the general objective of DSN can be formulated as follows , minF L ( φ ( F , { Co } ) ; D ) , ( 2 ) where L denotes the loss function and φ is the prediction function , of which a linear predictor or MLP can be used . The objective of the AutoDSN problem should be finding good sets of feature-interaction cells , { Co } , with o from 1 to O , formulated as follows . { Co } ∗ = min { Co } L ( φ ( F∗ , { Co } ) ; Dval ) , s.t . { F∗ = argminF L ( φ ( F , { Co } ) ; Dtra ) , |Co|≤ No , ∀o = 1 , 2 , ... , O , ( 3 ) where Dtra and Dval denote the training and validation datasets , respectively . For the AutoDSN problem , it is a challenging problem due to multiple aspects . First , the search space for C is relatively large , which burdens the search cost . Second , the desired sets are discrete , and there are also sparse constraints , making it an NP-hard problem . Third , the desired cross-features are closely co-related with each other , from lower orders to higher orders , which should be carefully considered .

Summary: This paper proposes a neural architecture search approach, PROFIT,  for deep sparse networks.  PROFIT introduces a distilled low-rank search space and a progressive search algorithm from the lower orders. The proposed method is evaluated on three benchmark datasets and the authors conduct extensive ablation study and discussion on the method. The paper is well written and very easy to follow even for people without a NAS background.