Fieldwise Factorized Networks for Tabular Data Classification

1 INTRODUCTION . Deep learning methods have recently made extraordinary progress by using architectures that utilize the regular structure of visual or linguistic data . However , one of the most common type of data consists of tables with arbitrarily ordered fields ( columns ) . Such tabular data are unable to exploit neural architectures with strong structural inductive biases . In fact , deep learning methods have not been able to convincingly outperform classical baselines on such datasets . As such , traditional machine learning methods , such as Gradient-Boosted Decision Trees ( Friedman ( 2000 ) ) and Factorization Machines ( Rendle ( 2010 ) ; Zhang et al . ( 2018 ) ; Pan et al . ( 2018 ) ; Sun et al . ( 2021 ) ) , still dominate even against recently proposed , specialized neural architectures ( Arik & Pfister , 2021 ; Yoon et al. , 2020 ) . In this paper , we first investigate the connection between shallow neural networks and factorization machine techniques , which are a dominant approach for solving benchmark tabular classification tasks , such as click-through-rate ( CTR ) prediction and user-movie recommendations . Based on this theoretical connection , we propose a neural network-based framework : fieldwise factorized neural networks ( F2NN ) . Our framework is general , and is aimed for tabular classification . By changing a single activation layer in F2NN , our framework transforms between being a factorization machine , a shallow ReLU network or a wide-and-deep model . Our framework consists of a set of fieldwise wide yet shallow neural networks , which are aggregated by summation and passed through non-linearity and classification layers . Differently from typical factorization machine methods , our framework can handle numeric and categorical data as well as multiclass outputs . Our framework is characterized by two main components : i ) learning a high-dimensional representation per-field by factorized networks , ii ) an adaptive activation layer acting on the representation aggregated from all fieldwise networks . Although the theoretical analysis shows that high-dimensional fieldwise embeddings are desired , mapping the high-dimensional features to them requires huge linear layers . Factorizing these layers using low-dimensional layers makes training such fieldwise networks feasible . We also show that for these fieldwise factorized networks , width is preferable to depth . Another key insight is that frameworks with different post-aggregation activation functions can be suitable for different types of tasks . While quadratic activations are optimal for user-product recommendations , ReLU networks are sometimes better for fully-numeric tasks . We thus seek a post-aggregation activation that can enjoy the best of both worlds , and demonstrate that GELU ( Hendrycks & Gimpel , 2016 ) satisfies these requirements . Our theoretical analysis shows that the GELU activation behaves as either quadratic and ReLU activations at different input scales and is thus suitable across the full range of tabular tasks . Consequently , GELU can be used as an efficient alternative for exhaustive search over the optimal activation or an ensemble approach . We present an extensive experimental analysis , evaluating our framework on a wide range of popular numeric and categorical tabular classification tasks as well as CTR prediction . We find that our simple method performs comparably or better than tree-based methods and recent complex deep learning method on general classification , and outperforms strong baselines on CTR prediction . Our results show that the network width and activation play a bigger role than depth or attention for tabular data . Our contributions include : ( 1 ) Establishing a connection between factorization machines and shallow neural networks . ( 2 ) Presentation of a general , theoretically grounded framework for tabular data classification . Our method can express many popular methods by simply modifying its activation function . It also makes learning high-dimensional fieldwise representations feasible by low-rank factorization . ( 3 ) Extensive evaluation on a range of tabular classification tasks and demonstrating above or comparable to state-of-the-art performance . 1.1 RELATED WORK . Tree-based models : Ensembles of decision trees , such as GBDT ( Gradient Boosting Decision Tree ) ( Friedman , 2001 ) , are typically the top-choice for tabular data classification . At the moment , there are several established GBDT libraries , such as XGBoost ( Chen & Guestrin ( 2016 ) ) , LightGBM ( Ke et al . ( 2017 ) ) , CatBoost ( Prokhorenkova et al . ( 2018 ) ) , which are widely used by both machine learning researchers and practitioners . Tree-based methods have some limitations that are addressed by other methods such as neural networks : requirement for feature engineering , poor scaling to high input cardinality , reduced representation transferability and GPU acceleration . FM-based models : Click-through rate ( CTR ) prediction plays a key role in recommendation systems and online advertising . Factorization machine models are a popular family of methods for CTR prediction , including : matrix factorization ( Koren ( 2008 ) ) , factorization machine ( FM , Rendle ( 2010 ) ) , Field-aware Factorization Machines ( FFM , Zhang et al . ( 2018 ) ) , Field-weighted Factorization Machine ( FwFM , Pan et al . ( 2018 ) ) and Field-matrixed Factorization Machines ( FmFM , Sun et al . ( 2021 ) ) . We will give a more in-depth introduction to FM-based methods in Sec . 2 . As a step toward combining deep neural network for this task , Wide & Deep ( Cheng et al . ( 2016 ) ) proposed to train a joint network that combines a linear model and a deep neural network . DeepFM ( Guo et al . ( 2017 ) ) suggest to learn a low-order feature interactions through the FM component instead of the linear model . Since then , various embedding-based neural networks have been proposed to improve the performance ( Deng et al . ( 2021 ) ; He & Chua ( 2017 ) ; Wang et al . ( 2017 ) ; Lian et al . ( 2018 ) ) . Unlike He & Chua ( 2017 ) , our framework is able to learn higher dimensional representations , as well as leveraging non-linear embeddings for the numerical fields . It is also more general , as it able to recover range of tabular models by changing the activation . Fieldwise models : Li et al . ( 2020 ) present a model for categorical data , that utilizes linear models with variance and low-rank constraints , and is also interpretable in a field-wise manner . Although we share the ideas of fieldswise low-rank factorization , our framework allows non-linearity , as well as handles both numerical and categorical fields . Luo et al . ( 2020 ) propose NON to take advantage of intra-field information and non-linear interactions . However , our components are much simpler yet effective . Our fieldwise networks rely on low-rank factorization of wider networks and our nonlinearity is based on an adaptive activation , while NON utilize only ReLU activation and ensembles multiple heavy aggregation mechanisms , such as attention and deep neural networks . Deep models for general tabular data : While classical methods are still the industry favorite , some recent work propose to use deep learning for tabular data . For example , TabNet ( Arik & Pfister ( 2021 ) ) uses neural networks to mimic decision trees by placing importance on only a few features at each layer , using modified attention layers . Yoon et al . ( 2020 ) propose VIME , which employs MLPs in a technique for pre-training based on denoising . Transformer models for more general tabular data include TabTransformer ( Huang et al . ( 2020 ) ) , which uses a transformer encoder to learn contextual embeddings only on categorical features . The main issue with this model is that numerical data do not go through the self-attention block , but are only fed to an MLP . SAINT ( Somepalli et al . ( 2021 ) ) address that issue by projecting numerical features to the higher dimensional embedding space and passing them , together with the categorical embeddings , through the transformer blocks . In addition , SAINT propose using attention in the rows level , to explicitly allow data points to attend each other . Critiques of deep neural networks for tabular data : Recently , several works criticized the proposed deep models for tabular data . For example , several works criticize MLP since it is unsuitable for modeling multiplicative interactions between features ( Rendle et al . ( 2020 ) ) . In addition , Shwartz-Ziv & Armon ( 2020 ) criticize deep tabular data models on claiming to outperform XGBoost , while their study shows that XGBoost outperforms these deep models across the datasets . Deep learning on unordered sets : Qi et al . ( 2017 ) suggest a unified architecture for handling point cloud data . From a data structure point of view , both point cloud and tabular data are an unordered set of vectors . The basic architecture of their network is surprisingly simple as in the initial stages each point is processed identically and independently . In addition , key to this approach is the use of max pooling for aggregating informative points of the point cloud . Our framework architecture shares similar architecture ideas , by the independent processing of each field , followed by aggregation . However , we use a different network for every field while they share the same network to all points ( due to the symmetric of point cloud data ) . 2 THE THEORETICAL CONNECTION BETWEEN FACTORIZATION MACHINES AND FIELDWISE NEURAL NETWORKS . In this section , we will briefly overview factorization machine approaches . We will then demonstrate a theoretical connection between them and shallow neural networks . This will form the basis for our final , generalized framework in Sec . 3 . 2.1 BACKGROUND . In this section we describe a common approach for classification of tabular data . For ease of explanation , we detail the case of binary classification of multi-field categorical data . However , there is no loss of generality , our final approach applies for categorical and numerical as well as for multi-class tabular datasets . Preliminaries : A training dataset consists of S labelled samples { ( x ( 1 ) , y ( 1 ) ) , ( x ( 2 ) , y ( 2 ) ) .. ( x ( S ) , y ( S ) ) } , where x and y are a sample and label respectively . Each sample x is specified by C categorical features f1 , .. fC and J different fields F1 , ... FJ . Each field may contain multiple features , while each feature belongs to only one field . An example of fields are ” Country ” or ” City ” , whereas features could be ” Japan ” or ” Rome ” . To simplify the notation , we use index i to represent the feature fi and F ( i ) to represent the field which fi belongs to . SF denotes the set of features belonging to field f and JF ( i ) represents their number . We denote x ( s ) i = 1 if feature i is active for this instance , otherwise x ( s ) i = 0 . We denote the one-hot vector of features per-field xF = concat { xi ∈ SF } . We denote the embedding of feature i as ei ∈ RK , where K is the ( usually small ) feature embedding dimension . We denote by EF ∈ RK×JF ( i ) the field embedding matrix whose rows are the embeddings ei of the features SF belonging to the field f . Logistic Regression ( LR ) is probably the most widely used model for this task . However , linear models lack the capability to represent feature interactions as cross features are often significant ( Chapelle et al . ( 2014 ) ) . Degree-2 Polynomials ( Poly2 ) ( Chang et al . ( 2010 ) ) model a general way to address this problem . Such models learn a dedicated weight , Wij , for each feature conjunction resulting in a field interaction matrix W ∈ RC×C . Factorization Machine Models : Estimating W is hard due to its huge dimensionality and missing values . Factorization-machine models propose to learn the effect of feature conjunctions by low-rank factorization of the interaction matrix W . Factorization machine methods approximate the feature interaction strength as the scalar product of their embeddings weighted by a matrix MF ( i ) , F ( j ) ∈ RK×K , that depends on the fields of the two features : Wi , j ≈ eTi MF ( i ) , F ( j ) ej . Different factorization machine methods are distinguished by their particular choice of matrix MF ( i ) , F ( j ) . We present the choices of MF ( i ) , F ( j ) taken by four representative factorization ma- chine methods in Eq . 1 : FM ( Koren , 2008 ) , FFM ( Zhang et al. , 2018 ) , FwFM ( Pan et al. , 2018 ) , FmFM ( Sun et al. , 2021 ) : MF ( i ) , F ( j ) = IK ΦFM PTF ( j ) PF ( i ) ΦFFM IK · rF ( i ) , F ( j ) ΦFwFM Entire Matrix ΦFmFM ( 1 ) Where IK is the identity matrix , PF ( i ) is a per-field sparse binary projection matrix , rF ( i ) , F ( j ) is a scalar value learned for every pair of fields ( F ( i ) , F ( j ) ) . For FM and FFM , M is not learned , while for FwFM a scalar per pair of fields is learned , and for FmFM the entire matrix is learned . See Appendix B for detailed formulation of M for each FM-based model .

Summary: - The authors proposed a novel deep learning model for tabular data classification. - The authors utilize the field-wise factorized networks to extract the meaningful information from both categorical and numerical features of the tabular data. Then, using those embeddings to solve the classification problem. - The experimental results showed that the proposed method achieved comparable performances in comparison to SOTA deep learning models for tabular data (SAINT).