Fieldwise Factorized Networks for Tabular Data Classification

1 INTRODUCTION . Deep learning methods have recently made extraordinary progress by using architectures that utilize the regular structure of visual or linguistic data . However , one of the most common type of data consists of tables with arbitrarily ordered fields ( columns ) . Such tabular data are unable to exploit neural architectures with strong structural inductive biases . In fact , deep learning methods have not been able to convincingly outperform classical baselines on such datasets . As such , traditional machine learning methods , such as Gradient-Boosted Decision Trees ( Friedman ( 2000 ) ) and Factorization Machines ( Rendle ( 2010 ) ; Zhang et al . ( 2018 ) ; Pan et al . ( 2018 ) ; Sun et al . ( 2021 ) ) , still dominate even against recently proposed , specialized neural architectures ( Arik & Pfister , 2021 ; Yoon et al. , 2020 ) . In this paper , we first investigate the connection between shallow neural networks and factorization machine techniques , which are a dominant approach for solving benchmark tabular classification tasks , such as click-through-rate ( CTR ) prediction and user-movie recommendations . Based on this theoretical connection , we propose a neural network-based framework : fieldwise factorized neural networks ( F2NN ) . Our framework is general , and is aimed for tabular classification . By changing a single activation layer in F2NN , our framework transforms between being a factorization machine , a shallow ReLU network or a wide-and-deep model . Our framework consists of a set of fieldwise wide yet shallow neural networks , which are aggregated by summation and passed through non-linearity and classification layers . Differently from typical factorization machine methods , our framework can handle numeric and categorical data as well as multiclass outputs . Our framework is characterized by two main components : i ) learning a high-dimensional representation per-field by factorized networks , ii ) an adaptive activation layer acting on the representation aggregated from all fieldwise networks . Although the theoretical analysis shows that high-dimensional fieldwise embeddings are desired , mapping the high-dimensional features to them requires huge linear layers . Factorizing these layers using low-dimensional layers makes training such fieldwise networks feasible . We also show that for these fieldwise factorized networks , width is preferable to depth . Another key insight is that frameworks with different post-aggregation activation functions can be suitable for different types of tasks . While quadratic activations are optimal for user-product recommendations , ReLU networks are sometimes better for fully-numeric tasks . We thus seek a post-aggregation activation that can enjoy the best of both worlds , and demonstrate that GELU ( Hendrycks & Gimpel , 2016 ) satisfies these requirements . Our theoretical analysis shows that the GELU activation behaves as either quadratic and ReLU activations at different input scales and is thus suitable across the full range of tabular tasks . Consequently , GELU can be used as an efficient alternative for exhaustive search over the optimal activation or an ensemble approach . We present an extensive experimental analysis , evaluating our framework on a wide range of popular numeric and categorical tabular classification tasks as well as CTR prediction . We find that our simple method performs comparably or better than tree-based methods and recent complex deep learning method on general classification , and outperforms strong baselines on CTR prediction . Our results show that the network width and activation play a bigger role than depth or attention for tabular data . Our contributions include : ( 1 ) Establishing a connection between factorization machines and shallow neural networks . ( 2 ) Presentation of a general , theoretically grounded framework for tabular data classification . Our method can express many popular methods by simply modifying its activation function . It also makes learning high-dimensional fieldwise representations feasible by low-rank factorization . ( 3 ) Extensive evaluation on a range of tabular classification tasks and demonstrating above or comparable to state-of-the-art performance . 1.1 RELATED WORK . Tree-based models : Ensembles of decision trees , such as GBDT ( Gradient Boosting Decision Tree ) ( Friedman , 2001 ) , are typically the top-choice for tabular data classification . At the moment , there are several established GBDT libraries , such as XGBoost ( Chen & Guestrin ( 2016 ) ) , LightGBM ( Ke et al . ( 2017 ) ) , CatBoost ( Prokhorenkova et al . ( 2018 ) ) , which are widely used by both machine learning researchers and practitioners . Tree-based methods have some limitations that are addressed by other methods such as neural networks : requirement for feature engineering , poor scaling to high input cardinality , reduced representation transferability and GPU acceleration . FM-based models : Click-through rate ( CTR ) prediction plays a key role in recommendation systems and online advertising . Factorization machine models are a popular family of methods for CTR prediction , including : matrix factorization ( Koren ( 2008 ) ) , factorization machine ( FM , Rendle ( 2010 ) ) , Field-aware Factorization Machines ( FFM , Zhang et al . ( 2018 ) ) , Field-weighted Factorization Machine ( FwFM , Pan et al . ( 2018 ) ) and Field-matrixed Factorization Machines ( FmFM , Sun et al . ( 2021 ) ) . We will give a more in-depth introduction to FM-based methods in Sec . 2 . As a step toward combining deep neural network for this task , Wide & Deep ( Cheng et al . ( 2016 ) ) proposed to train a joint network that combines a linear model and a deep neural network . DeepFM ( Guo et al . ( 2017 ) ) suggest to learn a low-order feature interactions through the FM component instead of the linear model . Since then , various embedding-based neural networks have been proposed to improve the performance ( Deng et al . ( 2021 ) ; He & Chua ( 2017 ) ; Wang et al . ( 2017 ) ; Lian et al . ( 2018 ) ) . Unlike He & Chua ( 2017 ) , our framework is able to learn higher dimensional representations , as well as leveraging non-linear embeddings for the numerical fields . It is also more general , as it able to recover range of tabular models by changing the activation . Fieldwise models : Li et al . ( 2020 ) present a model for categorical data , that utilizes linear models with variance and low-rank constraints , and is also interpretable in a field-wise manner . Although we share the ideas of fieldswise low-rank factorization , our framework allows non-linearity , as well as handles both numerical and categorical fields . Luo et al . ( 2020 ) propose NON to take advantage of intra-field information and non-linear interactions . However , our components are much simpler yet effective . Our fieldwise networks rely on low-rank factorization of wider networks and our nonlinearity is based on an adaptive activation , while NON utilize only ReLU activation and ensembles multiple heavy aggregation mechanisms , such as attention and deep neural networks . Deep models for general tabular data : While classical methods are still the industry favorite , some recent work propose to use deep learning for tabular data . For example , TabNet ( Arik & Pfister ( 2021 ) ) uses neural networks to mimic decision trees by placing importance on only a few features at each layer , using modified attention layers . Yoon et al . ( 2020 ) propose VIME , which employs MLPs in a technique for pre-training based on denoising . Transformer models for more general tabular data include TabTransformer ( Huang et al . ( 2020 ) ) , which uses a transformer encoder to learn contextual embeddings only on categorical features . The main issue with this model is that numerical data do not go through the self-attention block , but are only fed to an MLP . SAINT ( Somepalli et al . ( 2021 ) ) address that issue by projecting numerical features to the higher dimensional embedding space and passing them , together with the categorical embeddings , through the transformer blocks . In addition , SAINT propose using attention in the rows level , to explicitly allow data points to attend each other . Critiques of deep neural networks for tabular data : Recently , several works criticized the proposed deep models for tabular data . For example , several works criticize MLP since it is unsuitable for modeling multiplicative interactions between features ( Rendle et al . ( 2020 ) ) . In addition , Shwartz-Ziv & Armon ( 2020 ) criticize deep tabular data models on claiming to outperform XGBoost , while their study shows that XGBoost outperforms these deep models across the datasets . Deep learning on unordered sets : Qi et al . ( 2017 ) suggest a unified architecture for handling point cloud data . From a data structure point of view , both point cloud and tabular data are an unordered set of vectors . The basic architecture of their network is surprisingly simple as in the initial stages each point is processed identically and independently . In addition , key to this approach is the use of max pooling for aggregating informative points of the point cloud . Our framework architecture shares similar architecture ideas , by the independent processing of each field , followed by aggregation . However , we use a different network for every field while they share the same network to all points ( due to the symmetric of point cloud data ) . 2 THE THEORETICAL CONNECTION BETWEEN FACTORIZATION MACHINES AND FIELDWISE NEURAL NETWORKS . In this section , we will briefly overview factorization machine approaches . We will then demonstrate a theoretical connection between them and shallow neural networks . This will form the basis for our final , generalized framework in Sec . 3 . 2.1 BACKGROUND . In this section we describe a common approach for classification of tabular data . For ease of explanation , we detail the case of binary classification of multi-field categorical data . However , there is no loss of generality , our final approach applies for categorical and numerical as well as for multi-class tabular datasets . Preliminaries : A training dataset consists of S labelled samples { ( x ( 1 ) , y ( 1 ) ) , ( x ( 2 ) , y ( 2 ) ) .. ( x ( S ) , y ( S ) ) } , where x and y are a sample and label respectively . Each sample x is specified by C categorical features f1 , .. fC and J different fields F1 , ... FJ . Each field may contain multiple features , while each feature belongs to only one field . An example of fields are ” Country ” or ” City ” , whereas features could be ” Japan ” or ” Rome ” . To simplify the notation , we use index i to represent the feature fi and F ( i ) to represent the field which fi belongs to . SF denotes the set of features belonging to field f and JF ( i ) represents their number . We denote x ( s ) i = 1 if feature i is active for this instance , otherwise x ( s ) i = 0 . We denote the one-hot vector of features per-field xF = concat { xi ∈ SF } . We denote the embedding of feature i as ei ∈ RK , where K is the ( usually small ) feature embedding dimension . We denote by EF ∈ RK×JF ( i ) the field embedding matrix whose rows are the embeddings ei of the features SF belonging to the field f . Logistic Regression ( LR ) is probably the most widely used model for this task . However , linear models lack the capability to represent feature interactions as cross features are often significant ( Chapelle et al . ( 2014 ) ) . Degree-2 Polynomials ( Poly2 ) ( Chang et al . ( 2010 ) ) model a general way to address this problem . Such models learn a dedicated weight , Wij , for each feature conjunction resulting in a field interaction matrix W ∈ RC×C . Factorization Machine Models : Estimating W is hard due to its huge dimensionality and missing values . Factorization-machine models propose to learn the effect of feature conjunctions by low-rank factorization of the interaction matrix W . Factorization machine methods approximate the feature interaction strength as the scalar product of their embeddings weighted by a matrix MF ( i ) , F ( j ) ∈ RK×K , that depends on the fields of the two features : Wi , j ≈ eTi MF ( i ) , F ( j ) ej . Different factorization machine methods are distinguished by their particular choice of matrix MF ( i ) , F ( j ) . We present the choices of MF ( i ) , F ( j ) taken by four representative factorization ma- chine methods in Eq . 1 : FM ( Koren , 2008 ) , FFM ( Zhang et al. , 2018 ) , FwFM ( Pan et al. , 2018 ) , FmFM ( Sun et al. , 2021 ) : MF ( i ) , F ( j ) = IK ΦFM PTF ( j ) PF ( i ) ΦFFM IK · rF ( i ) , F ( j ) ΦFwFM Entire Matrix ΦFmFM ( 1 ) Where IK is the identity matrix , PF ( i ) is a per-field sparse binary projection matrix , rF ( i ) , F ( j ) is a scalar value learned for every pair of fields ( F ( i ) , F ( j ) ) . For FM and FFM , M is not learned , while for FwFM a scalar per pair of fields is learned , and for FmFM the entire matrix is learned . See Appendix B for detailed formulation of M for each FM-based model .

Summary: The paper proposes a neural network architecture, inspired from factorization machines, for classification task from tabular data. In particular, the proposed factorized neural networks (F2NN) leverage the capabilities of shallow networks for learning factorized low-level representations that can approximate (the typically) high-dimensional representations of each field. This,in turn, together with different choices of activation factions, provides a generalization over different factorization approaches, and can lead to good (albeit not always better) performance on different classification tasks. Results from several (16) experiments on binary and multi-class classification tasks and two CTR tasks, comparing F2NN to different benchmark algorithms, show that it has comparable performance to some of standard (SOTA) methods.