Year: 2021

Abstract:
Tabular data is one of the most common data-types in machine learning, however, deep neural networks have not yet convincingly outperformed classical baselines on such datasets. In this paper, we first investigate the theoretical connection between neural network and factorization machine techniques, and present fieldwise factorized neural networks (F2NN), a neural network architecture framework that is aimed for tabular classification. Our framework learns high-dimensional field representations by a low-rank factorization, and handles both categorical and numerical fields. Furthermore, we show that simply by changing our penultimate activation function, the framework recovers a range of popular tabular classification methods. We evaluate our method against state-of-the-art tabular baselines, including tree-based and deep neural network methods, on a range of tasks. Our findings suggest that our theoretically grounded but simple and shallow neural network architecture achieves as strong or better results than more complex methods.

heading: 1 INTRODUCTION
text: Deep learning methods have recently made extraordinary progress by using architectures that utilize the regular structure of visual or linguistic data. However, one of the most common type of data consists of tables with arbitrarily ordered fields (columns). Such tabular data are unable to exploit neural architectures with strong structural inductive biases. In fact, deep learning methods have not been able to convincingly outperform classical baselines on such datasets. As such, traditional machine learning methods, such as Gradient-Boosted Decision Trees (Friedman (2000)) and Factorization Machines (Rendle (2010); Zhang et al. (2018); Pan et al. (2018); Sun et al. (2021)), still dominate even against recently proposed, specialized neural architectures (Arik & Pfister, 2021; Yoon et al., 2020).
In this paper, we first investigate the connection between shallow neural networks and factorization machine techniques, which are a dominant approach for solving benchmark tabular classification tasks, such as click-through-rate (CTR) prediction and user-movie recommendations. Based on this theoretical connection, we propose a neural network-based framework: fieldwise factorized neural networks (F2NN). Our framework is general, and is aimed for tabular classification. By changing a single activation layer in F2NN, our framework transforms between being a factorization machine, a shallow ReLU network or a wide-and-deep model. Our framework consists of a set of fieldwise wide yet shallow neural networks, which are aggregated by summation and passed through non-linearity and classification layers. Differently from typical factorization machine methods, our framework can handle numeric and categorical data as well as multiclass outputs.
Our framework is characterized by two main components: i) learning a high-dimensional representation per-field by factorized networks, ii) an adaptive activation layer acting on the representation aggregated from all fieldwise networks. Although the theoretical analysis shows that high-dimensional fieldwise embeddings are desired, mapping the high-dimensional features to them requires huge linear layers. Factorizing these layers using low-dimensional layers makes training such fieldwise networks feasible. We also show that for these fieldwise factorized networks, width is preferable to depth. Another key insight is that frameworks with different post-aggregation activation functions can be suitable for different types of tasks. While quadratic activations are optimal for user-product recommendations, ReLU networks are sometimes better for fully-numeric tasks. We thus seek a post-aggregation activation that can enjoy the best of both worlds, and demonstrate that GELU
(Hendrycks & Gimpel, 2016) satisfies these requirements. Our theoretical analysis shows that the GELU activation behaves as either quadratic and ReLU activations at different input scales and is thus suitable across the full range of tabular tasks. Consequently, GELU can be used as an efficient alternative for exhaustive search over the optimal activation or an ensemble approach.
We present an extensive experimental analysis, evaluating our framework on a wide range of popular numeric and categorical tabular classification tasks as well as CTR prediction. We find that our simple method performs comparably or better than tree-based methods and recent complex deep learning method on general classification, and outperforms strong baselines on CTR prediction. Our results show that the network width and activation play a bigger role than depth or attention for tabular data.
Our contributions include: (1) Establishing a connection between factorization machines and shallow neural networks. (2) Presentation of a general, theoretically grounded framework for tabular data classification. Our method can express many popular methods by simply modifying its activation function. It also makes learning high-dimensional fieldwise representations feasible by low-rank factorization. (3) Extensive evaluation on a range of tabular classification tasks and demonstrating above or comparable to state-of-the-art performance.

heading: 1.1 RELATED WORK
text: Tree-based models: Ensembles of decision trees, such as GBDT (Gradient Boosting Decision Tree) (Friedman, 2001), are typically the top-choice for tabular data classification . At the moment, there are several established GBDT libraries, such as XGBoost (Chen & Guestrin (2016)), LightGBM (Ke et al. (2017)), CatBoost (Prokhorenkova et al. (2018)), which are widely used by both machine learning researchers and practitioners. Tree-based methods have some limitations that are addressed by other methods such as neural networks: requirement for feature engineering, poor scaling to high input cardinality, reduced representation transferability and GPU acceleration.
FM-based models: Click-through rate (CTR) prediction plays a key role in recommendation systems and online advertising. Factorization machine models are a popular family of methods for CTR prediction, including: matrix factorization (Koren (2008)), factorization machine (FM, Rendle (2010)), Field-aware Factorization Machines (FFM, Zhang et al. (2018)), Field-weighted Factorization Machine (FwFM, Pan et al. (2018)) and Field-matrixed Factorization Machines (FmFM, Sun et al. (2021)). We will give a more in-depth introduction to FM-based methods in Sec. 2. As a step toward combining deep neural network for this task, Wide & Deep (Cheng et al. (2016)) proposed to train a joint network that combines a linear model and a deep neural network. DeepFM (Guo et al. (2017)) suggest to learn a low-order feature interactions through the FM component instead of the linear model. Since then, various embedding-based neural networks have been proposed to improve the performance (Deng et al. (2021); He & Chua (2017); Wang et al. (2017); Lian et al. (2018)). Unlike He & Chua (2017), our framework is able to learn higher dimensional representations, as well as leveraging non-linear embeddings for the numerical fields. It is also more general, as it able to recover range of tabular models by changing the activation.
Fieldwise models: Li et al. (2020) present a model for categorical data, that utilizes linear models with variance and low-rank constraints, and is also interpretable in a field-wise manner. Although we share the ideas of fieldswise low-rank factorization, our framework allows non-linearity, as well as handles both numerical and categorical fields. Luo et al. (2020) propose NON to take advantage of intra-field information and non-linear interactions. However, our components are much simpler yet effective. Our fieldwise networks rely on low-rank factorization of wider networks and our nonlinearity is based on an adaptive activation, while NON utilize only ReLU activation and ensembles multiple heavy aggregation mechanisms, such as attention and deep neural networks.
Deep models for general tabular data: While classical methods are still the industry favorite, some recent work propose to use deep learning for tabular data. For example, TabNet (Arik & Pfister (2021)) uses neural networks to mimic decision trees by placing importance on only a few features at each layer, using modified attention layers. Yoon et al. (2020) propose VIME, which employs MLPs in a technique for pre-training based on denoising. Transformer models for more general tabular data include TabTransformer (Huang et al. (2020)), which uses a transformer encoder to learn contextual embeddings only on categorical features. The main issue with this model is that numerical data do not go through the self-attention block, but are only fed to an MLP. SAINT (Somepalli et al. (2021))
address that issue by projecting numerical features to the higher dimensional embedding space and passing them, together with the categorical embeddings, through the transformer blocks. In addition, SAINT propose using attention in the rows level, to explicitly allow data points to attend each other.
Critiques of deep neural networks for tabular data: Recently, several works criticized the proposed deep models for tabular data. For example, several works criticize MLP since it is unsuitable for modeling multiplicative interactions between features (Rendle et al. (2020)). In addition, Shwartz-Ziv & Armon (2020) criticize deep tabular data models on claiming to outperform XGBoost, while their study shows that XGBoost outperforms these deep models across the datasets.
Deep learning on unordered sets: Qi et al. (2017) suggest a unified architecture for handling point cloud data. From a data structure point of view, both point cloud and tabular data are an unordered set of vectors. The basic architecture of their network is surprisingly simple as in the initial stages each point is processed identically and independently. In addition, key to this approach is the use of max pooling for aggregating informative points of the point cloud. Our framework architecture shares similar architecture ideas, by the independent processing of each field, followed by aggregation. However, we use a different network for every field while they share the same network to all points (due to the symmetric of point cloud data).

heading: 2 THE THEORETICAL CONNECTION BETWEEN FACTORIZATION MACHINES AND FIELDWISE NEURAL NETWORKS
text: In this section, we will briefly overview factorization machine approaches. We will then demonstrate a theoretical connection between them and shallow neural networks. This will form the basis for our final, generalized framework in Sec. 3.

heading: 2.1 BACKGROUND
text: In this section we describe a common approach for classification of tabular data. For ease of explanation, we detail the case of binary classification of multi-field categorical data. However, there is no loss of generality, our final approach applies for categorical and numerical as well as for multi-class tabular datasets.
Preliminaries: A training dataset consists of S labelled samples {(x(1), y(1)), (x(2), y(2))..(x(S), y(S))} , where x and y are a sample and label respectively. Each sample x is specified by C categorical features f1, ..fC and J different fields F1, ...FJ . Each field may contain multiple features, while each feature belongs to only one field. An example of fields are ”Country” or ”City”, whereas features could be ”Japan” or ”Rome”. To simplify the notation, we use index i to represent the feature fi and F (i) to represent the field which fi belongs to. SF denotes the set of features belonging to field f and JF (i) represents their number. We denote x (s) i = 1 if feature i is active for this instance, otherwise x (s) i = 0. We denote the one-hot vector of features per-field xF = concat{xi ∈ SF }. We denote the embedding of feature i as ei ∈ RK , where K is the (usually small) feature embedding dimension. We denote by EF ∈ RK×JF (i) the field embedding matrix whose rows are the embeddings ei of the features SF belonging to the field f .
Logistic Regression (LR) is probably the most widely used model for this task. However, linear models lack the capability to represent feature interactions as cross features are often significant (Chapelle et al. (2014)). Degree-2 Polynomials (Poly2) (Chang et al. (2010)) model a general way to address this problem. Such models learn a dedicated weight, Wij , for each feature conjunction resulting in a field interaction matrix W ∈ RC×C . Factorization Machine Models: Estimating W is hard due to its huge dimensionality and missing values. Factorization-machine models propose to learn the effect of feature conjunctions by low-rank factorization of the interaction matrix W . Factorization machine methods approximate the feature interaction strength as the scalar product of their embeddings weighted by a matrix MF (i),F (j) ∈ RK×K , that depends on the fields of the two features: Wi,j ≈ eTi MF (i),F (j)ej . Different factorization machine methods are distinguished by their particular choice of matrix MF (i),F (j). We present the choices of MF (i),F (j) taken by four representative factorization ma-
chine methods in Eq. 1: FM (Koren, 2008), FFM (Zhang et al., 2018), FwFM (Pan et al., 2018), FmFM (Sun et al., 2021):
MF (i),F (j) =  IK ΦFM PTF (j)PF (i) ΦFFM
IK · rF (i),F (j) ΦFwFM Entire Matrix ΦFmFM
(1)
Where IK is the identity matrix, PF (i) is a per-field sparse binary projection matrix, rF (i),F (j) is a scalar value learned for every pair of fields (F (i), F (j)). For FM and FFM, M is not learned, while for FwFM a scalar per pair of fields is learned, and for FmFM the entire matrix is learned. See Appendix B for detailed formulation of M for each FM-based model.

heading: 2.2 FACTORIZATION-BASED MODELS AS A FIELDWISE FACTORIZED NEURAL NETWORK
text: In this section, we show that factorization machines are a special case of shallow neural networks. Let us denote the second order interaction term as q2 = ∑
i ∑ j xixj · eTi MF (i),F (j)ej . We demon-
strated above that the crux of factorization machine methods is in the representation of the interaction matrix MF (i),F (j). Let M be the matrix that consists of the matrices MF (i),F (j) for all pairs of fields. Since there is no order between the fields, for each pair only one matrix is learned, meaning MFiFj == MFjFi, while the diagonal block matrices are zeros. See Appendix B for a detailed formulation of M. We observe that M is symmetric matrix, and therefore an eigen-decomposition can be applied to it: M = UTΛU . Here we propose to utilize a reduced-rank factorization of M resulting in a field-wise factorization: MF (i),F (j) = UTF (i)ΛUF (j), where UF (i) ∈ R
d×K . Note that the full rank dimension of M is equal to the number of fields multiplied by the number of dimensions per-embedding. Using this factorization the feature interaction term q2 can now be written:
q2 = ∑ i ∑ j (xiUF (i)ei) TΛ(xjUF (j)ej) (2)
We rewrite the sums, as the sum over the field f and the sum over the per-field indices i ∈ SF : q2 = ∑ F ∑ F ′ ( ∑ i∈SF xiei) TUTF ΛUF ′( ∑ j∈SF ′ xjej) (3) Note that as xF is one-hot (only a single element has a non-zero value), the product EFxF =∑ j∈SF ′ xjej can be efficiently computed using an embedding layer EFxF :
q2 = ( ∑ F UFEFxF ) TΛ( ∑ F ′ UF ′EF ′xF ′) (4)
As the right-hand left-hand vectors are equal, it yields the simple expression: q2 = diag(Λ) · ( ∑ F UFEF xF ) 2 (5)
Where the above square is elementwise. As an intuitive explanation, the second order interactions are modelled by several steps: i) embedding of the one-hot per-field feature using linear layer EF . ii) projection of the feature embedding to a higher dimension using the per-field projection matrix UF . iii) summation over the projected embeddings of all fields. iv) computing the scalar product of their square with the diagonal of the matrix Λ. Note that this can be expressed by a shallow neural network that first learns a per-field representation, then aggregates the representations by summation, passes the result through a non-linearity, and then a linear layer. Factorization machines are a particular instantiation of this neural network, when the activation function is a quadratic function. This will be generalized in the next section.

heading: 3 FIELDWISE FACTORIZED NETWORKS FOR TABULAR CLASSIFICATION
text: We propose fieldwise-factorized neural-networks, a general framework for tabular classification.

heading: 3.1 GENERAL FRAMEWORK
text: In this section, we overview our proposed framework: fieldwise-factorized neural-networks (F2NN), while detailing its components in the following sections. We learn a dedicated neural network ϕF for each field that takes as input the values of the field xF (one-hot for categorical fields, scalar for numeric fields) and returns a high-dimensional embedding ϕF (xF ). The architecture of the fieldwise neural network will be described in Sec. 3.2. The aggregated high-dimensional feature is obtained by summing over the fields ∑ F ϕf (xF ). It is then passed through an activation function σ. Our activation σ allows different activation functions for different dimensions. The post-activation results are finally multiplied by output linear layer Wout, mapping it to the output logits zout:
zout = Woutσ( ∑ F ϕF (xF )) (6)
Note that differently from factorization machine, but similarly to other neural networks, our framework is able to handle both binary and multiclass classification tasks.
The framework is characterized by two main components: per-field representation by factorized networks ϕF , and an adaptive activation σ on the aggregated representations of all fields. We will describe them in detail in the next sections.

heading: 3.2 FIELDWISE FACTORIZED NETWORKS
text: The theoretical basis in Sec. 2 suggests that the learned field embeddings should be of highdimension to factorize a full-rank MF (i),F (j), but that these representations should be low-rank factorized. The motivation behind these design choices is to reduce sample complexity. Modeling field interactions requires a high-dimension while there may not be a sufficient number of samples per-feature for estimating it. Complexity is reduced: i) by learning a per-field representation that does not take other features into account ii) by using low-rank factorization inside these fieldwise networks. Since we are able to use other activation than just than quadratic, d is allowed to be even larger than the full rank dimension. Our framework is able to handle both categorical and numeric fields:
Categorical fields: Following Sec. 2.2, we learn fieldwise networks ϕcatF (xF ) = UFEFxF . We choose the per-feature embedding (output of EF ) to be low-dimensional as there are often many features and limited data per-feature, while matrix UF projects this to high-dimension.
Numeric fields: Differently from categorical fields, numeric fields are ordered which enables learning more complex functions. We choose to learn a factorized one-hidden layer for each field. Our network first projects the scalar value to a high-dimension tF = vF xF + bF (where vt, bF ∈ Rl). The results are passed through a ReLU network, and mapped to the per-field embedding. We found that a high dimensional tF is important for achieving strong performance. However, as tF and the output field-embedding ϕF have a high-dimension, the second linear layer becomes very large. Instead, we choose to low-rank factorize the second layer. The post-activation ReLU(tF ) is projected to a low-dimension, by linear layer EF and is then projected using linear layer UF to the highdimensional field embedding ϕF . Note that as both EF and UF are linear, with no intermediate non-linearity - they are equivalent to a single (low-rank) linear layer. The entire network is therefore a factorized one-hidden layer neural network.
In summary, the fieldwise factorized networks for categorical and numeric variables are given by:
ϕ(xF ) =
{ UFEFxF Categorical
UFEFReLU(vF xF + bF ) Numeric (7)
Note that although SAINT (Somepalli et al. (2021) also use a one-hidden layer network for numeric values, its formulation does not use the low-rank factorization and therefore cannot handle high width, which we show is key to the performance of our method. Also note that our formulation can easily handle deeper fieldwise architectures, but we did not observe benefits from deeper networks.

heading: 3.3 OUR FRAMEWORK CAN EXPRESS POPULAR TABULAR CLASSIFICATION METHODS
text: Despite the simplicity of our framework, it is very general. By changing the fieldwise factorized network ϕF and activation σ, it can express several popular tabular classification methods:
Factorization machine methods: When choosing a quadratic activation σ(x) = x2 and the fieldwise network ϕF (xF ) = UFEFxF our framework becomes a factorization machine-based model. This proof was detailed in Sec. 2.2. Therefore, factorization machine models are expressible by our framework.
One-hidden Layer ReLU Networks: When choosing the activation σ(x) = ReLU(x) and the fieldwise networks as simple linear layers ϕF (x) = UFx, our framework becomes a one-hidden layer neural network. Although more layers can be easily added to our framework after the activation layer σ, we did not find this beneficial in practice. We will present empirical results in Sec. 4.2 for showing this.
Wide-and-Deep: When selecting a fraction of dimensions of σ to have quadratic activations, while the rest are selected to have ReLU activation, our framework becomes a wide-and-deep model (as it is the sum of a ReLU network and a factorization machine). Although the original wide-anddeep method used full rank for the feature interaction Wij , later methods (e.g. Guo et al. (2017)) use different factorization machine varieties. By modifying the choice of dimensionality of the perfeature embedding ei and per-field projection d, all the above methods are expressible by our model. One caveat is that the deep part of our framework only has a single hidden layer, but as mentioned previously, adding further layers has not improved results in our experiments.

heading: 3.4 ADAPTIVE NON-LINEARITY
text: In Sec. 3.3 we established that by choosing different activation function σ, our framework can express a range of popular tabular classification methods. We will show in Sec. 4.3, that in different classification problems, either σ(x) = ReLU(x) or σ(x) = x2 yields significantly better performance. By the analysis in Sec. 3.3 these are cases where either ReLU neural networks or factorization machine achieve better results. By selecting σ a concatenation of ReLU and quadratic functions, we may be able to deal with all cases. However this may not be an efficient solution as it doubles the number of parameters. It would naturally be attractive to use a non-linearity that may be able to automatically adapt to the setting most beneficial to a particular dataset. Here we suggest using the GELU activation function (Hendrycks & Gimpel, 2016) as a bridge between the two paradigms. At the limits, GELU(x) has the following attractive properties:
GELU(x) ≈
{ 0.5x+ √ 1 2πx
2 ∥x∥ << 1 ReLU(x) ∥x∥ → ∞
(8)
Therefore, by varying the magnitude of the input, and utilizing the GELU activation, our framework is able to switch between factorization machine and ReLU network behavior. The adaptive behavior will be demonstrated empirically in Sec. 4.3. GELU can be used as an efficient alternative for exhaustive search over the optimal activation function or an ensemble approach (wide-and-deep).

heading: 3.5 PRACTICAL CONSIDERATIONS: THE DEVIL IS IN THE DETAILS
text: The field of deep learning has made significant steps forward by increasing the depth of neural networks and by using attention-based architectures such as transformers. It might be expected that increasing network depth or using transformers might be beneficial for tabular data. Unfortunately, in our experiments, we did not find evidence to support this. We do not claim that it is impossible to gain by using the two mechanisms, but this is not trivial. More optimistically, in our experiments we found that there are lower hanging fruit. Specifically, we obtained significant gains by choosing suitable width d, l and activation σ, as well as tuning the regularization strength (either dropout or weight decay). We suggest GELU and full rank projections UF (d = J ·K) as robust default values. Although tuning the embedding dimension K can also help, we kept it in-line with the baselines.

heading: 4 EXPERIMENTS

heading: 4.1 PERFORMANCE COMPARISON

heading: 4.1.1 GENERAL TABULAR CLASSIFICATION
text: Evaluation protocol We evaluated our method on the same datasets as SAINT 1 (Somepalli et al. (2021)). The datasets include 14 binary classification tasks and 2 multiclass classification tasks. We did not include Arcene and Arrythmia datasets from the reported results as they are very small datasets and their results have high variance (Arrhythmia contains 452 samples and Arcene contains 200 samples). In addition, we use exactly the same pre-processing as in the official SAINT implementation. Details of these datasets and the pre-processing are provided in Appendix C.
Hyperparameters: Our optimization hyper-parameters follow SAINT. We run a grid search on the hyperparameters presented in Table 5 with seed=1, and selected the best model by the validation set. For fair comparison, we used the same EF dimensionality as SAINT defaults.
Baselines We compare against the baselines reported in SAINT. They include traditional methods such as logistic regression and random forests, as well as powerful boosting libraries XGBoost (Chen & Guestrin (2016)), LightGBM (Ke et al. (2017)), and CatBoost (Prokhorenkova et al. (2018)). They also compare against deep learning methods, like multi-layer perceptrons, VIME (Yoon et al. (2020)), TabNet (Arik & Pfister (2021)), and TabTransformer (Huang et al. (2020)). The numbers were copied from the SAINT paper. In order to compare the performance on the same exact seeds, we re-run the SAINT models on 10 different seeds (1-10). For a fair comparison, we trained SAINT with our best numerical embedding hidden dimension l and their default selection (100), and reported the best of the two settings.
Results The averaged performance for all methods is reported in Table 1, the results per dataset are detailed in Table 9. We reported ROC-AUC for binary classification, and class prediction accuracy or multi-class classification. Our results are slightly better or comparable to SAINT variations, except from on multiclass tasks, where SAINT-s achieves poor performance. Our method also outperforms all of SAINT’s baselines on average. SAINT-s and our framework use exactly the same learning setting (optimization hyper-parameters, datasets). The difference between the architectures is the post-EF architecture. While we use a set of linear projections that are only summed and passed through activation and classification layers, SAINT-s uses a massive six layer Transformer. This highlights that for robust and accurate neural models for tabular data, it does not appear that complex methods have an edge over well-tuned simple models such as ours.

heading: 4.1.2 CTR PREDICTION
text: CTR prediction is a fundamental tabular binary classification task. FM-based models are the most common approach for tackling it. As our framework generalizes FM-based models, we also examined if CTR prediction can benefit from our framework.
Datasets We used Avazu and Criteo which are common CTR prediction benchmarks.
Baselines We compared the framework implementation against FmFM, which is the shallow stateof-the-art of FM-models, and two deep variations - DeepFwFM and DeepFmFM. We implemented our framework on the FmFM public code-base 2, and used their data processing and baselines implementation. We split the data as described in FmFM paper, and applied the same splits for all the models. We conducted a hyper-parameter search for both our method and FmFM, and found the optimal hyperparameters for both were the same. We thus applied the same hyperparameters for the deep baselines. Other implementation details are provided in Appendix C.
Results The ROC-AUC results are reported in Table 1. We reported the results of using the papers hyperparameters and using our tuned hyperparameters, as well as quoted the reported results from the paper. We can observe that even after improving the baselines by a better tuning, our framework outperforms all other baselines.
1https://github.com/somepago/saint 2https://github.com/yahoo/FmFM

heading: 4.2 ABLATION STUDIES
text: In this section we will explore the effect of the framework components on its performance. The experiments in the following section were conducted using SAINT evaluation datasets and process. From resources constraints, we run the experiments on seed=1 only. The results are reported in Table 2, averaged over the datasets, more details are found in Tables 10-11.
Per-field Projections: To examine the importance of the per-field networks, we evaluated sharing the projection matrix UF across fields. We can observe that all datasets (besides HTRU2), gain from the fieldwise projections. In multi-class datasets, the difference is much significant.
Low rank projections: In this experiment we reduced the dimension of the projection matrix UF to that of the output of EF . Overall, we can observe, that the factorized high-dimensional representation is crucial, especially in Forest, Volkert and MNIST.
Numeric fieldwise networks: We tested the following alternatives: (i) a K-dimensional linear layer for each field, (ii) applying non-linearity, without a high-dimensional projection, meaning vF dimension is equal to EF dimension, (iii) projecting to high-dimensional space and reduce to Kdimensional, without a non-linearity. The averaged results over the datasets that contain continuous fields are reported in Table 2 (Right). We can observe that the non-linearity is an important aspect, and boost the performance. In addition, by controlling the width for the high-dimensional space, we gain a further improvement over the low-dimensional non-linear network.
Deeper model: We added 2 extra layers after the activation σ with equal width as field embedding, ϕ(xF ), and matching the model activation and dropout. Since quadratic activation performs poorly when deepening the network, we tested all activations for the datasets that selected a quadratic in their shallow variation, and report the best. While we observed improvement on specific datasets, such as Volkert and Forest, in most of the datasets the results were comparable of even worse than our shallow model with the linear classification head.
The activation after aggregating the fieldwise representations In the original selected models, the selected activations are divided as follow: 6 datasets with a quadratic activation, 4 datasets with ReLU activation, and 4 with GELU activation. In order to explore the effect of the activation, we vary the activation for each dataset, while keeping the other hyperparameters fixed. The averaged

heading: 16 0.6876 0.6972 0.6974 0.6937

heading: 192 0.7164 0.7217 0.7280 0.7278
text: results are reported in Table 2. In many datasets the particular activation was immaterial. However, in some datasets, activation has made a significant difference, so it should be carefully chosen.

heading: 4.3 GELU AS AN AUTOMATICALLY ADAPTED ACTIVATION
text: We conducted an empirical study on the advantage of using GELU activation, by examining two cases - a case where quadratic activations are preferred over ReLU, and a case where ReLU activations are preferred. We demonstrate that by using GELU, we can get the closest result to the preferred activation.
Quadratic is better than ReLU: Rendle (2010) showed that a simple matrix factorization model performs better than MLP and NeuMF models (Zhao et al. (2021)) for collaborative filtering. We applied our method on the authors’ dataset and settings. We tested the effect of different choices of σ on the ability of our framework to express a dot product between fields. Dedicated choices of initialization and regularization were used, see details in Appendix C. We report the results for embedding sizes K = 16 and K = 192, the minimal and maximal embedding sizes that were tested in Rendle (2010). Note that the reported result for matrix factorization of Rendle (2010) is 0.7294 (on embedding size of 192). As expected, we can observe that for each embedding size, quadratic activation performs the best, while ReLU performs poorly. However, GELU achieves similar results to the quadratic performance, demonstrating its robustness and adaptive behaviour.
ReLU is better than Quadratic: On the other hand, from our experiments on forest dataset, we observed that a quadratic activation performs dramatically worse than ReLU ( 94.84% and 99.4%, respectively). However, GELU obtains 99.47% which is in line with ReLU.

heading: 5 DISCUSSION AND CONCLUSIONS
text: In this paper, we presented a theoretically grounded, general framework for handling tabular data classification. Our framework can be extended in many ways. Two promising ideas are custom activations and representation transfer:
Activation for tabular tasks. We highlighted the importance of the activation σ and demonstrated both theoretically and empirically, that GELU activation is suitable across the full range of tabular tasks. This suggests that developing an activation function for tabular classification is very promising.
Exploring the learned representations. As a by-product of tabular data classification, our framework learns fieldwise representations. Future work should examine if the representations are useful for transfer learning for related tasks.

heading: A APPENDIX - F2NN FRAMEWORK ILLUSTRATION
text: An illustration of our framework can be found in Figure 5.

heading: B APPENDIX - FM-BASED MODELS FORMULATIONS
text: Embedding form: FM, FwFM and FmFM, learn C vectors, one for each feature, ei ∈ RK .
FFM learns (J−1) vectors for each feature, eF (j)i ∈ R K J , overall C ·(J−1) vectors. Note that they select much higher K than all other approaches, therefore KJ > 1. ei ∈ R K places these (J − 1) vectors in the relevant indices, while 0K J in its field indices. For example, e1 = [0K J , e21, ..., e J 1 ], e2 = [e 1 2,0K , ..., e J 2 ]. Then, PF (j) ∈ RK×K (RJ· K J ×K) extracts the embedding eF (j)i . For
example, P2 is of the form: P2 =  0K J IK J 0K J .
. 0K
J  Interaction matrix form: In general, for every FM-based model, M takes the following form:
M = 1
2 ·  0K M1,2 M1,3 ... M1,J M1,2 0K M2,3 ... M2,J . . . ... . . . . ... . . . . ... .
M1,J M2,J M3,J ... 0K
 (9)

heading: C APPENDIX - EXPERIMENTAL SETTINGS AND IMPLEMENTATION DETAILS
text: C.1 PERFORMANCE EVALUATION
C.1.1 GENERAL TABULAR DATA
Datasets Detailed information on the evaluated datasets is reported in table 4. It can be seen that the evaluated datasets are diverse, in terms of their size and the amount of fields, and contains both categorical and numerical features. Each of these datasets is publicly available from either UCI 3 or AutoML 4. We use the exact processing as is SAINT implementation, i.e. all the continuous features are Z-normalized, and all categorical features are label-encoded before the data is passed on to the embedding layer.
Implemetation details We implemented our framework in the SAINT public code-base 5. Linear and Embedding layers were initialized using the PyTorch default. We used dropout before UF and after the activation, and a bias for UF and for the numerical EF and tF . As in factorization
3http://archive.ics.uci.edu/ml/datasets.php 4https://automl.chalearn.org/data 5https://github.com/somepago/saint
machine methods, we observe minor benefits from adding linear per-field terms to the final logits. We implemented this by adding to the per-field final layer WF , an identity matrix with the same dimension as the output of EF . The activation sigma for the resulting dimension was chosen to be identity σ(x) = x.
Training Our optimization hyper-parameters follow SAINT. We used the AdamW optimizer with β1 = 0.9, β2 = 0.999, decay = 0.01, and with a learning rate of 0.0001 with batches of size 256 (except for MNIST, which they use a batch size of 32). We trained for 100 epochs. We split the data into 65%, 15%, and 25% for training, validation, and test splits, respectively.
Hyperpararmter search: We performed architecture search on the hyperparameters presented in Table 5 with seed=1. We selected the best model by the validation set. Our experiments employed a fixed embedding and projection sizes to all of the fields (although tuning this might improve results).
C.1.2 CTR PREDICTION
Datasets: The first one is the Criteo CTR data set, it is a well-known benchmark dataset which used for the Criteo Display Advertising Challenge. There are 45 million samples and each sample has 13 numerical fields and 26 categorical fields. The second data set is the Avazu CTR data set, it was used in the Avazu CTR prediction competition, which predicts whether a mobile ad will be clicked. There are 40 million samples and each sample has 23 categorical fields.
Implementation details We implemented our framework on FmFM public code-base 6, and used their data processing and baselines implementation. We split the data as described in FmFM paper, and applied the same splits for all the models. For our implementation, we adopted FmFM model implementation, and changed the main component to a Tensorflow linear layers, an activation between them, and bias for the per-field projection layer. For the weights initialization, FmFM implementation uses weights that are sampled from a normal distribution. For the reported frame-
6https://github.com/yahoo/FmFM
work instance performance, in Criteo we used ReLU activation and a full rank per-field projection layer, and for Avazu we used GELU activation and factor of 5 of the full rank dimension. In this settings, we did not use a dropout between the layers. Note that since the numerical fields in Criteo datasets are discrete, we used the same numerical data processing as in the baselines and consider them as categorical.
Hyperparameters search We found that the hyperparameters in this task are crucial, included the standard deviation of the weights initialization. Therefore, we performed an hyperparameters search both for our framework and for FmFM, though obtained that they should use the same hyperparameters. Thus, we applied this hyperparameters for the deep baselines too. For the hyperparameters search, we implemented our framework by a GELU activation and a full rank per-field projection layer. The exact hyperparameters of each baseline is reported in Table 6.
C.1.3 ACTIVATION STUDY
Rendle (2010) public code is optimized for simplicity and not for efficiency, therefore we have implemented an efficient keras version of it, using Dot layer, and adjusted the hyperparamters to fit their reported results. We used Adam optimizer, batch size of 10k and L2 regularization strength of 2.5e−7, while kept the negative sampling on 8, and a learning rate of 0.002.
Next, since matrix factorization is a special case of our framework, we implemented it by our framework, using quadratic activation and the eigen vectors and values of 0.5 · [ 0K IK IK 0K ] for the per-field projections and for the final classification layer, respectively. The eigenvectors were partitioned with respect to the fields. When freezing these weights, we get an exact implementation of a dot product, which also reflected by a replication of the performance between both of the implementations.
In order to explore the ability of other activation to approximate the results of a dot product based model, we used the eigen decomposition as an initialization and regularization, and searched the best regularization strength for each activation. The regularization strength of the reported results are reported in Table 7.
We reported this experiments results on embedding sizes of 16 and 192, the minimal and maximal embedding sizes that were tested in Rendle (2010).

heading: D APPENDIX - RESULTS PER DATASET
text: Performance comparison: The performance comparison per dataset are reported in Table 9. The results are averaged over seeds 1-10. We reported also the standard deviation.
Ablation studies: Tables 10-11 report the results of the ablation studies per dataset

heading: E APPENDIX - INFERENCE TIME COMPARISON
text: We also examined the runtime between our framework and SAINT. We computed the average inference time over all batches of size 256. The results on three datasets are reported in Table 8. We can observe that using a transformer slows inference without an improvement to the performance.
Ta bl
e 9:
Pe rf
or m
an ce
co m
pa ri
so n
pe rd
at as
et ,A
ve ra
ge d
re su
lts ov
er se
ed s
1- 10 . R O C -A U
C C re di t H T R U 2 Q SA R B io Sh ru tim
e Sp
am ba
se Ph
ili pp
in e
K D
D 99
SA IN
T 97
.8 9 ± 0. 85
97 .9
8 ± 0. 4 2
93 .2
8 ± 1 .4
86 .3
7 ± 1. 0 7
98 .3
7 ± 0. 4 4
81 .2
2 ± 1
10 0 ± 0
SA IN
Ti
97 .9
2 ± 0. 85
98 .0
2 ± 0. 3 5
92 .9 ± 1. 7 9
86 .2
3 ± 1. 2 3
98 .2 ± 0. 3 4
80 .5
6 ± 1. 4 8
10 0 ± 0
SA IN
Ts
98 .0
9 ± 0. 72
97 .9
9 ± 0. 4 4
92 .6
4 ± 1. 4 1
86 .1
8 ± 1. 1 1
98 .2
8 ± 0. 2 1
78 .4
6 ± 1. 8 1
10 0 ± 0
F2 N
N 97
.0 1 ± 1. 31
98 .1
3 ± 0. 3 9
92 .8
9 ± 1. 4 8
86 .4
7 ± 1. 1 5
98 .3
1 ± 0. 3 9
80 .8
1 ± 1. 8 8
10 0 ± 0
R O
C -A
U C
A cc
ur ac y B an k B la st ch ar Fo re st Sh op pe rs In co m e Vo lk er t
M N
IS T
SA IN
T 92
.9 3 ± 0 .4
84 .0
3 ± 0. 9 3
98 .2 ± 0. 2 4
93 .2 ± 0. 4 7
91 .4
4 ± 0 .3 6
69 .5
7 ± 0 .4 5 4 4
97 .8
6 ± 0. 1 0 3 5
SA IN
Ti
93 .1
2 ± 0 .4
83 .9
8 ± 0 .9
94 .7
4 ± 0. 9 7
92 .8
9 ± 0 .4 1
91 .4 ± 0. 3 8
70 .3
4 ± 0 .2 7 4 9
97 .7
1 ± 0. 1 7 7 4
SA IN
Ts
93 .6
7 ± 0. 25
83 .9
9 ± 0. 9 8
99 .7
1 ± 0. 0 2
93 .0
6 ± 0. 5 1
91 .5
6 ± 0. 3
49 .7
1 ± 3. 1 5 6
94 .1
6 ± 0. 3 2 3 2
F2 N
N 93
.4 3 ± 0. 27
84 .0
9 ± 0. 9 4
99 .4
9 ± 0. 0 2
92 .9
7 ± 0. 4 8
91 .5
4 ± 0. 3 6
70 .2
2 ± 0 .3 7 8 4
97 .5
3 ± 0. 1 2 6 6
Ta bl
e 10
:A bl
at io
ns pe
rd at
as et
R O
C -A
U C
A cc
ur ac y C re di t H T R U 2 Q SA R B io Sh ru tim e Sp am ba se Ph ili pp in e K D D 99 B an k B la st ch ar Fo re st Sh op pe rs In co m e Vo lk er t M
N IS
T
F2 N
N 97
.7 3
98 .4
2 94
.1 2
86 .6
98 .4
4 82
.4 7
10 0
93 .0
9 82
.3 8
99 .4
7 93
.5 3
91 .6
2 70
.3 5
97 .7
L ow
-r an
k 96
.9 5
98 .5
7 93
.4 9
86 .6
9 98
.0 9
81 .4
1 10
0 92
.8 8
82 .4
1 96
.3 5
93 .2
91 .6
7 65
.2 7
92 .6 6 Sh ar ed 95 .9 3 98 .4 2 93 .4 8 85 .5 1 98 .0 2 81 .0 2 10 0 92 .0 9 82 .3 99 .4 2 92 .9 6 91 .5 62 .4 4 78 .5 D ee p 97 .4 8 98 .2 5 93 .5 4 85 .3 9 98 .2 4 81 .4 7 10 0 93 .1 1 82 .3 99 .5 8 93 .5 4 91 .7 7 71 .9 93 .2 4 Q ua dr at ic 97 .7 3 98 .4 7 93 .8 1 86 .6 98 .1 1 82 .4 7 10 0 92 .9 5 82 .3 8 94 .8 4 93 .4 91 .6 2 66 .9 3 97 .7 R eL U 97 .8 9 98 .4 2 94 .1 2 85 .9 9 98 .4 4 81 .7 6 10 0 93 .0 6 82 .2 5 99 .4 93 .5 3 91 .7 4 69 .9 3 95 .2 2 G E L U 97 .7 1 98 .4 2 93 .7 7 85 .9 3 98 .4 81 .7 3 10 0 93 .0 9 82 .2 6 99 .4 7 93 .4 9 91 .6 9 70 .3 5 94 .8 8
Ta bl
e 11
:N um
er ic
al ne
tw or
ks ab
la st
io ns
pe rd
at as
et .(
i) st
an ds
fo rU
F E
F x F
,( ii)
re pr
es en
ts l =
K ,a
nd (i
ii) re
pr es
en ts U F E
F (v
F x F + b F )
R O
C -A
U C
A cc
ur ac y C re di t H T R U 2 Q SA R B io Sh ru tim e Sp am ba se Ph ili pp in e K D D 99 B an k B la st ch ar Fo re st Sh op pe rs In co m e Vo lk er t
F2 N
N 97
.7 3
98 .4
2 94
.1 2
86 .6
98 .4
4 82
.4 7
10 0
93 .0
9 82
.3 8
99 .4
7 93
.5 3
91 .6
2 70
.3 5
(i )
97 .8
4 98
.4 8
92 .1
1 85
.2 5
97 .7
8 79
.5 5
10 0
92 .8
6 81
.8 9
99 .1
3 92
.7 6
91 .1
9 70
.7 8
(i i)
97 .8
8 98
.5 5
92 .8
9 86
.3 5
97 .9
6 81
.6 1
10 0
93 .0
2 82
.4 8
99 .3
9 93
.1 2
91 .4
70 .5 5 (i ii) 96 .0 4 98 .3 94 .5 8 85 .4 9 98 .0 1 79 .4 4 10 0 92 .7 2 81 .9 6 99 .1 93 .1 8 91 .2 7 70 .2 6

Review summaries:
- The authors proposed a novel deep learning model for tabular data classification. - The authors utilize the field-wise factorized networks to extract the meaningful information from both categorical and numerical features of the tabular data. Then, using those embeddings to solve the classification problem. - The experimental results showed that the proposed method achieved comparable performances in comparison to SOTA deep learning models for tabular data (SAINT).

This paper presents a general neural network framework for tabular data classification, that can recover a range of popular classification methods. The main components are per-field representation by factorized network and an activation function on aggregated representations of all fields. Experiments on general tabular classification and CTR prediction demonstrate its effectiveness, although performance gain is marginal.   

The paper describes an approach for tabular data classification. In particular, authors proposed the Fieldwise Factorized neural network to tackle the problem defining a general framework in which it is possible to express other tabular classification methods. The proposed approach permits to manage categorical and numerical fields and seems to outperform other methods (on average). Moreover the experiments seem to demonstrate the importance of per-field network and the improvement in the results using GELU instead of the ReLU of Quadratic activations.

The paper proposes a neural network architecture, inspired from factorization machines, for classification task from tabular data. In particular, the proposed factorized neural networks (F2NN) leverage the capabilities of shallow networks for learning factorized low-level representations that can approximate (the typically) high-dimensional representations of each field. This,in turn, together with different choices of activation factions, provides a generalization over different factorization approaches, and can lead to good (albeit not always better) performance on different classification tasks. Results from several (16) experiments on binary and multi-class classification tasks and two CTR tasks, comparing F2NN to different benchmark algorithms, show that it has comparable performance to some of standard (SOTA) methods.