Fieldwise Factorized Networks for Tabular Data Classification
Authors: Chen Almagor
Yedid Hoshen
Venue: ICLR 2022 Submitted (2022)

Abstract: Tabular data is one of the most common data-types in machine learning, however, deep neural networks have not yet convincingly outperformed classical baselines on such datasets. In this paper, we first investigate the theoretical connection between neural network and factorization machine techniques, and present fieldwise factorized neural networks (F2NN), a neural network architecture framework that is aimed for tabular classification. Our framework learns high-dimensional field representations by a low-rank factorization, and handles both categorical and numerical fields. Furthermore, we show that simply by changing our penultimate activation function, the framework recovers a range of popular tabular classification methods. We evaluate our method against state-of-the-art tabular baselines, including tree-based and deep neural network methods, on a range of tasks. Our findings suggest that our theoretically grounded but simple and shallow neural network architecture achieves as strong or better results than more complex methods.

Under review as a conference paper at ICLR 2022
FIELDWISE FACTORIZED NETWORKS FOR TABULAR DATA CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Tabular data is one of the most common data-types in machine learning, however, deep neural networks have not yet convincingly outperformed classical baselines on such datasets. In this paper, we first investigate the theoretical connection be- tween neural network and factorization machine techniques, and present fieldwise factorized neural networks (F2NN), neural network architecture framework that is aimed for tabular classification. Our framework learns high-dimensional field representations by low-rank factorization, and handles both categorical and nu- merical fields. Furthermore, we show that simply by changing our penultimate activation function, the framework recovers range of popular tabular classifica- tion methods. We evaluate our method against state-of-thh-arr tabular baselines, including tree-based and deep neural network methods, findings suggest that our theoretically grounded but simple and shallow neural net- work architecture achieves as strong or better results than more complex methods.
range of tasks. Our
INTRODUCTION
Deep learning methods have recently made extraordinary progress by using architectures that uti- lize the regular structure of visual or linguistic data. However, one of the most common type of data consists of tables with arbitrarily ordered fields (columns). Such tabular data are unable to exploit neural architectures with strong structural inductive biases. In fact, deep learning methods have not been able to convincingly outperform classical baselines on such datasets. As such, tradi- tional machine learning methods, such as Gradient-Booster Decision Trees (Friedman) (2000) and Factorization Machines (Rendle) (2010p; Zhang et al. (2018); Pan al. (2088); (Sun al. (2021) ) still dominate even against recently proposed, specialized neural architectures (Arik & Pfister 2021 Yoon et al. 2020).
In this paper, we first investigate the connection between shallow neural networks and factorization machine techniques, which are a dominant approach for solving benchmark tabular classification tasks, such click- throuhhrouuh--aee (CTR) prediction and user-movie recommendations, Based this theoretical connection, we propose a neural network-bard based framework: fieldwise factorized neural networky (F2NN). Our framework is general, and is aimed for tabular classification. By changing single activation layer in F2NN, our framework transforms between being a factorization machine, a shallow ReLU network or a wide-and-deep model. Our framework consists of a set of fieldwise wide yet shallow neural networks, which are aggregated by summation and passed through non-linearity and classification layers. Differently from typical factorization machine methods, our framework can handle numeric and categorical data as well as multiclass outputs.
Our framework is characterized by two main components: i) learning high-dimensional representa- tion per-field by factorized networks, ii) an adaptive activation layer acting on the representation ag- gregated from all fieldwise networks. Although the theoretical analysis shows that digh-dimensional fieldwise embeddings are desired, mapping the digh-dimensional features to them requires huge lin- ear layers. Factorizing these layers using low-dimensional layers makes training such fieldwise networks feasible. We also show that for these fieldwise factorized networks, width is preferable to depth. Another key insight is that frameworks with different post-aggregation activation functions can be suitable for different types of tasks. While quadratic activations are optimal for user- uroduproduct recommendations, ReLU networks are sometimes better for fully nulleri tasks. We thus seek a post-aggregation activation that enjoy the best of both worlds, and demonstrate that GELU

---

Under review as a conference paper at ICLR 2022
(Hendrycks & Gimpel 2016) satisfies these requirements. Our theoretical analysis shows that the GELU activation behaves as either quadratic and ReLU activations at different imput scales and is thus suitable across the full range of tabular tasks. Consequently, GELU can be used as efficient alternative for exhaustive search over the optimal activation or an ensemble approach.
We present an extensive experimental analysis, evaluating our framework on a wide range of popular numeric and categorical tabular classification tasks as well as CTR prediction. We find that our simple method performs comparably or better than tree-based methods and recent complex deep learning method on general classification, and outperforms strong baselines on CTR prediction. Our results show that the network width and activation play a bigger role than depth or attention for tabular data.
Our contributions include: (1) Establishing a connection between factorization machines and shal- low neural networks. (2) Presentation of a general, theoretically grounded framework for tabular data classification. Our method can express many popular methods by simply modifying its activa- tion function. It also makes learning high-dimensional fieldwise representations feasible by low-rank factorization. (3) Extensive evaluation on a range of tabular classification tasks and demonstrating above or comparable to state-of-the-art performance.
1.1 RELATED WORK
Ensembles of decision trees, such as GBDT (Gradient Boosting Decision Tree) (Friedman, 2001), are typically the top-choice for tabular data classification. . At the moment, there are several established GBDT libraries, such as XGBoost (Chen & Guestrin (016)). LightGBM (Ke et al. (2017), CatBoost (Prokhorenkova et al. (2018)). which are widely used by both machine learning researchers and practitioners.
Tree-based models:
methods have some limitations that are addressed by other methods such as neural networks: requirement for feature engineering, poor scaling to high input cardinality, reduced representation transferability and GPU acceleration.
FM-based models: Click-through rate (CTR) prediction plays a key role in recommendation sys- popular family of methods for CTR prediction, including: matrix factorization (Koren ),008), factorization machine (FM, Rendle 2010). Field-aware Factorization Machines (FFM. Zhang et al. (2018). Field-weighted Factoriza- tion Machine (FwFM, Pan et al.] (2018)) and metrixed Factorization Machines (FmFM, Sun al. (202I)). We will give a more in- in-tepth introduction FM-based methods Sec. 2 As step toward combining deep neural network for this task, Wide & Deep (Cheng et al. (2016) proposed to train a joint network that combines linear model and deep neural network. DeepFM (Guo et al. (2077) suggest to learn a low-order feature interactions through the FM component instead of the linear model. Since then, various embedding- based neural networks have been proposed to improve the performance (Deng et al. (202T); He & Chua (017); Wang et al. (2017); Lian et al. (2018). Unlike He & Chua (2017). our framework is able to lear higher dimensional representations, as well as leveraging non-linear embeddings for the numerical fields. It is also more general, it able to recover range of tabular models by changing the activation.
tems and online advertising. Factorization machine models
Fieldwise models: [i et al. 2020) present model for categorical data, that utilizes linear models with variance and low-rank constraints, and also interpretable in a field-wise manner. Although share the ideas of fieldswise low-rank factorization, our framework allows non-linearity, as well as handles both numerical and categorical fields. Luo et al. (2020) propose NON to take advantage of intra-field information and non-linear interactions. However, our components are much simpler yet effective. Our fieldwise networks rely on low-rank factorization of wider networks and our non- linearity is based on an adaptive activation, while NON utilize only ReLU activation and ensembles multiple heavy aggregation mechanisms, such as attention and deep neural networks.
Deep models for general tabular data: While classical methods are still the industry favorite, some recent work propose to use deep learning for tabular data. For example, TabNet (Arik & Pfister (02II) uses neural networks to mimic decision trees by placing importance on only few features at each layer, using modified attention layers. Yoon et al. (2020) propose VIME, which employs MLPs in a technique for pre-training based on demoising. Transformer models for more general tabular data include Tab Transformer (Huang et al. 20200), which uses transformer encoder to learn contextual embeddings only on categorical features. The main issue with this model is that numerical data do not go through the self-attention block, but are only fed to MLP. SAINT (Somepalli et al. (202))
2

---

Under review as a conference paper at ICLR 2022
address that issue by projecting numerical features to the higher dimensional embedding space and passing them, together with the categorical embeddings, through the transformer blocks. In addition, SAINT propose using attention in the rows level, to explicitly allow data points to attend each other.
Critiques of deep neural networks for tabular data: Recently, several works criticized the pro- posed deep models for tabular data. For example, several works criticize MLP since it is unsuit- able for modeling multiplicative interactions between features (Rendle et al. (2020). In addition, Shwartz-Ziv & Armon (2020) criticize deep tabular data models on claiming outperform XG- Boost, while their study shows that XGBoost outperforms these deep models across the datasets.
Deep learning on unordered sets: Qi et al. (017)) suggest unified architecture for handling point cloud data. From data structure point of view, both point cloud and tabular data are an unordered set of vectors. The basic architecture of their network surprisingly simple as in the initial stages each point is processed identically and independently. In addition, key to this approach is the use of max pooling for aggregating informative points of the point cloud. Our framework architecture shares similar architecture ideas, by the independent processing of each field, followed by aggregation. However, we use a different network for every field while they share the same network to all points (due to the symmetric of point cloud data).
THE THEORETICAL CONNECTION BETWEEN FACTORIZATION MACHINES AND FIELDWISE NEURAL NETWORKS
In this section, will briefly overview factorization machine approaches. We will then demonstrate a theoretical connection between them and shallow neural networks. This will form the basis for our final, generalized framework in Sec.
2.1 BACKGROUND
In this section we describe common approach for classification of tabular data. For ease of explana- tion, we detail the case of binary classification of multi-field categorical data. However, there is no loss of generality, our final approach applies for categorical and numerical as well for multi-class tabular datasets.
((x(") (y)), (2))) training dataset consists of S labelled samples (()))) where and y sample and label respectively. Each sample is specified by C' categorical features fi. and J different fields Fy., Each field may contain multiple features, while each feature belongs only one field. An example of fields are "Country" or "City", whereas features could be "Japan" or "Rome". To simplify the notation, we use index represent the feature f; and F(i) to represent the field which fi belongs to. SF denotes the set of features belonging to field f and JF()) represents their number. We denote c. = 1 if feature is active for this instance, otherwise x!"' = O. We denote the one-hot vector of features per-field oF = concat(2; E SF}. We denote the embedding of feature as e4 E R where K is the (usually small) feature embedding dimension. We denote by Ep E IR x the field embedding matrix whose rows are the embeddings e; of the features Sp belonging to the field f.
Preliminaries:
A
Logistic Regression (LR) is probably the most widely used model for this task. However, linear models lack the capability to represent feature interactions as cross features are often significant (Chapelle et al.] (2014). Degree-2 Polynomials (Poly2) (Chang al.) (2010) model a general way to address this problem. Such models learn a dedicated weight, Wij. for each feature conjunction resulting in field interaction matrix W E CxC
Factorization Machine Models: Estimating W is hard due to its huge dimensionality and miss- models propose to leam the effect of feature conjunctions by
ing values. factorization-macll low-rank factorization of the interaction matrix W. Factorization machine methods approximate the feature interaction strength as the scalar product of their embeddings weighted by a matrix Mr().F) k that depends the fields of the two features: W., e?'M F()) ey- Different factorization machine methods are distinguished by their particular choice of matrix M MF)) We present the choices of M FF)) taken by four representative factorization ma-
3

---

Under review as a conference paper at ICLR 2022
chine methods in Eq. I FM (Kornn 2008 FFM
Zhang al.
2018 FWFM (Pan et al. 2018
FmFM (Sun et al. 2021):
TK PFG) PF()) 1K TF((.FF))
$ PP
S FFFM
Mr(.F()
(1)
FWFM
Entire Matria
PmMF
Where Ix is the identity matrix, PF(1) is per-field sparse binary projection matrix, (F().F() is scalar value learned for every pair of fields (F(i), F()). For FM and FFM, M is not learned, while for FWFM a scalar per pair fields learned, and for FmFM the entire matrix is learned. See Appendix B for detailed formulation of M for each FM-based model.
2.2 FACTORIZATION-BANE MODELS AS A FIELDWISE FACTORIZED NEURAL NETWORK
In this section, we show that factorization machines are a special case of shallow neural networks.
Let us denote the second order interaction term as 02 E, T. ef FF) We demon- strated above that the erux of factorization machine methods is in the representation of the interac- tion matrix M F().FF()- Let M be the matrix that consists the matrices M F()) for all pairs of fields. Since there is order between the fields, for each pair only matrix is learned, meaning Mpiry = Mrjre, pú, while the diagonal block matrices are zeros. See Appendix B for detailed formulation of M. We observe that M is symmetric matrix, and therefore an eigen-decomposittinn can be applied to it: M = AU Here we propose to utilize a reduced-rank factorization of M resulting in field-wise factorization: MF().F() = UT P(a) AUF where UF(() E Rd Note that the full rank dimension of M is equal to the number of fields multiplied by the number of dimensions per-embedding. Using this factorization the feature interaction term 42 can now be written:
22
M(r,Uuyu))
(2)
We rewrite the sums, the sum over the field f and the sum over the per-field indices Sp:
Q =Σ 2,ee)==AUU
Σ zzee)
1ESP
JESp
Note that Fp is one-hot (only single element has non-zero value), the product EFæF Ejesp I₁ej can be efficiently computed using an embedding layer EpoF:
42 Σ EF A E EF
(4)
As the right-hand left-hand vectors are equal, it yields the simple expression:
-2 diag(A) E UFEEF xp)
(5)
Where the above square is elementwise. As intuitive explanation, the second order interactions are modelled by several steps: i) embedding of the one-hot per-field feature using linear layer Ep. ii) projection of the feature embedding iii) summation over the projected embeddings of all fields. iv) computing the scalar product of their square with the diagonal of the matrix A. Note that this can be expressed by shallow neural network that first learns a field representation, then aggregates the representations by summation, passes the result through non-linearity, and then linear layer. Factorization machines are particular instantiation of this neural network, when the activation function is quadratic function. This will be generalized in the next section.
higher dimension using the per-field projection matrix Up-
FIELDWISE FACTORIZED NETWORKS FOR TABULAR CLASSIFICATION
We propose fieldwiseffactrrized neural-networkss general framework for tabular classification.
4

---

Under review as a conference paper at ICLR 2022
3.1 GENERAL FRAMEWORK
In this section, we overview our proposed framework: fielddvise-fuctoiized neural-nelworks (F2NN), while detailing its components in the following sections. We learn a dedicated neural network or for each field that takes as input the values of the field xp (one-hot for categorical fields, scalar for numeric fields) and returns a dineesionnnin embedding (x(xF). The architecture of the fieldwise neural network will be described in Sec. 3.2] The aggregated high-dimensional feature is obtained by summing over the fields Σr & (xr) is then passed through an activation function o. Our activation a allows different activation functions for different dimensions. The post-activation results are finally multiplied by output linear layer W out, mapping to the output logits Zout:
Bout Woutd Σ ΦF(EF))
Note that differently from factorization machine, but similarly to other neural networks, our frame- work is able to handle both binary and multiclass classification tasks.
The framework is characterized by two main components: per-field representation by factorized networks Φr, and an adaptive activation a on the aggregated representations of all fields. We will describe them in detail in the next sections.
3.2 FIELDWISE FACTORIZED NETWORKS
The theoretical basis in Sec. suggests that the learned field embeddings should of high- dimension to factorize full-rank ((),F(() but that these representations should be low-rank factorized. The motivation behind these design choices is to reduce sample complexity. Modeling field interactions requires a high-dimension while there may not be sufficient number of samples per-feature for estimating it. Complexity is reduced: i) by learning a per-field representation that does take other features into account ii) by using low-rank factorization inside these fieldwise networks. Since we are able to use other activation than just than quadratic, d is allowed to be even larger than the full rank dimension. Our framework able to handle both categorical and numeric fields:
we learn fieldwise networks dgat (#f) = Up EpTF. We choose the per-fcature embedding (output of Ep) to be low-dimensional as there are often many features and limited data per-feature, while matrix Ur projects this to high-dimension.
Categorical fields:
Following Sec. 2.2
Numeric fields:
Differently from categorical fields, numeric fields are ordered which enables learn- ing more complex functions. We choose learn a factorized one-hidden layer for each field. Our network first projects the scalar value high-dimension tF 2F + by (where Ut. by E R²). The results are passed through a ReLU network, and mapped to the per-field embedding. We found that a high dimensional f. is important for achieving strong performance. However, as tr and the output fild-embedding &p have a high-dimension, the second linear layer becomes very large. In- stead, we choose to low-rank factorize the second layer. The post-activation Re is projected to a low-dimension, by linear layer EF and is then projected using linear layer UF to the high- dimensional field embedding ór. Note that as both EF and Up are linear, with no intermediate non-linearity they are equivalent to single (low-rank) linear layer. The entire network is therefore a factorized one-hidden layer neural network.
In summary, the fieldwise factorized networks for categorical and numeric variables given by:
(UFÉFTF UeEERRLLUUpp + br) Numeric
Categorical
Note that although SAINT (Somepalli et al. (202T) also use a one-hidden layer network for numeric values, its formulation does not use the low-rank factorization and therefore cannot handle high width, which we show is key to the performance of our method. Also note that our formulation can easily handle deeper fieldwise architectures, but we did not observe benefits from deeper networks.
3.3 OUR FRAMEWORK CAN EXPRESS POPULAR TABULAR CLASSIFICATION METHODS
Despite the simplicity of our framework, it is very general. By changing the fieldwise factorized network dy and activation a, it can express several popular tabular classification methods:
5

---

Under review as a conference paper at ICLR 2022
Factorization machine methods: When choosing quadratic activation ((x) x² and the fieldwise network OF (xp) = E our framework becomes a factorization machine- baasiee-baadd model. This proof was detailed in Sec. 2.2 Therefore, factorization machine models are expressible by our framework.
One-hidden Layer ReLU Networks: When choosing the activation (xx) (eLU(x) and the field- wise networks as simple linear layers 0 = UT, our framework becomes a one-hidden layer neural network. Although more layers can be easily added to our framework after the activation layer o, we did not find this beneficial in practice. We will present empirical results in Sec. 4.2 for showing this.
Wide-and- Deep: When selecting fraction of dimensions of a to have quadratic activations, while the rest are selected have ReLU activation, our framework becomes a wide-and-deep model (as it is the sum of a ReLU network and a factorization machine). Although the original wide-and- deep method used full rank for the feature interaction Wij. later methods (e.g. Guo al. (2017)) use different factorization machine varieties. By modifying the choice of dimensionality of the per- feature embedding e; and per-field projection all the above methods are expressible by our model. One caveat is that the deep part of our framework only has single hidden layer, but mentioned previously, adding further layers has not improved results in our experiments.
3.4 ADAPTIVE NON-LINEARITY
In Sec. 3.3
we established that by choosing different activation function a, our framework can ex- press range of popular tabular classification methods. We will show in Sec. 4.3 that in different classification problems, either (x)) (x) or axx) = x² yields significantly better perfor- mance. By the analysis in Sec. 3.3 these cases where either ReLU neural networks or factoriza- tion machine achieve better results. By selecting a concatenation of ReLU and quadratic functions, doubles
we may be able deal with all cases. However this may not be an efficient solution the number of parameters. It would naturally be attractive to use a non-linearity that may be able to automatically adapt to the setting most beneficial to a particular dataset. Here we suggest using the GELU activation function (Hendrycks & Gimpel 2016) as bridge between the two paradigms. At the limits, (ELU(x) has the following attractive properties:
0.5x V
I x²
| <<>
GELU(x) ~
(8)
(x:
|||||| +
Therefore, by varying the magnitude of the input, and utilizing the GELU activation, framework is able to switch between factorization machine and ReLU network behavior. The adaptive behavior will be demonstrated empirically in Sec. 4.3 GELU can be used as an efficient alternative for exhaustive search over the optimal activation function or ensemble approach (wide-and-deepy
3.5 PRACTICAL CONSIDERATIONS THE DEVIL IS IN THE DETAILS
The field of deep learning has made significant steps forward by increasing the depth of neural networks and by using attention-based architectures such as transformers. might be expected that increasing network depth or using transformers might be beneficial for tabular data. Unfortunately, in our experiments, we did not find evidence to support this. We do not claim that impossible to gain by using the two mechanisms, but this is not trivial. More optimistically, in our experiments we found that there are lower hanging fruit. Specifically, we obtained significant gains by choosing suitable width d, I and activation 0, well as tuning the regularization strength (either dropout or weight decay). We suggest GELU and full rank projections Up (d = J- K) as robust default values. Although tuning the embedding dimension K can also help, kept in-line with the baselines.

---

Under review as a conference paper at ICLR 2022
4
EXPERIMENTS
4.1 PERFORMANCE COMPARISON
4.1.1 GENERAL TABULAR CLASSIFICATION
Evaluation protocol We evaluated our method the same datasets SAINT Somepalli et al. (20TT). The datasets include 14 binary classification tasks and 2 multiclass classification tasks. We did not include Arcene and Arrythmia datasets from the reported results as they are very small datasets and their results have high variance (Arrhythmia contains 452 samples and Arcene con- tains 200 samples). In addition, we use exactly the same pre-processing as the official SAINT implementation. Details of these datasets and the pre- pre-processing are provided in Appendix
Our optimization hyper- parameerrss follow SAINT. We run a grid search on the hyperparameters presented in T seed=1, and selected the best model by the validation set. For fair comparison, we used the same Fy dimensionality as SAINT defaults.
Hyperparameters:
Baselines We compare against the baselines reported in SAINT. They include traditional methods such as logistic regression and random forests, as well as powerful boosting libraries XGBoost (Chen) & Guestrin ) LightGBM (Ke et al. (2017)), and CatBoost (Prokhorenkova et al. (2018). They also compare against deep learning methods, like multi-layer perceptrons, VIME (Yoon et al. (2200), ) TabNet (Arik & Pfister (2021)). and Tab Transfformrr (Huang et al. (2020). The numbers were copied from the SAINT paper. In order to compare the performance on the same exact seeds, re-run the SAINT models on 10 different seeds (1-10). For fair comparison, trained SAINT with our best numerical embedding hidden dimension 1 and their default selection (100), and re- ported the best of the two settings.
The averaged performance for all methods is reported in Table I the results per dataset are detailed in Table We reported ROC-AUC for binary classification, and class prediction accuracy or multi-class classification. Our results are slightly better or comparable to SAINT variations, except from on multiclass tasks, where SAINT-s achieves poor performance. Our method also outperforms all of SAINT's baselines on average. SAINT-s and our framework use exactly the same learning setting (optimization hyper-parameters. datasets). The difference between the architectures is the post-EE architecture. While we use set of linear projections that are only summed and passed through activation and classification layers, SAINT-s uses a massive six layer Transformer. This highlights that for robust and accurate neural models for tabular data, it does not appear that complex methods have an edge over well-tuned simple models such as ours.
Results
4.1.2 CTR PREDICTION
CTR prediction is fundamental tabular binary classification task. FM-based models the most common approach for tackling it. As our framework generalizes FM-based models, we also exam- ined if CTR prediction benefit from framework.
Datasets We used Avazu and Criteo which are common CTR prediction benchmarks.
Baselines We compared the framework implementation against FmFM, which is the shallow state- of-the-art of FM-models, and two deep variations DeepFwFM and DeepFmFM. We implemented framework the FmFM public code-base ? and used their data processing and baselines im- plementation. We split the data as described FmFM paper, and applied the same splits for all the models. We conducted hyper-parameter search for both our method and FmFM, and found the optimal hyperparameters for both were the same. We thus applied the same hyperparameters for the deep baselines. Other implementation details are provided Appendix C
We reported the results of using the papers hyperparameters and using our tuned hyperparameters, as well as quoted the reported results from the paper. We can observe that even after improving the baselines by a better tuning, our framework outperforms all other baselines.
Results The ROC-AUC results are reported in Table I
7

---

Under review as a conference paper at ICLR 2022
Performance comparison. Left: general tabular data classification (Average over all datasets). Above line results taken from SAINT paper evaluated on unknown seeds. Below line mean and standard deviation of SAINT results computed using the authors' code averaged over seeds 1-10, and of our method evaluated on exactly the same data. Right: CTR prediction (ROC- AUC %). results reported by authors, II results of running the model on our data split with paper's hyperparameters, III result of running the model on our data split with our tuning for the model. DeepEwFM results are without a pruning mechanism.
Table 1:
All
Logistic Regression
86.822
Random Forest
89.086
Criteo II
XGBoost
91.038
Model
III 81.166
LightGBM
89.658
FmFM
81.09
81.11
CatBoost
90.197
DeepFmFM
81.175
ML.P
84.106
DeepFwFM
81.16
81.105
81.130
VIME
78.482
Ours
81.245
TabNet
86.515
Tab Transformer
88.654
Avazu
SAINT
91.884
Model
II
III
SAINT-i
91.716
FmFM
77.63
77.343
78.635
SAINT-s
90.669
DeepFmFM
78.647
DeepFwFM
78.93
78.519
78.596
SAINT
91.596 10.
Ours
78.701
SAINT-i
91.286 142
SAINT-s
89.821 +0.230
Ours
91.635 +0.158
4.2
ABLATION STUDIES
In this section we will explore the effect of the framework components on its performance. The experiments in the following section were conducted using SAINT evaluation datasets and process. From resources constraints, we run the experiments on seed=1 only. The results are reported in Table 2 averaged over the datasets, more details are found in Tables 10
Per-field Projections: To examine the importance of the per-field networks, we evaluated sharing the projection matrix Up across fields. We can observe that all datasets (besides HTRU2), gain from the fieldwise projections. In multi-class datasets, the difference much significant.
Low rank projections: In this experiment reduced the dimension of the projection matrix Ur to that of the output of Ep. Overall, we can observe, that the factorized high-dimensional represen- tation crucial, especially in Forest, Volkert and MNIST.
Numeric fieldwise networks: We tested the following alternatives: (i) a K K-dimensional linear layer for each field, (ii) applying non-linearity, without high-dimensional projection, meaning uF dimension is equal to Ep dimension, (ii) projecting to high-ding -igh-dimeniional space and reduce to K- dimensional, without non-linearity. The averaged results over the datasets that contain continuous fields are reported in Table 2 (Right). We can observe that the non-linearity is an important aspect, and boost the performance. addition, by controlling the width for the high dimedesiiiiin space, we gain further improvement over the low-dimensional non-linear network.
We added 2 extra layers after the activation a with equal width field embedding, ((xF). and matching the model activation and dropout. Since quadratic activation performs poorly when deepening the network, we tested all activations for the datasets that selected a quadratic in their shallow variation, and report the best. While we observed improvement on specific datasets, such as Volkert and Forest, in most of the datasets the results were comparable of even worse than shallow model with the linear classification head.
Deeper model:
The activation after aggregating the fieldwise representations In the original selected models, the selected activations are divided follow: 6 datasets with quadratic activation, 4 datasets with ReLU activation, and 4 with GELU activation. In order to explore the effect of the activation, we vary the activation for each dataset, while keeping the other hyperparameters fixed. The averaged
8

---

Under review as a conference paper at ICLR 2022
Table 2: Averaged ablation studies. Left: General framework architecture ablations. Right: Nu-
meric fieldwise networks ablations. Variation
Binary (ROC-AUC)
Multiclass (Aceuracy)
All
Full method
93.16
84.02
91.85
Variation
Num. DS
Shared field matrix
92.55
70.47
89.40
Full method
91.40
Deep
92.89
82.57
91.41
(i) UFEFFF
90.74
Low rank
92.64
78.97
90.69
(ii K
91.17
Quadratic
92.70
82.31
91.21
(iii) UpEp(Upzz bbr
90.80
ReLU
93.05
82.58
91.55
GELU
93.00
82.61
91.51
Table 3:
GELU as adapted activation HR@@ 10 on Movielens dataset
Emb. size
ReLU
GELU
Quad.
MF
16
0.6876
0.6972
0.6974
0.6937
192
0.7164
0.7217
0.7280
0.7278
results are reported in Table 2
In many datasets the particular activation was immaterial. However, in some datasets, activation has made significant difference, it should be carefully chosen.
4.3 GELU AS AN AUTOMATICALLY ADAPTED ACTIVATION
We conducted empirical study on the advantage of using GELU activation, by examining two cases case where quadratic activations are preferred over ReLU, and a case where ReLU ac- tivations are preferred. We demonstrate that by using GELU, can get the closest result to the preferred activation.
Quadratic is better than ReLU: Rendle (201)) showed that simple matrix factorization model performs better than MI.P and NeuMF models (Zhao et al, (202T)) for collaborative filtering. We the authors' dataset and settings. We tested the effect of different choices
applied our method of a on the ability of our framework to express a dot product between fields. Dedicated choices of initialization and regularization were used, details in Appendix C We report the results for embedding sizes K 16 and = 192, the minimal and maximal embedding sizes that were tested Rendle (2010). Note that the reported result for matrix factorization of Rendle (2010) is 0. 0.7294 (on embedding size of 192). As expected, we can observe that for each embedding size, quadratic activation performs the best, while ReLU performs poorly. However, GELU achieves similar results to the quadratic performance, demonstrating its robustness and adaptive behaviour.
ReLU is better than Quadratic: On the other hand, from our experiments on forest dataset, observed that quadratic activation performs dramatically worse than ReLU (9.884% and 99.4%, respectively). However, GELU obtains 99, 99.47% which is in line with ReLU.
DISCUSSION AND CONCLUSIONS
In this paper, we presented theoretically grounded, general framework for handling tabular data classification. Our framework can be extended in many ways. Two promising ideas are custom activations and representation transfer:
Activation for tabular tasks. We highlighted the importance of the activation and demonstrated both theoretically and empirically, that GELU activation is suitable across the full range of tabular tasks. This suggests that developing activation function for tabular classification is very promis- ing.
Exploring the learned representations. As by-product of tabular data classification, framework leamss fieldwise representations. Future work should examine the representations are useful for transfer learning for related tasks.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Sercan Ó. Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In AAAI, 2021.
Yin-Wen Chang, Cho-Jui Hsieh, Kai-Wei Chang, Michael Ringgaard, and Chih-Jen Lin. Training and testing low-degree polynomial data mappings via linear SVM. J. Mach. Learn. Res., 2010.
Olivier Chapelle, Eren Manavoglu, and Rómer Rosales. Simple and scalable response prediction for display advertising. ACM Trans. Intell. Syst. Technol., 2014.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In KDD, 2016.
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. Wide & deep learning for recommender systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, DLRS@ RecSys 2016, Boston, MA, USA, September 15, 2016, 2016.
Wei Deng, Junwei Pan, Tian Zhou, Deguang Kong, Aaron Flores, and Guang Lin. Deeplight: Deep lightweight feature interactions for accelerating CTR predictions in ad serving. In WSDM, 2021.
Jerome H. Friedman. Greedy function approximation: gradient boosting machine. Annals of Statistics, 2000.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: A factorization- machine based neural network for CTR prediction. In IJCAI, 2017.
Xiangnan He and Tat-Seng Chua. Neural factorization machines for sparse predictive analytics. In SIGIR, 2017.
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, 2016.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar S. Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. CoRR, 2020.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-
Yan Liu. Lightgbm: highly efficient gradient boosting decision tree. In NIPS, 2017.
Ychuda Koren. Factorization meets the neighborhood: multifaceted collaborative filtering model. In KDD, 2008
Zhibin Li, Jian Zhang, Yongshun Gong, Yazhou Yao, and Qiang Wu. Field-wise learning for multi- field categorical data. In Hugo Larochelle, Mare Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Ties Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL /RR/ntttps///prrcedding neurips. cc/ http:ipp..pppppppppe.epe cc/paaper/222//aaht
.html
Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In KDD, 2018
Yuanfei Luo, Hao Zhou, Wei- Wei Tu, Yuqiang Chen, Wenyuan Dai, and Qiang Yang. Network on network for tabular data classification in real-world applications. In Jimmy Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (eds.), Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 2317-2326. ACM, 2020. doi: 3401437
10.1145/339727 1..0111.... URL htt.::ttts:/att or wrg..o.ggg...... 110111145333
Junwei Pan, Jian Xu, Alfonso Lobos Ruiz, Wenliang Zhao, Shengjun Pan, Yu Sun, and Quan Lu. weighted factorization machines for thiouhthrough rate prediction in display advertising. In www. 2018.
10

---

Under review as a conference paper at ICLR 2022
Liudmila Ostroumova Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In NeurlPS, 2018.
Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In CVPR, 2017.
Steffen Rendle. Factorization machines. In ICDM, 2010.
Steffen Rendle, Walid Krichene, Li Zhang, and John R. Anderson. Neural collaborative filtering vs. matrix factorization revisited. In RecSys, 2020.
Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. CoRR, 2020.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT: improved neural networks for tabular data via row attention and contrastive pre-training. CoRR, 2021.
Yang Sun, Junwei Pan, Alex Zhang, and Aaron Flores. FM2: felldmatrixed factorization machines recommender systems. In WWW, pp. 2828-2837, 2021.
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In ADKDD, 2017.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela der Schaar. VIME: extending the success of self- and semi-supervised learning to tabular domain. In NeurlPS, 2020.
Zhiyuan Zhang, Yun Liu, and Zhenjiang Zhang. Field-aware matrix factorization for recommender systems. IEEE Access, 2018.
Xinke Zhao, Wei Zeng, and Yixin He. Collaborative filtering via factorized neural networks. Appl. Soft Comput., 2021.
A APPENDIX F2NN FRAMEWORK ILLUSTRATION
An illustration of our framework can be found in Figure 5
B APPENDIX FMMBASED MODELS FORMULATIONS
Embedding form: FM, FwFM and FmFM, learn C' vectors, for each feature, es R
FFM learns (J-1) 1) vectors for each feature, e F()) R overall C- (J-1) vectors. Note that they select much higher than all other approaches, therefore 1 > e, RK places these (J 1) vectors in the relevant indices, while 0gg in its field indices. For example, e₁ = [0g . ef}, jc2 0K O..0,,,.ee .... Then, Pr(j) E IR KxK 1- xK) extracts the embedding e F()) For 0
1x
0 0g
example, P2 is of the form: Pa
0g
Interaction matrix form:
In general, for every FM-based model, M takes the following form:
11

---

Under review as a conference paper at ICLR 2022
4
8g
Rel.
Ral.
Figure 1: Fieldwise factorized neural networks (F2NN) architecture. Toy example of binary clas- sification: Fields 3 and 4 are numerical, while fields 1 and 2 are categorical, with six and four categories, respectively.
0K
M1,2
M1.3
MI.J
M1,2
0K
M2.3
M2.J
1 M
(9)
2
Mi,
M2.5
Ma.3
0k
C APPENDIX EXPERIMENTAL SETTINGS AND IMPLEMENTATION DETAILS
C.I PERFORMANCE EVALUATION
C.I.1 GENERAL TABULAR DATA
Datasets Detailed information on the evaluated datasets is reported in table can be seen that the evaluated datasets diverse, in terms of their size and the amount of fields, and contains both categorical and numerical features. Each of these datasets publicly available from either UCI or AutoML We use the exact processing as is SAINT implementation, i.e. all the continuous features are Z-normalized, and all categorical features are label-encoded before the data is passed on to the embedding layer.
Implemetation details We implemented our framework in the SAINT public code-base Linear and Embedding layers were initialized using the PyTorch default. We used dropout before UF and after the activation, and a bias for Up and for the numerical EF and tr. As in factorization
https://github.com
https///gittuu..coommmmeoeoo wttp://///momaa///aaaaaaa.mm
12

---

Under review as a conference paper at ICLR 2022
Table 4: General tabular datasets
Dataset
Task
#Fields
#Categ.
#Numer.
Size
#Positives
#Negatives
% of positives
Income
Binary
14
8
32,561
7,841
24,720
24.08
Bank
Binary
16
9
45,211
5,289
39,922
11.7
BlastChar
Binary
17
7,043
1,869
5,174
26.54
Credit
Binary
29
0
29
284,807
492
284,315
0.17
Forest
Binary
49
0
49
495,141
283,301
211,840
57.22
HTRU2
Binary
8
O
8
17,898
1,639
16,259
9.16
KDD99
Binary
39
3
36
494,021
97,278
396,743
19.69
Shoppers
Binary
17
2
15
12,330
1,908
10,422
15.47
Philippine
Binary
308
0
308
5,832
2,916
2,916
50
QSAR Bio
Binary
41
0
41
1,055
356
699
33.74
Shrutime
Binary
11
3
8
10,000
2,037
7,963
20.37
Spambase
Binary
57
57
4,601
1,813
2,788
39.4
Volkert
Multiclass(10) 10)
147
0
147
58,310
MNIST
Multiclass(10) (10)
784
784
60,000
Table 5: Saint datasets Architecture hyperparameters search. d factor and fuctor stands for factors of the full rank dimensionality of Ep and tp respectively.
Parameter
Values
d factor factor
{0.1, 0.25, 0.5, 0.75, 1,2,3,4,,
(1,2,3, [ReLU, Quadratic, GELU) {0, 0.,0.1,00 0.25, 0.5, 0.75)
Activation
Dropout
machine methods, observe minor benefits from adding linear per-field terms the final logits. We implemented this by adding to the per-field final layer WF, an identity matrix with the same dimension as the output of EF. The activation sigma for the resulting dimension was chosen be identity (xx) x.
Training Our optimization pappe-pprameters follow SAINT. We used the AdamW optimizer with 3 0.9, 82 = 0.999, decay 0.01, and with a learning rate of 0.0001 with batches of size 256 (except for MNIST, which they use a batch size 32). We trained for 100 epochs. We split the data into 65%, 15%, and 25% for training, validation, and test splits, respectively.
Hyperpararmter search: We performed architecture search on the hyperparameters presented in Table 5 with seed=1. We selected the best model by the validation set. Our experiments employed fixed embedding and projection sizes to all of the fields (although tuning this might improve results).
C.1.2 CTR PREDICTION
Datasets:
The first one is the Criteo CTR data set, it is well-known benchmark dataset which used for the Criteo Display Advertising Challenge. There are 45 million samples and each sample has 13 numerical fields and 26 categorical fields. The second data set is the Avazu CTR data set, it was used the Avazu CTR prediction competition, which predicts whether a mobile ad will be clicked. There are 40 million samples and each sample has 23 categorical fields.
Implementation details We implemented our framework on FmFM public code-base and used their data processing and baselines implementation. We split the data described in FmFM pa- per, and applied the same splits for all the models. For our implementation, we adopted FmFFM model implementation, and changed the main component tion between them, and bias for the per-field projection layer. For the weights initialization, FmFMM implementation uses weights that are sampled from normal distribution. For the reported frame-
Tensorflow linear layers, activa-
https:://itthnn htwp:hwbw..coo/oooo///////
13

---

Under review as a conference paper at ICLR 2022
Table 6: CTR prediction baselines hyperparameters: learning rate(?), regularization strength (A), standard deviation of the weights normal initialization (σ), batch size (bs). I results reported by authors, II results of running the model on our data split with paper's hyperparameters, III result of running the model on our data split with our tuning for the model.
Criteo II
Model FmFM
7 = le x= le
- le X = le-s;
Y= - 1e,s,
0.01, bs 1024
a 0.01, bs 1024
== 0.2, bs 2000
DeepFmFM
Y = le= À le -5
0.2, bs 2000
DeepEwFM
7 = le 30-3, A 3e-7
7 = le-1, le
7 le-1, A 1e-5
0.01, bs 2048
= 0.01, bs 2048
0.2, bs 2000
Ours
7 le 4 A lE le-5
0 0.2, bs = 2000
Avazu
Model FmFM
III y= lc 1e- 20-0
le A = 1e-6
= = X = le-5
a 0.01, bs 1024
α 0.01, bs 1024
0.2, bs 5000
DeepFmIM
7 le-3, 26-6
8 = 0.2, bs = 5000
DeepFwFM
7 Le -3 3, x= Ge
= - 1e-5,
== le A 2c
0 = 0.01, bs = 2048
a = 0.01, bs = 2048
o = 0.2, bs = 5000
Ours
== le 1e- 2
0.2, bs 5000
work instance performance, in Criteo used ReLU activation and full rank per-field projection layer, and for Avazu we used GELU activation and factor of the full rank dimension. In this settings, we did use dropout between the layers. Note that since the numerical fields in Criteo datasets are discrete, we used the same numerical data processing as in the baselines and consider them as categorical.
Hy plperparaaeters search We found that the hyperparameters in this task are crucial, included the standard deviation of the weights initialization. Therefore, we performed an hyperparameters search both for our framework and for FmFM, though obtained that they should use the same hyperparam- eters. Thus, we applied this hyperparameters for the deep baselines too. For the hyperparameters search, we implemented our framework by GELU activation and full rank per-field projection layer. The exact hyperparameters each baseline reported in Table 6
C.1.3 ACTIVATION STUDY
Rendle (2010) public code is optimized for simplicity and not for efficiency, therefore we have implemented an efficient keras version of it, using Dot layer, and adjusted the hyperparamters fit their reported results. We used Adam optimizer, batch size of 10k and L2 regularization strength of 2.5e -7, while kept the negative sampling on and learning rate of 0.002.
Next, since matrix factorization is a special case of our framework, we implemented it by our frame-
Ox TK for the per-field OK
work, using quadratic activation and the eigen vectors and values of 0.5.
IK
projections and for the final classification layer, respectively. The eigenvectors were partitioned with respect to the fields. When freezing these weights, we get an exact implementation of dot product, which also reflected by replication of the performance between both of the implementations.
14

---

Under review as a conference paper at ICLR 2022
Table 7: GELU adapted activation regularization strength of the reported results
MF initialization & regularization
Emb. size
ReLU
GELU
Quad.
16
A le-5
X le-b
À ==
192
A = 5e 3
A = 5e
A =11
In order to explore the ability of other activation to approximate the results of dot product based model, we used the eigen decomposition as an initialization and regularization, and searched the best regularization strength for each activation. The regularization strength of the reported results are reported in Table
We reported this experiments results on embedding sizes of and 192, the minimal and maximal embedding sizes that were tested in Rendle (2010).
D APPENDIX RESULTS PER DATASET
Performance comparison:
The The performance comparison per dataset are reported in Table 9
results are averaged over seeds 1-10. We reported also the standard deviation.
Ablation studies:
Tables 10
report the results of the ablation studies per dataset
E APPENDIX INFERENCE TIME COMPARISON
We also examined the runtime between our framework and SAINT. We computed the average infer- ence time over all batches of size 256. The results on three datasets are reported in Table 8 We can observe that using a transformer slows inference without an improvement to the performance.
Table Inference time comparison (milliseconds per batch on average). F2NN is faster than SAINT variations.
Income
Forest
Bank
SAINT-s
6.890
7.621
5.764
SAINT-i
1.502
3.327
1.348
SAINT
2.449
4.480
2.107
F2NN
0.660
0.933
0.575
15

---

Under review as a conference paper at ICLR 2022
MNIST
97.7
92.66
78.5
93.24
97.7
95. 15.22
94.88
Accuracy
Accuracy Volkert
Volkert
70.35
65.27
62.44
71.9
66.93
69.93
70.35
70.35
70.78
70.55
70.26
Table 11: Numerical networks ablastions per dataset. (i) stands for Up EFæF, (ii) represents = K, and (iii) represents Er UFFF + 1bb)))
Income
91.62
91.67
91.5
91.77
91.62
91.74
91.69
Income
91.62
91.19
91.4
91.27
97.86 +0.1035
97.71 #0.1774
94.16 E0.3232
97.53 10.1266
KDD19
100 10
100 =0
100 +0
100 +:
MNIST
Shoppers
93.53
93.2
92.96
93.54
93,4
93.53
93.49
Shoppers
93.53
92.76
93.12
93.18
Accuracy
Philippine
81.22 +1
80.56 1:448
78.46 #1.81
11.88
69.57 +0.4544
70.34 +0,2749
49.71 13.156
10.3784
Forest
99.47
96.35
99.42
99.58
94.84
99.4
99.47
Forest
99. 19.4
99.13
99. 39
Table 9: Performance comparison per dataset, Averaged results over seeds 1-10.
99.1
Volkert
80.81
Blastchar
70.22
82.38
82.41
82.3
82.3
82.38
82.25
82.26
Blastchar
82.35 $2.38
81.89
82.48
81.96
Spambase
98.37 10.44
98.2 +0.34
98. 08.28 +0021
98.31 +0.39
Income
91.44 +0.36
91.4 +0.38
91.56 +0.3
91.54 10.36
Bank
93 3.009
92.88
92.09
93,11
92.95
93.06
93.09
Bank
93.00 00
92.86
93.02
92.72
Table 10: Ablations per dataset
ROC-AUC Shrutime
86.37 11.07
+1.23
86.18 +1.11
86.47 1115
KDD99
100
100
100
100
100
100
100
KDD99
100
100
100
100
93.2 +0.47
92.89 #0.41
93.06 +0.51
92.97 10.48
86.23
Shoppers
ROC-AUC AUC Philippine
ROC-AUC
82.47
81.41
81.02
81.47
82.47
81.76
81.73
Philippine
QSAR Bio
93.28 11.4
92.9 92.9 +1.79
92.64 +1.41
92.89 11.88
22.47
79.55
81.61
79.44
ROC-AUC
+0.24
94.74 +0.97
99.71 +0.02
10.02
Forest
Spambase
98.09
98.02 02
98.24
98.11 11
98. 98.44 44
Spambase
98.2
99.49
98.44
98.4
98. 98.44
97.78
97.96
98.01
+0.42
+0.35
+0.44
10.39
HTRU2
97.98
98.02
97.99
98.13
Blastchar
84.03 +0.93
83.98 +0.9
83.99 10.98
10.94
Shrutime
86.6
86.69
85.51
85.39
86.6
85.99
85.93
Shrutime
86.6
85.25
86.35
85.49
84.09
97.89 +0.85 .5
97.92 +0.85
98.09 +0.72
11.31
QSAR Bio
Credit
QSAR Bio
92.93 =0.4
93.12 =0.4
93.67 +0.25
93.43 10.27
94.12
93.49
93.54
93.81
94.12
93.77
97.01
93.44
94.12
92.11
92. 89
94.58
Bank
SAINT-I
SAINT'S
HTRU2
SAINT
F2NN
SAINT-i
SAINT-S
98 98.42 .42
98.57
98 98.42 42
98 98.25
98.47
98.42 98.42
98.42
HTRU2
98.42
98. 48
98.55
98.3
SAINT
F2NN
Credit
97.73
96.95
95.93
Deep 97.48
97.73
97.89
97.71
Credit
97.73
97.84
97.88
96.04
Low-rank
Quadratic
F2NN
Shared
ReLU GELU
F2NN
(i)
(ii)
(iii)
16


Reviews:
[{"review_id": "kN2Whjy7AEw", "reviewer": "Reviewer_EGGP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper, in general, is well written, however the structure (in particular the RW and problem setup) can be improved.  The contributions, especially on studying and investigating the variants of F2NN (wrt architecture and activation functions) are interesting and seem solid. \n\nstrengths: \n- an elegant method \n- well documented ablation study\n- well written paper\n\nweaknesses:\n- weak experimental setup\n- weak (and potentially misleading) discussion of the findings\n\n--\nThe major concerns I see, relate to the experimental setup wrt the comparisons to other methods and the reporting of the results. Namely, the authors report comparisons (on the classification problems) to several benchmark methods, typically applied to these tasks, but in reality they use results from a previous paper. It seems that none of the other methods have been tuned nor their hyper-parameters reported. Many of these benchmark methods are sensitive to their parameters, and their performance may improve significantly when properly used. On the other hand, F2NN, and one of the methods (SAINT) seem to be properly evaluated. This may lead to over-optimistic results, and misleading conclusions. \n\nAnother issue is how these results have been reported. The authors report mean performance, averaged over all 16 tasks (without reporting even std). This can also be misleading, since these are tasks with different properties, and the performance of different methods may substantially vary between them. IMO, analyzing and reporting them separately, not only will be more correct, but it also may provide other insights related to the performance of F2NN (which is given in Appendix C, but not discussed). Moreover, while I enjoyed the ablation study, the GELU adaptivity experiments seem a bit incomplete. The authors simulate the experimental setting reported by Rendle (2010), which is ok to begin with, but IMO should have investigated this further wrt other embedding-sizes and architecture designs (since one of the contributions is the GELU adaptivity).\n\nThere are several 'deep' FM/FFM approaches highlighted in the related work. On the other hand, the authors evaluate on some such methods (implemented within a public repo) but never discuss nor connect these two. Are some of the methods discussed (eg DeepFM) used in the experiments (eg. DeepFmFM)? If so, which ones and can you provide more details on them? Also for instance the one of Guo et al (2017) seems to be (partially) approximated by F2NN, but not really evaluated. How does F2NN compare to it? \n\nThe related work can also be improved in order to better serve the motivation for the work. Namely, personally, I don't see how the 'critiques of DL' paragraph fits, since this work doesn't seem to really support/nor dispute the stated claims. On a similar note, IMO the relation to Qi et al (2017) in the last paragraph, as it reads now - seems a bit of a stretch. \n\nConsidering, the 'devil is in the details' section, there is a statement regarding the benefits of the network depth for transformers. It is unclear, whether the referred experiments (that dispute this) concern F2NN or the transformer-based models, since it seems SAINT was tested with the same architecture, and TabTransformer not evaluated.", "summary_of_the_paper": "The paper proposes a neural network architecture, inspired from factorization machines, for classification task from tabular data. In particular, the proposed factorized neural networks (F2NN) leverage the capabilities of shallow networks for learning factorized low-level representations that can approximate (the typically) high-dimensional representations of each field. This,in turn, together with different choices of activation factions, provides a generalization over different factorization approaches, and can lead to good (albeit not always better) performance on different classification tasks. Results from several (16) experiments on binary and multi-class classification tasks and two CTR tasks, comparing F2NN to different benchmark algorithms, show that it has comparable performance to some of standard (SOTA) methods. ", "summary_of_the_review": "The paper, in general, is well written, however the structure (in particular the RW and problem setup) can be improved.  The contributions, especially on studying and investigating the variants of F2NN (wrt architecture and activation functions) are interesting and seem solid. However, I have some concerns regarding the experimental set-up as well as the discussion of related work, which are reflected in my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_dXrSwG-7OX", "reviewer": "Reviewer_J2FM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors present an interesting framework useful for tabular classification task. \nThey have compared their method with other classifiers in a fair way, with the experiments that are well designed and well described with the right amount of details useful for the reproducibility (I hope the authors share their code). \nMoreover, the ablation studies are very interesting and seem to confirm the importance of the per-field network.\n\nThe results of the proposed approach are quite good even though it is difficult to be sure on which model is better because the results are really near each other and there isn't a difference statistically significant. \nNonetheless, the experiments have shown some interesting insights and the importance of some design choices.\n\nMy main concern is about the formalism they have used in the description of the method. I think that a revision of the math and formalism could make the work more readable and reproducible. \nIn the following some details.\n\nThe formalism introduced in Section 2 is not completely coherent and in some parts misleading. \nThere is not a different text formatting used for the scalar values and vectors (for instance bold for vectors). This good practice is instead followed in some parts of Section 3.\nJust for example (but all the section 2 and 3 should be revised carefully in order to make the paper more rigorous): \n- $x_f$ is a vector and $x_i$ is a scalar. \n- $f$ is used to indicate the categorical features but also to indicates the generic field (in definition of $S_f$ for example). Probably it is better to use F to indicate the generic field.\n\nOther comments:\n- In §2.2 insert a reference to the Appendix where $M$ is appropriately described.\n- In Equation 2 the authors introduce also $x_i$ and $x_j$ even though the previous defined factorization don't mention them. The authors should describe the motivation and the validity of this operation.\n \nSome minor typos:\n- Page 7: XGBoost Chen & Guestrin (2016) --> XGBoost (Chen & Guestrin (2016))\n- Page 8: immaterial --> do you mean not significant?\n\nSome doubts:\n- $E_f$ is the field embedding matrix, and the embeddings are the rows (as described in §2.1). Before the Equation 4 authors stated that: $E_fx_f$ is the embedding of the one hot vector $x_f$ but, if $x_f$ is column vector, it should be $E_f^T x_f$. If I'm right the authors should carefully revised all equations accordingly.\n- Equation 1 presents four Factorizazion Machine Models. If I understood well, only FmFM has been tested (probably is the best model among others). If it is correct the author should be clarify this point. Moreover, in Table 1 it is used the name FmFm but in other parts it is indicated as FmFM (also §4.1.2). \n\nMissed references:\n- [1] Yuanfei Luo et al. Network On Network for Tabular Data Classification in Real-world Applications 2020\n- [2] Zhibin Li et al., Field-wise Learning for Multi-field Categorical Data, 2020\n\nThe [1] and [2] present two approaches to tackle the same problem. Moreover [2] seems to use a similar approach using a field-wise network. Could the authors describe the differences? Moreover, the models in [1] and [2] are not tested, why didn't they tested also these two methods? Do the other tested methods outperform them?  ", "summary_of_the_paper": "The paper describes an approach for tabular data classification.\nIn particular, authors proposed the Fieldwise Factorized neural network to tackle the problem defining a general framework in which it is possible to express other tabular classification methods.\nThe proposed approach permits to manage categorical and numerical fields and seems to outperform other methods (on average). Moreover the experiments seem to demonstrate the importance of per-field network and the improvement in the results using GELU instead of the ReLU of Quadratic activations.", "summary_of_the_review": "There are some interesting aspects and the insights are supported by a good experimental set. In my opinion, with a careful revision of the formalism in the description of the method, the paper can be accepted for ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LxUUqmgWPrV", "reviewer": "Reviewer_c4rf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This theoretical connection between FM-based models and field wise neural networks is impressive. The generalization next to it is convincing and solid.\n\nThe line \"M and is made symmetric\" should be typo. The equivalence of upper triangular M and symmetric M should be stated explicitly.   \n\nThe experiment shows quite marginal performance improvement. Do the authors have comments? This paper may pose new directions to improve current popular models given the theoretical connection. Any preliminary thoughts? the conclusion section doesn't include next steps. Increasing lowered rank in factorization can be explored. \n\nIn Appendix A, it said \"FM, FwFM and FmFM, learn C vectors for each feature\", should be \"FM, FwFM and FmFM, learn C vectors, one for each feature\"?", "summary_of_the_paper": "This paper presents a general neural network framework for tabular data classification, that can recover a range of popular classification methods. The main components are per-field representation by factorized network and an activation function on aggregated representations of all fields. Experiments on general tabular classification and CTR prediction demonstrate its effectiveness, although performance gain is marginal.   ", "summary_of_the_review": "The paper provides new insights on a unified neural network framework that can cover a range of popular tabular classification methods. The derivation is dense but solid. Although performance gain is marginal from experiments, the framework itself deserves more attention. It would be even better to pose new directions to improve models further.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4YdU8y36op2", "reviewer": "Reviewer_jCBE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Experimental results\n- Based on Table 1 and 8, the overall performances are very similar with SAINT.\n- Definitely, the performances are not \"statistically significant\" better than SAINT.\n- What are the main advantages of the proposed method in comparison to SAINT? It seems like in terms of the performances, the advantages seem very marginal and we cannot claim that the proposed method is better than SAINT.\n- Also, based on Table 1, it seems like the differences from XGBoost and GBM are also marginal. In that case, are there any advantages to using deep learning models instead of those traditional models?\n\n2. Standard deviations\n- It is always good to show the standard deviations of the performances.\n- We can see it in Table 1 (left) but not in Table 1 (right).\n- Based on the marginal performance improvements, it would be more important to demonstrate this standard deviation.\n\n3. Figures.\n- It would be good if the authors provide a figure that overview the proposed networks.\n- As the authors mentioned in the discussions, it would be more informative if the authors show some tSNE analyzes about the learned representations.\n\n4. Intuitions\n- The authors showed various results and ablation studies to discover the rules of thumbs for different cases.\n- However, the intuitions for those conclusions are not enough. For instance, why is wide more effective than deep?\n\n5. Extensions of the model\n- It seems like the outputs do not need to be a categorical variable.\n- In that point of view, can we extend this method to the regression problem?", "summary_of_the_paper": "- The authors proposed a novel deep learning model for tabular data classification.\n- The authors utilize the field-wise factorized networks to extract the meaningful information from both categorical and numerical features of the tabular data. Then, using those embeddings to solve the classification problem.\n- The experimental results showed that the proposed method achieved comparable performances in comparison to SOTA deep learning models for tabular data (SAINT).", "summary_of_the_review": "Strength:\n- The authors proposed a novel and well-grounded deep learning model for tabular data classification problem.\n- The authors provided extensive experimental results to understand the performances of the proposed method.\n\nWeakness:\n- The experimental results are not that promising. The performances are similar (not noticeably better) than SOTA. If the authors can claim the unique advantages of the proposed method in comparison to SOTA, I think I will increase my scores.\n- The intuitions of the proposed method are limited. \n\n-----------------------------------------\n\nThank you for the detailed answers from the authors.\nI carefully read the authors' responses and other reviewers' reviews.\nOverall, most reviewers are concerned on the weakness of the experimental results. This is also my main concern about this paper.\nAlthough the authors claimed that the proposed method is \"faster\" than another baseline, it does not mean that this method is \"faster\" than other alternatives (we can easily find other traditional methods that are faster than the proposed method with similar performance).\nFor XGBoost and GBM, the authors said that those are not suitable for CTR prediction due to the high cardinality, However, the authors did not show how they are working in those kinds of datasets in the classification problem. If that claim is valid, it would be much better to show some datasets with that characteristic and show the failure of XGBoost and GBM.\nBased on these, I am going to stay on my original score (5) because the performance concerns are not well resolved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a deep learning technique aimed for tabular data via a unified view of factorization machines and other DNN approaches. The reviews are overall positive when discussing the provided technique, the motivation behind it and the writing. However, there are major concerns related to the experiments. The most dominant one is that of significance, meaning the advantage of the provided method when compared to existing literature. Other claims such as unclear details or different methods of reporting might be possible to resolve via minor edits, but this concern was not resolved in the rebuttal period. Before the paper can be published in a venue such as ICLR, it should provide a clearer comparison against previous works showing exactly where it improves upon them. At its current state, it doesn’t seem to be ready.
Decision: Reject