Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a general neural network framework for tabular data classification, that can recover a range of popular classification methods. The main components are per-field representation by factorized network and an activation function on aggregated representations of all fields. Experiments on general tabular classification and CTR prediction demonstrate its effectiveness, although performance gain is marginal.   
main_review: This theoretical connection between FM-based models and field wise neural networks is impressive. The generalization next to it is convincing and solid.

The line "M and is made symmetric" should be typo. The equivalence of upper triangular M and symmetric M should be stated explicitly.   

The experiment shows quite marginal performance improvement. Do the authors have comments? This paper may pose new directions to improve current popular models given the theoretical connection. Any preliminary thoughts? the conclusion section doesn't include next steps. Increasing lowered rank in factorization can be explored. 

In Appendix A, it said "FM, FwFM and FmFM, learn C vectors for each feature", should be "FM, FwFM and FmFM, learn C vectors, one for each feature"?
summary_of_the_review: The paper provides new insights on a unified neural network framework that can cover a range of popular tabular classification methods. The derivation is dense but solid. Although performance gain is marginal from experiments, the framework itself deserves more attention. It would be even better to pose new directions to improve models further.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a neural network architecture, inspired from factorization machines, for classification task from tabular data. In particular, the proposed factorized neural networks (F2NN) leverage the capabilities of shallow networks for learning factorized low-level representations that can approximate (the typically) high-dimensional representations of each field. This,in turn, together with different choices of activation factions, provides a generalization over different factorization approaches, and can lead to good (albeit not always better) performance on different classification tasks. Results from several (16) experiments on binary and multi-class classification tasks and two CTR tasks, comparing F2NN to different benchmark algorithms, show that it has comparable performance to some of standard (SOTA) methods. 
main_review: The paper, in general, is well written, however the structure (in particular the RW and problem setup) can be improved.  The contributions, especially on studying and investigating the variants of F2NN (wrt architecture and activation functions) are interesting and seem solid. 

strengths: 
- an elegant method 
- well documented ablation study
- well written paper

weaknesses:
- weak experimental setup
- weak (and potentially misleading) discussion of the findings

--
The major concerns I see, relate to the experimental setup wrt the comparisons to other methods and the reporting of the results. Namely, the authors report comparisons (on the classification problems) to several benchmark methods, typically applied to these tasks, but in reality they use results from a previous paper. It seems that none of the other methods have been tuned nor their hyper-parameters reported. Many of these benchmark methods are sensitive to their parameters, and their performance may improve significantly when properly used. On the other hand, F2NN, and one of the methods (SAINT) seem to be properly evaluated. This may lead to over-optimistic results, and misleading conclusions. 

Another issue is how these results have been reported. The authors report mean performance, averaged over all 16 tasks (without reporting even std). This can also be misleading, since these are tasks with different properties, and the performance of different methods may substantially vary between them. IMO, analyzing and reporting them separately, not only will be more correct, but it also may provide other insights related to the performance of F2NN (which is given in Appendix C, but not discussed). Moreover, while I enjoyed the ablation study, the GELU adaptivity experiments seem a bit incomplete. The authors simulate the experimental setting reported by Rendle (2010), which is ok to begin with, but IMO should have investigated this further wrt other embedding-sizes and architecture designs (since one of the contributions is the GELU adaptivity).

There are several 'deep' FM/FFM approaches highlighted in the related work. On the other hand, the authors evaluate on some such methods (implemented within a public repo) but never discuss nor connect these two. Are some of the methods discussed (eg DeepFM) used in the experiments (eg. DeepFmFM)? If so, which ones and can you provide more details on them? Also for instance the one of Guo et al (2017) seems to be (partially) approximated by F2NN, but not really evaluated. How does F2NN compare to it? 

The related work can also be improved in order to better serve the motivation for the work. Namely, personally, I don't see how the 'critiques of DL' paragraph fits, since this work doesn't seem to really support/nor dispute the stated claims. On a similar note, IMO the relation to Qi et al (2017) in the last paragraph, as it reads now - seems a bit of a stretch. 

Considering, the 'devil is in the details' section, there is a statement regarding the benefits of the network depth for transformers. It is unclear, whether the referred experiments (that dispute this) concern F2NN or the transformer-based models, since it seems SAINT was tested with the same architecture, and TabTransformer not evaluated.
summary_of_the_review: The paper, in general, is well written, however the structure (in particular the RW and problem setup) can be improved.  The contributions, especially on studying and investigating the variants of F2NN (wrt architecture and activation functions) are interesting and seem solid. However, I have some concerns regarding the experimental set-up as well as the discussion of related work, which are reflected in my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: - The authors proposed a novel deep learning model for tabular data classification.
- The authors utilize the field-wise factorized networks to extract the meaningful information from both categorical and numerical features of the tabular data. Then, using those embeddings to solve the classification problem.
- The experimental results showed that the proposed method achieved comparable performances in comparison to SOTA deep learning models for tabular data (SAINT).
main_review: 1. Experimental results
- Based on Table 1 and 8, the overall performances are very similar with SAINT.
- Definitely, the performances are not "statistically significant" better than SAINT.
- What are the main advantages of the proposed method in comparison to SAINT? It seems like in terms of the performances, the advantages seem very marginal and we cannot claim that the proposed method is better than SAINT.
- Also, based on Table 1, it seems like the differences from XGBoost and GBM are also marginal. In that case, are there any advantages to using deep learning models instead of those traditional models?

2. Standard deviations
- It is always good to show the standard deviations of the performances.
- We can see it in Table 1 (left) but not in Table 1 (right).
- Based on the marginal performance improvements, it would be more important to demonstrate this standard deviation.

3. Figures.
- It would be good if the authors provide a figure that overview the proposed networks.
- As the authors mentioned in the discussions, it would be more informative if the authors show some tSNE analyzes about the learned representations.

4. Intuitions
- The authors showed various results and ablation studies to discover the rules of thumbs for different cases.
- However, the intuitions for those conclusions are not enough. For instance, why is wide more effective than deep?

5. Extensions of the model
- It seems like the outputs do not need to be a categorical variable.
- In that point of view, can we extend this method to the regression problem?
summary_of_the_review: Strength:
- The authors proposed a novel and well-grounded deep learning model for tabular data classification problem.
- The authors provided extensive experimental results to understand the performances of the proposed method.

Weakness:
- The experimental results are not that promising. The performances are similar (not noticeably better) than SOTA. If the authors can claim the unique advantages of the proposed method in comparison to SOTA, I think I will increase my scores.
- The intuitions of the proposed method are limited. 

-----------------------------------------

Thank you for the detailed answers from the authors.
I carefully read the authors' responses and other reviewers' reviews.
Overall, most reviewers are concerned on the weakness of the experimental results. This is also my main concern about this paper.
Although the authors claimed that the proposed method is "faster" than another baseline, it does not mean that this method is "faster" than other alternatives (we can easily find other traditional methods that are faster than the proposed method with similar performance).
For XGBoost and GBM, the authors said that those are not suitable for CTR prediction due to the high cardinality, However, the authors did not show how they are working in those kinds of datasets in the classification problem. If that claim is valid, it would be much better to show some datasets with that characteristic and show the failure of XGBoost and GBM.
Based on these, I am going to stay on my original score (5) because the performance concerns are not well resolved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes an approach for tabular data classification.
In particular, authors proposed the Fieldwise Factorized neural network to tackle the problem defining a general framework in which it is possible to express other tabular classification methods.
The proposed approach permits to manage categorical and numerical fields and seems to outperform other methods (on average). Moreover the experiments seem to demonstrate the importance of per-field network and the improvement in the results using GELU instead of the ReLU of Quadratic activations.
main_review: The authors present an interesting framework useful for tabular classification task. 
They have compared their method with other classifiers in a fair way, with the experiments that are well designed and well described with the right amount of details useful for the reproducibility (I hope the authors share their code). 
Moreover, the ablation studies are very interesting and seem to confirm the importance of the per-field network.

The results of the proposed approach are quite good even though it is difficult to be sure on which model is better because the results are really near each other and there isn't a difference statistically significant. 
Nonetheless, the experiments have shown some interesting insights and the importance of some design choices.

My main concern is about the formalism they have used in the description of the method. I think that a revision of the math and formalism could make the work more readable and reproducible. 
In the following some details.

The formalism introduced in Section 2 is not completely coherent and in some parts misleading. 
There is not a different text formatting used for the scalar values and vectors (for instance bold for vectors). This good practice is instead followed in some parts of Section 3.
Just for example (but all the section 2 and 3 should be revised carefully in order to make the paper more rigorous): 
- $x_f$ is a vector and $x_i$ is a scalar. 
- $f$ is used to indicate the categorical features but also to indicates the generic field (in definition of $S_f$ for example). Probably it is better to use F to indicate the generic field.

Other comments:
- In §2.2 insert a reference to the Appendix where $M$ is appropriately described.
- In Equation 2 the authors introduce also $x_i$ and $x_j$ even though the previous defined factorization don't mention them. The authors should describe the motivation and the validity of this operation.
 
Some minor typos:
- Page 7: XGBoost Chen & Guestrin (2016) --> XGBoost (Chen & Guestrin (2016))
- Page 8: immaterial --> do you mean not significant?

Some doubts:
- $E_f$ is the field embedding matrix, and the embeddings are the rows (as described in §2.1). Before the Equation 4 authors stated that: $E_fx_f$ is the embedding of the one hot vector $x_f$ but, if $x_f$ is column vector, it should be $E_f^T x_f$. If I'm right the authors should carefully revised all equations accordingly.
- Equation 1 presents four Factorizazion Machine Models. If I understood well, only FmFM has been tested (probably is the best model among others). If it is correct the author should be clarify this point. Moreover, in Table 1 it is used the name FmFm but in other parts it is indicated as FmFM (also §4.1.2). 

Missed references:
- [1] Yuanfei Luo et al. Network On Network for Tabular Data Classification in Real-world Applications 2020
- [2] Zhibin Li et al., Field-wise Learning for Multi-field Categorical Data, 2020

The [1] and [2] present two approaches to tackle the same problem. Moreover [2] seems to use a similar approach using a field-wise network. Could the authors describe the differences? Moreover, the models in [1] and [2] are not tested, why didn't they tested also these two methods? Do the other tested methods outperform them?  
summary_of_the_review: There are some interesting aspects and the insights are supported by a good experimental set. In my opinion, with a careful revision of the formalism in the description of the method, the paper can be accepted for ICLR.
review_title: null

Meta-review: The paper provides a deep learning technique aimed for tabular data via a unified view of factorization machines and other DNN approaches. The reviews are overall positive when discussing the provided technique, the motivation behind it and the writing. However, there are major concerns related to the experiments. The most dominant one is that of significance, meaning the advantage of the provided method when compared to existing literature. Other claims such as unclear details or different methods of reporting might be possible to resolve via minor edits, but this concern was not resolved in the rebuttal period. Before the paper can be published in a venue such as ICLR, it should provide a clearer comparison against previous works showing exactly where it improves upon them. At its current state, it doesn’t seem to be ready.
Decision: Reject