Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes several approaches for numerical feature embeddings for tabular deep learning models and experimentally investigates the gains that such approaches provide. Methodologically, simple linear embeddings, piece-wise linear embeddings, and periodic-activation-based embeddings are proposed and experimentally evaluated on a suite of many benchmark tabular datasets. The experimental results showcase an improved performance competitive with gradient boosted decision trees (GBDT) on GBDT-biased datasets.
strengths_and_weaknesses: Strengths:
* Possible performance improvements from using embeddings for numerical features are showcased
* The experiments are conducted on a large suite of datasets
* Good average rank improvements over GBDT on a GBDT-friendly benchmark are achieved by using numerical embeddings
* The paper is very well-written and easy-to-follow
* The code is in good shape

Weaknesses:
* These embeddings for numerical features are essentially feature engineering. Comparison to GBDT with feature engineering would dramatically improve the paper and the experimental results.
* Methodological contributions of piece-wise linear embeddings are similar to feature binning [1] as the paper mentions in the introduction.
* I believe automated feature engineering methods such as e.g. [2,3,4] should be mentioned in related work.
* While the proposed numerical feature embedding approaches are compared to each other, it is unclear if they provide any benefit over e.g. SAINT [5] with their dense layer or small MLP embeddings.


[1] Dougherty, J., Kohavi, R. and Sahami, M., 1995. Supervised and unsupervised discretization of continuous features. In Machine learning proceedings 1995 (pp. 194-202). Morgan Kaufmann.

[2] Khurana, U., Samulowitz, H. and Turaga, D., 2018, April. Feature engineering for predictive modeling using reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1).

[3] Kaul, A., Maheshwary, S. and Pudi, V., 2017, November. Autolearn—Automated feature generation and selection. In 2017 IEEE International Conference on data mining (ICDM) (pp. 217-226). IEEE.

[4] Khurana, U., Turaga, D., Samulowitz, H. and Parthasrathy, S., 2016, December. Cognito: Automated feature engineering for supervised learning. In 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW) (pp. 1304-1307). IEEE.

[5] Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C.B. and Goldstein, T., 2021. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342.
questions: * Line 286: "However, after coupling DL models with embeddings for numerical features, the situation changes and the MI dataset remains the only one where all DL models lag behind GBDT." It is true that all deep models lag behind only on a single dataset. However, this seems like a one-to-many comparison. Do you have recommendations for a single best model or a good rule of thumb? What should a practitioner use? 

* Table 6: Simple MLP achieves the best performance (with PLR embeddings). In general, from Table 6 we see that MLP>ResNet>Transformer when numerical embeddings are used. Do you have any explanation/intuition about why?


limitations: The authors stated that the work focuses on a generic aspect of deep learning models and that therefore the negative societal impact discussion does not apply. The limitations have been adequately addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new encoding scheme for embedding numerical features in neural network models. The authors report that for tabular data in many cases neural methods have failed to improve on Gradient Boosted Decision Trees (GBDT), thus necessitating more research into how to represent the tabular data, in particular features that are numerical as opposed to categorical. The authors propose a variation of a binning/bucketing scheme which they call Piecewise Linear Encoding (PLE), which is more informative than a one-hot encoding scheme, where a numerical feature is simply represented by the embedding corresponding to the bin/bucket id it belongs to. This is clearly a bad scheme for numerical features, since the ordering information is lost: if x_1 > x_2, then there is no ordering relationship between the embedding(x_1) and embedding(x_2), so the model must first learn this implicit relationship. Such a scheme therefore loses some information in the case of numerical data. The authors propose their PLE scheme as follows: for the data x, if it falls between bin b_t and b_{t + 1}, then embedding_i = 0 if i < t, embedding_i = 1 if i > t, and embedding_t = (x - b_t)/(b_{t+1} - b_t). Thus this is a piece-wise linear scheme which explains the naming of the scheme. The authors also consider some variations inspired by the position encoding used in Transformer models for text, where embedding_i also has a sinusoidal component. 

The authors show experimental results using this scheme where they show that on several tabular data-sets this scheme improves on the baseline MLP models using simple linear embeddings and is competitive with decision trees. 

strengths_and_weaknesses: Strengths: This paper explores a few different representation schemes to make neural networks work better for tabular data and present empirical results demonstrating that it improves on other simpler representation schemes such as linear or one-hot encoding schemes. The paper is also clearly written and the notation is easy to follow.

Weaknesses: Several of the data-sets used are likely very small scale so it is possible that in the abundant data regime the gains from this method are small. I would also imagine that the reason the decision trees are better than neural networks on tabular data of the kind the authors study is also likely because it seems like a data limited setting. In large(r) CTR datasets like CRITEO, it is not clear that 1) decision trees would be better than MLPs and 2) the encoding scheme would give as big of a "win". 
questions: Comments:
- Line 14: "Specifically, after proper embeddings, simple MLP-like models can perform on par with the attention-based architectures"

- Line 44: "In particular, we show, that after proper embeddings, simple MLP-like architectures often provide the performance comparable to the state-of-the-art attention-based models."

The statements in line 14 and line 44 should be qualified with a "for numerical data", since that is the correct scope of the claim as presented in this work.

- Line 55 "3. On a number of public benchmarks, we achieve the new state-of-the-art of tabular DL." -> "state-of-the-art on tabular DL" Typo, please fix.

References:

The piecewise encoding scheme is quite similar to the "thermometer encoding scheme" proposed in [1] for representing image intensities, with the main difference being the absence of the piece-wise linear term (x - b_t)/(b_{t+1} - b_t). The motivation was to see whether this representation conferred any adversarial robustness (which it did not), but improvements in standard (vanilla) accuracy of ResNets was observed using this scheme on MNIST, CIFAR-10 and SVHN datasets (see Table 2, 3, 4 corresponding to "vanilla" and "clean").

[1] @inproceedings{buckman2018thermometer,
  title={Thermometer encoding: One hot way to resist adversarial examples},
  author={Buckman, Jacob and Roy, Aurko and Raffel, Colin and Goodfellow, Ian},
  booktitle={International Conference on Learning Representations},
  year={2018}
}

Suggestions: It would make the work more convincing if the authors reported results on larger CTR datasets like CRITEO.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper address tabular data prediction by using deep models. The authors argue that embedding for numerical features are critical but underexplored in previous work. In practice, two embedding modules termed piecewise linear encoding and periodic activations are proposed to improve the performance of the current deep models for tabular data. Extensive experimental results demonstrate the effectiveness of the proposed two embedding schemes.
strengths_and_weaknesses: Strengths
+ The paper is clearly written and easy to follow
+ The proposed two embedding schemes are straightforward and have practical effectiveness.
+ Extensive experimental results illustrate the performance of the proposed method on various deep model architectures.

Weaknesses
- What’s the insight of designing such two embedding modules? It is more like the practical solutions that depend on massive experiments, which makes me wonder about the intuition behind these choices.

- What’s the relationship between these two proposed embedding modules? Should we need to consider them as independent contributions, and why? Are there any different application scenarios for these two modules? Which one is more effective in real applications?

- Is there any more deep architectures for tubular data analysis, and how about their performance by adopting the proposed two modules. If the authors aim to demonstrate the generality of the proposed module, more recent proposed deep networks (but not the simple Resnet or Transformer) should be considered as the baseline models, and their performance improvement by using proposed modules should be clearly reported.

questions: See the weaknesses. (1)The motivation of the module design is still not clear; (2) How about the performance gain of the proposed two modules for more recent proposed deep architecture.
limitations: The authors partially discussed the limitations of the current work in the conclusion and future work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper presents methods for embedding numerical features for tabular deep learning methods. The paper describes the current best practices in the tabular deep learning literature, and then proposes two basic approaches for embedding numerical features and ways to combine and augment them. First, the authors propose Piecewise Linear Encoding (PLE), where a number is mapped to a T-dimenionsal vector by partitioning the range of values into T bins corresponding to the coordinates, where a coordinate is equal to: 1 if the number is below the respective bin, 0 if the number is above the bin, and otherwise the relative location of the number in the given bin, i.e., (x - low) / (high - low). This is effectively a continuous relaxation of treating numeric features as categorical via quantization. Different strategies for how to compute the bins are considered. The second method is based on mapping to a vector with periodic activation functions, i.e., mapping x -> [cos(w_1 * xx), sin(w_1 * x), ..., cos(w_n * xx), sin(w_n * x)], similar to prior proposal for positional embeddings. This paper proposes that the two base feature embeddings can be combined with some sequence of linear and ReLU activation layers before they are fed into the model.

For evaluating the different embedding methods, the paper considers three base models, a simple MLP, a ResNet, and a Transformers architecture. Various combinations of models and embedding methods are tested on a variety of tabular datasets. The main findings are that with well-chosen numerical feature embedding, deep learning-based approaches can nearly match or exceed the performance of Gradient Boosted Decision Trees. Moreover, it is shown that numerical feature embeddings are important for all model types, and even an MLP can perform exceptionally well.
strengths_and_weaknesses: The main strengths of the paper are its comprehensive set of experiments, including not just one variant but many possible combinations of the proposed embeddings. Such a detailed set of results could prove useful for practitioners that might consider adopting one of the proposed methods based on similarity to one of the given datasets in the evaluation. Moreover, tabular deep learning is still a relatively underexplored subject, and any push to establish better baselines could prove important.

However, the paper ignores the literature on embedding numerical features found in other domains but effectively addresses the same fundamental problem. Namely, there are many papers on how to treat numerical data found in natural language, and specifically how to embed it in the same space as conventional word embeddings. Many suggestions have been proposed over the years, some are identical to the methods in this paper, e.g., the use of periodic activation functions. While the authors note that it was proposed for positional embeddings, it was also proposed for embedding plain numbers found in the text. See [1] for a survey on the various methods used in natural language processing.

It is also difficult to discern what the bottom line is from the many experimental results, regardless of whether the proposed embedding methods are novel or not. It is unclear which method is preferred (if any), and why one method would work better than another on a given dataset. There is little attempt to analyze the results or draw any conclusions. The paper would be significantly improved if it was concentrated on one specific method rather than all possible combinations of a set of methods.

Finally, it is unclear why the paper uses the specific datasets and not others. The only explanation given is that these datasets are "GBDT-friendly". However, it is unclear what are the qualifications of a dataset to be considered as "GBDT-friendly", and why we should ignore other datasets that were previously explored in related literature on tabular deep learning. For instance, in [2] a different set of datasets is considered (though there is some overlap), where they already showed that deep learning methods can beat in most cases GBDT. In my view, this submission lacks a thorough explanation of their methodology for selecting the datasets they analyze and why it is okay to ignore others that have appeared in other papers.

[1] - Thawani et al., Representing Numbers in NLP: a Survey and a Vision, NAACL 2021.

[2] - Gorishniy et al., Revisiting Deep Learning Models for Tabular Data, NeurIPS 2021.
questions: Could the authors explain which method is preferred and under what circumstances? Why would one method perform better on one dataset than on another, or is it all just statistical noise? I would strongly advise to move the bulk of the various variants tested to the appendix and keep only the one or two leading variants (their naming scheme is also quite confusing). It could be worthwhile to display more varied results in the context of an ablation study.

Could the authors please explain their methodology for picking the datasets to evaluate their methods? What does "GBDT-friendly dataset" mean?

I would also suggest the authors to include in their related works section a paragraph on the related methods using the NLP literature for embedding numbers.

As a very minor complaint, I would also recommend the authors ensure each time initials are introduced that their full meanings are also included. For example, GBDT appears in the abstract without explanation. CTR appears in the body without ever explaining what it stands for (Click-through Rate I guess), as well as DL (Deep Learning). Moreover, I would suggest the authors to refrain from using "dramatic" wordings (e.g., "game changer", "epic difference", "we can't help") -- let your results speak for themselves.
limitations: The authors discussed the runtime costs associated with using deep learning-based methods relative to decision trees, as well as the difficulty in drawing far reaching conclusions based on the selection of datasets and their sample size.
review_title: null

Meta-review: metareview: The authors propose and study the use of embedding scheme to apply deep learning to tabular problems. According to reviewers HiMf and 5oG4 and reading the submission, the method is simple and clearly explained. The experiments are comprehensive and demonstrates empirical improvements on small scale datasets. Moreover, discussions with reviewers have allowed the authors to provide additional relevant experiments providing comparisons with other methods. 
I recommend this paper for acceptance.
confidence: Less certain
Decision: Accept