Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper devises an approach to semantically encode ordinal categorical features as numerical features by using pre-trained language models when preprocessing tabular data, named BERT-Sort. In addition, this paper introduces a new benchmark that is collected from 10 different public data sets with 42 different ordinal features. Upon this benchmark, BERT-Sort significantly outperforms traditional encoding methods such as OrdinalEncoder.
overall_review: N/A for reproducibility reviewers
review_summary: N/A for reproducibility reviewers
review_title: null

final_score: rating: 3: Marginally below the acceptance threshold (use sparsely)
review_content: summary_of_contributions: This paper considers preprocessing of categorical features by treating them as text, to which a pretrained Transformer like BERT is applied to identify what would be the most likely ordering of these words in a natural corpus. Then the categories are ordinally encoded according to this ordering. 


overall_review: This paper introduces a very simple idea with promising looking results. However, there are not enough comparisons/explanations of the details to convince me their results will be as useful in real-world AutoML as presented in the paper.  

First off, there should be more baselines. Eg. one option is just to use pretrained (or fine-tuned) BERT to embed the categories' text into a higher-dimensional vector which can be fed into AutoML systems. This is considered in the following paper which should be discussed:

Benchmarking Multimodal AutoML for Tabular Data with Text Fields (2021)
https://arxiv.org/abs/2111.02705

Another option is to represent categories in a label-dependent way via target mean encoding. A third option is to use a feedforward network for the dataset with embedding layers for each categorical variable that learn embeddings of its categories from scratch. These embeddings could then easily be fed into the AutoML system.


Other questions:

- For reproducibility, what versions of each AutoML tool are you using, what machine are you running them on, are you running each tool as the only process on the machine?

- How much extra time does BERT-Sort add to the overall training of each AutoML system?

- Why did you choose F1 score as the predictions' evaluation metric?  I would be curious if the BERT-Sort gains remain across other evaluation metrics like AUC, accuracy.

- Why are there so many AutoML failures on your chosen datasets? Are you running on a machine with enough RAM, or are these datasets peculiar in some fashion?

- How do you handle missing values in your approach?

- How do you handle previously unseen category in the test data during inference?
review_summary: The presented idea is very simple the implement and can be combined with any AutoML system (although the authors should comment more on its additional computational complexity).  My biggest concern is around the trustworthiness and generalizability of the results (in particular the boost to AutoML systems' accuracy which seems like the major contribution of this paper).  If the authors can assuage my concerns, I am open to increasing more score, as the results do look promising and I like the simplicity.
review_title: null

final_score: rating: 3: Marginally below the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The authors introduce Bert-Sort, a novel method to semantically encode categorical values with zeros-hot Masked Language Models for tabular data. They compare their approach to existing techniques used in modern AutoML systems both in terms of accuracy of ordinal encodings and end-to-end AutoML performance via 10 datasets and show improvement in both areas.
overall_review: The authors propose a novel and interesting idea to leverage pre-trained language models for more accurate ordinal encoding in tabular datasets. This has strong potential to improve end-to-end AutoML systems by improving the quality of the data fed to downstream models. In particular, this problem has no easy solution outside of naive approaches as described by the authors currently implemented in AutoML systems, so leveraging pre-trained models to order the values is a step in the right direction, and shows encouraging results via Table 4.

However, I cannot ignore the major issues in Table 5 as I discuss in detail in earlier sections. Unfortunately this drastically hinders the paper due to a combination of misleading numbers, lack of information on reasons for dataset failures, a lack of overall per-dataset scores, a lack of reproducible code to run the experiments, and an incorrect conclusion that fails to discuss any drawbacks to the method or inconclusive findings from the experiments.

Because of these issues, I cannot accept the paper in its current form.
review_summary: This is a good paper hindered by a flawed experimental result analysis and conclusion which unfortunately cannot be overlooked in its severity. Because of these issues, I vote to reject. Refer to the other sections for detailed comments.

[Update from Rebuttal] I have increased my score from 2 to 3 due to the improvements made in the rebuttal.
review_title: null

final_score: rating: 3: Marginally below the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The authors' primary contribution is the development of a method of creating more meaningful feature embeddings for categorical variables that may carry (hidden) ordinal information. For instance, the words [mild, moderate, severe] make the most sense in terms of an ordinal relationship, and the authors posit that creating an embedding for this variable that respects this order could make ML models more successful at detecting relationships between these and other variables.

To enable these embeddings, the authors leverage MLM (masked language modeling) models, which are commonly used in NLP to predict a masked token in the context of other (unmasked) tokens. These predictions are then used to determine which ordering of the variables is most appropriate, in effect leveraging the MLMs' knowledge about what order these words appear in the corpora on which they are trained. The authors test many different choices for MLMs and emphasize that practitioners can choose from among many pre-trained models in the application of their technique.

Furthermore, the authors suggest that the method they use to determine the most likely ordering can be leveraged as a kind of metric to indicate whether a sequence of variables exhibits an ordinal relationship. This assertion, along with the primary one, is tested using different AutoML technologies, which the authors claim demonstrate the superiority of their solution over the default categorical embeddings found in these libraries.
overall_review: The authors have found a problem that many ML practitioners will find interesting and their solution seems to be a good one to me. I appreciate how the authors draw from the knowledge of NLP researchers to aid in other fields. The proposed solution is just complex enough to be interesting, while still being tractable to implement for any experienced ML user.

The paper needs a lot of attention to the language and technical aspects, however. The authors are unclear about what their main results are, which obscures from the reader whether they are meaningful. Mathematical and notational mistakes are common enough that they require the reader to make leaps to understand what the authors are saying in places. Finally, some results (e.g. the results about using MLMs as a metric of ordinal relation among tokens) seem to fail in spite of the text stating the opposite. 

I would suggest that the authors carefully rewrite their paper with more clear results and some editing and resubmit at a future venue.
review_summary: Due to the clarity issues and the many technical errors and typos throughout the paper, I do not recommend the paper be published in its current state. The idea seems promising and I believe the authors could have made a significant contribution to the field, but in its current state, it fails to deliver on its promises or convince me that it was performed thoroughly and competently.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The authors introduce the BERT-Sort algorithm to semantically encode ordinal categorical values via a zero-shot pre-trained masked language model. The authors also construct benchmarks from 10 public sets for sorting ordinal categorical values, where their proposed method seems to outperform existing autoML approaches.
overall_review: - The authors should provide an additional ablation study on the necessity of the two approaches detailed in Section 3.2. The design detailed in Section 3.2 seems to indicate that the Bert-Sort algorithm is not even able to 1) sort [Hot, Boiling Hot, Lava Hot, Cold] into [Cold < Hot < Boiling Hot < Lava Hot] or 2) sort months according to their ordinal numbers, which seem to be extremely simple when a tremendously large model such as BERT is used. I think the author should provide experiments regarding Bert-Sort without using the two approaches mentioned above. Otherwise, I would presume it is the inductive bias in the first and the second approaches that induces the results in Table 4. 

- In Table 5, I do not see a significant increase in the number of successful trained models when BERT-Sort is used. To be more specific, the number of successfully trained models is 23/50 w/ the BERT-Sort algorithm and 21/50 w/o the algorithm. Moreover, for AutoGluon, FLAML, and H2O, there is no impact when BERT-Sort is used.
review_summary: I like the ideas proposed by the authors, which I believe to be novel in the field of AutoML.
But I only recommend weak acceptance at this moment, as the empirical results are not extensive (i.e., do not provide ablation study), convincing (i.e., do not repeat over different random seeds), and significant (i.e., the results in Table 5 are far from being prominent).
review_title: null

Meta-review: metareview: This paper considers preprocessing of categorical features by treating them as text, to which a pretrained Transformer like BERT is used to identify what would be the most likely ordering of these words in a natural corpus. 

I would like to thank the authors for actively addressing reviewers' comments or providing clarifications whenever it was needed. As a result, ARU2 and 61b6, and 96Ap have raised their score and are inclined to (weakly) accept the paper. However, one of the reviewers raised many clarity issues, technical errors, and typos and suggested a weak reject. 

I would recommend assigning a shepherd to this paper to address the reviewer's 6K9H in terms of presentations and writing. 
confidence: 3: The area chair is somewhat confident
Decision: Accept