Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposes a new benchmark on evaluating AutoML methods on tabular+text data. The paper accumulated 18 datasets from industrial applications that combine numeric/feature fields with free text-form fields and are either regression or classification tasks. As a motivation for this benchmark the paper shows that adding inputs from text-columns helps increase the performance on this benchmark.

--- 
After the author response, I feel like my concerns have been partically addressed and I remain at the rating 6. The reason for not going above is that I still believe that the paper does not provide enough analysis of the presented results. It would be great if the authors can state this part stronger and expand on it in their final version, as this will motivate readers of this paper to work on and develop tabular+text methods, which is the goal of this benchmark.
strengths: The paper is well written and easy to understand.

The paper evaluates various baselines including simple "hacks" on the proposed benchmark and also proposes a method that works best on this dataset.

The benchmark, consisting of 18 datasets seems sufficient for evaluating models in this Tabular+Text domain.


weaknesses: What are the performances of the methods without using the text field? I can see performances of using only text and the combination, but am not sure about the performance without text.

Lack of analysis. While the paper shows that using multi-modal embeddings can help, it’s not obvious why or when. Especially since this paper advocates for the usage of text-inputs, it’d be interesting to understand when it helps and when it does not. Also, the fact that “the majority of additional gains produced by exponential decay”, is something that is hard to understand why this is the case for text models.

Lack of clear learnings. The differences between the methods are quite small and there do not seem to be any major insights gained from using this “new” source of inputs. It'd be great if the paper makes the findings (that results from this dataset) clearer and better differentiate it from the methods applied.
correctness: Yes.
clarity: Yes.
relation to prior work: Yes, as far as I can tell.
additional feedback: The paper reads a bit like an advertisement, e.g.
> The data files are currently hosted in AWS Simple Cloud Storage 150 (S3), which is a reliable and highly-available medium.


> modeling is simply done via AutoGluon-Tabular, an easy-to-use and highly accurate open-source tool for automated supervised learning on tabular data

or 
> For real-time applications with latency constraints, AutoGluon provides options to accelerate ensemble inference via pruning or distillation [17] 
which seems completely unrelated to what the paper is proposing.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors present a benchmark for measuring the progress of AutoML algorithms on already-existing datasets with multimodal features, such as text and categorical/numerical ones. The authors systematically evaluate the single- and multimodal models, provide a wide variety of baselines and demonstrate empirically that understanding both modalities is crucial for achieving good performance.

strengths: Authors tackle the vital problem of measuring how robust an AutoML algorithm is on a wide variety of tabular tasks. The implications of the achieved baseline results can spur further research ideas. In addition, the benchmark itself has chances of being adopted by the research community as an essential metric. The authors' good choice is to align the datasets with the real-world needs, thus helping to increase the capabilities of using AutoML in industrial applications.

weaknesses: My main concern with the proposed benchmark is the lack of presented criteria for selecting these particular datasets from all available tabular datasets. The only rationale why these 18 datasets should be considered together is presented using a very vague statement regarding their diversity and real-world relatedness. Still, I would expect that the authors performed a broader search, that baseline algorithms with multiple modalities were evaluated, and that based on the explicitly formulated criteria (e.g., dataset size, scores improvement of multimodal models) some of them were selected, and others were rejected. The lack of assuring the stated qualities during the selection process weakens the overall benchmark's impact. **Why should researchers use the provided aggregation of datasets instead of choosing a different subset of available datasets?**

**Another point worth questioning is whether some of the datasets (e.g., ae, jigsaw, fake, house) has enough room for improvement as the baseline model achieves up to 98% accuracy on some of them.** But, again, this seems related to the previous point of explicitly stating qualities of considered datasets, and the datasets featured in the benchmark should remain challenging to measure the field's progress.

Additionally, the `prod, IMDb, channel, house` datasets seem to not provide reliable information for measuring improvements of a multimodal model. After adding the textual information, the score improvement is not significant for AG models. **So, how will these four datasets allow us to understand the importance of multimodal reasoning?**

Another concern I have regarding the experimentation and comparison of different models - **Is the comparison between fuse-early and fuse-late models fair considering parameter count and the number of operations performed?** From what I see in the appendix, the fuse-early method adds more tokens for tabular data, whereas fuse-late approaches allocate more representation capacity to the multimodal information. If the two methods differ significantly in the costs (i.e., if the bigger model performed better), then some of the statements presented in the discussion section may not hold. However, clarifying that they are comparable because, e.g., they feature the same amount of parameters will clear my concerns.

Overall, the paper's contribution to the "benchmarking" side of AutoML models is not rich enough or not described. It seems like the authors gathered some datasets (diversity appears to be a natural side effect and not a planned feature) and performed extensive evaluations on them to prove that they correctly chose them. However, even some of the results point against it. The problem I see is that the experimentation section of the paper should not be the main contribution of a benchmark paper. There are plenty of low hanging fruits that authors can improve in designing the benchmark itself:
1. provision of human performance (to understand the score at which tasks will be considered solved),
2. description of a dataset search process, e.g., where did authors look for the datasets and what phrases/keywords were used? (to understand that this benchmark cover the most important ones)
3. selection process and elimination rules (to understand criteria for choosing the datasets)

**TL;DR**
Despite this, the paper seems to be a solid submission for a top conference, but some of the design choices in the benchmark itself lack rationale. Most importantly – why should one follow your preferences of the 18 datasets presented here? Additionally, how important is it to measure the improvement on already solved tasks?


correctness: The experiment seems sound and solid, with only one minor concern about the fairness of comparison. The author presented and described a wide variety of models, and the choices made there are clear. In the benchmark itself, there are some critical issues raised above, but also many design choices are correct (e.g., metrics, the mix of classification vs. regression tasks, datasets from real applications)
clarity: The paper is extremely well written, a paragraph-to-paragraph transition is very nice, and reading many multi-sentence passages is a pleasure.
relation to prior work: Yes, the authors explicitly compare to previous contributions in section 2.
additional feedback: This question does not impact the score, but addressing it may be considered helpful for further reading:
1. The second point in the discussion seems to be highly related to the third point, that in general, stacking models later helps rather than fusing low-level features. Do you have any rationale or intuition behind the lower performance or any suggestions regarding what can be improved for the `Fuse-Early` type of approaches?
review_title: Important topic, rich experimentation section, but the benchmark itself lacks some of the qualities of a good benchmark.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors present a novel benchmark of 18 tabular classification/regression datasets, consisting of tables with numerical, categorical, and at least one text column. All datasets are publicly available and have been aggregated from different sources, mainly Kaggle competitions, covering different domains, sample sizes, and feature numbers with the goal to provide a robust benchmark for real application deployment of AI methods -- evaluating a single model on the entire collection of datasets without manual adjustments. The authors evaluate various baselines and more sophisticated model aggregations on their benchmark, highlighting the heterogeneity of their dataset. Their best performing model, a stack ensemble of tabular models and multimodal transformers, also indicates the complementary nature of both methods.

strengths: - Collection of diverse datasets for three different tasks
- Comprehensive evaluation of several tabular/text-based models and their combinations
- Results highlight the heterogeneous nature of the benchmark
weaknesses: - While the datasets themselves are quite diverse, I think the evaluation's methodology does not consider an important aspect: training a model with data from each dataset in the benchmark and showing that it performs well still only tells us that the model performs well on domains it has already seen. It is intractable to cover all the domains and applications  (and as the authors note, their datasets are largely related to business applications), so it might be worth considering an evaluation where some datasets (of already trained tasks) remain completely hidden. To me, this would be a more accurate evaluation of an AutoML model -- as I normally do not have any training data for the application I want to use my model for. 
- As different metrics have been employed for different tasks,  averaging the scores (as in Table 3) over all datasets does not seem sound
- It would be nice to know whether the format of the tables has been restricted in some datasets, e.g. that headers are always only the first row of a table.
- Some peculiarities of selected datasets (such as the extraordinary low scores on the pop benchmark) were not addressed.
correctness: See above. Experimental design is sound and described in detail in supp. material.
clarity: Clearly written, although some points in the experiments section should rather belong into the discussion (e.g. l.340 -l.344). A conclusion is also missing and must be added in the case of a camera-ready version.
relation to prior work: - It is mentioned in l.76-77 that recent work on transformers for understanding structured text tables, such as TaBert or TAPAS addresses a different task than the one in the paper, it is not explained in what way this line of research is different. Is it that these models have been largely studied in the context of question answering? It would be good if the authors could explain their argument, as to me these models seem like a very natural candidate to explore in the proposed benchmark.  
- It would be good if the authors could cite relevant papers for each dataset they are using, or directly refer to the dataset page. This information can be found in the repository but seems appropriate to be mentioned in the paper itself.
additional feedback: - It appears all citations after page 1 are somewhat broken as clicking them sends me back to page 1. The issue was reproduced in two different pdf readers.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper describes a benchmark focused on assessing AutoML strategies on datasets with features of multiple modalities, i.e. numerical, categorical and textual. The benchmark consists of 18 datasets originally from platforms like Kaggle, cover classification and regression tasks, and match each task to a standard metric. The evaluation of several multimodal AutoML methods show that extracting signal from textual features improves over numerical-only, which confirms the importance of this benchmark.

----

I thank the authors for their response and appreciate the revised paper. I have updated my rating accordingly.
strengths: It seems valuable to evaluate, hence motivate, AutoML strategies for their capability to handle features from multiple modalities. The experiments demonstrate the incremental predictive performance once models are tailored towards leveraging textual features.
The discussion of the importance/impact of this in practice is not made very concrete in the introduction. Adding this discussion would strengthen the paper.
weaknesses: Little analyses are presented that give insight into the datasets besides the number of features per data type. A better understanding of these datasets would be informative for analysing future methods benchmarked on these datasets.

The choice for metrics per dataset is weakly motivated; the standard metrics per prediction task class are chosen, disregarding for example class imbalance per-dataset. A few more comments on this in “correctness”.

Although the argument for AutoML benchmarks with a variety of data types is convincing, the motivation in the paper could be strengthened.  For example, an aggregate of feature importances across datasets, e.g. based on the analyses in Appendix C, could support the motivation. Alternatively, a reference to Appendix C in the main paper would be good.
correctness: The methods benchmarked in this paper seem implemented correctly. I would expect that important preprocessing steps as implemented in the benchmark datasets would instead be part of an AutoML pipeline assessed with this benchmark. For example, it is noted that “we simply rescale and center numeric features and impute their missing values via their average.” But it makes more sense if these are left as a challenge for AutoML pipelines, e.g. imputing missing values with a feature’s average might not necessarily be the best strategy for all datasets. Similar for removing “non-predictive features”, duplicated rows, and highly correlated features, as mentioned in the main paper. I would expect pipelines to solve these tasks automatically.

The evaluation metrics correspond to “standard” metrics for these modeling tasks. I have a few remarks about this choice:
- Some of the datasets are originally accompanied with other metrics (e.g. *house* and *qaa*), it is not explained why other choices are made for these datasets.
- For multiclass classification tasks accuracy might not be accurate, especially in cases of class imbalance. In these cases, I believe more common metrics are precision, recall, F1-scores, etc.
- R2 for regression seems fine to use although it is not bounded by [0,1] as stated.

clarity: The paper, in general, reads well it is hard to make comparisons across datasets and methods based on Table 3. Should this table be exhaustive, or provide easy comparison and key takeaways of evaluations of a few methods? The latter seems more interesting to the reader in this paper. A visual representation of the evaluation would improve this, although some methods might need to be suppressed.
relation to prior work: Sufficient detail is provided to understand the added value of the benchmark. Prior benchmarks for AutoML incorporate only categorical and/or numerical variables. The cornerstone of this benchmark is that textual variables exhibit significant signals towards prediction tasks. A few such datasets exist but are not considered fit for purpose due to their size or representativeness of “modern applications”.
additional feedback: The descriptions of the datasets seem inconsistent. All descriptions cover the prediction task, but some include more information of the dataset source and what they are used for. Are the other datasets from Kaggle not used in competitions?
review_title: Paper introduces a valuable benchmark

Meta-review: This paper presents a collection of tabular+text datasets and also introduces new methods for handling the combination of tabular+text. This doesn’t provide a perfect focus on the actual new dataset collection, which was the primary issue that reviewers criticized. The authors added much new information about the actual datasets in an extensive rebuttal phase with all reviewers, in the end leaving the reviewers satisfied. I therefore recommend acceptance now. I encourage the authors to add any further documentation and statistics about the datasets they think could be useful.
Decision: Accept