Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: There's a huge problem with benchmarks in that there is no way to account for differences in performance among models viz. confounding variables. They respond to this issue with control variables, for example fixed hyperparameters, data splits, datasets, out of the box models, and so on. Even more so, the current benchmark community's focus on accuracy without considering other variables is insufficient to address benchmarking needs of the NLP and other communities. As such, the authors provide an array of evaluation metrics that go beyond performance, targeting what they term "value-drive communities". 
strengths: The authors don't explicitly mention this aspect although it is apparent, but their benchmark actually can highlight inequality and climate impact in a lot of ways because they provide a mechanism to show 'hey, their model performed better because they had a pod of TPUs at their trillion dollar tech company'. I think the note for a carbon footprint is extremely timely, especially considering Bender et al's paper at FAccT 2021 about the high carbon footprint of NLP models. I love that this paper also says 'let's go beyond accuracy!'. Obviously, there is more to a model than how well it optimizes for loss. In addition, there is a mechanism to keep track of criteria to potentially understand WHY one model outperforms another.  
weaknesses: I would love to see an explanation of how one would account for differences in models where there are multiple variables at play. For example, X team had two different variables than Y team; which variable was most impactful, if any? Also, if the model is customized, how can one keep track of which of the criteria were impacted by such customization? How do we compare two different models that were not trained on the same datasets or don't use the same evaluation metrics? Or do we not?

As the authors point out, the benchmark only works on supervised tasks because of it's dependence on Ludwig, but to be fair evaluating unsupervised tasks is tricky and mostly non-standard. However, there is a growing interest in unsupervised deep generative and contrastive models, so it's still a weakness nonetheless.
correctness: Are the claims made in the submission correct? 
The authors are definitely biased toward their previous work and don't critique it the same way they do other works, but I didn't find any glaringly wrong claims.

If it is a benchmark, are the evaluation methods and experiment design appropriate and performed correctly?
Sure, there's more than one way to skin a cat.
clarity: I think the paper is well-written, but I would lowkey like to have seen some sections or statements appear earlier. However, this might be my reviewer hat rather than a general audience need.

One point of the paper where I'm a bit confused about is whether this is an NLP benchmark for text data or a generic benchmark for tasks across specialized AI domains, especially because RNN/LSTMs can be used for computer vision tasks. I noticed that the GitHub only included text dataset samples, but the paper does not explicitly refer to this benchmark as an NLP benchmark, whereas for example ImageNet is very explicitly a computer vision benchmark. It would be great if the authors could clarify about their vision of the scope of the benchmark in specialized AI tasks because the paper doesn't mention images in LBT until 5 pages in. The architectures that are provided are mostly used for NLP tasks in practice.

I'm also wondering what variety of tasks the fixed hyperparameters will perform well across different conditions (e.g., datasets, tasks, models)?

What scale is this benchmark measuring? There's a huge difference in the size of the models for a research paper using MNIST digits and an industry model that takes several months to train using an ungodly amount of hyperparameters. Another question is can this benchmark scale to ginormous models in industry or other places?

If a model performs great on this benchmark, is that the end all be all? What else should benchmarkers be looking for?

For some reason, Figure 1 in the paper shows MNIST and CIFAR-10 in a diagram of LBT, but I cannot find these as "default" datasets on the GitHub although I believe users have the option to add them on their own. However, it's kind of misleading, especially as the first graphic the reader sees.

Will there be no leaderboard whatsoever to show progress in the community, or will the results of the benchmark be internal, e.g. a company keeps track of their own internal leaderboard? I know the authors said this toolkit complements the leaderboards and that there is an Elasticsearch database. I also wonder if there's a way to log the submitters or if they can also stay anonymous?

I think there might be some missing or mislabeled figures that kind of threw off my ability to fully understand a few things??

I notice the authors bring up "robustness", but there isn't much discussion in the way of the relationship between accuracy, generalization, distributional shifts, etc. If you're going to talk about robustness, then a robust model goes beyond accuracy or even subgroup accuracy. So it seems LBT does not go beyond accuracy but rather is in addition to accuracy. I think there was a reference to adversarial examples, but the discussion on robustness was not unified at all. On that note, I don't know how the authors are defining robustness. While I sympathize with the page limit, it left me wanting. I think even the convenience that RobustnessGym already provides its own robustness metric, e.g. subpopulation performance, is a good reason to define robustness in a certain way. This was one of those things that made the paper not really self contained.
relation to prior work: The author(s) of the paper have also created DawnBench which is relatively well known. There is a website to compare models: https://dawn.cs.stanford.edu/benchmark/ImageNet/train.html. DawnBench uses the following criteria to compare models: Model	Time to 93% Accuracy, Cost (USD), Max Accuracy, Hardware, Framework (i.e., ML software). However, the only dataset is ImageNet, and thus measures accuracy for computer vision models only. Either way, they're pretty similar, but LBT has distinct contributions and takes it a step further. The authors cover other related work in the paper and provide justifications for how the work differs from previous contributions.
additional feedback: I think the paper solves a really good problem by providing control variables for some semblance of standardization across benchmarks.
review_title: Personalized Benchmarking with the Ludwig Benchmarking Toolkit

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper provides a toolkit for benchmarking models in a uniform way. To enable this they include infrastructure to standardize model training and and hyper parameter optimization. 
The paper also claims to include the ability to use it for multi objective evaluations.  This includes cost/ efficiency latency training speed etc ...

The paper then uses the framework to perform a case study. Here the paper re-uses existing datasets to benchmark classifiers. And in the case study they use 5 pre-trained models and 2 models trained from scratch. 
- They show that the a higher latency model does not always perform better than a lower latency model (But not information on how the latency was measured is in the main paper). 
- They show that there is not a single model that is best for all tasks.
- They provide results on robustness for different models but no in depth analysis
- They give results that shows it is hard to predict which models will converge further.

I am very conflicted to give this paper a score. On the one hand, there is nothing fundamentally wrong with the paper. On the other hand, there is no case study that provides new insight. This makes the paper sit right on the edge for me. I could agree with acceptance and rejections. 
strengths: - I have the impression that the toolbox itself can be very useful
- The paper is mainly well written and easy to understand but often I would want to have additional/more precise information
weaknesses: The main weakness of the paper is the lack of insight from the case study.
Ideally in a benchmark paper I would prefer to gain additional knowledge. Unfortunately when I read this paper I do not get the feeling that I learned something new. 

On top of that several aspects are not clear to me:
- The paper claims to support latency/cost measurements. it is not clear to me for which devices this is possible or how this is implemented. Are these hard measurements/simulators/analytical models or are hooks present for the users to provide this information. 
-It would be good to provide an overview of the models that are already available in the toolbox. 
correctness: Evaluation seems reasonable and correct
clarity: Paper is clear
relation to prior work: The paper re-uses existing datasets. it is unclear which insights coming from the combined evaluation are truly novel. This is the main weakness of the paper
additional feedback: N/A
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors introduce an extension to a popular open-source framework. This framework, Ludwig, allows users to easily build and train models by specifying simple configuration files. The authors' extension of this framework allows for researchers to specify a set of benchmarking tests for these their own models.

The authors' benchmarking framework allows users evaluate the effects of choosing different hyperparameters, optimizers, hardware, allotted training time and model on several relevant objectives. These objectives comprise both model performance (e.g. accuracy, robustness) and computational resource usage and characteristics (e.g. cost, training speed, latency). The value of this framework comes from allowing users to perform these tests by simply specifying the desired datasets, models, metrics and objectives in configuration files mirroring those already in general use.
strengths: This is a useful tool for researchers, and is directly aligned with the spirit of the NeurIPS benchmarks and datasets track. This benchmarking framework is specifically designed to make it easier for small research groups (who  lack the resources afforded by well-funded teams and training hardware found at large organizations) to carry out research in machine learning in a more principled and reproducible manner. Thus, it is directly aimed at making at making machine learning research more equitable and accessible.

The authors provide relevant code and configuration file examples that demonstrate how to use the benchmarking framework. The authors' choice to leverage open-source software infrastructure already in common use as a foundation for their framework is a wise one, and increases the likelihood that the ideas in this paper will stand the test of time.

Even if the authors' actual code is not directly used by other other researchers, the ideas and challenges outlined in the paper will provide useful insight to others who wish to build on them. In particular, I think the focus on easy user-configurability and the emphasis on measuring computational resource usage as well as accuracy is important. I hope other common machine learning benchmarking software takes note of these ideas.

The code is freely available, and is of decent quality as far as research code goes. 
weaknesses: No major weaknesses that are not already accounted for in the paper. In fact, the authors include a refreshingly up-front discussion of the limitations of their own work.

In particular, the authors' reliance on other software for implementing the benchmarking limits the applicability of the framework primarily to supervised models and a limited selection of datasets. In addition, this paper does little to help users address the thorny tradeoff the accuracy and conclusions drawn from the benchmarks vs. the increased computational resources required to explore more of the parameter space.

Both of these points are addressed in the paper.
correctness: The authors use their framework to run an ensemble of text classification experiments. They train seven different models on nine different datasets on a variety of hyperparameters. They have several interesting findings, including some evidence that pre-training models does not have as great an effect on model performance as is commonly believed. The experiments are an appropriate test for the benchmarking framework.
clarity: The paper writing and layout is excellent. The authors clearly outline their goals and target audience, and the potential uses of this framework.  They provide code examples in the paper along with experimental results.
relation to prior work: This authors provide a very good discussion of related benchmarking software, and where their framework fits in context. In particular it emphasizes that this paper is more focused on enabling multi-objective benchmarks that can be reproduced easily. The present work is focused on giving holistic measures of performance, both predictive and computational, as opposed to merely predictive measures.
additional feedback: The following would enrich the paper overall:
* A brief discussion of how other benchmark makers (gym, etc) might incorporate lessons learned over the course of this work into their libraries.
* Remove filler words like "easy", "simple", etc.
review_title: A solid contribution to this conference

Meta-review: The paper provides a software framework for conducting machine learning benchmarks. The reviewers agree that such software will be useful for the community, and that the paper is of high quality. The main concern is whether a paper on software, as opposed to a dataset or benchmark, is in scope for the datasets & benchmarks track. Due to the importance of open source software for machine learning research, I lean towards a broad interpretation of the datasets & benchmarks track to include software for benchmarking, and hence recommend to accept the paper. 
Decision: Accept