Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper collects and adapts previously published datasets to construct a standardized benchmark for TinyML methods, which previously suffer from poor comparability due to software/hardware heterogeneity. The tasks include keyword spotting, visual wake words, image classification, and anomaly detection. The benchmark has two divisions, one allowing any changes to the pipeline and one only allowing post training optimization. It measures accuracy, latency and energy use.
strengths: 1. The benchmark contains a wide range of tasks, despite lacking NLP-related ones. The tasks are also oriented at low-resource environments and applications can be found in real life (e.g., single person detection for doorbells). 
2. The measurement considers latency, energy and accuracy. The use of a quality threshold instead of the commonly used tradeoff curve comparisons greatly simplifies the comparison. 
3. The benchmark has two divisions: an open division that allows the change of model, data, training, and a closed division that focuses more on post-training optimization. This allows different methods that focus on different parts of the whole optimization process to show their benefits more clearly.
4. The challenges for constructing a standardized TinyML benchmark are well summarized in the related work section. 
weaknesses: The running and submission process to the benchmark, as documented in the Appendix, requires significantly more effort than a typical benchmark submission, where submitters usually only submit the test results. This can limit the number of submitters. However, this is perhaps a tradeoff that has to be made for more comparability for the widely heterogenous TinyML works.

It would also be nice to include a summary of the execution process in the main paper instead of only documenting in Appendix.


correctness: Yes. The benchmark is constructed from previously published datasets, with adaptations made for low-resource and simplicity considerations. The evaluation and experiment design are well documented in paper.
clarity: The paper is well written.
relation to prior work: Yes.
additional feedback: Please see strengths and weaknesses.
review_title: Standardized and Diverse Benchmarks for TinyML

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents MLPerf Tiny, the first industry-standard benchmark suite for ultra-low-power tiny machine learning systems. MLPerf Tiny measures the accuracy, latency, and energy of machine learning inference to properly evaluate the tradeoffs between systems.
strengths: This paper presents the first industry-standard benchmark suite for ultra-low-power tiny machine learning systems, it could push the development for compact model design.
weaknesses: Specific experiments are lacking, such as the accuracy, latency, and energy of the ML models (Mobilenet，MicroNets，RNN, AutoEncoder) on MLPerf Tiny Benchmark.
correctness: Yes.
clarity: Yes, the paper is well written.
relation to prior work: Yes, this paper compares the proposed dateset with previous contributions.
additional feedback: It would be better if there are more specific experiments, such as the accuracy, latency, and energy of the ML models (Mobilenet，MicroNets，RNN, AutoEncoder) on MLPerf Tiny Benchmark.
review_title: Good paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper focuses on the field on tiny machine learning which are used in ultra-low-power devices. The first industry-standard 
 benchmark is introduced in this paper named "MLPerf Tiny". The details of the new benchmark are presented in various aspects, including tasks, datasets, models, evaluation process, and system design.

This paper makes the following contributions:

1. A new benchmark on tiny machine learning is established. 
2. The related work is introduced thoroughly. The motivation of building such a benchmark is clear.
3. The benchmark design is detailed with a modular implementation.
4. It includes various tasks including image classification, keyword spotting, etc.
strengths: + The contributions of the new benchmark on tiny ML are significant since the widespread use of ML models in smart devices require the fair comparisons between different methods and models. Establishing a new benchmark in this field may be valuable.
+ The benchmark is relevant to a broad research community, and especially from the industry perspective.
+ The benchmark is open-sourced and will be maintained in the future.
weaknesses: I'm not an expert in this area. From my perspective, I have few concerns about the benchmark.

My biggest question is about the evaluation. Although the authors have introduced many aspects of the benchmark, I do not see many evaluation results done in this benchmark. As there may be many existing methods/models performing tiny ML, it is expected to conduct comprehensive studies to compare the existing methods. Based on the evaluation, could you have some findings or the limitations of existing methods?

For Figure 1, can you provide more illustrations about the connections to the benchmark? E.g., how do you deal with each level of Figure 1 in your benchmark.
correctness: This benchmark includes various tasks, all of which use the public datasets. There is no data collection process in this benchmark. However, this benchmark only introduces the techniques without many experiments. So the experiment design cannot be judged correctly. 
clarity: This paper is mostly well-written.
relation to prior work: This paper claimed that it is the first benchmark. The related work is introduced thoroughly (to me).
additional feedback: Please address the concerns about the evaluation.
review_title: Review of "MLPerf Tiny Benchmark"

Meta-review: The paper proposes a benchmark for machine learning models running on low-power devices. All reviewers saw value in the proposed benchmark, but initially had some minor concerns and requested more details about how to run the benchmark, wanted to see more evaluation results, and wanted to see more discussion about connections between different layers of the TinyML stack (Figure 1). The authors were able to address these concerns through their responses and revisions to the paper, and in the end all reviewers voted to accept the paper. Congratulations on having your paper accepted to the NeurIPS 2021 Datasets and Benchmarks Track! The authors are encouraged to use the additional space in the camera-ready version of the paper to continue refining the language and presentation in light of the new additions suggested by reviewers.
Decision: Accept