Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper emphasizes the importance of context and nuance when discussing “efficiency”. The authors explain different efficiency metrics and the distinction between training and inference efficiency. They then illustrate how assessments of the “efficiency” of different models can be misleading or even contradictory depending on factors such as the choice of efficiency metric, baseline, architecture, hardware, optimizer, and other experimental design choices. The authors finish with a few suggestions for best practices.
main_review: The motivation for this paper is excellent. The measurement and discussion of “efficiency” in contemporary ML research is typically simplistic, and sometimes misleading. The ML community is in dire need of work that explains relevant quantities and considerations, provides concrete examples of pitfalls and poor practices, and emphasizes the importance of context and nuance when assessing “efficiency”. Detractors might accuse this paper of not containing sufficient original research for publication in a venue such as this one. My response to this criticism is that ML research would be in a better place if researchers took more time to reflect on the state of research and identify ways in which our practices and scholarship could be made more robust and rigorous. That is what this paper does.

Unfortunately, I don’t think this paper fully realizes its potential. It could be clearer and more focused; it reads like it was written quickly. Many of the examples don’t feel sufficiently explained, and some don’t feel well-integrated. In its current state, I am not confident that it is suitable for publication. However, I am optimistic that it could be after revision.


**Feedback**

My primary criticism is that the paper is insufficiently concrete. It would be substantially more impactful with more real examples, ideally with accompanying plots and/or tables. For example, on page 5 the authors state

>Some hyper-parameters, such as learning-rate and weight-decay, can be tuned so as to reach good quality quickly but then plateau lower than in “slower” settings that eventually reach higher quality...it could well be that method B gets to X+epsilon faster than A, or worse, A may never reach X+epsilon.

These concepts could be illustrated with a plot showing learning curves for different hyperparameters and/or methods, ideally with real data. This also applies to the case studies presented in Section 3.2.1, all of which currently feel too light, and would be greatly improved with plots and/or tables.

The ViT model configurations section *Degree of parallelism: Scaling Depth (D) vs. Scaling Width (W)* requires additional explanation. Do all of the other hyperparameters remain constant? What is the width for the variable-depth models, and what is the depth for the variable-width models? Furthermore, the authors show a wide range of depth and width values but only discuss one depth-width pair. I think a more accurate and meaningful statement is something to the effect of “When using FLOPs or # params as a metric, increasing width is more beneficial for smaller param/FLOPs budgets, while increasing depth becomes more beneficial for larger param/FLOPs budgets. However, this effect is much less clear when using msec/image as a metric. Performance is similar for when scaling width vs. depth”. Additionally, the effects of scaling width vs. depth could become clearer if the authors sampled more datapoints between D6-D32, and beyond W3072.

At the top of page 8, the authors state “for a similar number of FLOPs, EfficientNet has fewer parameters than RegNet and SwinTransformers.” This statement would be more impactful if FLOPs and # params were directly plotted against each other.

There is at least one company—[MosaicML](www.mosaicml.com)—founded with the goal of improving the efficiency of training deep learning models. Their research seems relevant—they present a [framework for analyzing “efficiency"](https://www.mosaicml.com/blog/methodology). The existence of this company could be taken as evidence in favor of the relevance of the present work, and as such the authors may want to discuss/cite it.

Page 6, *Introducing Sparsity*. The authors write

>...while for unstructured sparsity, it is not possible for the corresponding low-level operations to reach the same efficiency as their dense counterparts on current hardware, where memory access is significantly more expensive than compute.

I think this point could be made more strongly: no common accelerator can extract significant efficiency gains from unstructured sparsity. This could be illustrated with real data showing the change in wall clock time per step as a function of % sparsity for two models, one in which the sparsity is structured and one in which the sparsity is unstructured, ideally for multiple different accelerators.

On page 5, the authors state

>However, counting training cost in terms of steps can be problematic as step count in itself is not a meaningful metric, and can be stretched arbitrarily with optimizers introducing multi-step lookahead schemes.

This statement could also be emphasized or strengthened. I’m having a hard time thinking of a scenario in which step count is informative.

The statement of contributions (2nd to last paragraph in intro) should be more specific. I suggest the authors enumerate (or provide examples of) specific observations and suggestions/recommendations.

The final paragraph of the introduction feels redundant.

The concept of “Pareto-optimality” is used but not explained. This concept should be explained if the present work is attempting to be a go-to reference for understanding efficiency.

The authors detail many different efficiency metrics, which is excellent. However to make room for the additional text and figures that I think this paper needs, I suggest moving to the appendix descriptions of metrics that aren’t used elsewhere in the main text. I also think Section 3.3 (Cost Indicators for Architecture Search) could be expanded and moved to the appendix.

Page 6, “Sharing parameters”: The authors write “When we introduce a form of parameter sharing, it is clear that compared to the same model with no parameter tying...”. Please be consistent in your terminology; pick “parameter sharing” or “parameter tying” and use one exclusively.

Typo: “These differences are not surprising, given that Fig. is...” the Figure is not referenced. I assume it should “Fig. 3”.

summary_of_the_review: The paper is well-motivated and attempts to fulfill a critical need in the field, but feels a bit rushed and incomplete in its current form.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work addresses the problem of measuring and reporting efficiency of machine learning models. The authors show that there is a series of efficiency metrics which are not fully correlated, but, in many of literature, only few of them are reported making confusing claims on production-level usage of proposed models. The main claim is a proposed list of recommendations for reporting of efficiency of new ML models and approaches.
main_review: Strengths:
-	Good overview (systematization) of efficiency metrics and the connection between them
-	Cases of misleading report in recent ML/AI studies 

Weaknesses:
-	It is not clear the direct contribution to the AI/ML industry. The contribution looks as a meta-contribution to the environment in which the ML/AI science behaves. E.g., the metrics are known, the efficiency aspects to take care while considering ML/AI models are also well known. I believe any engineer from the industry takes care of any of the aspects. So, this paper states: “Please, report properly the result when publish”. For instance, in numerical math and algorithms, the differences between memory, parallelism, etc are well known and are not mixed.  For sure, there are works that report only few of aspects, but stating that they sometimes mislead, does not help the industry (this more relates to education?). 
-	If this paper is about how the existing papers have misleading reporting, then it would be nice to have some quantitative analysis (based on some statistics). E.g., it is not clear how many times bad efficiency reporting occurs in the recent bunch of AI/ML papers.

summary_of_the_review: Good paper that describes the issues in results reporting in ML/AI papers. However, given the weaknesses listed above, this work seems to have a notable room for improvement.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper provides a detailed discussion on various cost indicators for “efficiency” of neural network models. It looks at both training and inference processes and argues, based on several examples and arguments, that no single cost metric is sufficient to provide a complete picture of a models’s efficiency, due to the behavior of cost indicators not necessarily being correlated or highly dependent on the platform. It concludes with recommendations to include multiple cost indicators in papers and to clearly state (restrict) efficiency claims.
main_review: The main strength of the submission is that it clarifies and demonstrates the insufficiency of any single cost metric regarding neural network model efficiency. It lists a fairly broad set of cost indicators in Section 2, then focuses on demonstrating issues with the most common ones in the subsequent sections. Three sets of experiments (Fig. 1, 2, 3) show that different cost indicators may not necessarily be correlated.
The paper is overall clear and well written and lays out potential issues with evaluation and reporting of (and resulting claims from) the various indicators precisely.

While the insights provided in this submission may not be novel, this is the first paper to provide a comprehensive argumentation of advantages and disadvantages of cost indicators. There is little research insight beyond this, which, given the “survey-like” nature of the submission, is acceptable. The main weakness of the submission is the brevity of the discussion and suggestions (Section 4), which reduces to a plea to do rigorous science (report multiple metrics, don’t overstate claims).

Minor comments:
- On page 8, first paragraph, a figure reference is missing (presumably to Fig. 3).
- Pg. 9, last paragraph: wholistic -> holistic
summary_of_the_review: The submission contains an assessment of advantages and drawbacks of various model efficiency cost indicators. It comprehensively lays out issues and argues for a more rigorous and precise handling of reported metrics in academic work. Novelty and research insights remain limited, yet publication has good scientific value for means of awareness, reference and citation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper address the problem of model efficiency indicators. The paper brings up the risks of reporting only few efficiency indicators and points out this erroneous common practice.  
The authors investigate how reporting only few cost indicators might lead i) to partial or incorrect conclusions about the model efficiency and  ii) to unfair model comparison. Finally, the authors give recommendations about how to report efficiency indicators.

main_review: The motivation behind the work is clear and valuable to the community. The authors provide relevant reflections and point out common mistakes. However, I believe that the technical content of the paper is not sufficient for acceptance at ICLR. While warning the community is surely important, I would encourage the authors to propose more  practical and detailed guidelines, hence, provide some complete and detailed examples of how model efficiency should be assessed and compared. It would be interesting if such examples were taken from current papers that draw wrong conclusions on model efficiency.

I would suggest to reduce the “Introduction” section (which contains come repetition) and  the “training or inference cost?” section (especially the "Gaming with training time" subsection) and enlarge the discussion section by giving complete and practical example of how model efficiency should be quantified and how two models should be properly compared. As already mentioned, it would be interesting if such examples were taken from current papers that draw wrong conclusions on model efficiency.

The authors should clearly define what “model capacity” is. They mention it, but do not define it properly. 
The authors should clearly define “parameter-matched” and “compute-matched”.

In figures 1 and 2, apart from the parameters of the models that are changed, all the other parameters are set to the default values given by the referenced papers? It should Please specify if this is the case.

What is the hardware used for computing the msec/example in figures 1 and 2? Please specify it.

MINOR COMMENT:
Add a reference to Mesh Tensorflow.
Typo: Figure 3. “Ac curacy”; in Section 3.1 the figure number is missing in the text.

summary_of_the_review: I agree with the authors that it is important to raise awareness and to point out the issues related to the adoption of only a few cost indicators. I believe that the key message of the paper is valuable to the community. However, I believe that the paper does not qualify for acceptance at ICLR because of its limited technical contribution.
review_title: null

Meta-review: Overall the reviewers like the ideas in this paper.  It calls out some of the issues with the current line of thinking in the ML/AI community.  There were some concerns, but overall this paper offers a new way to think about, present, and question efficiency results.  This could be quite infulential.  I think this is interesting enough to warrent publicaiton.
Decision: Accept (Poster)