The Efficiency Misnomer
Authors: Mostafa Dehghani
Yi Tay
Anurag Arnab
Lucas Beyer
Ashish Vaswani
Venue: ICLR 2022 Poster (2022)
Keywords: Efficiency in Machine Learning
FLOPs
Number of Parameters
Throughput

Abstract: Model efficiency is a critical aspect of developing and deploying machine learning models. 
Inference time and latency directly affect the user experience, and some applications have hard requirements. In addition to inference costs, model training also have direct financial and environmental impacts.
Although there are numerous well-established metrics (cost indicators) for measuring model efficiency, researchers and practitioners often assume that these metrics are correlated with each other and report only a few of them.
In this paper, we thoroughly discuss common cost indicators, their advantages and disadvantages, and how they can contradict each other.
We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models. We further present suggestions to improve reporting of efficiency metrics.

Published as a conference paper at ICLR 2022
THE EFFICIENCY MISNOMER
Mostafa Dehghani", Anurag Anurag Arnab", Lucas Beyerr,Ashish Google Research {dehghani, aarnab, lbeyer, avaswani, yitay)?google.com
Ashish Vaswani, Yi Tay"
ABSTRACT
Model efficiency is a critical aspect of developing and deploying machine learning models. Inference time and latency directly affect the user experience, and some ap- plications have hard requirements. In addition to inference costs, model training also have direct financial and environmental impacts. Although there are numerous well- established metries (cost indicators) for measuring model efficiency, rescarchers and practitioners often assume that these metrics are correlated with each other and report only few of them. In this paper, thoroughly discuss common cost indicators, their advantages and disadvantages, and how they can contradict each other. We demon- strate how incomplete reporting of cost indicators lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models. We further present suggestions to improve reporting of efficiency metries.
INTRODUCTION
Aside from model quality, the efficiency (Menghani, 2021) of a model is often an important aspect consider and is commonly used measure the relative utility of different methods. After all, training time spent on accelerators is directly linked to financial costs and environmental impact. Mcanwhile, the speed of a model may be directly linked to user experience. To this end, there have been well-established ways in the literature to assess and report the efficiency of a model such as number of trainable parameters, number of floating-poit operations (FLOPs), and speed/throughput.
While it is commonly assumed that these cost indicators are correlated (e.g., a lower number of pa- rameters would translate to a higher throughput) we show that this might not necessarily be the case. Therefore, incomplete reporting across the spectrum of cost indicators may lead to an incomplete picture of the metrics, advantages and drawbacks of the proposed method. To this end, we show that it may also be possible to, perhaps unknowingly, misrepresent model's 's efficiency by only reporting favorable cost indicators. Moreover, the choice of cost indicators may also result in unfair, incomplete, or partial con- clusions pertaining model comparisons. We refer to this phenomenon the 'efficiency misnomer'.
The overall gist of the efficiency misnomer is that no single cost indicator is sufficient. Incomplete reporting (e.g., showing only FLOPs, or the number of trainable parameters) measure of efficiency can be misleading. For example, a model with low FLOPs may not actually be fast, given that FLOPs does not take into account information such degree of parallelism (e.g., depth, recurrence) or hardware-related details like the cost of a memory access. Despite this, FLOPss has been used as the most common cost indicator in many research papers, especially the recent computer vision literature, to quantify model efficiency (Szegedy et al., 2015; He et al., 2016; Tan and Le, 2019; Feichtenhofer et al., 2019; Fan et al., 2021).
Likewise, the number of trainable parameters (size of the model) despite being commonly used as the de-facto cost indicator in the NLP community (Devlin et al., 2018; Liu et al., 2019; Lan al., 2019) and previously the vision community (Krizhevsky et al., 2012; Simonyan and Zisserman, 2015; Huang et al., 2017; Tan and Le, 2019), can also be misleading when used as a standalone measure of efficiency. Intuitively, a model can have very few trainable parameters and still be very slow, for instance when the parameters are shared among many computational steps (Lan et al., 2019; Dehghani et al., 2018). often be insightful to decide if a model fits in memory, it
While the number of trainable parameters is unlikely to be useful as a standalone cost indicator. That said, it still common practice to parameter- match models to make 'fair' comparisons (Mehta et al., 2020; Lee-Thorp et al., 2021: Tay et al., 2020a; Xue et al., 2021; Wightman et al., 2021), even if one model is in reality slower or faster than another.
" Equal contribution.

---

Published as a conference paper at ICLR 2022
SuperGlue Accuracy [%]]
75
70-
65-
Transformer
Switch Tr
60
Universal Tr
55
100
1000
10
0.3
1.0
3.0 5.0
Million Parameters
TFLOPs
msec/example
Comparison of standard Transformers, Universal Transformers and Switch Transformers in terms of three common cost metries: number of parameters, FLOPs, and throughput. Relative ranking between models is reversed between the two cost indicators. Experiments and the computation of cost metrics were done with Mesh Tensorflow (Shazeer al., 2018) 8), using 64 TPU-V3.
Figure I:
Meanwhile, using throughput/speed as the primary indicator of efficiency can also be problematic since this tightly couples implementation details, hardware optimizations, and infrastructure details (e.g., input pipeline latency) into the picture. Hence, this might not present an apples-to-apples comparison certain methods or worse, across different infrastructures or hardware.
Given that the landscape of research on model architectures is diverse, the relationship between cost indicators may strongly deviate from the norm, and learning how to fairly compare models within the context of efficiency-based cost indicators is crucial. For instance, there seems to be rising trend towards sparse models (Fedus et al., 2021; Riquelme et al., 2021) which usually have an incredibly large number of trainable parameters but maintain the FLOPs and speed of dense models. On the contrary, there are models that are considered lightweight due to their small number of trainable parameters (Lan al., 2019; Dehghani et al., 2018) but, in actual practice, consume similar amounts of compute. Figure 1 shows an example of how the scaling behavior of a model with respect to parameter count can look favorable, while taking the FLOPs or throughput the cost indicator, different model scales much better. This example shows how looking at one metric can be deceptive and cost lot of time and resources, e.g. by choosing the wrong candidate for scaling up.
Besides the fact that different cost indicators capture different aspects, they can be chosen to reflect either the cost of training or the cost at inference time. Both training and inference costs can be crucial, depending on the context. Also, single cost indicator can favor model over another during inference, but not training (or vice versa). For instance, model that shares parameters depth particularly memory- -effrrreeffiientt during inference, but during training, the size of activation that need to be kept for the backward pass is just as large as for a similar model with no parameter sharing.
The overarching conundrum here is that first of all, no single cost indicator captures holistic view that is universally useful to all practitioners or researchers. We show that the trade-offs between cost indicators fall far from the standard assumptions and can be non-trivial to navigate. Moreover, argue that cost indicators that one cares about strongly depend on the applications and setup in which the models supposed be used. For example, for embedded application, inference speed is paramount, while for deployed recommendation systems, training cost can be extremely important as models are constantly being retrained.
The overall contributions of this paper are as follows: We call out the intrinsic difficulty of measuring model efficiency within the context of deep neural networks. We review the most common cost indicators and present the advantages and disadvantages of each and discuss why they might be insufficient as standalone metrie. While obvious, we show examples of how model efficiency might be misrepresented by incomplete reporting of these cost indicators. We characterize this problem, coin the term 'efficiency misnomer's and show that it more prevalent than imagined.
We present experiments where comparing model efficiency strongly depends on the choice of cost indicator, like scenarios where there is parameter sharing, sparsity, or parallelizable operations the model. Moreover, we briefly review some of the current common practices in the literature and discuss how existing work report comparisons of different models and analyze the efficiency of algorithms. Along with the discussion and analyses, we provide that we believe would help researchers and practitioners draw more accurate conclusions about the efficiency of different models.
concrete suggestions and recommendations
2

---

Published as a conference paper at ICLR 2022
2 A PRIMER ON COST INDICATORS
One of the main considerations in designing neural network architectures is quality-cost trade- off (Paleyes et al., 2020). In almost all cases, the more computational budget is given to method, the better the quality of its outcome will be. To account for such a trade-off, several cost indicators are used in the literature of machine learning and its applications to showcase the efficiency of different models. These indicators take different points of view to the computational costs.
FLOPs: A widely used metric as the proxy for the computational cost of a model is the number of floating Doating--pinttin multiplication aad---nd-ddeed addition operations (Johnson, 2018; Kim et al., 2021; Arnab et al., 2021; Tay et al., 2021a;b; Narayanan et al., 2021; Liu et al., 2021). Alternttve FLOPs, the number of multiply-accumulat (MAC') as a single unit of operation is also used in the literature (Johnson, 2018). Reported FL,PPs are usually calculated using theoretical values. Note that theoretical FLOPs ignores practical factors, like which parts of the model can be parallelized.
Number of Parameters: Number of trainable parameters also used as an indirect indicator of computational complexity as well as memory usage (during inference) (Kim et al., 2021; Arnab et al., 2021: Tan and Le, 2019; Liu et al., 2021; Guo et al., 2020; Mahabadi et al., 2021 Houlsby et al., 2019). Many research works that study the scaling law (Kaplan et al., 2020; Hernandez et al., 2021; Tay et al., 2021b), especially in the NLP domain, use the number of parameters as the primary cost indicator (Devlin et al., 2018; Raffel et al., 2019; Liu et al., 2019; Xue et al., 2021).
Speed: Speed is one the most informative indicator for comparing the efficiency of different models (So et al., 2021; Dosovitskiy et al., 2020; Arnab al., 2021; Tay et al., 2021b; Kim et al., 2021; He et al., 2021a: Narayanan et al., 2021; Lagunas et al., 2021; Liu et al., 2021; Tay et al., 2020c;b). In some setups, when measuring speed, the cost of "pipeline" is also taken into account which better reflects the efficiency in real-world scenario. Note that speed strongly depends on hardware and implementation, so keeping the hardware fixed or normalizing based on the amount of resources used is the key for a fair comparison. Speed is often reported in various forms:
Throughput refers to the number of examples (or tokens) that are processed within a specific period of time, e.g.. "examples (or tokens) per second".
Latency usually refers to the inference time (forward pass) of the model given an example or batch of examples, and is usually presented as "seconds per forward pass". The main point about latency is that compared to throughput, it ignores parallelism introduced by batching examples. As an example, when processing a batch of 100 examples in 1 second, throughput is 100 examples per second, while latency is second. Thus, latency is an important factor for real-time systems that require user input.
Wall-clock time/runtime measures the time spent to process fixed set of examples by the model. This is often used to measure the training cost, e.g., total training time up to convergence. Pipeline bubble is the time that computing devices are idle at the start and end of every batch (Narayanan et al., 2021), which indirectly measures the speed of the non-pipeline parts of the process.
Memory Access Cost (MAC) corresponds to the number of memory accesses. It typically makes up a large portion of runtime and is the actual bottleneck when running on modern platforms with strong computational power such as GPUs and TPUs (Ma et al., 2018).
The cost indicators we discussed above present different perspectives efficiency. However, some of these cost indicators may depend on factors that are not inherent to the design of the model, but on the hardware, the model runs (e.g., CPU, GPU, or TPU), the framework that the model is implemented in (e.g., JAX, Py PyTorch, or Flowo or even programming skill. These confounding factors add up the difficulty of comparisons. For instance, theoretical FLOPs provides hardware-inddeenden comparison, however, it does not necessarily translate to the speed of a model as it does not capture the sequential dependencies of operations and memory accesses in the model. On the other hand, throughput and peak memory usage, which could better reflect the model's efficiency in real- world scenario, strongly depend on the hardware and the implementation. Software support can also be a limiting factor in achieving the best possible hardware performance for model. In (Barham and Isard, 2019), the authors make an excellent case for improving the programmability of software stack for modern accelerators to enable a wider class of models.
MAC may also refer to the memory access cost (Ma al., 2018)
3

---

Published as a conference paper at ICLR 2022
In the rest of the paper, mainly focus the number of parameters, FLOPs and speed since they capture most common use cases and are most commonly encountered in the literature. Appendix B presents other cost indicators that could be important depending on the use case.
2.1 TRAINING OR INFERENCE COST?
When talking about costs, we can disentangle the cost of training and the cost of inference. Based on the estimate from NVIDIA (Leopold, 2019) and Amazon (Barr. 2019) as major cloud service providers, 80-90% of the MI. workload is inference processing. Thus, more often, the cost of model during infer- ence is taken as the real cost with the argument that the amortized per-usage cost of training can be really small compared to the inference cost when a model is deployed to be used by many users. However, with the trend of improving the performance of models by scaling up their computational budget and/or train- ing data, well as the fast progress and frequency of the emergence of new models, the training cost can be seen as a relevant concern. Moreover, in some cases, we have to frequently retrain models due to, for instance, privacy requirements, or where new data is being continuously generated like in many recommender systems. The importance of inference efficiency is already clear and here, we will discuss more the importance of training efficiency as well potential issues with reporting training costs.
The are several arguments in favor of the importance of reporting training costs and the need for models that train efficiently. For instance, from research and development point of view, when a class of models is efficient during training, it is more likely to see improvements in their performance, due to ease of iterating on ideas around them. Moreover, when a model shows merit in terms of performance, usually some posthoc modifications can be applied to improve its inference efficiency.
Besides, the memory requirements of different models can be wildly different during training, although their inference memory consumption is comparable. For instance, different optimizers use different amounts of memory on the device, for instance, SGD--Mmenttm (Qian, 1999) vs SAM (Foret et al., 2020). If the success of a model is strongly tied to using an optimizer with high memory cost, it can become a bottleneck during training when we plan to scale that model up compared to the model that uses more memory-efficient optimizer.
Another example is a model that has a high degree of parameter sharing, which could be extremely efficient in terms of inference memory usage, but during training, the size of activation we need to keep for the backward pass is as big as a similar model with no parameter sharing. Activation and their statisties form big portion of memory usage compared to parameter size, decreasing the "number of parameters" by parameter sharing may not lead to any significant decrease in the memory usage during training (Dehghani al., 2018).
Gaming with training time Whilst "training time" can be a great cost indicator, it is also prone to Goodhart's Law: When used the main metric, it can and will be gamed and lose its meaning. First of all, it is difficult to compare architectures with respect to training time, since training time includes the whole "recipe" and different ingredients may be better fits for different architectures. For instance, MobileNet and Ef- ficientNets almost exclusively work well with the RMSProp optimizer (Pham, 2021).
80
longer schadule
75
shorter schadule
70
65
65
60
55
55
50
Second, because the full training recipe is inevitably involved, the only meaningful claim to be made is achieving higher accuracy with smaller total training cost; good example of this is Table in Wightman et al. (2021). The training cost may be in terms of any of the indicators described above. However, counting training cost in terms of steps can be problematic as step count in itself is not a meaningful metric, and can be stretched arbitrarily with optimizers introducing multi- lookahead schemes (Zhang et al., 2019).
45
0.0M
0.5M
1.0M 1.5M 2.0M
Training steps
Figure 2: The learning progress of ResNet-101 x 3 on JFT-300M with short and long schedules. obtained from (Kolesnikov al., 2020). Decaying the learning rate too carly leads to higher performance in lower steps, but the final performance is significantly worse.
Conversely, it may be tempting to claim that a method A performs almost as well as another method with dramatically reduced training cost. Such claim is not valid, method A may not have been optimized for training cost, and may well just be re-tuned with that in mind and outperform method such learning rate and weight decay, can be tuned such
B. For example, training
4

---

Published as a conference paper at ICLR 2022
that they achieve good quality quickly. but then plateau to lower points than in "slower" settings that eventually reach higher quality. This is illustrated by Figure 2, obtained from Kolesnikov et al. (2020). where the "long" training schedule (which is not optimized for training cost) for ResNet- 101x3 achieves the best performance, but the "short" schedule converges significantly faster to a lower final accuracy. Another example, in the context of reinforcement learning, is the Rainbow baseline of Kaiser et al. (2019) which was subsequently re-tuned in van Hasselt et al. (2019), and shown to benefit from significantly longer training schedules too.
Some works, like the MLPerf benchmark (Mattson et al., 2019), aim at decreasing the training cost required to reach a fixed quality
70
weight decay: 1e-5 weight decay: 1e-4
Upstream perfomance
60
X with a fixed model M. While this is a good way of fixing the many moving pieces of training a deep learning model, should
50
X+
40
be noted that due to Goodhart's s law, results do not mean more than "reaching quality X with model M". Specifically, if a method A reaches quality X twice as fast method B while both us- ing model M, this can neither be used to imply anything about their comparison when using model N. nor imply anything on their comparison with the target quality X+c: it could well be that method gets X+e faster than A, or worse, may never reach X+c. Figure 3 shows another concrete example, obtained from (Kolesnikov et al., 2020) where we see faster initial conver- gence of a ResNet model when trained with lower weight decay, which may trick the practitioner into selecting a sub-optimal value, while using higher weight decay leads to a slower convergence, but a better final performance.
:30
20
10
25k
50k
75k
100k
Traini ng steps
Figure 3: The learning progress of ResNet-101 3 JFT-300M with different weight decays, obtained from (Kolesnikov et al., 2020). Lower weight decay leads to acceleration of convergence, while eventually results in an under-performing final model.
A final concern is that, methods which seemingly train faster tend to be used more often and hence get optimized more over time, with the danger of getting stuck in local minimum (Hooker, 2020; Dehgann et al., 2021b).
2.2 POTENTIAL DISAGREEMENT BETWEEN COST INDICATORS
In this section, we will discuss some of the cases which there could be disagreement between some of the cost indicators discussed before.
Sharing parameters When we introduce form of parameter sharing, it is clear that compared to the same model with no parameter sharing, we end up having fewer trainable parameters, while the number of FLOPs or speed stays the same. An example is the comparison of the Universal Transformer (UT) (Dehghani et al., 2018) with vanilla Transformer (Vaswani et al., 2017) Figure 1. UT shares parameters of the model in depth, thus stays close to the frontiers of quality- However, when looking at the FLOPs, to maintain a similar capacity in terms of parameter count, UT requires more computation, which makes not so efficient from the FLOPs point of view.
of parameters.
Introducing Sparsity Sparsity is becoming one of the main ways of both scaling up and down deep neural networks. One needs to distinguish between at least two broad classes of sparse neural networks: structured and unstructured.
Structured sparse models replace large, dense parts of a model by a collection of much smaller, still dense parts. Examples include variants as simple as replacing dense convolutions by grouped (Xie et al., 2017) or separable (Howard et al., 2017) ones, or as complicated as replacing large blocks by many smaller "expert" blocks and routing examples through the best suited ones only (Fedus et al., 2021; Riquelme et al., 2021). The latter, often called Mixture of Experts (MoF), allows growing the capacity in terms of parameter count, while keeping the computational cost small and constant. Figure 1 compares the quality- of Switch Transformer (Fedus et al., 2021), a MoE, to that of vanilla Transformer (Vaswani et al., 2017). While Switch falls short in terms of quality vs parameter count, offers great tra with respect to quality vs FLOPs and speed.
Unstructured sparse models are models where weights of dense operations are made to contain many (almost) zeros (Gale et al., 2019; Evci et al., 2020), which do not contribute to the operation's 's result and thus, in principle, can be skipped. This keeps the overall structure of the original model, while significantly reducing the FL.PSs of the most expensive operations.
5

---

Published as a conference paper at ICLR 2022
ImageNet Accuracy [%]
60
D32
D48
W4096
D32
D48
W4096
D32
D48
W4096
D24
D24
D24 W3372
W3072
W3072
D16
D16
D16
50
W1536
W1536
W1536
W1024
W1024
D8
W1024
D8
D8
40
D6
D6
D6
W768
W768
W768
10
20
30
50
100
0.3 0.5
2
5
0.1
0.2
0.4
0.6
Million Parameters
GFLOPs
msecfimg
Figure 4: Comparison of scaling a small ViT in depth (D, number of encoder blocks) vs scaling in width (W. hidden dimension). Which architecture appears "better", in terms of cost-quality trade-off, changes depending on which indicator is considered. Experiments and the computation of cost metrics were done with Scenic (Dehghani et al., 2021a), using 64 TPU-V3,
Both types of sparse models result in large reductions in theoretical FLOPs, often of several orders of magnitude. However, these do not translate to equally large speed-ups. Difficulties for structured sparse models (especially for the MoF type) include the overhead of routing, and the inability to effectively use batched operations, while for unstructured sparsity, it is not possible for the corresponding low-level operations to reach the same efficiency as their dense counterparts on current hardware, where memory access is significantly more expensive than compute (Gale et al., 2020).
Degree of parallelism: Scaling Depth (D) vs. Scaling Width (W) When scaling up the model size, different strategies can be used. Although these different strategies may have a similar effect in terms of parameter count, and even quality, they can have different effects on the cost in terms of FLOPs and throughput. The most common knobs for scaling up models are changing depth (number of layers) and width (hidden dimension) of models (Tay et al., 2021 To study such an effect, we ran a set of controlled experiments with Vision Transformers (Dosovitskiy al., 2020), where we scale the width by increasing number of heads (while maintaining the hidden dimensions of heads fixed) as well as that of the FFN, and we also scale up the depth, by only increasing the number of encoder blocks. Note that when changing depth or width of the model (see Table 2 in Appendix A for the exact configurations) all other peerepparaeeters are kept fixed based on the default values given by the referenced papers. Figure 4 shows the accuracy of these models with respect to different cost indicators.
In general, we see that considering FI.OPs number parameters as the cost indicator, increasing width is beneficial for smaller budget regions, while increasing depth gives better quality with lower cost when we scale up to higher budget regions. However, this is not necessarily the case when considering speed (msec/img) as the cost indicator. As an example, comparing D48 (a ViT with 48 layers) to W3072, (a ViT with FFN dimension 3072 and QKV dimension 768 split across heads) in terms of FLOPs or number of parameters, they have more or less similar cost, suggesting the D48 as a clear Pareto efficient model?. However, when looking at the speed (msec/img), we observe that W3072 is not necessarily worse than D48, since there are less sequential and more parallelizable operations.
Target platform, and implementation In some cases, certain design in the hardware may lead to discrepancies between the cost indicators. As an example, tensor factorization is used as one of the common techniques to accelerate the matrix multiplication and used for making neural network models more efficient (Zhang et al., 2015b; Jaderberg et al., 2014). Weight decomposition, regardless of its effect on the quality of the model, can reduce the number of FLOPs in neural networks by 75% (Zhang et al., 20) However, it has been shown that using weight decomposition CUDNN that is optimized for 3 3 (He et al., 2017) convolutions and they prefer single large matrix multiplication instead of several small ones. FNet Theorhhor et al., 2021) also shows how certain architectures can have significantly different speed in different hardware (e.g., GPUs and TPUs). When using specifie hardware and compiler, some small design choices significantly affect the cost of model. As an example, Zhai et al. (2021) show that for ViT, using global average pooling instead of a CLS as the representation of the input can significantly reduce the memory cost TPU-V3. This is because the current TPU hardware pads the length dimension of the inputs to multiple of 128, which may result in up to a 50% memory overhead. Another example for how implementation details can affect the efficiency is presented in (Vaswani et al., 2021). where the author discusses how carefully designed implementation of the local neighborhood gathering function for local attention might reduce memory usage while avoiding unnecessary extra computation.
GPUs can be slower as
2 A model counts Pareto efficient model when having lowest cost compared to all the models with similar quality, or highest quality compared to all the models with similar cost.

---

Published as a conference paper at ICLR 2022
VIT (DeiT)
Swin Transformer
RegNet
EfficentNet
84-
Accuracy [%)
82
100
3
10
Million Parameters
GFLOPs
msec/example
Accuracy versus different cost indicators.
Million Parameters
90
20
msselexang secefexample
msse/exxmple
5
20
3
10
30
10
30
20 30
90
GFLOPs
GFLOPs
Million Parameters
(b) Correlation between different cost indicators.
Figure 5: Accuracy and value of different cost indicators for different models on ImageNet dataset. Values for accuracies and cost indicators are obtained from (Steiner et al., 2021; Liu et al., 2021). Cost indicators are based on Py PyTToch implementation of the included models and the throughput is measured on a V100 GPU, using timm (Wightman, 2019).
3 DISCUSSION
The most common use of cost indicators in the MI community is for (1) comparing different models in terms of efficiency using cost indicator, (2) evaluating different models in terms of quality while fixing the cost across models to have a fair comparison, and (3) choosing models with the right trade-off for the context at hand, e.g. in architecture search. While (1) and (2) are just two sides of the same coin, the focus in the first case on finding models with better quality, while the second case is concerned with comparing quality while keeping a certain efficiency metric constant.
3.1 ON PARTIAL CONCLUSIONS FROM INCOMPLETE COMPARISONS
As we showed in Section 2, claiming model is more efficient by reporting better scores on subset of cost indicators can lead to partial, incomplete and possibly biased conclusions.
Figure 5a compares the FL.OSs, parameters and run time for several models versus their accuracy on the image classification task. As seen previously in Figure 1, the relative positions of different model architectures EfficientNets (Tan and Lc, 2019) are on the corent--fronirrrteeo of accuracy-paramtter and accuracy-FLOP trade-offs. On the other hand, the accuracy accuracy-througe -hroughput curve is dominated by Twanfraanformerss (Liu et al., 2021). ) As such, there is no model that is clearly more efficient here and conclusions may differ strongly depending on which efficiency metric is employed.
not consistent the performance metric is varied. For example, one hand,
Note that when comparing variants of a single model family, most of cost indicators correlate. Hence, it could be sufficient consider of these cost indicators However, comparisons using single cost indicator between models with completely different architectures are more complicated. Figure 5b shows the correlation between the cost indicators.
For example, we can see that for a similar number of FLOPs, EfficientNet has fewer parameters than other model families such as RegNet and SwinTransformer. Conversely, for a similar number of FLOPs. EfficientNet has slower throughput (i.e., higher msec/examples) than RegNet and SwinTransformer These differences are not surprising, given that we are comparing transformer- based (Dosovitskiy et al., 2020; Liu et al., 2021) and convolution-basee 2020). Moreover, some of these models are found via architecture search when optimizing for specific cost-indicators (c.g., EfficientNets are optimized for FLOPs).
architectures (Tan and Le, 2019; Radosavovic et al.,
Another observation from Figure is that some variants of transformer based models have the same parameter count, while significantly different GFLOPs, throughput and accuracy. This is due to the
3 Note that this is not always the case, as we have observed in Figure 4 that scaling depth depth of a single model can lead different observations with respect different cost indicators.
7

---

Published as a conference paper at ICLR 2022
change in the model's 's input resolution (e.g., from 224 x 224 to 384 384), leading different number of input tokens to the transformer encoder. This again indicates how a change in the setup could affect some of the cost indicators significantly while barely impacting others.
3.2 MAKING FAIR COMPARISONS THROUGH AN EFFICIENCY LENS
Efficiency metrics and cost indicators are often used to ground comparisons between two or more models or methods (e.g., a proposed model and baselines). By keeping one or more cost indicators fixed, one would often list models side by side in order to make comparisons fair. There are two main strategies here, namely (I) parameter-matcheds comparisons, where configuration of all models are chosen to have similar number of trainable parameters; and/or (2) flop/cpuuueee matched comparisons, where all models have similar computational budget, e.g., configurations are chosen to have similar FLOPs. Whilst there is no straightforward answer to which strategy is the better one, we highlight two case studies and discuss implications and general recommendations.
3.2. THE ISSUES WITH PARAMETER MATCHED COMPARISONS
We delve into some of the potential issues with the parameter-matched comparisons. The gist of many of these issues is that not every parameter is created equal, causing many intricate complexities that might complicate fair comparisons among models. Here we present situations where parameter matched comparison could go wrong.
Token-Free Models Token-free models (Xue et al., 2021; Tay et al., 2021c; Jaegle al., 2021) get rid of the large subword vocabulary by modeling at the character or byte level. A large number of parameters originating from the embedding matrix therefore dropped when transiting to token- models. Hence, it remains an open question of how to fairly compare these class of models with their subword counterparts. ByT5 (Xue et al., 2021) proposed to up-scale the Transformer stack order to compensate for lost parameters in the embedding layer. While this argument was made in the spirit of 'fairnes'' (e.g., comparing both methods at parameter-man matched setup), the up-scaling causes a substanrial reduction in model speed. This is largely because parameters in the embedding matrix typically do not incur much computation while increased parameters in other parts of the Transformer stack (more depth/winn hepth//wdth)) lead to relatively substantial compute cost. What is referred to as base or small size here refers to model that is significantly more computationally costly (in terms of speed, throughput and FLOPs) than their other counterparts and hence the naming alone can be very misleading to practitioners. To make this fair, ideally, the authors would have to present results that are both parameter matched and compute-matee matched.
Gicodde-Decooee vs Decoder-Only Transformers When it comes to pre-trained language models, the choice of backbone architecture, encoder-decodee vs decoder-only, plays an important role in the efficiency of the model with respect to different cost indicators. When comparing these models, we need to know that encoder-decoder model with I. encoder and I. decoder layers has approximately similar amount of parameters as a decoder-only model with 2 layer, while it has half a compute and is twice faster (R (Raffel et al., 2019). This is due a form of model sparsity in encoder-decoder architecture. Thus parameter- match comparison in this setup might be unfair to encoder-decoder models, especially when the speed more of concern than the memory consumption, e.g., when scaling up.
Sparse Models and Mixture-of- Mixture-of-Experts A defining feature of sparse models (Lample et al., 2019; Fedus al., 2021) 1) is that they remain compute- matched while enabling scaling to large number of parameters. Hence, parameter matched comparisons do not make sense for sparse models and parameter-matchhe
sparse models can be seen as unfair method of unnecessarily downplaying the strengths of sparse models. This has been also shown in Figure 1, where comparing a switch transformer with vanilla transformer with the same number of parameters always goes in favor of vanilla transformer. Note that many works in the literature employ compute-matchi matched comparisons for comparing sparse models (Narang et al., 2021: Fedus al., 2021: Lample et al., 2019).
Vision Transformers and Sequence Length Models with a flexible sequence length, such as ViTs when varying patch size, have the opposite property of sparse models: one can instantiate architectures with significantly different computational cost (e.g.. FLOPs and speed) while keeping the parameter count similar. Thus, this type of models should not be compared based parameter count either. Table presents the number of parameters, FLOPs, and inference speed of ViT using different patch sizes. We can see inverse an correlation between parameter size and both FLOPPs and speed. Model
8

---

Published as a conference paper at ICLR 2022
Model
Input sequence length
Million parameters
GFLOPs
msec/example
ViT-B/8
785
86.5
78.54
7.17
ViT-B/10
197
86.6
17.63
1.30
ViT-B/32
50
88.2
4,42
0.39
ViT-B/64
17
95.3
0.93
0.11
Table 1: Parameters FLOPs and speed of ViT- Base with different patch sizes (i.e., 8, 16 16, 32 x 32, and 64 64). Models are fed by input images of size 224 224 3, thus the input sequence length is (224/pater size)²+1 11 (for the CLS token). Numbers in the table are reported using the code in Scenic (Dehghani et al., 2021a) when running on 64 TPU-V3.
with bigger patch sizes have less FLOPs and are faster, while having more parameters, due to the larger patch embedding module.* 4
3.2.2 THE ISSUES WITH COMPUTE MATCHED COMPARISONS
We have previously discussed how parameter matched comparisons may be unfair. This section discusses a case where compute matched comparisons may raise concerns. Consider a scenario where the proposed method achieves compute saving by an architectural design that does not influence model parameters at all (e.g., downsampling sequences). A seemingly fair comparison here would be to take standard model and remove layers or hidden dimensions until both models are compute matched. However, this comparison runs the risk of a baseline model that is handicapped by significantly insufficient model capacity in terms of parameter count and therefore, substantially underperform the proposed method. This is evident in Perceiver I0 (Jaegle et al., 2021) where the baseline BERT is shrunk to mere 20M parameters (compared to 425M in the proposed approach) in a compute matched comparison setup. Moreover, the baseline BERT is also handicapped in terms of depth (6 layers vs 40 layers) which is shown in (Tay et al., 202 lb) to be unfavorable. Note that depth is not taken into account for FLoP-matched comparisons and if the authors were to account for speed-match then the baseline here would be substantially faster than the proposed method. Overall, we do note that making fair compute matched comparisons is clearly nontrivial and challenging problem. However, our recommendation is that when there is just no easy way to make a fair comparison, we encourage authors to make the b effort in finding a best 'matched' setup and show multiple alternatives if possible.
3.3 COST INDICATORS FOR ARCHITECTURE SEARCH
Aside from being used for comparing models, many architecture search studies add cost indicator to the loss function as resource constraint (He et al., 202 and account for efficiency. To this end, these studies mostly use the parameter size (Pham al., 2018). FLOPs (Hsu et al., 2018), memory access cost (MAC) (Ma et al., 2018) 8), or real latency (Tan et al., 2019; Wu et al., 2019). Given that we have shown earlier how cost indicators may disagree with one another or lead to impartial conclusions, we suggest that practitioners place extra caution in choosing cost indicators for architecture search algorithms especially given its computational cost and sensitivity to the selected cost indicator. Here, making assumptions that cost indicators are always interchangeable runs the risk of conducting massive search for finding models that are impractical.
4 SUGGESTIONS AND CONCLUSION
A lot of recent work has focused on comparing different model architectures on the basis of cost indicator like parameter count or FL.OPs. We have demonstrated in this paper that using any cost indicator alone can be misleading, with parameter count being the most problematic one. Oftentimes, parameter count is used to imply "model capacity": however, when varying model architecture in any nontrivial way, this is wrong. Correctly estimating and comparing capacity across model architectures is an open research problem.
Since each indicator stands for something different and comes with its own pros and cons, suggest always reporting and plotting curves using all available cost indicators, and refraining from highlighting results using, just single one. Moreover, given that it is basically impractical to provide holistic report of all cost metrics (for instance, runtime on all possible hardwares) narrowing down the efficiency claims to the exact setup that models are evaluated and avoiding overgeneralized conclusions in comparisons can already provide a much more clear picture to the community.
Using patch size of n. X the number of parameters of the patch embedding module is n xxemb_din c.g., for Vil T-B/ 64, it is 64 x x 768.

---

Published as a conference paper at ICLR 2022
REFERENCES
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.
Vivit: A video vision transformer. arXiv preprint 1.XXiv:2003.1699/ 2021.
Paul Barham and Michael Isard. Machine learning systems are stuck in a rut. In Proceedings the
Workshop on Hot Topics in Operating Systems, pages 177-183,
Jefff Barr. Amazon ec2 htt.a//aw...........oooooooooooa
with-aww-infferntii--hiis wiiissateeeiiiiioooooooooomoeoee 2019, Accessed: 2021-
6-1.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv:2109. 2021.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and łukasz Kaiser. Universal transformers. arXiv preprint
2018.
Mostafa Dehghani, Alexey Gritsenko, Anurag Arnab, Matthias Minderer, and Yi Tay, Scenic: A JAX library for computer vision research and beyond. arXiv preprint (xXiv:2110.11400) 2021a.
Mostafa Dehghani, Yi Tay, Alexey A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. arXiv preprint arXiv:2107.0000 202.. 2021b.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1at:::1800.48800 2018.
Piotr Dollár, Mannat Singh, and Ross Girshick. Fast and accurate model scaling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 924 114-992,42 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 1929, 2020.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Mak- ing all tickets winners. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Re- search, pages 2943-2952. PMLR, 2020. URL http:tp:/ / proceedings. mll. palr.press/vv1/ evci20a. html.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. arXiv preprint arXiv:2104.11227, 2021.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference compuler vision, pages
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.014122 2020.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. CoRR, abs/1902.09974, 2019. URL http: / /aaxivi.vv////////// org/ abs hwtp//912...0 09574.
Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse GPU kernels for deep learning. In Christine Cuicchi, Irene Qualters, and William T. Kramer, editors, Proceedings of the International Conference for High Performance Computing. Networking, Storage and Analysis, SC 2020, Virtual Event/ / Atlanta, Georgia, USA, November 9-19, 2020, page 17. IEEE/ACM, 2020. doi: 10.1 4C441400.000000001 UtLhttpp:: : wobi..rg//ie 1109 /////C444000 2020. 00021.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07440 2020.
10

---

Published as a conference paper at ICLR 2022
Junxian He, Graham Neubig, and Taylor Kirk-Kirkaatiick. Efficient nearest neighbor language models. arXiv preprint arX1v:2009.04221, 202
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual leamning for image recogniton. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
Xin He, Kaiyong Zhao, and Xiaowen Chu, Automl: survey of the state-of-the--at, Knowledge-Based Systems, 212: 106622, 2021b.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01223 2021.
Sara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. arXiv preprint 10Xiv:/9002.00511,200
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 170.:02704. 2017.
Chi-Hung Hsu, Hhan Chang, Jhao-Hong Liang. Hsin-Ping Chou, Chun-Hao Liu, Shih-Chier Chang. Jia- Yu Pan, Yu-Ting Chen, Wei Wei, and Da-Cheng Juan. Monas: Multi-objective neural architecture search using reinforcement learning. arXiv preprint arXiv: 10332, 2018.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference Computer Vision and Puttern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2261-2269. IEEE Computer Society, 2017. doi: 10. 10.1109/CVPR.2017.220 7.34.. URL https: / doi org/ 10. 1111///VPPE 2017 2433.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint
2014.
Andrew Jaegle, Sebastian Borgeaud, Jean- can-BBatisste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14799. 2021.
Jeff Johnson.
Rethinking floating point for deep learning. arXiv preprint arxiv:1811.01721.200 2018.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model-based reinforcement learning for atari. CoRR, abs/ htss/9903.00344, 2019. URL http: /ttp://aaavvv/ea///a//eeeeaaavva org/ abs/ /1903. 00374.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint 200i 000000033,22000
Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz. Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. Scalable and efficient moe training for multitask multilingual models. arXiv preprint 1rXiv:2109.100 10465, 2021.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer VisV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pages 491-507, Springer, 2020.
Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16020-16030, 2021.

---

Published as a conference paper at ICLR 2022
Alex Krizhevsky, llya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In Peter L. Bartlett, Fernando N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Pro- ceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages nwwripss.p....
1106-1114, 2012. URL https: httttps://proceedingg wtt../pp...pppppe/////0..... wccppapprr//1//aaatt.... c399862d3b9ddbp 6084668843332 76844669244664455 Abstract html.
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. Block pruning for faster transformers. arXiv preprint arXiv:2109904838, 2021.
Guillaume Lample, Alexandre Sablayrolles, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Large memory layers with product keys. arXiv preprint arXiv: 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for sulf-super learning of language representations. arXiv preprint arXiv: 1942, 2019.
James Lee-Thorp, Joshua Ainslie. Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.03822 2021.
George Leopold. Aws to offer nvidia's gpus for ai inferencing. www.lppwireecomm/209999/////// hrgprsdss-ssggp---aaaa-d-----eee
2019. Accessed: 2021-6-1.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / 1692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In arXiv preprint arXiv:2103. 14030, 2021.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 1 116-131, 2018.
Practical guidelines for
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. arXiv preprint arXiv:2106.0 04647, 2021a.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint arXiv:2106.00 04489, 2021b.
Peter Mattson, Christine Cheng, Cody Coleman, Greg Diamos, Paulius Micikevicius, David A. Patter- son, Hanlin Tang, Gu- Yeon Wei, Peter Bailis, Victor Bittorf, David Brooks, Dehao Chen, Debojyoti Dutta, Udit Gupta, Kim M. Hazelwood, Andrew Hock, Xinyuan Huang, Bill Jia, Daniel Kang. David Kanter, Naveen Kumar, Jeffery Liao, Guokai Ma, Deepak Narayanan, Tayo Oguntebi, Gennady Pekhimenko, Lillian Pentecost, Vijay Janapa Reddi, Taylor Robie, Tom St. John, Carole-Jean Wu. Lingjic Xu, Cliff Young, and Matei Zaharia. Miperf training benchmark. CoRR, abs//991001 2019. URL htw.thtpp///ee.tt //tp:///ariv..///e org abs/ 1 1910. 01500.
Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Delight: Deep and light-weight transformer. arXiv preprint arXiv:2008,0065 00623, 2020.
Gaurav Menghani. Efficient deep learning: A survey on making deep learning models smaller, faster, and better. arXiv preprint arXiv:2106.0888 08962, 2021.
Sharan Narang, Hyung Won Chung. Yi Tay. William Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel. Noam Shazeer, Zhenzhong Lan, et al. Do transformer modifications transfer across implementations and applications? arXiv preprint arXiv: 102. /972, 2021.
12

---

Published as a conference paper at ICLR 2022
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vi- jay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernuuer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters, arXiv preprint arXiv:210.004477 2021.
Andrei Paleyes, Caoul-Gabriel Urma, and Neil D Lawrence. Challenges in deploying machine learning: a survey of case studies. arXiv preprint arXiv:201 1.09926, (199966,2221 2020.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis- Muii-M Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arx: 104, 10350, 2021.
Hieu Pham. MS Windows NT kernel description. https: http::///weba..civeeeeee org/ web.:o021100552222/httts:: //tt::/twwtttee com/hieupham789 //ttt/ttsst/////////ss// 1371891950110007200000
2021. Accessed: 2021-05-10.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, pages 4095 4104. PMLR, 2018.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12(1): 145-151, 1999.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10428-10436, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv:1910. 10683, 2019.
Carlos Riquelme, Joan Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. arXiv preprint arXiv:2106 05974, 2021.
Or Sharir, Barak Peleg, and Yoav Shoham. The cost of training nlp models: A concise overview. arXiv preprint arXiv:2004.08M0 08900000000 2020.
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414 14- 10423, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL aarxiv.vviv..////vv///aaa/a/aa.a wwt.//voooooooooooooo.ooo.ooo abs 14994.99.00 1556.
David So, Wojeiech Manke, Hanxiao Liu, Zihang Dai. Noam Shazeer, and Quoc V Le. Primer: 2021.
Searching for efficient transformers for language modeling. arXiv preprint
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270 2022.
2021.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv: 2019.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference Computer Vision and Pattern Recognition, 2015, Boston, MA. USA, June 7-12, 2015, pages 1-9. JEEE Computer Society, 2015. doi: 10.1109/CVPR..2055.2998 URL https: / wwwoo.oog///. / 1109 ///// 201 5. 7298594.
13

---

Published as a conference paper at ICLR 2022
Mingxing Tan and Quoc Le. Efficienteet: Rethinking model scaling for convolutional neural networks. In JCML,, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-awar neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820-2828 2019.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models. arXiv preprint arXiv:2005.00743: 2020a.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: benchmark for efficient transformers. arXiv preprint arXiv:20 1.04006, 2020b.
Yi Tay, Mostafa Dehghami. Dara Bahri, and Donald Metzler. Efficient transformers: A survey, arXiv preprint ar arxiv:2009.06732 2020c.
Yi Tay, Mostafa Dehghani, Vamsi Aribandi, Jai Gupta, Philip Pham, Zhen Qin, Dara Bahri, Da-Cheng Juan, and Donald Metzler. Omninet: Omnidirectional representations from transformers. arXiv preprint ar arXiv:2/03.01000 103.01075, 202la..
2021a.
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2 /0686, 2021b.
Yi Tay, Vinh Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient- based subword tokenization. arXiv preprint XrXiv:2 106. 12672, 2021c.
Hado van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforeement lcarning? In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' A'Alhhé-Bu,, Emily B. Fox, and Roman Gamett, editors, Advances in Neu- ral Information Processing Systems 32: Annual Conference Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages neurips httpiaps.ccpppppe/peee wttpp/pppp//////////eep html.
14322-14333, 2019. URL httLtttss//pocceeeee 1b742 5adfl 8b7584454440 887
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 6008, 2017.
infoommtioonrooessinng
Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12894 12904, 2021.
Ross Wightman. Pytorch image models.
com wttp:wwttttaan//////ot...o
sytorch-image-me
s, 2019.
Ross Wightman, Hugo Touvron, and Herve Jegou. Resnet strikes back: An improved training procedure in timm. arXiv preprint 10..x:21100004477 2021.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-awen aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu. and Kaiming He. Aggregated residual transformations for deep neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards a token-free future with pre-trained byte-to-byte models. arXiv preprint arXiv:2 /3626, 2021.
14

---

Published as a conference paper at ICLR 2022
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint 20X1v:2100.04550,,20000 2021.
Michael R. Zhang, James Lucas, Jimmy Ba, and Geoffrey E. Hinton. Lookahead optimizer: steps forward, I step back. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alchh-Buue, Emily B. Fox, and Roman Gamett, editors, Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 9593-9604, 2019. URL https: 90fd4f88 httpps//s: /twpee/ee//occoeeeseeeeeoeee neurips cc/paper/201199 20//2/0019/hhhh/ 9/hash/ html.
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence, 38(10): (0): (00))194319555 2015a.
Xiangyu Zhang, Jianhua Zou, Xiang Ming, Kaiming He, and Jian Sun. Efficient and accurate approximations of nonlinear convolutional networks. In Proceedings the IEEE Conference on Computer Vision and pattern Recognition, pages 1984-1992, 2015b.
15

---

Published as a conference paper at ICLR 2022
Appendix
A EXPERIMENTAL SETUP: SCALING DEPTH VS. SCALING WIDTH
Detailed configurations and scores for the experiments on scaling Vision Transformer, by increasing the depth vs increasing the width of the model.
Configuration
Cost
Quality ImageNot Accuracy
Model
Num layers
FFN FFN
QKV dim
Nam heads
Million Parameters
GFOPs
msse/ting
D6
1024
384
18.89
0.61
0.09
37.5
DB
1024
384
22.44
0.79
0.11
42.4
D16
16
1024
384
36.63
1.52
0.22
51.5
D24
1024
384
50.83
2.25
0.32
55.7
D32
32
1024
384
65.03
2.98
0.43
58.8
D48
1024
0.64
W768
768
192
9.47
0.31
0.11
34.4
K1024
1024
384
24.81
0.92
0.16
45.2
X1536
1536
512
42.51
1.70
0.22
50.3
X3072
3072
101.52
4.44
0.35
58.3
X4D96
12
4096
1024
16
173.10
7.80
0.68
63.3
Table 2: Detailed configuration as well as cost versus quality scores for experiments on scaling depth or width of Vision Transformer.
B ADDITIONAL COST INDICATORS
Depending on the use case, there are other cost indicators that may play a key role in determining the efficiency of model:
Sample efficiency that can be expressed as training data size, e.g. number of data points (or tokens for language models Kaplan et al. (2020)) that a model needs in order to reach a reasonable performance. Being sample efficient depends many factors, like inductive biases of the model or the training curriculum.
Carbon footprint of a model, during training or inference that proxy of the environmental impact. Quantifying this cost directly is not straightforward, but (Strubell et al., 2019) proposed the approximate environmental costs of an ML model as Footprint= (ee + queries x eEinference) x CO2edinceneer KKun KWh, where, ee indicate the electrical energy (Patterson et al., 2021).
Monetary i.e., the expenses for training serving models, which is reported in the form of figures (Sharir al., 2020). As an example, the figures for training can be computed as total-train-time x total number of chips x prices offered by cloud solutions This cost indicator is more often used business documents, than in scientific articles.
number of model activations, proposed by Dollár et al. (2021) as complexity metric. Model activations refers to the number of elements in the output tensors from the building blocks of the model, e.g., output size of convolutional layers ResNet. Dollár activations is reliable predictor for the model runtime and showed that compared to FL.OPs, it is more strongly correlated with the runtime on memory- baadwiddth limited hardware.
(2021) argue that the number of
Memory consumption one the main dimensions of efficiency and has been used cost indicator in various research works (Bondarenko et al., 2021; Dosovitskiy et al., 2020; Kondratyuk al., 2021). Reporting memory footprint often done in form of "peak memory usage", during training that takes into account the memory consumption by the model, optimizer, and the pipeline. Another form that is also common comparing the maximum batch size that fits on a specific device for specific task. Unlike model activation size, memory footprint depends on the hardware and implementation. It has a direct implication of whether a model can fit on a device, which a hard constraint in some cases.
NNte that the prices largely depend the price of electricity.
16


Reviews:
[{"review_id": "zNQFsGx7dw9", "reviewer": "Reviewer_AyyT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strength of the submission is that it clarifies and demonstrates the insufficiency of any single cost metric regarding neural network model efficiency. It lists a fairly broad set of cost indicators in Section 2, then focuses on demonstrating issues with the most common ones in the subsequent sections. Three sets of experiments (Fig. 1, 2, 3) show that different cost indicators may not necessarily be correlated.\nThe paper is overall clear and well written and lays out potential issues with evaluation and reporting of (and resulting claims from) the various indicators precisely.\n\nWhile the insights provided in this submission may not be novel, this is the first paper to provide a comprehensive argumentation of advantages and disadvantages of cost indicators. There is little research insight beyond this, which, given the “survey-like” nature of the submission, is acceptable. The main weakness of the submission is the brevity of the discussion and suggestions (Section 4), which reduces to a plea to do rigorous science (report multiple metrics, don’t overstate claims).\n\nMinor comments:\n- On page 8, first paragraph, a figure reference is missing (presumably to Fig. 3).\n- Pg. 9, last paragraph: wholistic -> holistic", "summary_of_the_paper": "The paper provides a detailed discussion on various cost indicators for “efficiency” of neural network models. It looks at both training and inference processes and argues, based on several examples and arguments, that no single cost metric is sufficient to provide a complete picture of a models’s efficiency, due to the behavior of cost indicators not necessarily being correlated or highly dependent on the platform. It concludes with recommendations to include multiple cost indicators in papers and to clearly state (restrict) efficiency claims.", "summary_of_the_review": "The submission contains an assessment of advantages and drawbacks of various model efficiency cost indicators. It comprehensively lays out issues and argues for a more rigorous and precise handling of reported metrics in academic work. Novelty and research insights remain limited, yet publication has good scientific value for means of awareness, reference and citation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TnaaupxcPK", "reviewer": "Reviewer_GqNw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation for this paper is excellent. The measurement and discussion of “efficiency” in contemporary ML research is typically simplistic, and sometimes misleading. The ML community is in dire need of work that explains relevant quantities and considerations, provides concrete examples of pitfalls and poor practices, and emphasizes the importance of context and nuance when assessing “efficiency”. Detractors might accuse this paper of not containing sufficient original research for publication in a venue such as this one. My response to this criticism is that ML research would be in a better place if researchers took more time to reflect on the state of research and identify ways in which our practices and scholarship could be made more robust and rigorous. That is what this paper does.\n\nUnfortunately, I don’t think this paper fully realizes its potential. It could be clearer and more focused; it reads like it was written quickly. Many of the examples don’t feel sufficiently explained, and some don’t feel well-integrated. In its current state, I am not confident that it is suitable for publication. However, I am optimistic that it could be after revision.\n\n\n**Feedback**\n\nMy primary criticism is that the paper is insufficiently concrete. It would be substantially more impactful with more real examples, ideally with accompanying plots and/or tables. For example, on page 5 the authors state\n\n>Some hyper-parameters, such as learning-rate and weight-decay, can be tuned so as to reach good quality quickly but then plateau lower than in “slower” settings that eventually reach higher quality...it could well be that method B gets to X+epsilon faster than A, or worse, A may never reach X+epsilon.\n\nThese concepts could be illustrated with a plot showing learning curves for different hyperparameters and/or methods, ideally with real data. This also applies to the case studies presented in Section 3.2.1, all of which currently feel too light, and would be greatly improved with plots and/or tables.\n\nThe ViT model configurations section *Degree of parallelism: Scaling Depth (D) vs. Scaling Width (W)* requires additional explanation. Do all of the other hyperparameters remain constant? What is the width for the variable-depth models, and what is the depth for the variable-width models? Furthermore, the authors show a wide range of depth and width values but only discuss one depth-width pair. I think a more accurate and meaningful statement is something to the effect of “When using FLOPs or # params as a metric, increasing width is more beneficial for smaller param/FLOPs budgets, while increasing depth becomes more beneficial for larger param/FLOPs budgets. However, this effect is much less clear when using msec/image as a metric. Performance is similar for when scaling width vs. depth”. Additionally, the effects of scaling width vs. depth could become clearer if the authors sampled more datapoints between D6-D32, and beyond W3072.\n\nAt the top of page 8, the authors state “for a similar number of FLOPs, EfficientNet has fewer parameters than RegNet and SwinTransformers.” This statement would be more impactful if FLOPs and # params were directly plotted against each other.\n\nThere is at least one company—[MosaicML](www.mosaicml.com)—founded with the goal of improving the efficiency of training deep learning models. Their research seems relevant—they present a [framework for analyzing “efficiency\"](https://www.mosaicml.com/blog/methodology). The existence of this company could be taken as evidence in favor of the relevance of the present work, and as such the authors may want to discuss/cite it.\n\nPage 6, *Introducing Sparsity*. The authors write\n\n>...while for unstructured sparsity, it is not possible for the corresponding low-level operations to reach the same efficiency as their dense counterparts on current hardware, where memory access is significantly more expensive than compute.\n\nI think this point could be made more strongly: no common accelerator can extract significant efficiency gains from unstructured sparsity. This could be illustrated with real data showing the change in wall clock time per step as a function of % sparsity for two models, one in which the sparsity is structured and one in which the sparsity is unstructured, ideally for multiple different accelerators.\n\nOn page 5, the authors state\n\n>However, counting training cost in terms of steps can be problematic as step count in itself is not a meaningful metric, and can be stretched arbitrarily with optimizers introducing multi-step lookahead schemes.\n\nThis statement could also be emphasized or strengthened. I’m having a hard time thinking of a scenario in which step count is informative.\n\nThe statement of contributions (2nd to last paragraph in intro) should be more specific. I suggest the authors enumerate (or provide examples of) specific observations and suggestions/recommendations.\n\nThe final paragraph of the introduction feels redundant.\n\nThe concept of “Pareto-optimality” is used but not explained. This concept should be explained if the present work is attempting to be a go-to reference for understanding efficiency.\n\nThe authors detail many different efficiency metrics, which is excellent. However to make room for the additional text and figures that I think this paper needs, I suggest moving to the appendix descriptions of metrics that aren’t used elsewhere in the main text. I also think Section 3.3 (Cost Indicators for Architecture Search) could be expanded and moved to the appendix.\n\nPage 6, “Sharing parameters”: The authors write “When we introduce a form of parameter sharing, it is clear that compared to the same model with no parameter tying...”. Please be consistent in your terminology; pick “parameter sharing” or “parameter tying” and use one exclusively.\n\nTypo: “These differences are not surprising, given that Fig. is...” the Figure is not referenced. I assume it should “Fig. 3”.\n", "summary_of_the_paper": "This paper emphasizes the importance of context and nuance when discussing “efficiency”. The authors explain different efficiency metrics and the distinction between training and inference efficiency. They then illustrate how assessments of the “efficiency” of different models can be misleading or even contradictory depending on factors such as the choice of efficiency metric, baseline, architecture, hardware, optimizer, and other experimental design choices. The authors finish with a few suggestions for best practices.", "summary_of_the_review": "The paper is well-motivated and attempts to fulfill a critical need in the field, but feels a bit rushed and incomplete in its current form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KHmLhAgjJ2T", "reviewer": "Reviewer_6vKw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-\tGood overview (systematization) of efficiency metrics and the connection between them\n-\tCases of misleading report in recent ML/AI studies \n\nWeaknesses:\n-\tIt is not clear the direct contribution to the AI/ML industry. The contribution looks as a meta-contribution to the environment in which the ML/AI science behaves. E.g., the metrics are known, the efficiency aspects to take care while considering ML/AI models are also well known. I believe any engineer from the industry takes care of any of the aspects. So, this paper states: “Please, report properly the result when publish”. For instance, in numerical math and algorithms, the differences between memory, parallelism, etc are well known and are not mixed.  For sure, there are works that report only few of aspects, but stating that they sometimes mislead, does not help the industry (this more relates to education?). \n-\tIf this paper is about how the existing papers have misleading reporting, then it would be nice to have some quantitative analysis (based on some statistics). E.g., it is not clear how many times bad efficiency reporting occurs in the recent bunch of AI/ML papers.\n", "summary_of_the_paper": "This work addresses the problem of measuring and reporting efficiency of machine learning models. The authors show that there is a series of efficiency metrics which are not fully correlated, but, in many of literature, only few of them are reported making confusing claims on production-level usage of proposed models. The main claim is a proposed list of recommendations for reporting of efficiency of new ML models and approaches.", "summary_of_the_review": "Good paper that describes the issues in results reporting in ML/AI papers. However, given the weaknesses listed above, this work seems to have a notable room for improvement. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5lLDIyRnRTG", "reviewer": "Reviewer_sinW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation behind the work is clear and valuable to the community. The authors provide relevant reflections and point out common mistakes. However, I believe that the technical content of the paper is not sufficient for acceptance at ICLR. While warning the community is surely important, I would encourage the authors to propose more  practical and detailed guidelines, hence, provide some complete and detailed examples of how model efficiency should be assessed and compared. It would be interesting if such examples were taken from current papers that draw wrong conclusions on model efficiency.\n\nI would suggest to reduce the “Introduction” section (which contains come repetition) and  the “training or inference cost?” section (especially the \"Gaming with training time\" subsection) and enlarge the discussion section by giving complete and practical example of how model efficiency should be quantified and how two models should be properly compared. As already mentioned, it would be interesting if such examples were taken from current papers that draw wrong conclusions on model efficiency.\n\nThe authors should clearly define what “model capacity” is. They mention it, but do not define it properly. \nThe authors should clearly define “parameter-matched” and “compute-matched”.\n\nIn figures 1 and 2, apart from the parameters of the models that are changed, all the other parameters are set to the default values given by the referenced papers? It should Please specify if this is the case.\n\nWhat is the hardware used for computing the msec/example in figures 1 and 2? Please specify it.\n\nMINOR COMMENT:\nAdd a reference to Mesh Tensorflow.\nTypo: Figure 3. “Ac curacy”; in Section 3.1 the figure number is missing in the text.\n", "summary_of_the_paper": "The paper address the problem of model efficiency indicators. The paper brings up the risks of reporting only few efficiency indicators and points out this erroneous common practice.  \nThe authors investigate how reporting only few cost indicators might lead i) to partial or incorrect conclusions about the model efficiency and  ii) to unfair model comparison. Finally, the authors give recommendations about how to report efficiency indicators.\n", "summary_of_the_review": "I agree with the authors that it is important to raise awareness and to point out the issues related to the adoption of only a few cost indicators. I believe that the key message of the paper is valuable to the community. However, I believe that the paper does not qualify for acceptance at ICLR because of its limited technical contribution. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall the reviewers like the ideas in this paper.  It calls out some of the issues with the current line of thinking in the ML/AI community.  There were some concerns, but overall this paper offers a new way to think about, present, and question efficiency results.  This could be quite infulential.  I think this is interesting enough to warrent publicaiton.
Decision: Accept (Poster)