The Efficiency Misnomer

Model efficiency is a critical aspect of developing and deploying machine learning models . Inference time and latency directly affect the user experience , and some applications have hard requirements . In addition to inference costs , model training also have direct financial and environmental impacts . Although there are numerous wellestablished metrics ( cost indicators ) for measuring model efficiency , researchers and practitioners often assume that these metrics are correlated with each other and report only few of them . In this paper , we thoroughly discuss common cost indicators , their advantages and disadvantages , and how they can contradict each other . We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models . We further present suggestions to improve reporting of efficiency metrics . 1 INTRODUCTION . Aside from model quality , the efficiency ( Menghani , 2021 ) of a model is often an important aspect to consider and is commonly used to measure the relative utility of different methods . After all , training time spent on accelerators is directly linked to financial costs and environmental impact . Meanwhile , the speed of a model may be directly linked to user experience . To this end , there have been well-established ways in the literature to assess and report the efficiency of a model such as number of trainable parameters , number of floating-point operations ( FLOPs ) , and speed/throughput . While it is commonly assumed that these cost indicators are correlated ( e.g. , a lower number of parameters would translate to a higher throughput ) we show that this might not necessarily be the case . Therefore , incomplete reporting across the spectrum of cost indicators may lead to an incomplete picture of the metrics , advantages and drawbacks of the proposed method . To this end , we show that it may also be possible to , perhaps unknowingly , misrepresent a model ’ s efficiency by only reporting favorable cost indicators . Moreover , the choice of cost indicators may also result in unfair , incomplete , or partial conclusions pertaining to model comparisons . We refer to this phenomenon as the ‘ efficiency misnomer ’ . The overall gist of the efficiency misnomer is that no single cost indicator is sufficient . Incomplete reporting ( e.g. , showing only FLOPs , or the number of trainable parameters ) as a measure of efficiency can be misleading . For example , a model with low FLOPs may not actually be fast , given that FLOPs does not take into account information such as degree of parallelism ( e.g. , depth , recurrence ) or hardware-related details like the cost of a memory access . Despite this , FLOPs has been used as the most common cost indicator in many research papers , especially in the recent computer vision literature , to quantify model efficiency ( Szegedy et al. , 2015 ; He et al. , 2016 ; Tan and Le , 2019 ; Feichtenhofer et al. , 2019 ; Fan et al. , 2021 ) . Likewise , the number of trainable parameters ( size of the model ) despite being commonly used as the de-facto cost indicator in the NLP community ( Devlin et al. , 2018 ; Liu et al. , 2019 ; Lan et al. , 2019 ) and previously the vision community ( Krizhevsky et al. , 2012 ; Simonyan and Zisserman , 2015 ; Huang et al. , 2017 ; Tan and Le , 2019 ) , can also be misleading when used as a standalone measure of efficiency . Intuitively , a model can have very few trainable parameters and still be very slow , for instance when the parameters are shared among many computational steps ( Lan et al. , 2019 ; Dehghani et al. , 2018 ) . While the number of trainable parameters can often be insightful to decide if a model fits in memory , it is unlikely to be useful as a standalone cost indicator . That said , it is still common practice to parametermatch models to make ‘ fair ’ comparisons ( Mehta et al. , 2020 ; Lee-Thorp et al. , 2021 ; Tay et al. , 2020a ; Xue et al. , 2021 ; Wightman et al. , 2021 ) , even if one model is in reality slower or faster than another . Meanwhile , using throughput/speed as the primary indicator of efficiency can also be problematic - since this tightly couples implementation details , hardware optimizations , and infrastructure details ( e.g. , input pipeline latency ) into the picture . Hence , this might not present an apples-to-apples comparison of certain methods or worse , across different infrastructures or hardware . Given that the landscape of research on model architectures is diverse , the relationship between cost indicators may strongly deviate from the norm , and learning how to fairly compare models within the context of efficiency-based cost indicators is crucial . For instance , there seems to be a rising trend towards sparse models ( Fedus et al. , 2021 ; Riquelme et al. , 2021 ) which usually have an incredibly large number of trainable parameters but maintain the FLOPs and speed of dense models . On the contrary , there are models that are considered lightweight due to their small number of trainable parameters ( Lan et al. , 2019 ; Dehghani et al. , 2018 ) but , in actual practice , consume similar amounts of compute . Figure 1 shows an example of how the scaling behavior of a model with respect to parameter count can look favorable , while taking the FLOPs or throughput as the cost indicator , different model scales much better . This example shows how looking at one metric can be deceptive and cost a lot of time and resources , e.g . by choosing the wrong candidate for scaling up . Besides the fact that different cost indicators capture different aspects , they can be chosen to reflect either the cost of training or the cost at inference time . Both training and inference costs can be crucial , depending on the context . Also , a single cost indicator can favor a model over another during inference , but not training ( or vice versa ) . For instance , a model that shares parameters in depth is particularly memory-efficient during inference , but during training , the size of activation that need to be kept for the backward pass is just as large as for a similar model with no parameter sharing . The overarching conundrum here is that first of all , no single cost indicator captures a holistic view that is universally useful to all practitioners or researchers . We show that the trade-offs between cost indicators fall far from the standard assumptions and can be non-trivial to navigate . Moreover , we argue that cost indicators that one cares about strongly depend on the applications and setup in which the models are supposed to be used . For example , for an embedded application , inference speed is paramount , while for deployed recommendation systems , training cost can be extremely important as models are constantly being retrained . The overall contributions of this paper are as follows : We call out the intrinsic difficulty of measuring model efficiency within the context of deep neural networks . We review the most common cost indicators and present the advantages and disadvantages of each and discuss why they might be insufficient as a standalone metric . While obvious , we show examples of how model efficiency might be misrepresented by incomplete reporting of these cost indicators . We characterize this problem , coin the term ‘ efficiency misnomer ‘ , and show that it is more prevalent than imagined . We present experiments where comparing model efficiency strongly depends on the choice of cost indicator , like scenarios where there is parameter sharing , sparsity , or parallelizable operations in the model . Moreover , we briefly review some of the current common practices in the literature and discuss how existing work report comparisons of different models and analyze the efficiency of algorithms . Along with the discussion and analyses , we provide some concrete suggestions and recommendations that we believe would help researchers and practitioners draw more accurate conclusions about the efficiency of different models . 2 A PRIMER ON COST INDICATORS . One of the main considerations in designing neural network architectures is quality-cost tradeoff ( Paleyes et al. , 2020 ) . In almost all cases , the more computational budget is given to a method , the better the quality of its outcome will be . To account for such a trade-off , several cost indicators are used in the literature of machine learning and its applications to showcase the efficiency of different models . These indicators take different points of view to the computational costs . FLOPs : A widely used metric as the proxy for the computational cost of a model is the number of floating-point multiplication-and-addition operations ( Johnson , 2018 ; Kim et al. , 2021 ; Arnab et al. , 2021 ; Tay et al. , 2021a ; b ; Narayanan et al. , 2021 ; Liu et al. , 2021 ) . Alternative to FLOPs , the number of multiply-accumulate ( MAC1 ) as a single unit of operation is also used in the literature ( Johnson , 2018 ) . Reported FLOPs are usually calculated using theoretical values . Note that theoretical FLOPs ignores practical factors , like which parts of the model can be parallelized . Number of Parameters : Number of trainable parameters is also used as an indirect indicator of computational complexity as well as memory usage ( during inference ) ( Kim et al. , 2021 ; Arnab et al. , 2021 ; Tan and Le , 2019 ; Liu et al. , 2021 ; Guo et al. , 2020 ; Mahabadi et al. , 2021a ; b ; Houlsby et al. , 2019 ) . Many research works that study the scaling law ( Kaplan et al. , 2020 ; Hernandez et al. , 2021 ; Tay et al. , 2021b ) , especially in the NLP domain , use the number of parameters as the primary cost indicator ( Devlin et al. , 2018 ; Raffel et al. , 2019 ; Liu et al. , 2019 ; Xue et al. , 2021 ) . Speed : Speed is one the most informative indicator for comparing the efficiency of different models ( So et al. , 2021 ; Dosovitskiy et al. , 2020 ; Arnab et al. , 2021 ; Tay et al. , 2021b ; Kim et al. , 2021 ; He et al. , 2021a ; Narayanan et al. , 2021 ; Lagunas et al. , 2021 ; Liu et al. , 2021 ; Tay et al. , 2020c ; b ) . In some setups , when measuring speed , the cost of “ pipeline ” is also taken into account which better reflects the efficiency in a real-world scenario . Note that speed strongly depends on hardware and implementation , so keeping the hardware fixed or normalizing based on the amount of resources used is the key for a fair comparison . Speed is often reported in various forms : • Throughput refers to the number of examples ( or tokens ) that are processed within a specific period of time , e.g. , “ examples ( or tokens ) per second ” . • Latency usually refers to the inference time ( forward pass ) of the model given an example or batch of examples , and is usually presented as “ seconds per forward pass ” . The main point about latency is that compared to throughput , it ignores parallelism introduced by batching examples . As an example , when processing a batch of 100 examples in 1 second , throughput is 100 examples per second , while latency is 1 second . Thus , latency is an important factor for real-time systems that require user input . • Wall-clock time/runtime measures the time spent to process a fixed set of examples by the model . This is often used to measure the training cost , e.g. , total training time up to convergence . • Pipeline bubble is the time that computing devices are idle at the start and end of every batch ( Narayanan et al. , 2021 ) , which indirectly measures the speed of the non-pipeline parts of the process . • Memory Access Cost ( MAC ) corresponds to the number of memory accesses . It typically makes up a large portion of runtime and is the actual bottleneck when running on modern platforms with strong computational power such as GPUs and TPUs ( Ma et al. , 2018 ) . The cost indicators we discussed above present different perspectives on efficiency . However , some of these cost indicators may depend on factors that are not inherent to the design of the model , but on the hardware , the model runs on ( e.g. , CPU , GPU , or TPU ) , the framework that the model is implemented in ( e.g. , JAX , PyTorch , or TensorFlow ) , or even programming skill . These confounding factors add up to the difficulty of comparisons . For instance , theoretical FLOPs provides a hardware-independent comparison , however , it does not necessarily translate to the speed of a model as it does not capture the sequential dependencies of operations and memory accesses in the model . On the other hand , throughput and peak memory usage , which could better reflect the model ’ s efficiency in a real-world scenario , strongly depend on the hardware and the implementation . Software support can also be a limiting factor in achieving the best possible hardware performance for a model . In ( Barham and Isard , 2019 ) , the authors make an excellent case for improving the programmability of software stack for modern accelerators to enable a wider class of models . 1MAC may also refer to the memory access cost ( Ma et al. , 2018 ) In the rest of the paper , we mainly focus on the number of parameters , FLOPs and speed since they capture most common use cases and are most commonly encountered in the literature . Appendix B presents other cost indicators that could be important depending on the use case .

Summary: The paper address the problem of model efficiency indicators. The paper brings up the risks of reporting only few efficiency indicators and points out this erroneous common practice.   The authors investigate how reporting only few cost indicators might lead i) to partial or incorrect conclusions about the model efficiency and  ii) to unfair model comparison. Finally, the authors give recommendations about how to report efficiency indicators.