The Efficiency Misnomer

Model efficiency is a critical aspect of developing and deploying machine learning models . Inference time and latency directly affect the user experience , and some applications have hard requirements . In addition to inference costs , model training also have direct financial and environmental impacts . Although there are numerous wellestablished metrics ( cost indicators ) for measuring model efficiency , researchers and practitioners often assume that these metrics are correlated with each other and report only few of them . In this paper , we thoroughly discuss common cost indicators , their advantages and disadvantages , and how they can contradict each other . We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models . We further present suggestions to improve reporting of efficiency metrics . 1 INTRODUCTION . Aside from model quality , the efficiency ( Menghani , 2021 ) of a model is often an important aspect to consider and is commonly used to measure the relative utility of different methods . After all , training time spent on accelerators is directly linked to financial costs and environmental impact . Meanwhile , the speed of a model may be directly linked to user experience . To this end , there have been well-established ways in the literature to assess and report the efficiency of a model such as number of trainable parameters , number of floating-point operations ( FLOPs ) , and speed/throughput . While it is commonly assumed that these cost indicators are correlated ( e.g. , a lower number of parameters would translate to a higher throughput ) we show that this might not necessarily be the case . Therefore , incomplete reporting across the spectrum of cost indicators may lead to an incomplete picture of the metrics , advantages and drawbacks of the proposed method . To this end , we show that it may also be possible to , perhaps unknowingly , misrepresent a model ’ s efficiency by only reporting favorable cost indicators . Moreover , the choice of cost indicators may also result in unfair , incomplete , or partial conclusions pertaining to model comparisons . We refer to this phenomenon as the ‘ efficiency misnomer ’ . The overall gist of the efficiency misnomer is that no single cost indicator is sufficient . Incomplete reporting ( e.g. , showing only FLOPs , or the number of trainable parameters ) as a measure of efficiency can be misleading . For example , a model with low FLOPs may not actually be fast , given that FLOPs does not take into account information such as degree of parallelism ( e.g. , depth , recurrence ) or hardware-related details like the cost of a memory access . Despite this , FLOPs has been used as the most common cost indicator in many research papers , especially in the recent computer vision literature , to quantify model efficiency ( Szegedy et al. , 2015 ; He et al. , 2016 ; Tan and Le , 2019 ; Feichtenhofer et al. , 2019 ; Fan et al. , 2021 ) . Likewise , the number of trainable parameters ( size of the model ) despite being commonly used as the de-facto cost indicator in the NLP community ( Devlin et al. , 2018 ; Liu et al. , 2019 ; Lan et al. , 2019 ) and previously the vision community ( Krizhevsky et al. , 2012 ; Simonyan and Zisserman , 2015 ; Huang et al. , 2017 ; Tan and Le , 2019 ) , can also be misleading when used as a standalone measure of efficiency . Intuitively , a model can have very few trainable parameters and still be very slow , for instance when the parameters are shared among many computational steps ( Lan et al. , 2019 ; Dehghani et al. , 2018 ) . While the number of trainable parameters can often be insightful to decide if a model fits in memory , it is unlikely to be useful as a standalone cost indicator . That said , it is still common practice to parametermatch models to make ‘ fair ’ comparisons ( Mehta et al. , 2020 ; Lee-Thorp et al. , 2021 ; Tay et al. , 2020a ; Xue et al. , 2021 ; Wightman et al. , 2021 ) , even if one model is in reality slower or faster than another . Meanwhile , using throughput/speed as the primary indicator of efficiency can also be problematic - since this tightly couples implementation details , hardware optimizations , and infrastructure details ( e.g. , input pipeline latency ) into the picture . Hence , this might not present an apples-to-apples comparison of certain methods or worse , across different infrastructures or hardware . Given that the landscape of research on model architectures is diverse , the relationship between cost indicators may strongly deviate from the norm , and learning how to fairly compare models within the context of efficiency-based cost indicators is crucial . For instance , there seems to be a rising trend towards sparse models ( Fedus et al. , 2021 ; Riquelme et al. , 2021 ) which usually have an incredibly large number of trainable parameters but maintain the FLOPs and speed of dense models . On the contrary , there are models that are considered lightweight due to their small number of trainable parameters ( Lan et al. , 2019 ; Dehghani et al. , 2018 ) but , in actual practice , consume similar amounts of compute . Figure 1 shows an example of how the scaling behavior of a model with respect to parameter count can look favorable , while taking the FLOPs or throughput as the cost indicator , different model scales much better . This example shows how looking at one metric can be deceptive and cost a lot of time and resources , e.g . by choosing the wrong candidate for scaling up . Besides the fact that different cost indicators capture different aspects , they can be chosen to reflect either the cost of training or the cost at inference time . Both training and inference costs can be crucial , depending on the context . Also , a single cost indicator can favor a model over another during inference , but not training ( or vice versa ) . For instance , a model that shares parameters in depth is particularly memory-efficient during inference , but during training , the size of activation that need to be kept for the backward pass is just as large as for a similar model with no parameter sharing . The overarching conundrum here is that first of all , no single cost indicator captures a holistic view that is universally useful to all practitioners or researchers . We show that the trade-offs between cost indicators fall far from the standard assumptions and can be non-trivial to navigate . Moreover , we argue that cost indicators that one cares about strongly depend on the applications and setup in which the models are supposed to be used . For example , for an embedded application , inference speed is paramount , while for deployed recommendation systems , training cost can be extremely important as models are constantly being retrained . The overall contributions of this paper are as follows : We call out the intrinsic difficulty of measuring model efficiency within the context of deep neural networks . We review the most common cost indicators and present the advantages and disadvantages of each and discuss why they might be insufficient as a standalone metric . While obvious , we show examples of how model efficiency might be misrepresented by incomplete reporting of these cost indicators . We characterize this problem , coin the term ‘ efficiency misnomer ‘ , and show that it is more prevalent than imagined . We present experiments where comparing model efficiency strongly depends on the choice of cost indicator , like scenarios where there is parameter sharing , sparsity , or parallelizable operations in the model . Moreover , we briefly review some of the current common practices in the literature and discuss how existing work report comparisons of different models and analyze the efficiency of algorithms . Along with the discussion and analyses , we provide some concrete suggestions and recommendations that we believe would help researchers and practitioners draw more accurate conclusions about the efficiency of different models . 2 A PRIMER ON COST INDICATORS . One of the main considerations in designing neural network architectures is quality-cost tradeoff ( Paleyes et al. , 2020 ) . In almost all cases , the more computational budget is given to a method , the better the quality of its outcome will be . To account for such a trade-off , several cost indicators are used in the literature of machine learning and its applications to showcase the efficiency of different models . These indicators take different points of view to the computational costs . FLOPs : A widely used metric as the proxy for the computational cost of a model is the number of floating-point multiplication-and-addition operations ( Johnson , 2018 ; Kim et al. , 2021 ; Arnab et al. , 2021 ; Tay et al. , 2021a ; b ; Narayanan et al. , 2021 ; Liu et al. , 2021 ) . Alternative to FLOPs , the number of multiply-accumulate ( MAC1 ) as a single unit of operation is also used in the literature ( Johnson , 2018 ) . Reported FLOPs are usually calculated using theoretical values . Note that theoretical FLOPs ignores practical factors , like which parts of the model can be parallelized . Number of Parameters : Number of trainable parameters is also used as an indirect indicator of computational complexity as well as memory usage ( during inference ) ( Kim et al. , 2021 ; Arnab et al. , 2021 ; Tan and Le , 2019 ; Liu et al. , 2021 ; Guo et al. , 2020 ; Mahabadi et al. , 2021a ; b ; Houlsby et al. , 2019 ) . Many research works that study the scaling law ( Kaplan et al. , 2020 ; Hernandez et al. , 2021 ; Tay et al. , 2021b ) , especially in the NLP domain , use the number of parameters as the primary cost indicator ( Devlin et al. , 2018 ; Raffel et al. , 2019 ; Liu et al. , 2019 ; Xue et al. , 2021 ) . Speed : Speed is one the most informative indicator for comparing the efficiency of different models ( So et al. , 2021 ; Dosovitskiy et al. , 2020 ; Arnab et al. , 2021 ; Tay et al. , 2021b ; Kim et al. , 2021 ; He et al. , 2021a ; Narayanan et al. , 2021 ; Lagunas et al. , 2021 ; Liu et al. , 2021 ; Tay et al. , 2020c ; b ) . In some setups , when measuring speed , the cost of “ pipeline ” is also taken into account which better reflects the efficiency in a real-world scenario . Note that speed strongly depends on hardware and implementation , so keeping the hardware fixed or normalizing based on the amount of resources used is the key for a fair comparison . Speed is often reported in various forms : • Throughput refers to the number of examples ( or tokens ) that are processed within a specific period of time , e.g. , “ examples ( or tokens ) per second ” . • Latency usually refers to the inference time ( forward pass ) of the model given an example or batch of examples , and is usually presented as “ seconds per forward pass ” . The main point about latency is that compared to throughput , it ignores parallelism introduced by batching examples . As an example , when processing a batch of 100 examples in 1 second , throughput is 100 examples per second , while latency is 1 second . Thus , latency is an important factor for real-time systems that require user input . • Wall-clock time/runtime measures the time spent to process a fixed set of examples by the model . This is often used to measure the training cost , e.g. , total training time up to convergence . • Pipeline bubble is the time that computing devices are idle at the start and end of every batch ( Narayanan et al. , 2021 ) , which indirectly measures the speed of the non-pipeline parts of the process . • Memory Access Cost ( MAC ) corresponds to the number of memory accesses . It typically makes up a large portion of runtime and is the actual bottleneck when running on modern platforms with strong computational power such as GPUs and TPUs ( Ma et al. , 2018 ) . The cost indicators we discussed above present different perspectives on efficiency . However , some of these cost indicators may depend on factors that are not inherent to the design of the model , but on the hardware , the model runs on ( e.g. , CPU , GPU , or TPU ) , the framework that the model is implemented in ( e.g. , JAX , PyTorch , or TensorFlow ) , or even programming skill . These confounding factors add up to the difficulty of comparisons . For instance , theoretical FLOPs provides a hardware-independent comparison , however , it does not necessarily translate to the speed of a model as it does not capture the sequential dependencies of operations and memory accesses in the model . On the other hand , throughput and peak memory usage , which could better reflect the model ’ s efficiency in a real-world scenario , strongly depend on the hardware and the implementation . Software support can also be a limiting factor in achieving the best possible hardware performance for a model . In ( Barham and Isard , 2019 ) , the authors make an excellent case for improving the programmability of software stack for modern accelerators to enable a wider class of models . 1MAC may also refer to the memory access cost ( Ma et al. , 2018 ) In the rest of the paper , we mainly focus on the number of parameters , FLOPs and speed since they capture most common use cases and are most commonly encountered in the literature . Appendix B presents other cost indicators that could be important depending on the use case .

Summary: This work addresses the problem of measuring and reporting efficiency of machine learning models. The authors show that there is a series of efficiency metrics which are not fully correlated, but, in many of literature, only few of them are reported making confusing claims on production-level usage of proposed models. The main claim is a proposed list of recommendations for reporting of efficiency of new ML models and approaches.