The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment
Authors: Jared Fernandez
Jacob Kahn
Clara Na
Yonatan Bisk
Emma Strubell
Venue: EMNLP 2023 Main (2023)
Keywords: efficiency
latency
inference
TL;DR: Software frameworks impose bottlenecks to efficiency improvements on NLP model architectures.

Abstract: Increased focus on the computational efficiency of systems in natural language processing has motivated the design of efficient model architectures and improvements to underlying hardware accelerators.  However, the resulting increases in computational throughput and reductions in floating point operations have not directly translated to improvements in wall-clock inference latency.  We demonstrate that  these discrepancies can be largely attributed to bottlenecks introduced by deep learning frameworks. We denote this phenomena as the framework tax, and observe that the disparity is growing as hardware speed increases over time. In this work, we examine this phenomena through a series of case studies analyzing the effects of model design decisions, framework paradigms, and hardware platforms on total model latency.  Based on our findings, we provide actionable recommendations to researchers and practitioners aimed at narrowing the gap between efficient NLP model research and practice.

The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment
Jared Fernandez!
Jacob Kahn Clara Na Yonatan Bisk'
Emma Strubell
LLanguage Technologies Institute, Carnegie Mellon University
Allen Institute for Artificial Intelligence SFAIR
Cjaredfern, Strubel1J@cmu.. edu, {csna, ybisk)@cs. cmu. edu, jacobkahnefb.com com
Abstract
RT-BBaee
Increased on the computational efficiency of NLP systems has motivated the design of ef- ficient model architectures and improvements underlying hardware accelerators. How- ever, the resulting increases in computational throughput and reductions in floating point op- erations have directly translated to improve- ments in wall-clock inference latency. We demonstrate that these discrepancies can be largely attributed to bottlenecks introduced by deep learning frameworks. We denote this phenomenon as the framework tax, and hard-
Punnel transformer
Albert
0.05
MoblleBERT
Latency (s)
Sqq SqueeceGERT
0.04
MACS are not proportional
0.03
Latency
0.02
0.01
0.6
MACS
1e10
that the disparity is growing ware speed increases over time. In this work, we examine this phenomenon through a se- ries of case studies analyzing the effects of model design decisions, framework paradigms, and hardware platforms on total model latency. Code is available at https://ee thub. com/ JaredFern/F eatfeerr///aaork---
Figure 1: Latency function of model's multiply cumulate operations (MACs) on an Nvidia 2080ti GPU, Expected relationships do not hold; model complexity and hardware capabilities fail to predict latency due to hamnewor--boundednee
to make up 80 to 90% of ML cloud computing demand (Barr, 2019; Leopold, 2019). In these set- tings, metrics of model speed such as latency and throughput essential for inference workloads that are subject to real wall-clock time constraints such as real-time natural language generation and automated speech recognition (Reddi et al., 2020).
1 Introduction
Natural language processing systems have bene- fited from improvements in performance driven by scaling training data and number of model parameters (Kaplan et al., 2020; Alabdulmohsin et al., 2022; Tay et al., 2021, 2022). However, the accompanying increases in computation raise con- cemss to the efficiency of these systems due to associated environmental costs of development and deployment (Schwartz al., 2020; Strubell al., 2019).
These concerns have motivated research in de- signing more efficient neural network model archi- tectures and faster hardware accelerators. In the past five years alone, the number of papers that mention the terms efficient or efficiency in top ma- chine learning venues has grown by over 2.5x and even more so at venues in natural language process- ing, increasing by 8.3x in the same span.' This has spurred innovations in the design of efficient neu- ral network architectures for language aiming reduce the number of trainable model parameters, floating point or (MACs) (landola et al., 2020; Dai et al., 2020; Sun
In particular, efficiency is especially important in inference settings where models are used re- peatedly and at scale, in contrast to training which poses a single upfront computational cost. For ex- ample, Meta reports that inference workloads make up 70% of their Al power consumption, with the remaining 30% due training and development (Wu et al., 2022), while Google attributes 60% of their ML energy consumption to inference (Pat- terson et al., 2022). Inference also estimated
operations
BBased on publications at MI. (ICLR, ICML, NeurlPS) and NL.P conferences (ACL, EMNLP) between 2017 and 2022

---

al., 2020). Over the same period, GPU hardware accelerators have seen similar performance gains with the number of floating point operations per second (FLOPS) growing by over 175%.
lutional neural newtork models in eager execution Py just-in-ting compiled TorchScript, and ahead-of-time compiled ONNX runtime using a CUDA execution provider. We perform our study across seven different GPUs from the Pascal, Tur- ing, and Ampere Nvidia GPU microarchitectures. Based on our findings, provide series of recommendations for NLP researchers and practi- tioners presented through collection of case stud- ies. Among these, we recommend usage of static or ahead-of-time inference runtimes when batch sizes are small as they can substantially reduce framework overhead. Alternatively, when using frameworks for inference, recommend increasing model width or batch size at no cost to latency and take advantage of per- formance gains associated with increased model capacity (Zagoruyko and Komodakis, 2016). For example, hidden dimensions in self-attention and fully connected layers can be doubled to increase model capacity without affecting latency, imply- ing that model designers have an extra degree of freedom often overlooked when designing around parameters or FLOPs. We hope that our analy- sis and recommendations will help bridge the gap between efficient NLP research and practice.
Despite this progress, the supposed gains offered by higher performance hardware and more effi- cient models are often not realized in inference settings, where wall-clock model speed has not re- liably improved, as seen in Figure 1. We show that this misalignment is primarily attributable to overhead incurred by deep learning frameworks used to implement and execute models. In the past, the execution of large neural networks common to natural language processing has been assumed to be compute- houuded by computationally intensive tensor operations (Li et al., 2020). However, as the speed of hardware increases, the overhead in- troduced by the deep learning frameworks used to implement and deploy these models is no longer negligible and imposes bottlenecks on inference. In this work, we conduct systematic investiga- tion in which we show that improvements to neural network architectures and hardware have not trans- lated into reductions in inference latency. We show that this breakdown is largely due to overhead in- troduced by deep learning frameworks. We refer to this phenomenon as the framework tax, and show that it exists across all deep learning framework paradigms (e.g. eager execution, just-in-time, and ahead--f-tinee compilation).
eager execution-
2 Related Work
2.1 Efficiency Metries & Cost Indicators
Previous efforts report efficiency often utilize proxy metrics for amount of computation, such the number of floating point (FLOPs) or multiply- accumulate (MACs) operations (Schwartz et al., 2020). Similarly, number of trainable parameters is a frequently reported as proxy for memory utiliza- tion (Lan et al., 2019). Unfortunately, these proxy metrics are often not predictive of realworld effi- ciency. For example, total FLOPs does not account for the varying extent which different operations can be parallelized and techniques such weight tying can reduce parameter counts without reduc- ing the amount of required computation (Lan al., 2019). From the perspective of device utilization, hardware and model FLOPs utilization (Chowdh- cry et al., 2022) are reported as the ratio between observed FLOPs per second and a hardware plat- forms peak theoretical FLOPs.
At small batch sizes and sequence lengths, we show that fixed framework overhead dominates in- ference time leading proxy measurements of model efficiency, such as MACs and parameter count, to breakdown as predictors of inference latency. Fur- thermore, we note that existing efforts to improve model efficiency by designing lower FLOP model architectures and faster GPU kernels do not reduce latency, when total execution is bound by fixed-cost framework overhead. Moreover, we note that as hardware performance increases, NLP systems will become increasingly framework-bound at larger batch sizes.
An exhaustive comparison of the rapidly grow- ing space of models and hardware platforms is out of scope, so we instead identify the most popular model components, inference environments, and hardware accelerators for testing (i.e. isolating the cases most likely to mislead a practitioner). We analyze the performance of transformer and convo-
Previous works examining the relationship be- tween efficiency measures showed that different cost indicators do not correlate well with each other during neural network training (Dehghani et al.,
PPer theoretical FLOPS of Nvidia GPUs (2017-2022).

---

2021). In particular, it has been hypothesized that discrepancies between FLOPs and wallelock infer- ence latency primarily are primarily compute bounded by kernel execution or memory-bound by data movement as opposed to framework bottle- necks (Langerman et al., 2020). These previous works have largely focused convolutional neural networks (CNNs) in computer vision. We extend these analyses to the inference setting and study transformer-basee neural networks for NLP, and show that FL.OP-based proxy metries breakdown for due to additional performance bottlenecks in- troduced by deep learning frameworks.
stract away the underlying frameworks, compilers. backends, and hardware platforms. While general improvements in hardware and software kernels may lead to improvements across all models, it has been argued that solely focusing on performance optimization of a limited set of model architec- tures and runtimes may lead to overspecialization (Hooker, 2021).
Previous analysis of the computational proper- ties of hardware accelerators has largely focused on the training setting in which larger kernels and batch sizes hide framework overhead that emerges in the inference setting (Wang et al., 2020c; Zhu et al., 2020, 2018). Other analyses of end-to-end systems analyses has primarily focused on domain- specific applications in reinforcement learning and recommendation systems (Gleeson et al., 2021; Lin et al., 2022), where simulation and memory access dominate execution time. Additionally, these prior efforts are restricted to small sets of reference mod- els and have not directly examined the relationship between model architectures and platforms.
2.2 Efficient Model Design
Desire to develop computationally efficient models for language processing has led to the development of a variety of model architectures that achieve comparable task performance under fixed FLOP budgets. For example, compression of input se- quences and intermediate representations has been used to reduce the computational cost of long text sequences (Dai et al., 2020; Goyal et al., 2020) and distillation has been used to reduce model size (Sanh et al., 2019; Hou et al., 2020). Other work has sought to design efficient model archi- tectures by developing low-FLOP substitutes for standard, dense self-attention and convolution op- erations (landola et al., 2016; Zhang et al., 2018; Sandler al., 2018; Sun et al., 2020; Xiong et al., 2021; Wang et al., 2020b).
Preliminaries
3.1 Neural Network Frameworks
To take advantage of massively parallcl hardware accelerators, inference with variable length text and speech sequences are padded to fixed length tensors that are processed with neural network frameworks. These frameworks provide implementations and APIs for tensor operations, gradient calculation, and construction of neural network computational graphs. Frameworks generally fall into the follow- ing design paradigms (Kahn et al., 2022):
Additionally, direct efficiency metrics, such as wall-clock latency and energy usage have been in- corporated into the objectives of neural architec- ture search (NAS) and AutoML methods (Wu et al., 2019; Tan et al., 2019; Wang et al., 2020a). Man- ual inspection of the learned models shows that NAS often implicitly learns to take advantage of supported parallelism, learning wider architectures on GPU devices and deeper models on CPU de- vices (Cai et al., 2018; Tan et al., 2019; Sandler et al., 2018). However, it is often impossible to control for the hardware systems used for collect- ing these metrics leading to conclusions that may not generalize across deployment settings.
Eager Execution: The computational graph is constructed from a series of operations that are executed as soon as called from an interpreter. Ex- amples include: PyTorch (Paszke et al., 2019) and Chainer (Tokui et al., 2015).
Deferred Execution:
A series of operations are defined and executed on sample data to generate a dataflow graph that can then be just-in-time (JiT) compiled. Examples include: TorchScript, Jax (Bradbury et al., 2018). Theano (Al-Rfou et al., 2016), Caffe (Jia et al., 2014).
2.3 Platform Performance Analysis
Efforts establish common benchmarks leverage reference models and hardware platforms with tar- get latency or accuracy (Reddi al., 2020; Zhou et al., 2020). Although these efforts have led to improvement in end-to-end latency, they often ab-
The computational graph is pre-defined, compiled, and executed inside a specialized run- time; allowing for aggressive, global ahead-of-timn (AoT) compiler optimizations. Examples include:
Static:

---

Lekency
0.030
CUDA Kernel Execution Time
Framework Overhead
GPU
fare
0.025
Time
Latency
0.020
CPU
0.035
GPU
0.030
Kerwll
Konnel 3
Figure 2: Profiles where execution is frumework bound by CPU kernel dispatch operations (above) and compute- bound by GPU kernel operations (below). Small com- pute kernels occur in inference at lower batch sizes. Boxes with dashed lines represent framewoork overkead.
60
Batch Size
Figure 3: Framework overhead of BERT-Base in Py- Torch for various batch sizes on RTX-8000. Al- though small batch sizes require fewer FLOPs, these re- ductions do not translate to speedups in latency. Frame- work overhead is substantial at low batch sizes but only small constant at large batch sizes.
ONNX Runtime, TensorFlow 1.0, MXNet, Ten- SorRT, and TVM (Chen et al., 2015, 2018).
Eager execution frameworks compute each layer separate operations which each incur additional overhead from CPU kernel dispatches. To accel- erate execution, deferred execution and static in- ference frameworks compile the neural network's computational graph to combine multiple opera- tions together (e.g. fusing attention layers and GELU activations) or remove unnecessary com- putation (i.c. scalar folding). Removal of kernel launch operations can reduce the memory footprint and result in more efficient kernels, thus decreasing overall framework overhead.
See Figure (Aminabadi et al., 2022).
Kernel serialization with optimizations such CUDA Graphs remove the overhead from multiple kernel dispatches by capturing and replaying the entire computational graph as a single operation. However, kernel serialization requires that models are graph safe (i.e. static shapes and static con- trol flow). This can pose challenges for NLP sys- tems which often leverage dynamic computational graphs to deal with variable length sequences, parse tree depths, and batch sizes (Looks et al.. 2017).
While deferred and static frameworks are com- monly used deployment settings, the NLP re- search community relics heavily on cager mode frameworks during the development of new mod- els for their case of use. This further exacerbates the community divide, where models are designed under different assumptions than deployed.
When the execution of GPU kernel computation is largely blocked by CPU framework operations such as kernel dispatches, the model's execution becomes framework-boune boond.. In this setting, latency is constant regardless of batch size number of MACs computed. For settings where latency is de- pendent the execution of computational kernels and data movement, models are compute-bound.
3.2 Framework Overhead
Deep learning frameworks asynchronously dis- patch computation for execution on highly paral- Jelized hardware accelerators, as shown Figure 2. For sufficiently large compute kernels, such as those during training, models achieve near max- imum GPU utilization measured as the differ- ence between total execution time and active GPU time (Zhu et al., 2018). 1).. However, during inference, smaller input sizes lead to suboptimal GPU utiliza- tion the rapid executing kernels do not saturate the fixed cost framework overhead incurred from CPU operations such as kernel launch, graph con- struction, control flow, and device synchronization;
Experiments
We evaluate models frameworks from each of the major paradigms: eager execution PyTorch, de- ferred execution TorchScript, and statically com- piled ONNX Runtime with a CUDA backend.
We use PyTorch 1.12.1 with CUDA 11.6 and Python 3.8.13. We use ONNX Runtime 1.7.0 with CUDA 11.1.1 and cuDNN v8.0.4.3. Baseline exper- on compute node with Nvidia RTX-800 800000 GPU and an Intel Xcon E5-2630 CPU with 32 GB of DDRAM memory.
iments
We measure latency, GPU utilization over a

---

Reskiet: (p32))
RosNet
BERT (fp32)
BERT
BERT
10
10
PyTorch Sortpp
OlVtO Surtiime
10-2
10-3
10-1
10
10"
1.0
Batch
Batch
Batch
Sequance Longth
Figure 4: Latency vs. (batch size and sequence lengths) for baseline models in FP16 and FP32 on RTX-8000. Framework boundedness exists for all models at small input sizes where framework overhead dominates runtime and results in constant latency regardless of input size. Framework overhead is most prominent in smaller models executed half precision on slower frameworks.
range of batch sizes to capture common inference use cases stated in (Reddi et al., 2020): sin- gle example inferences is common in streaming on-device applications, such as autocomplete and automatic speech recognition; large batch inference is typical in offline server settings.
to overtake fixed framework overhead. As re- sult, half precision inference is framework bound for larger batch sizes. For inference with larger compute- batch sizes, latency observes ex- pected speedups from using half precision.
Although models based on convolutions (ResNet-50) and self-attention (BERT-Base operations both exhibit framework- bound behavior, they transition to compute boundedness at different batch sizes. The difference model behavior can be attributed to differences in the rate at which compute kernels overtake CPU dispatch operations. For the well-optimized operations (Conv2Ds and GEMMs) that make up ResNet-50 and BERT, the time per FLOP is reasonably consistent.
We simulate text data by randomly generating token sequences of length 128, as commonly used in sentence classification tasks (Liu et al., 2019; Izsak al., 2021). We report averaged metrics from 100 forward passes after 10 warm-up passes initialize model weights and data. GPU core and memory utilization are measured with the Nvidia Nsight Compute and Nvidia Management Library.
We select BERT-Base (Devlin et al., 2018) and ResNet-50 (He et al., 2016) representative mod- els for encoder-only and convolutional neural net- work architectures commonly used for sentence and image classification tasks (Reddi et al., 2020; Janapa Reddi et al., 2022). We evaluate model ar- chitectures for natural language generation, speech, and vision and show that they all observe frame- work bound behavior in Appendices C, D and E.
4.1 Framework Design Decisions
In Figure 4, we observe that frameworks from all execution paradigms exhibit framework bound be- haviors. However, deferred execution TorchScript and static ONNX Runtime, which support com- putational graph compilation (e.g. operator fu- sion), exhibit less framework overhead and provide speedups over eager PyTorch. These increases are especially pronounced at low batch sizes where inference is framework-bound For batch size 1, TorchScript and ONNX provide an average FP16 speed up of 34.16% and 71.38% over PyToorc, respectively. As batch sizes increase and models become compute bound, there is minimal differ- ence in latency across frameworks the majority of execution time spent on kernel execution.
Analysis The number of FLOPs required for model inference scales with the input batch size and sequence length. As such, one would expect that latency scales accordingly as well. However, seen in Figure models exhibits framework boundedness for both small sequence lengths and batch sizes, where latency is constant regardless of input size.
Computation with mixed and half precision (FP16) often increases training throughput over single precision (FP32), we observe that frame- work overhead results in latency bottlenecks re- gardless of the precision during inference. As half precision computation is faster due to reduced data movement, GPU kernel execution time takes longer
Additionally, consider both static, serial- ized CUDA graphs and Py BetterTransformer framework optimizations in Figure BetterTrans- former provides speedups through additional kernel fusion and sparse tensor operations that take advan- tage of sparse sequence lengths and padding tokens

---

10a1
buge
IaLpper Boc
Torch
Nested Tensors
10
10
CUDA Graphs
Latency (s)
Latency
10-1
10
10th 10"
10"
10'
MACS
Batch Size
Figure 6: Comparison of latency for BERT variants that scale model width and depth. Increases in model depth add more framework overhead, whereas increases in model width lead to faster transitions to compute boundedness.
10'
10
107
Batch Size
Figure 5: Different framework optimizations lead to la- tency improvements in different regimes for CUDA Graph kernel serialization reduces launch over- head in the framework bound regime, whereas sparse computation reduces latency at larger batch sizes.
We compare our baseline models to variants that scale both model width and depth. We examine 12-layer BERT-Base against its 6-layer DistilBERT (Sanh et al., 2019) variant and experiment across parameterized BERT models, varying the number of encoder layers as well as width of their fully connected and self-attention layers.
to remove redundant computation.
To construct sparse inputs, we simulate sam- ples by generating variable length sequences and padding to the maximum sequence length of 128. Sentences are randomly generated according the sequence length distribution of the Penn Treebank (Taylor et al., 2003), with an average length of 20.92 and a standard deviation of 10.18 tokens.
Analysis When scaling model depth, we observe that latency increases both the framework- and compute-bouee bound regimes as each added layer op- eration requires an additional CPU-GPU dispatch. Deeper model variants have larger fixed latency in the framework-bourd regime as seen in Figure 6. Gounter-intuitivele wider model variations see increase in latency at low batch sizes. As model execution is framework bound, total runtime is stant despite wider operations requiring more float- ing point operations. Instead, increased per-layer kernel execution time causes these wider models to become compute-bound at lower batch sizes. In the compute- bound regime, latency scales more rapidly with batch size for wide models.
Analysis Utilization of CUDA Graphs substan- tially reduce latency at low batch sizes when infer- ence is bounded by framework overhead from ker- nel launches. However, larger batch sizes, nested tensor operations can leverage sparsity in padded variable sequence lengths to provide substantial latency reductions when inference compute- bounded.
4.2 Model Design Decisions
We examine variety of common model architec- ture design decisions and investigate their align- ment with commonly reported efficiency proxies and empirically observed latency.
4.2.2
Downsampling and Hierarchical Pooling
10
10
BEffT-Bas
Funnel
4.2.1 Scaling Model Depth & Width
Latency
Assumption Scaling the dimensionality and number of hidden layers is commonly used in NLP and computer vision as means to explore tradcoffs between model performance and com- putational requirements (He et al., 2016; Touvron al., 2021; Zhou al., 2021). Recent work has shown that model end-task performance scales dif- ferently along each axis (Tay et al., 2021, 2022; Nguyen et al., 2021).
10
10
10"
10'
30'"
10'
10'
MACS
Batch Size
Figure 7: Comparison BERT and Funnel Transformer latency. Despite using fewer total MAC operations. Funnel is slower than BERT for inference to the introduction of additional intermediate pooling layers.

---

Assumption Self-attention layers in Transformer architectures are notoriously computationally ex- pensive as their complexity scales quadratically with input sequence length. To reduce this com- putational bottleneck, researchers have developed efficient transformer architectures that seek to re- duce sequence lengths via methods such as down- sampling, low rank approximations, and locality sensitive hashing (Dai et al., 2020; Kitaev al., 2020; Wang et al., 2020b; Xiong al., 2021).
Assumption Model architectures that target de- ployment on edge devices, such as mobile phones. are often designed to decrease the total number of FLOPs or MACs under the assumption that such reductions translate to decreased latency. Towards this end, operations such as grouped convolutions (Zhang et al., 2018; landola et al., 2020), inverted residual bottlenecks (MBConvs) (Sandler et al., 201 8; Sun et al., 2020), and squeeze and excitation layers (landola et al., 2016) have all been proposed as substitutes for dense convolution, self-attention, and linear operations. However, in practice these operations often lack the highly optimized frame- work and hardware support developed for more standard operations and as a result exhibit higher per-flop latency and poor memory utilization.
We examine the performance of the Funnel Transformer which applies average pooling to per- form sequence length reduction every 4 layers. The model achieves comparable accuracy to BERT on downstream tasks while requiring 42% fewer total MAC operations through sequence length reduc- tion. This model achieves similar downstream task performance to BERT-Base and trains 33% faster based on wall-clock time.
We examine this assumption with models that use grouped convolutions in SqueezeBERT (lan- dola et al., 2020) inverted bottleneck layers and Mobile BERT (Sun al., 2020).
Analysis While Funnel Transformer reduces to- tal FLOPs and obtains substantial speedups in large- scale training. this speedup does not translate to increased speed of inference as seen in Figure In practice, the average pooling layers used to perform sequence length reductions add additional opera- tions to the computation graph and increase the model's framework overhead. At low batch sizes, Funnel Transformer is framework bound at much higher latency than BERT, and remains slower even at larger batch sizes. While some architectural inno- vations decrease the total number of model FLOPs, some approaches increase the size of tensor opera- tor graphs (c.g. vis-a-vis additional layers) which can ultimately increase inference latency.
Analysis To achieve comparable accuracy on downstream language tasks with low-FLOP opera- tions, efficient BERT variants require much deeper model architectures which results in much higher fixed framework overhead seen in Figure 8. Ad- ditionally, these models exhibit worse FLOP per second due to poor memory utilization compared to conventional dense linear and convolution op- erations. These operations can lead to slowdowns in deployment settings where depthwise and point- wise convolutions may have limited hardware and framework backend support.
4.3 Hardware Considerations
10.
Nvitia 10
4.2.3 Efficient Mobile Architectures
Nvidia 205071
3030
Nvidia V100
Latency (s)
A100
2CfT-ous MoDRetERT SQUeCOREENT
10
Latency
10a2
10-2
10P
10L
102
10f
103
10a
Batch Size
Batch Size
10
20
1030
2011
2012
107
10a
10a
MACS
Batch Size
Figure 9: Framework overhead occurs acToSs genera- tions GPU hardware, with increasing prominence as hardware speeds increase with newer generations.
Figure 8: Latency of Transformer models using efficient variations of convolution and self-attention operations. All of the variants observe lower latency at large batch sizes, but have worse FLOP utilization. Efficient Trans- former variants are slower than BERT at small batch sizes due to the introduction of more layers.
In Figure 9, we observe that framework framework-bounddd behaviors during inference across multiple gener-
PPointwise convolutions are often memory bound: https: //pytorch. org/ttttrraata tutorialss/eeeaaae htt////pee///cpppeeeeeeee//// www.pppctttpnngg..ce htnl
bbund::

---

ations of consumer, workstation, and datacenter Nvidia GPUs. As the speed of the accelcrator in- creases, the relative execution speed of the com- pute kernels decreases while the total framework overhead due to CPU kernel dispatch operations remains constant. We observe that GPUs that lack Tensor Core support for half precision operations, such as the 1080Ti, are notably slower and less framework-boundd than newer GPUs. Conversely, this leads faster GPUs, such as the RTX 3090 and the A100, to remain framework bound at larger batch sizes for both ResNet-50 and BERT. These observations indicate that framework bounds on model execution will continue to worsen as hard- ware improves unless deep learning frameworks are commensurately improved. For example, BERT- Base is not framework bound for older 1080Ti GPUs but is on newer 3090 GPUs.
Dynamic computational graphs can be faster for input sentences with variable lengths. Dy- namic computational graphs can leverage input sparsity to reduce latency when processing vari- able length text. For example, PyTorch with sparse tensor optimizations reduces the latency of static CUDA graphs by 80.56% at batch size 128 when processing sparse inputs.
At large input sizes, framework overhead from graph operations is negligible. For batch sizes larger than 16, we find that there is minimal la- tency difference across models, inference runtimes, and frameworks. In the compute-bound regime, number of FLOPs is still poor latency predictor due to variable execution time for different opera- tions. For example, efficient mobile architectures that depend on inverted-residual layers mem- ory inefficient and are much slower per-FLOP than standard convolution and linear layers.
Framework boundedness is mainly caused by bottlenecks due to CPU dispatch operations. As result, the fixed latency for framework- mod- els is product the entire system configuration, dependent on hardware specifications such as CPU and memory speeds. In contrast, the latency of compute-bound models is mainly determined by the properties of kernels: operation type, kernel size, and their supported GPU FLOPs per second with faster GPUs yielding faster execution. Ad- ditional details on hardware system configurations are provided in Appendix A.
For framework-boumn models, model depth is a reasonable proxy for latency. Number of float- ing point operations a poor indicator of latency in a framework-boue is generally constant and tied to framework over- heads and the size of the computational graph. In framework bound models, the size of the computa- tional graph is related model depth.
setting, as total runtime
Estimations of latency for models deployed in production settings must account for their target framework and hardware platform. Model de- velopment frequently occurs using eager execution research frameworks. However, deployment often occurs in inference runtimes and on mobile devices or specialized hardware. This misalignment can mislead the development of "efficient" models and result in claimed gains that do not translate real- world deployment settings. As such, researchers should be clear in specifying the setting of their "ef- ficiency" gains, such as the target frameworks and hardware platform, when developing new methods. For example, techniques such as hardware-aware neural architecture search which leverage direct latency measures must also control for framework choices to account for this mismatch.
5 Discussion
Computational graph optimizations and com- pilation improve latency. Removal of host lan- guage dependencies and graph optimizations pro- vides substantial speedups over eager frameworks for inference low batch sizes. However, fea- ture completeness for operators and control flow varies across graph optimizers and compilers. For example, the FNet architecture (Lee-Thorp et al., 2021) relies on FFTs as a deterministic swap-in for self-attention. FFT operations are not currently supported by ONNX or TorchSeript As expected, FNet executed in Py PyTorch outperforms BERT exe- cuted in ONNX despite less framework overhead and numerous static optimizations with a 10.3 % speedup at batch size For improvements in re- search translate to deployment, additional invest- ment can be directed towards support for complex control flow and operators in inference runtimes.
Throughput and input size can be increased at minimal cost for framework-bound models. For a given model, latency is constant regardless of batch size until compute kernels saturate and exceed CPU launch costs. If computation is bot-

---

SA Dim
FC Dim
Batch
Seq
Latency
TP
ResNet-500,, we show that all models exhibit frame- work boundedness. Additionally, we observe that these inefficiencies are becoming more apparent as hardware accelerator speeds increase.
768
3072
1
128
0.0136
0.0136
768
3072
4
128
0.0134
0.0034
768
3072
1
512
0.0134
0.0134
1536
6144
128
0.0134
0.0134
We introduce the concept of the framework tax to describe when improvements in hardware speed and reductions in required computation fail to trans- late to speedups in model latency and throughput due to bottlenecks incurred by deep learning frame- works. We hope that these observations raise aware- ness of the impact and limitations created by choice of deep learning frameworks on model develop- ment and deployment.
Table I: Latency and throughput (TP) BERT Py PyTrrh models RTX-8000. Scaling along batch sizes and model width shows no increase in latency.
tlenecked by framework overhead, the batch size or input size can be increased without increases in overall runtime. In practice, this can lead to the pro- cessing of larger batch sizes and sequence lengths at no additional latency cost until kernel operations saturate framework overhead.
Limitations
In this work, study the inference efficiency of speech and language models using GPU hardware accelerators. While GPUs are the most common general purpose hardware accelerators, there ex- ist domain specific architectures such as Google TPU's, GraphCore IPUs, and custom ASICs which present additional settings for future investigation. Additionally, our study evaluates efficiency via model latency and our claims do not necessarily translate to other metrics of efficiency, such power consumption or power output.
Model width can be increased at cost for framework-bound models. For given batch size, individual layers a framework bound model can be made wider by increasing hidden dimension or filter size without impacting latency. Wider mod- els are known to exhibit better task performance due to increased parameter count and expressivity (Zagoruyko and Komodakis, 2016).
Model designers can leverage wider architec- tures with their target inference framework and hardware setting in mind achieve higher utiliza- tion. For example, models processing few exam- plcs during inference can leverage wider layers to avoid framework bottlenecks. See Table 1.
As models continue to scale in size, they often require model or data parallelism techniques that require computation across several nodes which introduce overhead from multi-device synchromiza- tion and network communication. Additionally, we do not study latency in the training setting where the per-layer computation is larger due to the com- putation of gradients and losses.
Using Bigher-perfomiigg hardware does not necessarily improve end-to-end performance. Framework overhead limits the impact of improved hardware as it limits utilization. This trend will con- tinue as ML-specific hardware advances without ef- forts to address software bottlenecks. For example, single-example inference with both BERT is slower using an A A100 than using V100 GPU despite a 2.75x increase in peak computational throughput.
A Acknowledgements
This work was supported in part by a grant from the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE2140739. We thank the anonymous reviewers for their valuable feedback. We would also like to thank lab members and faculty for helpful feed- back during discussions and revisions, including: Daniel Fried, Han Guo, Jeremiah Milbauer, Sanket Vaibhav Mehta, and Saujas Vaduguru.
6 Conclusion
We conduct an extensive study of neural networks from the convolutional and transformer architec- ture paradigms across a variety of software and hardware platforms. We show that inference per- formed with these large neural networks, which was previously assumed to be compute bounded, is in fact limited by overhead incurred by deep learning frameworks. While wider transformer ar- chitectures (c.g. BERT-Base) exhibit less bounded- ness behaviors than narrower, deeper CNNs (e.g.
References
Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller, Dzmitry Bahdanau, Nicolas Ballas, Frédéric Bastien, Justin Bayer, Anatoly Be- likov, Alexander Belopolsky, et al. 2016. Theano: A

---

python framework for fast computation of mathemat- ical expressions. arXiv e-prints, pages arXiv-1605.
Mostafa Dehghani, Anurag Amnb, Lucas Beyer, Ashish Vaswani, and Yi Tay. 2021. The efficiency misnomer. arXiv preprint arXiv: 10. 12894.
Ibrahim M Alabculmohsin, Behnam Neyshabur, and Xiaohua Zhai. 2022. Revisiting neural scaling laws in language and vision. Advances in Neural Informa- tion Processing Systems, 35:22300-22312.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
Reza Yazdani Aminabadi. Samyam Rajbhandari. Am- mar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022. Deepspeed-ig inference: ieepseeeddiffeernnce:: enabling efficient inference of transformer models unprece- dented scale. In Proceedings of the International Conference on High Performance Computing, Net- working. Storage and Analysis, pages 1-15.
James Gleeson, Moshe Gabel, Gennady Pekhimenko, Eyal de Lara, Srivatsan Krishnan, and Vijay Janapa Reddi. 2021. RI-scope: Cross-stack profiling for deep reinforcement learning workloads. Proceed- ings of Machine Learning and Systems, 3:783-799.
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020. Power-bert: Accelerating bert inference via progressive word-vector elimina- tion. In International Conference on Machine Learn- ing. pages
Jeff Barr. 2019. Amazon ec2 update-infl instances with aws inferentia chips for high performance cost- effective inferencing-
PMLR.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Landermman-iinne and Qiao Zhang. 2018. JAX: composable transformations of Python+NumPy programs.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Han Cai, Ligeng Zhu, and Song Han. 2018 Proxy- lessnas: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations.
Sara Hooker. 2021. The hardware lottery. Communica- tions of the ACM, 64(11))58-65.
Lu Hou, Zhiqi Huang. Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33:9782-977 9793.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Se- lected Topics in Signal Processing, 16(6):1505-1588.
Forrest N landola. Song Ham, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. 2016. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07366)
Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang. Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. 2015. Mxnet: A flexi- ble and efficient machine learning library for het- erogeneous distributed systems. arXiv preprint arXiv:1512.01274.
Forrest N landola, Albert E Shaw, Ravi Krishna, and Kurt W Keutzer. 2020. Squeezebert: What can computer vision teach nlp about efficient neural net- works? arXiv preprint wxXiv2000..13366
Tianqi Chen, Thierry Morcau, Ziheng Jiang, Lianmin Zheng, Eddie Yan. Haichen Shen. Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, al. 2018. [TVM): An automated (End-to-Fam Endd optimizing compiler for deep learning. In 13th USENIX Sympo- sium Operating Systems Design and Implementa- tion (OSDI 18), pages 578-594.
Peter Izsak, Moshe Berchansky, and Omer Levy. 2021. How to train bert with an academic budget. In Pro- ceedings of the 2021 Conference Empirical Meth- ods in Natural Language Processing, pages 10644- 10652.
Vijay Janapa Reddi, David Kanter. Peter Mattson, Jared Duke, Thai Nguyen, Ramesh Chukka, Ken Shir- ing, Koan-Sin Tan, Mark Charlebois, William Chou, et al. 2022. MIperf mobile inference benchmark: An industry-standard open-source machine learning benchmark for on-device ai. Proceedings of Machine Learning and Systems, 4:352 369.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0231
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Con- volutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international confer- ence on Multimedia, pages 675-678.
Zihang Dai, Guokun Lai, Yiming Yang, and Quoe Le, Filtering out sequential redundancy for efficient language processing, Ad- vances in neural information processing systems, 33:4271-4288
2020.

---

Jacob D Kahn, Vineel Pratap, Tatiana Likhoma- nenko, Qiantong Xu, Awni Hanmun, Jeff Cai, Paden Tomasello, Ann Lee, Edouard Grave, Gilad Avidov, al. 2022. Flashlight: Enabling innovation in tools for machine learning. In International Conference on Machine Learning, pages 10557-10574. PMLR.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. 2021. Do wide and deep networks learn the same things? uncovering how neural network representa- tions vary with width and depth. In International Conference on Learning Representations.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess. Rewon Child, Scott Gray. Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint ar 2rXiv:200100830
David Patterson, Joseph Gonzalez, Urs Hölzle, Quoe Le, Chen Liang. Lluis-Miquel Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. 2022. The carbon footprint of machine learning train- ing will plateau, then shrink. Computer, 55(7):18- 28.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXi::2001.044: 04451..
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel. Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learn- ing of language representations. arXiv preprint arXiv:1909.119 11942.
Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, et 2020. MIperf infer- ence benchmark. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pages 446-459. IEEE
David Langerman, Alex Johnson, Kyle Buettner, and Alan D George. 2020. Beyond floating-point ops: Cnn performance prediction with critical datapath length. In 2020 IEEE High Performance Extreme Computing Conference (HPEC), pages 1-9, IEEE,
Mark Sandler, Andrew Howard, Menglong Zhu, An- drey Zhmoginov, and Liang-Chieh Chen. 2018. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEFF conference on computer vision and pattern recognition, pages 4510-4520.
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. 2021. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.03822
George Leopold. 2019. Aws to offer nvidia's 14 gpus for ai inferencing, URL: https://web. archive. hpewire.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of hert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 1910.01 108.
wgg/we//202200000002/////ssss//2 comm/00090////9//sssungneeee -pp------aa------att infereecce-lafffomm/iiisee on 2022-04-19).
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2020. Green ai. Communications of the ACM, 63(12):54-63.
Cheng Li, Abdul Dakkak, Jinjun Xiong, Wei Wei, Lingjie Xu, and Wen-mei Hwu. 2020. Xsp: Across- stack profiling and analysis of machine learning mod- els on gpus. In 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 326-327. IEEE
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages (645-3650, Florence, Italy. Asso- ciation for Computational Linguistics.
Zhongyi Lin, Louis Feng, Ehsan K Ardestani, Jaewon Lee, John Lundell, Changkyu Kim, Arun Kejariwal, and John D Owens. 2022. Building performance model for deep learning recommendation model train- ing on gpus. arXiv preprint arXiv:2201.0783
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang. and Denny Zhou. 2020. Mobilebert: a compact task-agnostic bert for resource-limited de- vices. arXiv preprint arXiv:2004.0222 029844
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 10w::1997.1192.
Mingxing Tan, Bo Chen, Ruoming Pang. Vijay Vasude- van, Mark Sandler, Andrew Howard, and Quoe Le. 2019. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 2820-2828.
Moshe Looks, Marcello Herreshoff, DeLesley Hutchins, and Peter Norvig. 2017. Deep learning with dynamic computation graphs. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24.26, 2017, Conference Truck Pro- ceedingy. OpenReview.net.
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler, 2022. Scaling laws vs model architectures: How

---

does inductive bias influence scaling? arXiv preprint vrXiv:2207.10551.
the AAAI Conference on Artificial Intelligence, vol- ume 35, pages -1188-44148.
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama. Ashish Vaswani, and Donald Met- zler. 2021. Scale efficiently: Insights from pre- training and fine-tuntin -uniin transformers. arXiv preprint arXiv:2109. 10686.
Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. arXiv preprint arXiv: 1605.001444 07146.
Xiangyu Zhang. Xinyu Zhou, Mengxiao Lin, and Jian Sun. 2018. Shufflenet: An extremely efficient con- volutional neural network for mobile devices. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1848-6856.
Ann Taylor Mitchell Marcus, and Beatrice Santorini. 2003. The penn treebank: an overview. Treebanks: Building and using parsed corpora, pages 5-22.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. 2021. Deepvit: Towards deeper vision trans- former. arXiv preprint arXiv:2103. 11886.
Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clay- ton. 2015. Chainer: next-generation open source framework for deep learning. In Proceedings of work- shop on machine learning systems (LearningSss) in the twenty-ninte anmual conference on neural infor- mation processing systems (NIPS), volume pages 1-6.
Xiyou Zhou, Zhiyu Chen, Xiaoyong Jin, and William Yang Wang. 2020. Hulk: An en- engy efficiency benchmark platform for responsi- ble natural language processing. arXiv preprint arXiv:2002.05822
Hugo Touvron, Matthicu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. 2021. Going deeper with image transformers. In Proceedings of the IEEF/CVE International Conference on Com- puter Vision, pages 42.42.
Hongyu Zhu, Mohamed Akrout, Bojian Zheng, Andrew Pelegris, Amar Phanishayee, Bianca Schroeder, and Gennady Pekhimenko. 2018. Tbd: Benchmarking and analyzing deep neural network training, arXiv preprint
Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. 2020a. Hat: Hardware-aware transformers for efficient natural lan- guage processing. arXiv preprint ar XrXiv:2005. 14187.
Hongyu Zhu, Amar Phanishayee, and Gennady Pekhi- menko. 2020. Daydream: Accurately estimat- ing the efficacy of optimizations for (DNN} train- ing. In 2020 USENIX Annual Technical Conference (USENIX ATC 20), pages 337-352.
Sinong Wang, Belinda 7 Li, Madian Khabsa, Han Fang, and Hao Ma. 2020b. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.c 046.046800
A Hardware Platforms
Yu Wang. Gu- Yeon Wei, and David Brooks. 2020c. A systematic methodology for analysis of deep learning hardware and software platforms. Proceedings of
Full details for the various hardware platforms and GPUs used for evaluation in Section 4.3 are de- scribed in Table 2.
Machine Learning and Systems, 2:30-43.
B Hardware and Utilization
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. 2019. Fbnet: Hardware-awart efficient convnet design via differ- entiable neural architecture search. In P Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10734-10742.
In Table 3, we examine GPU hardware utilization of various frameworks in terms of the percentage of SMs active (i.c. number of compute units utilized) and warp occupancy (i.e. amount of time each com- pute unit is active). Graph compilations and ker- nel serialization reduce framework overhead and increase hardware utilization. All frameworks ob- serve suboptimal hardware utilization
Carole-Jean WU, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Glo- ria Chang, Fiona Aga, Jinshi Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anas- tasia Melnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin Lee, Bu- gra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim Hazelwood. 2022. Sus- tainable ai: Environmental implications. challenges and opportunities. In Pruceedings of Machine Learn ing and Systems, volume 4, pages 795-813.
C Additional Experiments: Natural Language Generation
In Figure 10, we examine the GPT-2 decoder-only transformer model architecture and observe that with encoder-only architectures generative models encounter framework boundedness for small input sizes. Evaluations are performed across batch sizes from 1 to 128 and a fixed generation length of 16 tokens.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. 2021. Nyströmforner: nyström-based algorithm for approximating self-attention, In Proceedings of

---

CPU
GPU
GPU Arch
Core Count
Tensor
Clock Rate
Memory (GB)
Mem (GB/s)
BW
FP16
Intel Xeon Silver 4110
10SITT
Pascal
3584
1.38
11
484.4
22.78
Xeon
Volta
32
Intel Xeon ES-2630
2080T1
Turing
3584
544
1.35
11
616
26.90
Intel Xeom K5-2630
KIX-0000
Turing
48
AMD EPYC 7282
3090
Ampere
10496
328
1.35
24
936
35,89
Intel Xeon Silver 4110
A6000
Ampere
10752
336
141
48
768
38.71
Intel Xcon
A100
Ampere
432
Table 2: Details on hardware platforms used in our experiments, ordered by Nvidia microarchitecture generation.
Framework
Graph Compilation
Kernel Serialization
Latency
SMs Active
Warp Occupancy
Porch
Nome
None
10.54 ms
2.6%%
0.9%
Py Tarch with TarchScript
Just-in-Time
None
6.14 ms
18.5%
3.0%
Py Torch with Graphs
Yes
ONNX RT
Alead-of-Timme
None
2.56 ms
22.3%
9.595
ONNX RT with CUDA Graphs
Ahead-of Time
Yes
2.11
59%
20.3%
Table 3: Comparison of SM Activity Across Frameworks for BERT-Bas batch size 1.
GPT-2 (fp16)
fculictigg Mauilatie cTiceo rhet
Pyllorch TorchScript
Latency
Shuftalte
Latency (s)
10-2
10a
10
107
1013
109
10P
30
102
8x10
MACs
Batch Size
Figure 12: Latency of vision models using efficient
18
variations of convolution and self-attention operations.
107
Batch Size
E Additional Experiments: Speech
Figure 10: Latency of generative language models for varying batch sizes.
In Figure 13, examine the behavior of the WavLM (Chen al., 2022) model which consists of CNN encoder followed by transformer encoder layers. Audio inputs are simulated as second sequences sampled at 16 kHz to create 32,000- dimensional floating point inputs. Figure 13, we observe that WavLM exhibits framework bound behavior but quickly transitions to being compute- bound due to the large audio sequence lengths.
D Additional Experiments: Vision
In Figures 11 and 12, we examine vision models utilizing "efficient" model design choices through scaling and efficient operation variants. Image in- puts simulated by randomly generating three- channel 224 224 RGB images. As with language models, deeper models introduce additional frame- work overhead and low-FI Now-FLOP alternatives are more framework bound.
Mevi (fp16)
WevLM (jp32)
10-1
Latency
Notic: 101
NONS
Whie Rollls
10-1
10-1
16-1
10-7
Latency
109
10"
Size
10
10"2
Figure 13: Transformer-based speech models exhibit
framework boundedness but transition to compute- bound at small batch sizes due long sequence lengths.
MACS
Batch Size
Figure 11: Latency of vision models that scale model depth and number of hidden dimensions.


Reviews:
[{"review_id": "yO0MXMsglI", "reviewer": "Reviewer_aBkf", "summary": "", "questions": "The contribution should be highlighted at the end of Introduction. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s positive assessment of our work. As the reviewer notes, this work is the first to our knowledge to systematically examine the impact of deep learning frameworks on the efficiency of NLP systems. As we observe in our Related Work, there exist performance analyses of deep learning-based systems in other fields such as recommendation systems and simulation-based reinforcement learning [1, 2]. For the final revision, we will be sure to more clearly include comparisons with these works and to emphasize our main contribution in the introduction.\n\n### Reference\n\n[1] James Gleeson, Moshe Gabel, Gennady Pekhimenko, Eyal de Lara, Srivatsan Krishnan, and Vijay Janapa Reddi. 2021. Rl-scope: Cross-stack profiling for deep reinforcement learning workloads. Proceedings of Machine Learning and Systems, 3:783–799.\n\n[2] Zhongyi Lin, Louis Feng, Ehsan K Ardestani, Jaewon Lee, John Lundell, Changkyu Kim, Arun Kejariwal, and John D Owens. 2022. Building a performance model for deep learning recommendation model training on gpus. arXiv preprint arXiv:2201.07821.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Presents a series of case studies to analyze the framework tax phenomenon.", "reasons_to_reject": "1. There is no comparison with other studies. \n", "questions_to_authors": "The contribution should be highlighted at the end of Introduction. ", "paper_topic_and_main_contributions": "This paper presents a series of case studies to analyze the framework tax phenomenon existing in deep learning frameworks and its effect on model efficiency. Overall, there is no comparison with other studies. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qemzQVotpg", "reviewer": "Reviewer_fRoK", "summary": "", "questions": "A. The results focus heavily on PyTorch, TorchScript, and ONNX. While these are undoubtedly popular, what about other major frameworks, such as TensorFlow 2.0? Have you observed any similar disparities in those frameworks?\n\nB. In the \"Experiments\" section, you've focused on specific GPU microarchitectures (Pascal, Turing, Ampere). While this selection covers a broad range, it would be enlightening to understand the rationale behind excluding certain GPU architectures or why these specific architectures were chosen.\n\nC. While the discussion emphasizes the disadvantages of the framework tax, are there potential advantages or reasons why these overheads exist in the first place? For instance, might any trade-offs related to functionality, flexibility, or developer ease-of-use justify some amount of this tax?\n\nD. The \"Limitations\" section briefly touches upon other hardware accelerators like TPUs and ASICs but doesn't delve into the potential implications of the framework tax in those settings. Given the growing popularity of domain-specific architectures for AI, could you speculate or provide preliminary observations about how the framework tax might manifest in those contexts?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to the reviewer for their positive comments and comprehensive feedback. As the reviewer notes, there are many components and modeling choices that contribute to the design of efficient NLP systems. While the framework tax and its associated framework overhead are by no means the sole bottleneck on efficiency, we believe that our study provides valuable analysis of common workloads and identifies an often ignored bottleneck not captured by existing metrics (e.g. FLOPs/MACs, parameter counts).\n\n## Questions & Concerns\n`While the \"framework tax\" is an interesting observation, the paper treats all NLP models uniformly. Given the diversity of NLP tasks and associated models, it's possible that not all models will exhibit similar \"framework tax\" phenomena.`\n\nAlthough there exist settings in which other factors  may limit system latency, such as user interaction in human in the loop systems, many modern NLP systems rely on large neural network transformer-based models – a setting where we show framework overhead can often impose bottlenecks on inference efficiency. Additionally, we show that framework boundedness is not limited to encoder-only models and with CNNs and hybrid CNN-transformer architectures exhibiting the same behaviors in Appendix C and D. \n\nFurthermore, in additional experiments with decoder-only GPT-2 (for maxlen 16), we find that framework overhead also bottlenecks seq2seq language generation small batch sizes \n\n| Batch Size | Latency (ms) |\n|-|-|\n|1|\t6.993\n|2|\t7.987\n|4|\t8.629\n|8|\t8.241\n|16|\t8.791\n|32|\t9.237\n|64|\t13.72\n|128| 18.78\n\n`The paper measures efficiency primarily through model latency. This ignores other crucial efficiency aspects such as power consumption, cost-effectiveness, and overall throughput.`\n\nWhile other efficiency measures are important in assessing the efficiency of NLP systems, inference latency is a common real-world bottleneck in deployment and frequently reported measurement among existing efficiency benchmarks [3, 4]. We leave these investigations as potential directions for future work as efficiency cost indicators are often uncorrelated in practice [2]. \n\n`The results focus heavily on PyTorch, TorchScript, and ONNX. While these are undoubtedly popular, what about other major frameworks, such as TensorFlow 2.0? Have you observed any similar disparities in those frameworks?` \n\nWe select PyTorch (eager), TorschScript (JiT), and ONNX RT (AoT) as representative examples of frameworks from various design paradigms. As an exhaustive evaluation of all deep learning frameworks is beyond the scope of this work, our goal is to provide insight into the overhead associated with each paradigm. As TF2.0 is based around the eager execution, we expect to observe high framework overhead as observed with eager execution PyTorch.\n\n` In the \"Experiments\" section, you've focused on specific GPU microarchitectures (Pascal, Turing, Ampere). While this selection covers a broad range, it would be enlightening to understand the rationale behind excluding certain GPU architectures or why these specific architectures were chosen.` \n\nThe Pascal, Turing, and Ampere microarchitectures and the associated GPUs are representative of publicly available consumer  (eg. 1080 Tis, 2080Ti, 3090) and enterprise  (e.g. v100, RTX8000, A6000)  GPUs at the time experiments were conducted. They were selected to reflect increases in hardware speed (e.g. via increases in number of CUDA core counts, clock speed, etc) as well as the introduction of new computing technology (e.g. the introduction of tensor cores in the Turing generation). \n\n`While the discussion emphasizes the disadvantages of the framework tax, are there potential advantages or reasons why these overheads exist in the first place? For instance, might any trade-offs related to functionality, flexibility, or developer ease-of-use justify some amount of this tax?`\n\nAs the reviewer notes, efficiency tradeoffs are often made to increase usability. For example, PyTorch allows for rapid prototyping via eager construction of computational graphs in Python – but incurs host language overheads and forgoes optimizations from graph compilation. While these tradeoffs are often beneficial to research velocity, one of our goals in highlighting these sources of overhead to motivate the design of frameworks that are both lightweight and usable.\n\n`Since the paper focuses heavily on GPUs, it might unintentionally endorse or bias attendees towards GPU-based architectures, neglecting other emerging and efficient architectures.`\n\n`The \"Limitations\" section briefly touches upon other hardware accelerators like TPUs and ASICs but doesn't delve into the potential implications of the framework tax in those settings. Given the growing popularity of domain-specific architectures for AI, could you speculate or provide preliminary observations about how the framework tax might manifest in those contexts?`\n\nIn our study we primarily focus on the GPUs due to their widespread availability and high usage as deep learning hardware accelerators. Unfortunately, we did not have access to domain specific hardware. Based on prior performance analyses of custom hardware accelerators [1], we expect that CPU and framework overhead would also impose bottlenecks on execution with TPUs as they follow a similar design based around a host CPU and a TPU compute device. \nAdditionally, in additional experiments on CPU-only devices where there is no need for operation dispatch to GPUs, we find that framework overhead is less pronounced.\n\n### References\n\n[1] Wang, Yu, Gu-Yeon Wei, and David Brooks. \"A systematic methodology for analysis of deep learning hardware and software platforms.\" Proceedings of Machine Learning and Systems 2 (2020): 30-43.\n\n[2] Dehghani, Mostafa, et al. \"The Efficiency Misnomer.\" International Conference on Learning Representations. 2021.\n\n[3] Peng, Hao, et al. \"Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation.\" arXiv preprint arXiv:2307.09701 (2023).\n\n[4] Reddi, Vijay Janapa, et al. \"Mlperf inference benchmark.\" 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses the significant yet overlooked problem of the \"framework tax,\" where improvements in hardware and model architectures do not necessarily translate to reduced inference latencies.\n2. Through systematic investigation, the authors evaluate different model architectures across multiple frameworks and GPUs, providing a robust and varied assessment of the phenomenon.\n3. The paper offers actionable insights and recommendations, like the suggestion to use static or ahead-of-time inference runtimes for small batch sizes, that can greatly benefit NLP researchers and practitioners.\n4. By highlighting the disparities between NLP research and real-world deployment, the paper fosters an understanding that could bring research and practice closer, ensuring that efficiency gains in the research stage translate well in deployment.\n\nBenefits to the NLP community:\n1. Researchers and developers will become more aware of the overhead introduced by deep learning frameworks, potentially influencing their choice of frameworks and methodologies.\n2. Model designers can use the findings to create both computationally efficient models and have reduced inference latencies, leading to faster real-world applications.\n3. For developers with existing deployments, the insights from this paper can guide adjustments to current systems for better performance.\n4. The paper sets the stage for future research on framework optimization, model-hardware co-design, and a deeper understanding of the intricate play between model design, hardware, and frameworks.", "reasons_to_reject": "1. The paper focuses predominantly on GPUs for evaluation. Given the wide variety of hardware accelerators used in the industry, such as TPUs, IPUs, and custom ASICs, the study's findings might not generalize across all hardware platforms.\n   \n2. The paper measures efficiency primarily through model latency. This ignores other crucial efficiency aspects such as power consumption, cost-effectiveness, and overall throughput.\n\n3. While the \"framework tax\" is an interesting observation, the paper treats all NLP models uniformly. Given the diversity of NLP tasks and associated models, it's possible that not all models will exhibit similar \"framework tax\" phenomena.\n\n4. While the paper mentions that certain operations and design decisions could potentially reduce latency or framework overhead, some of these claims lack empirical results to back them up, relying on theoretical reasoning alone.\n\nRisks of presenting at the conference:\n\n1. Presenting the idea of a \"framework tax\" might oversimplify the challenges faced in deploying NLP models. Attendees might walk away thinking that framework overhead is the only significant issue, which might not be the case.\n\n2. Since the paper focuses heavily on GPUs, it might unintentionally endorse or bias attendees towards GPU-based architectures, neglecting other emerging and efficient architectures.\n\n3. If attendees focus too much on the \"framework tax\" and neglect other aspects of model efficiency, it might misdirect research and development efforts towards only one aspect of the problem.\n\n4. Given the emphasis on the inefficiencies of existing frameworks, the paper might receive pushback or defensive reactions from developers or proponents of the said frameworks, leading to polarized discussions.", "questions_to_authors": "A. The results focus heavily on PyTorch, TorchScript, and ONNX. While these are undoubtedly popular, what about other major frameworks, such as TensorFlow 2.0? Have you observed any similar disparities in those frameworks?\n\nB. In the \"Experiments\" section, you've focused on specific GPU microarchitectures (Pascal, Turing, Ampere). While this selection covers a broad range, it would be enlightening to understand the rationale behind excluding certain GPU architectures or why these specific architectures were chosen.\n\nC. While the discussion emphasizes the disadvantages of the framework tax, are there potential advantages or reasons why these overheads exist in the first place? For instance, might any trade-offs related to functionality, flexibility, or developer ease-of-use justify some amount of this tax?\n\nD. The \"Limitations\" section briefly touches upon other hardware accelerators like TPUs and ASICs but doesn't delve into the potential implications of the framework tax in those settings. Given the growing popularity of domain-specific architectures for AI, could you speculate or provide preliminary observations about how the framework tax might manifest in those contexts?", "paper_topic_and_main_contributions": "Despite improvements in NLP model architectures and hardware accelerators, there's a discrepancy between the computational efficiency improvements and the real-world (wall-clock) inference latency. The paper attributes this discrepancy mainly to overheads introduced by deep learning frameworks, terming it the \"framework tax\".\n\nKey contributions:\n\n1. Demonstrated that advancements in neural networks and hardware don't necessarily reduce inference latency due to the \"framework tax, \" which increases as hardware speeds up.\n   \n2. Investigated the effects of model design decisions, deep learning framework paradigms (eager execution, just-in-time, and ahead-of-time compilation), and hardware platforms on total model latency.\n\n3. Evaluated transformer and convolutional neural network model performance across various environments (PyTorch, TorchScript, ONNX) and GPUs. Findings revealed that framework overhead dominates at small batch sizes, leading to proxies like MACs and parameter counts not predicting inference latency effectively.\n\n4. Suggested using ahead-of-time inference runtimes for small batch sizes to reduce overhead. For those using eager execution, increasing model width or batch size doesn't cost extra latency, providing extra flexibility in model design.\n\n5. Highlighted the divergence between research settings and deployment, emphasizing that researchers should specify efficiency gains contextually (frameworks, hardware).\n\n6. Pointed out that more performant hardware doesn't necessarily equate to improved end-to-end performance because of the \"framework tax\".", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fkcl3M91rE", "reviewer": "Reviewer_GHwB", "summary": "", "questions": "1. Why did you not look at quantization?\n2. Do you see any difference across fp16 to bf16?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’ positive feedback and agree that our insights on bottlenecks introduced by deep learning frameworks can provide immediately useful guidance in the deployment of efficient NLP systems! We hope that our work can help researchers better understand practical bottlenecks and ensure that efficiency gains attained in research settings translate to deployed NLP systems.\n\n## Questions and Concerns\n\n`The paper does not analyze any generative models such as Llama.`\n\nWe focus our analysis on BERT-like sequence classifications models to investigate the effectiveness of efficient model architecture designs which  have largely targeted encoder-only models (e.g. MobileBERT, SqueezeBERT, DistilBERT, etc). \n\nIn preliminary experiments with decoder-only LMs, we find that the framework tax also leads to inference bottlenecks of generative models.  For sequence generation, we observe framework bound behaviors with roughly fixed latency for batch sizes smaller than 16.\n\nGPT-2 Base in PyTorch (SeqLen = 16 Tokens):\n| Batch Size | Latency (ms) |\n|-|-|\n|1|\t6.993\n|2|\t7.987\n|4|\t8.629\n|8|\t8.241\n|16|\t8.791\n|32|\t9.237\n|64|\t13.72\n|128| 18.78\n\n`The paper does not analyze quantized models in either int8 or int4. Why did you not look at quantization?`\n\nWe primarily focus on optimizations from computational graph compilation and sparsity, as quantization is not currently natively supported for GPU operations in PyTorch nor by ONNX Runtime’s CUDA Execution Provider. \n\nRef: https://pytorch.org/docs/stable/quantization.html#frequently-asked-questions\n\n`The evaluation does not explore optimized runtimes like Faster Transformers, which commonly lead to 10x speedup, especially in small models.`\n\nIn Section 4.1, we investigate sparse transformer and nested tensor accelerations with PyTorch’s BetterTransformer optimizations. BetterTransformer provides FlashAttention and FastPath optimizations which are similar to the optimizations provided by Nvidia’s FasterTransformer. \n\nAs with both the JiT and AoT runtimes, we observe that FasterTransformer yields reduces latency but still suffers from framework boundedness at lower batch sizes. We will be sure to clarify implementation details in future revisions.\n\n`Do you see any difference across fp16 to bf16?`\n\nWith an Nvidia RTX-8000 GPU, we observe that for both float16 and bfloat16, inference latency exhibits framework bounded behavior.  However, inference with bfloat transitions to compute boundedness at smaller batch sizes. This discrepancy can likely be attributed to the lack of bfloat16 support by the Turing generation of Nvidia GPUs which results in slower computation. \n\nRef: https://www.nvidia.com/en-us/data-center/tensor-cores/\n\n| Batch Size | Float16 Latency (ms) | BFloat Latency (ms) | \n| - | - | -|\n1 | 6.24| 12.75\n2 | 8.58 | 11.75\n4 | 8.24  | 15.18\n8 | 8.26  | 24.95\n16 | 9.28 | 49.17\n32 | 18.03 | 92.93\n64 | 32.5 | 184.65\n128 | 63.8 |  365.03\n\n`The paper does not study the impact of sequence length on framework vs compute bounding`\n\nWhile we primarily study the impact of batch size scaling, framework boundedness is not only limited to small batch sizes but also bottlenecks inference on short sequences. In additional experiments (below), we observed the latency of BERT-Base for various sequence lengths at batch size 128 and observe boundedness for sequences shorter than 8 tokens. \n\n|Seq Len |\tLatency for BS=128  (ms)\n|--|--|\n1\t|\t6.05 \n2|\t        8.39 \n4|\t         8.40 \n8|\t8.30 \n16|\t9.73 \n32|\t17.07\n64|\t30.58\n128 | 63.63", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Paper provides an easy-to-read, thorough evaluation of the impacts to model latency and how frameworks and computations play together.\n2. Paper evaluates many sub-components of latency using different models, batch sizes, etc. \n3. Evaluation is on many types of computing hardware", "reasons_to_reject": "1. The paper does not analyze any generative models such as Llama.\n2. The paper does not analyze quantized models in either int8 or int4.\n3. The evaluation does not explore optimized runtimes like Faster Transformers, which commonly lead to 10x speedup, especially in small models. \n4. The paper does not study the impact of sequence length on framework vs compute bounding", "questions_to_authors": "1. Why did you not look at quantization?\n2. Do you see any difference across fp16 to bf16?", "paper_topic_and_main_contributions": "The authors introduce a thorough evaluation on estimating the cost of deep learning model frameworks vs the actual computation which occurs on CPU and GPU. as a part of their study, the demonstrate that many improvements in FLOPs lead to no improvements in wall clock, which they attribute to the frameworks used. They then go on to posit that the framework tax will increase over time despite improvements in hardware speed. \nMoreover, this paper introduces well-formulated tidbits that can be useful to researchers deploying language models such as: \n1. At small batches inference is compute bound but as batch size scales models become framework bound. \n2. The use of specialized models are often not faster in wall clock as they are memory bound.\n3. Older GPUs are not framework bound but novel more expensive GPUs are. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies why improvements in throughput and FLOPs of recent efficient models and hardware architectures do not directly translate to reductions in model inference latency. The paper attributes this gap to the overhead imposed by some deep learning frameworks, to which it refers as "framework tax." Overall, the reviewers unanimously rate this paper as strong both with regards to soundness and excitement.

The reviewers praise this paper for the broad insights it provides on a practically relevant topic. They also praise the novelty and systematic investigation, and note that this can be widely useful for the NLP community.

Most of the reviewer criticism centers around extending the study to other accelerators, models (beyond encoder-only transformers), and other aspects such as to investigate different sequence lengths. While the present paper does not cover all possible experiment settings, it nevertheless has been praised for its broadness by the reviewers. This suggests that the identified gaps could motivate further work on this subject in the future. Moreover, the authors provide preliminary insights in their responses on some of the settings requested by the reviewers, such as decoder-only transformers, and regarding sequence length. These results could provide additional value to the submission.
Decision: Accept-Main