Amdahl’s Law for LLMs: A Throughput-Centric Analysis of Extreme LLM Quantization
Authors: Jinendra Malekar
Ramtin Zand
Venue: Accepted by TMLR ()

Abstract: The emergence of 1-bit large language models (LLMs) has sparked significant interest, promising substantial efficiency gains through extreme quantization. However, these benefits are inherently limited by the portion of the model that can be quantized. Specifically, 1-bit quantization typically targets only the projection layers, while the attention mechanisms remain in higher precision, potentially creating significant throughput bottlenecks. To address this, we present an adaptation of Amdahl's Law specifically tailored to the LLMs, offering a quantitative framework for understanding the throughput limits of extreme quantization. Our analysis reveals how improvements in quantization can deliver substantial throughput gains, but only to the extent that they address critical throughput-constrained sections of the model. Through extensive experiments across diverse model architectures and hardware platforms, we highlight key trade-offs and performance ceilings, providing a roadmap for future research aimed at maximizing LLM throughput through more holistic quantization strategies.

Published in Transactions on Machine Learning Research (09/2025)
Amdahl's Law for LLMs: A Throughput-Centric Analysis of Extreme LLM Quantization
Jinendra Malekar
jmalekar@emaillsc.dd
Deportment of Computer Science and Engineering University of South Carolina
Ramtin Zand
ramtin@kce..sc.co
Deportmend of Computer Science and Engineering University of South Carolina
Reviewed on OpenReview: https: // openreview. net/ pet//frum?? id= jdrJU@QU
Abstract
The emergence of 1-bit large language models (LLMs) has sparked significant interest, promising substantial efficiency gains through extreme quantization. However, these benefits are inherently limited by the portion of the model that can be quantized. Specifically, 1-bit quantization typically targets only the projection layers, while the attention mechanisms re- main higher precision, potentially creating significant throughput bottlenecks. To address this, we present an adaptation of Amdahl's Law specifically tailored the L.MMs, offering a quantitative framework for understanding the throughput limits of extreme quantization. Our analysis reveals how improvements in quantization can deliver substantial throughput gains, but only to the extent that they address critical throughpuu--constrannee sections of the model. Through extensive experiments across diverse model architcetures and hardware platforms, we highlight key trade-ofTs and performance ceilings, providing a roadmap for future research aimed at maximizing LLM throughput through more holistic quantization strategies.
1 Introduction
Generative Large language models (LLMs) such as GPT (Radford et al. 2019), OPT (Zhang et al. 2022) and LLaMA (Touvron et al. 2023) have attracted significant attention in recent years bocause of their
impressive performance across various lasks, including but not limited to machine translation (Xu el. al.
2024a), conversational chatbots (Sánchez Cuadrado al. 2024). question answering (Tan al. 2023). and even code generation (Ugare ct al. 2024).
However, the remarkable performance of LLMs has come with increasing computational and energy costs. This necessitates expanding tighh-performacc computing resources in data centers, potentially delaying green energy commitments and causing adverse environmental impacts (Stojkovic et al., 2024). Consequently, recent research has focused on optimizing the energy footprint of LLMs through various model optimization and compression techniques like pruning (Ma al. 2023) and quantization (Xiao 2023:
Shao et al.
2023
Ma et al. 2024
In addition to enhancing the energy efficiency of rumning LLMs in data centers, model optimization and compression can facilitate the deployment. of LLMs on mobile and edge computing devices for real-time
processing (Reidy et al. 2023 Ardakani et al.] 2025). This advancement could benefit. various emerging
applications, including social roboties (Addlesee et al. 2024 Esteban-Loor Lozano et al. 2024), and augmented
and virtual reality (Asadi et al.
2024
Morales Showalter Bucher 2023
Among LLM compression approaches, quantization has become a focal point, driven by works such Smoothquant. (Xiao al. 2023 Omniquant (Shao et. al. 2023 and more recently ShiftAddLLLM (You)

---

Published in Transactions on Machine Learning Research (09/2025)
et a.. 2024) which emphasize post-training quantization (PTQ) a cost-effective approach avoiding the processes of retraining and fine-tuning which can be particularly costly for LLMs.
A newly emerging approach to optimizing and compressing models through quantization-aware training (QAT) involves the extreme quantization of certain portions of LLMs, using binary {-1,1} and ternary (-1,0, t} weights (Wang et al. 2023 Ma et al., 2024).. This development has initiated the era of 1-bit LLMs. Besides the memory utilization advantages, extreme quantization transforms the costly matrix multiplication (MatMul) operations into more efficient addition and subtraction operations, and thus leading to MatMul- free operations. However, it is important to note that not all MatMul operations can undergo extreme quantization due to the resulting drop in accuracy. Specifically, MatMul operations in the attention heads still require higher precisions, such as 16-bit floating point (FP16) or 8-bit integer (INT8). A follow-up work [Zhu et al., 2024) has built upon BitNet (Wang et al. 2023 aiming to eliminate MatMul operations in attention heads by using Hadamard products.
The advent of 1-bit LLMs has paved the way for various new research directions. However, since they currently address only fraction of the model, crucial question arises:
What effects do partial enhancements in 1-bit LLMs have on the overall performance of the model?
Answering this question is vital for guiding future research effectively and avoiding misguided ineffective goals. For example, if the MatMul operations that are replaced with MatMul-free operations in current 1-bit LLMs account for the majority of the model's computation and memory usage, then focusing on optimizing the relatively minor MatMul operations in attention heads may be less impactful. In this case, prioritizing custom hardware development to fully leverage extreme quantization would be more sensible. Conversely, if the conversion of the fraction of MatMul operations to MatMullfree operations in current 1-bit LLMs does not significantly affect overall computation and memory usage, it would be more prudent to focus on optimizing the MatMul operations in the attention heads- not optimized in 1-bit LLMs than investing in hardware development for current 1-bit LLMs.
In this study, we address the highlighted research question through extensive experiments and analyses on various LLMs. We explore various model hyperparameters across two types of hardware designed for edge and cloud environments. In addition, we propose an adaptation of Amdahl's Law for LLMs to identify how partial improvements LLM can translate into overall enhancements for the entire model. Our findings reveal important nuances that can guide future research in the 1-bit LLMs era.
2
Related Work
2.1 Transformer Quantization
Transformer quantization can be categorized into weight-only and weiglt-and-accivvtion quantization. Weight-only quantization reduces memory requirements, while weight-and-acctivation quantization also en- hances computational efficiency. SmoothQuant (Xiao et al.] 2023) supports both activation and weight quantization, based on the principle that the difficulty of activation quantization can be mathematically transformed. This method demonstrates a 1.5x speedup and 2x memory reduction for LLMs with negligi- ble loss, supporting W8A8 (8-bit weight, 8-bit activation) quantization. Omniquant (Shao et al. 2023] is another method supporting a broader spectrum of weight-activvaionn quantization configurations, including W4A4, W4A16, W3A16, and W2A16. This is achieved through learnable weight clipping, which optimizes the clipping threshold, and learnable equivalent transformation, which mitigates activation outliers by shift- ing them to weights. LLM.ntt() (Dettmers et al. 2022) loads the model in 8-bit format, with 99.9% of operations performed in 8-bit, except for emergent outliers. It uses vector-wise quantization with different normalization constants to preserve model performance. GPTQ (Frantar et al. 2023) focuses solely on weight quantization, achieving 3-bit and 4-bit quantization with speedups of 3.24x x and 4.53x on A6000 and A100 GPUs, Qul## (Tseng et al. 2024) is a weight-only quantization method that achieves state-of- thetth-arrrtt performance sub-4-bit quantization. It employs randomized Hadamard transform combined with vector quantization and then fine-tunes the model to enhance its fidelity. All the methods discussed above are related to PTQ
2

---

Published in Transactions on Machine Learning Research (09/2025)
4
Output
LeyerNorm
Feed Forword Layers
xx1
W:
Decoder Block
W
Add Norm
GELU
Self Attention
Concotonate
df x.
Add & Norm
dx
N
Decoder Block
Feed Forward
EEX
d/hx1
Concalenate
d/b X
4/6
LayerNom
Ado Norm
Decader Input
d/A x
Concalenete
4/h
Softmax
Transpoose
d.x d
4/A x
d/j
Inout Tokeess
Wg
4x1 1
Previous [-1 dx
Value Tokens
Decoder output
d
MatMul Operations
Key Tokens
The typical architecture of decoder-only LLM models and its underlying operations. enization and embedding layers are not shown in the figure.
Figure 1:
The tok-
2.2 Era of 1-bit LLMs
Before the LLM era, the concept of binary weight and activation quantization was explored in works such Binarized Neural Networks (BNN) (Hubara et al. 2016) and Ternary Neural Networks (TNN) (Alemdar ct al. 2017). While these studics did not focus on generative language tasks, they achieved significant per performance improvements, with BNNs demonstrating up to 7x performance gains and TNNs showing up to 3.1x energy efficiency improvements. These findings underscore the remarkable ability of neural networks to operate effectively with just 1-bit precision.
Recently, following the rise of LLMs and advancements in model performance, the BitNet (Wang et al. 2023 introduced 1-bit transformer quantization technique for L.Mss. This method replaces the conventional 'inerr layer in PyTorch with and activations are represented in 8-bit precision. Despite this quantization, other components like self- attention remain in 8-bit. format.. BitNet's design suggests that it can scale to even larger transformer modols, following scaling laws similar to those used for full-procision transformers.
BitLiner layer, where weights are restricted to either
-1,
variant of Bit.Net, known as BitNet 1.58 (Ma et al., 2024); employs ternary weights (-1, 0, 1) and achieves perplexity and end- task performance comparable to full-precision transformers (FP16 or BF16). In terms of the computation, 1-bit LLMs transform MatMul operations into addition operations, due to the 1-bit nature of the weights, except for layers like attention, which need to remain in high precision to maintain performance. Additionally, 1-bit. LLMs address the challenge of transferring model parameters from DRAM to on-chip accelerators such as SRAM, prompting the development of architectures optimized for the efficient operation of 1-bit LLMs.
3 Approach
3.1
Demystifying the Underlying Operations of LLMs
The overall architecture of the generative LLMs shown in Figure which includes N decoder blocks, each consisting of self-attention and feedforward layers followed by add and normalization operations (Vaswani et al. 2017). The core of the LLM is the self-attention mechanism with multiple heads (h). For attention head, the computation begins with three linear projections of the token vector to form the Key (K), Query (Q), and Value (V) vector sequences:
Q @= WQ* II K= W V W
(1)
3

---

Published in Transactions on Machine Learning Research (09/2025)
Block
Description
Dimonsion
We
Attention
WK
(d d) * (d 1)
Projections
Wv
Wx
Attention
Q K
(xxd/h) (d/h x
Head
V * Score
(d/h x 1) (1
Intermediate FF
(dFF d) * (d x
FFN
Output FF
xd drF) * (drr x
Table 1: MatMul Operations and their dimensions in docoder-only LLMs. The parameters d, h, 1, and dFF denote embedding dimension, number of attention heads, sequence length, and the size of the feed-forward layers, respectively.
where I is the input token vector and Wq: WK, and Wy trainable weight matrices with d d dimensions where d is the embedding dimension. The operator * denotes the MatMul operation.
The generated K, Q, and V vectors are then divided into h vectors with reduced dimensionality of d/h where h is the number of attention heads. At this stage, the value and key vectors are concatenated with the previous - value and key tokens that are cached from previous token generation iterations to form a d/h x l matrix in each head, where / is the sequence length. Next, the attention scores are computed using the scaled dot-product of the queries and keys multiplied with the generated value matrix (Vaswani et al. 2017 This step includes two MatMul (x) operations:
Attention (Q.KVV = softmar( Q+ KT
(2)
Vd
Subsequently, the output of the attention heads are concatenated and linearly transformed as follows:
MultiHead(Q.. K, V) = (onaa heady) Wx
where head,- = Attention (Qi, K2, Vi), and Wx is a d x d matrix with trainable elements.
Finally, the attention output followed by a feed-forwe ford-fo network (FFN) that involves two linear and one nonlinear transformations. As shown in Figure Gaussian error linear unit (GELU) (Hendrycks Gimpel 2016) activation function is frequently utilized in LLMs to introduce nonlinearity into FFN. Therefore, the FFN computation can be described as follows, which involves two more MatMul operations:
FFN(x) GELU(x * W,+ + by) * Wo + bo
(4)
where Wi and Wo trainable matrices with deF x d and d dep dimensions, respectively, where deF is the leed-forwarl layer size. Also, the is MultiHead (Q, K, V). Both the self-attention and FFN layers are followed by layer normalization which can also introduce some nonlinearity in the process of division by the standard deviation.
In summary, the computation decoder blocks LLMs involves a combination of nonlinear operations (LayerNorm, softmax, GELU), and lincar MatMul operations. In particular, there are total of 2h + 6 MatMul operations in decoder block including six weight-to-activattion MatMuls (Wq, WK, Wv, Wx, WI, and Wo projections) plus 2h head (h). In decoder-only LLMs, MatMuls are matrix-vector multiplications since the inference is performed iteratively, processing one input token per iteration and the keys and values from previous iterations are cached as shown in Figure I Table lists the dimensions of each of these MatMul operations.
MatMuls to compute Attention K, V) in each
Previous works Kim et al. 2023 have shown that designing dedicated hardware compute nonlincar operations in LLMs can make their computation overhead negligible compared MatMul operations. Con- sequently, optimizing MatMul operations can lead to significant speedup in LLM computation. The 1-bit
4

---

Published in Transactions on Machine Learning Research (09/202))
Feed Forward Layere
Layerhlorm
W
W
Add
GELU
Self Attention
Concelanate
Wo
Concal
Layerboorm
Softmax
Concet
Input
Transpose
W
Output
Wa
MotMul-Free
Cached Value Tokens
Cached Key
MathAu
Figure The 1-bit LLMs divide the model into two portions: attention heads with MatMul operations (shown in rod) and MatMul-free projection layers (shown in green).
Ma et al. 2024 Xu al. 2024b recent approach that has attracted consid- erable attention involves extreme quantization of MatMuls in LLMs, except for the attention heads, which require higher precision to maintain accuracy. Quantizing all the weight matrices (Wq, WK, Wv, Wx. W,, and Wo) in the LLMs to include only binary {} or ternary {-1,0, I] elements transforms the weight- to-activation MatMul operations to simple addition and subtraction operations, dividing the LLM models into two portions one with MatMul and one without MatMul (or (aatMul-free)) as shown Figure 2
LLMs (ang et al. 2023
Due to variations in the model hyperparametens (d, h, and dep) across different types of LLMs, the pro- portion of the linear projections (weight-to-activation MatMuls) and attenton head aactivationtto-activall MatMuls) computation relative to the entire model can vary significantly. Therefore, performance analysis with layer-wise granularity, targeted in this work, can illuminate the effectiveness of the 1-bit LLMs and help determine future research directions.
3.2 Design of LLM-Specific Hardware
While GPUs are commonly used for LLM inference and training due to their general-purpose parallelism, we focus on TPUs to explore design trade-offs specific LLM workloads. Unlike GPUs, TPUs allow directly control memory hicrarchy and dataflows critical for modeling 1-bit LLM performance. Results from GPU-based setups, discussed in Appendix D further contextualize our findings.
For the performance analysis, we leverage TPU architectures which are specifically designed to accelerate the MatMul operations dominant in machine learning workloads. TPUs maximize data reuse while minimizing
data transfer by utilizing systolic arrays at their core (Jouppi 2017 Jouppi et al. 2017).
A systolic array typically consists of two-dimensional arrays of processing elements (PEs). Each PE performs muuliply-and-cccumulat (MAC) operation, multiplying weights and inputs using a multiplier circuit and adding the result to previously computed partial sums using an accumulator circuit. The MAC result is either retained within the same PE or broadcast to other PEs for further computations, depending the dataflow architecture. This MAC operation is executed in every PE of the systolic array, enabling efficient MatMul operations by maximizing data reuse and minimizing additional data transfer overhead.
The dataflow systolic array is a mapping scheme determined by the microarchitecture of the PEs, which dictates how input data fed into the array and how partial results and outputs are generated and stored. Rather than repeatedly loading and storing data to and from memory, each PE typically follows of
5

---

Published in Transactions on Machine Learning Research (09/2025)
Main Controller
Dataflow Generator
Main Memory (DRAM)
Input Memory
Buffers
PE
PE
PE
Nonlinear Functional Unit
Weight Memory
Oulput Memory
Buffers
Systolic Array
The overall architecture of the TPU that is designed for accelerating LLMs, featuring dedicated hardware to support nonlinear operations.
Figure 3:
these dataflow architectures: (1) Input Stationary (IS): The inputs (or activations) remain fixed in the PEs while the weights are sequentially fed into the PEs; (2) Output Stationary (OS): The outputs are attached with the MAC units, while the imputs and weights circulate among the PEs. Inputs and weights are loaded, multiplied, and the results accumulated with partial sums held in the PR. (3) Weight Slationary (WS): Each weight is preloaded into register within cach PE. During each cycle, inputs are multiplied by the weights and broadcast across the PEs.
Figure 3 shows the architecture of TPU, consisting of weight, imput, and output memories, and systolic array of size $ x PEs surrounded by the first-in-first-ont (FIFO) buffers. Additionally, our TPU includes a Nonlinear Punctional Unil, featuring custom hardware to support nonlinear operations the LLMs. The Dataflow Generator block generates the memory read/write addresses to store or retrieve the inputs, weights, and outputs according to the selected dataflow. The Main Controller manages the data transfer between memories, FIFOs, and the systolic array.
As previously discussed, the MatMul operations in generative LLMs involve matrix-vector multiplications. Consequently, the sizes of the input and output vectors are always smaller than those of the weight matrices. For
MatMuls in the attention head, where there are no weight values (See Figure D. we store the concatenated Value and Key matrices (with d/h x and I d/h dimensions, respectively) in the weights memory, while the Query and attention score vectors are stored in the input memory. Based the paltern in the size of the input, weight, and output tensors in matrix-vector multiplications involved in LLMs (mentioned in Table D TPU design with larger weight memory compared to input and output memories would be more efficient, as it reduces the need for costly accesses to the main DRAM memory to load the weights.
For the dataflow architecture, we conducted comprehensive experiments utilizing IS, OS, and WS dataflows. Based on the results obtained (refer to Appendix A ), the OS dataflow architccture demonstrated the best performance. The OS dataflow is particularly advantageous for accumulating results since partial remain stationary and do not need to be moved frequently. Additionally, once weight and input values are fetched from their respective memories, they are reused by passing from one PE to another, leveraging the spatial dataflow capabilities of TPUs.
3.3 Amdahl's Law of LLMs
Since current 1-bit LLMs only improve a part of the model (the projections without enhancing the at- tention heads), we need a mechanism determine how these partial improvements translate into overall enhancements for the entire model. This is crucial for addressing the main research question of the paper.

---

Published in Transactions on Machine Learning Research (09/2025)
The Amdahl's Law provides framework for such scenarios. Amdahl's Law is formula used to find the maximum improvement in system when only part of it is improved. Tt can be expressed as:
Stotal
(5)
1 F
F
Spartiol
where F the fraction of the system that is improved, Spartiat is the factor by which the part F is improved, and Stotal is the overall improvement of the entire system.
In the context of 1-bit LLMs, we define F as the fraction of the MatMul operations that can be replaced with MatMul-free operations by using extreme quantization, relative to all MatMul operations in the model. This formulation enables practitioners to reason about speedup potential (Sportiat) as a function of hardware- specific parameters (c.g., systolic array size) and model hyperparamoters such as embedding size d, context length 1, leedforward dimension dFF, and number of attention heads h. Here, our focus is on quantifying the value of F across various LLMs and hardware configurations. Enhancing Spartial is closely tied to the design of custom hardware accelerators for binary and ternary operations, which beyond the scope of provide context, recent processing-in-mennt (Malekar et al., 2025)) targeting the acceleration of projection layers of 1-bit LLMss demonstrated up to an 80x increase in throughput. In Appendix [F] we provide a more generalized variation of the proposed Amdahl's Law of LLM to enable a more general and hardware-agnostic analysis.
this paper. Nonetheless,
(PIM)- hardware design
4 Experiments
4.1 Simulation Setup
For our experiments, we study 13 different LLMs including GPT, OPT, and LLaMA models. Our work assumes W1A8, W2A8 quantization as the basclinc quantization schemes throughout the analysis. Specif- ically, projection weights are represented using binary or ternary quantization (i.e., 1-2 bits for weights), while activations are maintained at 8-bit integer precision (INT8). This reflects the implementation princi- ples of systems like BitNet (Wang et al., 2023) and BitNet 1.58 (Ma et al. 2024), which preserve accuracy by keeping activations moderate precision while aggressively quantizing weights. Table 2 lists all models and their corresponding hyperparameters. To space in the main body of the paper, only provide the results for the seven OPT models, as they represent a diverse range of model hyperparameter combinations. The results for GPT and LLaMA models are provided in the Appendix
B
For the hardware, we designed two TPUs tailored for different applications: cloud and edge processing. The cloud TPU features a 256 x 256 systolic array with 16MB of SRAM, while the edge TPU has a 32 x 32 systolic array with SMB of SRAM. Both systolic arrays employ an OS datallow. Also, in both designs, 2MB of memory is allocated for internal use, including storing control and configuration data, tracking computation states, managing data flow, and ensuring scamless data movement. All memorics use double buffering to mask the latency associated with SRAM access. Table 3 provides the memory distribution of both edge and cloud TPU designs. We utilize the cycle-accurate SCALF-Sin framework (Samajdar et al. 2018 2020 to measure compute cycles and memory accesses in various LLMs.
Our experiments investigate the distribution of computation and memory utilization between the attention heads, which require MatMul operations, and the projections, which can be binarized or ternarized and consequently do not involve MatMuls. In the results presented, the terms "MatMul" and "MatMul-Free" refer to these respective parts of the LLM computation.
4.2 Performance Analysis on Cloud Setup
In the first set of experiments, we examine various language models deployed on the cloud TPU setup to determine the fraction of the models that can become MatMul-Free 1-bit L.Mss, ie., projection layers, through extreme quantization. To achieve this, we compare seven OPT models of various sizes (ranging from 350M to 66B parameters) with different sequence lengths (ranging from 128 to 4096). We vary the
7

---

Published in Transactions on Machine Learning Research (09/2025)
Hyperparameters
Models
Param
d
h
drr
125M
768
12
768
128-4096
355M
1024
16
1024
128-4096
GPT
774M
1280
20
1280
128-4096
1.5B
1600
25
1600
128-4096
350M
1024
16
4096
128-4096
1.3B
2048
32
8192
128-4096
2.7B
2560
32
10240
128-4096
OPT
6.7B
4096
32
16384
128-4096
13B
5120
40
20480
128-4096
30B
7168
56
28672
128-4096
66B
9216
76
36864
128-4096
7B
4096
32
11008
128-4096
LLaMA
13B
5120
40
13824
128-4096
Table Model
used for simulations.
Systolic Array
Memory Capacity
Design
Size
Input
Output
Weight
Cloud
256 256
4MB
4MB
8MB
Edge
32 x 32
2MB
2MB
4MB
Table 3: The specification of the edge and cloud TPUs.
soquence length (1) because it only affects the computation in the attention heads and determincs the size of the remaining MatMul operations in 1-bit LLMs (refer to Table I
Figure ! exhibits the fraction of MatMul-Free operations in the OPT models deployed on the cloud setup, measured in terms of compute cycles and memory accesses. The fraction of MatMul-Free operations varies from roughly 23% to 98% for compute cycles and from 59% to 99.8% for memory access across different configurations. In general, MatMull-Free operations increase with model size and decrease with sequence length.
4.2.1 Compute Cycle Analysis.
Typically, the context length of the OPT models is equal to 2048. The MatMul-Free compute cycles for these OPT models can be observed in the 2048 row of Figure 4 (a). For smaller language models like OPT 350M, approximately 63% of the computation occurs in the attention heads, involving MatMul operations, while only 37.1 % of the computation benefits from MatMul-Free operations provided by 1-bit LLMs. Moreover, the OPT 1.3B model with a sequence length of 2048 is particularly noteworthy, as half of its computation can be MatMul-Free, while the other half involves MatMul operations.
Models like LLaMA 7B and 13B generally
larger sequence lengths of 4096. In terms of model size, they are comparable to OPT 6. 7B and 13B models. As illustrated in the 4096 row of Figure (a), MatMul-Free operations account. for 64.5% and 69% of the computation for OPT 6.7B and OPT 13B models, respectively. A detailed analysis of the LLaMA models can be found in the Appendix B
4.2.2 Memory Access Analysis.
1-bit LLMs, such as BitNet (Wang et al. 2023 and BitNet 1.58 Ma et al. 2024), can achieve up to 16x and 8x reductions weight memory utilization compared FP16 LLMs because they use just bit and bits represent binary and ternary weights, respectively. Besides memory capacity, another crucial factor affecting LLM throughput is the number of memory accesses required for inference. Therefore, we also analyzed the

---

Published in Transactions on Machine Learning Research (09/2023)
128
0.777
0.858
0.900
0.963
0.973
4096 2048 1024 512 256 128
0.978
0.989
0.991
0.995
0.996
0.997
0.998
4096 2048 1024 512 256
0.759
0.845
0.891
0.950
0.959
0.970
0.976
0.958
0.979
0.983
0.990
0.992
0.994
0.995
Sequence Length
Sequence Length
0.661
0.771
0.834
0.921
0.935
0.952
0.962
0.921
0.959
0.967
0.979
0.983
0.988
0.991
0.524
0.656
0.739
0.868
0.890
0.917
0.934
0.853
0.921
0.936
0.959
0.967
0.976
0.982
0.371
0.505
0.603
0.778
0.812
0.855
0.883
0.745
0.854
0.880
0.922
0.937
0.954
0.964
0.234
0.346
0.440
0.645
0.690
0.754
0.795
0.593
0.745
0.786
0.856
0.881
0.912
0.930
1024
2048
2560
4096
5120
7168
9216
1024
2048
2560
4096
5120
7168
9216
350M
1.3B
2.7B
6.7B
138
30B
350M
1.3B
2.78
6.7B
13B
30B
66B
Embedding Dimension
Embedding Dimension
(a) Compute Cycles
(b) Memory Access
Figure 4: Fraction of Mut-ul-free operations in the OPT models deployed on the cloud setup.
OPT 668
7.5
7.0
-PT
6.5
OPT 0PT-138
6.0
OPT 6.78
0
5.5
-2.78
5.0
OPT 1.38
4.55
OPT--3000
4.0
3.5
3.0
1.5
1.0
20
60
100
Partial Improvement
Figure 5: Amdahl's Law of LLMs for cloud deployment scenario. The dashed lines and solid lines show the effect of partial improvement in projection layers and partial improvement in attention layers, respectively.
memory access of the MatMul-Free and MatMul components of the 1-bit LLM architecture. Figure (b) shows the ratio of memory reads and writes associated with the projection layers MMatMull-Free parts) to those of the entire model.
The results reveal a trend similar the compute cycle analysis. However, unlike compute cycles, for all cases, the majority of memory accesses associated with the projection layers. For example, for OPT models with sequence length of 2048, the fraction of memory access for the OPT 350M is 74%, increasing to 96% for the OPT 66B. The memory access analysis suggests opportunities for research into hybrid memory hierarchy designs, where the components of the model dominating memory accesses can be offloaded to faster memory tochnologics. For example, in OPT 6.7B with sequenco length of 4096, moving the projection layers' weight and activation data to faster memory can speed up more than 85% of all memory accesses.
4.2.3 Amdahl's Law of LLMs.
Here, we leverage Amdahl's Law of LLMs proposed in Equation 5 to show how partial improvements in the LLMs can euhance overall performance. In particular, we vary Spartial from 1 to 100, and using the fractions (F) shown in Figure 4 calculate the overall improvement model (Stota)). Figure demonstrates an Amdahl's Law analysis when improvements are applied to either attention layers (MatMul parts) or the projections layers (MatMul-Free parts) of LLMs. The dashed lines show the effects of improving the projection layers, while the solid lines represent the impact of improvements to the attention layers. This

---

Published in Transactions on Machine Learning Research (09/2023)
128
0.967
0.983
0.986
0.994
0.996
0.997
4096 2048 1024 512 256 128
0.979
0.989
0.992
0.995
0.996
0.997
0.998
4096 2048 1024 512 256
Sequence Length
0.940
0.968
0.974
0.986
0.989
0.992
0.994
Sequence Length
0.959
0.979
0.983
0.990
0.992
0.994
0.995
0.890
0.941
0.951
0.974
0.979
0.985
0.988
0.921
0.959
0.967
0.979
0.983
0.988
0.991
0.805
0.890
0.909
0.950
0.960
0.971
0.977
0.855
0.921
0.936
0.960
0.967
0.977
0.982
0.677
0.804
0.835
0.906
0.923
0.944
0.956
0.746
0.854
0.880
0.922
0.937
0.954
0.964
0.512
0.673
0.718
0.829
0.858
0.894
0.916
0.595
0.746
0.786
0.856
0.881
0.912
0.930
1024
2048
2560
4096
5120
7168
9216
1024
2048
2560
4096
5120
7168
9216
350M
1.3B
2.7B
6.7B
138
30B
66B
350M
1.3B
2.78
6.7B
13B
30B
66B
Embedding Dimension
Embedding Dimension
(a) Compute Cycles
(b) Memory Access
Figure 6: Fraction of MatMul-froe operations the OPT models deployed on the edge setup.
analysis is based on OPT models with a typical sequence length of 2048. For Amdahl's Law analyses with other sequence lengths, please refer to the Appendix C
The Amdahl's Law analysis of LLMs deployed on the cloud setup reveals three key takeaways:
1. For smaller language models like OPT 350M, improving the attention heads has greater impact than enhancing the projection layers. Therefore, using 1-bit. LLM paradigms to improve these models may result in limited overall performance gains.
2. Medium-sized LLMs, such OPT 1.3B and 2.7B, can benefit from combining 1-bit LLM im- provements with enhancements to the attention heads, such as replacing MatMul operations with Hadamard products and additive operators as proposed in (Zhu el al. 2024
3. For larger models like OPT 6.7B and above, improvements to the attention heads have minimal effect overall performance. In these cases, employing 1-bit LLM methods alone can lead to significant gains in performance and throughput.
4.3 Performance Analysis on Edge Setup
Figure 10 presents the compute and memory analysis for the edge setup. In all cases, most of the compu- tations occur in the MatMul-Free portion. The only instance where the computation relatively evenly distributed between the MatMul and MatMul-Frre parts is for OPT 350M with 4096 sequence length. A similar pattern is observed in memory accesses. This happens because, in the edge setup, the smaller 32 x 32 systolic array is less efficient at handling the larger matrices typically found in projection layers. Meanwhile, the MatMul operations in the attention heads are smaller due to the splitting of large matrices among mul- tiple heads (refer to Figure [ making these operations more manageable even with smaller systolic arrays. Consequently, this increases the ratio computation the projection layers compared to the attention heads.
The difference between the cloud and edge setups stems from their scaling characteristics. In cloud TPUs (256x256 arrays), large MatMuls in projection layers efficiently handled due ample compute and memory bandwidth, Icading to pronounced compute savings for MatMullfree sections. In contrast, edge TPUs (32x32 arrays) suffer degraded efficiency on large projections due to limited parallelism and SRAM, capping compute benefits despite similar memory access trends.
Additionally, the cloud sctup faces underutilization of processing elements when executing smaller MatMuls in attention heads, whereas the edge setup rarely encounters this issue. This helps explain the observed difference in the compute cycle fraction (F) between edge and cloud environments, though the memory
10

---

Published in Transactions on Machine Learning Research (09/2025)
OPT-66B
OPT 300
2 15
14
OPT-6.78
E 13
0,7-
Overa 1mm110
0011
11
OPT 1.38
10
OPT 350M
0
20
40
60
80
100
Partial Improvement
Figure 7: Amdahl's Law of LLMs for edge deployment sccnario. The dashed lines and solid lines show the effect of partial improvement in projection layers and partial improvement in attention layers, respectively.
access patterns remain largely similar across both. We include additional experiments in Appendix E where the embedding dimension is fixed (d = 4096) and the number of attention heads is varied, to isolate the impact of attention head MatMul dimensions on the compute and memory balance.
4.3.1 Amdahl's Law of LLMs.
Figure 7 illustrates the Amdahl's Law analysis for the OPT models with 2048 sequence length deployed the edge TPU setup. The results indicate that, unlike in the cloud setup, across all cases, enhancing the projection layers leads to significantly greater overall improvements in the entire model. Conversely, improving the attention layers yields only marginal gains. This suggests that 1-bit LLM approaches targeting optimization of projection layers are significantly more efficient in the edge setup. Therefore, a promising research direction would be to focus on developing efficient custom hardware for implementing extremely quantized projection layers, rather than concentrating on algorithmic and hardware innovations to enhance computation in the attention heads.
5 Conclusion and Future Work
The proposal of 1-bit LLMs has opened several research avenues, including the development of custom hard- ware for 1-bit LLMs as well as algorithmic innovations to enhance aspects of LLM computation that 1-bit LLMs cannot address. Here, we aimed to provide a roadmap to avoid fundamentally incorrect or inelficient research goals in this field. We introduced the concept of Amdahl's Law LLMs, which helps determine how partial improvements from 1-bit LLMs translate into overall model enhancements. We conducted extensive evaluations across different LLMs with various hyperparameters to identify relevant patterns. The results reveal a significant dependency of 1-bit LLM efficacy on model sizes, hyperparameters, and hardware con- figurations. Key findings include: (i) 1-bit LLM paradigms have limited impact on smaller language models, particularly when the context length is large, (ii) for medium-sizae sized LLMs, 1-bit LLM methods show benefits, but further algorithmic innovations are needed to enhance parts that 1-bit L.M approaches cannot improve, and (iii) for large-scale LLMs, extreme quantization from 1-bit LLMs alone can improve the majority of computations, in some cases by more than 99%. While the observation that linear projection layers increas- ingly dominate compute at larger model scales may be expected from dimensional analysis, contribution lies in quantifying this shift precisoly across a wide range of real-world LLM configurations (OPT, GPT, LLaMA) using cycle-accurate sinulations on custom-designed TPU architectures. This approach moves beyond theoretical speculation and establishes an empirical foundation for throughput bottlenecks under extreme quantization regimes.
The proposed rescarch opens up several promising directions for future work, including: (i) expanding the de- sign space to encompass emerging model variants, such multi-query attention, Mixture-of -fxpretsss (MoF),
11

---

Published in Transactions on Machine Learning Research (09/2025)
and linear attention, and studying their impact on F, the overall bottleneck profile, and optimal hardware al- location; and (ii) integrating memory-efficient attention mechanisms (e-g., Flash Attention (Dao et al. 2022) PagedAttention @Kwon et al., 2023)) ) and refining the analysis within hardware-speciffe implementations to evaluate their effects on memory access patterns.
Acknowledgments
This work supported by the National Science Foundation (NSF) under grant numbers 2340249 and 2409697.
References
Angus Addlesce, Neeraj Cherakara, Nivan Nelson, Daniel Hernández García, Nancic Gunson, Weronika Sieińska, Marta Romeo, Christian Dondrup, and Oliver Lemon. A multi-parly conversational social robot using 1mms. In Companion of the 2024 ACM/TEEE International Conference on Human- Robot Interaction, pp. 1273 1275, 2024.
Hande Alemdar, Vincent Leroy, Adrien Prost-Boucle, and Frédéric Pétrot. Ternary neural networks for resource-efficint ai applications. In 2017 international joint conference on neural networks (LJCNN, pp. 2547-2554. IEEE, 2017.
Mahsa Ardakani, Jinendra Malekar, and Ramtin Zand. Llmpi: Optimizing Ilms for high-throughput raspberry pi. In Proceedings of the Computer Vision and Pottern Recognition Conference, pp. 8879-6388, 2025.
Amir Resa Asadi, Joel Appiah, Siddique Abubakr Muntaka, and Jess Kropezynski. Actions, not apps: To- ward using lims to reshape context aware interactions in mixed reality systems. In International Conference on StmannCoompule Interaction, pp. 166-176. Springer, 2024.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-effiiiet exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16359, 2022.
Tim Dettmers, Mike Lewis, Youncs Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318 15330038-30332, 2022.
Iván Esteban-Lozano, Álvaro Castro-González, and Paloma Martínez. Using a llm-based conversational agent in the social robot mini In International Conference on Human- Computer Interaction, pp. 15-26. Springer, 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for gen- erative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2016. URL https://arxiv..rrg abs/1606.0844
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv,, and Yoshua Bengio. Binarized neural networks. Advances neural information processing systems, 29, 2016.
Norm Jouppi. Quantifying the performance of the tpu, our first machine learning chip. Google Cloud Platform Blog, Google, 5, 2017.
Norman Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proc. of the 44th Annual Int. Symp. on Comput. Architecture, pp. 12, 2017.
Schoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Gene, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael w Mahoney, et al. Full stack optimization of transformer inference: survey. arXio preprint arXi:2200..14000 14017, 2023.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonza- lez, Hao Zhang, and Ton Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp. 611 626, 2023.
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit Ilms: All large language models are in 1.58 bits. arXiv preprinl or 21v:24002. 17764, 2024.
Xinyin Ma, Gonglan Fang, and Xinehao Wang- Llm-pruner: On the structural pruning of large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (ods.), Advances in Neural Information Processing Systems, volume 36, pp. 21702-21720. Curran Associates, Inc., 2023. URL httpp:://proceedinggn wttp::s..rrrrrpcccccc...o httppppppr..llee/ppp//////////// 1.958511400674992441122878888880 Poapeaareeeppppeerroooorreeeeeee Conference...
Jinendra Malekar, Peyton Chandarana, Md Hasibul Amin, Mohammed E Elbtity, and Ramtin Zand. Pim- Ilm: A high-throughput hybrid pim architecture for 1-bit llms. arXiv preprint arXiv:2504 0199// 2025.
Peter Morales and Richard Alexander Showalter-Bucher Towards large language models at the edge on mobile, augmented reality, and virtual reality devices with unity. In System and Architecture for Generative AT on the Edge/ Mooble Platforms (SAGE), 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Tyy Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Brendan C Reidy, Mohammadreza Mohammadi, Mohammed E Elbtity, and Ramtin Zand. Efficient de- ployment of transformer models on edge tpu accelerators: real system evaluation. In Architecture and System Support for Transformer Models (ASSYST@ ISCA 2023), 2023.
Ananda Samajdar, Yuhao Zhu, Paul Whatmough, Matthew Mattina, and Tushar Krishna. Scale-sim: Sys-
tolic cmn accelerator simulator. arXiv preprint arXit:1811.02885 2018.
Ananda Samajdar, Jan Moritz Joseph, Yuhao Zhu, Paul Whatmough, Matthew Mattina, and Tushar Kr- ishna. A systematic methodology for characterizing scalability of dnn accelerators using scale-sim. In 2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 58 68. IEEE, 2020.
Jesús Sánchez Cuadrado, Sara Pórez-Soler, Esther Guerra, and Juan De Lara. Automating the development of lask-oriented Ilmebased chatbots. In Proceedings of the 6th ACM Conference Conversational User Interfaces, pp. 1-10, 2024.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidiroctionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13133 13137, 2023.
Jovan Stojkovie, Esha Choukse, Chaojie Zhang, Inigo Goiri, and Josep Torrellas Towards greener 1lms: Bringing energy-efficiency to the forefront of Ilm inference. arXiv preprint w:Xiv:2403.20366 2024.
Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. Can chatgpt replace traditional kbqa models? an in-depth analysis of the question answering performance the gpt 11m family. In International Semantic Web Conference, pp. 348 367. Springer, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ann Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint Xrx:::2302.1999 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better Ilm quantization with hadamard incoherence and lattice codebooks, 2024. URL htttps://arxtt: http://////xvvvvoo///..../ 2402. 04396
13

---

Published in Transactions on Machine Learning Research (09/2025)
Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. Improving Ilm code generation with grammar augmentation. arXiv preprint #Xiu:2403.01692 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXi::2310.1145. 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint arXiv:2401.08/17 2024a.
Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely bit large language models. arXiv preprint 1Xx::2402.11295 2024b.
Haoran You, Yipin Guo, Yichao Fu, Wei Zhou, Huihong Shi, Xiaofan Zhang, Souvik Kundu, Amir Yazdan- bakhsh, and Yingyan Lin. Shiftaddllm: Accelerating pretrained 1lms via post-training multiplication-les reparameterization.. arXiv preprint arxiv:2406. 05981, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprinl or 220. 01068, 2022.
Rui-Jie Zhu, Yu Zhang, Ethan Sifferman, Tyler Sheaves, Yiqiao Wang, Dustin Richmond, Peng Zhou, and Jason K Eshraghian. Scalable matmul-free language modeling. arXio preprint or Xix:22000 06. 02528, 2024.
14

---

Published in Transactions on Machine Learning Research (09/202))
A TPU Dataflow Analysis
12 .1M
Source
14.0M
Output Stationery
Input Stationery
12.0M
Weight Stationery
Total Compute Cycles
10.0M
7.9M
8.0M
6.0M
4.17M
75M
9.7M
3.16M
4.0M
6.33
1.77M
1.31M
2.0M
133
4.46
1 6.98M
5.59
0 0.68M
1.1M
DOO
OPT-125MOPT-350M OPT-1.38 OPT-2.7B OPT-6.7B
OPT-138
OPT-30B
OPT-66B
Model
(a): Cloud Setup
98 06M
Source
105M
Output Stationery
Input Stationery
90M
Weight Stationery
Total Compute Cycles (M)
75M
59 M2M
60M
45M
31. 13M
MMM
33,73M
7.13M 20.23M
20.5 S5M
20.73 73M
30M
10.88M
15M
9.41M 1M
0.95M
.00
5.45M
8.466M 8.
7.02M
Blnn
OPT-125MMPT-350M
OPT-1.3B
OPT-2.7B OPT-6.7B
OPT-13B
0PT-30B
OPT-66B
Model
(b): Edge Setup
Figure 8: Total cycles for different dataflow of OPT models.
All experiments were conducted on a computing setup with a single node, 48 cores, and 200 GB of memory. The Scale-Sim V2 tool was installed, and experiments were executed using specific configurations for both cloud and edge setups.
15

---

Published in Transactions on Machine Learning Research (09/2025)
B
GPT and LLaMA Results
128
0.613
0.655
0.689
0.746
0.943
0.953
4096 2048 1024 512 256 128
0.946
0.959
0.967
0.973
0.993
0.995
Context Length
4096 2048 1024 512 256
0.589
0.632
0.667
0.726
0.938
0.949
Sequence Length
0.898
0.921
0.936
0.948
0.987
0.989
0.469
0.621
0.815
0.880
0.902
0.334
0.375
0.412
0.481
0.838
0.864
0.688
0.746
0.786
0.821
0.949
0.959
0.211
0.244
0.273
0.332
0.734
0.773
0.525
0.595
0.647
0.696
0.903
0.921
0.122
0.143
0.163
0.205
0.588
0.637
0.356
0.424
0.479
0.534
0.823
0.853
gpt2 smail
768
gpt2 medium
1024
gpt2-Harge
1280
1600 gpt2-xl
Ilamaa7b
4096
Ilama-130
5120
gpt2 small
768
gpt2 -mediu
1024
gpt2 gpt2-4arge
1280
1600 gpt2-xl
4096 llamaa11 7b
5120 Ilama-13) 13b
Embedding Dimension
Embedding Dimension
(a): Compute Cycles
(b): Memory Access
Figure 9: Fraction of MatMul-free operations in the GPT and LLaMA models for the cloud setup.
128
0.919
0.937
0.948
0.958
0.990
0.992
128
0.946
0.959
0.967
0.973
0.993
0.995
Context Length
4096 2048 1024 512 256
0.859
0.888
0.907
0.924
0.982
0.986
Sequence Length
4096 2048 1024 512 256
0.898
0.921
0.936
0.948
0.987
0.989
0.759
0.805
0.836
0.863
0.967
0.973
0.815
0.854
0.880
0.902
0.974
0.979
0.616
0.677
0.722
0.763
0.937
0.949
0.688
0.746
0.786
0.821
0.949
0.959
0.448
0.515
0.567
0.619
0.883
0.904
0.525
0.595
0.647
0.696
0.903
0.921
0.289
0.348
0.397
0.449
0.792
0.826
0.356
0.424
0.479
0.534
0.823
0.853
gpt2-small small
768
gpt2 gpt2-medium medium
1024
gptZ-large
1280
1600
gpt2-xl
Ilama-7b
A096
Illma-11 -1b
5120
gpt2-smal small
768
gpt2 2112-medium
1024
gpt2-large
1280
1600
gpt2-xl
llama-7b
4096
5120 Baaa-11 11b
Embedding Dimension
Embedding Dimension
(a): Compute Cycles
(b): Memory Access
Figure 10: Fraction of MatMul-fres operations in the GPT and LLaMA models for the edge setup.
All experiments were conducted on a computing setup with a single node, 48 cores, and 200 GB of memory. The Scale-Sim Sim V2 tool was installed, and experiments were executed using specific configurations for both cloud and edge setups.
16

---

Published in Transactions on Machine Learning Research (09/2025)
C Amdahl's Law Analysis of OPT Models for Various Sequence Lengths
OPT 66B
-68
30
OPT 308
-PB
Overall Improvement
OPT
Overall Improvement
OPT
OPT 6.78
OPT 6.78
20
OPT 2.78
20
OPT 2.78
OPT 1.38
OPT 1.38
15
OPT
15
OPT
0
20
40
60
80
100
0
20
40
60
80
100
Partial Improvement
Partial Improvement
(a): Sequence Length 128
(b): Sequence Length 256
20.0
OPT 66B
OPT 66B
30B
12
3PB
OPT
OPT
Overall Improvement
OPT 6.78
10
OPT 6.78
OPT 2.78
OPT 2.78
12.5
OPT 1,38
OPT-1.38
10.0
OPT 350M
OPT 350M
7.5
5.0
2.5
0
20
40
60
80
100
0
20
40
60
80
100
Partial Improvement
Partial Improvement
(c): Sequence Length 512
(d): Sequence Length 1024
OPT 668
4.5
OPT 668
7
-3B
OPT 30B
Overall Improvement
OPT 13B
4.0
-
6
6.78
OPT 6.78
3.5
-27-2.7B
OPT 2.78
OPT 1.38
3.0
OPT 1.38
OPT 350M
2.5
2.0
2
1.5
1.0
20
40
60
80
100
20
40
60
80
100
Partial Improvement
Partial Improvement
(e): Sequence Length 2048
(f): Sequence Length 4096
Figure 11: Amdahl's law for different sequence lengths in the OPT models for the cloud setup.
17

---

Published in Transactions on Machine Learning Research (09/2025)
OPT 668
60
OPT-66B
70
OPT 308
OPT
Overail Improvement
OPT 138
Overail Improvement
50
OPT
OPT 6.7B
OPT 6678
OPT 2.78
OPT
40
1.38
-
30
30
10
10
0
20
40
60
80
100
0
20
40
60
80
100
Partial Improvement
Partial Improvement
(a): Sequence Length 128
(b): Sequence Length 256
OPT 668
OPT 668
40
OPT 30B
OPT
Overail Improvement
OPT 13B
Overall improvementt
25
OPT
OPT 6.78
OPT 6.78
OPT 2.77
OPT 2.78
OPT 1.38
OPT 1.38
15
350M
20
10
10
0
20
40
60
80
100
0
20
40
60
80
100
Partial Improvement
Partial Improvement
(c): Sequence Length 512
(d): Sequence Length 1024
O 66B
OPT 666
17.5
OPT 30B
OPT
Overall Improvement
15.0
OPT
Overall Improvement
OPT 13B
OPT-6.78
OPT 6.78
12.5
OPT 2.78
OPT 2.78
-
OPT 1.38
10.0
OPT 350M
OPT
7.5
5.0
2.5
20
40
60
80
100
0
20
40
60
80
100
Partial Improvement
Partial Improvement
(c): Sequence Length 2048
(f): Sequence Length 4096
Figure Amdahl's law for different sequence lengths the OPT models for the edge setup.
All experiments were conducted on a computing setup with a single node, 48 cores, and 200 GB of memory. The Scale-Sim V2 tool installed, and experiments were exccuted using specific configurations for both cloud and edge setups.
18

---

Published in Transactions on Machine Learning Research (09/2025)
D GPU Results
128
0.953
0.978
0.988
0.978
0.991
0.995
128
0.948
0.979
0.985
0.992
256
0.949
0.981
0.988
0.981
0.992
0. 996
Sequence Length
Sequence Lengtth
256
0.967
0.979
0.985
0.990
512
0.960
0.988
0.993
0.963
0.996
0.998
512
0.957
0.979
0.986
0.988
0.996
0.991
0.995
0.989
0.998
0.990
1024
0.987
0.992
0.999
2048
0.985
0.998
0.998
0.995
0.999
0.999
768
1024 gpt2-medium 9pt2-large Embedding Dimension
1280
1600
768
2048
4096 Liama-78 Embedding Dimension
2560
4096
7168
gpt2
gpt2-xl
OPT-125M
OPT-1:38
OPT-2.78
OPT-6.78
OPT-138
(b): GPT Latency
(a): OPT and Llama Latency
Figure 13: Percentage of latency of projection layers measured using GPU. All experiments were conducted on a computing setup with a single node, 40 cores, NVIDIA Tesla V100 with 32 GB of RAM.
3pt2
3pt2
50
gpt2-medium
gpt2-medium gpt2 apt2-large
gpt2
Overall improvement
gpt2.xl
Overall Improvement
gpt2.xl
40
30
20
40
60
80
100
0
20
40
60
80
100
Partial Improvement
Partial Improvement
(a): Sequence Length 128
(b): Sequence Length 256
gpt2
gpt2
gp(2-medium
gpt2-medium
gpt2 large
80
gpt2 large
Overall Improvement
gpt2-xl
Overall Improvement
gpt7-xl
30
40
10
20
40
60
80
100
0
20
40
60
80
100
Partial Improvement
Partial Improvement
(c): Sequence Length 512
(d): Sequence Length 1024
Figure 14: Amdahl's law analysis on GPU of GPT models.
19

---

Published in Transactions on Machine Learning Research (09/2025)
70
Ueme-70
70
Uame-78
OFT-175M
GFT-125M
0FT-1.30
60
OFT-1.38
Overall Improvement
0BT:2.BB
Overall Improvement
OFT-2.78
OFT-6 76
50
OFT-13B
40
40
30
20
0
20
40
60
80
100
0
20
40
60
80
100
Partial Improvement
Partial Improvement
(a): Sequence Length 128
(b): Sequence Length 256
80
Uama-78
Uama-78
GPT-125M
GPT-125M
70
36
OFT-1.38
Overall Improvement
50
OPT-2.76
Overall Improvement
OFT-2.70
60
OFT:13B
OFT:13B
40
20
10
0
20
40
60
80
100
0
20
40
60
80
100
Partial Improvement
Partial Improvement
(c): Sequence Length 512
(d): Sequence Length 1024
Uama-78
OPT-125M
OPT-1.38
Overall Improvement
OFT-2.70
6.76
60
DFT:138
40
20
0
20
40
60
80
100
Partial Improvement
(c): Sequence Length 2048
Figure 15: Amdahl's law analysis GPU of OPT and LLama models.
The GPU latency analysis is intended to contextualize the TPU-centric findings and highlight that GPU-optimized transformer kernels the projection layers remain the dominant latency contributor, especially
in large models. For instance, Figure 12(a) shows that in LLaMA and OPT models, projection layers account for more than 95% total latency, reinforcing our claim that quantization of projection layers yields substantial benefits across hardware backends.
20

---

Published in Transactions on Machine Learning Research (09/2025)
E
Analysis of Attention Head Count
128
0.964
0.960
0.952
0.916
0.851
0.746
4096 2048 1024 512 256 128
0.990
0.990
0.990
0.989
0.989
0.988
256
0.960
0.956
0.948
0.908
0.838
0.726
0.980
0.979
0.979
0.979
0.978
Sequence Length
Sequence Length
512
0.939
0.931
0.915
0.858
0.761
0.621
0.960
0.960
0.960
0.959
0.959
0.957
4096 2048 1024
0.900
0.885
0.857
0.772
0.643
0.483
0.923
0.923
0.923
0.922
0.921
0.918
0.831
0.806
0.761
0.643
0.492
0.334
0.857
0.857
0.856
0.855
0.853
0.849
0.720
0.684
0.621
0.483
0.334
0.207
0.750
0.750
0.749
0.748
0.744
0.739
16
32
128
16 Number of Heads
128
Number of Heads
(a): Compute Cycles
(b): Memory Access
Figure 16: Cloud TPU Setup. Heatmaps illustrate the fraction of Mattuul-Free operations the projec- tion layers across different numbers of attention heads and sequence lengths, with the embedding dimension fixed at d 4096. As the number of heads increases (with fixed d), the MatMul dimensions within each allention head decrease, leading to under-utilization of large (256x256) systolic arrays in the cloud setup. This results in larger compute cycles in the attention heads, for instance, with h 128, 79.93% of compute cycles occur in the attention heads, while only 20.7% 10.%% are in the projection layers.
128
0.987
0.987
0.986
0.985
0.983
0.978
4096 2048 1024 512 256 128
0.990
0.990
0.990
0.990
0.989
0.989
256
0.977
0.976
0.975
0.973
0.968
0.959
0.980
0.979
0.979
0.979
0.979
0,979
Sequence Length
Sequence Length
4096 2048 1024 512
0.957
0.956
0.954
0.949
0.940
0.923
0.960
0.950
0.960
0.960
0.959
0.958
0.920
0.918
0.914
0.905
0.889
0.859
0.923
0.923
0.923
0.922
0.921
0.920
0.854
0.850
0.843
0.829
0.754
0.857
0.857
0.856
0.856
0.854
0.852
0.746
0.740
0.730
0.709
0.671
0.607
0.750
0.750
0.749
0.748
0.746
0.742
16
32
64
128
16
32
64
128
Number Heads
Number Heads
(a): Compute Cycles
(b): Memory Access
Figure 17: Edge TPU Sctup. Hcatmaps show the ratio of MatMul-Free operations in projection layers across varying attention head counts and sequence lengths, with the embedding dimension fixed at d 4096. On edge TPUs, smaller attention head MatMuls can be efficiently handled by small systolic arrays (e.g.. 32x32) without incurring the overhead of data movement between processing elements seen in larger arrays. As a result, most the clock cycles still happen the projection layers.
21

---

Published in Transactions on Machine Learning Research (09/2025)
F Expanding the Generalization and Applicability of Amdahl's Law for LLMs
While our simulations currently focus on TPUs as representative hardware bascline, we recognize that modern LLM deployment spans a diverse range of hardware largets. These include general-purpose GPUs (such as NVIDIA's 's A100 and H100), reconfigurable platforms like FPGAs, application-specific integrated circuits (ASICs), and emerging memory-centric architectures such as processing-in- in-memory (PIM) accel- erators. Each of these platforms introduces unique tradcoffs compute throughput, memory bandwidth, energy efficiency, and support for sparsity or quantization.
To enable more general and hardware-agnostic analysis, the proposed Amdahl's Law of LLM can be expanded to explicitly factor in the non-uniform acceleration characteristies of various LLM components. Instead of assuming a monolithic speedup across the entire workload, this formulation distinguishes among different layers or operations within the modcl, specifically, the projection layers, attention heads, and leedforward networks (FFNs), and allows their respective speedups to be expressed independently. This can be modeled as:
Speedup Non-Quantized Portion + f(pro) Sa
(6)
fthead)
((FF) Say
SA.
Here, f(-) denotes the fractional contribution of each component to the overall computation time in the baseline (unaccelerated) system. The terms Sa, Sn, and Say represent the realized speedup of the pro- joction layers, attention heads, and feedforward components, respectively, on the target hardware. The "Non-Quantized Portion" lerm accounts for operations that remain unaccelerated either due to algorith- mic constraints or hardware limitations (e.g., softmax normalization, residual connections, or embedding lookups). These speedup terms are determined by the characteristics of the underlying hardware architec- ture. For example, in our case, the speedups reflect the performance of systolic arrays, which vary in size and efficiency across cloud and edge platforms. However, our model is flexible enough to incorporate other architectural paradigms. For instance, PIM-based accelerators may be particularly effective at accelerating projection layers, while systolic arrays or GPUs may offer greater advantages for malrix-heavy feedforward operations. This flexibility enables modeling of hybrid systems, where different layers of the LLM are mapped to specialized hardware for optimal efficiency.
By explicitly decoupling the model architecture from the hardware acceleration characteristics, this formu- lation allows practitioners to reason about performance tradeoffs in modular way and extend the analysis to heterogencous or mixed-procision systems.
22


Reviews:
[{"review_id": "AtfOKO1hWC", "reviewer": "Reviewer_mZtb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The paper provides a simple yet general quantitative framework to reason about the performance limits of partial model optimization.\n\n2. The study is supported by extensive experiments covering different LLMs across the OPT, GPT, and LLaMA families, with varying sequence lengths. Analyzing performance on two distinct, custom-designed hardware platforms (cloud and edge) further strengthens the conclusions.\n\n3. The results provide a clear and valuable insights. The key takeaway—that the optimal focus of optimization shifts from attention heads to projection layers depending on model size and hardware—is a practical insight that can help guide research efforts.\n\nWeaknesses: \n\n1. The paper does not explicitly state whether its analysis assumes weight-only or weight-and-activation quantization. While it mentions that activations remain in 8-bit precision in some 1-bit models, a clearer definition of the assumed quantization scheme (e.g., W1A8 or W1A16) is needed.\n\n2. The analysis focuses heavily on calculating the fraction of the model that can be improved (F) but states that analyzing the speedup factor of that part is beyond its scope. The paper would be strengthened by a sensitivity analysis or discussion on how different, realistic speedup values for MatMul-free hardware might alter the conclusions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an adaptation of Amdahl's Law, to provide a quantitative framework for analyzing the throughput limits of 1-bit LLMs. Through experiments on different models and hardware platforms, the authors reveal that the primary performance bottleneck shifts between attention heads and projection layers depending on model size and hardware. This analysis provides a roadmap for focusing future research on the more impactful component.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clarify the assumed quantization scheme or methods.\n\n2. Include a sensitivity analysis for partial speedup improvement.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BO9v0jfBH4", "reviewer": "Reviewer_VFaP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**strengths**\n- the topic of improving LLM performance via quantization and related optimization strategies is of interest to the audience of this journal\n- the paper is easy to follow, and for the most part well organized\n\n**weaknesses**\n- in my opinion, innovation is very limited: the observation that linear layer operations are more dominant for larger model sizes and smaller sequence lengths can be derived from basic mathematical considerations on tensor sizes, and are thus in line with expectation of practitioners. Amdahl's law application does provide a quantitative expectation on the improvement, but it is simply being applied to the problem at hand, with no particular \"adaptation\" nor being \"tailored\" (unlike the authors' claims). TPU measurements are of interest but I have concerns regarding the generality of the results (see next point). Consequently, the practical value of this work is also very limited, in my view\n- memory access are strongly impacted by the choice of attention implementation (such as various memory-efficient variants on TPU or, on GPU, scaled dot product attention, flash attention, paged attention, etc.). However, this aspect is not discussed\n- the implications of using Cloud vs. Edge setup for TPU are not well explained, beyond listing systolic array size and memory capacity in Table 3: why does the Cloud setup show more pronounced trends in terms of compute, while memory access are almost identical (fig 4 vs fig 6)?\n- GPU latency results in appendix D show hardly any dependence and are not being discussed, so it's hard to argue that they contextualize the TPU measurements\n\n**other minor comments**\n- fig 4 is unintuitively plotted with the sequence length axis decreasing from bottom to top\n- fig 5 and 7 (Amdahl's law plots) are not easy to read, with several overlapped lines and a legend that doesn't include the solid lines\n- typo in section 4.3: Figure 10 should be Figure 6", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an analysis on the relative impact of linear layer operations vs. attention matmuls in 1-bit quantized transformers. With binary or ternary weights (1-2 bits), linear layer operations can be replaced by more efficient additions and substraction, and are thus denoted \"MatMul-free\". The authors apply Amdahl's law to estimate how an improvement in these MatMul-free operations vs. attention matmuls would translate in overall model improvement, thus drawing model-dependent recommendations on where optimization efforts should be placed. Compute cycles and memory accesses are examined across 3 model architectures (GPT, OPT, Llama) of varying sizes, with primary focus on 2 TPU setups. GPU results are provided in appendix.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Could the authors please comment on the weaknesses brought up above?", "broader_impact_concerns": "Not discussed in the manuscript. No concerns on my end.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lj0tLkRB13", "reviewer": "Reviewer_dcc9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- the authors provide an analysis of how basic model architecture choices (embedding dimension, sequence length) affect the tradeoffs\n- the authors provide extensive numeric data for the aspects of model/hardware design they explored, based on their simulations\n- the authors are careful to pay attention to memory access costs, as these too can be a bottleneck or source of inefficiency in DL systems\n- the discussion offers clear, actionable advice for when to use this kind of quantization, and the simulations offer a realistic perspective on what the benefits really are\n\nWeaknesses:\n- I am not sure if some of the conclusion here provides useful insights that could not be inferred from Amdahl's law; in other words. It seems like a somewhat trivial conclusion that the speedup from quantization would be limited by those operations that could not be quantized.\n- I believe the value here is from the authors' use of simulation, because their numbers appear to be derived from a model of real-world inefficiencies, idiosyncrasies, and processes that otherwise make them hard to guess a-priori. But the paper does not apply much effort towards explaining these intricacies, and what interesting insights - if any - the authors learned from simulations compared to plugging estimates of model FLOPs directly into the formula for Amdahl's law. For example, were there any cases where the hardware struggled with a particular architecture, or where energy/throughput was worse than expected because of how the Transformer architecture/scale interacted with the systolic array behavior? Did certain implementation details affect the numbers? The base idea of Amdahl's law is obvious, but findings about the quirks of hardware and models in simulation might provide more ideas about why these tradeoffs occur and what could be done about them. Currently, it is unclear to me what new knowledge, if any, is gained by doing the simulations.\n- Relatedly, because others are aware of the limitations of accelerating non-attention operations in Transformers, people have tried implementing alternative architectures to work around these issues. What would be the results of simulating on multi-query attention, sparse-expert, or linear-attention models? These I think would be useful takeaways from an Amdahl's-law-style analysis: these are much newer architectures claiming to produce an advantage by reducing some sort of attention-related overhead, for which there is far less rigorous work studying their real-world benefits based on benchmarks and their effects on hardware utilization. Some of these are key advances in the design of Transformer models that are now used a lot in practice, and so I believe that investigating them within this framework is necessary for the results of this work to be useful to practitioners.\n- Models may reduce the memory intensiveness of multi-head attention by using larger and fewer attention heads. In terms of studying the effect of transformer design, the number of attention heads is an important hyperparameter that I do not believe is sufficiently considered in this work, given that attention itself can often be the main performance bottleneck of interest. Indeed, hyperparameters like attention heads do change in the models the authors simulate, but appear not to have their effects analyzed explicitly, such as in Figure 4/5. Also, when plotting results in terms of embedding size and sequence length, it is hard to disentangle whether the final numbers are affected by the change in these two parameters or change in number of attention heads.\n\nThe result of these limitations is that the analysis of specific hardware simulation and only certain architecture variables in LLMs mean that the scope of the conclusions' applicability to real-world cases is limited; readers may have a hard time figuring out if the results here are actually predictive for their particular setup or choice of hyperparameters; there is not enough evidence and cases considered to make a convincing general argument about what Amdahl's law looks like for LLMs in different configurations of model and hardware, in my opinion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present an analysis of the computational benefits of aggressively quantizing the MLP layers of large-scale Transformer models, using simulations of machine-learning accelerator hardware. The key point is that Amdahl's law limits the benefits that can be had based on the model size, because some portion of the computation/overhead is due to the attention blocks, which are harder and less commonly quantized. This leads to conclusions about when and where quantization can have a benefit in language models, with concrete numbers and data for practitioners to reference.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This work provides an analysis of LLMs from the perspective of Amdahl's law and its interaction with quantization, but does analyze how different model designs, hardware, and other factors interact thoroughly enough to be broadly useful. My main concerns are these: Does the use of simulation yield any new insights outside of the already-established theory of how LLM performance should scale? And, do the simulations cover enough configurations/model architectures used in practice, with sufficient discussion about the effects of certain design decisions, for the results to be applicable?\n\nI therefore believe the work could benefit from more thorough investigation of different architecture types (both neural-network and hardware). Covering all commonly-tweaked hyperparameters such as attention head size and discussing the effects of architecture changes more thoroughly would make the results of the authors' approach more useful for real-world deployments. Otherwise I believe many readers might question if the results/numbers presented here apply to their use case. For example: while attention can be a bottleneck, the authors do not investigate what happens when the design/shape of this layer is changed, even though this is arguably just as commonly done as changing sequence length or embedding dimension. There is also an opportunity to further research the specific details of how hardware works with these systems, deriving more general conclusions based on hardware-level design decisions that help readers understand Amdahl's law for LLMs overall rather than just as it pertains to the specific hardware discussed in the paper. For example, more hardware than just TPUs use systolic arrays, so it is worth advertising and justifying that the findings here are more broadly applicable. By understanding *why* certain decisions affect the numbers, readers could build an understanding ofthe behavior of Amdahl's law in LLMs in general, which they could then apply broadly.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)