LLMCarbon: Modeling the End-to-End Carbon Footprint of Large Language Models
Authors: Ahmad Faiz
Sotaro Kaneda
Ruhan Wang
Rita Chukwunyere Osi
Prateek Sharma
Fan Chen
Lei Jiang
Venue: ICLR 2024 oral (2024)
Keywords: carbon footprint modeling
large lanaguage models
Primary area: societal considerations including fairness, safety, privacy
TL;DR: we propose a carbon footprint modeling tool for large language models.

Abstract: The carbon footprint associated with large language models (LLMs) is a significant concern, encompassing emissions from their training, inference, experimentation, and storage processes, including operational and embodied carbon emissions. An essential aspect is accurately estimating the carbon impact of emerging LLMs even before their training, which heavily relies on GPU usage. Existing studies have reported the carbon footprint of LLM training, but only one tool, mlco2, can predict the carbon footprint of new neural networks prior to physical training. However, mlco2 has several serious limitations. It cannot extend its estimation to dense or mixture-of-experts (MoE) LLMs, disregards critical architectural parameters, focuses solely on GPUs, and cannot model embodied carbon footprints. Addressing these gaps, we introduce \textit{\carb}, an end-to-end carbon footprint projection model designed for both dense and MoE LLMs. Compared to mlco2, \carb~significantly enhances the accuracy of carbon footprint estimations for various LLMs. The source code is released at \url{https://github.com/SotaroKaneda/MLCarbon}.

Published as a conference paper at ICLR 2024
LLMCARBON: MODELING THE END-TO-END CAR- BON FOOTPRINT OF LARGE LANGUAGE MODELS
Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi', Prateek Sharma, Fan Chen, Lei Jiang Indiana University JJackson State University {afaiz, a skaneda, ruhwang, prateeks, fc?, Jiang60 edu 1:009670398students jsums edu
ABSTRACT
The carbon footprint associated with large language models (LLMs) is a sig- nificant concern, encompassing emissions from their training, inference, exper- imentation, and storage processes, including operational and embodied carbon emissions. An essential aspect is accurately estimating the carbon impact of emerging LLMs even before their training, which heavily relies on GPU usage. Existing studies have reported the carbon footprint of LLM training, but only one tool, mlco2, can predict the carbon footprint new neural networks prior physical training. However, mlco2 has several serious limitations. It can- not extend its estimation to dense or mixture-of-experts (MoE) LLMss, disregards critical architectural parameters, focuses solely GPUs, and cannot model em- bodied carbon footprints. Addressing these gaps, we introduce LLMCarbon, an end-to-end carbon footprint projection model designed for both dense and MoE LLMs. Compared to mlco2, LLMCarbon significantly enhances the accuracy of carbon footprint estimations for various LLMs. The source code released httpr//aaaaaaaaa///aaabbaaa /MICCrbbn
https: http//tttttuuuutu.////////tttt.
INTRODUCTION
Large language models (LLMs) have established their supremacy in addressing wide spectrum of natural language processing tasks (Brown et al. 2020). However, the proliferation of these models, coupled with increasingly expansive datasets (Sanderson 2023 Anil et al. (023), has woven LLM inferences into the fabric of everyday life (Campello de Souza et al.) [2023). This surge in LLM adoption has. in turn, exacerbated the already considerable environmental impacts associated with machine learning (ML) (Thompson et al.) 2021). For instance, the creation of a transformer with 213 million parameters through neural architecture search has been likened the carbon dioxide equivalent (CO2eq) emissions of five cars over their entire lifespans (Strubell et al. 2019).
Given the ecological implications of LLMs, it becomes essential for both cloud service providers and regular users to gain profound understanding of the carbon footprint of emerging LLMs. This awareness particularly critical before embarking on resource-intensive training endeavors that en- tail the utilization of thousands of GPUs. During the initial design phase, key parameters such the LLM's parameter count, hardware configurations, and the energy efficiency the hosting data cen- ter need to be factored into robust carbon footprint projection model. This model should possess the capability to swiftly and accurately estimate the carbon footprint, encompassing both opera- tional and embodied carbon emissions. Moreover, it should provide valuable insights into metrics like test loss, training duration, and inference latency, all crucial aspects of LLM performance. The existence of such a carbon footprint projection model empowers cloud providers to intelligently ex- plore the trade-off between test loss and carbon footprint when designing new LLMs. Additionally, encourages everyday users to adopt practices that mitigate LLM carbon footprints by facilitating quantitative comparisons across various L.LM configurations.
Currently, there a notable void in the availability of a comprehensive end-to-end earbon footprint projection model tailored specifically for LLMs. Prior research efforts (Henderson et al.) 2020; Wu ct al. 2022 Anthony et al. 2020 Schwartz et al. 2020 Patterson et al., 2021 |
Dodge et al. 2022
"This work supported in part by CCF-2105972, and NSF CAREER AWARD CNS-2143120.

---

Published as a conference paper at ICLR 2024
Strubell et al. 2019: Lakim et al. 2022) have predominantly focused on recording and reporting the carbon footprint associated with the training phase of ML models. To date, only one tool, mlco2 (Lacoste al. 2019). has emerged capable of predicting the carbon footprint of an ML task based on parameters like GPU usage, training duration, and data center efficiency. However, mlco2 exhibits several serious limitations. Firstly, it is confined to convolutional neural networks (CNNs) and cannot extend its estimations to include the carbon footprint of LLMs. Secondly, mlco2 neglects crucial architectural aspects of ML models, such as parameter counts, resulting in overestimated projections. Thirdly, it exclusively considers GPUs, disregarding specialized ML hardware like TPUs (Jouppi et al., 2017). and assumes uniform peak computing throughput across GPUs, leading to inaccuracies in its carbon footprint assessments. Lastly, although the embodied carbon footprint of an MI task holds equal significance to its operational carbon footprint (Wu et al. 2022), mlco2 incapable of modeling the embodied carbon footprint of an LLM based on its hardware resources.
In this paper, we propose an end-to-end carbon footprint projection model, LLMMCarbon, which can accurately predict the carbon footprint of both dense and MoE LLMs during their training, infer- ence, experimentation, and storage phases. LLMCarbon incorporates critical LLM, hardware, and data center parameters, such as LLM parameter count, hardware type, system power, chip area, and data center efficiency. to model both operational and embodied carbon footprints of an LLM. When validated against Google's published LLM carbon footprints, the results generated by LLM- Carbon exhibit differences of only S 8.2%, and thus are more accurate than those of mlco2.
2
BACKGROUND
LLM Carbon Footprint The carbon footprint of a LLM comprises two fundamental compo- nents (Gupta et al. 2022): the operational footprint, encompassing emissions stemming from hard- ware energy consumption, and the embodied footprint, encapsulating emissions arising from hard- ware manufacturing. Previous investigations (Henderson et al.] 20200] Wu et al. 2022 Anthony
ett al., 2020; [Shwwarzz et al. 2020 Patterson al., 2022 Dodge al. 2022 Strubell et al.) 2019)
have predominantly focused on recording and reporting the operational carbon footprint of various
ML tasks. A notable exception is Wu et al. (2022), which delved into the embodied carbon footprint of ML tasks and revealed that within Meta data center, the embodied carbon footprint of an LLM constitutes 50% of its operational carbon footprint.
Neural Scaling Law. The Neural Scaling Law (Kaplan et al. 2020) delineates a power-law rela- tionship linking an LLM's sss test loss to three key factors: the number of model parameters, the scale of the training dataset, and the computational resources utilized during training. This relationship holds across diverse architectures and downstream MI. tasks, spanning zero-shot, prompted, and fine-tuned scenarios (Caballero et al. 2023).
Reducing LLM Carbon Footprint. Efforts on reducing LLM carbon footprints have been chan- neled into 4 domains. Firstly, sparse MoE architectures (Fedus et al. 2022) have been proposed to enhance LLM performance by increasing model parameters while maintaining a similar computa- tional load. Secondly, the adoption of specialized ML hardware, such as TPUs (Jouppi al. (017), has emerged as a more energy-efficient alternative to power-hungry GPUs. Thirdly, MI-focused data centers have optimized their facilities into large-scale systems, reducing cooling and infras- tructure overhead to enhance power usage effecttienesss (PUE) (Liu et al. 2020). Lastly, these data centers are transitioning renewable energy sources like solar and wind power (Acun et al. 2023) to mitigate the operational carbon footprint of LLMs. However, the recent proliferation of ML- specific hardware within these data centers, driven by the diverse demands of ML tasks, widening the gap between operational and embodied carbon footprints in the near future (Wu et al. 2022).
Parallelism in LLM Processing. Effective processing of LLMs necessitates the utilization of mul- tiple computing devices, such as GPUs or TPUs, owing to significant L.MM parameter counts. Four types of parallelism, i.e., data, tensor, pipeline, and expert, are commonly employed to enhance hardware efficiency, quantified as actual throughput relative to peak throughput.
In data parallelism (Xing et al. 2015), the full LLM model is distributed to each computing device, while the input dataset is divided among these devices. Periodic gradient aggregation ensures that all devices maintain consistent model weights.
Data Parallelism:
Tensor Parallelism: Tensor parallelism (Narayanan et al., 2021) involves distributing an LLM's layers across multiple devices. Within a transformer layer, the self-attention block partitions key, query, and value matrices through column-wist division. The output linear layer directly handles
2

---

Published as a conference paper at ICLR 2024
Table 1: The comparison of LLMCarbon against prior work.
predictive
MoE
architectural specialized operational embodicd parameters
scheme
modeling support
hardware
carbon carbon
mlco2
x
X
others
x
X
X
LLMCarbon
the attention operation's partitioned output, with weight matrix partitioning by rows. In the two- layer MLP, the first layer is divided along columns, and the second along rows. Efficient data coordination among partitions on different devices achieved through two all-reduce operations in forward and backward passes.
Pipeline Parallelism: In pipeline parallelism (Narayanan et al.) 2021), an LLM's layers are dis- tributed across multiple devices. Each device handles an equal number of layers, and microbatches split a batch for pipelined execution. Synchronous weight updates are ensured through pipelining. However, periodic pipeline flushes to synchronize steps across devices introduce "pipeline bub- bles" at batch starts and ends, which need to be minimized for efficient pipeline model parallelism. Expert Parallelism:
Expert parallelism (Kim et al. [20T]) is tailored for parallelizing the train- ing of MoE LLMs. This approach involves distributing distinct experts across various devices, enabling parallel execution. However, due the separation of experts across multiple computing devices, explicit communication using all-to-all primitives becomes essential.
3 RELATED WORK
Table provides a comparison between LLMCarbon and existing research endeavors. The predom- inant focus of prior studies (Henderson et al. 2020 Wu et al. 2022] Anthony et al. [2020] [Schwartz jet al. 2020 Dodge et al. 2022, Strubell et al. 2019) has been the measurement and reporting of carbon footprints associated with the actual training phase of ML models, denoted as "others" in table. Notably, only one previous model, mlco2 (Lacoste al. 2019), possesses the capability to predict the carbon footprint of ML task based metrics like GPU utilization, training dura- tion, and data center efficiency. Nevertheless, mlco2 encounters four significant limitations. Firstly, mlco2 cannot estimate the carbon footprint of LLMs, particularly sparse MoE LLMs. Secondly, overlooks essential architectural attributes of LLMs, such as L.LM parameter count, resulting in exaggerated predictions. Thirdly, mlco2 exclusively considers GPUs and neglects specialized ML hardware like TPUs (Jouppi et al. 2017), assuming uniform peak computing throughput across all GPUs, thereby yielding imprecise carbon footprint estimations. Lastly, mlco2 cannot model the embodied carbon footprint of an LLM based on its hardware configuration.
4 LLMCARBON
4.1 OVERVIEW
Figure I presents overview of LLMCarbon for predicting the carbon footprint of an LLM. The in- puts to I.LMCarbon encompass the LLM's archi- tectural description, data center specification, and
LLM architecture configuration
hardware configuration
data center configuration
V :parameter model
data, tensor, pipe., & expert parallelism
hardware configuration. To output the LLM's car- bon footprint, LLMCarbon employs a series of mod- els, each processing specific input details. LLMCar-
parameter
neural
FLOP model
hardware efficiency carbon model
embodied
scaling
bon can use the parameter model to determine the LLM's parameter count based on its architectural attributes, or directly accept the LLM's parameter count input. With the LLM's parameter count and
test loss
FLÖPs
model
operational
hardware efficiency
embodied
carbon model
CO2eq
operational CO2eq total CO2eq
training token count, LLMCarbon calculates the test loss by the neural scaling law (Kaplan et al., 2220).. and employs the FLOP model to estimate the volume of FLOPS required for LLM processing. Through the parameter count, LLMCarbon generates the optimal data, tensor, pipeline, and expert parallelism setting. Taking into account the parallelism setting and hardware configuration, LLM- Carbon's 's hardware efficiency model computes the hardware efficiency, representing the real com- puting throughput divided by the peak computing throughput. Utilizing data center details, hardware efficiency, and FLOP count, LLMCarbon applies the operational carbon model to derive the LLM's operational carbon footprint. Similarly, by considering the hardware configuration, LLMCarbon's
Figure 1: The overview of LLMCarbon.
3

---

Published as a conference paper at ICLR 2024
10
10
data
pipe
10
10"
expert
pipe
22.
3.5
3.0
derse
56
dense
#20
HW efficlenn
48
10
tensor
10
tensor
115
MoE
10
10
data
100
MoE
40
0.5
36
10"
800 (8)000
10
400
600
800
1000
0.0
700
800
parameters
parameter #(B)
parameters (B)
parameter #BB)
Figure 2: The paral-
Figure The paral-
Figure 4: The com- puting device number for processing LLMs.
Figure 5: The hard- ware efficiency for processing LLMs.
lelism setting for pro- cessing dense LLMs.
lelism setting for pro- cessing MoE LLMs.
embodied carbon model yields the LLM's embodied carbon footprint. The overall carbon footprint of the LLM is then computed by summing both the operational and embodied carbon footprints.
4.2 PARAMETER MODEL
Among all LLM architectural attributes, the LLM parameter count has the largest impact on test loss (Kaplan et al. 2020). To reduce projection errors, LLMCarbon can take the parameter count as direct input. or estimate the parameter count by the parameter model. The parameter model's I's input comprises the L.M's architectural parameters including the hidden size (h), the number of layers (l), the vocabulary size (V), and the number of experts (Ne). For a dense LLM,, we calculate its parameter count (Pa) by Equation [ (Narayanan et al.) 2021). An MoE LLM (Rajbhandari et al.) 2022) replaces (p (0, 1)) feed-forward layers in its counterpart dense LLM with MoE layers. An MoE layer's parameter count is the sum of the expert parameter count (Pezp 8h N.) and the self-attention parameter count (Patt = 4hr2,, the parameter count (P.) of an MoE LLM can computed using Equation 2 The parameter model of L.MSs adopting an encoder-decoder architecture can be viewed in Appendix
Pa RJ 12th2 + Vh
(1)
4.3 NEURAL SCALING LAW
The neural scaling law (Kaplan et al. 2020) predicts an LLM's test loss based on its parameter count P and the training dataset size D. For ensuring the comparability test losses across various models, sizes, and datasets, we adopt the Chinchilla scaling law (Hoffmann et al. (2022) formulated as Equation where A, B, a. B, and E' are fitting constants. The test loss L equals the summation of an irreducible term E and reducible term diminishing through the scaling of P and D.
A B L(P,D) = pa
+ D8
+
(3)
TC = 6PD
(4)
1C = 2PD
(5)
4.4 FLOP MODEL
The FLOP model receives two inputs: the count of parameters (P) and the number of tokens (D) processed by the LLM processing. The primary component of FLOPs is the multiply-accumulate operations involving LLM weights and intermediate results. Within our FLOP model. the FLOP count necessary for training a dense LLM (TC) is estimated using Equation 4 For dense LLM inferences, the FLOP count (1C) approximated for MoE LLM processing, we input the parameter number of the dense base model (Rajbhandari al. 2022) of the MoE LLM into Equations and respectively.
per Equation [S] To compute the FLOP count
4.5 HARDWARF EFFICIENCY MODEL.
Efficient processing of LLMs relies on achieving high hardware efficiency, which is calculated as the actual computing throughput divided by the peak throughput. This efficiency is largely deter- mined by the optimal configuration of data, tensor, pipeline, and expert parallelism, along with the number of devices used for the task. Using too few or too many devices or improperly configuring parallelism can lead to reduced hardware efficiency. For example, achieving optimal parallelism for GPT-3 with 175 billion parameters requires 1.5K V100 GPUs, resulting in a hardware efficiency of 47% (Narayanan al. 2021). Conversely, unoptimized configuration using 10K V100 GPUs yields a substantially lower hardware efficiency of only 19.7% (Patterson et al. 2021).
Optimal Parallelism Setting. The optimal parallelism setting is represented (p. t, d, e), where each variable corresponds to a degree of pipeline, tensor, data, and expert parallelism, respectively. For dense L.Mss, optimal settings derived from (Narayanan et al. 2021), depicted in Figure
4

---

Published as a conference paper at ICLR 2024
where 1 is omitted. Initially, increase tensor parallelism (t) up 2 (e.g., 8) when employing z-device servers (Narayanam et al. 2021), each containing 2 interconnected devices. This increment in t is confined to avoid exceeding communication bandwidth limits. Once 2 is reached, further scaling for larger LLMs involves increasing pipeline parallelism (p) (Narayanan et al., 2027]. However, the product of i and p (t-p) must not exceed certain threshold to ensure that L.LM parameters and intermediate data fit into device memory. The number of devices required to achieve optimal hardware efficiency for dense LLM processing is calculated as n f p (Narayanan et al., 2021), and can be viewed in Figure + A polynomial regression model is used to predict optimal hardware efficiency based on these parameters. For MoE LLMs, the optimal parallelism settings are adopted from (Chen et al., 2023). As Figure shows, assuming 64 experts within an MoE L.M,, expert parallelism (e) always set to 64, intertwining d and for a uniform expert distribution. To reduce inter-device all-to-all communications, d is fixed at 1. Scaling MoE LLM parallelism is achieved by increasing pipeline parallelism (p). The number of devices required for optimal hardware efficiency in MoE LLM processing is also calculated as n f. d. As Figure 4 exhibits, MoE LLMs require fewer devices compared to dense LLMs with equivalent parameter counts due to their lower computational overhead. The optimal hardware efficiency during MoE LLM processing is represented in Figure 5) MoE LLMs achieve ~ 80% (Chen et al. (023) of the optimal hardware efficiency of their dense base models, due extra host-device memory swaps.
70 nE
eff n n
re < n
TFLOP
eff rc -
n
(6)
&ev
(7)
eff n + re
Tddex FLOP peak eff
Fewer or More Computing Devices. When the number of computing devices is not equal to t p d, the hardware efficiency decreases. The efficiency (effre with devices can be calculated using Equation 6 where "No ~ 122 are fitting constants, eff', means the highest hardware efficiency, and n indicates the number of devices that can achieve eff
energy hard
Σ
(Pc. clf. n4 .4) (8)
energy oper = energy PUE
(9)
&Chardwaree.co
4.6 OPERATIONAL CARBON MODEL
By using the FLOP count (TFLOP), the hardware efficiency (eff), and the computing device num- ber (Ndew), we can determine the execution time of a device through Equation 7 where FLOP peak represents the device peak throughput. The total energy (energyhava) consumed by all hardware units can be calculated using Equation 8 where P: denotes the peak power of hardware unit i; eff, represents the hardware efficiency of hardware unit i; n. indicates the count of hardware unit i; and t; means the execution time of hardware unit i. Hardware units encompass a range of components, including CPUs, LLM computing devices, memories, SSDs, and others.
CO2eq eneryy oper -corb_ii biinee (10)
CO2eg chip area CPA
(11)
PUE. Power Usage Effectiveness (PUH) (Henderson et al. 2020) serves as the industry standard metric for evaluating a data center's energy efficiency. It defined as the ratio of the total energy consumption of the data center, including all auxiliary components like cooling, to the energy consumed solely by the computing hardware within the data center. The operational energy (emeryy oper)) associated with LLM processing can be calculated using Equation 9 where energy hand denotes the energy used by the computing hardware within a data center, and PUE indicates the PUE of the specific data center.
4 CO2eg
CO2eq emb
(12) CO2eq CO2eq oper + CO2eq emb (13)
lifetime;
iEhardware_sen
Carbon Intensity. Carbon intensity a metric that assesses the environmental impact of data cen- ter's energy consumption. Carbon-free energy (CFE) denotes the proportion of renewable, carbon- free energy utilized within data center. As a data center increases its utilization of renewable energy, it experiences increase in CFE and corresponding decrease in carbon intensity. Table provides insights into the carbon intensity and CFE values for some data centers. The operational carbon footprint (CO2eg oper) attributed to LLM processing is calculated using Equation 10, where energy oper represents the operational energy for LLM processing, and carb inten denotes the car- bon intensity of the specific data center.
5

---

Published as a conference paper at ICLR 2024
Table 3: The comparison of embodied carbon footprints.
Table 2: The data center efficiency.
hardware
description
unit
CPA
data
carbon free
carbon
CPU
TSMC 16nm
147 mmm?
1 cm
center
intensity
name
energy
gCO2eq/ Wh
DRAM
Micron 18nm
256 GB
0.4 kgCO2/ GB
asia-east2
28%
360
SSD
Samsung 20nm
32 TB
0.01 0.018kyO22/G GB
c
I us-centrall
91%
127
TPUv3
TSMC 16nm
700 mm?
kg cm 1.6 kgCO2/ e
97%
394
TPUv4
TSMC 7nm
400 mmm
us-south1 40%
296
V100
TSMC 12nm
815 mm?
1.2 kgCO2/ cm2
H100
TSMC 4nm
814 mem?
1.8 kgCO2/ cm²
4.7 EMBODIED CARBON MODEL
To quantify the chip's embodied carbon footprint (CO2eg chip) within a specific hardware unit, Equation [1 is employed, where area represents the chip's area. The Carbon emitted Per unit Area (CPA) is contingent on various semiconductor fabrication parameters, including yield, energy con- sumption per unit area during manufacturing, emissions from chemicals utilized in hardware pro- duction, and emissions associated with raw material sourcing for fabrication. Specific values for area and CPA for distinct hardware units are elaborated in Table 3 where area values for CPU, DRAM, SSD, TPU, and GPU are drawn from sources such as (Singh et al.) 2020). (Choel 2021). (Wiki) 2023b), and [Wiki] 2023a). CPA values for Micron, Samsung, and TSMC are extracted from (Gar- cia Bardon et al.) 2020), and ISMC.20199 The total embodied carbon footprint (CO2eg cmb orig- inating from all hardware units involved in LLM processing is assessed using Equation 12 where CO2cqcchip. denotes the chip's embodied carbon footprint for hardware unit i, lifetime, means the lifespan of hardware unit i, and ( represents the execution duration of hardware unit i. The hardware units mentioned in Equation 12 iincuue CPUs, LLM computing devices, memories, SSDs, and other components. Notably, Meta's data centers achieve an average utilization rate of 60% throughout the 5-year lifespan of hardware units [Wu et al. 2022).
4.8
TOTAL CARBON FOOTPRINT
The total carbon footprint (CO2eg) resulting from LLM processing is determined using Equation 13 where CO2eq oper indicates the operational carbon footprint of the LLM, and CO2eq emb denotes the embodied carbon footprint of the LLM.
VALIDATION
We employ LLMCarbon to compute the operational footprints of five LLMs, including dense and MoE architectures, developed by Google, OpenAL, and Meta during their training phases. We also compute the operational footprint of another LLM, Noor (Lakim al. 2022), during its storage phase. To validate the predictions of I..MCarbon, we compare our calculated operational footprint values with the previously published data for these LLMs. Morcover, we utilize LLMCarbon to predict the embodied footprint of an LLM developed by Meta and validate the result by comparing with the actual embodied footprint data.
5.1 OPERATIONAL CARBON FOOTPRINT VALIDATION
Training Phase. Table presents the validation results of LLMCCrbon's predictions on the train- ing operational carbon footprint. To validate the training operational carbon footprint estimations
yielded by LLMCarbon, we selected five LLMs: T5 (Raffel et al. 2020), GPT-3 (Brown al., 2020).
GShard (Lepikhin et al.) 2021). Switch (Fedus et al. (022). and XLM (Conncau et al. (020). We
list the inputs and outputs of MCarbon in Table 4 Within the table, "device TPD (W) indicates
the Chip Thermal Design Power of computing device, while "avg. system power (W)" conveys the
average system power per computing device, including TPU/GPU, host CPU, DRAM, and network interface. The inputs on the parameters of LLMs, hardware, and data centers, and the actual train- ing operational carbon footprint values of these LLMs were collected from (Patterson et al. 2021) architectural param-
and (Wu et al.] 2022). Since the parameter count of an LLM is considered eter the LLM in (Patterson et al.) 202T) and (Wu et al. 2022). we skipped the parameter model and directly used the parameter count input to M..CCarbon. The validation of the parameter

---

Published as a conference paper at ICLR 2024
Table 4: The validation the operational carbon footprints of various L.Mss.
LLM
T5
GPT3
GShard
Switch
XLM
reference
Patterson et al.
2021
(Wu et al.
2022)
developer
Google dense
OpenAl
Google
Google MoE
Meta
type
dense
MoE
dense
parameter (B)
11
175
619
1500
0.55
base model param. # (B)
2.3
7.41
token (B)
500
300
1K
2K
7K
CO2eq/ KWh
0.545
0.429
0.177
0.33
0.413
PUE computing device
1.12
1.1
1.09
1.1
1.1
TPUv3
V100
TPUv3
TPUv3
V100
device TPD (W)
450
300
450
450
300
avg. system power (W)
310
330
288
245
342
peak TFLOPs/s
123
125
123
123
125
achieved TFLOPs/s
45.6
24.6
48
34.4
26.5
hardware efficiency
37%
19.7%
39%
28%
21.2%
device # total Zots
512
10K
1K
1K
512
40.5
314
13.3
82.2
23.9
training days
20
14.8
3.1
27
20.4
actual 1CO2eg
46.7
552.1
4.3
59.1
39
mlco2 predicted tCO2eg
89.4
955.2
8.4
137.3
66.96
mlco2 A
+91.3%
+73%
+5.3%
+132%
+.99
LLMCarbon predicted tC02eq
45.66
553.87
4.46
63.9
37.6
LLMCarbon A
-2.22%
+ +0.32%
+3.8%
+8.2%
-3.54%
model of LLMCarbon can be found in Appendix[B, Owing to the adoption of suboptimal parallelism settings, the hardware efficiencies for training these LLMs hover within the range of 39% to 19. 19.7%, lower than the hardware efficiencies achieved with optimal parallelism configurations. Comparing the predicted operational carbon footprints to actual data, LLMCarbon's projections display dispari- ties of < 8.2%. When predicting the operational carbon footprint during the training of MoF. LLMs, I.LMCarbon incurs a higher margin of error, due the intricacy MoE architectures. On the con- trary. when compared to actual data, the training operational carbon footprint estimations made by mlco2 (Lacoste et al. 2019) suffer from huge disparities of more than 69%, because mlco2 assumes all devices consistently operate the peak computing throughput and consume the peak power.
Inference Phase. validate operational carbon footprint predictions generated by LLMCarbon, consider the inferences of GPT3 with 175B parameters (Yu al., (022). These inferences were carried out on A100 GPUs, using a batch size of 32 and an input size of 128 tokens (Yu et al. the hardware efficiency model, this specific hardware configuration yields hardware efficiency of 9.26%. Achieving the optimal hardware efficiency for GPT3 requires ~ 1..5K GPUs, which is significantly more than what was used for these inferences. LLMCarbon's predicted latency for this inference batch is 1s while the actual latency for this inference batch is 3s (Yu et al.) 2022). We assume the inference experiments took place in data center with PUE of I.I and carbon intensity of 0.429 CO2eg/ Wh. The difference between the predicted and actual inference operational carbon footprints does not exceed +3.3%.
2022). According
Storage Phase. The typical power consumption of cloud storage is reported as 11.3W/TB (Posani al.) 2018), while the power consumption for data transfer within a data center is around 1.48W/TB (Baliga et al. 2011). Over six-month storage phase, the Noor LLM (Lakim et al.) 2022) encompasses 32.7 TTB of storage data, comprising curated data, bulk data, and the model. Ad- ditionally, it transfers a data volume of 277.4TB. Based LLMCarbon's 's estimations, the storage data energy is predicted as 1.596MWh (compared to the actual 1.69MWh (Lakim et al. 2022), ), while the energy consumption attributed to data transfer is projected to be 1.77MWh (compared to 1.8MWh (Lakim et al., 2022)). Notably, the projection accuracy LLMCarbon regarding the operational energy during the storage phase showcases
margin of less than 3.6%.

---

Published as a conference paper at ICLR 2024
400
stbrage D experiment inference
carton footprint tCO [gg
400
.350
trairing embooded
Caltoon boleitne ICO
300
jet plane round trip
300
storage experiment inference training
embodidd
250
San Fran New York
200
150
100
100
<5
XLM
Swatch
XLM Switch
KLM Switch
V100
TPUN
TPUN4
via HIGD TPUY
TPUV4
v100 J Switch
TPU14
whole in cycle
renewable energy optimal para setting
75
KLM
Figure 6: The carbon footprint of three
Figure 7: The carbon footprint of GPT3
LLMs in case studies.
trained by different computing devices.
Experimentation Phase. The experimentation phase consisting of various actívities of training. inference, and storage (Wu et al., (202)). And we have validated the training phase, inference phase, and storage phase of an L.M in previous sections.
5.2 EMBODIED CARBON FOOTPRINT VALIDATION
Table presents the validation results of the embodied carbon footprint estimated by LLMCarbon in comparison to the pub- lished data of XLM et al22 This the only publicly available data regard- ing the embodied carbon footprint a LLM training hardware infrastructure our best knowledge. The setup consists of 512 V100 GPUs organized into 64 8- GPU servers, each equipped with a CPU, 32TB SSD disk, and 256GB DRAM main memory system. Using the unit and CPA data from Table 3 we computed the values of CO2eq presented in Table 5 The training duration of XLM is 20.4 days, and Wu
Table 5: The embodied carbon footprint validation against Meta XLM.
CO2eg chip (kgCO 24
CO2eq emb
hardware number
Wfetiie
(tCO2eg)
GPU
512
9.78
1.12%
0,056
CPU
64
1.47
1.12%
0.0018
SSD
64
576
1.12%
0.412
DRAM
64
102.4
1.12%
0.073
others
64
148.2
1.12%
0.096
predicted sum
0.64
actual 0.66 LCO2eg, A -3.05%
al. 2002 assumed a hardware unit lifetime of years. Consequently, the time fifetime values for all hard ward units were determined to be 1.12%. Apart from CPU, GPU, SSD, and DRAM, other hardware components (others) such the motherboard, chassis, and PSU collectively contribute to 15% (Tannu & Nair 2022) of the anticipated total embodied carbon footprint. In contrast to the reported embodied carbon footprint of XLM (Wu et al., 2022), the predictions produced by LLM- Carbon reveal a disparity of -3.05%
CASE STUDIES USING LLMCARBON
We used LLMCCrbon to demonstrate the following case studies.
Large Embodied Carbon Footprint. The embodied carbon footprint throughout the life-cycle of an I.M is significant. Even when no computing activities occur, the L.M still incurs embodied car- bon overhead due to the idle hardware allocated to the LLM. As illustrated in Figure the embodied carbon footprint of an LLM across its entire life-cycle contributes to approximately 24% ~ 35% of the overall carbon footprint (including embodied, training, inference, experimentation, and storage carbon footprints) of the LLM. We adopted the ratio between training, inference, and experimen- tation activities from (Wu et al.) 2022). Furthermore, as data centers progressively shift towards adopting renewable energy sources, the embodied carbon footprint of an L.IM will dominate the entire life-cycle carbon footprint of the I.I.M the near future. For instance, 97% of the operational energy in Meta data center (Wu et al. (022)) is provided by renewable sources. The embodied carbon footprints of diverse LLMs operating within this data center constitute 92% ~ 95% of their entire life-cycle carbon footprints. This underscores the pivotal role of accounting for embodied carbon in the sustainability evaluation LLMs.
Optimal Parallelism Setting. As discussed in Section 5.1 the training processes of the LLMs used in our validation lacked optimized parallelism settings. By using LLMCarbon, we pinpoint the optimal configurations for data, tensor, pipeline, and expert parallelism pertaining these three
8

---

Published as a conference paper at ICLR 2024
LLMs. As illustrated in Figure 6 the adoption of these optimal parallelism settings leads to noteworthy decrease (i.e., 16% ~ 39%) in their operational carbon footprints.
New Accelerators. When employing distinctive computing devices for the LLM processing, the operational carbon footprints of an LLM tend to differ, while the embodied carbon footprints re- main similar. Figure 7 showcases the outcomes derived from training, inferring, and experimenting with three LLMs utilizing V100 GPU, H100 GPU, TPUv3, and TPUv4. Their embodied carbon footprints exhibit similarity, as the embodied carbon emissions of SSD and DRAM dominate their total embodied carbon footprints. However, compared to V100 GPUs, the operational carbon foot- prints of these LLMs are notably curtailed by 71% and 41% when employing H100 and TPUv4 accelerators, respectively. Embracing novel computing devices for LLMs presents a pragmatic path to mitigate their operational carbon footprints.
Training Carbon Footprint Scaling. In addition to the LLMs (i.e.. T5, GPT3, GShard, Switch, XLM, and Noor) we used in validations, we also included other
2.25
XLM
2.20
PR-MoE
LLMs in our analysis, such as PaLM (Chowdhery et al.
2.15
Neer
2022), Gopher (Rae et al.) (202T) Chinchilla (Hoff-
Eest loss
mann et al., 2022). LaMDA (Thoppilan et al.; 2022).
2.10
FB-MoF
T5 LaMDA
urassic
Jurassic-1 (Lieber et al. 202T). MT-NLG (Smith et al.
2.05
Gopher
2022), Bloom (Scao et al. 2022). YaLM (Yandex (022).
YaLM
GPT3
2.00
NLG
GLM (Zeng al.] 2023). GLaM (Du al. 2022), FB-
Gshard
GLM Bloom
MoE (Artetxe et al. (021). ST-MoE (Zoph et al. (2022), and PR-MoE (Rajbhandari et al.) 2022). Among these LLMs, GShard, Switch, GLaM, FB-MoE, ST-MoE, and
1.95
ST-MoE
ChinChill
PaLM
1.90
GLaM
Switch
PR-MoE use sparse MoE architectures, while the other LLMs adopt dense architectures. We do not aim to di- rectly compare the accuracy and carbon emissions of these original LLMs,, since they were trained by different datasets and in different data centers. Instead, we study the test losses and training operational carbon footprints of some new LLM designs adopting the same architectures as these LLMs. We assume these new LLM designs are trained using the same dataset and the same hardware infrastructure in the same data center. We present the test losses and training operational carbon footprints of these LLMs in Figure 8 To compute the test loss,
1.85
10
102
10 10t
training carbon tCO2eq
Figure 8: The trade-off between train- ing carbon footprint and test loss.
adopt the fitting constants including a 0.34, 3 0.28, A 406.4, .,.44, B- 410.7, 7, and E- = 1.69 for Equation 3 from (Hoffmann et al. (022). Since the test loss of an MoE LLM with P parameters is similar to that of its dense counterpart with only P/8 parameters (Rajbhandari et al. 2022). we decreased the P of MoE LLMs to P/S in Equation The training processes of all LLMs use their optimal parallelism settings and the corresponding numbers of V100 GPUs hosted by a data center where PUE is 1.1 and CO2eg/ KWh is 0.431. Overall, an LLM with a larger number parameters and trained on more tokens achieves a lower test loss but also consumes a larger training operational carbon footprint. Compared to dense LLMs, the Pareto front of MoE LLMs is closer to the origin point, indicating that an MoE LLM can obtain lower test loss by the same training carbon footprint.
CONCLUSION
In this paper, we propose LLMCarbon, an end-to-end carbon footprint modeling tool for dense and MoE LLMs, which contribute significantly to carbon emissions during training, inference, experi- mentation, and storage processes. IL.MCarbon can accurately assess the operational and embodied carbon footprints of an L.M,, enabling efficient exploration of the design space by considering the trade-off between carbon footprint and test loss. It also promotes the adoption of carbon footprint reduction measures by facilitating quantitative comparisons among various LLM configurations.
REFERENCES
Bilge Acun, Benjamin Lee, Fiodar Kazhamiaka, Kiwan Maeng, Udit Gupta, Manoj Chakkaravarthy, David Brooks, and Carole-Jean Wu. Carbon explorer: A holistic framework for designing carbon aware datacenters. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pp. 118-132, 2023.

---

Published as a conference paper at ICLR 2024
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint 2rXh:2235..0443, 2023.
Lasse F Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. Carbontracker: Track- ing and predicting the carbon footprint of training deep learning models. arXiv preprini arXiv:2007.03051, 2020.
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victo- ria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:21 12. 12.6::2412.10644, 2021.
Jayant Baliga, Robert W. A. Ayre, Kerry Hinton, and Rodney S. Tucker. Green cloud computing: Balancing energy in processing, storage, and transport. Proceedings of the IEEE, 99(1):149-167, 2011.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020.
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In The Eleventh International Conference on Learning Representations, 2023. URL https:/ //
openreview net/ htt.:/forummmme (i=.oruneiisskkkeeeec..
Bruno Campello de Souza, Agostinho Serrano de Andrade Neto, and Antonio Roazzi. Are the new ais smart enough to steal your job? iq scores for chatgpt, microsoft bing, google bard and quora poe. IQ Scores for ChatGPT, Microsoft Bing, Google Bard and Quora Poe (April 7, 2023), 2023.
Xin Chen, Hengheng Zhang, Xiaotao Gu, Kaifeng Bi, Lingxi Xie, and Qi Tian. Pipeline moe: A flexible moe implementation with pipeline parallelism. arXiv preprint 1XXv:2304.11444 2023.
Jeongdong Choe. Memory technology 2021: Trends & challenges. In 202/ International Con- ference on Simulation of Semiconductor Processes and Devices (SISPAD), pp. 111-115. IEEE, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Oit, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Annual Meeting of the Association for Computational Linguistics. pp. 8440 8451, July 2020.
Jesse Dodge, Taylor Prewitt, Remi Tachet des Combes, Erika Odmark, Roy Schwartz, Emma Strubell, Alexandra Sasha Luccioni, Noah A. Smith, Nicole DeCario, and Will Buchanan. Mea- suring the carbon intensity of ai in cloud instances. In ACM Conference on Fairness, Accountabil- ity, and Transparency, pp. 1877--1894, New York, NY, USA, 2022, Association for Computing Machinery. ISBN 1781450393522.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pp. 5547- 5569. PMLR, 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1): 5232-5270, 2022.
10

---

Published as a conference paper at ICLR 2024
M. Garcia Bardon, P. Wuytens, L.-A Ragnarsson, G. Mirabelli, D. Jang. G. Willems, A. Mallik, A. Spessot, J. Ryckaert, and B. Parvais. Dico including sustainability: Swwer-perforancee -rea- cost-environmmentl score (ppace) analysis for logic technologies. In IEEE International Electron Devices Meeting, pp.
41.4, $1.4.1 1..11441..444 .44, 2020.
Udit Gupta, Young Geun Kim, Sylvia Lee, Jordan Tse, Alsien-Hsin S. Lee, Gu- Yeon Wei, David Brooks, and Carole-Jean Wu. Chasing carbon: The elusive environmental footprint of computing. IEEE Micro, 42(():37 47, jul 2022.
Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. To- wards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research, 21(1). jan 2020. ISSN 1532-4435.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute- compute-optimal large language models. arXiv preprint arXiv:2203. 15556 2022.
Norman P Jouppi, Cliff Young. Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. JEEE/ACM International symposium on computer architecture, pp- 1-12, 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08360 2020.
Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. Scalable and effi- cient moe training for multitask multilingual models. arXiv preprint arXiv: 10465, 2021.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09000 2019.
Imad Lakim, Ebtesam Almazrouei, Ibrahim Abualhaol, Merouane Debbah, and Julien Launay. A holistic assessment of the carbon footprint of noor, a very large Arabic language model. In Pro- ceedings of BigScience Episode #5 Workshop on Challenges Perspectives Creating Large Language Models, pp. 84-94, may 2022.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazecr, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference Learning Representations, 2021. URL bttps: http::::////eeevveeeeeeee http:eewwwwwweeeeeooeeeeeeee....
net/ http//oormmmmmmmm0m Xwt.ddg=wweeexemybb..
Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-1: Technical details and evalua- tion. White Paper. A121 Labs, 1, 2021.
Yanan Liu, Xiaoxia Wei, Jinyu Xiao, Zhijie Liu, Yang Xu, and Yun Tian. Energy consumption and emission mitigation prediction based on data center traffic and pue for global data centers. Global Energy Interconnection, 3(8)2722-22,, 2020.
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vi- jay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-lm. In ACM International Conference for High Performance Computing, Net- working. Storage and Analysis, 2021.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Mique Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2 104. 10350, 2021.
David Patterson, Joseph Gonzalez, Urs Hölzle, Quoe Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David R So, Maud Texier, and Jeff Dean. The carbon footprint of machine learning training will plateau, then shrink. Computer, 55(7):18-28, 2022.
11

---

Published as a conference paper at ICLR 2024
Lorenzo Posani, Alessio Paccoia, and Marco Moschettini. The carbon footprint of distributed cloud storage. arXiv preprint 1803v.8803.00 2018.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 14446 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
wttppl/.oc/ppppes//22///// 201221220004-- html
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-ng mee::: Advancing mixture-of-expene inference and training to power next next-generation scale. In International Conference on Machine Learning, pp. 18332-18346, 2022.
Katharine Sanderson. Gpt-4 is here: what scientists think. Nature, 615(7954):773, 2023.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 2022.
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai. Communications of the
ACM, 63(12):54-66 nov 2020.
Teja Singh, Sundar Rangarajan, Deepesh John, Russell Schreiber, Spence Oliver, Rajit Seahra, and Alex Schaefer. zen 2: The amd 7nm energy-efficient high-performance x86-64 microprocessor core. In 2020 IEEE International Solid-State Circuits Conference-(ISSCCC, pp. 42-44. IEEE, 2020.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deep- speed and megatron train megatron-turine nig 530b, large-scale generative language model. arXiv preprint arXiv:2201.11990 2022.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In Annual Meeting of the Association for Computational Linguistics, pp.
2019.
Swamit Tannu and Prashant Nair. The dirty secret of ssds: Embodied carbon. In The 1st Workshop on Sustainable Computer Systems Design and Implementation, 2022.
Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F. Manso. Deep learning's 's diminishing returns: The cost of improvement is becoming unsustainable. IEEE Spectrum, 58 (10):50-55, 2021. doi: 10. 19/10M/SCCC22001 201...P9E...221119999544444
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08233 2022.
TSMC. TSMC Corporate Social Responsibility Report. https: https::/esgg eegg......... tsmc com/
2019.
Wiki. h
Ampere$20 htw.://occcccceccteteeeeeaaaaaa. 2023a.
Wiki. Tensor Processing Unit.
http:////nnxx.pppttiiec [Btoort000ceess22000001.11111113
2023b.
Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, al. Sustainable ai: Environmental impli- cations, challenges and opportunities. Proceedings of Machine Learning and Systems, 4:795-813, 2022.
12

---

Published as a conference paper at ICLR 2024
Eric P. Xing, Qirong Ho. Wei Dai, Jin Kyu Kim, Jinliang Wei, Seunghak Lee, Xun Zheng, Peng- tao Xie, Abhimanu Kumar, and Yaoliang Yu. Petuum: new platform for distributed machine learning big data. IEEE Transactions on Big Data, 1(2):49-67,20) 2015.
Yandex. Yalm 100b. httpps//ggghhu...com///aaaaa///a
2022.
Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gov Chun. Orea: A distributed serving system for (Transformer-Basee)) generative models. In USENIX Symposium on Operating Systems Design and Implementation, pp. 521-55215 2022.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, 2023.
Barret Zoph, Irwan Bello, Sameer Kumar. Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08996 2022.
13

---

Published as a conference paper at ICLR 2024
Table 6: The architectural details of dense LLMSs for validations and explorations. The dense LLMs
we selected include T5 (Raffel et al.] [2020), GPT-3 (Brown et al.
2020). XLM (Conneau et al.
2020 Noor (Lakim 2022 LaMDA (Thoppilan et al.
20022) PaLM (Chowdhery et al.) 2022),
Gopher (Rae et al., (021)) Chin-
chilla (offmann et al.
2022), Jurassic-1 1 (Lieber et al. 2021
MT-NLG (Smith al. 2022).
Bloom (Seao et al. 2022). YaLM Yandex 2022). and GLM Zeng
al. 2023).
Name
Param.
V
h
d
dhead
N Nhead
(
Equ.
Pa (B)
Diff.
T5
11
32K
1024
65536
128
128
24
14
11.3
+2.79%
GPT3
175
51.2K
12288
49152
128
96
96
174.58
-0.24%
XLM
0.55
250K
1024
4096
64
16
24
0.557
+1.45%
Noor
13
PaLM
540
256K
18432
73728
256
48
118
15
539.24
-0.14%
Gopher
280
51.2K
16384
65536
128
128
80
258.54 -7.66%
Chinchilla
70
51.2K
8192
32768
128
64
80
64.84
-7.36%
LaMDA
137
51.2K
8192
65536
128
128
64
137.86 +0.63%
Jurassic-1
178
256K
13824
55296
144
96
76
175
-1.68%
MT-NLG
530
51.2K
20480
81920
160
128
105
529.53
-0.09%
Bloom
176
51.2K
14336
57344
128
112
70
173.37
-1.49%
YaLM
100
GLM
130
51.2K
12288
49152
128
70
127.46
-1.55%
A MORE ON THE LLM PARAMETER MODEL
We listed the architectural parameters of dense LLMs we selected in Table and the architectural parameters of MoE LLMs we used in Table
GPT3-like Dense LLMs: The parameter count for most dense LLMS structured on a GPT3-like architecture (Brown et al.) 2020) can be determined using Equation [ In each layer of these dense LLMs, there exists self-attention layer and a feed-forward layer. The Wy. WR. and W. matrices of the self-attention layer possess dimensions of h.Nneaddread. with f representing the hidden size, Nhead indicating the number of heads, and dhead denoting the head dimension. The W. matrix that links the self-attention layer to the feed-forward layer also has dimension of h.hheadheead- In the feed-forward layer, two hdys weight matrices are used, where dff signifies the dimension of the feed-forward layer. In a conventional LLM architecture, we have Theeathhead drs 4h. Consequently, the parameter count for single dense LLM layer can be calculated as 4h Nheaddhhed + 2hd = 12 Additionally, a dense LLM possesses Vh token embedding parameters, where V denotes the vocabulary size. In total, dense LLM utilizing GPT3-like architecture incorporates 12/ + Vh parameters, where I stands for the number of layers.
h and
Encoder-Decoder Dense LLMs: Certain dense LLMs from Google, such as T5 (Raffel et al., 2020), employ an encoder-decoder transformer architecture. Within a single layer of these LLMs, there exist both an encoder and a decoder. The encoder comprises a self-attention layer and feed-forward layer, while the decoder includes two self-attention layers and a foed-forward layer. The parameter count for the encoder is 4h.Nheaddhead + 2hd whereas the parameter count for the decoder is 8h Sheahddhheadtt 2hd Therefore, the total parameter count for a single LLM resembling T5 becomes 12hNheadChead + 4hd,y. In the case of a T5-like LLM,, where Theeadhhead # h and d + 4h, cannot derive further simplified equation. The overall parameter count for T5-like LLM can be estimated as:
Pa (12hN 122hNheaadhhen. + 4hddz)) 1 Vh.
(14)
For some dense LLMs like LaMDA (Thopiilan 2022). which consist of only a decoder in each layer, the total parameter count for a LaMDA- LLM is:
Pa (ShNheaddhead 2ld Vh.
(15)
MoE LLMs: In the case certain MoE LLMs, especially those developed by Google, we also encounter scenarios where Wheadthoad o h and dyf + 4h. Consequently, within an MoE layer,
14

---

Published as a conference paper at ICLR 2024
Table 7: The architectural details of MoE LL for validations and explorations. The MoE LLMs
we selected include Gshard (Lepikhin et al. 2021). Switch (Fedus et al. 2022). GLaM (Du et al.
2022 FB-MoE (Artetxe et 2021). ST-MoE Zoph et 2022). and PR- MRM Rajbhandari al.
2022
Name
Param.(B)
Pa (B)
h
d5s
dread
Nhead
Ne
Equ.
Pe (B)
D
Gshard
600
2.3
0.5
1024
8192
128
16
36
2048
16
618.47
+3.07%
Switch
1571
7
2048
6144
32
64
15
2048
T6
1546.19
-1.58%
GLaM
1200
95
0.5
8192
32768
128
128
64
64
2
1133.87 -5.51%
FB-MoE
1100
2.3
0.5
4096
16384
128
32
512
2
1103.81
+0.35%
ST-MoE
269
32
0.25
5120
20480
128
64
27
64
T6
273.17
+1.55%
PR-MoE
1.3
0.5
2048
8192
128
16
64/128
T6
31.8
+2.5%
2 while the self-attention parameter count can be determined as P Patt 4hhheaddhead- The overall parameter count for such an MoE LLM can be estimated follows:
compute the expert parameter count Peep
Pe (1 Pp + p(2hd,y 4heheaddhead)
(16)
B PARAMETER MODEL VALIDATION
Validation of Dense LLMs: We present the architectural parameters of dense L.MSs in Table It's
worth noting that while Noor was utilized in the validation of training operational energy and YaLM LLM scaling, their original papers (Lakim al. 2022 Yandex 2022) do not provide architectural specifications, thus preventing
from determining their parameter count using LLMCarbon. In Table 66 we apply Equation T to calculate the parameter count models such as GPT3, XLM, Gopher, Chinachilla, Jurassic-1. MT-NLG, Bloom, and GLM. Additionally.
Equation 14 to estimate the parameter count for T5, and Equation [5] for PaLM and LaMDA. Among all dense LLMs, Gopher and Chinchilla exhibit the most substantial disparities between the predicted and actual parameter numbers. This deviation is primarily attributed to the usage of positional encoding mechanisms in these LLMs, with the weights employed their relative positional encodings not included in our calculations. For instance, Gopher incorporates 21.5 billion weights in its relative positional encoding, contributing to this observed difference.
We present the architectural parameters of MoE LLMs in Table 7 To compute the parameter count for GLaM, and FB-MoE, we utilize Equation For Gshard, Switch, ST-MoE, and PR-MoE, we apply Equation[I6 In PR-MoE, portion of MoE layers have 64 experts, and the other MoE layers have 128 experts. In the table, Gshard, GL.aM, and PR-MoE encounter the largest disparities between the predicted and actual parameter counts. The deviation is caused by the usage of positional encoding mechanisms in these MoE LLMs, and the unaccounted parameters in their routing networks.
Validation of MoE LLMs:
15


Reviews:
[{"review_id": "TOEP5u59ej", "reviewer": "Reviewer_aKac", "summary": "This paper introduces LLMCarbon, a comprehensive cost model developed to estimate the carbon footprint associated with various stages of LLM computation, such as training, inference, experimentation, and storage. LLMCarbon extends its applicability beyond previously established models by incorporating support for a broader range of LLM architectures, with a specific focus on the mixture-of-experts architecture and more types of hardware. The paper provides an in-depth discussion of LLMCarbon’s cost model, covering aspects like parameter count, FLOPs, and hardware efficiency. To validate LLMCarbon’s efficacy, we present experimental results across different LLMs and hardware configurations.", "questions": "- For figuring out the optimal parallelism strategy using LLMCarborn, how different are the strategies returned by LLMCarborn compared to the one solved using the compilation approach, e.g., the one proposed in [1]?\n- Instead of scaling law, recent LLM pre-training efforts usually leverage a certain training number of tokens, e.g., 1T/1.4T in LLaMA. Would it be sufficient to directly use the number of training tokens instead of loss/perplexity?\n- How can one use LLMCarborn to guide the designs of future generations of LLM architecture/training procedures or even AI hardware design?\n\n[1] https://www.usenix.org/system/files/osdi22-zheng-lianmin.pdf  \n[2] https://arxiv.org/abs/2302.13971", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is both well-written and well-motivated, addressing the critical issue of environmental sustainability by studying the carbon footprint of LLM computation. The authors have proposed a cost model that appears to be both fine-grained and carefully designed, reflecting a thorough approach to this significant topic.", "weaknesses": "- the procedure of how LLMCarborn figures out the optimal parallelism is not quite clear.\n- It is not clear how to use LLMCarborn to guide the design of future generations of LLM architectures and figure AI accelerators.\n- The relationships and connections between the proposed hardware efficiency and metrics like arithmetic intensity and MFU are not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eufWCUElAK", "reviewer": "Reviewer_KiRY", "summary": "In this paper, the authors proposed an end-to-end carbon footprint predictor for llm services. Specificall, the predicted carbon footprint is the sum of two parts: operational carbon footprint and embodied carbon footprint. The former one is calculated with a model by taking model parameters, hardware efficiency as input; The latter one is calcuated with another model by taking hardware type, chip area, system power as input. Through extensive comparison experiments, the predicted results shows a better performance with at most 8.2% error.", "questions": "See above weakness part.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Compared with baselines, the proposed llmcarbon generalize the carbon prediction to various network architectures (dense llm and moe llm), various hardwares (gpu and tpu) and various phase (training, inference, experimentaiton and storage).\n\n- The key submodels in llmcarbon are elaborated with mathematic formulation and detail description. All the architecture is easy to follow.", "weaknesses": "- Some parameter is crucial to the final predicted results, the process of setting parameters should be clarified. For example, in equation 3, \\alpha, \\beta are the fitting parameters. However, the fitting dataset and fitting method are missing.\n\n- Apart form the parameters discussed in the paper, more factors also affect the footprint. e.g., the paramter precision (fp16, int8, int4) and the implementation of kernel operation.\n\n- There are not any text description about figure 3 and figure 4.\n\n- The proposed llmcarbon is somewhat like a system design with many prior experience from sota, the algorithm contribution are not notable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OP6ihqed4j", "reviewer": "Reviewer_5gUj", "summary": "The paper provides a tool for estimating the carbon footprint of large language models (LLMs). The tool accounts for each life-cycle phase of an LLM: embodied carbon, training, inference, storage. The energy used by hardware is accounted for by considering data center characteristics and model architecture attributes. Comparison with published carbon numbers shows less than 10% error in estimation.", "questions": "- Are ground truth carbon footprint values based on actual measurements? \n- Did you tune your equations or its parameters to reduce the error? If so, how do we know that LLMCarbon is not \"overfitting\"?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Problem statement is clear\n- Carbon footprint considers a comprehensive view of LLM life-cycle\n- Use of real-world data points from literature to perform carbon footprints\n- Validation against published carbon footprint numbers\n- Low error rates in estimation", "weaknesses": "- It is not clear how the \"ground truth\" carbon footprint numbers were established by prior works. Did they perform actual measurements? This is a concern because if the numbers are estimated, and there is an overlap in estimation methods, then it's not surprising that the error rate is low.\n- It is difficult to separate the contributions of the paper from the prior works. Much of the equations and parameters of the equations are based on prior work. Is the contribution of the paper to sum up carbon numbers from prior work equations? \n- There is no measure of utility of an LLM. For example, one can reduce the emissions by increasing test loss, but a high test loss can lead to poor performance which cannot be used in the real-world. It'll be interesting to measure the utility across classical ML models such as those used for classification, regression, translation, etc. \n- It would be good to discuss the utility of the measurement tool. Is the expectation that model designers pick an appropriate data center, architecture, or training dataset to use based on carbon footprint? Much of the time such decisions are constrained by other requirements. For example, model designers have little control over the embodied footprint of the ML model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gUeF1UPEgH", "reviewer": "Reviewer_pKfr", "summary": "The main contribution of this paper is to present a comprehensive model for end-to-end carbon footprint of LLMs. Observing that LLMs can be a major contributed to overall carbon footprint, the authors model its carbon footprint from both operational and embodied perspectives, and presents detailed results for different types of architectures.", "questions": "While the paper certainly addresses an important topic, there are at least three issues with it, from the perspective of this reviewer:\n\n\n1) It seems to me that the authors definition of embodied carbon is the carbon emitted during the production of the hardware on which LLM models run. If that is the case, why? Because that hardware is not just used for running LLM workloads; it is. probably used for running other workloads as well. In my humble opinion, the embodied carbon -- in the case of LLMs -- should represent the carbon emitted during the training, as trained model is the main product/output of LLM. What do authors think about this?\n\n2) It is not immediately evident to me how carbon (emission) factor is accounted for in this modeling. In principle, this coefficient is location dependent and consequently depending on where you are ruining your LLMs, you should get a different carbon footprint. Can the authors clarify that?\n\n3) While I appreciate the importance of carbon modeling of LLMs, it is not clear from this work i) whether the paper is relevant to ICLR as it does not address any core topic in learning, and ii) how it can be used for optimization purposes.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Modeling carbon footprint is an interesting (and increasingly important) research direction. This submission is probably one of the early papers on the topic.", "weaknesses": "Not clear whether the authors' definition of embodied carbon is accurate.\nNot clear where the carbon coefficient factors in in the model formulation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g0sFMwwnOT", "reviewer": "Reviewer_12is", "summary": "The work builds an analytic model that predicts carbon equivalent emissions of LLM training and deployment. The model is extensive and contains variables and functions such as scaling laws, parallelism used, mixture of experts, data center efficiencies, and embodied carbon footprint.\n\nRecommendation: While the model could further be extended to include more detailed variables, it is a groundbreaking effort that will lay the foundation for all future carbon models. I recommend this work to be highlighted at the conference.", "questions": "Comments:\n- Figure 6 and 7 caption seem to be swapped\n\nQuestions:\n- why do you use 16 A100s, batch size 32 and 128 tokens for inference benchmarking? I would assume the most common deployment strategies for inference are (1) 8x A100 + NVSwitch with batch size of 64-128 for token-by-token generation and (2) personal deployment with batch size 1 (this can be significant for T5 and other open source LLMs which enjoy large widespreadnn use)", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- a very comprehensive model that captures previously neglected variables that led to widely inaccurate carbon emission predictions\n- uses scaling laws and other variables to make accurate predictions of the carbon emissions of any training run\n- validation of the model with reported carbon footprints in the literature.", "weaknesses": "- the mixture of expert models seems to use regular scaling laws equations for performance prediction. More appropriate would be to use the equations from the routed scaling laws paper.\n- some factors are not very well discussed. For example, experimentation can have a large variance of CO2eq used. For deployment, more experimentation is usually undertaken to improve model efficiency through speculative decoding and distillation. A short discussion on the most CO2eq intensive factors and how they might differ between companies/institutions would be appropriate (no need to model this)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)