LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferences

## Abstract

Abstract Throughout its lifecycle, a large language model (LLM) generates a substantially larger carbon footprint during inference than training. LLM inference requests vary in batch size, prompt length, and token generation number, while cloud providers employ different GPU types and quantities to meet diverse service-level objectives for accuracy and latency. It is crucial for both users and cloud providers to have a tool that quickly and accurately estimates the carbon impact of LLM inferences based on a combination of inference request and hardware configurations before execution. Estimating the carbon footprint of LLM inferences is more complex than training due to lower and highly variable model FLOPS utilization, rendering previous equation-based models inaccurate. Additionally, existing machine learning (ML) prediction methods either lack accuracy or demand extensive training data, as they inadequately handle the distinct prefill and decode phases, overlook hardware-specific features, and inefficiently sample uncommon inference configurations. We introduce LLMCO 2 , a graph neural network (GNN)-based model that greatly improves the accuracy of LLM inference carbon footprint predictions compared to previous methods.

## 1 Introduction

Large language models (LLMs) (BigScience, ; Meta, ; Mistral, ) have demonstrated high efficacy across various generative Natural Language Processing (NLP) tasks, such as code completion (Nam et al., ), question-answering (Shao et al., ), and text summarization (Pilault et al., ). Their integration into daily activities (e.g., web browsing (Campello de Souza et al., )) highlights their increasing prevalence. However, this widespread adoption has led to significant carbon dioxide equivalent (CO2eq) emissions (Luccioni et al., ). For instance, training the Google T5 LLM generates 40% more carbon emissions than a round-trip flight between San Francisco and New York (Faiz et al., ).

Inferences for LLMs can produce an even larger carbon footprint than their initial training. Conservative estimates suggest that OpenAI handles over 270 million daily inference requests (Chien et al., ), with an average prompt length of 1.2K tokens per request (Patel et al., ). Training GPT-4 requires approximately 13 trillion tokens (OpenAI, ), with a single epoch requiring three times the FLOPs of an inference (Faiz et al., ). Consequently, the carbon emissions from 121 days of serving GPT-4 inferences equate to those of its training. As the volume of daily inference requests rises, and with increased adoption of LLMs across various applications (OpenAI, ), the period required for inference emissions to match training emissions is rapidly decreasing.

Given the significant environmental impact, it is essential for both end users and cloud providers to understand the carbon emission costs of different service-level objectives (SLOs) (Patel et al., ) for LLM inference accuracy and latency. An accurate carbon footprint prediction tool is crucial before initiating inference requests, enabling users to assess trade-offs between accuracy, latency, and carbon emissions. This tool would also help cloud providers justify billing policies transparently, promoting low-carbon practices among users.

However, there is a lack of modeling tools for accurately estimating the carbon footprint of LLM inferences. Users submit LLM inference requests with varying configurations (e.g., batch size, prompt length, and token generation number) to cloud services, while cloud providers employ different GPU types and quantities to meet diverse SLOs for accuracy and latency. Prior studies (Nguyen et al., ; Luccioni et al., ) have reported LLM inference carbon emissions on a limited range of GPUs, but exhaustively profiling all possible configurations is impractical. Although a carbon footprint model for LLM training (Faiz et al., ) exists, its equation-based approach fails to accurately capture LLM inference emissions due to lower and highly variable model FLOPS utilization (MFU) (Pope et al., ). While machine learning (ML)-based tools have been developed to predict inference latency (Liu et al., ; Zhang et al., ) and energy (Tu et al., ) for neural networks on mobile devices, applying them to LLM inference carbon footprints results in low accuracy or requires extensive training data for the following reasons:

- •
Lack of autoregressive consideration. Existing tools treat CNN inference as a single task, failing to account for the distinct autoregressive phases of LLMs: the compute-intensive prefill and the memory-bound decode (Patel et al., ). This oversight reduces prediction accuracy or necessitates large training datasets.
- •
Neglect of hardware-specific features. Prior tools overlook critical hardware characteristics such as GPU peak computing throughput, memory bandwidth, and network bandwidth. Accurate predictions thus require exhaustive profiling across different GPU platforms (Pope et al., ), making these methods inefficient and less reliable for new hardware.
- •
Disregard for prevalent configurations. Previous tools treat all inference configurations equally, ignoring that most real-world LLM inference requests, such as those in Azure Cloud (Microsoft, ), feature small batch sizes, shorter prompts, and limited token generation. This approach fails to improve prediction accuracy for typical usage scenarios.

We introduce LLMCO2, an accurate carbon footprint regression model for LLM inferences. LLMCO2 employs a novel graph embedding technique that represents each transformer layer’s kernels as a graph, with nodes indicating kernels and edges capturing data dependencies. Node features for the prefill and decode phases are encoded separately, incorporating each node’s Roofline performance as a hardware-specific feature. We also develop a focused data sampling algorithm that emphasizes common configurations of inference requests, LLM architectures, and GPU setups. LLMCO2 improves carbon footprint prediction accuracy by 51%-123% over existing ML-based energy predictors for LLMs across diverse inference requests and GPU configurations.

Figure: Figure 1: A decoder-only LLM’s autoregressive inference consists of prefill and decode phases.
Refer to caption: x1.png

## 2 Background

Autoregressive LLM Inferences. As shown in Figure 2, during autoregressive inference (Pope et al., ) of a decoder-only LLM, all input tokens are processed in parallel during the first iteration, generating the first token—this is the prefill phase (Patel et al., ). The context from the LLM’s attention layers is stored in the key-value (KV) cache for future iterations. Subsequent tokens are then generated using the latest token and the KV cache as inputs, forming the decode phase (Patel et al., ).

Distinct Characteristics of Two Phases. In autoregressive LLM inference, the prefill phase computes and stores context in the KV cache, while the decode phase primarily accesses this cache. The prefill phase is compute-bound, relying on GPU cores, whereas the decode phase is memory-bound, relying on GPU memory. Consequently, the phases differ in latency, energy consumption, and carbon footprint. As shown in Figure 2, during a Bloom-3b inference (BigScience, ) with a batch size of 1 on an Nvidia L4 GPU, the prefill phase’s energy use is negligible for requests with many generated tokens but dominates when fewer tokens are generated. Treating both phases as a single task without sufficient training data significantly reduces energy prediction accuracy.

Kernels in a transformer Layer. As illustrated in Figure 4, a transformer layer (Vaswani et al., ) comprises a masked multi-head attention (MHA) layer and a feed-forward (FF) layer framed by normalization layers. In the MHA layer, the attention mechanism is executed via $Q_{proj}$, $K_{proj}$, and $V_{proj}$ kernels. Techniques like Grouped-Query Attention (Ainslie et al., ) and Flash Attention (Dao, ) reduce memory overhead in LLM inferences by employing fewer key-value heads and fusing attention computations ($fuse_{atten}$). The FF layer operates as a two-layer MLP. To enable tensor parallelism (Narayanan et al., ), two all-reduce kernels are employed to distribute General Matrix Multiply (GEMM) operations across GPUs.

Figure: Figure 3: An example of a all-reduce kernel running on four GPUs.
Refer to caption: x3.png

Figure: Figure 4: The kernels in a transformer layer.
Refer to caption: x4.png

All-reduce. Given the huge memory demands of LLMs and the limited capacity of individual GPUs, multiple GPUs connected via PCIe or NVLink (Patel et al., ) are crucial for LLM inferences. Tensor parallelism (Aminabadi et al., ) splits tensors across GPUs and replicates all layers, providing a significant speedup over other parallelism strategies. To support tensor parallelism, two all-reduce kernels are incorporated into each transformer layer. An all-reduce kernel (Hidayetoglu et al., ) consists of a reduce-scatter operation followed by an all-gather operation, as shown in Figure 3. For instance, a $4\times 4$ matrix, evenly distributed across four GPUs (each holding a column), undergoes reduce-scatter, where each row is assembled and summed on one GPU, followed by all-gather, where the summed values are shared across all GPUs.

Figure: (a) memory.
Refer to caption: x5.png

Roofline model. The Roofline model (Cardwell & Song, ) is a performance analysis tool that estimates a kernel’s performance on a GPU, measured in operations per second (OPs/s). It considers factors like peak GPU throughput, peak memory and network bandwidth, and the kernel’s memory- and network-specific arithmetic intensities. The memory and network Roofline models for an Nvidia V100 GPU (Nvidia, ) are shown in Figures 5(a) and 5(b), respectively. The X-axes represent arithmetic intensity, calculated as total kernel operations divided by total memory or network bytes transferred. A ridge point, or ‘balance’ point, indicates where compute and data movement performance meet. The Nvidia V100 supports FP32, FP16, and INT8 operations with HBM or GDDR memory, resulting in six ridge points in Figure 5(a). Two network interfaces (NVLink and PCIe) yield two ridge points in Figure 5(b). Kernels with arithmetic intensities below the ridge point are memory- or network-bound, while those above are compute-bound. For instance, in Figure 5(a), the INT8 kernel 0 ($K_{0}$) is compute-bound with HBM but memory-bound with GDDR.

Figure: (a) code prompt.
Refer to caption: x7.png

Request characterization. In real-world LLM serving clouds like Microsoft Azure, the distribution of inference configurations such as prompt length and generated token count is not uniform. We analyzed public production traces (Microsoft, ) from Azure’s code completion and conversation services. The prompt length distributions are shown in Figures 6(a) and 6(b), respectively. Code completion prompts are often longer, with a median length of 1.5K tokens, compared to chat prompts with a 1.02K median, as they include substantial chunks of existing code. Most requests have prompt lengths under 3K tokens. The distributions of generated tokens are in Figures 6(c) and 6(d); the median is 13 tokens for code completion and 129 for chat, with most requests generating under 600 tokens. Major LLM serving clouds, including Azure, use a mixed continuous batching policy (Agrawal et al., ), keeping batch sizes typically at $\leq 2$ (Patel et al., ).

**Table 1: The comparison of LLMCO2 against prior work.**
| advantage | LLMCarbon | nn-Meter | DeepEn | NNLQP | ours |
| --- | --- | --- | --- | --- | --- |
| ML-based regression technique | ✗ | ✓ | ✓ | ✓ | ✓ |
| prefill and decode phases | ✗ | ✗ | ✗ | ✗ | ✓ |
| hardware (core, mem, & net) features | ✓ | ✗ | ✗ | ✗ | ✓ |
| tensor parallelism on multi-GPUs | ✗ | ✗ | ✗ | ✗ | ✓ |
| sampling common inference configs | ✗ | ✗ | ✗ | ✗ | ✓ |

## 3 Related Work

We compare LLMCO2 with prior work in Table 1. LLMCarbon (Faiz et al., ) employs an equation-based approach using FLOPs to estimate LLM inference carbon footprints, resulting in inaccuracies. Other models like nn-Meter (Zhang et al., ), DeepEn (Tu et al., ), and NNLQP (Liu et al., ) use random forests or neural networks to predict latency or energy for CNNs and transformers but treat predictions as a single task, overlooking the autoregressive nature of LLM inferences and failing to distinguish between prefill and decode phases. These methods focus solely on architecture-specific features like input sizes and network structures, relying on brute-force sampling across hardware platforms, yet they omit hardware-specific features such as GPU peak throughput, memory bandwidth, and network bandwidth, resulting in reduced accuracy for unseen hardware configurations. Furthermore, they do not address tensor parallelism across multiple GPUs and uniformly sample inference configurations, neglecting common settings like medium prompt lengths and small batch sizes. In contrast, LLMCO2 separates prefill and decode phases, incorporates hardware-specific features, supports multi-GPU tensor parallelism, and prioritizes frequently occurring inference configurations, achieving more precise carbon footprint predictions.

## 4 LLMCO2

Figure: Figure 7: The overview of LLMCO2.
Refer to caption: x11.png

We introduce LLMCO2, an accurate carbon footprint regression model for LLM inferences, as outlined in Figure 7. Firstly, ❶ we propose a novel graph embedding method that encodes a transformer’s layer, running on one or more GPUs during inference, as a graph. In the graph, nodes correspond to the kernels within the transformer layer, while edges depict data dependencies between kernels. Each node is annotated with architectural features such as operation count, memory access size, and network transfer size, along with a hardware-specific feature, i.e., its Roofline performance. Node features are divided into two sets, one for the prefill phase and the other for the decode phase. We then employ graph convolution layers to process this graph. Global LLM architectural features, including total operation count, memory access, and network transfer size, are integrated with the processed graph features, and two linear layers use these to predict the LLM inference’s operational energy. To convert operational energy consumption to carbon footprint, we apply the equation from Faiz et al. ():

$$ $CO2eq_{oper}=energy_{oper}\cdot PUE\cdot carb_inten,$ (1) $$

where $CO2eq_{oper}$ represents the operational carbon footprint, $PUE$ denotes the power usage effectiveness, and $carb_inten$ indicates the carbon intensity of the data center. The embedded carbon footprint is calculated using the equations provided in Faiz et al. (). Secondly, ❷ we introduce a focused data sampling technique that targets common inference configurations, LLM architectures, and GPU setups frequently encountered by commercial cloud users. This technique generates numerous combinations of inference, LLM architecture, and GPU configurations to build the training dataset. For each combination, the configurations extract kernels in each transformer layer and global LLM features, following various kernel optimizations. These kernels, global LLM features, and hardware configurations form the input data for LLMCO2, while ground truth energy data is gathered from real GPUs, adhering to the specified configurations.

Figure: Figure 8: Converting the kernels in a transformer layer into a graph.
Refer to caption: x12.png

### 4.1 Graph Embedding

We introduce a graph embedding technique for LLMCO2 to represent a transformer layer as a directed acyclic graph (DAG) of kernels: $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ is the set of nodes (kernels), and $\mathcal{E}$ denotes the edges (data dependencies between kernels). Each edge’s source kernel supplies input data to the destination kernel. The key advantages of this technique are: (1) each node has two distinct feature sets—one for the prefill phase and the other for the decode phase, and (2) each node includes a hardware-specific feature, i.e., Roofline performance. Figure 8 provides an example of this graph embedding for a transformer layer.

Node features. A node $v\in\mathcal{V}$ corresponds to a kernel in the transformer layer. To accurately estimate the carbon footprint of the two distinct phases in an LLM inference, each node is assigned two sets of features: one for the prefill phase and the other for the decode phase. Each feature set concatenates the following elements:

- •
$T$: Defines the type of the kernel. For instance, the value projection kernel $V_{proj}$ multiplies the input with its weight matrix $\mathbf{W}_{V}$. More kernel types are detailed in Tables 6 and 7 of Appendix A. A one-hot vector is used to encode $T$.
- •
$S$: Represents the dimensions of the kernel’s input, weight, and output.
- •
$O$: Refers to the number of operations of the kernel. While tools like CALFLOPS (Ye, ) can compute operation counts for kernels based on LLM architecture, they do not differentiate between the prefill and decode phases, or support the all-reduce kernel. Therefore, we provide equations for calculating $O$ for the kernels used (excluding the all-reduce kernel) in Appendix A. For the same kernel, the operation count in the decode phase ($O^{dec}$) is proportional to the number of generated tokens ($N_{gT}$), while in the prefill phase ($O^{pre}$), it is proportional to the input sequence length ($L_{seq}$). For the all-reduce kernel operating on an $n\times m$ matrix distributed across $l$ GPUs (each holding an $n/l\times m$ portion), the operation count during the decode phase is given by Equation 2, and during the prefill phase by Equation 3. These operations occur during the reduce-scatter step.
$O_{allr}^{dec}=n\cdot\frac{m}{l}\cdot(N_{gT}-1)$
(2)
$O_{allr}^{pre}=n\cdot\frac{m}{l}\cdot L_{seq}$
(3)
- •
$M$: Denotes the total memory footprint accessed by the kernel. We also provide equations for computing $M$ (excluding the all-reduce kernel) in Appendix A. As with $O$, the memory footprint during the decode phase ($M^{dec}$) is proportional to $N_{gT}$, while in the prefill phase ($M^{pre}$), it is proportional to $L_{seq}$. For the all-reduce kernel, the total memory footprint during the decode phase is determined by Equation 4, where $D_{A}$ represents the data type of activations (e.g., FP16), and in the prefill phase by Equation 5.
$M_{allr}^{dec}=2O_{allr}^{dec}\cdot D_{A}$
(4)
$M_{allr}^{pre}=2O_{allr}^{pre}\cdot D_{A}$
(5)
- •
$I$: Indicates the data size transferred over the GPU network interface for an all-reduce kernel. When the all-reduce kernel operates on an $n\times m$ matrix distributed across $l$ GPUs, the data size transferred during the decode phase is calculated using Equation 6, and during the prefill phase using Equation 7.
$I_{allr}^{dec}=\frac{n}{l}\cdot m\cdot(l-1)\cdot D_{A}\cdot(N_{gT}-1)$
(6)
$I_{allr}^{pre}=\frac{n}{l}\cdot m\cdot(l-1)\cdot D_{A}\cdot L_{seq}$
(7)
- •
$P$: Denotes the kernel’s Roofline performance. The memory-related arithmetic intensity (MAI) for a non-all-reduce kernel is calculated as $O/M$, while the network-related arithmetic intensity (NAI) for an all-reduce kernel is $O/I$. The memory-related ridge point (MRP) of a GPU is computed as $Th_{max}/BW_{max}$, and the network-related ridge point (NRP) as $Th_{max}/NET_{max}$. Here, $Th_{max}$ is the GPU’s peak computational throughput for a specific data type (e.g., INT8), $BW_{max}$ is the maximum memory bandwidth, and $NET_{max}$ represents the peak network bandwidth. The kernel’s Roofline performance is determined by Equation 8. For non-all-reduce kernels, if the MAI is below the MRP, $P$ equals the product of $BW_{max}$ and the MAI; otherwise, $P$ equals $Th_{max}$. For all-reduce kernels, if the NAI is less than the NRP, $P$ is the product of $NET_{max}$ and the NAI; otherwise, $P$ is $Th_{max}$. Including $P$ as a node feature is essential to account for hardware-specific characteristics, such as $Th_{max}$, $BW_{max}$, and $NET_{max}$, in LLM inference energy modeling.

$$ $P=\begin{cases}BW_{max}\cdot MAI\text{ (}NET_{max}\cdot NAI\text{)}&\text{if }MAI<MRP\text{ (}NAI<NRP\text{)}\\ Th_{max}&\text{otherwise}\end{cases}$ (8) $$

Global LLM features. In addition to node features, LLMCO2 incorporates global LLM features, covering the overall architecture of an LLM, batch size, prompt length, generated token count, total FLOP count, memory read/write data size, and network transfer data size. The architectural details include quantization bitwidth, hidden size, intermediate size, head count, and layer count.

Graph Conv. We adopted the GraphSAGE layer (Hamilton et al., ) for LLMCO2.

### 4.2 Focused Energy Data Sampling

Figure: Algorithm 1 Focused energy data sampling.

To train a carbon footprint regression model, it is crucial to construct a training dataset by sampling a variety of LLM architectural, inference, and hardware configurations. However, random sampling proves ineffective, often resulting in suboptimal predictive models due to the exclusion of vital data related to LLM architecture and inference configurations. Creating a new LLM demands considerable resources, including substantial training data and computational power. Consequently, most LLMs in academia and industry are based on a few foundational models (Meta, ), sharing similar architectural attributes, such as head count, layer number, hidden size, and intermediate size. Neglecting these established architectural parameters and relying on random sampling leads to poor regression accuracy. Additionally, as depicted in Figure 6, the distributions of input prompt lengths and generated token counts are non-uniform in major cloud-based LLM services. Treating all inference configurations uniformly and sampling evenly across all possible inference configurations yields an ineffective training dataset, failing to capture the prevalent configuration ranges of inference requests, thereby compromising the accuracy of carbon footprint predictions.

Focused data sampling. Instead of employing random sampling, we propose a focused energy data sampling strategy, as outlined in Algorithm 1, to identify the most influential data points within the space of LLM architectural, inference, and hardware configurations that significantly affect prediction accuracy. Our approach deliberately omits rarely encountered configurations, such as an LLM with a hidden size of 32 or an inference request generating 16K tokens. Instead, our method iteratively samples additional data around points with high prediction errors. In line 1 of Algorithm 1, we initially select $A=50K$ data points from the combined space of LLM architecture $\mathcal{A}$, inference $\mathcal{I}$, and hardware $\mathcal{H}$ configurations. We curated $\mathcal{A}$ using 17 state-of-the-art LLMs, adjusting parameters such as head count, layer count, intermediate size, quantization bitwidth, and hidden size. Inference configurations ($\mathcal{I}$) were derived from public LLM inference request traces (Microsoft, ), while four GPU configurations (as detailed in Table 2) were considered for $\mathcal{H}$. Our experimental setup is further explained in Section 5.1. To assess the quality of the sampled data, we train a GNN-based predictor and construct a test dataset (lines 1-1), incorporating 20% of the newly sampled data in each iteration. We then perform fine-grained sampling on data points with significant regression errors (lines 1-1). For each data point, we randomly sample $B=100$ data points within a range of $\pm C$ from their original values in $\mathcal{A}\times\mathcal{I}\times\mathcal{H}$, where $C$ varies for different configurations. For instance, $C=10$ for input prompt length, $C=1$ for generated token count, and $C=1$ for layer number. This iterative process continues until the energy predictor’s accuracy reaches the desired target (lines 1-1).

**Table 2: The configuration of GPUs used for LLM inferences.**
| GPU | max throughput (TOPs/s) | memory | network | power | node | area | tech |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FP32 | FP16 | INT8 | (GB/s) | (GB/s) | ($W$) | size | ($mm^{2}$) | (nm) |  |
| T4 | 8.1 | 65 | 130 | 320 | 64 | 70 | 4 | 545 | 12 |
| L4 | 121 | 242 | 485 | 300 | 64 | 72 | 4 | 294 | 5 |
| A100 | 312 | 624 | 1248 | 2039 | 600 | 400 | 4 | 826 | 7 |
| H100 | 989 | 1979 | 3958 | 3350 | 900 | 700 | 4 | 814 | 5 |

## 5 Evaluation

### 5.1 Experimental methodology

Dataset construction. We developed an energy dataset to evaluate the performance of various energy prediction methods, selecting six LLM series: Bloom (Bloom-560m, Bloom-1b1, Bloom-1b7, Bloom-3b, Bloom-7b1) (BigScience, ), Gemma (Gemma-2b, Gemma-7b) (Team et al., ), Gemma2 (Gemma2-2b, Gemma2-9b, Gemma2-27b) (Team et al., ), Qwen2 (Qwen2-0.5b, Qwen2-1.5b, Qwen2-7b, Qwen2-72b) (Yang et al., ), Llama3.1 (Llama3.1-8b, Llama3.1-70b) (Meta, ), and Mixtral (Mixtral-8×7b) (Jiang et al., ). The LLM architectural design space was explored by varying parameters such as quantization bitwidth, hidden size, intermediate size, head count, and layer count. Public Azure LLM serving traces (Microsoft, ) were used to generate inference requests, encompassing two distinct traces: one for chat (19,336 entries) and the other for code completion (8,199 entries), for LLMCO2. We sampled inference configurations by adjusting input prompt lengths and generated token numbers, ensuring batch sizes remained under two. We adopted random sampling to generate inference requests with different input prompt lengths and generated token numbers for our baseline schemes. LLM inferences were executed using the GPU configurations detailed in Table 2, with the number of GPUs per inference ranging from the minimum required to a maximum of four, regardless of each hardware configuration’s total GPU capacity. For the training dataset, we considered L4, A100, and H100 GPUs, while all four GPU configurations were included in the test dataset.

Measurement and implementation. We used the Nvidia Management Library (NVML) (NVIDIA, ) to measure the energy consumption of LLM inferences on the target GPUs. Each inference was executed 5 times, and the average energy consumption was recorded as the ground truth. LLMCO2 consists of two graph convolution layers and two linear layers. LLMCO2 was developed using the PyG package (Team, ) and trained on a Tesla L4 GPU. The model training used the Adam optimizer, with a learning rate of 0.001 and a batch size of 512.

Evaluation metrics. We evaluated prediction accuracy using the Mean Absolute Percentage Error (MAPE) and Error Bound Accuracy (EBA($\delta$)). MAPE quantifies the average absolute percentage deviation between predicted and actual energy values, with lower values indicating higher accuracy. EBA($\delta$) represents the percentage of predictions within a specified error bound $\delta$ of the ground truth, with higher values reflecting greater regression precision.

Baseline Schemes. To assess the performance of LLMCO2, we compared it against three baseline schemes: LLMCarbon (Faiz et al., ), DeepEn (Tu et al., ), and NNLQP (Liu et al., ). LLMCarbon is an equation-based approach that estimates carbon footprint by counting only the FLOPs involved in an LLM inference. DeepEn (Tu et al., ) samples the energy consumption of each kernel across different GPUs and utilizes this dataset to train a random forest-based predictor. The only distinction between DeepEn and nn-Meter (Zhang et al., ) is that nn-Meter is trained to predict inference latency instead of energy consumption. NNLQP (Liu et al., ), a GNN-based energy predictor, represents each model layer as a graph. However, neither DeepEn nor NNLQP distinguishes between the prefill and decode phases of LLM inferences, incorporates hardware-specific features, or emphasizes sampling common LLM architecture, inference, and GPU configurations frequently used in cloud environments.

### 5.2 Operational energy results

**Table 3: The mean absolute percentage error (MAPE) comparison between various energy predictors. (The lower, the better)**
| Scheme | Gemma | Bloom | Gemma2 | Qwen2 | Mixtral | Llama3.1 | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- |
| LLMCarbon | 89.2% | 98.5% | 101.2% | 71.3% | 156% | 233% | 124.9% |
| DeepEn | 34.3% | 39.1% | 41.7% | 33.8% | 23.2% | 19.7% | 31.9% |
| NNLQP | 26.8% | 30.6% | 31.6% | 21.5% | 34.1% | 26.2% | 28.5% |
| LLMCO2 | 15.8% | 11.9% | 21.1% | 6.3% | 19.4% | 18.3% | 15.5% |

MAPE. Table 3 presents the comparison of Mean Absolute Percentage Error (MAPE) between LLMCO2 and various baseline schemes. On average, LLMCO2 achieves the lowest MAPE values across different LLMs. LLMCarbon estimates operational energy consumption based solely on FLOP counts, neglecting memory accesses within transformer layers and critical hardware-specific features, such as peak GPU memory and network bandwidth, resulting in the highest MAPE values for all LLMs. The two ML-based predictors, DeepEn and NNLQP, exhibit comparable average MAPE values. Due to a smaller training energy dataset size for Mixtral and Llama3.1, DeepEn, leveraging its random forest model, performs slightly better than NNLQP on these LLMs. Overall, LLMCO2 outperforms DeepEn and NNLQP, reducing the average MAPE by 51.4% and 45.6%, respectively, by treating the prefill and decode phases separately, incorporating kernel-specific Roofline performance, and training with energy data derived from public Azure LLM serving traces.

**Table 4: The error bound accuracy (EBA) comparison between various energy predictors. (The higher, the better)**
| Metric | Scheme | Gemma | Bloom | Gemma2 | Qwen2 | Mixtral | Llama3.1 | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EBA($30\%$) | LLMCarbon | 7.8% | 6.1% | 3.8% | 11.9% | 4.2% | 3.5% | 6.2% |
| DeepEn | 58.5% | 48.9% | 44.8% | 60.5% | 41% | 36.6% | 48.4% |  |
| NNLQP | 60% | 50.1% | 45.6% | 65.2% | 37.5% | 33.4% | 48.6% |  |
| LLMCO2 | 77.9% | 85.3% | 71.3% | 80.1% | 64.8% | 62.6% | 73.6% |  |
| EBA($10\%$) | LLMCarbon | 2.1% | 1.9% | 0.8% | 7.7% | 1.3% | 0.8% | 2.4% |
| DeepEn | 25% | 13% | 12.5% | 30.2% | 13.2% | 11.6% | 17.6% |  |
| NNLQP | 32.6% | 15.7% | 19.3% | 36.5% | 9.8% | 9.2% | 20.5% |  |
| LLMCO2 | 53.3% | 60% | 40.3% | 53.1% | 35.3% | 32.1% | 45.7% |  |
| EBA($5\%$) | LLMCarbon | 0.1% | 0% | 0.1% | 0.3% | 0% | 0% | 1.1% |
| DeepEn | 11.6% | 8.2% | 4.7% | 16.1% | 7.2% | 6.8% | 9.1% |  |
| NNLQP | 11.9% | 13.2% | 7.4% | 19.2% | 6.3% | 6.1% | 10.7% |  |
| LLMCO2 | 27.1% | 46.7% | 12.1% | 33.7% | 9.3% | 7.3% | 22.7% |  |

EBA. Table 4 presents the Error Bound Accuracy (EBA) for various energy predictors at 5%, 10%, and 30% error bounds. LLMCO2 consistently achieves the highest EBA values across all error bounds and LLMs. In contrast, LLMCarbon records the lowest EBA values, as it fails to account for memory accesses and network data transfers, resulting in poor performance, especially with LLMs featuring complex decode phases, such as Gemma2, and those constrained by all-reduce kernels, like Mixtral and Llama3.1. Although DeepEn and NNLQP exhibit similar MAPE values, NNLQP achieves higher EBA values at smaller error bounds due to its more effective GNN model in energy data regression when ample training data is available. Ultimately, LLMCO2 outperforms DeepEn and NNLQP, improving the average EBA($10\%$) by 160% and 123%, respectively.

### 5.3 Ablation studies

We conducted ablation studies on EBA($10\%$) to evaluate the contribution of each component of LLMCO2, as summarized in Table 5. By using distinct node features for the prefill and decode phases, LLMCO2 improves EBA($10\%$) by 67% compared to NNLQP. In real-world LLM serving clouds, most inference requests involve medium-length prompts and fewer generated tokens, leading to significant errors when combining the two phases for carbon overhead prediction. Incorporating Roofline performance as a node feature further boosts LLMCO2’s EBA($10\%$) by 13.1%, as this feature facilitates knowledge transfer from L4 to T4 GPUs in the test dataset. Finally, the focused energy data sampling technique elevates LLMCO2’s EBA($10\%$) improvement to 123% of NNLQP, since training with data distributions that mirror real-world LLM inference configurations enhances its performance on test datasets with similar prompt lengths and generated token distributions.

**Table 5: The EBA($10\%$) comparison between various components of LLMCO2.**
| Scheme | Gemma | Bloom | Gemma2 | Qwen2 | Mixtral | Llama3.1 | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- |
| +prefill/decode | 41.5% | 38.5% | 29.5% | 44.6% | 26.9% | 24.6% | 34.3% |
| +Roofline | 45.6% | 46.3% | 33.5% | 47.9% | 30.8% | 28.6% | 38.8% |
| +focused sample | 53.3% | 60% | 40.3% | 53.1% | 35.3% | 32.1% | 45.7% |

Figure: Figure 9: The comparison of carbon per GPU per second.
Refer to caption: x13.png

## 6 User Case Studies

Carbon comparison between training and inference. LLM training requires numerous GPUs operating at high throughput over extended periods. For example, Mixtral-8×7b training uses 512 A100 GPUs at about 50% peak throughput for three months (Jiang et al., ). In contrast, INT8 Mixtral-8×7b inference utilizes four A100 GPUs at 10%–40% peak throughput for around 4 seconds. Figure 11 shows the normalized carbon footprint per GPU per second for Mixtral-8×7b training and inference, relative to the training baseline. The embodied carbon per GPU per second is consistent across all configurations since A100 GPUs are used. The prefill phase of an inference with a batch size of 4 and a 6K prompt length has a similar operational carbon footprint per GPU per second to training, while the decode phase with a batch size of 1 and a 1K prompt length has a much lower footprint. Unlike training, the embodied carbon overhead dominates the decode phase of Mixtral-8×7b inferences.

Inference on multiple GPUs. Additional GPUs can accelerate LLM inference (Pope et al., ), but using more GPUs than necessary is often unsustainable, particularly for real-world cloud inference configurations with small batch sizes and short prompts. Figure 11 shows the operational carbon footprint of Bloom-7b1 inferences across different GPU counts with varying prompt lengths and batch sizes. For example, although an inference with a batch size of 4 and a 1K-token prompt benefits from 2 or 4 GPUs, lowering the per-GPU carbon footprint, an inference with a batch size of 1 and a 64-token prompt incurs increased latency and higher per-GPU operational carbon due to the communication overhead of all-reduce kernels when using more GPUs. As shown in Figure 11, the total carbon overhead rises considerably with more GPUs for inferences with smaller batch sizes and prompts.

## 7 Conclusion

LLM inference produces a larger carbon footprint than training, necessitating accurate estimation tools for both users and cloud providers. Existing models fall short due to their inability to capture LLM autoregressive behaviors, hardware-specific features, and real-world configuration distribution. We presented LLMCO2, a GNN-based model to address these challenges, offering improved accuracy in predicting the carbon footprint of LLM inferences compared to prior methods.

## Appendix A The kernels in a transformer layer

Linear kernels. The kernels in a transformer layer without kernel optimizations are illustrated in Figure 12. The configuration with flash-attention is shown in Figure 12(a), while the configuration without flash-attention is depicted in Figure 12(b). The operation counts and GPU memory usage for all kernels in a transformer layer, with and without flash-attention, are provided in Table 6 and Table 7, respectively. The operation count for all linear kernels during the decode phase ($O_{linear}^{dec}$) is computed as:

$$ $O_{linear}^{dec}=2\cdot|B|\cdot d_{in}\cdot d_{out}\cdot(N_{gT}-1)/N_{GPU},$ (9) $$

where $|B|$ is the batch size, $d_{in}$ is the input dimension, $d_{out}$ is the output dimension, $N_{gT}$ is the number of generated tokens and $N_{GPU}$ is the number of GPUs. During the decode phase, each linear kernel loads weights, the size of which is given by:

$$ $M_{W,load,linear}^{dec}=d_{in}\cdot d_{out}\cdot D_{W}\cdot(N_{gT}-1)/N_{GPU},$ (10) $$

where $D_{W}$ represents the data type of the weights, such as FP16. Each linear kernel reads activation values, the size of which can be calculated as:

$$ $M_{A,load,linear}^{dec}=d_{in}\cdot|B|\cdot D_{A}\cdot(N_{gT}-1)/N_{GPU},$ (11) $$

where $D_{A}$ is the data type of the activation values. Only the $K_{proj}$ and $V_{proj}$ kernels need to store activation values, with the size computed as:

$$ $M_{A,store,linear}^{dec}=d_{out}\cdot|B|\cdot D_{A}\cdot(N_{gT}-1)/N_{GPU}.$ (12) $$

For other linear kernels, $M_{A,store,linear}^{dec}=0$. No linear kernel reads from the KV cache, but most types of linear kernels store data in the KV cache, except for the $K_{proj}$ and $V_{proj}$ kernels, where $M_{KV,store,linear}^{dec}=0$. The size of KV cache data stored by other linear kernels during the decode phase is given by:

$$ $M_{KV,store,linear}^{dec}=d_{out}\cdot|B|\cdot D_{KV}\cdot(N_{gT}-1)/N_{GPU},$ (13) $$

where $D_{KV}$ represents the data type of the KV cache values. The total memory access for linear kernels during the decode phase can be computed as:

$$ $M^{dec}=M_{W,load,linear}^{dec}+M_{A,load,linear}^{dec}+M_{A,store,linear}^{dec}+M_{KV,store,linear}^{dec}.$ (14) $$

In contrast, for all linear kernels, the operation count during the prefill phase ($N_{op,linear}^{pre}$) can be calculated as:

$$ $N_{linear}^{pre}=2\cdot|B|\cdot d_{in}\cdot d_{out}\cdot L_{seq}/N_{GPU},$ (15) $$

where $L_{seq}$ represents the input prompt sequence length. During the prefill phase, each linear kernel loads weights, with the size calculated as:

$$ $M_{W,load,linear}=d_{in}\cdot d_{out}\cdot D_{W}/N_{GPU},$ (16) $$

where $D_{W}$ is the data type of the weights, such as FP16. The size of activation values for each linear kernel during the prefill phase is computed as:

$$ $M_{A,load,linear}^{pre}=d_{in}\cdot|B|\cdot D_{A}\cdot L_{seq}/N_{GPU}.$ (17) $$

The activation value size in the prefill phase can be computed as:

$$ $M_{A,store,linear}^{pre}=d_{out}\cdot|B|\cdot D_{A}\cdot L_{seq}/N_{GPU}.$ (18) $$

No linear kernel reads from the KV cache, but most types of linear kernels store data in the KV cache. Only the $K_{proj}$ and $V_{proj}$ kernels do not store data in the KV cache, so $M_{A,store,linear}^{pre}=0$ for these kernels. The size of data stored in the KV cache by other linear kernels during the prefill phase can be computed as:

$$ $M_{KV,store,linear}^{pre}=d_{out}\cdot|B|\cdot D_{KV}\cdot L_{seq}/N_{GPU}.$ (19) $$

The total memory access for linear kernels during the prefill phase can be calculated as:

$$ $M^{pre}=M_{W,load,linear}+M_{A,load,linear}^{pre}+M_{A,store,linear}^{pre}+M_{KV,store,linear}^{pre}.$ (20) $$

**Table 6: The computation, memory access, and network transfer of kernels with flash attention.**
| kernel | OPs | memory | network |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| prefill | decode | prefill | decode | prefill | decode |  |
| $norm_{attn}$ | Eq 51 | Eq 43 | Eq 57 | Eq 49 | 0 | 0 |
| $Q_{proj}$ | Eq 15 | Eq 9 | Eq 20 | Eq 14 | 0 | 0 |
| $K_{proj}$ | Eq 15 | Eq 9 | Eq 20 | Eq 14 | 0 | 0 |
| $V_{proj}$ | Eq 15 | Eq 9 | Eq 20 | Eq 14 | 0 | 0 |
| $fuse_{attn}$ | Eq 38 | Eq 27 | Eq 42 | Eq 31 | 0 | 0 |
| $out_{proj}$ | Eq 15 | Eq 9 | Eq 20 | Eq 14 | 0 | 0 |
| $add_{attn}$ | Eq 52 | Eq 44 | Eq 57 | Eq 49 | 0 | 0 |
| $norm_{mlp}$ | Eq 51 | Eq 43 | Eq 57 | Eq 49 | 0 | 0 |
| $gate_{proj}$ | Eq 15 | Eq 9 | Eq 20 | Eq 14 | 0 | 0 |
| $up_{proj}$ | Eq 15 | Eq 9 | Eq 20 | Eq 14 | 0 | 0 |
| $act_{mlp}$ | Eq 53 | Eq 45 | Eq 58 | Eq 50 | 0 | 0 |
| $down_{proj}$ | Eq 15 | Eq 9 | Eq 20 | Eq 14 | 0 | 0 |
| $add_{mlp}$ | Eq 52 | Eq 44 | Eq 57 | Eq 49 | 0 | 0 |
| all reduce | Eq 3 | Eq 2 | Eq 5 | Eq 4 | Eq 7 | Eq 6 |

Kernels in attention. The attention head dimension $d_{h}$ equals the hidden size ($size_{h}$) divided by the number of attention heads ($n_{h}$). For a transformer layer with no flash attention, in the decode phase, the operation number of a kernel of $matmul_{SV}$ or $matmul_{QK}$ ($O_{matmul}^{dec}$) can be calculated as

$$ $O_{matmul}^{dec}=|B|\cdot d_{h}\cdot n_{h}\cdot(2L_{seq}+N_{gT})\cdot N_{gT}/N_{GPU},$ (21) $$

where $L_{seq}$ is the input prompt length, while $N_{gT}$ is the number of newly generated tokens. The operation number in a softmax activation in the decode phase can be computed as

$$ $O_{softmax}^{dec}=5|B|\cdot n_{h}\cdot(2L_{seq}+N_{gT})\cdot N_{gT}/2/N_{GPU}.$ (22) $$

These kernels do not need to load weights or store data in the KV cache. In the decode phase, the kernel of $matmul_{SV}$, $matmul_{QK}$ or softmax loads or stores activation values, whose size is

$$ $M_{A,load/store,matmul/softmax}^{dec}=|B|\cdot n_{h}\cdot D_{A}\cdot(2L_{seq}+N_{gT})\cdot N_{gT}/2/N_{GPU},$ (23) $$

where $D_{A}$ is the data type of activations. The kernel of $matmul_{SV}$ or $matmul_{QK}$ also needs to load data from the KV cache and the size of these data can be computed as

$$ $M_{KV,load,matmul}^{dec}=|B|\cdot d_{h}\cdot n_{kv}\cdot D_{KV}\cdot(2L_{seq}+N_{gT})\cdot N_{gT}/2/N_{GPU},$ (24) $$

where $n_{KV}$ is the number of key-value heads, while $D_{KV}$ indicates the data type of KV cache data. In the decode phase, the total memory access size of $matmul_{SV}$ or $matmul_{QK}$ can be computed as

$$ $M_{matmul}^{dec}=M_{A,load,matmul}^{dec}+M_{A,store,matmul}^{dec}+M_{KV,load,matmul}^{dec},$ (25) $$

while the total memory access size of softmax is

$$ $M_{softmax}^{dec}=M_{A,load,softmax}^{dec}+M_{A,store,softmax}^{dec}.$ (26) $$

For a transformer layer with flash attention, during the decode phase, the operation count for the $fuse_{attn}$ kernel can be calculated as:

$$ $O_{fuse}^{dec}=2O_{matmul}^{dec}+O_{softmax}^{dec}.$ (27) $$

The total size of activation values loaded by $fuse_{attn}$ can be computed as:

$$ $M_{A,load,fuse}^{dec}=d_{h}\cdot|B|\cdot n_{h}\cdot D_{A}\cdot(N_{gT}-1)/N_{GPU}.$ (28) $$

The total size of activation values stored by $fuse_{attn}$ is given by:

$$ $M_{A,store,fuse}^{dec}=2d_{h}\cdot|B|\cdot n_{h}\cdot D_{A}\cdot(N_{gT}-1)/N_{GPU},$ (29) $$

The total size of data loaded from the KV cache by $fuse_{attn}$ is:

$$ $M_{KV,load,fuse}^{dec}=2|B|\cdot s_{block}\cdot d_{h}\cdot n_{KV}\cdot D_{KV}\cdot(2L_{seq}+N_{gT})\cdot N_{gT}/2/N_{GPU}.$ (30) $$

where $s_{block}$ is the number of KV-heads that can be stored in the GPU on-chip memory, and $n_{KV}$ is the number of key-value heads. The total memory access size of $fuse_{attn}$ during the decode phase is:

$$ $M_{fuse}^{dec}=M_{A,load,fuse}^{dec}+M_{KV,load,fuse}^{dec}+M_{KV,load,fuse}^{dec}$ (31) $$

In contrast, during the prefill phase, the operation count for the kernel of $matmul_{SV}$ or $matmul_{QK}$ ($O_{matmul}^{pre}$) can be calculated as:

$$ $O_{matmul}^{pre}=2|B|\cdot d_{h}\cdot n_{h}\cdot L_{seq}/N_{GPU}.$ (32) $$

The operation count for a softmax activation in the prefill phase is computed as:

$$ $O_{softmax}^{pre}=5|B|\cdot n_{h}\cdot L_{seq}/N_{GPU}.$ (33) $$

These kernels do not need to load weights or store data in the KV cache. In the prefill phase, the kernel of $matmul_{SV}$, $matmul_{QK}$, or softmax loads or stores activation values, the size of which is:

$$ $M_{A,load/store,matmul/softmax}^{pre}=|B|\cdot n_{h}\cdot D_{A}\cdot L_{seq}/N_{GPU},$ (34) $$

The kernel of $matmul_{SV}$ or $matmul_{QK}$ also loads data from the KV cache, with the size computed as:

$$ $M_{KV,load,matmul}^{pre}=|B|\cdot d_{h}\cdot n_{kv}\cdot D_{KV}\cdot L_{seq}/N_{GPU}.$ (35) $$

In the prefill phase, the total memory access size for $matmul_{SV}$ or $matmul_{QK}$ can be computed as:

$$ $M_{matmul}^{pre}=M_{A,load,matmul}^{pre}+M_{A,store,matmul}^{pre}+M_{KV,load,matmul}^{pre},$ (36) $$

while the total memory access size for softmax is:

$$ $M_{softmax}^{pre}=M_{A,load,softmax}^{pre}+M_{A,store,softmax}^{pre}.$ (37) $$

For a transformer layer with flash attention, in the prefill phase, the operation count for the $fuse_{attn}$ kernel can be computed as:

$$ $O_{fuse}^{pre}=(2O_{matmul}^{pre}+O_{softmax}^{pre})\cdot L_{seq}.$ (38) $$

The total size of activation values loaded by $fuse_{attn}$ can be calculated as:

$$ $M_{A,load,fuse}^{pre}=d_{h}\cdot|B|\cdot n_{h}\cdot D_{A}\cdot L_{seq}/N_{GPU},$ (39) $$

while the total size of activation values stored by $fuse_{attn}$ is:

$$ $M_{A,store,fuse}^{pre}=2d_{h}\cdot|B|\cdot n_{h}\cdot D_{A}\cdot L_{seq}/N_{GPU},$ (40) $$

The total size of data loaded from the KV cache by $fuse_{attn}$ is:

$$ $M_{KV,load,fuse}^{pre}=2|B|\cdot s_{block}\cdot d_{h}\cdot n_{KV}\cdot D_{KV}\cdot L_{seq}/N_{GPU}.$ (41) $$

The total memory access size for $fuse_{attn}$ during the prefill phase is:

$$ $M_{fuse}^{pre}=M_{A,load,fuse}^{pre}+M_{KV,load,fuse}^{pre}+M_{KV,load,fuse}^{pre}$ (42) $$

**Table 7: The computation, memory access, and network transfer of kernels without flash attention.**
| kernel | OPs | memory | network |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| prefill | decode | prefill | decode | prefill | decode |  |
| $norm_{attn}$ | Eq 51 | Eq 43 | Eq 57 | Eq 49 | 0 | 0 |
| $Q_{proj}$ | Eq 15 | Eq 9 | Eq 20 | Eq 14 | 0 | 0 |
| $K_{proj}$ | Eq 15 | Eq 9 | Eq 20 | Eq 14 | 0 | 0 |
| $V_{proj}$ | Eq 15 | Eq 9 | Eq 20 | Eq 14 | 0 | 0 |
| $matmul_{QK}$ | Eq 32 | Eq 21 | Eq 25 | Eq 36 | 0 | 0 |
| $softmax$ | Eq 33 | Eq 22 | Eq 26 | Eq 37 | 0 | 0 |
| $matmul_{SV}$ | Eq 32 | Eq 21 | Eq 25 | Eq 36 | 0 | 0 |
| $add_{attn}$ | Eq 52 | Eq 44 | Eq 57 | Eq 49 | 0 | 0 |
| $norm_{mlp}$ | Eq 51 | Eq 43 | Eq 57 | Eq 49 | 0 | 0 |
| $gate_{proj}$ | Eq 15 | Eq 9 | Eq 20 | Eq 14 | 0 | 0 |
| $up_{proj}$ | Eq 15 | Eq 9 | Eq 20 | Eq 14 | 0 | 0 |
| $act_{mlp}$ | Eq 53 | Eq 45 | Eq 58 | Eq 50 | 0 | 0 |
| $down_{proj}$ | Eq 15 | Eq 9 | Eq 20 | Eq 14 | 0 | 0 |
| $add_{mlp}$ | Eq 52 | Eq 44 | Eq 57 | Eq 49 | 0 | 0 |
| all reduce | Eq 3 | Eq 2 | Eq 5 | Eq 4 | Eq 7 | Eq 6 |

normalization, residual add, and MLP act. For these layers, no weights are loaded, and there is no access to the KV cache. For each normalization layer during the decode phase, the operation count can be computed as:

$$ $O_{norm}^{dec}=7|B|\cdot size_{h}\cdot(N_{gT}-1)/N_{GPU},$ (43) $$

where $size_{h}$ represents the hidden size, and $N_{gT}$ is the number of generated tokens. For each residual add layer during the decode phase, the operation count can be computed as:

$$ $O_{add}^{dec}=|B|\cdot size_{h}\cdot(N_{gT}-1)/N_{GPU}.$ (44) $$

For each MLP activation layer during the decode phase, the operation count is:

$$ $O_{act}^{dec}=2|B|\cdot size_{h}\cdot(N_{gT}-1)/N_{GPU}.$ (45) $$

The size of activation values loaded or stored by a normalization or residual add layer during the decode phase can be computed as:

$$ $M_{A,load/store,norm/add}^{dec}=|B|\cdot size_{h}\cdot D_{A}\cdot(N_{gT}-1)/N_{GPU}.$ (46) $$

For an MLP activation layer, the size of activation values loaded during the decode phase is:

$$ $M_{A,load,act}^{dec}=2|B|\cdot size_{h}\cdot D_{A}\cdot(N_{gT}-1)/N_{GPU},$ (47) $$

and the size of activation values stored is:

$$ $M_{A,store,act}^{dec}=|B|\cdot size_{h}\cdot D_{A}\cdot(N_{gT}-1)/N_{GPU}.$ (48) $$

Thus, for each normalization or residual add layer, the total memory access size during the decode phase is:

$$ $M_{norm/add}^{dec}=2M_{A,load/store,norm/add}^{dec}.$ (49) $$

For each MLP activation layer, the total memory access size is:

$$ $M_{act}^{dec}=3M_{A,load,act}^{dec}.$ (50) $$

For each normalization layer during the prefill phase, the operation count can be computed as:

$$ $O_{norm}^{pre}=7|B|\cdot size_{h}\cdot L_{seq}/N_{GPU},$ (51) $$

For each residual add layer during the prefill phase, the operation count can be computed as:

$$ $O_{add}^{pre}=|B|\cdot size_{h}\cdot L_{seq}/N_{GPU}.$ (52) $$

For each MLP activation layer during the prefill phase, the operation count is:

$$ $O_{act}^{pre}=2|B|\cdot size_{h}\cdot L_{seq}/N_{GPU}.$ (53) $$

The size of activation values loaded or stored by a normalization or residual add layer during the prefill phase can be computed as:

$$ $M_{A,load/store,norm/add}^{pre}=|B|\cdot size_{h}\cdot D_{A}\cdot L_{seq}/N_{GPU}.$ (54) $$

For an MLP activation layer, the size of activation values loaded during the prefill phase is:

$$ $M_{A,load,act}^{pre}=2|B|\cdot size_{h}\cdot D_{A}\cdot L_{seq}/N_{GPU},$ (55) $$

and the size of activation values stored is:

$$ $M_{A,store,act}^{pre}=|B|\cdot size_{h}\cdot D_{A}\cdot L_{seq}/N_{GPU}.$ (56) $$

Thus, for each normalization or residual add layer, the total memory access size during the prefill phase is:

$$ $M_{norm/add}^{pre}=2M_{A,load/store,norm/add}^{pre}.$ (57) $$

For each MLP activation layer, the total memory access size is:

$$ $M_{act}^{pre}=M_{A,load,act}^{pre}+M_{A,store,act}^{pre}.$ (58) $$

## Peer Reviews

confidence: 1
contribution: 2
initial_rating: 6
presentation: 3
questions: -  Could you provide more details about the exact training setup of the GNN, specifically: what is the target label during training, how is it measured and at what granularity, and how do you handle the separation of prefill and decode phases in your ground truth measurements?
soundness: 3
strengths: - Addresses the critical issue of carbon footprint estimation for LLM inferences
- This paper separate modeling of prefill/decode phases and provides more accurate estimate
- Use GNN to predict the carbon footprint and shows promising accuracy.
summary: This paper introduces LLMCO$_2$, a GNN-based pipeline to estimate the carbon footprint of LLM inference.
LLMCO$_2$, separate LLM's prefill and decode stage during inference, and uses focused sampling targeting common inference configurations. The system demonstrates significant improvement in prediction accuracy compared to existing methods across various LLM architectures and GPU configurations.
weaknesses: - This is good work. However,  this paper is more suitable for an HPC-related conference; I didn't see much relation to the submitted track of `alignment, fairness, safety, privacy, and societal considerations'. 
- While the paper presents several innovations, it looks like combining existing methods such as GNNs, the Roofline model, and active learning. It may lack of novelty.

---

confidence: 4
contribution: 1
initial_rating: 1
presentation: 2
questions: Q1: Given the discrepancies in the cited data (as outlined in W1, W2, and W3), how do you plan to reassess and strengthen the motivation for your research?

Q2: Regarding the issues raised in W4, W6, W7, and W8, could you elaborate on your decision-making process for including or excluding certain factors in your model? What were the trade-offs you considered, and how do you plan to address these concerns in future iterations of your work?

Q3: Why were real-world deployment types outlined in W5 not included? The results from your model are not representative if even one of the techniques is used, potentially making the model void outside of the exact specifications under which it was developed in this paper. My current understanding points me towards a mathematical model based on empirical measurements rather than a regression-based predictor for energy consumption. Even if the mathematical model might not be perfectly accurate, I am fairly confident in estimating a tight lower and upper bound of energy consumption with any of these techniques used.
soundness: 2
strengths: S1: The authors found a research gap that the previous SOTA did not address.

S2: This is an important topic to work on, and I am glad it is being tackled from a solid basis in empirical research (performing a roofline analysis, considering multiple models, etc.).

S3: An interesting approach to use the computation graph as input to a GNN. I feel there is a lot of value in this following this further in this line of research.
summary: This paper presents LLMCO2, a graph neural network that provides the inference energy consumption cost of LLMs based on the computation graph and deployed hardware of said LLM. They account for the prefill- and decode-phase of the LLM, utilize hardware characteristics, and keep tensor parallelism in mind, which limits the mean absolute error percentage compared to previous state-of-the-art to ~ 20% w.r.t. ground truth.
weaknesses: W1: I seriously doubt the credibility of the papers cited in this line of research, like Faiz et al., which is an important building block in the argumentation of this paper (see first and second paragraph). They cite Strubell et al. 2019, which was famously (or at least I thought) debunked by Patterson et al. (https://arxiv.org/pdf/2104.10350) as providing an 88x higher estimate of the energy consumption compared to what was really used, highlighting how hard it is to perform this research in practice. Patterson et al. was also not cited, which, from my understanding, is one of the few papers of people with actual end-to-end access to all production metrics, making this more trustworthy than most other papers that estimate costs. I take specific issue with general statements "(...) with a single epoch requiring three times the FLOPs of an inference", "(...) the period required for inference emissions to match training emissions is rapidly decreasing." and "For instance, training the Google T5 LLM generates 40% more carbon emissions than a round-trip flight between San Francisco and New York (Faiz et al., 2024)." While the first two statements would have to have a large caveat of the specific use-case attached to it and potentially not being representative at the current time, the last statement is, to my understanding, entirely wrong. Patterson et al. states that T5 training took 46.7 tCO2e (Table 4), which is 26% of a roundtrip between NYC and SF (180 tCO2e). The authors argue that this took 140%, an error by a factor of 140/26 ~= 5x.

W2: I question the addition of empirical data from older hardware towards the training dataset of LLMCO2. Inference for frontier models like OpenAI's (which was used in the motivation) likely does not happen on older GPU architectures due to their missing support of sparsity and reduced FLOP and memory bandwidth per $ performance, making that analysis void. When arguing that inference is costly at scale, one needs to analyze the actual costs of said scale and not use older hardware for that comparison.

W3: The authors cite the Azure LLM trace as a basis for their decision to use a <2 batch size and argue that this is representative. The same paper (Patel et al. '24) also says this only counts for the prefill (Sec 3.D) but not for decoding, which scales for that particular model-hardware setup until a batch size of 64 (only limited by memory in that specific instance, making the results even more questionable as this is dependent on the context lengths of the requests due to them affecting the KV-cache). This is not addressed by LLMCO2, making its prediction very likely not applicable to other providers and use cases. Additionally, Patel et al. argue that the largest latency (hence, time and energy consumption at ~50% energy consumption) comes from the decoding phase, so they split these two phases to run on different hardware setups. However, the authors do not consider this use case (at least to my understanding) when generating the dataset to train LLMCO2, discounting their own motivation of "Disregard for prevalent configurations", while Patel et al. was used to argue this exact motivation.

W4: The idle time of the nodes was not considered, which is also likely to be a large chunk of energy spent if services are not utilized to capacity. Arguably, this can be omitted by assuming that on-demand up-/downscaling of hardware works efficiently. However, this was never stated. Generally, the assumptions of when LLMCO2 will likely provide accurate results and when it does not are not stated concretely enough.

W5: While I appreciate the empirical-based approach of this paper, it is lacking on multiple fronts. No SOTA deployment methods are discussed (while citing Patel et al., which proposes one instance when splitting prefill and decoding phase on different hardware generations), and continuous batching was barely addressed in how it affects energy consumption. No sparsity, no speculative decoding or streaming, no TensorRT, no NVIDIA Triton, no vLLM, no torch.compile, and a very limited definition of workloads. Just as an example, if speculative decoding improves the decoding phase by 2x, its carbon emissions will be approximately halved (due to the smaller model being multiple magnitudes faster and cheaper to run, making it a negligible cost). This is a very common technique and is likely to be used by most inference providers, which is not addressed at all.

W6: Peak FLOP is wrong for the H100 in Table 2 (see the H100 datasheet https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet). FP32 is stated as 989 TFLOP, while it is 67 TFLOP (presumably, the authors mistyped TF32, which would also be wrong as this is the performance with sparsity). FP16 performance is stated as 1979 TFLOP, but this is 989 TFLOP, as this is the performance with sparsity enabled. Same thing for INT8. Same issue for all precisions for the A100. This makes me question the other results from this paper if such an important consideration was overlooked.

W7: The second paragraph states, "(...) the period required for inference emissions to match training emissions is rapidly decreasing." While I agree that this can be the case, the way this is argued is not conclusive. We see a trend that compute utilization is doubling every 6 months (https://epochai.org/trends#compute). To argue the point of inference costs becoming closer to the training costs, the same analysis would need to happen for inference (presumably with AI usage numbers, which are likely to be kept private at large).  However, due to the issues from W5, the trace used in this paper would not be representative of the real world, making it probably impossible to argue outside of specific deployments.

W8: Eq. 1, "energy_per_operation x PUE x carb_intensity." I doubt that this is the correct way to estimate energy consumption. This excludes the energy for the nodes itself, including networking, PC internals, and cooling. A DGX H100 node uses roughly 10.2kW, while 8xH100 use 8x700W=5.6kW, making your resulting estimate off by a factor of 2x (https://docs.nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html)

W9: What were the exact dataset splits? The dataset and evaluation sections seem to suggest that all evaluated models were part of the training dataset, making me afraid of an overfit happening. 

W10 (Summary and final notes): While the basic premise of the paper is interesting to use a computational-graph analysis and using a GNN to process it, the application of this model would lead to the following problems:
- A misguided understanding of how energy costs come about from LLM inference due to very limited real-world application scenarios
- Potentially misleading results about being energy-efficient in theory or inference being much more grave w.r.t. energy-consumption than training. It is important to note that I am not averse to thinking this might be the case, but how it is stated here is definitely wrong. I fear this work being misunderstood and misused, similar to Strubell et al. (and how prior cited work like Faiz et al. was used similarly by the authors).

Minor issues:
- Figures 9, 10, and 11 are squished and are hardly readable.
- Figure 10 was referenced when the authors wanted to reference Figure 11 in " Figure 10 shows the operational carbon footprint of Bloom-7b1 (...)".
- Figure 11: It is hard to understand what training is and what is inference due to the legend's labels.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: The paper’s focused data sampling strategy is likely to cause biases in the model’s predictions. The sampling method omits rarely encountered configurations, which might negatively affect the model's robustness when faced with less common but real-world scenarios, potentially leading to inaccuracies in edge cases.
soundness: 2
strengths: 1. The organization of this paper is good.
2. It can improve the accuracy of LLM inference carbon footprint predictions compared to previous methods.
summary: The paper proposes a graph neural network-based model, called LLMCO2, which aims to improve the accuracy of carbon footprint prediction in the LLM inference process.
weaknesses: 1. The paper proposes a GNN-based model to predict the carbon footprint of LLM inferences. The integration of graph embedding, data sampling, and the Roofline model does not introduce fundamentally new concepts but rather repurposes established techniques in a novel application context. This approach lacks substantial innovation. The primary contribution is an improvement in prediction accuracy over previous models. However, these improvements are incremental, and the research lacks a clear breakthrough in model efficiency or in introducing a new predictive paradigm.

2. While the model aims to predict the carbon footprint, it does not adequately address the broader trade-offs in energy efficiency versus accuracy. The model's usage of multiple GPUs and its impact on real-world energy savings are not thoroughly discussed.