Kati8/ICLR_2023_2025 huggingface dataset:
authors: Ahmad Faiz
authors: Sotaro Kaneda
authors: Ruhan Wang
authors: Rita Chukwunyere Osi
authors: Prateek Sharma
authors: Fan Chen
authors: Lei Jiang
reviews: 
Comments: The work builds an analytic model that predicts carbon equivalent emissions of LLM training and deployment. The model is extensive and contains variables and functions such as scaling laws, parallelism used, mixture of experts, data center efficiencies, and embodied carbon footprint.

Recommendation: While the model could further be extended to include more detailed variables, it is a groundbreaking effort that will lay the foundation for all future carbon models. I recommend this work to be highlighted at the conference.
Strengths: - a very comprehensive model that captures previously neglected variables that led to widely inaccurate carbon emission predictions
- uses scaling laws and other variables to make accurate predictions of the carbon emissions of any training run
- validation of the model with reported carbon footprints in the literature.
Weaknesses: - the mixture of expert models seems to use regular scaling laws equations for performance prediction. More appropriate would be to use the equations from the routed scaling laws paper.
- some factors are not very well discussed. For example, experimentation can have a large variance of CO2eq used. For deployment, more experimentation is usually undertaken to improve model efficiency through speculative decoding and distillation. A short discussion on the most CO2eq intensive factors and how they might differ between companies/institutions would be appropriate (no need to model this)
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: The main contribution of this paper is to present a comprehensive model for end-to-end carbon footprint of LLMs. Observing that LLMs can be a major contributed to overall carbon footprint, the authors model its carbon footprint from both operational and embodied perspectives, and presents detailed results for different types of architectures.
Strengths: Modeling carbon footprint is an interesting (and increasingly important) research direction. This submission is probably one of the early papers on the topic.
Weaknesses: Not clear whether the authors' definition of embodied carbon is accurate.
Not clear where the carbon coefficient factors in in the model formulation.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 3 good
reviews: 
Comments: The paper provides a tool for estimating the carbon footprint of large language models (LLMs). The tool accounts for each life-cycle phase of an LLM: embodied carbon, training, inference, storage. The energy used by hardware is accounted for by considering data center characteristics and model architecture attributes. Comparison with published carbon numbers shows less than 10% error in estimation.
Strengths: - Problem statement is clear
- Carbon footprint considers a comprehensive view of LLM life-cycle
- Use of real-world data points from literature to perform carbon footprints
- Validation against published carbon footprint numbers
- Low error rates in estimation
Weaknesses: - It is not clear how the "ground truth" carbon footprint numbers were established by prior works. Did they perform actual measurements? This is a concern because if the numbers are estimated, and there is an overlap in estimation methods, then it's not surprising that the error rate is low.
- It is difficult to separate the contributions of the paper from the prior works. Much of the equations and parameters of the equations are based on prior work. Is the contribution of the paper to sum up carbon numbers from prior work equations? 
- There is no measure of utility of an LLM. For example, one can reduce the emissions by increasing test loss, but a high test loss can lead to poor performance which cannot be used in the real-world. It'll be interesting to measure the utility across classical ML models such as those used for classification, regression, translation, etc. 
- It would be good to discuss the utility of the measurement tool. Is the expectation that model designers pick an appropriate data center, architecture, or training dataset to use based on carbon footprint? Much of the time such decisions are constrained by other requirements. For example, model designers have little control over the embodied footprint of the ML model.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 2 fair
reviews: 
Comments: In this paper, the authors proposed an end-to-end carbon footprint predictor for llm services. Specificall, the predicted carbon footprint is the sum of two parts: operational carbon footprint and embodied carbon footprint. The former one is calculated with a model by taking model parameters, hardware efficiency as input; The latter one is calcuated with another model by taking hardware type, chip area, system power as input. Through extensive comparison experiments, the predicted results shows a better performance with at most 8.2% error.
Strengths: - Compared with baselines, the proposed llmcarbon generalize the carbon prediction to various network architectures (dense llm and moe llm), various hardwares (gpu and tpu) and various phase (training, inference, experimentaiton and storage).

- The key submodels in llmcarbon are elaborated with mathematic formulation and detail description. All the architecture is easy to follow.
Weaknesses: - Some parameter is crucial to the final predicted results, the process of setting parameters should be clarified. For example, in equation 3, \alpha, \beta are the fitting parameters. However, the fitting dataset and fitting method are missing.

- Apart form the parameters discussed in the paper, more factors also affect the footprint. e.g., the paramter precision (fp16, int8, int4) and the implementation of kernel operation.

- There are not any text description about figure 3 and figure 4.

- The proposed llmcarbon is somewhat like a system design with many prior experience from sota, the algorithm contribution are not notable.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper introduces LLMCarbon, a comprehensive cost model developed to estimate the carbon footprint associated with various stages of LLM computation, such as training, inference, experimentation, and storage. LLMCarbon extends its applicability beyond previously established models by incorporating support for a broader range of LLM architectures, with a specific focus on the mixture-of-experts architecture and more types of hardware. The paper provides an in-depth discussion of LLMCarbon’s cost model, covering aspects like parameter count, FLOPs, and hardware efficiency. To validate LLMCarbon’s efficacy, we present experimental results across different LLMs and hardware configurations.
Strengths: This paper is both well-written and well-motivated, addressing the critical issue of environmental sustainability by studying the carbon footprint of LLM computation. The authors have proposed a cost model that appears to be both fine-grained and carefully designed, reflecting a thorough approach to this significant topic.
Weaknesses: - the procedure of how LLMCarborn figures out the optimal parallelism is not quite clear.
- It is not clear how to use LLMCarborn to guide the design of future generations of LLM architectures and figure AI accelerators.
- The relationships and connections between the proposed hardware efficiency and metrics like arithmetic intensity and MFU are not clear.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good