Energy and Carbon Considerations of Fine-Tuning BERT
Authors: Xiaorong Wang
Clara Na
Emma Strubell
Sorelle Friedler
Sasha Luccioni
Venue: EMNLP 2023 Findings (2023)
Keywords: energy costs
fine-tuning
efficiency evaluation
efficiency
BERT
transformer
TL;DR: We study energy costs of fine-tuning language models on various tasks and provide recommendations to researchers and practitioners who wish to predict or reduce these costs.

Abstract: Despite the popularity of the pre-train then fine-tune paradigm in the NLP community, existing work quantifying energy costs and associated carbon emissions has largely focused on language model pre-training. Although a single pre-training run draws substantially more energy than fine-tuning, fine-tuning is performed more frequently by many more individual actors, and thus must be accounted for when considering the energy and carbon footprint of NLP. In order to better characterize the role of fine-tuning in the landscape of energy and carbon emissions in NLP, we perform a careful empirical study of the computational costs of fine-tuning across tasks, datasets, hardware infrastructure and measurement modalities. Our experimental results allow us to place fine-tuning energy and carbon costs into perspective with respect to pre-training and inference, and outline recommendations to NLP researchers and practitioners who wish to improve their fine-tuning energy efficiency.

Energy and Carbon Considerations of Fine-Tuning BERT
Xiaorong Wang
Clara Na csnaêcs. cmu. edu
swangxr@gmail.cmm com
Emma Strubell strubell@cmu.en edu
Sorelle A. Friedler
Sasha Luccioni sasha luccioni@hf co
sorelle@cs haverford.cem edu
'Haverford College, "Carnegie Mellon University, 3 Allen Institute for AI, 'Hugging Face
Abstract
tive contributions due to different aspects of model development and use (Kaack et al., 2022).
Despite the popularity of the pre-train then fine- tune paradigm in the NLP community, existing work quantifying energy costs and associated carbon emissions has largely focused on lan- guage model pre-training. Although single pre-training run draws substantially more en- ergy than fine-tuning, fine-tuning is performed frequently by many tors, and thus must be accounted for when con- sidering the energy and carbon footprint of NI.P. In order to better characterize the role of fine- tuning in the landscape of energy and carbon emissions in NLP, we perform careful empir- ical study of the computational costs of fine- tuning across tasks, datasets, hardware infras- tructure and measurement modalities. Our perimental results allow us to place fine-tuning energy and carbon costs into perspective with respect to pre-training and inference, and out- line recommendations to NLP rescarchers and practitioners who wish to improve their fine- tuning energy efficiency.
In this work perform empirical study to quantify the energy requirements of language model fine-tuning, including in the context of pre- training and inference energy requirements. While this may seem like it should be straightforward calculation, there are several variables that can in- fluence compute time and energy consumption, ranging from: (1) the type of hardware used for both pre-training and fine-tuning (since this usually differs between the two), (2) the type of task and the type of computation required carry it out, and (3) intrinsic characteristies of the dataset, such as average sequence length, its similarity with the pre- trainining dataset, etc.
individual
In order isolate the factors that have the most influence on fine-tuning dynamics, we compare fine-tuning energy use across a suite of common supervised NLP datasets including the tasks of en- tailment, sentiment analysis, question answering, and named entity recognition, and with training data sizes ranging from 6K to 400K examples. We also measure energy across two different sets of hardware, using the CodeCarbon (Schmidt al., 2021) software package and physical energy mea- surement device at the wall, to quantify variance due to physical factors. To enable carefully con- trolled comparison of the roles pre-training and fine-tuning in NLP model lifecycle energy use, we additionally pre-train BERT variants from scratch on the same hardware. We find that pre-training BERT is equivalent to anywhere from 400 (MNLI) to 45,00 45,000 (RTE) fine-tuning runs depending the dataset size, and that number of training tokens a reasonable heuristic for estimating fine-tuning energy use. Further comparison of fine-tuning in-
1 Introduction
Fine-tuning pre-trained language models is fre- quent occurrence in natural language processing (NLP) research and practice, yet the vast major- ity of work quantifying the energy and carbon footprints of NLP workloads has focused on pre- training (Strubell al., 2019; Dodge et al., 2022; Luccioni et al., 2023) or inference (Desislavov et al., 2023; Luccioni et al., 2023). The typical lifecycle of an NLP model includes data inges- tion, pre-training, fine-tuning and inference, all of which contribute non-trivially to energy use and corresponding emisssoon (Patterson et al., 2021; Wu et al., 2022). Better understanding of the role that each phase plays in overall energy and car- bon footprint vital to inform policy decisions, yet we still lack basic data quantifying the rela-
'The "true" number of training tokens seen, accounting for dynamic padding of sequences to the maximum length in hatch, is a better predictor than relying on to mean or median number of tokens per example.
"Denotes equal contribution.

---

ference energy intensity across tasks confirms that example sequence length holds much stronger in- fluence on energy intensity in the fine-tuning phase than in the inference phase, in alignment with ex- pectations from previous work (Zhou et al., 2021). Together, our observations contextualize the en- ergy and carbon requirements of fine-tuning in the broader model lifecycle and highlight the need to study fine-tuning energy efficiency separately from pre-training and inference workloads in NLP mod- els. We hope that our careful measurement of the relative costs of different NLP workloads will serve as a valuable datapoint informing decision -making both within and beyond the NLP community.
quirements in the bigger picture of ML model life- cycle emissions.
Another related direction of research examines the dynamics of pre- me-ttrining and fine-tuning of language models and the influence of factors like random seeds and early stopping (Dodge et al., 2020), scaling (Tay et al., 2022) and learning dy- namics (Hao et al., 2020). While all of these stud- ies have shed important light these processes, in practice most of the decisions made remain empiri- cal, with practitioners either referring to previous work (when hyperparameters and other training de- tails are reported), or using techniques such as grid or random search (Bergstra and Bengio, 2012) to converge on optimal parameter values. Our own work builds upon both of these research directions. We study both the pre-training and fine-tuning pro- cess, and our experiments for studying their energy intensity are based on the works cited above.
2 Related Work
Measurement of energy consumption and car- bon emissions of NLP models has become an active area of research since was first identi- fied that modern state-of-the-art models based on deep learning can produce substantial greenhouse gas emissions due to the energy required to train them (Strubell et al., 2019; Schwartz et al., 2020). These measurements have mostly focused on two research directions. First, there has been a series of empirical studies on different model architec- tures, focused on estimating the carbon emissions generated by their training process and the rela- tive efficiency of different methods (Naidu et al., 2021; Patterson et al., 2021, 2022). Recent work by Luccioni et al. has built upon this, aiming to en- compass the embodied emissions of manufacturing computing hardware well those produced via the inference process (2023). There has also been complementary work measuring the energy used by Transformer models during inference and ways predicting those costs for different architectures and models (Cao et al., 2021; Ang et al., 2022).
Methodology
Full training details can be found in Appendix A.I. We release code for replicating our measurements. ? We encourage others to run our code on their hard- ware to add to repository of measurements across different hardware platforms.
3.1 Pre-training BERT
In this work we are interested in measuring the energy consumption of fine-tuning, compares to other stages of model use: pre-training and in- ference. In order to establish a comparable base- line for the energy consumption and carbon emis- sions of pre-training, we pre-trained a BERT-base model (Devlin et al., 2019) from scratch on the same hardware that we use for fine-tuning (Sec- tion 3.3). Although in practice pre-training and fine-tuning are often done separately and differ- ent hardware, we fixed the machine for both sets of experiments in order aid direct comparability of energy usage measurements. Following Devlin et al. (2019), we pre-train our model on the Book- Corpus (Zhu et al., 2015) and the 2020 version of Wikipedia (Foundation, 2020), both downloaded from HuggingFace Datasets (Lhoest et al., 2021). Our precise pre-training methodology differs slightly from Devlin et al. (2019): our data nec- essarily differs slightly because the original train- ing corpus was not released along with the model, and we only use the masked language modeling
Closest our work, the HULK bench- mark (Zhou et al., 2021) was proposed measure the relative efficiency-accuracy trade-off's of differ- ent pre-trained models, measuring the wall-clock time for different pre-trained models to reach a target accuracy on one of three fine-tuning tne-tuning tasks. Different from Zhou et al. (2021), our work explic- itly focuses on the energy and carbon required for fine-tuning (theirs uses time and financial cost as proxies), evaluates a wider variety of fine-tuning tasks and hardware settings in order to elucidate the factors that predict fine-tumin tinettunng energy requirements, and further contextualizes fine-tuning energy re-
Phttps: //gi thub. http///annxr/FT-eeeeeee

---

(MLM) objective without next sentence prediction (NSP) following Liu al. (2019), who found that removing NSP did not substantially impact end- task performance.
machine had four GPUs. Pre-training experiments used all GPUs in each machine.
All fine-tuning tasks were performed on the same machines, but using only one GPU. This re- flects the typical scenario where fine-tuning is done on a single GPU even if the machine itself has more GPUs. To better compare the energy usage results across pre-training and fine-tuning, we also report energy usage estimate for BERT-base pre- training on 1 RTX 8000 GPU with hyperparameters equivalent to training on 4 GPUs, extrapolated from a 12- 12-hour partial training run. Details are recorded in Appendix A.I.
In order to assess the relative impact of using a more efficient pre-trained BERT variant, we also followed the DistilBERT (Sanh et al., 2019) recipe, performing knowledge distillation our pre-trained BERT-base model.
3.2 Fine-tuning BERT
We evaluate the energy consumption and carbon emissions of the fine-tuning process on the tasks in Table 1. We deliberately chose this selection of end-tasks in order to vary fine-tuning dataset size, task type, and sequence length, while also aligning with tasks commonly used in NLP applications.
3.4 Measuring Energy and Carbon
To measure the energy consumed, we use the soft- ware tool CodeCarbon (Schmidt et al., 2021). Re- cent work has found that the existing libraries and code for estimating the carbon emissions of NLP techniques vary in terms of their accuracy and gen- cralizability to different types of hardware (Ban- nour al., 2021). To compensate for this, we calibrate the programmatic energy usage readings with a physical energy meter, with which we record energy readings directly from the compute node during experiments. Subsequently, we calculate a coefficient of expected power loss, 1.059, as the average proportion (over runs across fine-tuning tasks) of physical energy reading vs. programmatic energy measurement. Full results are given in Ap- pendix A.2. Thus, the energy consumed in kWh, denoted as E, is determined via the formula:
Soq. length
Dataset
Task
Examples
Med.
Batch
Wiki +Books
MLM
43M
128
128
RTE
NLI
6K
56
128
MNLI
NLI
433K
37
90
SQuADv1
QA
98K
159
336
SQUADy2
QA
142K
158
336
IMDB
Sent.
50K
128
128
SST2
Sent.
70K
10
40
CoNLL2003
NER
21K
15
53
CoNL
NER
143K
20
65
Table 1: Pre-training and fine-tuning dataset descrip- tions. We report two sequence length statistics: Median tokens per sequence in the dataset, and Batch, the aver- age maximum sequence length per batch as predicted by simulated sampling. The latter included as a more di- rect predictor of computation cost for dynamic padding.
E (kWh) 1.059 codecarbon kWh
Converting the power loss adjusted values to CO2 emissions is done through CodeCarbon us- ing coefficient specific to the energy grid based on the location of the server from the most recent EPA Power Profiler Data. The conversion factor for the server's location (Pittsburgh, PA) in Ta- ble is 1046.1 lbs/MWh, while the factor for the second server's location (Haverford, PA) 672.8 lbs/MWh. The total kilograms of CO2 emitted, denoted as C, is then determined via:
All models are fine-tuned on the BERT models described in $3.1. For each fine-tuning task, use typical fine-tuning hyperparameters specific to the task or user-reported hyperparameters on cur- rent fine-tuned models on HuggingFace, in order to mimic the common real-world use cases. For each task, we dynamically pad sequences to the maximum length in each batch. All fine-tuning hyperparameters are reported in Appendix A.
We also report average per-example energy use for inference. All the inference tasks are performed on 1 GPU with batch size on the same machines.
lb 1MWh
1kg
C E x 1046.1
X
X 2.20462lb
MWh 1000kWh We convert the CO2 emissions result human understandable reference values using the EPA Greenhouse Gas Equivalencies Calculator; in Ta- ble 2, we also show the equivalent CO2 emissions of miles driven by an average aasoline-pwwers passenger vehicle.
3.3 Hardware Platforms
To ensure reproducibility and measure variability across hardware platforms, we replicate experi- ments across two hardware platforms: One 1.00 machine and one RTX 8000 machine, where each

---

Training / Fine-tuning
Inference
Task
Dataset
Energy (kWh)
Emissions (kg CO2))
Time
Equiv. # PT
Equiv. miles
Energy
Equiv.
(kWh)
/ FT
runs
1000
runs
4 GPU (1m @ len 128)
270.9
128.6
27 hr
1.358
330
1 GPU (Im @ len 128)
419.6
199.1
673 hr
0.646
510
MLM
4 GPU (100k @ len 512)
124.1
58.90
114 hr
2.965
151
Total GPU
368.0
174.6
357 hr
1.000
408
NLI
RTE
0.008
0.004
s
45109
0.01
0.794c-3
MNLI
0.938
0.445
6700 8
392
1.14
7.349c-3
127k
SQuAD vl
0.537
0.255
3780 8
685
0.65
2.157c-3
249k
QA
SQuAD v2
0.795
0.377
5604 S
463
0.97
2.471e-3
322k
IMDB
0,151
0.072
1074
2441
0.18
0.849e-3
178k
Sent.
SST2
0.081
0.038
587 S
4540
0.10
0.691e-3
12k
CoNLL2003
0.021
0.010
149
17886
0.03
0.808e-3
3k
NER
CoNLL2012
0.207
0.098
1487 s
1778
0.25
0.878c-3
24k
Table 2: Energy consumption of pre-training BERT and fine-tuning the RTX8000 GPU machine. Energy is computed as raw energy measured by CodeCarbon multiplied by a coefficient to correct for power loss (Eq, 3,4). Equiv. miles refers the approximate number of vehicle miles driven resulting equivalent CO2 emissions. GPU pre-training costs are extrapolated from a shorter pre-training run lasting only few hours. Total cost of fine-tuning is derived from 900k steps pre-tre vee-training on sequences length 128 followed by 100k steps on sequences length 512, following (Devlin al., 2019). Energy consumption for inference is calculated using single-example batches.
4 Results and Discussion
runs of an MNLI-like task, to 47k fine-tunings on an RTE-like task. DistilBERT fine-tuning results on the A100 machine are in Appendix A.3.
Table 2 shows energy, carbon, and wall-clock time required to fine-tune BERT-base models the RTX8000 machine. Results the A100 machine are recorded in Appendix A.3 Tables 8 and
4.2
Comparing and Predicting Fine-tuning Emissions
Task
Dataset
Energy (kWh)
Time
Emiss. (kg COs)
We find that, controlling for hardware, energy con- sumption scales most predictably with wall clock time and mumber of tokens encountered during training (including the pad tokens added to se- quences to match the maximum sequence length in a batch). The linear relationship between en- ergy consumption and total number of tokens holds similarly on both machines (see Figure 1).Addi- tionally, we observe consistently higher energy consumption the RTX 8000 GPU machine. This is likely due to the higher energy overhead and the (in)efficiency of the hardware compared to the A 100 GPUs. Other figures in Appendix A.3 il- lustrate that, in contrast, energy requirements function of optimization steps of examples in the dataset can vary significantly across datasets and tasks.
Distil.
Books
187.74
175.5 hr
89.08
RTE
0.004
30 s
0.002
NLI
MNLI
0.481
3147 s
0.228
SQuAD v1
0,276
1954 s
0.131
QA
SQuAD v2
0.412
2916 s
0.196
IMDB
0.077
549
0.037
Sent.
SST
0.042
306
0.020
CONIL20003
0.011
78
0.005
NER
CoNLL2012
0.107
762
0.051
Table 3: Energy consumption of training (distillation) and fine-tuning DistilBERT on the RTX8000 machine. Energy calculations are the same as in Table 2. Distil- lation is performed on pre-trained model, and the true "total" cost includes the pre-traing pre-training cost as well.
number
4.1 Pre-training and Distillation
We observe that it requires an additional 50% of the energy cost of pre-training in order to perform knowledge distillation, but takes nearly 50% less energy fine-tune on the same tasks using Dis- tilBERT vs. normal BERT (see Table 3). By our estimate, one can fully amortize the ur-front cost distillation within anywhere from 86 fine-tuning
Note that cheaper inference is often the primary goal of knowledge distillation. Inference much cheaper than training and therefore requires more to amortize the initial cost of distillation, but inference also occurs much more frequently than training. Models running inference at scale are typically highly optimized with respect to specific deployment settings, so our estimates approximate a lower hound.

---

Energy consumed Vs tokens seen
the model.
Energy consumed (kWh) in log scale
squaze
deseet
On the other hand, there are many types of fine- tuning tasks where examples consist of sequences of varying lengths significantly shorter than the maximum length that the model has been trained on, as shown in Table 1. Since, effective sequence lengths are determined dynamically training time (where sequences are padded to the maximum length in each given batch), total training time is not as simple to extrapolate from standard measures of dataset size as in pre-training.
10a
10A
rit
con 1202
10a
tran speni
poe 2t spo
dilizer:
madens
10-7
A192 NENT
A0D Doiltee
10
m753333 sear
10c
10f
1044
Number of tokens in log scale
4.4 Fine-tuning Inference
Figure I: Total energy consumed (kWh) is strongly re- lated to number of tokens seen for BERT models on both A100 and RTX 8000 GPU machines, although the rela- tionship is more predictive on the RTX 8000 machine and energy usage is less consistent with DEST Note that both axes are in log scale. An alternative view of similar data in Figure 4 distinguishes pre-training workloads' energy consumption slightly from that of fine-tuning tasks.
Although we do observe that per-example inference efficiency costs are related to sequence lengths, there is overall less variation across datasets and tasks in inference costs compared to fine-tuning costs (see Table 4). This mirrors observation noted in the HULK benchmark (Zhou et al., 2021), though the best of our knowledge ours is the first to explicitly draw comparisons across task types and different aspects of dataset size (i.e. number of examples and examples' sequence lengths).
Seq.
length
kWh / Ik ex.
Dataset
Med.
Batch
Inf.
FT
RTE (NLI)
56
128
0.75e-3
1.09e-3
4.5 Single vs. Multiple GPUs
MNLI (NLI)
37
90
0.70e-3
0.80e-3
SQuADv1 (QA)
159
336
1.08e-3
3.04e-3
In general, typical hardware and data settings for pre-training and fine-tuning tend to differ. Though to the best of our knowledge it is less common to fine-tune causal LMs of this scale on multiple GPUs, we present additional results from multi- GPU fine-tuning on the x RTX8000 machine with the same fine-tuning tasks in Table 10 in Ap- pendix A.3. Our recommendations from an energy efficiency standpoint align with common rules thumb for effective utilization of hardware; if the resources would be idle otherwise, one could rea- sonably consider increasing batch size and learning rate to saturate the available hardware for both time- and energy-efficient training.
SQuAD,2 (OA)
158
336
1,24e-3
3.02e-3
IMDB (Sent.)
128
128
0.80c-3
1.21e-3
SST2 (Sent.)
10
40
0.65e-3
0.40e-3
CoNLL2003 (NER)
15
53
0.76e-3
0.49e-3
CoNLL2012 (NER)
20
65
0.83e-3
0.60e-3
Table Inference vs. fine-tuning energy requirements across end tasks. We see that fine-tuning energy usage varies according to sequence length much more widely than inference energy usage.
4.3 Fine-tuning Pre-training
Even for the more reliable predictors of energy consumption and carbon emissions (duration of training and number of tokens processed), the en- ergy cost profiles of pre-training vs. fine-tuning are different, likely due to differences in training infras- tructure, training objectives, and sequence lengths typically seen in pre-training vs. fine-tuning (see Figures 4, 3, 5, and 6). Pre-training in general is almost always performed over multiple GPUs which incurs energy costs from communication between GPUs, and often also with gradient accu- mulation to accommodate large batches. Moreover, sequences are packed together such that batches consist largely or entirely of sequences of identical length equal to the maximum sequence length for
Conclusion
We share a procedure for rigorous measurement of energy consumption from causal LM fine-tuning given multiple concrete hardware settings. We hope work is useful to researchers and prac- titioners who are interested in obtaining measure- ments for their own specific hardware, gaining intu- itions about factors affecting relative energy costs of different types of fine- tuning tasks, or under- standing these energy costs in context of the model lifecycle.

---

Limitations
DGE2140739, Any opinions, findings, and con- clusions or recommendations expressed in this ma- terial are those of the authors and do not necessarily reflect the views the sponsors.
While our work provides important first steps towards a clearer understanding of model fine- tuning's impact the environment, we note that our experimentation is limited to various token classification, sequence classification, and question answering tasks with BERT and DistilBERT- base models. We do not make claims or extrap- olations about much larger language models, or models with different architectures, as well as other types of tasks such as summarization. Future work in this direction can expand the number of tasks that we consider as well feature different archi- tectures such as RoBERTa (Liu et al., 2019).
References
Phyllis Ang. Bhuwan Dhingra, and Lisa Wu Wills. 2022. Characterizing the efficiency vs. accuracy trade-off for long-context NLP models. In Proceedings of NLP Power! The First Workshop on Efficient Bench- marking NLP, pages 113-121, Dublin, Ireland. Association for Computational Linguistics.
Nesrine Bannour, Sahar Ghannay, Aurelie Névéol, and Anne-Laure Ligozat. 2021. Evaluating the carbon footprint of NLP methods: a survey and analysis of existing tools. In EMNLP, Workshop SustaiNLP.
Additionally, the on-premises hardware infras- tructure used for our experimentation is realistic and typical of compute resources in academic set- tings, but we provide no firsthand evidence of fine- tuning emissions profiles expected from either local model training (where the impracticality of pre- training makes direct comparisons with fine-tuning emissions difficult) or fine-tuning on hardware that is part of much larger scale infrastructure such as on a public cloud. Furthermore, we expect that use of specialized hardware such as TPUs (as opposed to GPUs, which we use) would be associated with different emissions profiles.
James Bergstra and Yoshua Bengio. 2012. Random search for hyper- parameter optimization. Journal of Machine Learning Research, 13(10):281-305
Qingging Cao. Yash Kumar Lal, Harsh Trivedi, Aruna Balasubramanian, and Niranjan Balasubramanian. 2021. IrEne: Interpretable energy prediction for transformers. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 2145-2157, Online, Association for Computational Linguisties.
Radosvet Desislavov, Fernando Martinez-Plumed, and José Hermández-Orallo, 2023. Trends in ai inference energy consumption: Beyond the performance-ve parameter laws of deep learning. Sustainable Com- puting: Informatics and Systems, 18:100.
Ethics Statement
Training and deploying ML models and systems requires large amounts of energy, the production of which results in the emission of greenhouse gases. While the goal of our research is contribute to- wards better understanding of the factors that in- fluence these emissions, by carrying out our experi- ments, we were ourselves responsible for the emis- sion of 350 kg of carbon equivalents. We release our code and the data generated by our experiments to maximize the transparency and reproducibility of our work.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi. and Noah Smith. 2020. Fine-tuning pretrained language models: Weight ini- tializations, data orders, and early stopping. arXiv preprint arXiv:2002.06330
Acknowledgements
We are grateful to Victor Sanh for his time and patience in answering questions about BERT and DistilBERT pre-training. We would also like to thank our anonymous reviewers for taking the time provide helpful feedback.
Jesse Dodge, Taylor Prewitt, Remi Tachet des Combes, Erika Odmark, Roy Schwartz, Emma Strubell, Alexandra Sasha Luccioni, Noah Smith, Nicole DeCario, and Will Buchanan. 2022. Measuring the carbon intensity of ai in cloud instances. In 2022 ACM Conference on Fairness, Accountability. and Transparency, pages 1877-1894.
This work was supported in part by a grant from the National Science Foundation Graduate Research Fellowship Program under Grant No.
Wikimedia Foundation. 2020. Wikimedia downloads.

---

Yaru Hao. Li Dong, Furu Wei, and Ke Xu. 2020. In- vestigating learning dynamics of bert fine-tur tuning. In Proceedings of the 1st Conference of the Saia-Pccific Chapter of the Association for Computational Lin- guistics and the 10th International Joint Conference on Natural Language Processing, pages 87-92.
Victor Schmidt, Kamal Goyal, Aditya Joshi, Boris Feld. Liam Conell, Nikolas Laskaris, Ziyao Wang, Doug Blank, Jonathan Wilson, Sorelle Friedler, and Sasha Luccioni. 2021. Codecarbon: estimate and track carbon emissions from machine learning computing.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2020, Green AL. Communications of the ACM, 63(12):54 63.
Lynn H. Kaack, Priya L. Donti, Emma Strubell, George Kamiya, Felix Creutzig, and David Rolnick. 2022. Aligning artificial intelligence with climate change mitigation. Nature Climate Change, 12(6))518-
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645 3650, Florence, Italy. Asso- ciation for Computational Linguistics.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu. Lewis Tunstall, Joe Davison, Mario Saško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Samh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac. Thibault Goehringer. Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedingg of the 2021 Conference on Empirical Methods in Natural Language Procexs- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Met- zler. 2022. Scale efficiently: Insights from pretrain- ing and finetuning transformers. In International Conference on Learning Representations.
C.-J. Wu et al. 2022. Sustainable AI: Environmental im- plications, challenges, and opportunities. MLSys.
Xiyou Zhou, Zhiyu Chen, Xiaoyong Jin, and William Yang Wang. 2021. HULK: An energy effi- ciency benchmark platform for responsible natural language processing. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: System Demonstra- tions, pages 329-336, Online. Association for Com- putational Linguistics.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Alexandra Sasha Luccioni, Sylvain Viguier, and Anne- Laure Ligozat. 2023. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24
Yukun Zhu Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Con- ference on Computer Vision (ICCV).
Rakshit Naidu, Harshita Diddee, Ajinkya Mulay, Aleti Vardhan, Krithika Ramesh, and Ahmed Zamzam.
2021. Towards quantifying the carbon emissions of differentially private machine learning. arXiv preprint arXiv: 2000000709966
A Appendix
A.1 Pre-training and Fine-tuning Details
David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Mique Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. 2022. The carbon footprint machine learning train- ing will plateau, then shrink. Computer, 55(7):18- 28.
For pre-training BERT with MLM, we mainly fol- low what is listed in Devlin et al. (2019). The specific hyperparameters used for pre- pre-training are listed in Table 5 and were used on both machines.
Hyperparameters
David Patterson, Joseph Gonzalez, Quoe Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean, 2021. Carbon emissions and large neural network training, arXiv preprint arxiv: 2rXiv:2104.1033
maximum steps training batch size per device
1000000
61
evaluation hatch size per device
maximum sequence length
128
learning rate warmup steps weight decay
le-4
10000
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, distilled version of BERT: smaller, faster. cheaper and lighter. arXiy preprint arXiv:1910.01 1910.01108.
0.01
Table 5: Hyperparameters used for pre- pre-training BERT

---

In Table 6, list the set of hyperparameters used fine-tune each task. All fine-tuning tasks were run on both machines.
Hardware details The A100 machine is located in Haverford, Pennsylvania, USA, and has 4x NVIDIA A A100 GPUs with 40GB GDDR SDRAM, 376GB main memory and 32 Intel Xeon processors. The RTX 8000 machine located in Pittsburgh, Pennsylvania, USA, and has 4x NVIDIA Quadro RTX 8000 GPUs with 48GB GDDR SDRAM, 36 Intel Xeon processors and 251GB RAM.
Figure 2: Kill-A- Kilt-A-Watt wall reading measurement setup
A.3 Additional Results
BERT pre-training and fine-tuning results Ta- ble 8 records the energy consumption results from the 110 GPU machine, located Haverford, PA. The conversion factor for this location is 672.8 lbs COz/MWh. We then calculate the emissions using raw output from CodeCarbon, which is listed in Table 8.
A.2 Kill-A- Watt Measurements
Energy is lost during the process of transferring en- ergy from power source to the machine. The coef- ficient for the loss acquired through the readings from Kill-A-Watt devices. The device measures the instantancous Watt reading extracted from the wall and displays on the monitor. The measure- ments for the A100 machine were recorded only the single node of the cluster containing (the RTX8000 machine is not part of larger cluster). The GPU node connected to two outlets, and we plug separate Kill-A- Watt devices into both. For each instantancous reading, we read off and sum up the readings on both Kill-A-Wan devices. To best compare between the package reading and the wall reading, we read off of the device at the same 15 second interval that CodeCarbon records the energy consumed. To convert each instantaneous Watts readings into Kilowatt-Hours, we follow the formula:
DistilBERT results Table records the energy consumption and emission on the A100 GPU ma- chine. Distillation is not done this machine, and all the fine-tunings are done on the DisTiBBERT trained the RTX8000 machine. The results the A A100 machine shows similar trend that fine- tunings on DistilBERT takes around 50% less en- ergy than on BERT base models.
Energy consumad vs tokens seon an A100
citaset
0.ck
co1822003
ramed enthy recoystiin
cw182012
Squadu]
eeridment analyde
concumen (KWP))
0.8
Inds
squadu2
techusl entalimett
pre train spar)
queet amovers
masked anguage moceing
0.4
0.2
15 seconds
watts x hours
kWh
3600 seconds
1000
Tokens seen
We sum up all the calculations for the entire run of a fine-tuning experiment. Then, we can compare the sum of the wall readings with the Code Car- bon energy consumption recording. We divide the wall reading over the package reading to get a co- efficient, measuring the more realistic energy used. The setup of instantaneous reading on the A 100 machine shown Figure 2. The recordings of the wall readings on the A100 GPU machine are recorded in Table 7.
Figure Energy consumed (kWh) against total tokens seen A100 machine
Training duration and energy consumption From Figure 5 and 6, we see that there is a strong correlation between training time and energy con- sumption, which holds across our models and hard- ware settings (Figure 7). However, similarly to tokens seen, observe that pre-training exhibits slightly different energy consumption profile, do question answering tasks on the A100 machine (which have the longest sequences out of our fine- tuning tasks). When training time estimates are not feasible in advance (such as in certain hyperparam-
We ran these wall reading experiments for both machines and obtained a coefficient of 1.09 on the A100 machine, and a coefficient of 1.05 on the RTX machine.

---

RTE
MNLI
IMDB
SST2
SQuAD
SQuAD
CoNLL
CoNLL 2012
2003
epochs
3
3
5
3
2
2
3
3
train bs
32
32
16
32
32
32
32
32
eval bs
seq len
128
128
128
128
384
384
128
128
doc stride
128
128
2e-5
2e-5
2e-5
2e-5
3e-5
3e-5
5e-5
5
5e-5
Table 6: Hyperparameters used for fine-tuning tasks
Task
Random seed
Wall reading
Code reading (kWh)
Carbon
Power Loss Co- efficient
converted (kWh)
SST2 (no AC)
42
0.103
0.088
1.170
SST2
123
0.059
0.055
1.037
IMDB
42
0.070
0.061
1.131
IMDB
42
0.079
0.071
1.122
IMDB
123
0.0793
0.073
1.092
RTE
42
0.00641
0.00602
1.065
Table 7: Kill-A-Waa experiment recordings for the A100 GPU machine
Task
Dataset
Energy (kWh)
Emissions (kg CO2)
Training time
Equiv. runs
# PT
Equivalent emissions of miles driven
MLM
Wiki + Books
146.150
44.602
726553
144
0.002
0.002
19
58442
0.005
MNLI
0.356
0.239
2400
406
0.613
SQuAD
0.171
0.115
1061
846
0.299
SQuAD v2
0.263
0.177
1580
549
0.454
fine-tuning
IMDB
0.062
0.0190
478
2114
0.048
SST2
0.051
0.034
377
2858
0.087
CoNLL2003
0,011
0.007
78
13411
0.018
CoNIL2012
0.091
0.061
654
1584
0.156
Table 8: Energy consumption of pre-training BERT and fine-tuning the A100 GPU machine. Energy is computed as raw energy measured by CodeCarbon multiplied by coefficient to correct for power loss (Eq. 3.4). Equiv. miles refers the approximate number of vehicle miles driven resulting in equivalent CO2 emissions. Note that here, unlike in Table 2. the "baseline" pre- pre-training cost is fixed to the cost of 1 million steps of masked language modeling sequences length 128.
Task
Datasct
Energy
Time
Emiss.
eter sweeps), we recommend that researchers and practitioners use token counts estimates (including dynamic padding tokens) if they have reasonable knowledge of their data.
RTE
0.001
10
0.001
Entailment
MNLI
0.210
1539
0.141
SQuAD
0.091
566
0.061
SQuAD v2
0.141
915
0.095
IMDB
0.045
341
0.012
Sentiment
SST
0.034
262
0.023
As indicated in Figure and Figure 9, energy in- creases as the optimization steps increases. This is not surprising given the correlation between train- ing time and energy consumption. However, that for different tasks, the energy required for each step is very different. Each step of pre-training takes a longer time, likely due to the higher batch size than all fine-tuning tasks. For QA tasks, the per-step energy consumption is higher than other
20NL
0.007
57
0.005
NER
CoNLL
0.059
448
0.040
Table 9: Energy consumption of training (distillation) and fine-tuning DistilBERT on the A100 machine. Units and energy calculations are same as given in Table

---

20000 Energy consumed vs takens sean on RTX8000
Energy consumed vs duration
dhaset
colnent equansl
coribos
squate]
tomed entty mecoontion
cor12012
s0u2502
Jontiment anslysl
10a
Energy consumed (k90))
0.8
692
mail
Inde
precrsin (4 gaus. langer secr)
pretointe 16 gaust
precein (1 0x42
textual entsimonn:
macord anguage modeligg
quedtion inswedro
SAZ
Itr
mil
Ite
Enargy consumed (KWW))
12012
per-ooe 15 goas
21 geo
0.4
10
chilter:
A133
0.2
10-1
61-47
A133 Dsl
10
RTT0012 BE5
10a
10a
10a
10*
10a
109
Tokens seen
Fine tuning duration (5) in log scale
Figure 4: Energy consumed (kWh) against total tokens
Figure Total training time strongly predictive of total energy consumed (kWh) for both BERT and Dis- tiBBERT models, on both A100 and RTX 8000 GPU machines. Note that both axes are log seale. Dif- ferent from token counts, the trendlines themselves are similar as well across models and hardware settings.
seen RTX8000 machine
Energy training
oMtae::
rell
cnilly eccanicinn
Energy cansurmed (kWh)
Energy consumee vs training steps on A10D
calasst
taak
cont 2013
named
cont 2012
Sauide2 Sauode2
sertiment, ana ys
mebo
tostual mont
Energy consumed (kWh)
592
tran up
Quection anowering
manked language mode
0.05
0
200
400
600
800
1000
1200
1400
Training time
Figure 5: Energy consumed (kWh) against training time (8) on 100 machine, showing the first minutes
2000
4000
6000
8000
10000
Optimization steps
Energy consumed vs training time on RTX8000
0,40
dicaeet
G2233
sQuadv1
namred tion
Figure 8: Energy consumed (kWh) against optimization steps on A100 machine machine
Q212
SQUa212
sentament anatyda
imdb
1211
Loxtub: entaltment
Energy consureed (KWh))
5512
pre 1rin g01
Mulsitinn answoring
0_
milll
pre train spis. Innper seeel
madieed language modeting
Energy consumed vs training steps on RTX8000
catarst
0.20
cont 2013
$2u00v1
namod orite recagn timn
2012
Sauodv2
sore mont anabyis
0.15
ind
(e0 pretrain spos)
texisal entaiiment
Energy consumes (eWh))
(en-rain spot
quention answeding
0.10
mill
peetran spoi kmger segst
midded language modeling
11e
0.05
200
400
600
800
1000
1200
1400
Training (s)
Figure 6: Energy consumed (kWh) against training time (s) on RTX8000 machine, showing the first 30 minutes
2500
5000
7500
10000
12500
15000
17500
Optimization steps
tasks. This is likely due to the maximum sequence length of 384 being higher than for the other tasks. Figure 10 shows the number of examples of the task and the energy consumed log scale. Similar to Figure 1, we see a direct correlation between
Figure 9: Energy consumed (kWh) against optimization steps on RTX8000 machine

---

Energy consumed vs examples seen
Duration (8)
Energy (kWh)
debaet
Energy consumed (kWW) in log scale
Dataset
IGPU
4GPU
IGPU
4GPU
10a
BERT-base
10A
Itc
RTE (NLI)
59
27
8.
8,52e-3
MNLI (NLI)
6700
3014
9.38c-1
8.64e-1
con 1212
10a
tentaana spend spo
SQuADv1 (QA)
3780
1356
5.37c-1
4.28c-1
SQuADv2 (QA)
5604
2011
7.95e-1
6,16c-1
IMDB (Sent.)
1074
472
1.51e-1
1.32c-1
diliier
modens
SST2 (Sent.)
587
323
8.11e-2
8.39e-2
10
A133
CoNLL (NER)
149
737
2.06e-2
2.05e-1
NENT
CoNLL2012 (NER)
1487
737
2.07e-1
2.14e-1
A3D Doiiten
10
I773333 2ear
DistiBBFT
10f
10a 10f
RTE (NLI)
30
18
4.34c-3
4.72e-3
Number examples in log scale
MNLI (NLI)
3447
1760
4.81e-1
4,98e-1
SQUADv1 (QA)
1954
792
2.76e-1
2,40e-1
Figure 10: Energy consumed (kWh) against total num- ber of examples seen on both A100 and RTX8000 chines. Comparing with Figure 1, we see that merely counting number of training examples much less pre- dictive of energy consumption than accounting for ex- ample sequence lengths along with batch size (which affects the maximum sequence length in each batch)
SQuADyz (QA)
2916
1170
4.12e-1
3.46c-1
IMDB (Sent.)
549
272
7.75e-2
7.46e-2
SST2 (Sent.)
306
195
4.21e-2
5.49c-2
CoNLL (NER)
78
436
1.10c-2
1.18c-1
CoNLL2012 (NER)
762
434
1.07c-1
1.19c-1
Table 10: Single-GPU vs. BS and R--optimized 4-GPU fine-tuning energy requirements across end tasks on the RTX8000 machine, for BERT-base and DistilBERT.
higher number of training examples and higher energy usage on both machines,
does not always) use about twice as much energy in both BERT-base and DistilBERT. If we increase batch size x 4 with GPUs (and adjust learning rate accordingly), however, and compare single- GPU fine-tuning with multi-GPU fine-tuning (see Table 10), we observe that energy cost is typically similar or even less than when using 1 GPU, while taking around half much time or less. In both the "naive" and "optimized" multi-GPU settings, the single-vs- venuuli--- GPU difference in energy cost and job duration seems to be related to dataset sequence lengths. Tasks with longer sequences (such as QA tasks, and, to a lesser extent, IMDB and RTE) tend to exhibit more consistent and dramatic energy and time efficiency gains than the other tasks when ing GPUs. On the other hand, tasks with shorter sequences (such NER) tend to require more en- ergy with GPUs, even the wall-clock efficiency may be improved. One way one might interpret this is a large-enough per-device batch size and typical sequence length is necessary for multi-GPU train- ing to be "worth" the overhead of communication between GPUs.
Task accuracy vs emissions
0.93
0.92
scovorr
0.91
0.90
0.88
0.000
0.005
0.030
0.015: 0.020 0.025 0,030 0.035 0,060 0.045 Emissions (kg C02)
Figure II: CO2 emissions (kg) against task accuracy for SST-2 fine-tuning on RTX 8000 GPU machine
Figure 11 shows the accuracy of SST2 task as CO2 emissions (kg) increases. As shown previ- ously in Figure 6, energy consumption increases as time increases. Generally, as emission increases, accuracy increases. We can see that as emissions goes up, accuracy trending towards converging.
Single- vs. multi-GPU fine-tuning We tune-tune BERT-base and DistilBERT base models the RTX8000 machine as well. If the same hyperpa- rameters as the single-GPU setting are used naively (i.e. the same batch size is split over 4 GPUs), fine- tuning using GPUs can (but does not always) take even longer than using just I GPU, and can (but
In light of these observations, our general rec- ommendation is that, if one owns a machine with multiple GPUs, one should consider using all avail- able (idle) GPUs for energy- and time-efficient fine- tuning. Although is often sufficient to use a sin- gle GPU when fine-tuning models of scale similar to ours, and instantaneous energy usage may be

---

higher using more GPUs, total energy used may end up being less while also requiring less time, es- pecially if the training sequences tend to be longer. On the other hand, tasks with short sequences are likely best kept to a single GPU.


Reviews:
[{"review_id": "GRD7db9VmQ", "reviewer": "Reviewer_9NCQ", "summary": "", "questions": "A. Why it is important to measure energy cost of pre-training and fine-tuning separately? Do these two processes have significantly differences when it comes to energy cost?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback and suggestions. \n\n> **the experiments are only conducted with BERT, which is a relatively out-dated compared to more recent models such as RoBERTa, BART, T5, GPT, etc.**\n\nOur work focuses on causal language models. We would expect that a study on generative models’ energy considerations would be structured differently such as including much more substantial exploration of inference costs than in our study of BERT-like models. That being said, we would expect our BERT fine-tuning results to be relevant for RoBERTa fine-tuning and inference due to the shared base architecture even though pre-training takes significantly longer with RoBERTa.\n\n> **If the paper could propose a function to estimate the energy consumed without actually run the experiments, it would be an interesting contribution.**\n\nWe agree this would be a very valuable contribution! Unfortunately, precise energy usage is ultimately largely hardware-dependent (note the greater deviation from trend lines in the A100 experiments vs the RTX8000 experiments), so proposing a general function that completely obviates the need for empirical measurements is out of scope for this work. That being said, one could extrapolate to predict fine-tuning energy costs of new tasks after running only a couple fine-tunings for calibration. This would be particularly useful in settings where many fine-tuning runs over varied datasets is necessary (e.g. experimenting with different training data recipes in a multi-task learning scenario).\n\n\n> **the findings like the energy scales with the number of tokens, seem intuitive and not suprising**\n\nWe agree with the reviewer that our findings generally align with common intuitions, which, until now, were not supported by empirical evidence in a controlled study. That being said, there were multiple *different* alternative conclusions and findings that we were able to empirically rule out as *un*true through our experimentation, that we believe would have been equally intuitive and unsurprising had they turned out to be true instead of the findings presented in the paper. For instance, we would have been unsurprised if using fine-tuning dataset size alone was a sufficient predictor of the amount of energy consumed, perhaps because variation (though we observe this is *not* true in reality; in order to predict the amount of energy consumed in fine-tuning, we must specifically include the per-batch dynamically added padding tokens in the token count total).\n\nTo the best of our knowledge, there is no previous work containing our contributions, and our findings were non-trivial to measure empirically. We believe that many researchers and practitioners would find our work useful and interesting.\n\n\n\n> **A. Why it is important to measure energy cost of pre-training and fine-tuning separately? Do these two processes have significantly differences when it comes to energy cost?**\n\nOne common concern held by the reviewers was that it was unclear why it is valuable to study energy usage associated with fine-tuning separately from pre-training. We sincerely appreciate the opportunity to clarify – the following is a sentence we ended up cutting from a previous version in order to stay within the page limit, but we see now that it would be beneficial to add back in given an additional page:\n\n> While this may seem like it should be a straightforward calculation, there are several variables that can influence compute time and energy consumption, ranging from: (1) the type of hardware used for both pre-training and fine-tuning (since this usually differs between the two), (2) the type of task and the type of computation required to carry it out, and (3) intrinsic characteristics of the dataset, such as average sequence length, its similarity with the pre-training dataset, etc.\n\nTo elaborate further, the energy cost profiles are different, largely due to differences in training infrastructure, training objectives, and sequence lengths typically seen in pre-training vs fine-tuning. 1) Pre-training is almost always done using multiple GPUs which incurs additional energy costs from communication between GPUs vs the single GPU setup characteristic of many fine-tuning workloads. 2) Additionally, during pre-training a language modeling objective is used over much longer sequences than typically seen in fine-tuning, which impacts the way that total token count in a dataset factors into energy costs. Sequences are packed together such that batches consist largely or entirely of sequences of identical length equal to the maximum sequence length for the model. 3) On the other hand, there are many different types of fine-tuning tasks where examples consist of sequences of varying lengths significantly shorter than the maximum length that the model has been trained on (see Table 1). Since effective sequence lengths are determined dynamically at training time (where sequences are padded to the maximum length) in each given batch, total training time is not as simple to extrapolate from standard measures of dataset size as in pre-training. \n\nThe reviewers are in agreement that our methodology was described clearly and in detail, and that the claims we make are well-supported. To the best of our knowledge, no previous work provides such detailed estimates of energy costs and carbon emissions associated with fine-tuning BERT models. BERT and DistilBERT are widely used today by researchers and practitioners, and fine-tuning is a common workload that we believe is worth studying separately from pre-training and inference. Overall, we are hopeful that our work will be useful to those within and beyond the NLP community to inform decision-making.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "First, the paper is clearly written and easy to follow, which is appreciable. Second, the goal and claims of the paper are stated clear and supported with ambient empirical evidence. It does provide valuable empirical datapoints for energy cost of fine-tuning BERT that the community can refer to.", "reasons_to_reject": "The significance of this work might be limited. First, the experiments are only conducted with BERT, which is a relatively out-dated compared to more recent models such as RoBERTa, BART, T5, GPT, etc. If the paper could propose a function to estimate the energy consumed without actually run the experiments, it would be an interesting contribution. Second, the findings like the energy scales with the number of tokens, seem intuitive and not suprising.", "questions_to_authors": "A. Why it is important to measure energy cost of pre-training and fine-tuning separately? Do these two processes have significantly differences when it comes to energy cost?", "paper_topic_and_main_contributions": "This paper provide an empirical study of the computational costs of pre-training and fine-tuning BERT.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mjUN0huJAF", "reviewer": "Reviewer_NXfn", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback! We are glad you agreed that our paper provides a thorough analysis of energy and carbon intensity of pre-training and fine-tuning BERT models. We are hopeful that our work and its findings will be useful to researchers and practitioners. Please see our response below:\n\n> **Compared to inference, there are very few fine-tuning models**\n\nWe respectfully disagree with the reviewer’s statement that there few fine-tuned models – in applied ML settings, fine-tuning remains widely used - as can be seen by the almost 300,000 NLP models shared on Hugging Face, the vast majority of which are fine-tuned from a subset of base models - for instance, over 16,000 of them are fine-tuned versions of BERT. Furthermore, just recently Open AI announced that they would be offering a finetuning service for their GPT-series models, indicating that this remains a timely paradigm even in generative settings.\n\n>  **the author does not delve deeply into the carbon emissions resulting from the inference process**\n\nWe include emissions measurements from inference in Table 2 primarily for context – inference efficiency is much more well-studied than fine-tuning, and it is well-known that inference is much less energy-intensive than training. That being said, in the next version of the paper we can more explicitly relate our primary results to the inference measurements:\n\nAlthough we do observe that per-example inference efficiency costs are related to sequence lengths, there is overall *less variation across datasets and tasks* in inference costs compared to fine-tuning costs. This mirrors an observation noted in the HULK benchmark as well [1], although to the best of our knowledge ours is the first to explicitly draw comparisons across task types and different aspects of dataset size (i.e. number of examples and examples’ sequence lengths).\n\n| Dataset | Task | Median sequence length | Inference energy  (kWh) / 1000 ex. | Fine-tune energy (kWh) / 1000 ex. |\n| --- | --- | --- | --- | --- |\n| RTE | Entailment | 56 | 0.750e-3 | 3.276e-3 |\n| MNLI | Entailment | 37 | 0.704e-3 | 2.388e-3 |\n| SQuAD v1 | Question Answering | 159 | 1.078e-3 | 6.070e-3 |\n| SQuAD v2 | Question Answering | 158 | 1.235e-3| 6.034e-3 |\n| IMDB | Sentiment Analysis | 128 | 0.801e-3 | 6.030e-3 |\n| SST2 | Sentiment Analysis | 10 | 0.653e-3 | 1.204e-3 |\n| CoNLL2003 | Named Entity Recognition | 15 | 0.763e-3 | 1.465e-3 |\n| CoNLL2012 | Named Entity Recognition | 20 | 0.829e-3 | 1.791e-3 |\n\nFor the table above, inference was performed on a single RTX8000 with batch sizes of 1. Fine-tuning was performed on the same with batch sizes of 32 except for IMDB where bs=16. \nNote: the authors realized that there were some mistakes in the inference section of Table 2: the result for MNLI was a typo, and we had mistakenly applied a coefficient twice originally. The table above reflects the true values, and we will update the paper accordingly\n\n[1] Xiyou Zhou, Zhiyu Chen, Xiaoyong Jin, and William Yang Wang. 2021. [HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing](https://aclanthology.org/2021.eacl-demos.39). In *Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations*, pages 329–336, Online. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a comprehensive analysis of the energy and carbon footprint of both pre-training and fine-tuning BERT models. And it provides a more complete picture of the environmental impact of these widely used NLP models.\n", "reasons_to_reject": "The main reason to reject this paper is about the lack of real-world scenarios. Compared to inference, there are very few fine-tuning models, and the author does not delve deeply into the carbon emissions resulting from the inference process.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors compare fine-tuning energy use across a suite of standard supervised NLP datasets, including entailment, sentiment analysis, question answering, and NER tasks, with training data sizes ranging from 6K to 400K examples. They also measure energy use across two different sets of hardware. And they suggest increasing batch size for energy-efficient training.\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CCRIJ0ijWI", "reviewer": "Reviewer_ikfR", "summary": "", "questions": "Question A: If we want to improve the fine-tuning energy efficiency, what can we learn from your paper in addition to some well-known conclusions (e.g., increasing the batch size)?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. Please see our response below:\n\n> **why do we need to consider energy and carbon usage during the fine-tuning stage?**\n\nWe describe differences in assumptions between pre-training and fine-tuning below, but we would first like to address a commonly held belief that fine-tuning generally presents only trivial energy costs. We note in our abstract that, while it is indeed true that a single fine-tuning run’s energy requirements are likely to be negligible compared to pre-training costs, fine-tuning occurs much more frequently and is practiced by many more individuals than pre-training. It is not unrealistic to imagine scenarios where one would run enough fine-tunings (perhaps in a thorough hyperparameter sweep – see Table 2) that the total cost amounts to a sizable fraction of (or even exceeds) the cost of pre-training the model. \n\nEven in our own work, where 1) pre-training was a prominent workload and 2) precise fine-tuning  task accuracy was not a primary objective or consideration (and therefore substantial hyperparameter searches were unnecessary), we estimate 5% of our total emissions from our project are attributable to fine-tuning – equivalent to emissions from over 50 miles driven in a car. \n\nAs an aside, we have been involved in and reviewed submissions for previous projects where *only* fine-tuning was performed, where we estimate that the total energy costs easily met or exceeded the 350kg in carbon emissions from our current work. *In each case, the costs arose from reasonable and thorough experimentation across a variety of tasks, models, and methods – and reviewers still asked for additional experiments.*\n\nWe believe this is also relevant to the ethical concern raised by the reviewer: \n> **To explore the effect of carbon usage on fine-tuning PLMs, this work mimics the whole pre-training and then fine-tuning paradigm and emissions of 350 kg of carbon equivalents.**\n\nPre-training, the most energy intensive component of our work, was necessary for contextualizing the fine-tuning measurements on the same hardware. Our hope is that others who hope to contextualize their own fine-tuning costs can cite our findings without having to run the same full experiments themselves.\n\n> **What is the difference between pre-training and fine-tuning when considering energy and carbon usage?**\n\n>  **Compared with other learning strategies (e.g., prompt-tuning, in-context learning, low-rank adapter), fine-tuning is more similar to the pre-training stage since they both update all parameters of the LLM.**\n\nOne common concern held by the reviewers was that it was unclear why it is valuable to study energy usage associated with fine-tuning separately from pre-training. We sincerely appreciate the opportunity to clarify – the following is a sentence we ended up cutting from a previous version in order to stay within the page limit, but we see now that it would be beneficial to add back in given an additional page:\n> While this may seem like it should be a straightforward calculation, there are several variables that can influence compute time and energy consumption, ranging from: (1) the type of hardware used for both pre-training and fine-tuning (since this usually differs between the two), (2) the type of task and the type of computation required to carry it out, and (3) intrinsic characteristics of the dataset, such as average sequence length, its similarity with the pre-training dataset, etc.\n\nTo elaborate, there are differences in training infrastructure, training objectives, and sequence lengths typically seen in pre-training vs fine-tuning. 1) Pre-training is almost always done using multiple GPUs which incurs additional energy costs from communication between GPUs vs the single GPU setup characteristic of many fine-tuning workloads. 2) Additionally, during pre-training a language modeling objective is used over much longer sequences than typically seen in fine-tuning, which impacts the way that total token count in a dataset factors into energy costs. Sequences are packed together such that batches consist largely or entirely of sequences of identical length equal to the maximum sequence length for the model. 3) On the other hand, there are many different types of fine-tuning tasks where examples consist of sequences of varying lengths significantly shorter than the maximum length that the model has been trained on (see Table 1). Since effective sequence lengths are determined dynamically at training time (where sequences are padded to the maximum length) in each given batch, total training time is not as simple to extrapolate from dataset size and token count as in pre-training.\n\n\n> **What is the difference between different fine-tuning strategies?**\n\n> **it will be more valuable to explore carbon usage with learning strategies that do not update the LLM or only update fewer parameters.**\n\nWe agree that investigating other parameter-efficient fine-tuning methods or other learning strategies would be useful follow-up work (although most of the other learning strategies are not common in the masked language models which we experiment with). We focus on dense fine-tuning as a standard and widely practiced method; fine-tuning remains widely used - as can be seen by the almost 300,000 NLP models shared on Hugging Face, the vast majority of which are fine-tuned from a subset of base models - for instance, over 16,000 of them are fine-tuned versions of BERT. Furthermore, just recently Open AI announced that they would be offering a finetuning service for their GPT-series models, indicating that this remains a timely paradigm even in generative settings.\n\nWe view our current submission as a first contribution to a line of work investigating the energy demands of task-specific learning across a greater variety of settings. Although there is certainly room for extended follow-up work, we believe that ours is a useful and well-scoped initial finding that should be shared with the community.\n\n\n> **Question A: If we want to improve the fine-tuning energy efficiency, what can we learn from your paper in addition to some well-known conclusions (e.g., increasing the batch size)?**\n\nWe agree that improving fine-tuning energy efficiency is a worthy goal, though we do not attempt to propose direct solutions in our current work. Instead, we hope that our work 1) motivates others to consider fine-tuning as a non-trivial workload when planning their experimentation, 2) allows others to estimate fine-tuning costs much more easily than if starting from scratch and 3) encourages others to adopt the practice of estimating the energy/carbon emissions associated with their experimentation, even if each individual fine-tuning is not particularly carbon intensive.\n\n\n\n> **The organization of this paper can be improved (the conclusion section is missing).**\n\nThank you for this feedback. Given an extra page, we will devote much of it to highlighting specific conclusions from our work.\n\n> **It would be better if the authors could move the tables from the appendix to the contents while providing more in-depth discussions (and then extend this paper to a long one).**\n\nWe believe that our work is appropriate for a [short paper](https://2023.emnlp.org/calls/main_conference_papers/#short-papers). Under the types of acceptable papers in the [EMNLP 2023 CFP](https://2023.emnlp.org/calls/main_conference_papers/#contributions), we believe our paper may be considered an \"NLP engineering experiment.\" Historically (e.g. [EMNLP 2021](https://2021.emnlp.org/call-for-papers#short-papers), [ACL 2023](https://2023.aclweb.org/calls/main_conference/)), papers making a \"small, focused contribution\" using only \"a few pages with sufficient level of detail\" have been seen as appropriate for submission as a short paper. We are glad that the reviewer found our emissions and energy estimations professional and detailed; we are hopeful that many NLP researchers and practitioners will find our work useful.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper seems professional in emission estimation and provides a detailed emission result on BERT pre-training and fine-tuning.", "reasons_to_reject": "1. It is more likely to be a technical report or a blog rather than a research paper: why do we need to consider energy and carbon usage during the fine-tuning stage? What is the difference between pre-training and fine-tuning when considering energy and carbon usage? What is the difference between different fine-tuning strategies?\n2. Compared with other learning strategies (e.g., prompt-tuning, in-context learning, low-rank adapter), fine-tuning is more similar to the pre-training stage since they both update all parameters of the LLM. Therefore it will be more valuable to explore carbon usage with learning strategies that do not update the LLM or only update fewer parameters.\n3. The organization of this paper can be improved (the conclusion section is missing). It would be better if the authors could move the tables from the appendix to the contents while providing more in-depth discussions (and then extend this paper to a long one).", "questions_to_authors": "Question A: If we want to improve the fine-tuning energy efficiency, what can we learn from your paper in addition to some well-known conclusions (e.g., increasing the batch size)?", "paper_topic_and_main_contributions": "This paper performs a careful empirical study to estimate the carbon usage in the ''pre-train and then fine-tune'' paradigm by reimplementing the BERT pre-training and fine-tuning stage. They conduct experiments to measure carbon usage and provide empirical results.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "To explore the effect of carbon usage on fine-tuning PLMs, this work mimics the whole pre-training and then fine-tuning paradigm and emissions of 350 kg of carbon equivalents."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the energy cost of BERT during the fine-tuning process across various datasets and contextualizes this information in relation to pre-training costs. The reviewers praise the paper for the insightful empirical results and data points it provides on fine-tuning energy use. The reviewers highlight the potential value of these data points as a reference for the community, particularly in the context of hyperparameter optimization and assessing the overall cost associated with fine-tuning BERT models.

However, reviewers have also raised some concerns regarding the paper's limitations, particularly its focus on specific hardware and models. This limitation could restrict the broader applicability of the study's findings as a community reference. While this is an important shortcoming, it might be partially addressed in the next iteration of the paper, e.g., by discussing the observed differences between the two employed GPU variants. For instance, a discussion might provide some guidance to the reader to what extent the empirical results and which aspects of the present study will remain relevant in the future. The next iteration of this paper should also clarify/address the concerns raised by the reviewers regarding the distinction of multi-GPU (for pre-training) and single-GPU (for fine-tuning) results, and the impact of this distinction on the present study.
Decision: Accept-Findings