SlowGuess/NAACL_2022_OCR huggingface dataset:
In training different models we tracked energy consumption throughout each run and observed that the rate of energy consumption (power) is roughly constant after averaging over short intervals (one minute in this case). This is depicted in Figure 6 for four jobs with identical parameters but different power-cap limits. It can be expected that cutting training time by  $X$  percent will correspond to an  $X$  percent reduction in energy. We highlight this in consideration of common practices of significantly extending training times for marginal performance gains. For instance in (Devlin et al., 2019) doubling the number of training batches provided an additional  $1\%$  increase in performance on a particular benchmark test set. For certain applications or domains this additional training may make sense, but in cases where evaluation metrics include energy considerations, longer training for marginal performance improvements would be counterproductive and could incur significant energy expenditure.
3.4 Utilizing Efficient Datacenters
One last practice we address that can help researchers reduce their environmental impact is utilizing institutional shared datacenters and cloud computing resources for energy-intensive NLP applications. By considering this approach for applications as opposed to building and managing smaller, private HPC workstations or clusters, re
searchers can save money on equipment purchases and potentially energy bills depending on where resources are housed, as well as reducing the carbon footprint of their workloads. While there is convenience in having private computing resources that are accessible, this convenience comes at a cost. Generally speaking energy savings and impact is more easily obtained at larger scales. Datacenters and cloud computing providers make significant investments in the efficiency of their facilities. For instance, Google publishes data on its PUE, reaching a 12 month average of 1.10 in  $2021^{3}$ , and the National Renewable Energy Laboratory sets an annual goal of running their computing facility at a PUE of less than  $1.06^{4}$ , recently achieving a record of 1.036.
Additionally, many cloud providers are moving their energy supply towards more environmentally friendly and renewable energy sources in attempts to reduce their carbon output to zero (Barr, 2015). These types of improvements would be time-consuming and difficult to make for individual researchers, but by sacrificing some conveniences, AI researchers can reap these benefits without additional effort beyond moving their projects to these platforms.
4 Discussion and Recommendations
We believe the approaches proposed here offer easy-to-implement solutions for reducing the carbon footprint of NLP applications without significant algorithmic or software changes. Though they do not involve new algorithmic methods which are outside the scope of this article, these represent early steps towards more efficient NLP. Coupling them with algorithmic changes would further improve energy consumption. The goal of this article is to initiate a conversation between NLP researchers and those in the hardware and datacenter domains. Below, we list additional recommendations that may help shape such a conversation.
Understanding your computational environment's characteristics: Previous works highlighted the carbon footprint of computationally expensive NLP applications, and their recommendations of tracking and reporting energy usage was intended to encourage researchers to be aware of their individual impact. Similarly we highlight
the importance of datacenter characteristics and PUE variation to promote a deeper understanding of research energy requirements and the factors that constitute them. We hope this work leads NLP researchers to question assumptions about the datacenters where their workloads are running and what the relative efficiency of those datacenters are. For example, researchers should opt for energy-efficient datacenters and encourage their organizations to deploy or leverage energy-efficient datacenters. If possible, it would also be helpful for researchers to learn these operating characteristics of their datacenters or computing providers. Further we encourage the NLP community to work with their computing facility or datacenter to implement frameworks for tracking energy consumption like that outlined in Section 3 and other works (e.g. Henderson et al. (2020)).
Promoting better energy usage: In recent years, top conferences in AI and machine learning have introduced the requirement that papers include an ethics statement addressing the potential impact of their work on the broader society. However, one area that is currently lacking is the impact of AI on the environment. It may be difficult to account for every trial run or hyperparameter tuning when tracking and reporting estimates of energy usage. However, we hope that this practice promotes better awareness of AI energy consumption and fosters a greater focus on optimization pathways to reduce energy usage.
Alongside reporting energy consumption statistics, we make the additional recommendation that conference papers' statements discussing ethical considerations also identify steps undertaken to minimize energy consumption. We give an example of such an "energy statement" after concluding. Additionally, it is critical that energy-efficient NLP research be promoted in the research community, perhaps via specialized tracks or workshops focused on these problems.
Reducing the environmental impact: The purpose of this research is to educate NLP researchers on tools that can be used to reduce their energy usage and empower them to leverage those tools to minimize their carbon footprint. The methods discussed fit into a wider research effort to enable more efficient AI. Lastly, we also echo earlier calls for promoting more energy-conscious NLP practices and discourage overtraining or extensive hyperparameter searches. Reviews of conference sub
$^{3}$ https://www.google.com/about/datacenters/efficiency/ $^{4}$ https://www.nrel.gov/computational-science/measuring-efficiency-pue.html
missions should consider whether new methodologies are effective or the result of expensive optimization.
This article presents techniques that can improve the energy efficiency of training and inference for NLP applications. Importantly, the methods discussed can be used jointly with each other to achieve a compounding effect of energy savings. Future work relevant to these topics would include a wider survey of AI hardware and power-capping capabilities. While we focused on NVIDIA GPUs, evaluation of AI hardware from other vendors and cloud providers could have a potentially large impact for cloud computing as well as large shared high performance computing centers.
Energy Statement
The experiments performed in this work consumed a total of  $782\mathrm{kWh}$ . A majority of the experiments (approximately  $760\mathrm{kWh}$ ) were performed on our institution's high performance computing cluster, powered by largely carbon-free, hydroelectric power sources. To minimize energy consumption, much of these experiments were performed during system downtimes (e.g., when the system is undergoing scheduled maintenance and less busy) and when cooling needs are reduced.
The authors acknowledge the MIT Lincoln Laboratory Supercomputing Center (LLSC) for providing HPC resources that have contributed to the research results reported in this paper. The authors wish to acknowledge the following individuals for their contributions and support: Bob Bond, Tucker Hamilton, Jeff Gottschalk, Tim Kraska, Mike Kanaan, CK Prothmann, Charles Leiserson, Dave Martinez, John Radovan, Steve Rejto, Daniela Rus, Marc Zissman, Matthew L Weiss, David Bestor, Michael Jones, Albert Reuther, William Arcand, William Bergeron, Chansup Byun, Michael Houle, Matthew Hubbell, Hayden Jananthan, Jeremy Kepner, Kurt Keville, Anna Klein, Adam Michaleas, Peter Michaleas, Lauren Milechin, Julia Mullen, Charles Yee, Andrew Prout, and Antonio Rosa. We also acknowledge support from NSF awards 1920020, 2124897, and 1910601, the Massachusetts Green High Performance Computing Center, and Northeastern University.
Dario Amodei, Danny Hernandez, Girish SastryJack, Jack Clark, Greg Brockman, and Ilya Sutskever. 2018. Ai and compute. https://openai.com/blog/ai-and-compute/.
Rhonda Asciero and Andy Lawrence. 2020. Uptime institute global data center survey 2020. Technical report, Uptime Institute.
Jeff Barr. 2015. Cloud computing, server utilization, & the environment. https://aws.amazon.com/blogs/aws/cloud-computing-server-utilization-the-environment/.
Jeff Barr. 2019. Amazon ec2 update - inf1 instances with aws inferentia chips for high performance cost-effective inferencing. https://aws.amazon.com/blogs/aws/amazon-ec2-update-inf1-instances-with-aws-inferentia-chips-for-high-performance-cost-effective-inferencing/.
Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. 2016. An analysis of deep neural network models for practical applications. arXiv, abs/1605.07678.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186.
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2185-2194, Hong Kong, China. Association for Computational Linguistics.
Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations.
Nathan C. Frey, Baolin Li, Joseph McDonald, Dan Zhao, Michael Jones, David Bestor, Devesh Tiwari, Vijay Gadepally, and Siddharth Samsi. 2022. Benchmarking resource usage for efficient distributed deep learning. arXiv, abs/2201.12423.
Azzam Haidar, Heike Jagode, Phil Vaccaro, Asim YarKhan, Stanimire Tomov, and Jack Dongarra. 2019. Investigating power capping toward energy-efficient scientific applications. *Concurrency and Computation: Practice and Experience*, 31(6).
Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. 2020.
Towards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research, 21(248):1-43.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv, abs/1503.02531.
George Leopold. 2019. Aws to offer nvidia's t4 gpus for ai inferencing. https://www.hpcwire.com/2019/03/19AWS-upgrades-its-gpu-backed-ai-inference-platform/.
Joseph McDonald, Siddharth Samsi, Daniel Edelman, Chansup Byun, Jeremy Kepner, and Vijay Gadepally. 2021. Improved compression for word embeddings by scaling principal components. In 2021 IEEE High Performance Extreme Computing Conference (HPEC).
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In Proceedings of the 5th International Conference on Learning Representations (ICLR).
Jiaqi Mu and Pramod Viswanath. 2018. All-but-thetop: Simple and effective postprocessing for word representations. In International Conference on Learning Representations.
NVIDIA. 2021a. Nvidia data center GPU manager (dcfgm). https://developer.nvidia.com/dcfgm. Online; accessed November 2021.
NVIDIA. 2021b. Nvidia-smi. http://developerdownload.nvidia.com/compute/DCGM/docs/nvidia-smi-367.38.pdf. Online; accessed November 2021.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021. Carbon emissions and large neural network training.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Albert Reuther, Jeremy Kepner, Chansup Byun, Siddharth Samsi, William Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, Michael Jones, Anna Klein, Lauren Milechin, Julia Mullen, Andrew Prout, Antonio Rosa, Charles Yee, and Peter Michaleas. 2018. Interactive supercomputing on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC), pages 1-6. IEEE.
Siddharth Samsi, Matthew L Weiss, David Bestor, Baolin Li, Michael Jones, Albert Reuther, Daniel Edelman, William Arcand, Chansup Byun, John Holodnack, Matthew Hubbell, Jeremy Kepner, Anna Klein, Joseph McDonald, Adam Michaleas, Peter Michaleas, Lauren Milechin, Julia Mullen, Charles Yee, Benjamin Price, Andrew Prout, Antonio Rosa,
Allan Vanterpool, Lindsey McEvoy, Anson Cheng, Devesh Tiwari, and Vijay Gadepally. 2021. The mit supercloud dataset. In 2021 IEEE High Performance Extreme Computing Conference (HPEC), pages 1-8.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv, abs/1910.01108.
Roy Schwartz, Jesse Dodge, Noah Smith, and Oren Etzioni. 2020. Green ai. Communications of the ACM, 63:54 - 63.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F. Manso. 2020. The computational limits of deep learning.
Rob Toews. 2020. Deep learning's carbon emissions problem. https://www.forbes.com/sites/robtoews/2020/06/17/deep-learning-s-climate-change-problem/.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems, volume 33, pages 17283-17297. Curran Associates, Inc.