# Investigating Supercomputer Performance with Sustainability in the Era of Artificial Intelligence
## 1 Introduction  
Supercomputers are the main components driving innovations in the artificial intelligence (AI) boom, particularly in the development and inference of large language models (LLMs). Companies adopting supercomputers aiming for AI innovations typically gain a competitive edge, positioning themselves to lead in the race to dominate the AI industry. With the emergence of NVIDIA as the leader in AI, many organizations are increasingly shifting focus to high-performance computing (HPC) systems as a catalyst for rapid AI advancement. However, not all supercomputers are optimized for AI. Some supercomputers are specifically designed for handling both AI workloads and other scientific computations. Additionally, some supercomputers have the flexibility to be upgraded or extended to support AI capabilities [1]. For example, NVIDIA's Eos offered an impressive 18.4 exascale floating-point operations per second (Eflops) for AI computational performance and advancing the AI research field. NVIDIA's Selene extends its capabilities to support research in robotics, data analytics, and autonomous vehicle technology. Fujitsu's Fugaku operates at the exascale level. The integration of HPC and AI workloads has been instrumental in advancing AI research and development. Notably, the Japanese LLM is a product of Fugaku's capabilities. Aurora is another powerful HPC designed for scientific computations, including brain mapping [1]. In addition, other supercomputers with the potential to accommodate AI capabilities include Leonardo Frontier, the Large Unified Modern Infrastructure (LUMI), Sierra, Perlmutter, and Summit [1]. The HPC community has entered the era of exascale supercomputers. However, with the rapid evolution of AI technology, it is expected that exascale supercomputers may struggle to maintain pace with the increasing advancement of AI models in the future. To address this challenge, Japan has announced a five-year project for the development of zettascale supercomputers [2]. Typically, an enormous amount of energy is required to power supercomputers [3]. The amount of power is in the range of tens of thousands of kilowatts to almost 50,000 kw; for example, Aurora consumes 38,698 kw of power. LLMs consume a lot of power in different phases, such as pre-training, storage, data transfer, inference, and hardware manufacturing. The workload demand in data centers for the year 2023 was estimated at 1088 million computer instances, with an estimated power consumption of 411 terawatts per hour [4]. For years, power consumption in data centers has been stable despite the growing workload [5]. However, the AI evolution has sparked an increasing demand for energy to power supercomputers in data centers. It is estimated that the power demand will increase by 160% every year up to 2030, requiring 200 terawatts per hour. Europe is estimated to require USD 1 trillion to upgrade its power grid to cope with the increasing AI energy demand [4]. For example, GPT-3 was pre-trained on 10,000 graphical processing units (GPUs) at the Microsoft datacenter, and GLaM was pre-trained on TPUv4s at the Oklahoma datacenter [6]. The effectiveness of data center power usage for BLOOM, GPT-3, OPT, and Gopher is 1.2, 1.1, 1.092, and 1.08, respectively [7]. Currently, companies are investing billions of dollars into AI projects to gain a competitive edge over their competitors and win the AI race to become the leader in the industry. For example, Workday injected USD 500 million into AI, focusing on generative AI. In another financial war chest, Meta significantly increased its AI investment from USD 30 billion in 2023 to USD 40 billion in 2024. A consortium called the Global AI Infrastructure Investment Partnership comprising Microsoft, MGX, Global Infrastructure partners, NVIDIA as a technical adviser, and Blackrock is raising an ambitious amount ranging between USD 80 and USD 100 billion to invest in the construction of a data center and energy-efficient infrastructure for powering the data center [8].