# Investigating Supercomputer Performance with Sustainability in the Era of Artificial Intelligence
## 3 Literature Review  
Technology Trend Shift from Central-Processing-Unit-Dominated to Graphical-Processing-Unit-Dominated Supercomputers  
The era of AI has significantly disrupted the dominance of CPU-based supercomputers, triggering a shift toward GPU-dominated systems. GPUs have become the foundational technology for today's generative AI, providing the infrastructure required for large-scale model pre-training and inference. As a result, GPUs have recently emerged as the dominant computing core, driven by the growing demand fueled by the rapid evolution of AI. In today's rapidly evolving AI technology, GPUs are widely adopted for the pre-training and evaluation of deep neural networks such as transformer architecture [21; 22; 23]. The computation of a deep neural network based on mathematical operations is parallelized. The parallelization operations are successful as a result of the large volume of cores on GPUs making it possible for distributed computational operations on multiple cores to speed up operations. In the parallelization process, the processing units are required to operate independently. Parallelization operations on a PC with average hardware features comprising four or eight cores cannot be the same as parallelization on GPUs comprising thousands of cores. Parallelization is directly proportional to the number of cores. Operational speed increases with parallel processing, and GPUs are often used for the parallelization process. The parallelization capabilities of GPUs are higher than those of CPUs because GPUs have significantly more cores than CPUs [24]. The parallelization capabilities, high throughout, and thousands of cores of GPUs make them fit for the pre-training of deep neural networks on large-scale volumes of data where concurrent processing is required, unlike conventional CPUs optimized for sequential operations [25]. GPUs delivered high performance with greater EE compared to CPUs [26]. The 2019 investment of USD 1 billion in OpenAI in 2019 led to the development and deployment of the Azure AI supercomputer, designed for large-scale pre-training of deep neural networks. The Azure AI supercomputer is embedded with more than 285,000 CPUs and over 10,000 GPU cores; it became one of the most powerful AI supercomputers in the world [22]. GPU-based systems can be scaled up effectively to meet the requirements of supercomputing [26]. An increase in workload means increasing the number of GPUs in the system to allow increasing computational power to handle large models and high volumes of data and to allow the configuration of the system for model pre-training or inference. The AI supercomputing architecture designed to handle complex computation of AI applications heavily relied on GPUs to provide massive capabilities for large-volume data analytics [25]. The largest machine learning models in this era of AI are LLMs that have billions of parameters and consume millions of GPU hours during the pre-training process, emitting carbon dioxide. For example, the pre-training of BLOOM consumed 1,082,990 GPU hours and 433,196 kWh of power. GPT-3 consumed 1287 MWh of energy and emitted 502 tonnes of carbon dioxide. Gopher consumed 1066 MWh of power and emitted 352 tonnes of carbon dioxide, and OPT consumed 324 MWh of energy and emitted 70 tonnes of carbon dioxide. It is estimated that the pre-training of the BLOOM model emitted 24.7 tonnes of carbon dioxide and 50.5 tonnes of carbon dioxide for dynamic power consumption and the lifecycle of BLOOM, respectively [7]. Most of the LLMs in the world today, such as ChatGPT and most of the AI projects of Microsoft, Amazon, and Meta, are powered by GPU systems [27]. The demand for GPUs, driven by the evolution of AI, has enabled Nvidia (the maker of GPU accelerators) to overtake Amazon in market capitalization. Nvidia's stock is now worth USD 1.83 trillion, narrowly surpassing Google's USD 1.82 trillion valuation.