## 1 Introduction  
Artificial intelligence natural language processing models have developed rapidly based on the transformer-based model [1; 2; 3]. The transformer model is an artificial intelligence model that can understand and generate text like a human. These models perform well when more parameters are included [4; 5]. In order to train a model with many parameters, an AI supercomputer is required. NVIDIA DGX-A100 system [6] is a widely used AI supercomputer, consisting of 8 A100 Tensor Core GPUs with 40 GB of GPU memory. Recently, the size of language models has become very large, and in order to secure the GPU memory required for learning, dozens to hundreds of DGX-A100 clusters are used for training. However, obtaining a substantial number of these supercomputers is challenging for many researchers. This barrier often results in an inability to procure the ample GPU memory necessary for language model training. In this paper, we introduce a novel training algorithm designed to overcome GPU memory constraints in a single DGX-A100 system. This enhancement facilitates the training of larger language models and enables the expansion of batch sizes. Our approach manages the entire model by dividing it into sequential sub-models on the CPU and main memory [7; 8]. Each sub-model is then composed of parallel tasks, which are transmitted to a multi-GPU environment, executed simultaneously, and the results returned to the CPU. The proposed algorithm operates in an environment where frequent communication and computation between various resource types are essential. The asynchronous management and deployment of resources and tasks are crucial for efficient functioning. Processing performance is further optimized via strategic task placement across multiple execution streams (GPU, CPU, and IO). We have developed a comprehensive management system to effectively organize, synchronize, and execute complex tasks in this intricate environment. Additionally, we present a scheduling technique that organizes the workflow within the management system. This approach allows the training algorithm to utilize heterogeneous resources more efficiently, enabling the training of larger language models. Our methodology significantly reduces the GPU memory required for large-scale model training. It has been specifically optimized for the DGX-A100 system, allowing researchers to train more extensive models and larger batch sizes even within limited resources. The major contributions of this study are threefold. First, we focus on developing a training algorithm tailored to the heterogeneous environment of the DGX-A100 system. A core technology was implemented to enhance the system's overall efficiency, overcoming GPU memory limitations via a novel method of model division and parallelization using CPU and memory. Second, we have crafted a high-performance management system that ensures the smooth execution of our newly developed algorithm. This system orchestrates the training process and resources, managing and deploying computational tasks asynchronously. The result is not only optimized learning task efficacy but also a substantial enhancement in the overall learning process performance. Lastly, we introduce a strategic scheduling technique for the training algorithm within our management system. This method places tasks effectively across multiple execution streams and coordinates communication and computational tasks between heterogeneous resources. The coordination leads to a maximized overall training efficiency, aligning the process with the optimal resource usage [9]. The rest of this paper unfolds as follows: Section 2 establishes the need and uniqueness of our study by exploring the previous efforts to overcome GPU memory constraints and enable large-scale parallel learning.