# Efficient Use of GPU Memory for Large-Scale Deep Learning Model Training
## 6 Related Works
Our proposed scheme can manage memory more efficiently by managing deep learning data by dividing it into three types of model parameters, intermediate results, and input data. To overcome the limitations of large-scale deep learning model training, Microsoft developed Zero Redundancy Optimizer (ZeRO) by integrating existing data parallelization techniques and model parallelization techniques [22]. Through ZeRO-DP, Microsoft achieved both computational and communication efficiency, which has the advantage of data parallelism, and memory efficiency, which has the advantage of model parallelism, overcoming the limitations of existing parallelization techniques. Microsoft has also developed ZeRO-R to optimize three types of residual state memory: activations, temporary buffers, and unusable memory fragments. Microsoft integrated ZeRO-DP and ZeRO-R to implement ZeRO, a memory-optimized system for deep learning. ZeRO efficiently utilizes multiple GPUs to train large-scale models. Unlike ZeRO, we propose a technique for efficiently training large-scale models on a single GPU environment. In addition, we divide data used for deep learning into three types according to access patterns and apply memory advise to each type of data to efficiently utilize GPU memory. GPU-Enabled Memory-Aware Model-Parallelism System (GEMS) has been proposed to train large-scale deep learning models using high-resolution images, which are mainly used in digital pathology [23]. In their paper, four types of techniques are proposed: GEMS-Basic, GEMS-MAST, GEMS-MASTER, and GEMS-Hybrid. GEMS-MAST trains a replica of a part of the model by utilizing memory and computing resources that were not used in the existing model parallelization technique. GEMS-MASTER uses more replicas of model that GEMS-MAST used only two. Through this, GEMS-MASTER can train the model faster by overlapping computations and reducing the number of allreduce executions. They also achieved near-linear scalability with the GEMS-Hybrid. Unlike our work, GEMS requires the use of multiple GPUs to train large-scale models. Our work uses CUDA Unified Memory technology to train large-scale models even in a single GPU environment, and can efficiently manage memory to improve training performance. Out-of-Core DNN training framework (OC-DNN) is a deep learning framework that uses the new Unified Memory feature with Volta and Pascal GPUs [24]. The two main components of OC-DNN are the Interception Library(IL) and OC-Caffe. IL is an independent library that operates between CUDA Runtime/Driver and HPC Platform. IL allows applications to run using data that exceed GPU memory without modifying existing applications. OC-Caffe can save GPU memory and overlap data copy and computation by using Unified Memory buffer. In addition, OC-Caffe changes the existing device-to-device communication to Managed memory-to-Managed memory communication. Further, they use _cudaMemAdvise_ and _cudaMemPrefetch_ to optimize intra-GPU communication. OC-DNN and our study are similar in that both use CUDA Unified Memory technology. Unlike OC-DNN, We divide deep learning data into three types and use memory advise, a hint for memory management provided by NVIDIA, to efficiently manage memory to improve training performance.