# Efficient Use of GPU Memory for Large-Scale Deep Learning Model Training
## 6 Related Works  
Among the methods for optimizing GPU memory when performing deep learning, there is a swap in/out method between GPU memory and CPU memory. vDNN, a memory management system provided by NVIDIA, operates by swapping in/out of feature map data generated by deep learning operations to overcome the limitations of GPU memory [19]. During feed forwarding, vDNN swaps out feature map data not currently used in GPU memory to CPU memory. After that, vDNN swaps in the feature map data required for backpropagation from CPU memory to GPU memory; therefore, it is possible to learn a mini-batch larger than the size of the GPU memory. vDNN is similar to our proposed technique in that it improves the performance of deep learning through efficient memory management. However, unlike vDNN, our work utilizes CUDA Unified Memory to expand GPU memory. In addition, we apply memory advice according to data access patterns to efficiently use CUDA Unified Memory. Figure 11: GPU page fault pattern in single mini-batch training. Figure 10: This is a visualization of the profiling results using the NVIDIA Visual Profiler: (**a**) The profiling result when memory advise is not set; (**b**) A visualization of the profiling results when configured with memory advise that shows the best performance; (**c**) A visualization of profiling results when configured with memory advise, which shows the worst performance. [PERSON], [PERSON], [PERSON], [PERSON], [PERSON], [PERSON] et al. proposed a new memory management method for accelerating deep learning applications [11]. They extended NVIDIA's vDNN concept to address the computational performance degradation caused by PCIe bottlenecks in multi-GPU environments. To address the PCIe bottleneck, they limited the number of GPUs that swap feature map data generated in a specific hidden layer when performing deep learning. Further, they designed an algorithm to effectively prefetch the swapped out feature map data into GPU memory. Unlike their research, we propose a memory expansion method in a single GPU environment using CUDA Unified Memory technology and memory advise. [PERSON], [PERSON] and [PERSON]. et al. evaluated the performance of CUDA Unified Memory technology and memory advise on the Intel-Volta/Pascal-PCI platforms and the Power9-Volta-NVLink platform [13]. In their study, they evaluated the performance of CUDA Unified Memory and memory advise using eight applications such as matrix multiplication and convolution operations that use GPUs. Their study evaluated the performance of CUDA Unified Memory and Memory Advise on various hardware platforms. Through experiments, they showed that when the size of data exceeds the size of GPU memory, using memory advise for CUDA Unified Memory can improve performance. However, their study has a limitation in that there is no performance evaluation for the widely used deep learning workloads. Our study shows that the size of trainable mini-batch increases by applying CUDA Unified Memory to deep learning workloads, and that memory can be managed efficiently by using memory advise. Most of the datasets used in GPU applications such as deep learning are very large, which is not only larger than the GPU memory size but also larger than the host memory size [21]. In most HPC systems, data sets are stored in non-volatile memory (NVM) storage, which has a large capacity; therefore, for complex applications, data movement between GPU memory and NVM storage can be a difficult problem. To solve this problem, Direct Resource Access for GPUs Over NVM (DRAGON) was developed. DRAGON extended GPU memory to NVM storage, not just host memory, by extending NVIDIA's Unified Memory technology; they optimize the data movement by dividing the data access pattern into read-only and write-only, and separately manage the intermediate data generated through GPU operation. DRAGON optimizes only for read-only and write-only, so there is a limitation in that it is difficult to optimize according to data access patterns when performing deep learning.