# Efficient Use of GPU Memory for Large-Scale Deep Learning Model Training
## 1 Introduction  
To achieve high accuracy when performing deep learning, the size of deep learning models is increasing [1; 2; 3; 4; 5; 6]. However, there is a difficulty in using large-scale deep learning models that can achieve high accuracy due to the limitation of single-GPU memory [7; 8; 9; 10; 11; 12]. NVIDIA introduced CUDA Unified Memory technology from CUDA 6, allowing the GPU memory to be expanded by integrating the system's host (CPU) memory and device (GPU) memory [13; 14; 15]. CUDA Unified Memory enables automatic data migration between host memory and device memory via virtual memory. For example, if the GPU tries to access a virtual page that does not exist in GPU memory, a page fault occurs. After that, the page fault is resolved by mapping the accessed page to the physical page of the GPU and migrating the data existing in the host memory to the GPU memory. Through this, device memory and host memory are logically integrated to achieve the same effect as the memory of the GPU is expanded. Therefore, it is possible to train large-scale deep learning models with expanded GPU memory using CUDA Unified Memory technology. NVIDIA provides CUDA Unified Memory technology from CUDA 6.0 to expand GPU memory. CUDA Unified Memory technology integrates CPU memory and GPU memory so that it can be used as a single address space; therefore, CUDA Unified Memory enables the training of large-scale deep learning models by expanding the small GPU memory. In addition, starting with CUDA 8.0, memory advise can be used to efficiently manage data allocated to unified memory according to the access pattern [13; 16; 17]. NVIDIA divides memory advise into three types according to access patterns: _Read mostly_, _Preferred location_, and _Access by_. _Read mostly_ makes efficient use of read-intensive data. When accessing dataset to _Read mostly_, a replication of the data is created in the device that accessed the data. _Preferred location_ sets the device where the data will be located; therefore, the occurrence of page faults due to data access is minimized by locating the data in the device that will mainly use the data. _Access by_ sets the device to access the data. Devices set to _Access by_ are directly mapped to the data; therefore, when device accesses the data, a page fault does not occur. The data required for deep learning can be roughly classified into three types: model parameters, input data, and intermediate results. Model parameters must be updated with new values every iteration and must be on the GPU until training is complete. Input data are changed to new data every iteration and are not changed due to deep learning operations. Intermediate results are data that are newly generated for each iteration through deep learning operations, and are no longer needed after one iteration. The access pattern of data used in deep learning is similar to the memory advise supported by NVIDIA. Therefore, when deep learning is performed by applying CUDA Unified Memory technology, it is possible to efficiently perform deep learning using memory advise. In this paper, we extend GPU memory using CUDA Unified Memory technology for large-scale deep learning model training. To the best of our knowledge, our proposed scheme is the first approach to efficiently apply CUDA Unified Memory technology and hints for memory management provided by NVIDIA to deep learning. In addition, we propose an efficient GPU memory utilization method for large-scale training model training using memory advise. The main contributions of our work are as follows:  
* First, we propose the most efficient memory advise for the three types of data required for deep learning. Deep learning data can be divided into three types according to access patterns: model parameters, input data, and intermediate results. Further, memory advise can manage data by dividing it into three types: _Read mostly_, _Preferred location_, and _Access by_.