# Efficient Use of GPU Memory for Large-Scale Deep Learning Model Training
## 3 Efficient Data Management Scheme for Deep Learning  
In this study, we enable large-scale model training by expanding GPU memory by integrating CPU memory and GPU memory through CUDA Unified Memory technology in a single GPU environment. In this section, we propose a method to efficiently manage data used in deep learning using memory advise and data prefetch supported by CUDA 8.0. For example, an efficient convolution algorithm requires feature map data and layer weights required for feed forwarding, and an additional temporary buffer is required when using a fast-Fourier-transform-based convolution algorithm [19]. In this paper, we classify these data into two types: model parameters and intermediate results. Since each layer weight is part of the model parameters, the layer weights can be classified as model parameters. Feature map data and temporary buffer are data generated and used during feed forwarding or backpropagation, so we classified these data as intermediate results. However, the data required to perform deep learning are not only intermediate results, such as feature map data and temporary buffers, and model parameters, but also input data to be trained. Therefore, we can classify the data needed in deep learning into three types: model parameters, intermediate results, and input data. The three types of data needed to perform deep learning are created, modified, and used at different times. This means that the three types of data needed to perform deep learning each have different access patterns. The characteristics of the three types of data used in deep learning are as follows:  
* First, there are **model parameters**. Model parameters are generated and initialized before starting deep learning. Model parameters are weights for the features of the input data. When performing feed forwarding, the weight of each hidden layer(part of model parameters) generates feature map data to be passed to the next hidden layer through mathematical operation with the feature map data output from the previous hidden layer. After performing backpropagation, model parameters are updated with new values. The updated model parameters will be used by the GPU for training in the next iteration. In addition, in most deep learning models, model parameters account for the smallest proportion of the total data of deep learning [19]; therefore, we recommend that the model parameters remain on the GPU until training is complete. * Second, there are **intermediate results**. The intermediate result is the feature map data of each hidden layer that is generated as a result of mathematical operation of model parameters and input data when performing feed forwarding, and a temporary buffer used when backpropagation is performed. Therefore, intermediate results are newly generated and used by feed forwarding and backpropagation at every iteration, and are not used again after the iteration ends. Further, since the intermediate results are the largest among the three types of data required to perform deep learning, it is difficult to store all the intermediate results in the GPU memory [19]. Every intermediate result generated in each hidden layer during feed forwarding does not always have to be in the GPU memory. However, when performing backpropagation, the intermediate results generated during feed forwarding in each hidden layer must be in GPU memory before backpropagation of that hidden layer is performed. * Finally, there are **input data**. The input data are the data to be trained by passing them to the first layer (input layer) of the deep learning model. Input data are delivered to the GPU, which performs deep learning, with new data every iteration. In addition, the input data are not changed due to the deep learning operation and are no longer needed after one iteration. Therefore, the input data do not always have to be on the GPU until training is finished. The Table 1 summarizes the access patterns according to the types of deep learning data.