# Efficient Use of GPU Memory for Large-Scale Deep Learning Model Training
## 4 Implementation in PyTorch  
PyTorch, an open source deep learning framework, currently does not support CUDA Unified Memory technology; therefore, it is impossible to train a large-scale deep learning model that requires more than the GPU memory size through PyTorch in a single GPU environment. In this study, to overcome the limitations of existing PyTorch, CUDA Unified Memory technology is applied to PyTorch to enable large-scale deep learning model training. PyTorch performs computation in the Define-by-Run method [20]. PyTorch does not allocate the amount of memory required for operation in advance in the way that Tensorflow does, which operates in a Define-and-Run method, but rather allocates GPU memory when a user requests or requires additional memory space due to GPU operation. The user can copy a specific tensor to another device's memory by calling the _to()_ function. The user can call _to('cuda')_ to copy data to GPU memory. The tensor on which _to('cuda')_ is called is transferred to GPU memory, and the tensor can be used for computation using the GPU. Further, PyTorch automatically allocates additional GPU memory space for the computed results using the tensor in the GPU and stores the computed results in the GPU memory. Table 2 show the allocated GPU memory size according to the training progress when ResNet-50 is trained using PyTorch. The ResNet-50 model is a representative convolutional neural network (CNN) for image classification consisting of 50 convolution layers. Looking at Table 2, one can see that the GPU memory usage is 519 MiB when the ResNet-50 model is initialized in the GPU. Further, it can be seen that GPU memory usage increased to 6027 MiB after feed forwarding because PyTorch automatically allocated a memory space to additionally store intermediate results generated by performing feed forwarding. Finally, by performing backpropagation, additional intermediate results were generated, increasing the GPU memory usage to 7121 MiB. This shows that PyTorch allocates GPU memory when additional GPU memory is needed due to the user's _to('cuda')_ call and the result of GPU operation. As a result of analyzing the source code of PyTorch, we confirmed that all GPU memory allocation of PyTorch is performed through CUDACachingAllocator. CUDA Unified Memory technology can be applied to PyTorch simply by calling _cudaMallocManaged()_ instead of _cudaMalloc()_ in the _malloc()_ function of CUDACachingAllocator. When implemented in a simple way, the user cannot use the general GPU memory allocation method, but also has a problem where only fixed memory advise can be set; therefore, a problem arises where users have no choice but to use unified memory even when the memory required for deep learning is smaller than the GPU memory. Further, even when unified memory is used due to insufficient GPU memory, CUDA Unified Memory technology cannot be used efficiently because the same memory advise must be set for data with different access patterns. Therefore, we need to implement so that users can choose whether to apply CUDA Unified Memory and choose memory advise. In order to implement the \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Data Type**} & **Does Data Change Due to** & **Is Data Accessed by** & \\  & **Deep Learning Operation?** & **GPU on the Next Iteration?** & **Techniques for Managing Data** \\ \hline Model Parameters & Yes & Yes & _Preferred location_ \\ Input Data & No & No & _Read mostly_ or no memory advise \\ Intermediate results & Yes & No & Prefetch \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Characteristics of deep learning data. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Initialize Training Model** & **After Feed Forwarding** & **After Backpropagation** \\ \hline
519 MiB & 6027 MiB & 7121 MiB \\ \hline \hline \end{tabular}
\end{table} 
Table 2: GPU memory usage according to the training progress.