# Efficient Use of GPU Memory for Large-Scale Deep Learning Model Training
## 5 Experiment and Performance Analysis
### Execution Time According to Memory Advise  
Figures 6-9 show results of experiment. We performed 100 iterations of all experiments. The execution time shown in the graphs are the average time it takes to perform one iteration. The legends of the graphs are in the form of memory advise of model parameters-memory advise of input data. The legend of Original in Figure 6 is when unified memory is not used. In the graphs, PL stands for _Preferred location_. Further, RM means _Read mostly_ and NA means no memory advise. The x-axis of the graphs shows the memory advise setup of the intermediate results. For example, in the case of PL-RM and noadvise in the graph, model parameters are set to _Preferred location_, input data are set to _Read mostly_, and memory advise of intermediate results is not set. Figure 6 shows the training time when the data size is smaller than the GPU memory size. The y-axis of Figure 6 starts with 0.8. BERT-Large can be trained with the input mini-batch size of 16 and the max sequence size of 256 without using unified memory. When deep learning is performed without applying the Unified Memory, the time to perform one iteration is about 0.87 s, which shows the best performance. On the other hand, when unified memory is applied, performance is lower than when unified memory is not applied. In particular, when the model parameters are set as the _Preferred location_ and no advise is set in the input data, the training time when prefetching the intermediate results is about 0.98 s, showing the lowest performance. Further, when the memory advise of the model parameters and the memory advise of the input data are the same, prefetching  
Figure 6: Execution time when the data size is smaller than GPU memory size. (Note that the y-axis of this graph starts with 0.8.) the intermediate result shows the lowest performance. The reason for showing the lowest performance when prefetching intermediate results is that if the data size is smaller than the GPU memory size, the communication overhead due to prefetch is larger than the deep learning operation time on the GPU; therefore, if deep learning can be performed in GPU memory, it is better not to prefetch intermediate results. When model parameters are set as _Preferred location_ and input data are set to _Read mostly_, if _Preferred location_ or memory advise is not set for intermediate results, the execution time is 0.89 s, which is only 0.019 s longer than when the unified memory is not applied. This is the best performance when unified memory is applied. The reason for such good performance seems to be that, when memory advise is set as described above, no other overhead occurs except for the overhead caused by the initial page fault when deep learning is performed. Figure 7 shows the increase rate of training time due to the use of unified memory when deep learning can be performed in GPU memory. The learning time increases by up to 12.3% and at least by 2.3% when the unified memory is applied. Therefore, if deep learning is possible within the GPU memory, applying unified memory cannot efficiently perform deep learning. Further, it is better not to prefetch intermediate results when unified memory is applied. Figure 8 is a graph showing the execution time when deep learning cannot be performed without using unified memory because the data size is larger than the GPU memory size. Please note that y-axis of Figure 8 starts with 14. Figure 8 shows that the performance is the lowest when memory advise is not set for model parameters and input data; however, setting the model parameters to the _Preferred location_ gives the best performance. If deep learning can be performed in GPU memory, prefetching intermediate results shows low performance. On the other hands, when training within the GPU memory is not possible, prefetching intermediate results showed the best performance.