# Efficient Use of GPU Memory for Large-Scale Deep Learning Model Training
## 2 Background
### How to Use CUDA Unified Memory Efficiently  
For efficient use of CUDA Unified Memory technology, NVIDIA supports memory advise starting with CUDA 8.0. Memory advise can be divided into three types according to data access patterns [13; 16]. Figure 1 shows how memory advise works.  
* _Preferred location_ can set the physical location where data are actually stored. Data set as _Preferred location_ are fixed in the designated physical location. Therefore, if the data are accessed from another device, data are not migrated to the device but data are directly mapped. For example, as shown in Figure 1, if Device 1 attempts to access data fixed to Device 0, a page fault occurs in the first access. After the page fault, the data that are set as the _Preferred location_ and fixed to Device 0 are mapped directly to the Device 1 instead of unmapping them from the memory of Device 0 and migrating to Device 1 memory. Therefore, the next time the Device 1 accesses the data, a page fault will not occur. _Preferred location_ can operate efficiently when specific data needs to be physically fixed in specific device.
* _Access by_ specifies direct mapping to a specific device, as depicted in Figure 1; therefore, a device that is directly mapped by designating _Access by_ does not generate a page fault when accessing the corresponding data. The difference between data set as _Access by_ and data set as _Preferred location_ is that data set as _Access by_ are not fixed to a specific device but can be moved freely; therefore, the direct mapped device has the advantage that the data can be accessed without a page fault no matter where they are physically located; however, in the case of frequently accessed data, they may be  
Figure 1: Operation method according to the type of memory advise: (**a**) Data set as _Preferred location_ are directly mapped when accessed from other devices; (**b**) If data are set to _Access by_, a direct mapping is created with a specific device; (**c**) When another device accesses data set to Read mostly, a replica is created on the accessed device.  
an overhead because it takes a long time to access them because they are physically located on a different device.
* _Read mostly_ can be used efficiently when applied to read-intensive data. As you can see from Figure 1c, if data set to _Read mostly_ are accessed by a device that is not physically stored, a replica is created on the accessed device instead of migrating the accessed data to the device that accessed the data. Therefore, the data set to _Read mostly_ have the advantage that it can be accessed quickly from the device because there are copies in the device's memory after a page fault occurs in the first access; however, if the data set to _Read mostly_ are modified, there is a disadvantage that the data must be modified for all devices where replication exists.  
NVIDIA supports the data prefetch scheme in addition to memory advise to efficiently utilize CUDA Unified Memory technology. Before accessing the data allocated to unified memory, the user can call _cudaMemPrefetchAsync()_ to prefetch the data to the device where the data will be used. Data prefetched through _cudaMemPrefetchAsync()_ can be accessed without a page fault when accessed from the device because the data are already in the device. Data prefetch operates differently depending on the settings of memory advise. In the case of data set to Read mostly, if data prefetch is performed, a replica is created in the device. When the data set as the Preferred location are prefetched to a device other than the device where the data are fixed, the data are moved to another device instead of being fixed to the device.