# uDMA: An Efficient User-Level DMA for NVMe SSDs +
## 3 Background and Motivation
### Motivation  
To assess the effectiveness of the existing user-level DMA mechanism, we evaluated the state-of-the-art SPDK and the traditional memory copy method in Linux. We evaluated the overheads associated with memory page pinning and copying under various data sizes. First, we used SPDK's _spdk_dma_malloc_ each time to allocate and pin memory pages for different data sizes. Then, the pinned memory pages were released immediately. We performed this operation ten more times and calculated the average time cost. After that, we copied the data to the pinned memory for the same testing data sizes and calculated the average overhead caused by data copying. Figure 1 shows the statistical results.  
We identified two major shortcomings of the latest user-level DMA mechanism: (1) pinning memory page operation has a specific time overhead, and (2) a single user-level DMA method cannot be suitable for all I/O requests with various data sizes.  
(1) Pinning memory page operation has a specific time overhead. Figure 1 shows the cost of pinning memory pages. First, pinning one memory page (e.g., 1 KB or 4 KB) takes about 800 \(\upmu\)s. Second, with the increase in memory page size, the time cost of pinning memory pages increases gradually. In addition, as the number of pinning memory pages increases, so does the time overhead. Especially for I/O-intensive applications, frequently pinning memory pages can bring a huge time overhead, increasing the processing time of the whole I/O software stack in the OS. This can seriously degrade the potential of NVMe SSDs.  
(2) A single user-level DMA method cannot be suitable for all I/O requests with various data sizes. As shown in Figure 1, when the data size is less than 60 KB, the overhead of pinning memory pages is more significant than the memory copy. However, as the data size increases, the time taken to copy memory is gradually greater than the cost of pinning memory. In addition, the time overhead of the memory copy will continue to increase. Therefore, there is a tradeoff between pinning memory pages and memory copies.  
We make the following conclusions from the observation. First, for small data requests, copying data from pageable to pinned memory is shorter than pinning memory. Furthermore, because of the small quantity of data, only a few memory blocks need to be pinned to meet the requirement. It means we can pin a small piece of memory pages during the initialization phase and keep it pinned throughout the application's life cycle. Second, we can build a pinned memory pool for medium data requests to avoid frequent allocating, pinning, and freeing memory blocks. When a new I/O request arrives, if the available memory in the pinned memory pool meets the needs of the I/O request, we can directly fetch the corresponding memory block and assign it to the I/O request without allocating and pinning the new memory block. Finally, for the I/O requests of big data, the time consumption of pinning memory pages is less than that of the memory copy. However, pinning a large piece of memory blocks for a long time can lead to inefficient memory use and bring the lack of memory to other system applications. Therefore, we can only dynamically allocate and pin memory blocks for these requests.  
Figure 1: Time consumption of pin memory and memory copy.