# Performance Analysis of Container Effect in Deep Learning Workloads and Implications
## 5 Flushing Dirty Pages to Secondary Storage
To investigate the characteristics of write-back activities over time, we plot in Figure 4 the number of write-backs that occur for each memory page as time progresses. As shown in the figure, dirty pages that are being written back are evenly distributed across all pages in the early stages of deep learning, but after a certain time point, some limited pages are consistently flushed. This means that some limited data in deep learning is constantly modified, and it is necessary to efficiently manage these data flushes for stable performance of the deep learning training process. In the early stages of deep learning training, data is loaded, models are initialized, and weights are changed frequently. This means that there are a variety of dirty pages that need to be flushed back into storage [33]. However, as the model learns and the weights converge over time, the number of dirty pages decreases. That is, after a certain time point, the model repeatedly updates some limited data, which generates certain pages increasingly hotter as shown in Figure 4. Note that these hot \begin{table}
\begin{tabular}{l l l l} \hline \hline  & **Background** & **Periodic** & **Sync** \\ \hline SqueezeNet & 23,753 (42.1\%) & 32,542 (57.6\%) & 180 (0.3\%) \\ Mobile-BERT & 19,466 (34.9\%) & 36,106 (64.8\%) & 186 (0.3\%) \\ AlexNet & 4884 (28.5\%) & 12,020 (70.2\%) & 208 (1.2\%) \\ LSTM & 5361 (35.0\%) & 9773 (63.8\%) & 186 (1.2\%) \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Frequency and ratio of write-backs (multiple Dockers). \begin{table}
\begin{tabular}{l l l l} \hline \hline  & **Background** & **Periodic** & **Sync** \\ \hline SqueezeNet & 2400 (31.6\%) & 5182 (68.1\%) & 24 (0.3\%) \\ Mobile-BERT & 3040 (18.4\%) & 13,500 (81.5\%) & 20 (0.1\%) \\ AlexNet & 1674 (22.0\%) & 5866 (77.2\%) & 57 (0.8\%) \\ LSTM & 2157 (25.8\%) & 6155 (73.6\%) & 49 (0.6\%) \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Frequency and ratio of write-backs (single Docker). dirty pages can cause performance issues as they should essentially be flushed to storage frequently, wasting CPU and memory resources. To handle this situation, we suggest an intermediate non-volatile flushing layer residing between the main memory and the storage. Our architecture shown in Figure 5 has the mission of improving the write-back performance by making use of NVM as the front-end cache of secondary storage, leading to reduced I/O traffic and frequency of flush operations to slow disk storage. Specifically, our preliminary architecture utilizes NVMe as a secondary storage buffer to accelerate the write-back performances. As NVM is also a non-volatile medium like hard disks, we can eliminate storage flushing operations. To assess the effectiveness of this system architecture, we conduct simulation experiments for write-back activities in container-based deep learning workloads and compare the results of the original system and those with our NVM-added architecture. We use the parameters of a Toshiba DT01 ACA1 hard disk drive (HDD) with a read/write access latency of 8 milliseconds for secondary storage. For NVM media, we use the parameters of a phase change memory (PCM) with a write latency of 350 nanoseconds. Note that PCM is a well-known NVM media that can be placed in front of slow storage to accelerate I/O performances [16]. Figure 4: Write-back traces captured for each page number as time progresses. Figure 6 shows the I/O latency of the proposed NVM-added architecture in comparison with the traditional system that does not use NVM as workloads and system situations are varied. As shown in the figure, our preliminary architecture improves the I/O latency significantly in all cases. Specifically, the improvements for SqueezeNet, Mobile-BERT, AlexNet, and LSTM, are 83%, 90%, 84%, and 77%, respectively, in single Docker and 78%, 77%, 86%, and 83%, respectively, in multi-tenant Docker environments. The improvement is the largest in the single Docker Mobile-BERT dataset.