# Performance Analysis of Container Effect in Deep Learning Workloads and Implications
## 4 Analysis of Deep Learning Event Traces
### Dirty Pages and Write-Backs  
In this subsection, we track the frequency of write-back system calls to compare the incidence of dirty pages across containers, host processes, and multi-tenant environments. In a multi-tenant environment, we run four containers simultaneously. We set the flush interval for our Linux system according to the environment variables in Table 7. \begin{table}
\begin{tabular}{l r r} \hline \hline
**\# perf record -g-e block:block\_rq\_insert-p [pid] \&\& perf report** \\ \hline... & 17.77\% kworker/u40:5-f [kernel.kallysysms] [k] blk\_mq\_insert\_requests \\ ret\_from\_work & \\ kthread & \\ worker\_thread & \\ process\_one\_work & \\ wb\_workfn & \\ wb\_do\_writeback & \\ + wb\_writeback & \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Call graph of block request insert. \begin{table}
\begin{tabular}{l l r r r} \hline \hline  & **Event** & **Container Process** & **Container Daemon** & **Host Process** \\ \hline CPU & cpu\_core/cycles/ & 461,729,070,471,469 & 124,892,660,806 & 306,402,857,705,993 \\  & cpu\_atom/cycles/ & 93,706,213,673,857 & 93,207,046,124 & 61,663,593,615,121 \\  & cpu\_core/instructions/ & 412,465,430,821,263 & 198,780,273,963 & 178,197,584,510,724 \\  & cpu\_atom/instructions/ & 106,333,305,694,655 & 112,349,586,140 & 9,714,425,305,632 \\  & cpu\_core/cache-misses/ & 1,375,684,053,246 & 1,760,922,473 & 641,571,962,797 \\  & cpu\_atom/cache-misses/ & 539,034,658,626 & 1,663,241,340 & 61,020,677,212 \\ \hline Memory & page-faults & 1,685,309,171 & 9081 & 825,523,873 \\  & minor-faults & 342,203,571 & 9035 & 150,807,077 \\  & major-faults & 1,343,104,572 & 14 & 674,716,307 \\  & cpu\_core/mem-loads/ & 0 & 0 & 0 \\  & cpu\_atom/mem-loads/ & 1,688,256 & 0 & 3313 \\  & cpu\_core/mem-stores/ & 41,600,872,930,344 & 19,056,452,909 & 12,134,821,271,569 \\  & cpu\_atom/mem-stores/ & 11,791,995,380,030 & 10,442,649,500 & 511,089,910,826 \\ \hline System & block:block\_touch\_buffer & 281,259 & 1244 & 244,841 \\ call & block:block\_dirty\_buffer & 267,489 & 269 & 159,348 \\  & block:block\_rq\_complete & 977,810 & 143 & 1,100,795 \\  & block:block\_rq\_insert & 48,881 & 4 & 28,958 \\  & block:block\_rq\_issue & 971,675 & 528 & 1,096,962 \\  & block:block\_bio\_backmerge & 39,741 & 436 & 51,398 \\  & block:block\_bio\_frontmerge & 428 & 0 & 114 \\  & block:block\_bio\_queue & 1,007,074 & 993 & 1,145,499 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Event and system call traces extracted from container and host processes. Figure 3 shows the frequency intervals of write-back system calls invoked in a single container, multiple containers, and host systems. As shown in the figure, containers call write-back functions more frequently than host processes on average. Also, when we compare the standard deviation of write-back call intervals in Figure 3, containers exhibit a much larger variance than the host system. This implies that the estimation of write-back system calls is more difficult in container-based deep learning, causing potential performance penalties. When comparing single and multi-tenant container environments, however, there is no significant difference in the write-back system call interval. This implies that multiple containers are certain to flush more dirty pages than a single container per each write-back system call. That is, as containers need to perform global data synchronization consistently, multi-tenancy inherently increases I/O traffic to storage even if it does not increase write-back frequency [31]. Specifically, storage write traffic will be increased with multiple containers, as each container needs to synchronize its data with the data of other containers. In our experiments, the write-back system call interval of Docker containers is shorter than the default storage flush interval listed in Table 7. This implies that dirty pages are generated more frequently in the container than with the default storage flush setting, resulting in more frequent sync system calls.