# Check-QZP: A Lightweight Checkpoint Mechanism for Deep Learning Frameworks
## 1 Introduction  
Today, deep learning (DL) is emphasized even more as its technologies cover a wide spectrum of applications [1, 2]. Many researchers have carried out studies that enable DL technologies to upgrade legacy services or provide new applications [3, 4, 5, 6, 7, 8]. For example, some researchers have focused on autonomous vehicles or unmanned aerial vehicles (UAVs), incorporating DL technology to offer more intelligent services to clients [9, 10, 11, 12, 13]. Other researchers have enhanced custom-fit video recommendation systems using DL technology [14, 15, 16]. This research trend leads to excessive I/O operations for the learning and inference processes and an increase in data centers to store the vast amounts of data generated by applications or various sensors [17, 18, 19]. Unfortunately, the resources (e.g., CPU, GPU, memory, network, and storage devices) in the data centers must be shared among many applications in multi-tenant cloud environments [20, 21]. Therefore, applications with AI engines have to compete for limited resources in the data centers, resulting in interference across applications in terms of CPU/GPU preemption, memory allocation, and storage I/Os. Meanwhile, most platforms supporting AI engines sometimes write the bulk of their data to the underlying storage devices (e.g., HDDs or SSDs) while running the learningprocess [22, 23]. This is because the AI platforms have to store on-the-fly data that are generated during the training process to avoid losing the data in the event of a sudden power failure (i.e., _checkpoint_ operation) [6, 7]. In other words, after unexpected system crashes, the training process can be resumed using the data belonging to the last version of the checkpoint operation instead of performing the first step again. However, the data stored in the storage media are never reused unless the system crashes and are unavailable after running the next training epoch. In addition, the checkpoint operation to store the data can negatively affect a series of training processes, as it must wait for the write operations to finish; it may write more than several GB of data [6, 7, 24]. Unfortunately, the latency of the write operations can be delayed when I/O interference unintentionally occurs due to excessive I/O operations from other running applications; we call this situation a _checkpoint delay_ due to I/O conflict [24, 25]. In recent years, some studies have focused on designing the checkpoint operation to address this delay in terms of I/O bandwidth and recovery time [6, 7, 26, 27]. For example, the authors of Check-N-Run [7] adopted a quantization and parallelism mechanism into the checkpoint operation to deal with storage capacity and write bandwidth. The authors of CheckFreq [6] proposed a lightweight checkpointing framework for DNN training to speed up recovery time in the event of a sudden power failure. Unfortunately, previous studies do not consider the excessive I/O conflicts in cloud environments, where the latency of I/O operations from the checkpoint operations is limited because of interference from other running multi-tenant applications. In this paper, we propose Check-Q2P, which efficiently addresses the _checkpoint delay_ issue by reducing the total amount of checkpointed data under excessive I/O conflict scenarios. The contributions of this paper can be summarized as follows:  
* In order to understand how the training process works, we briefly introduce the internal procedure of the training process. Then, we describe why a _checkpoint delay_ occurs and what kinds of problems can arise. * We propose Check-Q2P, which consists of three features--quantization, parallelism, and compression--to efficiently optimize overall performance and storage space. * We implement Check-Q2P at the framework level and compare it with the default mechanism in multi-tenant scenarios where the ML model performs training and inference under high I/O pressure from the excessive FIO [28] benchmark. The rest of this paper is organized as follows.