# Check-QZP: A Lightweight Checkpoint Mechanism for Deep Learning Frameworks
## 2 Background and Related Works
### Training and Checkpoint Mechanism in ML model  
In general, a deep learning model comprises training and inference steps [29]. An inference step generates predictions based on the information transferred from the training step, which means that the training step is a requirement for making predictions.  
Therefore, we briefly explain how the training step works step by step (see Figure 1). The training step can be categorized into three phases: data loading, forward propagation, and backward propagation with weight updates [30]. The first phase is data loading, during which the data stored in the underlying storage media or remote storage devices are read. This phase includes the overheads from the I/O operations needed to read data through storage or network paths. After completing the data loading phase, the intermediate variable values in the neural network model, ranging from the input layer to the output layer, are calculated, resulting in the calculated training loss; we call this forward propagation. Once the forward propagation phase is complete, the backward propagation phase is triggered to update the network's weights and biases, which helps find the correct answer by computing the gradient of the loss. In the traditional approach, the training step is performed in a serialized manner and is repeated until a pre-defined threshold is satisfied; the threshold is pre-defined by a user and is calculated based on the number of epochs and mini-batches [31; 32]. Note that the training step can take a long time to complete, depending on the number of layers and the complexity of the problem to be solved. Unfortunately, this step does not provide persistence of the data because all variable values are stored in the host and GPU memory.  
A checkpoint mechanism is adopted to support the persistence of data, storing a set of variable values on the underlying storage media, such as an HDD or SSD [6; 33]. In an ML model, the checkpoint operation provides a full snapshot of in-memory values, including the weights, biases, and gradients of each layer. Therefore, the checkpoint in an ML model helps to quickly recover the last version of the intermediate variable values generated during the training of the ML model after a random crash or system failure occurs. Let us consider a simple example where the training step takes at least 7 days, and a system crash occurs 3 days after the start time of the training. In this example, if the checkpoint operation is available, we can recover the last version's data and resume the next process within a few seconds or minutes after the crash. Therefore, the training step can be completed after 4 days. Otherwise, after the crash occurs, the entire training step would need to be restarted, requiring 7 days to complete.  
However, the checkpoint mechanism causes two unavoidable problems: _checkpoint delays_ and waste of storage space [6; 7]. First, the checkpoint mechanism can postpone the training step to wait for the completion of I/O requests from the storage media and a flush operation for data persistence; we call this a _checkpoint delay_. Unfortunately, the _checkpoint delay_ is unpredictable and wastes hardware resources in that it leads to GPU resources being idle during the checkpoint operation. This is because current servers in cloud environments cannot guarantee I/O latency since they simultaneously handle sensitive data among multi-tenant applications by sharing their hardware resources. It is important to note that the data stored during checkpoint operations are available only  
Figure 1: Training process of an ML model in multi-tenant cloud environments.  
when system crashes or power failures occur. In other words, the checkpoint mechanism periodically issues a vast volume of data (e.g., at a granularity of MB or GB) to store values for the immediate variables on the storage media. Thus it can waste storage space and reduce the lifetime of storage devices in the case of SSDs.