# Check-QZP: A Lightweight Checkpoint Mechanism for Deep Learning Frameworks
## 3 Design and Implementation
### Quantization and Compression for Low Storage Costs  
Figure 2 shows the overall training process of Check-Q2P. For the checkpoint operation, Check-Q2P first performs a snapshot that copies in-memory values on the GPU (e.g., weights, biases, and gradients) to the host memory (denoted by ). After completing the snapshot operation, it triggers a series of operations to optimize the storage costs for data persistence. In general, quantization is used to store data at lower bit widths by converting floating-point values (e.g., FP32) to integer values without decimal points (e.g., INT8).  
Since the arithmetic of floating-point values is more complex than that of integer values, quantization helps save computing costs for the processor, reduce power consumption, and minimize memory usage. Therefore, we applied the quantization approach to the checkpoint operations to benefit the ML model; we call it the _Check-Q_ operation in this paper (denoted by ). This approach can reduce the amount of data issued to the storage media during each checkpoint operation, resulting in saved time and resources for storage I/O operations, and it can help avoid the I/O conflicts on the storage stacks. For quantization, we utilized the following formula:  
\[scale=\frac{x_{max}-x_{min}}{2^{N}-1} \tag{1}\]  
In Equation (1), _scale_ calculates the quantization range based on the quantization target \(x\). \(x_{max}\) and \(x_{min}\) are quantization targets (i.e., maximum and minimum values of model parameters). _scale_ is quantized to the size of \(N\) bits. \(x_{q}\) is a quantized model parameter. It becomes an element normalized to the _scale_ range for the quantization target \(x\). It is expressed in Equation (2):  
\[x_{q}=round\left(\frac{x-x_{min}}{scale}\right) \tag{2}\]  
Fortunately, the quantized data offer an opportunity to reduce the amount of data by an order of magnitude through data compression. In other words, after performing Check-Q operations, the data are generally encoded as integers in the form of bits without decimal points; thus, there is a high possibility of duplicated neighbors in the quantized data. The compression rate and performance highly rely on the duplication of the data. Therefore, to further reduce the amount of data, Check-QZP compresses the quantized data via the Check-Q operation once again; we call this operation _Check-Z_ in this paper (denoted by ). To implement the Check-Q operation, we utilized the gzip [47] library, which is widely used for the compression and decompression of data.  
Figure 2: An overview of Check-QZP.  
In summary, Check-QZP improves the overall performance of the training step by performing Check-Q and Check-Z operations and prevents the waste of storage space. Of course, these operations have the side effect of CPU costs for performing the above formula and compression, as well as potentially negatively affecting accuracy. Fortunately, any degradation in accuracy can be recovered by using fine-tuning approaches [44; 45; 46]. We also believe that the operations of Check-QZP can reduce the cost of I/O operations passing through the storage stack and increase the available storage space. This has a significant impact on I/O performance in multi-tenant cloud environments. Thankfully, the impact of the Check-Q and Check-Z operations is almost zero during normal execution because system crashes and power failures rarely occur in real-world ML scenarios.