## 3 Related Work
### Data Integrity  
The primary objective of big data transfer frameworks is to ensure the accuracy of transferred data. To accomplish this, extensive research has been conducted on the design and optimization of data integrity verification systems in different contexts, including storage systems [29; 30], cloud-based storage [31; 32], file systems [33; 34], and data transfer systems [35; 36; 37; 38].  
GridFTP [11; 12] and BBCP [13] are the two most popular and widely used data transfer tools in the big data community. These tools are file-based and transfer data sequentially. Simple checksum aggregation methods are commonly used to create file-level signatures and verify the integrity of data at the file level. This is possible because data transfer is a sequential process. However, this sequential, file-based method of verifying data integrity is not suitable for large datasets. If a dataset contains large files, the checksum computation may fail due to the size of the data, or it may take an extremely long time to complete. For this reason, a parallel checksum approach is necessary and preferable. Many researchers have proposed methods for transferring multiple files in parallel by exploiting pipelining and parallelism [35; 36; 37; 39] techniques. Unlike serial and file-based data transfer tools, this work considers transferring the workload as objects rather than files. Due to the object-based transfer and parallel processing, objects of the same logical file may be transferred out of order from the source to the destination. This out-of-order nature of the data transfer demands complex sorting and checksum aggregation methods to generate file- and dataset-level signatures used for data integrity verification.  
[PERSON] et al. [38] introduced _fSum_, a scalable parallel dataset checksumming tool designed specifically for long-term storage or archival. This method divides the workload into multiple chunks and distributes them to multiple processes, which calculate the chunk-level checksums in parallel. As a result of the parallel execution, the checksums are generated in a random order. Therefore, all the checksums must be sorted at the root process to generate a consistent dataset-level signature. However, this method is not scalable for large datasets as the root process must collect and store checksums of all processes in memory. This process requires a lot of memory and computation, which can become a bottleneck as the number of processes increases. A Bloom-filter-based data structure aggregates the chunk-level checksums to address these challenges. The insertion-order-independent nature of the Bloom filter data structure helps generate a single and consistent dataset-level signature without sorting. Therefore, this data integrity method for long-term storage or archival is more efficient in memory and computation than other approaches. However, the probabilistic nature of the Bloom filter can lead to false positives [18; 20; 40].  
The proposed data integrity framework is built based on the properties of space efficiency and insertion order independence of the Bloom filter. The proposed CRBF-based integrity verification framework complements the TPBF data integrity method in minimizing the memory and computational overhead while ensuring zero false-positive detections for the given workloads.