## 5 Cross-Referencing Bloom-Filter-Based Data Integrity Framework
### System Architecture  
The system architecture of the CRBF-based data integrity verification framework for object-based big data transfer systems is presented in Figure 4. This framework consists of two core components: the _file handler_ and _object handler_. The _file handler_ is responsible for managing file-level activities such as scheduling the transfer of file objects and performing file level integrity verification. Conversely, the _object handler_ is responsible for transferring objects and verifying the integrity of object data. As depicted in Figure 4, the DLBF [28], dataset Bloom filter (DSBF), and metadata Bloom filter (MDBF) are the core data structures of the proposed framework. Figure 3: Data- and layout-aware Bloom filter. (**a**) DLBF insert operation. (**b**) DLBF query operation. * **Data- and Layout-Aware Bloom filter (DLBF):** This Bloom filter [28] data structure stores the successfully transferred object information of an active file. The number of active files and active DLBFs depends on the number of I/O threads. As described in Section 4, DLBF is a modified data structure designed to prevent false positives of the standard Bloom filter. It consists of two segments: the layout and the Bloom filter, as shown in Figure 5. After a successful object transfer and subsequent integrity verification, the object layout bit (located in the layout segment) and \(k\) hashed bit positions, computed based on the object signature, are set to 1 in the DLBF. However, if the object fails the integrity verification, the _file handler_ reschedules the object for retransfer. This process will be continued until all file objects have been successfully transferred without any errors. During data transfer, if there is a fault and if the data transfer is resumed from a fault point, the proposed data integrity framework retrieves successfully transferred objects by querying the DLBF. Owing to the layout-aware nature of the data structure, the DLBF produces no false positives. Furthermore, dynamic changes to the objects and files can be detected using this data structure. * **Dataset Bloom Filter (DSBF):** This Bloom filter data structure maintains information regarding successfully transferred files. Upon transferring all file objects, a file signature is computed using the DLBF. If the file signature at both ends of the data transfer is the same, file-level integrity verification succeeds, and the \(k\) hashed bit positions computed on the file signature are set to 1 in the DSBF. If the file-level integrity verification fails, then the _file handler_ schedules all objects of that file for retransfer. The process of transferring files is continued until all files in the dataset have been transferred successfully and without any errors. If a fault occurs and the transfer resumes from the fault point, the DSBF is used to retrieve successfully transferred files. However, because Bloom filters are probabilistic in nature, there is a chance of false positives. To efficiently detect the false-positive errors of the DSBF, metadata Bloom filters (MDBF) at both ends of the data transfer are used as cross-referencing Bloom filters. If both the cross-referencing Bloom filters produce positive results, then the positive result of DSBF is considered as positive. Figure 4: Cross-referencing Bloom-filter-based data integrity verification system. Conversely, if any of the cross-referencing Bloom filters produces a negative result, then the positive result of DSBF is considered as negative. * **Metadata Bloom Filter (MDBF):** This Bloom filter stores the metadata of files that have been successfully transferred. File metadata in the following format is used at both ends of the data transfer. [File_Name, File_Size, TotalObjects, CreateTime, ModifyTime] Upon successful file-level integrity verification, both ends of the data transfer update MDBF with the \(k\) hashed bit positions computed on the file metadata.