## 1 Introduction
As a result, object signatures necessary to verify data integrity, are also computed in a non-sequential manner. Hence, complex sorting algorithms are necessary to generate file-level signatures, which has a significant impact on the total computational and memory requirements [17]. Table 1 summarizes the key differences between sequential and object based big data transfer systems. To address the aforementioned challenges with object-based big data transfer systems (OBDTS), researchers proposed a TPBF (two-phase Bloom filter)-based [17] framework for integrity verification. TPBF leverages the space efficiency and insertion order independence characteristics of the Bloom filters to facilitate efficient data integrity verification. Although TPBF successfully reduces the computational, storage, and memory overhead typically associated with data integrity verification, it is still susceptible to false-positive errors. These errors can lead to the possibility of incorrectly assuming that specific files or objects have been successfully transferred to the destination. This can result in data corruption, which makes the data unsuitable for subsequent processing. To overcome this challenge, this paper proposes an alternative framework called the cross-referencing Bloom filter (CRBF) \begin{table}
\begin{tabular}{l l l} \hline \hline
**Feature** & **Sequential Transfer** & **Object-Based Transfer** \\ \hline Data transfer order & Sequential & Out of order \\ Data integrity verification & Simple checksum aggregation & Complex sorting algorithms \\ Memory requirements & Low & High \\ Computational requirements & Low & High \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Differences between sequential and object-based data transfer. based data integrity verification framework. The primary objective of this framework is to effectively handle the data integrity requirements of OBDTS by eliminating false-positive errors. The following are the key findings and/or contributions of this work:  
* The proposed framework employs object-, file-, and dataset-level integrity verification, minimizing the data to be retransmitted due to integrity errors. * We analyzed the memory and false-positive overhead of the proposed framework. We concluded that the proposed framework is effective in eliminating false-positive errors while maintaining a similar memory footprint to the TPBF-based integrity framework. * In order to evaluate the efficacy of the proposed framework in detecting and recovering from integrity errors, we simulated integrity errors at the object, file, and dataset levels. The experimental results showed that the proposed framework is very accurate at detecting and re-transmitting erroneous objects, files, and datasets. * We simulated faults at different points in the data transmission process to analyze the recovery overhead and false-positive rate of the proposed framework. Faults were simulated after transmitting 20%, 40%, 60%, and 80% of the total data. The experimental findings demonstrated negligible recovery time and zero false positives for all workloads at all fault points. The structure of the remaining sections in this paper is as follows: In Section 2, background information on data integrity verification and the motivation for this research is provided. Section 3 explores related work pertaining to data integrity verification. Section 4 introduces the Data- and Layout-Aware Bloom filter (DLBF). The design and implementation details of the proposed end-to-end data integrity verification framework are presented in Section 5. The evaluation results of the framework are discussed in Section 6, and Section 7 concludes the work.