Design and Implementation of an Open-Source Deduplication Platform for

Data deduplication is a technique used to improve storage utilization by eliminating duplicate data. Duplicate data blocks are not stored and instead a reference to the original data block is updated. Unique data chunks are identified using techniques such as hashing, and an index of all the existing chunks is maintained. When new data blocks are written, they are hashed and compared to the hashes existing in the hash index. Various chunking methods exist, like fixed chunking or variable length chunking, which impact the effectiveness of identifying duplicates. Deduplication in storage systems can be done as a post-processing step, after the data has already been written to the storage device, or it can be performed in-line (i.e., before writing the data to the storage system). Deduplication can be implemented at various layers in the storage stack, such as application layer, file system layer, or block layer, each having their own trade-offs. In earlier deduplication studies, often researchers had to either implement their own deduplication platform, or had to use closed-source enterprise implementations. Due to this difficulty in finding an appropriate deduplication platform for research, we designed and developed our own deduplication engine and decided to open-source it. We also used our deduplication engine for further deduplication research. We present Dmdedup, a versatile and practical primary-storage deduplication platform suitable for both regular users and researchers. Dmdedup operates at the block layer, so it is usable with existing file systems and applications. Since most deduplication research focuses on metadata management, we designed and implemented a flexible backend API that lets developers easily build and evaluate various metadata management policies. We implemented and evaluated three backends: an in-RAM table, an on-disk table, and an on-disk COW B-tree. We have evaluated Dmdedup under a variety of workloads and report the evaluation results here. Although it was initially designed for research flexibility, Dmdedup is fully functional and can be used in production. Under many realworld workloads, Dmdedup’s throughput exceeds that of a raw block device by 1.5–6×. We have also submitted Dmdedup to the Linux community for inclusion in the Linux kernel mainline. Block-layer data deduplication allows file systems and applications to reap the benefits of deduplication without modifying each file system or application to support deduplication. However, important information about data context (e.g., data vs. metadata writes) is lost at the block layer. Passing such context to the block layer can help improve deduplication performance and reliability. We implemented a hinting interface to pass relevant context to the block layer, and evaluated two hints, NODEDUP and PREFETCH. To allow upper storage layers to pass hints based on the available context, we modified the VFS and file system layers to expose a hinting interface to user applications. Building on our open-source block-layer deduplication system, Dmdedup, we show that passing the NODEDUP hint can speed up applications by up to 5.3× on modern machines because the overhead of deduplication is avoided when it is unlikely to be beneficial. We also show that the PREFETCH hint can speed applications by up to 1.8× by caching hashes for data that is likely to be accessed soon.