Package: fastcdc
FastCDCThis package implements the "FastCDC" content defined chunking algorithm in
Python with optional cython support. To learn more about content
defined chunking and its applications, see the reference material linked below.RequirementsPythonVersion 3.7 and later. Tested on Linux, Mac and
WindowsInstalling$pipinstallfastcdcTo enable add additional support for the hash algorithms
(xxhashandblake3) use$pipinstallfastcdc[hashes]UsageCalculate chunks with default settings:$fastcdctests/SekienAkashita.jpghash=103159aa68bb1ea98f64248c647b8fe9a303365d80cb63974a73bba8bc3167d7offset=0size=22366hash=3f2b58dc77982e763e75db76c4205aaab4e18ff8929e298ca5c58500fee5530doffset=22366size=10491hash=fcfb2f49ccb2640887a74fad1fb8a32368b5461a9dccc28f29ddb896b489b913offset=32857size=14094hash=bd1198535cdb87c5571378db08b6e886daf810873f5d77000a54795409464138offset=46951size=18696hash=d6347a2e5bf586d42f2d80559d4f4a2bf160dce8f77eede023ad2314856f3086offset=65647size=43819Customize min-size, avg-size, max-size, and hash function$fastcdc-mi16384-s32768-ma65536-hfsha256tests/SekienAkashita.jpghash=5a80871bad4588c7278d39707fe68b8b174b1aa54c59169d3c2c72f1e16ef46doffset=0size=32857hash=13f6a4c6d42df2b76c138c13e86e1379c203445055c2b5f043a5f6c291fa520doffset=32857size=16408hash=0fe7305ba21a5a5ca9f89962c5a6f3e29cd3e2b36f00e565858e0012e5f8df36offset=49265size=60201Scan files in directory and report duplication.$fastcdcscan~/Downloads[####################################]  100%Files:1,332
ChunkSizes:min4096-avg16384-max131072UniqueChunks:506,077
TotalData:9.3GB
DupeData:873.8MB
DeDupeRatio:9.36%
Throughput:135.2MB/sShow help$fastcdc
Usage:fastcdc[OPTIONS]COMMAND[ARGS]...

Options:--versionShowtheversionandexit.--helpShowthismessageandexit.

Commands:chunkify*FindvariablesizedchunksforFILEandcomputehashes.benchmarkBenchmarkchunkingperformance.scanScanfilesindirectoryandreportduplication.Use from your python codeThe  tests also have some short examples of using the chunker, of which this
code snippet is an example:fromfastcdcimportfastcdcresults=list(fastcdc("tests/SekienAkashita.jpg",16384,32768,65536))assertlen(results)==3assertresults[0].offset==0assertresults[0].length==32857assertresults[1].offset==32857assertresults[1].length==16408assertresults[2].offset==49265assertresults[2].length==60201Reference MaterialThe algorithm is as described in "FastCDC: a Fast and Efficient Content-Defined
Chunking Approach for Data Deduplication"; see thepaper,
andpresentationfor details. There are some minor differences, as described below.Differences with the FastCDC paperThe explanation below is copied fromronomon/deduplicationsince this
codebase is little more than a translation of that implementation:The following optimizations and variations on FastCDC are involved in the chunking algorithm:31 bit integers to avoid 64 bit integers for the sake of the Javascript reference implementation.A right shift instead of a left shift to remove the need for an additional modulus operator, which would otherwise have been necessary to prevent overflow.Masks are no longer zero-padded since a right shift is used instead of a left shift.A more adaptive threshold based on a combination of average and minimum chunk size (rather than just average chunk size) to decide the pivot point at which to switch masks. A larger minimum chunk size now switches from the strict mask to the eager mask earlier.Masks use 1 bit of chunk size normalization instead of 2 bits of chunk size normalization.The primary objective of this codebase was to have a Python implementation with a
permissive license, which could be used for new projects, without concern for
data parity with existing implementations.Prior ArtThis package started as Python port of the implementation by Nathan Fiedler (see the
nlfiedler link below).nlfiedler/fastcdc-rsRust implementation on which this code is based.ronomon/deduplicationC++ and JavaScript implementation on which the rust implementation is based.rdedup_cdc at docs.rsAn alternative implementation of FastCDC to the one in this crate.jrobhoward/quickcdcSimilar but slightly earlier algorithm by some of the same researchers.Change Log[1.5.0] - 2023-03-13added python 3.10/3.11 supportremoved python 3.6 supportupdate dependencies[1.4.2] - 2020-11-25add binary releases to PyPI (Xie Yanbo)update dependencies[1.4.1] - 2020-09-30fix issue with fat option in cython versionupdated dependencies[1.4.0] - 2020-08-08add support for multiple path with scan commandfix issue with building cython extensionfix issue with fat optionfix zero-devision error[1.3.0] - 2020-06-26add newscancommand to calculate deduplication ratio for directories[1.2.0] - 2020-05-23Addedfaster optional cython implementationbenchmark command[1.1.0] - 2020-05-09Addedhigh-level APIsupport for streamssupport for custom hash functions[1.0.0] - 2020-05-07AddedInitial release (port ofnlfiedler/fastcdc-rs).