Package: fastchunking
What it isfastchunkingis a Python library that contains efficient and easy-to-use
implementations of string chunking algorithms.It has been developed as part of the work[LS16]at CISPA, Saarland University.Installation$ pip install fastchunkingNoteFor performance reasons, parts of this library are implemented in C++.
Installation from a source distribution, thus, requires availability of a
correctly configured C++ compiler.Usage and Overviewfastchunkingprovides efficient implementations for different string chunking
algorithms, e.g., static chunking (SC) and content-defined chunking (CDC).Static Chunking (SC)Static chunking splits a message into fixed-size chunks.Let us consider a random example message that shall be chunked:>>> import os
>>> message = os.urandom(1024*1024)Static chunking is trivial when chunking a single message:>>> import fastchunking
>>> sc = fastchunking.SC()
>>> chunker = sc.create_chunker(chunk_size=4096)
>>> chunker.next_chunk_boundaries(message)
[4096, 8192, 12288, ...]A large message can also be chunked in fragments, though:>>> chunker = sc.create_chunker(chunk_size=4096)
>>> chunker.next_chunk_boundaries(message[:10240])
[4096, 8192]
>>> chunker.next_chunk_boundaries(message[10240:])
[2048, 6144, 10240, ...]Content-Defined Chunking (CDC)fastchunkingsupports content-defined chunking, i.e., chunking of messages
into fragments of variable lengths.Currently, a chunking strategy based on Rabin-Karp rolling hashes is supported.As a rolling hash computation on plain-Python strings is incredibly slow with
any interpreter, most of the computation is performed by a C++ extension which
is based on thengramhashinglibrary by Daniel Lemire, see:https://github.com/lemire/rollinghashcppLet us consider a random message that should be chunked:>>> import os
>>> message = os.urandom(1024*1024)When using static chunking, we have to specify a rolling hash window size (here:
48 bytes) and an optional seed value that affects the pseudo-random distribution
of the generated chunk boundaries.Despite that, usage is similar to static chunking:>>> import fastchunking
>>> cdc = fastchunking.RabinKarpCDC(window_size=48, seed=0)
>>> chunker = cdc.create_chunker(chunk_size=4096)
>>> chunker.next_chunk_boundaries(message)
[7475, 10451, 12253, 13880, 15329, 19808, ...]Chunking in fragments is straightforward:>>> chunker = cdc.create_chunker(chunk_size=4096)
>>> chunker.next_chunk_boundaries(message[:10240])
[7475]
>>> chunker.next_chunk_boundaries(message[10240:])
[211, 2013, 3640, 5089, 9568, ...]Multi-Level Chunking (ML-*)Multiple chunkers of the same type (but with different chunk sizes) can be
efficiently used in parallel, e.g., to perform multi-level chunking[LS16].Again, let us consider a random message that should be chunked:>>> import os
>>> message = os.urandom(1024*1024)Usage of multi-level-chunking, e.g., ML-CDC, is easy:>>> import fastchunking
>>> cdc = fastchunking.RabinKarpCDC(window_size=48, seed=0)
>>> chunk_sizes = [1024, 2048, 4096]
>>> chunker = cdc.create_multilevel_chunker(chunk_sizes)
>>> chunker.next_chunk_boundaries_with_levels(message)
[(1049, 2), (1511, 1), (1893, 2), (2880, 1), (2886, 0),
(3701, 0), (4617, 0), (5809, 2), (5843, 0), ...]The second value in each tuple indicates the highest chunk size that leads to
a boundary. Here, the first boundary is a boundary created by the chunker with
index 2, i.e., the chunker with 4096 bytes target chunk size.NoteOnly the highest index is output if multiple chunkers yield the same
boundary.WarningChunk sizes have to be passed in correct order, i.e., from lowest to highest
value.PerformanceComputation costs forstatic chunkingare barely measurable: As chunking does
not depend on the actual message but only its length, computation costs are
essentially limited to a singlexrangecall.Content-defined chunking, however, is expensive: The algorithm has to compute
hash values for rolling hash window contents ateverybyte position of the
message that is to be chunked. To minimize costs, fastchunking works as follows:The message (fragment) is passed in its entirety to the C++ extension.Chunking is performed within the C++ extension.The resulting list of chunk boundaries is communicated back to Python and
converted into a Python list.Based on a 100 MiB random content, the author measured the following throughput
on an Intel Core i7-4770K in a single, non-representative test run using
Python 3.5 (Windows x86-64):chunk sizethroughput64 bytes118 MiB/s128 bytes153 MiB/s256 bytes187 MiB/s512 bytes206 MiB/s1024 bytes221 MiB/s2048 bytes226 MiB/s4096 bytes231 MiB/s8192 bytes234 MiB/s16384 bytes233 MiB/s32768 bytes234 MiB/sTestingfastchunkinguses tox for testing, so simply run:$ toxReferences:[LS16](1,2)Dominik Leibenger and Christoph Sorge (2016). sec-cs: Getting the
Most out of Untrusted Cloud Storage.arXiv:1606.03368