Package: fastzip
fastzipThis project lets you efficiently create and combine zip files.What is "fast" in "faszip"?Mainly two things:Multithreaded compression (DEFLATE), similar topigzThe ability to copy/merge zips without recompressing, similar tozipmergeIt's also pure python, able to stream the input (without needing the central
directory to be present), and features a rules-based compression method chooser,
to avoidincreasingthe size of tiny files.DemoSimilar API to ZipFilefrompathlibimportPathfromfastzip.writeimportWZipwithWZip(Path("out.zip"))asz:z.write(Path("a"))Merging existing filesfromfastzip.readimportRZipStreamfromfastzip.writeimportWZipwithWZip(Path("out.zip"))asz:forentryinRZipStream("zip1.zip").entries():z.enqueue_precompressed(*entry)forentryinRZipStream("zip2.zip").entries():z.enqueue_precompressed(*entry)If you'd like to customize the number of threads, choice of compression,
filename within the archive, etc those are all possible.  Right now look at the
source, but the basic gist is:fromioimportBytesIOfrompathlibimportPathfromfastzip.writeimportWZipfromfastzip.chooserimportCompressionChooserforce_store=CompressionChooser(default="store")withWZip(Path("out.zip"),threads=2,chooser=force_store)asz:z.write(Path("a"),archive_path=Path("inzipname"),synthetic_mtime=0,fobj=BytesIO(b"foo"))BenchmarkTested on 8- and 12-core xeons, it's able to use about 7.5 and 11 cores,
repsectively for sufficiently large workloads.  See the next section about bottlenecks.$ddif=/dev/urandomof=abs=1Mcount=512512+0recordsin512+0recordsout536870912bytes(537MB,512MiB)copied,2.17395s,247MB/s

$./demo.sh
+rm-fa.zipa1.zipa2.zip
+cata
+/usr/bin/timezip-oa.zipaadding:a(deflated0%)11.35user0.24system0:11.60elapsed99%CPU(0avgtext+0avgdata2892maxresident)k
24inputs+1048752outputs(1major+205minor)pagefaults0swaps
+/usr/bin/timepython-mzipfile-ca1.zipa12.61user0.44system0:14.53elapsed89%CPU(0avgtext+0avgdata11360maxresident)k
0inputs+1048920outputs(0major+1623minor)pagefaults0swaps
+/usr/bin/timepython-mfastzip-c-oa2.zipa20.39user0.56system0:02.78elapsed753%CPU(0avgtext+0avgdata1079140maxresident)k
0inputs+1048968outputs(0major+149864minor)pagefaults0swaps
+ls-la.zipa1.zipa2.zip
-rw-r--r--1timtim537034782Nov1613:47a1.zip
-rw-r--r--1timtim537037407Nov1613:47a2.zip
-rw-r--r--1timtim536957493Nov1613:01a.zipPerformance BottlenecksThere are potentially many.  Some that I've observed include:Slow overwrite performance (on ext4); this is why WZip only opens in 'xb' mode.Small files
a. Slow stat (~0.5mS even on a ramdisk)
b. zlib.crc32 doesn't release the gil < 5KiB
c. small files use up the file budget without making much forward progress
(reordering to have one large file every (file_budget-1) may reduce
stallsSmall output buffer (WZip opens with 1MB now)Seeks (which require flushing output buffer)ThreadPoolExecutor giant lock (submitting a future requires grabbing two
locks but one of them is a global lock to detect interpreter shutdown?)Deflate chunk size (smaller chunks means better parallelism on medium sized
files and less blocking on IO, but also scheduling many more futures, crc32
calls, and crc32_combine).  Larger chunk size also means a slightly better
compression ratio.Garbage collection (some recent changes to increase parallelism seem to
increase the quantity of GC)Mixing deflate and zstd -- although the number of threads is generally
shared between them, there are going to be stalls and overruns when
switching.When in doubt, generate a trace, and look for bottlenecks (either blank time,
which is not doing useful work, or basically anything besides a compress itself
that seems to take a long time.Sample TracesThis one only really performs well for the section in the middle marked
"acquire" -- that is where it's run out of some resource to be able to queue
more data, but also the compression threads are fully working.  The gold
seemingly-unlabeled events are GC, and you can see the cpu graph is generally
quite low.This is that middle section zoomed in, and you can see more clearly that the
compression threads are busy and the IO threads have some idle time.  The gaps
in MainThread imply that there is some hidden work.Future plans (from drawbacks)Near-term problems:Errors that happen in threads aren't always reported in the obvious way.Adding directories (rather than files) is serial.Files must be MMAP-able, and on a 32-bit system this will probably run out
of process address space.There is an open-file budget, but there is no formal memory budget; the
output futures are likely to buffer about 2x the size of the largest file,
in memory.  For example, the 512MB file above has a peak memory of about
1GB.WZipis too complicated and has many complicated methods.  This makes it
hard to refactor, and hard to extend.Not enough validation (e.g. duplicate filenames, filenames with ".." in
them), and not totally sure the archive name code is solid.NOTE: The API is expected to change several times before version 1.0; if adding
a dep edge to this project (e.g. if you use a hypothetical version0.4make
sure you specify>=0.4, <0.5.Licensefastzip is copyrightTim Hatch, and licensed under
the MIT license.  I am providing code in this repository to you under an open
source license.  This is my personal repository; the license you receive to
my code is from me and not from my employer. See theLICENSEfile for details.