Package: zipstream-ng
zipstream-ngA modern and easy to use streamable zip file generator. It can package and stream many files and
folders into a zip on the fly without needing temporary files or excessive memory. It can also
calculate the final size of the zip file before streaming it.Features:Generates zip data on the fly as it's requested.Can calculate the total size of the resulting zip file before generation even begins.Low memory usage: Since the zip is generated as it's requested, very little has to be kept in
memory (peak usage of less than 20MB is typical, even for TBs of files).Flexible API: Typical use cases are simple, complicated ones are possible.Supports zipping data from files, bytes, strings, and any other iterable objects.Keeps track of the date of the most recently modified file added to the zip file.Threadsafe: Won't mangle data if multiple threads concurrently add data to the same stream.Includes a clone of Python'shttp.servermodule with zip support added. Trypython -m zipstream.server.Automatically uses Zip64 extensions, but only if they are required.No external dependencies.Ideal for web backends:Generating zip data on the fly requires very little memory, no disk usage, and starts producing
data with less latency than creating the entire zip up-front. This means faster responses, no
temporary files, and very low memory usage.The ability to calculate the total size of the stream before any data is actually generated
(provided no compression is used) means web backends can provide aContent-Lengthheader in
their responses. This allows clients to show a progress bar as the stream is transferred.By keeping track of the date of the most recently modified file added to the zip, web
backends can provide aLast-Modifiedheader. This allows clients to check if they have the most
up-to-date version of the zip with just a HEAD request instead of having to download the entire
thing.Installationpip install zipstream-ngExamplesCreate a local zip file (simple example)Make an archive namedfiles.zipin the current directory that contains all files under/path/to/files.fromzipstreamimportZipStreamzs=ZipStream.from_path("/path/to/files/")withopen("files.zip","wb")asf:f.writelines(zs)Create a local zip file (demos more of the API)fromzipstreamimportZipStream,ZIP_DEFLATED# Create a ZipStream that uses the maximum level of Deflate compression.zs=ZipStream(compress_type=ZIP_DEFLATED,compress_level=9)# Set the zip file's comment.zs.comment="Contains compressed important files"# Add all the files under a path.# Will add all files under a top-level folder called "files" in the zip.zs.add_path("/path/to/files/")# Add another file (will be added as "data.txt" in the zip file).zs.add_path("/path/to/file.txt","data.txt")# Add some random data from an iterable.# This generator will only be run when the stream is generated.defrandom_data():importrandomfor_inrange(10):yieldrandom.randbytes(1024)zs.add(random_data(),"random.bin")# Add a file containing some static text.# Will automatically be encoded to bytes before being added (uses utf-8).zs.add("This is some text","README.txt")# Write out the zip file as it's being generated.# At this point the data in the files will be read in and the generator# will be iterated over.withopen("files.zip","wb")asf:f.writelines(zs)zipserver (included)A fully-functional and useful example can be found in the includedzipstream.servermodule. It's a clone of Python's built inhttp.serverwith the added ability to serve multiple files and folders as a single zip file. Try it out by
installing the package and runningzipserver --helporpython -m zipstream.server --help.Integration with a Flask webappA very basicFlask-based file server that streams all the
files under the requested path to the client as a zip file. It provides the total size of the stream
in theContent-Lengthheader so the client can show a progress bar as the stream is downloaded. It
also provides aLast-Modifiedheader so the client can check if it already has the most recent
copy of the zipped data with aHEADrequest instead of having to download the file and check.Note that while this example works, it's not a good idea to deploy it as-is due to the lack of input
validation and other checks.importos.pathfromflaskimportFlask,ResponsefromzipstreamimportZipStreamapp=Flask(__name__)@app.route("/",defaults={"path":"."})@app.route("/<path:path>")defstream_zip(path):name=os.path.basename(os.path.abspath(path))zs=ZipStream.from_path(path)returnResponse(zs,mimetype="application/zip",headers={"Content-Disposition":f"attachment; filename={name}.zip","Content-Length":len(zs),"Last-Modified":zs.last_modified,})if__name__=="__main__":app.run(host="0.0.0.0",port=5000)Partial generation and last-minute file additionsIt's possible to generate the zip stream, but stop before finalizing it. This enables adding
something like a file manifest or compression log after all the files have been added.ZipStreamprovides ainfo_listmethod that returns information on all the files added to the
stream. In this example, all that information will be added to the zip in a file named
"manifest.json" before finalizing it.fromzipstreamimportZipStreamimportjsondefgen_zipfile()zs=ZipStream.from_path("/path/to/files")yield fromzs.all_files()zs.add(json.dumps(zs.info_list(),indent=2),"manifest.json")yield fromzs.finalize()Comparison to stdlibSince Python 3.6 it has actually been possible to generate zip files as a stream using just the
standard library, it just hasn't been very ergonomic or efficient. Consider the typical use case of
zipping up a directory of files while streaming it over a network connection:(note that the size of the stream is not pre-calculated in this case as this would make the stdlib
example way too long).Using ZipStream:fromzipstreamimportZipStreamsend_stream(ZipStream.from_path("/path/to/files/"))The same(ish) functionality using just the stdlib:importosimportiofromzipfileimportZipFile,ZipInfoclassStream(io.RawIOBase):"""An unseekable stream for the ZipFile to write to"""def__init__(self):self._buffer=bytearray()self._closed=Falsedefclose(self):self._closed=Truedefwrite(self,b):ifself._closed:raiseValueError("Can't write to a closed stream")self._buffer+=breturnlen(b)defreadall(self):chunk=bytes(self._buffer)self._buffer.clear()returnchunkdefiter_files(path):fordirpath,_,filesinos.walk(path,followlinks=True):ifnotfiles:yielddirpath# Preserve empty directoriesforfinfiles:yieldos.path.join(dirpath,f)defread_file(path):withopen(path,"rb")asfp:whileTrue:buf=fp.read(1024*64)ifnotbuf:breakyieldbufdefgenerate_zipstream(path):stream=Stream()withZipFile(stream,mode="w")aszf:toplevel=os.path.basename(os.path.normpath(path))forfiniter_files(path):# Use the basename of the path to set the arcnamearcname=os.path.join(toplevel,os.path.relpath(f,path))zinfo=ZipInfo.from_file(f,arcname)# Write data to the zip file then yield the stream contentwithzf.open(zinfo,mode="w")asfp:ifzinfo.is_dir():continueforbufinread_file(f):fp.write(buf)yieldstream.readall()yieldstream.readall()send_stream(generate_zipstream("/path/to/files/"))TestsThis package contains extensive tests. To run them, installpytest(pip install pytest) and runpy.testin the project directory.LicenseLicensed under theGNU LGPLv3.