Package: aiozipstream
AioZipStreamThis is a fork ofZipStream. Simple python library for streaming ZIP files which are created dynamically, without using any temporary files.No temporary files, data is streamed directlySupporteddeflatecompression methodSmall memory usage, straming is realised using yield statementArchive structure is created on the fly, and all data can be created during streamFiles included into archive can be generated on the fly using Python generatorsAsynchronous AioZipStream and classic ZipStream are availableZip32 format compatible filesIndependent from python's standard ZipFile implementationAlmost no dependencies: onlyaiofilesin some circumstances (see AioZipStream section for details)Zip64 support is also planned in future (far future, because I never hitted 4GB file size limit ;-) )Required Python version:ZipStreamis compatible withPython 2.7.AioZipStreamrequirePython 3.6. For earlier versionsAioZipStreamis not available for import.Usage:List of files to archive is stored as list of dicts. Why dicts? Because there are possible additional parameters for each file, and more parameters are planned in future.Sample list of files to archive:files=[# file /tmp/file.dat will be added to archive under `file.dat` name.{'file':'/tmp/file.dat'},# same file as previous under own name: `completly_different.foo`# and will be compressed using `deflate` compression method{'file':'/tmp/file.dat','name':'completly_different.foo','compression':'deflate'}]It's time to stream / archive:zs=ZipStream(files)withopen("example.zip","wb")asfout:fordatainzs.stream():fout.write(data)Any iterable source of binary data can be used in place of regular files. Using generator as input for file must be represented bystreamfield instead offile, additionalnameparameter is also required.defsource_of_bytes():yieldb"123456789"yieldb"abcdefgh"yieldb"I am a binary data"files=[....# file will be generated dynamically under name my_data.bin{'stream':source_of_bytes(),'name':'my_data.bin'},]Keep in mind, that data should be served in chunks of reasonable size, because in case of using stream,ZipStreamclass is not able to split data by self.List of files to stream can be also generated on the fly, during streaming:importosfromzipstreamimportZipStreamdeffiles_to_stream_with_foo_in_name(dirname):# all files from selected firectoryforfinos.listdir(dirname):fp=os.path.join(dirname,f)ifos.path.isfile(fp):yield{'file':fp,'name':"foo_"+os.path.basename(fp)}# and our generator tooyield{'stream':source_of_bytes(),'name':'my_data.bin','compression':'deflate'}zs=ZipStream(files_to_stream_with_foo_in_name('\tmp\some-files'))Asynchronous AioZipStream:warning:To use asynchronous AioZipStream at least Python 3.6 version is required. AioZipStream is using asynchronous generator syntax, wchich is avilable from 3.6 version.To work with local files addtionalaiofileslibrary is required. If You plan to stream only dynamically generated content, thenaiofilesis not required.Seeaiofiles github repofor details aboutaiofiles.Sample of asynchronous zip streamingAny generator used to create data on the fly, must be defined asasync:asyncdefcontent_generator():yieldb'foo baz'asyncio.sleep(0.1)# we simulate little slow source of datadata=awaitremote_data_source()yieldbytes(data,'utf-8')# always remember to yield binary dataasyncio.sleep(0.5)yieldb"the end"Also zip streaming must be insideasyncfunction. Note usageaiofiles.openinstead ofopen, which is asynchronous and will not block event loop during disk access.fromzipstreamimportAioZipStreamasyncdefzip_async(zipname,files):aiozip=AioZipStream(files,chunksize=32768)asyncwithaiofiles.open(zipname,mode='wb')asz:asyncforchunkinaiozip.stream():awaitz.write(chunk)Here is going list of files to send:files=[{'file':'/tmp/car.jpeg'},{'file':'/tmp/aaa.mp3','name':'music.mp3'},{'stream':content_generator(),'name':'random_stuff.txt'}]Start asyncio loop and stream result to file:loop=asyncio.get_event_loop()loop.run_until_complete(zip_async('example.zip',files))loop.stop()ExamplesSeeexamplesdirectory for complete code and working examples of ZipStream and AioZipStream.