Package: repro-zipfile
repro-zipfileA tiny, zero-dependency replacement for Python'szipfile.ZipFilelibrary for creating reproducible/deterministic ZIP archives."Reproducible" or "deterministic" in this context means that the binary content of the ZIP archive is identical if you add files with identical binary content in the same order. It means you can reliably check equality of the contents of two ZIP archives by simply comparing checksums of the archive using a hash function like MD5 or SHA-256.This Python package provides aReproducibleZipFileclass that works exactly likezipfile.ZipFilefrom the Python standard library, except that certain file metadata are set to fixed values. See"How does repro-zipfile work?"below for details.You can also optionally install a command-line program,rpzip. See"rpzip command line program"below for more information.Installationrepro-zipfile is available from PyPI. To install, run:pipinstallrepro-zipfileIt is also available from conda-forge. To install, run:condainstallrepro-zipfile-cconda-forgeUsageSimply importReproducibleZipFileand use it in the same way you would usezipfile.ZipFilefrom the Python standard library.fromrepro_zipfileimportReproducibleZipFilewithReproducibleZipFile("archive.zip","w")aszp:# Use write to add a file to the archivezp.write("examples/data.txt",arcname="data.txt")# Or writestr to write data to the archivezp.writestr("lore.txt",data="goodbye")Note that files must be written to the archive in the same order to reproduce an identical archive. Be aware that functions that likeos.listdir,os.glob,Path.iterdir, andPath.globreturn files in a nondeterministic order—you should callsortedon their returned values first.Seeexamples/usage.pyfor an example script that you can run, andexamples/demo_vs_zipfile.pyfor a demonstration in contrast with the standard library's zipfile module.For more advanced usage, such as customizing the fixed metadata values, see the subsections under"How does repro-zipfile work?".rpzip command-line programYou can optionally install a lightweight command-line program,rpzip. This includes an additional dependency on thetyperCLI framework. You can install it either directly or using thecliextra with repro-zipfile:pipinstallrpzip# orpipinstallrepro-zipfile[cli]rpzip is designed to a partial drop-in replacement ubiquitouszipprogram. Userpzip --helpto see the documentation. Here are some usage examples:# Archive a single filerpziparchive.zipexamples/data.txt# Archive multiple filesrpziparchive.zipexamples/data.txtREADME.md# Archive multiple files with a shell globrpziparchive.zipexamples/*.py# Archive a directory recursivelyrpzip-rarchive.zipexamplesIn addition to the fixed file metadata done by repro-zipfile, rpzip will also always sort all paths being written.How does repro-zipfile work?ZIP archives are not normally reproducible even when containing files with identical content because of file metadata. In particular, the usual culprits are:Last-modified timestampsFile-system permissions (mode)repro_zipfile.ReproducibleZipFileis a subclass ofzipfile.ZipFilethat overrides thewrite,writestr, andmkdirmethods with versions that set the above metadata to fixed values. Note that repro-zipfile does not modify the original files—only the metadata written to the archive.You can effectively reproduce whatReproducibleZipFiledoes with something like this:fromzipfileimportZipFilewithZipFile("archive.zip","w")aszp:# Use write to add a file to the archivezp.write("examples/data.txt",arcname="data.txt")zinfo=zp.getinfo("data.txt")zinfo.date_time=(1980,1,1,0,0,0)zinfo.external_attr=0o644<<16# Or writestr to write data to the archivezp.writestr("lore.txt",data="goodbye")zinfo=zp.getinfo("lore.txt")zinfo.date_time=(1980,1,1,0,0,0)zinfo.external_attr=0o644<<16It's not hard to do, but we believeReproducibleZipFileis sufficiently more convenient to justify a small package!See the next two sections for more details about the replacement metadata values and how to customize them.Last-modified timestampsZIP archives store the last-modified timestamps of files and directories.ReproducibleZipFilewill set this to a fixed value. By default, the fixed value is 1980-01-01 00:00 UTC, which is the earliest timestamp that is supported by the ZIP format specifications.You can customize this value with theSOURCE_DATE_EPOCHenvironment variable. If set, it will be used as the fixed value instead. This should be an integer corresponding to theUnix epoch timeof the timestamp you want to set, e.g.,1704067230for 2024-01-01 00:00:00 UTC.SOURCE_DATE_EPOCHis astandardcreated by theReproducible Builds projectfor software distributions.File-system permissionsZIP archives store the file-system permissions of files and directories. The default permissions set for new files or directories often can be different across different systems or users without any intentional choices being made. (These default permissions are controlled by something calledumask.)ReproducibleZipFilewill set these to fixed values. By default, the fixed values are0o644(rw-r--r--) for files and0o755(rwxr-xr-x) for directories, which matches the common defaultumaskof0o022for root users on Unix systems. (The0oprefixis how you can write an octal—i.e., base 8—integer literal in Python.)You can customize these values using the environment variablesREPRO_ZIPFILE_FILE_MODEandREPRO_ZIPFILE_DIR_MODE. They should be in three-digit octalUnix numeric notation, e.g.,644forrw-r--r--.Why care about reproducible ZIP archives?ZIP archives are often useful when dealing with a set of multiple files, especially if the files are large and can be compressed. Creating reproducible ZIP archives is often useful for:Building a software package.This is a development best practice to make it easier to verify distributed software packages. See theReproducible Builds projectfor more explanation.Working with data.Verify that your data pipeline produced the same outputs, and avoid further reprocessing of identical data.Packaging machine learning model artifacts.Manage model artifact packages more effectively by knowing when they contain identical models.Related Tools and Alternativeshttps://diffoscope.org/Can do a rich comparison of archive files and show what specifically differshttps://github.com/timo-reymann/deterministic-zipCommand-line program written in Go that matches zip's interface but strips nondeterministic metadata when adding fileshttps://github.com/bboe/deterministic_zipCommand-line program written in Python that creates deterministic ZIP archiveshttps://salsa.debian.org/reproducible-builds/strip-nondeterminismPerl library for removing nondeterministic metadata from file archiveshttps://github.com/Code0x58/python-stripzipPython command-line program that removes file metadata from existing ZIP archives