Package: repro-apk
reproducible-apk-toolsfix-compresslevel.py,fix-newlines.py,sort-apk.py,sort-baseline.py,diff-zip-meta.py,dump-arsc.py,dump-axml.py,dump-baseline.py,list-compresslevel.py,zipinfo.py.scripts to make android apks reproduciblereproducible-apk-toolsis a collection of scripts (available as subcommands of
therepro-apkcommand) to help make APKs reproducible (e.g. by changing line
endings from LF to CRLF), or find out why they are not (e.g. by comparing ZIP
file metadata, or dumpingbaseline.proffiles).fix-compresslevel.pyRecompress with different compression level.Specify which files to change by providing at least one fnmatch-style pattern,
e.g.'assets/foo/*.bar'.If two APKs have identical contents but some ZIP entries are compressed with a
different compression level, thus making the APKs not bit-by-bit identical, this
script may help.$fix-compresslevel.py--help
usage:fix-compresslevel.py[-h][-v]INPUT_APKOUTPUT_APKCOMPRESSLEVELPATTERN[PATTERN...][...]$apksigcopiercomparesigned.apk--unsignedunsigned.apk
DOESNOTVERIFY[...]$fix-compresslevel.pyunsigned.apkfixed.apk6assets/foo/bar.js
fixing'assets/foo/bar.js'...
$zipalign-f4fixed.apkfixed-aligned.apk
$apksigcopiercomparesigned.apk--unsignedfixed-aligned.apk&&echoOK
OKNB: this builds a new ZIP file, preserving most ZIP metadata (and recompressing
entries not matching the pattern using the same compression level as in the
original APK) but not everything: e.g. copying the existing local header extra
fields which contain padding for alignment is not supported by Python'sZipFile, which is whyzipalignis usually needed.fix-newlines.pyChange line endings from LF to CRLF (or vice versa w/--from-crlf).Specify which files to change by providing at least one fnmatch-style pattern,
e.g.'META-INF/services/*'.If the signed APK was built on Windows and has e.g.META-INF/services/files
with CRLF line endings whereas the unsigned APK was build on Linux/macOS and has
LF line endings, this script may help.$fix-newlines.py--help
usage:fix-newlines.py[-h][--from-crlf][--to-crlf][-v]INPUT_APKOUTPUT_APKPATTERN[PATTERN...][...]$apksigcopiercomparesigned.apk--unsignedunsigned.apk
DOESNOTVERIFY[...]$fix-newlines.pyunsigned.apkfixed.apk'META-INF/services/*'fixing'META-INF/services/foo'...
fixing'META-INF/services/bar'...
$zipalign-f4fixed.apkfixed-aligned.apk
$apksigcopiercomparesigned.apk--unsignedfixed-aligned.apk&&echoOK
OKNB: this builds a new ZIP file, preserving most ZIP metadata (and recompressing
using the same compression level) but not everything: e.g. copying the existing
local header extra fields which contain padding for alignment is not supported
by Python'sZipFile, which is whyzipalignis usually needed.sort-apk.pySort (and w/o--no-realignalso realign) the ZIP entries of an APK.If the ordering of the ZIP entries in an APK is not deterministic/reproducible,
this script may help.  You'll almost certainly need to use it for all builds
though, since it can only sort the APK, not recreate a different ordering that
is deterministic but not sorted; see also the alignment CAVEAT.$sort-apk.py--help
usage:sort-apk.py[-h][--no-realign][--no-force-align][--reset-lh-extra]INPUT_APKOUTPUT_APK[...]$unzip-lunsigned.apk
Archive:unsigned.apkLengthDateTimeName
----------------------------62017-05-1511:24lib/armeabi/fake.so16722009-01-0100:00AndroidManifest.xml8962009-01-0100:00resources.arsc15362009-01-0100:00classes.dex
----------------41104files
$sort-apk.pyunsigned.apksorted.apk
$unzip-lsorted.apk
Archive:sorted.apkLengthDateTimeName
----------------------------16722009-01-0100:00AndroidManifest.xml15362009-01-0100:00classes.dex62017-05-1511:24lib/armeabi/fake.so8962009-01-0100:00resources.arsc
----------------41104filesNB: this directly copies the (bytes of the) original ZIP entries from the
original file, thus preserving all ZIP metadata.CAVEAT: alignmentUnfortunately, the padding added to ZIP local header extra fields for alignment
makes it hard to make sorting deterministic: unless the original APK was not
aligned at all, the padding is often different when the APK entries had a
different order (and thus a different offset) before sorting.Because of this,sort-apkforcefully recreates the padding even if the entry
is already aligned (since that doesn't mean the padding is identical) to make
its output as deterministic as possible.  The downside is that it'll often add
"unnecessary" 8-byte padding to entries that didn't need alignment.You can disable this using--no-force-align, or skip realignment completely
using--no-realign.  If you're certain you don't need to keep the old values,
you can also choose to reset the local header extra fields to the values from
the central directory entries with--reset-lh-extra.If you use--reset-lh-extra, you'll probably want to combine it with either--no-force-align(which should prevent the "unnecessary" 8-byte padding) or--no-realign+zipalign(which uses smaller padding).NB: the alignment padding used bysort-apkis the same as that used byapksigner(a0xd935"Android ZIP Alignment Extra Field" which stores the
alignment itself plus zero padding and is thus always at least 6 bytes), whereaszipalignjust uses plain zero padding.sort-baseline.pySortbaseline.profm(extracted or inside an APK).$sort-baseline.py--help
usage:sort-baseline.py[-h][--apk]INPUT_PROF_OR_APKOUTPUT_PROF_OR_APK[...]$diff-qsa/baseline.profmb/baseline.profm
Filesa/baseline.profmandb/baseline.profmdiffer
$sort-baseline.pya/baseline.profma/baseline-sorted.profm
$sort-baseline.pyb/baseline.profmb/baseline-sorted.profm
$diff-qsa/baseline-sorted.profmb/baseline-sorted.profm
Filesa/baseline-sorted.profmandb/baseline-sorted.profmareidentical$sort-baseline.py--apkunsigned.apksorted-baseline.apk
$zipalign-f4sorted-baseline.apksorted-baseline-aligned.apkNB: does not support all file format versions yet.NB: with--apk, this builds a new ZIP file, preserving most ZIP metadata (and
recompressing using the same compression level) but not everything: e.g. copying
the existing local header extra fields which contain padding for alignment is
not supported by Python'sZipFile, which is whyzipalignis usually needed.scripts to dump info from apks and related file formatsdiff-zip-meta.pyDiff ZIP file metadata.NB: this will not compare thecontentsof the ZIP entries, only metadata and
other non-contents bytes; to compare the contents of ZIP/APK files, use e.g.diffoscope.This will show differences in filenames, central directory headers, local file
headers, data descriptors, entry sizes, etc.Additional tests include compression level (if it can be determined), CRC32
checksum of compressed data, and extra data before entries or the central
directory; you can skip these (relatively slow) tests using--no-additional.Some differences make the output quite verbose and/or are usually the result of
other differences; you can skip/ignore these using--no-lfh-extra,--no-offsets,--no-ordering.$diff-zip-meta.py--help
usage:diff-zip-meta.py[-h][--no-additional][--no-lfh-extra][--no-offsets][--no-ordering]ZIPFILE1ZIPFILE2
$diff-zip-meta.pya.apkb.apk
---a.apk
+++b.apk
entryfoo:
-compresslevel=6+compresslevel=9-compress_crc=0x9ed711dc
+compress_crc=0xd9776b0c
$diff-zip-meta.pya.apkc.apk--no-offsets--no-ordering
---a.apk
+++c.apk
entries(sortedbyfilename):
-filename=META-INF/CERT.RSA
-filename=META-INF/CERT.SF
-filename=META-INF/MANIFEST.MF
centraldirectory:data_before:
-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
entryfoo:
-compresslevel=6+compresslevel=9-compress_crc=0x9ed711dc
+compress_crc=0xd9776b0c
entryMETA-INF/com/android/build/gradle/app-metadata.properties:data_before(entry):
-504b030400000000000021082102000000000000000000000000000066000000000000000000
-0000000000000000000000000000000000000000000000000000000000000000000000000000-0000000000000000000000000000000000000000000000000000000000000000000000000000-000000000000000000000000000000000000NB: work in progress; output format may change.dump-arsc.pyDumpresources.arsc(extracted or inside an APK) usingaapt2.$dump-arsc.py--help
usage:dump-arsc.py[-h][--apk]ARSC_OR_APK[...]$dump-arsc.pyresources.arsc
BinaryAPK
Packagename=com.example.appid=7f[...]$dump-arsc.py--apksome.apk
BinaryAPK
Packagename=com.example.appid=7f[...]dump-axml.pyDump Android binary XML (extracted or inside an APK) usingaapt2.$dump-axml.py--help
usage:dump-axml.py[-h][--apkAPK]AXML[...]$dump-axml.pyfoo.xml
N:android=http://schemas.android.com/apk/res/android(line=17)E:selector(line=17)E:item(line=18)[...]$dump-axml.py--apksome.apkres/foo.xml
N:android=http://schemas.android.com/apk/res/android(line=17)E:selector(line=17)E:item(line=18)[...]dump-baseline.pyDumpbaseline.prof/baseline.profm(extracted or inside an APK).$dump-baseline.py--help
usage:dump-baseline.py[-h][--apk][-v]PROF_OR_APK[...]$dump-baseline.pybaseline.prof
profversion=010Pnum_dex_files=4[...]$dump-baseline.pybaseline.profm
profmversion=002num_dex_files=4[...]$dump-baseline.pysome.apkentry=assets/dexopt/baseline.prof
profversion=010Pnum_dex_files=4[...]entry=assets/dexopt/baseline.profm
profmversion=002num_dex_files=4[...]NB: does not support all file format versions yet.list-compresslevel.pyList ZIP entries with compression level.$list-compresslevel.py--help
usage:list-compresslevel.py[-h]APK[...]$list-compresslevel.pysome.apkfilename='AndroidManifest.xml'compresslevel=9|6filename='classes.dex'compresslevel=Nonefilename='resources.arsc'compresslevel=None[...]filename='META-INF/CERT.SF'compresslevel=9|6filename='META-INF/CERT.RSA'compresslevel=9|6|4filename='META-INF/MANIFEST.MF'compresslevel=9|6|4NB: the compression level is not actually stored anywhere in the ZIP file, and
is thus calculated by recompressing the data with different compression levels
and checking the CRC32 of the result against the CRC32 of the original
compressed data.zipinfo.pyList ZIP entries (likezipinfo).This implementation aims for compatibility with the default and-loutput
formats of Info-ZIP'szipinfo; the-eextended output format is unique to
this implementation.  Other formats and options are (currently) not supported.Neither is the full variety of ZIP formats and extensions supported, just the
most common ones (UNIX, FAT, NTFS).The-l/--longoption adds the compressed size before the compression type;-e/--extendeddoes the same, adds the CRC32 checksum before the filename as
well, uses a more standard date format, and treats filenames ending with a/as directories.$zipinfo.py--help
usage:zipinfo.py[-h][-e][-l]ZIPFILE[...]$zipinfo.py-esome.apk
Archive:some.apk
Zipfilesize:5612bytes,numberofentries:8drw----2.0fat0bX2defN2017-05-1511:25:1800000000META-INF/
-rw----2.0fat77bl76defN2017-05-1511:25:18b506b894META-INF/MANIFEST.MF
-rw----2.0fat1672bl630defN2009-01-0100:00:00615ef200AndroidManifest.xml
-rw----1.0fat1536b-1536stor2009-01-0100:00:009987d5d8classes.dex
-rw----2.0fat29bl6defN2017-05-1511:26:52ff801cd1temp.txt
-rw----1.0fat6b-6stor2017-05-1511:24:3231963516lib/armeabi/fake.so
-rw----1.0fat896b-896stor2009-01-0100:00:004fcab821resources.arsc
-rw----2.0fat20bl6defN2017-05-1511:28:40c9983e85temp2.txt8files,4236bytesuncompressed,3158bytescompressed:25.4%
$zipinfo.py-lsome.apk
Archive:some.apk
Zipfilesize:5612bytes,numberofentries:8-rw----2.0fat0bX2defN17-May-1511:25META-INF/
-rw----2.0fat77bl76defN17-May-1511:25META-INF/MANIFEST.MF
-rw----2.0fat1672bl630defN09-Jan-0100:00AndroidManifest.xml
-rw----1.0fat1536b-1536stor09-Jan-0100:00classes.dex
-rw----2.0fat29bl6defN17-May-1511:26temp.txt
-rw----1.0fat6b-6stor17-May-1511:24lib/armeabi/fake.so
-rw----1.0fat896b-896stor09-Jan-0100:00resources.arsc
-rw----2.0fat20bl6defN17-May-1511:28temp2.txt8files,4236bytesuncompressed,3158bytescompressed:25.4%The fields are: permissions, create version, create system, uncompressed size,
extra info, compressed size (w/--longor--extended), compression type,
date, time, CRC32 (w/--extended), filename.The extra info field consists of two characters: the first isbfor binary,tfor text (uppercase for encrypted files); the second isXfor data
descriptor and extra field,lfor just data descriptor,xfor just extra
field,-for neither.See also:zipinfo(1),zipdetails(1).helper scriptsinplace-fix.pyConvenience wrapper for some of the other scripts likefix-newlinesthat makes
them modify the file in-place (and optionallyzipalignit too).$inplace-fix.py--help
usage:inplace-fix.py[-h][--zipalign]COMMANDINPUT_FILE[...][...]$inplace-fix.py--zipalignfix-newlinesunsigned.apk'META-INF/services/*'[RUN]python3fix-newlines.pyunsigned.apk/tmp/.../fixed.apkMETA-INF/services/*
fixing'META-INF/services/foo'...
fixing'META-INF/services/bar'...[RUN]zipalign4/tmp/.../fixed.apk/tmp/.../aligned.apk[MOVE]/tmp/.../aligned.apktounsigned.apkIfzipalignis not found on$PATHbut any of$ANDROID_HOME,$ANDROID_SDK, or$ANDROID_SDK_ROOTis set to an Android SDK directory, it
will usezipalignfrom the latestbuild-toolssubdirectory of the Android
SDK.NB: however, it will skipbuild-tools31.0.0and32.0.0becausetheirzipalignis broken.NB: this script is not available as arepro-apksubcommand, but as a seperaterepro-apk-inplace-fixcommand.gradle integrationYou can e.g. sortbaseline.profmduring thegradlebuild by adding something
like this to yourbuild.gradle:// NB: assumes reproducible-apk-tools is a submodule in the app repo's
// root dir; adjust the path accordingly if it is found elsewhere
project.afterEvaluate {
    tasks.compileReleaseArtProfile.doLast {
        outputs.files.each { file ->
            if (file.toString().endsWith(".profm")) {
                exec {
                    commandLine(
                        "../reproducible-apk-tools/inplace-fix.py",
                        "sort-baseline", file
                    )
                }
            }
        }
    }
}Alternatively, adding something like this allows you to modify the APK itself
after building (and re-sign it if necessary):// NB: assumes reproducible-apk-tools is a submodule in the app repo's
// root dir; adjust the path accordingly if it is found elsewhere
android {
    applicationVariants.all { variant ->
        variant.outputs.each { output ->
            variant.packageApplicationProvider.get().doLast {
                exec {
                    // set ANDROID_HOME for zipalign
                    environment "ANDROID_HOME", android.sdkDirectory
                    commandLine(
                        "../reproducible-apk-tools/inplace-fix.py",
                        "--zipalign", "fix-newlines", output.outputFile,
                        "META-INF/services/*"
                    )
                }
                // re-sign w/ apksigner if needed
                if (variant.signingConfig != null) {
                    def tools = "${android.sdkDirectory}/build-tools/${android.buildToolsVersion}"
                    def sc = variant.signingConfig
                    exec {
                        environment "KS_PASS", sc.storePassword
                        environment "KEY_PASS", sc.keyPassword
                        commandLine(
                            "${tools}/apksigner", "sign", "-v",
                            "--ks", sc.storeFile,
                            "--ks-pass", "env:KS_PASS",
                            "--ks-key-alias", sc.keyAlias,
                            "--key-pass", "env:KEY_PASS",
                            output.outputFile
                        )
                    }
                }
            }
        }
    }
}CLINB: you can just use the scripts stand-alone; alternatively, you can install therepro-apkPython package and use them as subcommands ofrepro-apk:$repro-apkdiff-zip-metaa.apkb.apk
$repro-apkdiff-zip-metaa.apkc.apk--no-offsets--no-ordering
$repro-apkdump-arscresources.arsc
$repro-apkdump-arsc--apksome.apk
$repro-apkdump-axmlfoo.xml
$repro-apkdump-axml--apksome.apkres/foo.xml
$repro-apkdump-baselinebaseline.prof
$repro-apkdump-baselinebaseline.profm
$repro-apkdump-baseline--apksome.apk
$repro-apkfix-compresslevelunsigned.apkfixed.apk6assets/foo/bar.js
$repro-apkfix-newlinesunsigned.apkfixed.apk'META-INF/services/*'$repro-apklist-compresslevelsome.apk
$repro-apksort-apkunsigned.apksorted.apk
$repro-apksort-baselinebaseline.profmbaseline-sorted.profm
$repro-apksort-baseline--apkunsigned.apksorted-baseline.apk
$repro-apkzipinfo-esome.apk
$repro-apkzipinfo-lsome.apkHelp$repro-apk--help
$repro-apkdiff-zip-meta--help
$repro-apkdump-arsc--help
$repro-apkdump-axml--help
$repro-apkdump-baseline--help
$repro-apkfix-compresslevel--help
$repro-apkfix-newlines--help
$repro-apklist-compresslevel--help
$repro-apksort-apk--help
$repro-apksort-baseline--help
$repro-apkzipinfo--helpInstallingUsing pip$pipinstallrepro-apkNB: depending on your system you may need to use e.g.pip3 --userinstead of justpip.From gitNB: this installs the latest development version, not the latest
release.$gitclonehttps://github.com/obfusk/reproducible-apk-tools.git
$cdreproducible-apk-tools
$pipinstall-e.NB: you may need to add e.g.~/.local/binto your$PATHin order
to runrepro-apk.To update to the latest development version:$cdreproducible-apk-tools
$gitpull--rebaseDependenciesPython >= 3.8 + click (repro-apkpackage only, the stand-alone scripts have
no dependencies besides Python).Thedump-arsc.pyanddump-axml.pyscripts requireaapt2.Debian/Ubuntu$aptinstallpython3-click
$aptinstallaapt# for dump-arsc.py & dump-axml.py$aptinstallzipalign# for realignment; see examplesLicense