Package: abarms
What isabarms?abarmsis ahandySwiss-army-knife-like tool/utility/console app for POSIX-compatible systems for manipulating Android Backup files (*.ab,*.adb) produced byadb backup,bmgr, and similar tools.abarmscan list contents, convert Android Backup files into TAR files and back (by decrypting, decompressing, and re-compressing said files), and split full-system dumps produced byadb backupinto per-app backups that can be given toadb restore.Basically, this is a simpler pure Python implementation (only requiressetuptoolsandcryptographymodules) ofandroid-backup-extractorand the parts ofandroid-backup-toolkitandandroid-backup-processorthat I use myself.Why doesabarmsexists?(TL;DR: read the parts in bold.)Did you know that your Android OS device already has an awesome built-in full-system phone-to-PC backup and PC-to-phone restore tool that does not require root access?adbutility of Android Platform Tools hasadb backupsubcommand that, in principle, can do basically everything you could possibly want there.Internally this is implemented via Android OS setuid root binary namedbu--- which you can run manually viaadb shell bu help--- that simply backs up every app on the device one by one and streams the resulting.abfile --- which is a wrapped PAX-formatted TAR file (see "EXTENDED DESCRIPTION" section inman 1 pax) --- to stdout.adb backupsubcommand is just a simple wrapper around it.But then Android Platform Tools bundle gives no tools to manipulate those backup files!So, if you make a full-system backup withadb backup, and then want to restore a single app out of 100+ you have installed on your device, you need third-party tools now.
This is kind of embarrassing, to be honest.
A tool to manipulate backup files should have been a standard utility in Android Platform Tools since Android version 0.1 or something.
(Seriously, are you not embarrassed? I'm embarrassed for the state of humanity thinking about how the most popular OS on the planet gives no widely accessible local backup and restore tools on par with what every user of 1970s-era UNIX mainframe had out of the box. I'm not asking for automatic opportunistic incremental quantum-safely encrypted full-system replication to cooperative nearby devices in a local mesh-network here!)Well, technically speaking, Android OS also has automatic scheduled non-interactive backup servicebmgr--- which can be controlled via Android settings menu andadb shell bmgr help, that does per-app backups and restores.
Internally,bmgrservice also generates.abfiles and then either uploads them to Google --- which is the default and the only option available through the settings menu --- or stores them locally under/data/data/com.android.localtransport/files/--- which requires root to access.
On old Android versions you could askbmgrto do a backup to an SD card directly from the settings menu, but Google removed that functionality to force users to use Cloud-based backups.So, basically, according to Google (and Samsung, which ship with their ownbmgr-like service in parallel withbmgr), to restore to a previous state of an app, or to migrate between phones you now apparently have to upload all your data to their servers in plain-text for their convenient data-mining and selling of your data to interested third parties.
Google even went as far as to hideadb backupsubcommand from their official Android documentation: compare theold manual foradbwith thecurrent one, Control+F for "backup".This resulted into every Android vendor now making their own vendor-specific phone-to-phone migration utilities, and a whole ecosystem of commercial apps that do whatadb backupalready does, but worse.This also resulted in usefulness ofadb backupitself being reduced because in Android version 6 Google made automatic daily file-based backups that get uploaded to Google the default when you attach your phone to your Google account.
So, most apps started to opt-out of those backups for privacy and security reasons -- which also started opting them out of being included inadb backupoutput, sincebmgrandbushare most of the infrastructure.
Some of those apps now implement their own in-app backup buttons hidden away in the settings menu somewhere, but most do not.Yes, this is stupid, seethis discussion on StackOverflow.
See also old Android developer docs that explained this fairly clearlyhereandhere.(You can also force an app to be included inadb backupby rebuilding its APK to enableandroid:allowBackupattribute in the manifest and installing the result manually, seethisfor more info.
But this will only work for newly installed apps as you will have to re-sign the resulting APK with your own private key and Android forbids app updates that change the signing key.)But, hopefully, eventually, some alternative firmware developer will fix the above bug and allowadb backupto backup all apps regardless ofandroid:allowBackupmanifest setting, as it should.Still,adb backupworks fine for a lot of apps and, hopefully, will eventually get back to working as well as it did before Android version 6 in the future.
Meanwhile,android-backup-toolkitallows you to split full-system dumps produced byadb backupinto per-app backups that can then be restored withadb restore.The problem is that, while I'm thankful thatandroid-backup-toolkitexists, I find it really annoying to use: it is a bundle of pre-compiled Java apps, binaries, and shell scripts that manages to work somehow, but modifying anything there is basically impossible as building all of those things from sources is an adventure I failed to complete, and then you need to install the gigantic Java VM and libraries to run it all.So, as it currently stands, to have per-app backups of your Android device you have to either:root your device;give up your privacy by uploading your backups to other people's computers (aka "the cloud"); orrepack all you APKs withandroid:allowBackup = trueand either run older Android firmware that can do backup to an SD card or runadb backupfrom your PC, and then extract per-app backups from its output (yes, this is not ideal, but it works, and does not need root).So, one day I was looking at all of this.
I couldn't root or change the firmware on a phone I wanted to keep backed up, but I could follow the last option and get most of what I wanted with almost no effort.
Except figuring out how to runandroid-backup-toolkitto do the very last step of this took me quite a while.
And so I thought, "Hmm, this seems overly complicated, something as simple as splitting and merging TAR files with some additional headers should be doable with a simple Python program."
So I made one.It turned out to be a bit less simple than I though it would be, mostly because Python'starfilemodule was not designed for this, so I had to make my own, and PAX-formatted TAR files are kind of ugly to parse, but it works now, so, eh.Hopefully,abarmsexisting will inspire more app and alternative firmware developers to supportadb backupproperly and so personal computing devices of late 2020s will finally reach feature parity with 1970s-era Tape ARchiving (TAR) backup technology.QuickstartInstallationInstall with:pip install abarmsand run asabarms --helpAlternatively, install it via Nixnix-env -i -f ./default.nixAlternatively, run without installing:python3 -m abarms --helpBackup all apps from your Android device, then restore a single app, without rootPrepare your PC and phoneBefore you make a full backup of your Android phone (or other device) you need toinstall Android Platform Tools (either fromthereor from you distribution),enable Developer Mode and USB Debugging (seeAndroid Docsfor instructions).then, usually, on your PC you need to runsudo adb kill-server
sudo adb start-serverunless, you added special UDev rules for your phone.Do a full backupTo do the backup, you need to unlock your phone, connect your it to your PC via a USB cable (in that order, otherwise USB Debugging will be disabled), confirm that the PC is allowed to do USB Debugging in the popup on the phone, then runadb backup -apk -obb -noshared -all -system -keyvalueon your PC, then (unlock your phone again and) press "Back up my data" button at the bottom of your screen.Now you need to wait awhile foradbto finish.
The result will be saved inbackup.abfile.If you want to backup to an explicitly named file, e.g. to note the date of the backup, runadb backup -f backup_20240101.ab -apk -obb -noshared -all -system -keyvalueinstead.Split it into piecesYou can view contents of the backup viaabarms ls backup_20240101.aband split it into per-app backups viaabarms split backup_20240101.abwhich will produce a bunch of files namedabarms_split_<filename>_<num>_<appname>.ab(e.g.abarms_split_backup_20240101_020_org.fdroid.fdroid.ab).Restore a single appA single per-app file can be fed back toadb restoreto restore that singe app, e.g.adb restore abarms_split_backup_20240101_020_org.fdroid.fdroid.abRebuild full backup from partsYou can also rebuild the original full-backup from parts viaabarms merge abarms_split_backup_20240101_*.ab backup_20240101.rebuilt.abto check that it produces exactly the same backup file# strip encryption and compression from the original
abarms strip backup_20240101.ab backup_20240101.stripped.ab

# compare to the stipped original and the rebuilt file
diff backup_20240101.stripped.ab backup_20240101.rebuilt.ab || echo differAlternativesandroid-backup-toolkitand friendsandroid-backup-extractoris a Java app that can decrypt and decompress Android Backup archives and convert them into TAR.android-backup-toolkitbuilds on top ofandroid-backup-extractorand provides a way to split full-system backup ADB files into per-app pieces.android-backup-processoris an older version ofandroid-backup-toolkit.If you have root on your deviceAssuming you have root on your Android phone, you can do# check if bmgr is enabled
adb shell bmgr enabled

# list bmgr transports
adb shell bmgr list transports
# localtransport should be there, enable it
adb shell bmgr transport com.android.localtransport/.LocalTransport

# enable bmgr
adb shell bmgr enable true

# do a full backup now
adb shell bmgr fullbackupand then take per-app backup files from/data/data/com.android.localtransport/files/.QuirksThe precise algorithm for how encrypted Android Backup files get their master key salted checksums computed remains a mystery to me even after reading all the related Android sources.Luckily, those checksums verify that the given passphrase is correct and can be ignored while reading.abfiles since the following encrypted Android Backup headers are verbose enough that a wrong passphrase will break parsing anyway.
None of my use cases ever need encrypted.abfiles and no firmware I know of requiresadb restoreinputs to be encrypted.So, after spending two days trying to figure those checksums out I decided thatabarmsdoes not support generating encrypted.abfiles by design.
(You are welcome to try and implement this, see comments in the__main__.py.)If it isn't clear,abarmsdoessupport encrypted.abfiles as inputs (because my phone always generates such regardless of my wishes).LicenseGPLv3+, small library parts are MIT.UsageabarmsA handy Swiss-army-knife-like utility for manipulating Android Backup files (*.ab,*.adb) produced byadb backup,bmgr, and similar tools.options:--version: show program's version number and exit-h, --help: show this help message and exit--markdown: show help messages formatted in Markdownpassphrase:-p PASSPHRASE, --passphrase PASSPHRASE: passphrase for an encryptedINPUT_AB_FILE--passfile PASSFILE: a file containing the passphrase for an encryptedINPUT_AB_FILE; similar to-poption but the whole contents of the file will be used verbatim, allowing you to, e.g. use new line symbols or strange character encodings in there; default: guess based onINPUT_AB_FILEtrying to replace ".ab" and ".adb" extensions with ".passphrase.txt"subcommands:{ls,list,strip,ab2ab,split,ab2many,merge,many2ab,unwrap,ab2tar,wrap,tar2ab}ls (list): list contents of an Android Backup filestrip (ab2ab): strip encyption and compression from an Android Backup filesplit (ab2many): split a full-system Android Backup file into a bunch of per-app Android Backup filesmerge (many2ab): merge a bunch of Android Backup files into oneunwrap (ab2tar): convert an Android Backup file into a TAR filewrap (tar2ab): convert a TAR file into an Android Backup fileabarms lsList contents of an Android Backup file similar to howtar -tvfwould do, but this will also show Android Backup file version and compression flags.positional arguments:INPUT_AB_FILE: an Android Backup file to be used as input, set to "-" to use standard inputabarms stripConvert an Android Backup file into another Android Backup file with encryption and (optionally) compression stripped away.
I.e. convert an Android Backup file into a simple unencrypted (plain-text) and uncompressed version of the same.Versioning parameters and the TAR file stored inside the input file are copied into the output file verbatim.Useful e.g. if your Android firmware forces you to encrypt your backups but you store your backups on an encrypted media anyway and don't want to remember more passphrases than strictly necessary.
Or if you want to strip encryption and compression and re-compress using something better than zlib.positional arguments:INPUT_AB_FILE: an Android Backup file to be used as input, set to "-" to use standard inputOUTPUT_AB_FILE: file to write the output to, set to "-" to use standard output; default: "-" ifINPUT_TAR_FILEis "-", otherwise replace ".ab" and ".adb" extension ofINPUT_TAR_FILEwith.stripped.aboptions:-d, --decompress: produce decompressed output; this is the default-k, --keep-compression: copy compression flag and data from input to output as-is; this will make the output into a compressed Android Backup file if the source is compressed; this is the fastest way tostrip, since it just copies bytes around as-is-c, --compress: (re-)compress the output file; this could take awhileabarms splitSplit a full-system Android Backup file into a bunch of per-app Android Backup files.Resulting per-app files can be given toadb restoreto restore selected apps.Also, if you do backups regularly, then splitting large Android Backup files like this and then deduplicating per-app files between backups could save lots of disk space.positional arguments:INPUT_AB_FILE: an Android Backup file to be used as input, set to "-" to use standard inputoptions:--prefix PREFIX: file name prefix for output files; default:abarms_split_backupifINPUT_AB_FILEis "-",abarms_split_<INPUT_AB_FILE without its ".ab" or ".adb" extension>otherwise-c, --compress: compress per-app output filesabarms mergeMerge many smaller Android Backup files into a single larger one.
A reverse operation tosplit.This exists mostly for checking thatsplitis not buggy.positional arguments:INPUT_AB_FILE: Android Backup files to be used as inputsOUTPUT_AB_FILE: file to write the output tooptions:-c, --compress: compress the output fileabarms unwrapConvert Android Backup header into a TAR file by stripping Android Backup header, decrypting and decompressing as necessary.The TAR file stored inside the input file gets copied into the output file verbatim.positional arguments:INPUT_AB_FILE: an Android Backup file to be used as input, set to "-" to use standard inputOUTPUT_TAR_FILE: file to write output to, set to "-" to use standard output; default: guess based onINPUT_AB_FILEwhile setting extension to.tarabarms wrap --output-versionConvert a TAR file into an Android Backup file by prepending Android Backup header and (optionally) compressing TAR data with zlib (the only compressing Android Backup file format supports).The input TAR file gets copied into the output file verbatim.Note that the above means that unwrapping a.abfile, unpacking the resulting.tar, editing the resulting files, packing them back with GNUtarutility, runningabarms wrap, and then runningadb restoreon the resulting file will probably crash your Android device (phone or whatever) because the Android-side code restoring from the backup expects the data in the packed TAR to be in a certain order and have certain PAX headers, which GNUtarwill not produce.So you should only use this on files previously produced byabarms unwrapor if you know what it is you are doing.Production of encrypted Android Backup files is not supported at this time.positional arguments:INPUT_TAR_FILE: a TAR file to be used as input, set to "-" to use standard inputOUTPUT_AB_FILE: file to write the output to, set to "-" to use standard output; default: "-" ifINPUT_TAR_FILEis "-", otherwise replace ".ab" and ".adb" extension ofINPUT_TAR_FILEwith.aboptions:--output-version OUTPUT_VERSION: Android Backup file version to use (required)-c, --compress: compress the output fileUsage notesGiving an encryptedINPUT_AB_FILEas input, not specifying--passphraseor--passfile, and not having a file named{INPUT_AB_FILE with ".ab" or ".adb" extension replaced with ".passphrase.txt"}in the same directory will case the passphrase to be read interactively from the tty.ExamplesList contents of an Android Backup file:abarms ls backup.abUsetarutil to list contents of an Android Backup file instead of runningabarms ls:abarms unwrap backup.ab - | tar -tvf -Extract contents of an Android Backup file:abarms unwrap backup.ab - | tar -xvf -Strip encryption and compression from an Android Backup file:# equivalent
abarms strip backup.ab backup.stripped.ab
abarms strip backup.ab# equivalent
abarms strip --passphrase secret backup.ab
abarms strip -p secret backup.ab# with passphrase taken from a file
echo -n secret > backup.passphrase.txt
# equivalent
abarms strip backup.ab
abarms strip --passfile backup.passphrase.txt backup.ab# with a weird passphrase taken from a file
echo -ne "secret\r\n\x00another line" > backup.passphrase.txt
abarms strip backup.abStrip encryption but keep compression, if any:# equivalent
abarms strip --keep-compression backup.ab backup.stripped.ab
abarms strip -k backup.abStrip encryption and compression from an Android Backup file and then re-compress usingxz:abarms strip backup.ab - | xz --compress -9 - > backup.ab.xz
# ... and then convert to tar and list contents:
xzcat backup.ab.xz | abarms unwrap - | tar -tvf -Convert an Android Backup file into a TAR archive:# equivalent
abarms unwrap backup.ab backup.tar
abarms unwrap backup.abConvert a TAR archive into an Android Backup file:# equivalent
abarms wrap --output-version=5 backup.tar backup.ab
abarms wrap --output-version=5 backup.tar