Package: extractcode
license: Apache-2.0copyright: copyright (c) nexB. Inc. and othershomepage_url:https://github.com/nexB/extractcodekeywords: archive, extraction, libarchive, 7zip, scancode-toolkit, extractcodeSupports Windows, Linux and macOS on 64 bits processors and Python 3.6 to 3.9.ExtractCode is a (mostly) universal archive extractor.Install with:pip install extractcode[full]Why another extractor?it will extract!ExtractCode will extract things where other archive and compressed file extractors may fail.ExtractCode supports one of largest number of archive formats listed in the
longList of supported archive formatsfound at the bottom of this document.Say you want to extract the tarball of the Linux kernel source code on Windows.
It contains paths that are the same when ignoring the case and therefore will
not extract OK on Windows: some file may be munged or the extract may file.Or a tarball (on any OS) may contain multiple times the exact same path. In
these cases the paths showing up earlier in the archive may be “hidden” and
overwritten by the same path showing up later in the archive giving the
impression that there is only one file.Or an archive may be damaged a little but most files can still be extracted.Or the extracted files are such permissions that you cannot read them and are
not owned by you.Or the archive may contain weird paths inluding relative paths that may be
problematic to extract.Or the archive may contain special file types (character/device files) that
may be problematic to extract.Or an archive may be a virtual disk or some file system(s) images that would
typically need to be mounted to be accessed, and may require root access
and guesswork to find out which partition and filesystem are at play and
which driver to use.In all these cases, ExtractCode will extract and try hard do the right thing to
obtain the actual archived content when other tools may fail.It can also extract recursively any type of (nested) archives-in-archives.As a downside, the extracted content may not be exactly what would be extracted
for a typical usage of the contained files: for instance some file may be
renamed, special files and symlinks are skipped, permissions and owners are
changed but this it is fine for prmary the use case which is analysis of file
content for software composition or forensic analysis.Behind the scene, ExtractCode uses multiple tools such as:the Python standard library,a custom ctypes binding to libarchive,the 7zip command line tool, andoptionally libguestfs on Linux.With these, it is possible to extract a large number of common and less common
archives and compressed file types. ExtractCode tries to extract things in the
same way on all supported OSes, including auto-renaming files that would have
invalid, non-extractible names on certain filesystems or when there are multiple
copies of the same path in a given archive (which is possible in a tar).The extraction is driven from  a “voting” system that considers the file
extension(s) and name, the filetype and mimetype (using a ctypes binding to
libmagic) to select the most appropriate extractor or decompressor function.
It can handle multi-level archives such as tar.gz and can extract recursively
any nested archives.Visithttps://aboutcode.organdhttps://github.com/nexB/for support and download.We run CI tests on:Azure pipelineshttps://dev.azure.com/nexB/extractcode/_buildInstallationTo install this package with its full capability (where the binaries for
7zip and libarchive are installed), use thefullextra option:pip install extractcode[full]If you want to use the version of binaries (possibly) provided by your operating
system, use theminimaloption:pip install extractcodeIn this case, you will need to provide a working and compatible libarchive and
7zip installed and configured in one of these ways such that ExtractCode can
find them:a typecode-libarchive and typecode-7z plugin: See the standard ones athttps://github.com/nexB/scancode-plugins/tree/main/builtinsThese can either bundle a libarchive library, a 7z executable or expose a
system-installed libraries.
It does so by providing plugin entry points asscancode_location_providerforextractcode_libarchivethat should point to aLocationProviderPluginsubclass with aget_locations()method that must return a mapping with
this key:‘extractcode.libarchive.dll’: the absolute path to alibarchiveshared object/DLLSee for example:https://github.com/nexB/scancode-plugins/blob/4da5fe8a5ab1c87b9b4af9e54d7ad60e289747f5/builtins/extractcode_libarchive-linux/setup.py#L40https://github.com/nexB/scancode-plugins/blob/4da5fe8a5ab1c87b9b4af9e54d7ad60e289747f5/builtins/extractcode_libarchive-linux/src/extractcode_libarchive/__init__.py#L17And in the same way, thescancode_location_providerforextractcode_7zipshould point to aLocationProviderPluginsubclass with aget_locations()method that must return a mapping with this key:‘extractcode.sevenzip.exe’: the absolute path to a7zipexecutableSee for example:https://github.com/nexB/scancode-plugins/blob/4da5fe8a5ab1c87b9b4af9e54d7ad60e289747f5/builtins/extractcode_7z-linux/setup.py#L40https://github.com/nexB/scancode-plugins/blob/4da5fe8a5ab1c87b9b4af9e54d7ad60e289747f5/builtins/extractcode_7z-linux/src/extractcode_7z/__init__.py#L18useenvironment variablesto point to installed binaries:EXTRACTCODE_LIBARCHIVE_PATH: the absolute path to a libarchive DLLEXTRACTCODE_7Z_PATH: the absolute path to a 7zip executablea system-installed libarchive and 7zip executableavailable in the systemPATH.The supported binary tools versions are:libarchive  3.5.x7zip 16.5.xDevelopmentTo set up the development environment:./configure --dev
source venv/bin/activateTo run unit tests:pytest -vvs -n 2To clean up development environment:./configure --cleanTo run the command line tool in the activated environment:./extractcode -hConfiguration with environment variablesExtractCode will use these environment variables if set:EXTRACTCODE_LIBARCHIVE_PATH : the path to thelibarchive.solibarchive
shared library used to support some of the archive formats. If not provided,
ExtractCode will look for a plugin-provided libarchive library path. Seehttps://github.com/nexB/scancode-plugins/tree/main/builtinsfor such plugins.
If no plugin contributes libarchive, then a final attempt is made to look for
it in the PATH using standard DLL loading techniques.EXTRACTCODE_7Z_PATH : the path to the7z7zip executable used to support
some of the archive formats. If not provided, ExtractCode will look for a
plugin-provided 7z executable path. Seehttps://github.com/nexB/scancode-plugins/tree/main/builtinsfor such plugins.
If no plugin contributes 7z, then a final attempt is made to look for
it in the PATH.EXTRACTCODE_GUESTFISH_PATH : the path to theguestfishtool from
libguestfs to use to extract VM images. If not provided, ExtractCode will look
in the PATH for an installedguestfishexecutable instead.Adding support for VM images extractionAdding support for VM images requires the manual installation of the
libguestfs-tools system package. This is suported only on Linux.
On Debian and Ubuntu you can use this command:sudo apt-get install libguestfs-toolsOn Ubuntu only, an additional manual step is required as the kernel executable
file cannot be read by users as required by libguestfish.Run this command as a temporary and immediate fix:sudo chmod 0644 /boot/vmlinuz-*
for k in /boot/vmlinuz-*
    do sudo dpkg-statoverride --add --update root root 0644 /boot/vmlinuz-$k
doneYou likely want both this temporary fix and a more permanent fix; otherwise each
kernel update will revert to the default permissions and ExtractCode will stop
working for VM images extraction.Therefore follow these instructions:1. As sudo, create the file /etc/kernel/postinst.d/statoverride with this
content, devised by Kees Cook (@kees) inhttps://bugs.launchpad.net/ubuntu/+source/linux/+bug/759725/comments/3#!/bin/sh
version="$1"
# passing the kernel version is required
[ -z "${version}" ] && exit 0
dpkg-statoverride --update --add root root 0644 /boot/vmlinuz-${version}Set executable permissions:sudo chmod +x /etc/kernel/postinst.d/statoverrideSee also these links for a complete discussion:https://bugs.launchpad.net/ubuntu/+source/linux/+bug/759725https://bugzilla.redhat.com/show_bug.cgi?id=1670790https://bugs.launchpad.net/ubuntu/+source/libguestfs/+bug/1813662/comments/24AlternativeThese other tools are related and were considered before creating ExtractCode:These tools provide built-in, original extraction capabilities:https://libarchive.org/(integrated in ExtractCode) (BSD license)https://www.7-zip.org/(integrated in ExtractCode) (LGPL license)https://theunarchiver.com/command-line(maintenance status unknown) (LGPL license)These tools are command line tools  wrapping other extraction tools and are
similar to ExtractCode but with different goals:https://github.com/wummel/patool(wrapper on many CLI tools) (GPL license)https://github.com/dtrx-py/dtrx(wrapper on a few CLI tools) (recently revived) (GPL license)List of supported archive formatsExtractCode can extract the folowing archives formats:Archive format kind: docsname: Office docextensions: .docx, .dotx, .docm, .xlsx, .xltx, .xlsm, .xltm, .pptx, .ppsx, .potx, .pptm, .potm, .ppsm, .odt, .odf, .sxw, .stw, .ods, .ots, .sxc, .stc, .odp, .otp, .odg, .otg, .sxi, .sti, .sxd, .sxg, .std, .sdc, .sda, .sdd, .smf, .sdw, .sxm, .stw, .oxt, .sldx, .epubfiletypes : zip archive, microsoft word 2007+, microsoft excel 2007+, microsoft powerpoint 2007+mimetypes : application/zip, application/vnd.openxmlformatsname: Dia diagram docextensions: .diafiletypes : gzip compressedmimetypes : application/gzipname: Graffle diagram docextensions: .grafflefiletypes : gzip compressedmimetypes : application/gzipname: SVG Compressed docextensions: .svgzfiletypes : gzip compressedmimetypes : application/gzipArchive format kind: regularname: Tarextensions: .tarfiletypes : .tar, tar archivemimetypes : application/x-tarname: Zipextensions: .zip, .zipxfiletypes : zip archivemimetypes : application/zipname: Java archiveextensions: .war, .sar, .earfiletypes : zip archivemimetypes : application/zip, application/java-archivename: xzextensions: .xzfiletypes : xz compressedmimetypes : application/x-xzname: lzmaextensions: .lzmafiletypes : lzma compressedmimetypes : application/x-xzname: Gzipextensions: .gz, .gzip, .wmz, .arzfiletypes : gzip compressed, gzip compressed datamimetypes : application/gzipname: bzip2extensions: .bz, .bz2, bzip2filetypes : bzip2 compressedmimetypes : application/x-bzip2name: lzipextensions: .lzipfiletypes : lzip compressedmimetypes : application/x-lzipname: RARextensions: .rarfiletypes : rar archivemimetypes : application/x-rarname: ar archiveextensions: .arfiletypes : current ar archivemimetypes : application/x-archivename: 7zipextensions: .7zfiletypes : 7-zip archivemimetypes : application/x-7z-compressedname: cpioextensions: .cpiofiletypes : cpio archivemimetypes : application/x-cpioname: Zextensions: .zfiletypes : compress’d datamimetypes : application/x-compressArchive format kind: regular_nestedname: Tar xzextensions: .tar.xz, .txz, .tarxzfiletypes : xz compressedmimetypes : application/x-xzname: Tar lzmaextensions: tar.lzma, .tlz, .tarlz, .tarlzmafiletypes : lzma compressedmimetypes : application/x-lzmaname: Tar gzipextensions: .tgz, .tar.gz, .tar.gzip, .targz, .targzip, .tgzipfiletypes : gzip compressedmimetypes : application/gzipname: Tar lzipextensions: .tar.lz, .tar.lzipfiletypes : lzip compressedmimetypes : application/x-lzipname: Tar lz4extensions: .tar.lz4filetypes : lz4 compressedmimetypes : application/x-lz4name: Tar zstdextensions: .tar.zst, .tar.zstdfiletypes : zstandard compressedmimetypes : application/x-zstdname: Tar bzip2extensions: .tar.bz2, .tar.bz, .tar.bzip, .tar.bzip2, .tbz, .tbz2, .tb2, .tarbz2filetypes : bzip2 compressedmimetypes : application/x-bzip2name: lz4extensions: .lz4filetypes : lz4 compressedmimetypes : application/x-lz4name: zstdextensions: .zst, .zstdfiletypes : zstandard compressedmimetypes : application/x-zstdname: Tar 7zipextensions: .tar.7z, .tar.7zip, .t7zfiletypes : 7-zip archivemimetypes : application/x-7z-compressedname: Tar Zextensions: .tz, .tar.z, .tarzfiletypes : compress’d datamimetypes : application/x-compressArchive format kind: packagename: Ruby Gem packageextensions: .gemfiletypes : .tar, tar archivemimetypes : application/x-tarname: Android appextensions: .apkfiletypes : zip archivemimetypes : application/zipname: Android libraryextensions: .aarfiletypes : zip archivemimetypes : application/zipname: Mozilla extensionextensions: .xpifiletypes : zip archivemimetypes : application/zipname: iOS appextensions: .ipafiletypes : zip archivemimetypes : application/zipname: Springboot Java Jar packageextensions: .jarfiletypes : bourne-again shell script executable (binary data)mimetypes : text/x-shellscriptname: Java Jar packageextensions: .jar, .zipfiletypes : java archivemimetypes : application/java-archivename: Java Jar packageextensions: .jarfiletypes : zip archivemimetypes : application/zipname: Python packageextensions: .egg, .whl, .pyz, .pexfiletypes : zip archivemimetypes : application/zipname: Microsoft cabextensions: .cabfiletypes : microsoft cabinetmimetypes : application/vnd.ms-cab-compressedname: Microsoft MSI Installerextensions: .msifiletypes : msi installermimetypes : application/x-msiname: Apple pkg or mpkg package installerextensions: .pkg, .mpkgfiletypes : xar archivemimetypes : application/octet-streamname: Xar archive v1extensions: .xarfiletypes : xar archivemimetypes : application/octet-stream, application/x-xarname: Nugetextensions: .nupkgfiletypes : zip archive, microsoft ooxmlmimetypes : application/zip, application/octet-streamname: Static Libraryextensions: .a, .lib, .out, .kafiletypes : current ar archive, current ar archive random librarymimetypes : application/x-archivename: Debian packageextensions: .deb, .udebfiletypes : debian binary packagemimetypes : application/vnd.debian.binary-package, application/x-archivename: RPM packageextensions: .rpm, .srpm, .mvl, .vipfiletypes : rpmmimetypes : application/x-rpmname: Apple dmgextensions: .dmg, .sparseimagefiletypes : zlib compressedmimetypes : application/zlibArchive format kind: file_systemname: ISO CD imageextensions: .iso, .udf, .imgfiletypes : iso 9660 cd-rom, high sierra cd-rommimetypes : application/x-iso9660-imagename: SquashFS disk imageextensions:filetypes : squashfsmimetypes :name: QEMU QCOW2 disk imageextensions: .qcow2, .qcow, .qcow2c, .imgfiletypes : qemu qcow2 image, qemu qcow imagemimetypes : application/octet-streamname: VMDK disk imageextensions: .vmdkfiletypes : vmware4 disk imagemimetypes : application/octet-streamname: VirtualBox disk imageextensions: .vdifiletypes : virtualbox disk imagemimetypes : application/octet-streamArchive format kind: patchesname: Patchextensions: .diff, .patchfiletypes : diff, patchmimetypes : text/x-diffArchive format kind: special_packagename: InstallShield Installerextensions: .exefiletypes : installshieldmimetypes : application/x-dosexecname: Nullsoft Installerextensions: .exefiletypes : nullsoft installermimetypes : application/x-dosexec