Package: xar
XARXAR lets you package many files into a single self-contained executable file.
This makes it easy to distribute and install.A.xarfile is a read-only file system image which, when mounted, looks like
a regular directory to user-space programs.  This requires a one-time
installation of a driver for this file system
(SquashFS).XAR is pronounced like "czar" (/t͡ʂar/).  The 'X' in XAR is meant to be
a placeholder for all other letters as at Facebook this format was originally
designed to replace ZIP-based PAR (Python archives), JSAR (JavaScript archives),
LAR (Lua archives), and so on.Use CasesThere are two primary use cases for XAR files.  The first is simply collecting
a number of files for automatic, atomic mounting somewhere on the filesystem.
Using a XAR file vastly shrinks the on-disk size of the data it holds.
Compressing to below 20% of the original size is not unheard of.  This can save
multiple gigabytes per machine and reduce random disk IO.  This is especially
important on machines with flash storage.The second use case is an extension of the first -- by making the XAR file
executable and using thexarexechelper, a XAR becomes a self-contained
package of executable code and its data.  A popular example is Python
application archives that include all Python source code files, as well as
native shared libraries, configuration files, other data.This can replace virtualenvs and PEX files with a system that is faster, has
less overhead, is more compatible, and achieves better compression.
The downside is that it requires a setuid helper to perform the mounting.Advantages of XAR for Python usageSquashFS looks like regular files on disk to Python. This lets it use regular
imports which are better supported by CPython.SquashFS looks like regular files to your application, too. You don't need to
usepkg_resourcesor other tricks to access data files in your package.SquashFS with Zstandard compression saves disk space, also compared to a ZIP
file.SquashFS doesn't require unpacking of.sofiles to a temporary location like
ZIP files do.SquashFS is faster to start up than unpacking a ZIP file. You only need to
mount the file system once. Subsequent calls to your application will reuse
the existing mount.SquashFS only decompresses the pages that are used by the application, and
decompressed pages are cached in the page cache.SquashFS is read-only so the integrity of your application is guaranteed
compared to using virtualenvs or unpacking to a temporary directory.BenchmarksOptimizing performance (both space and execution time) was a key design goal for
XARs. We ran benchmark tests with open source tools to compare PEX, XAR, and
native installs on the following metrics:Size:file size, in bytes, of the executableCold start time:time taken when we have nothing mounted or extractedHot start time:time taken when we have extracted cache or mounted XAR squashfsThe PEXs are built withpython3 setup.py bdist_pex --bdist-all, and the XARs
are built withpython3 setup.py bdist_xar --xar-compression-algorithm=zstd.Console scriptSizeCold start timeHot start timedjango-admin (native)22851072 B-0.220 sdjango-admin.pex8529089 B1.705 s0.772 sdjango-admin.xar5464064 B (-36%)0.141 s (-92%)0.122 s (-84%)black (native)1020928 B-0.245 sblack.pex677550 B0.737 s0.619 sblack.xar307200 B (-55%)0.245 s (-67%)0.219 s (-65%)jupyter (native)64197120 B-0.399 sjupyter.pex17315669 B2.152 s1.046 sjupyter.xar17530880 B (+1%)0.213 s (-90%)0.181 s (-83%)The results show that both file size (withzstd compression) and start times
improve with XARs. This is an improvement when shipping to large number of
servers, especially with short-running executables, such as small data
collection scripts on web servers or interactive command line tools.RequirementsXAR requires:Linux or macOSPython >=2.7.11& >=3.5squashfs-toolsto build XARssquashfuse>= 0.1.102withsquashfuse_llto run XARsComponents of XARbdist_xarThis is a setuptools plugin that lets you package your Python application
as a .xar file.  It requiressquashfs-tools.  Install it from PyPI to get
the stable version:pip install xaroryou can install it from this repository:python setup.py installAfter installation go to your favorite Python project with a console script and
run:python setup.py bdist_xarThe setuptools extensionbdist_xarhas options to configure the XAR, most
importantly--interpretersets the Python interpreter used. Runpython setup.py bdist_xar --helpfor a full list of options.xarexec_fuseThis is a binary written in C++ used to mount a SquashFS image.
It requiressquashfuseinstalled. Note that the currentsquashfusepackage
on Ubuntu doesn't includesquashfuse_ll, so you will have to install fromsource.You can build this part of the code with:mkdir build && cd build && cmake .. && make && [sudo] make installExamplesbdist_xarSimply run:python /path/to/black/setup.py bdist_xar [--xar-compression-algorithm=zstd]
/path/to/black/dist/black.xar --helpmake_xarXAR provides a simple CLI to create XARs from Python executables or directories.
We can create a XAR from an existing Python executable zip file, like a PEX.make_xar --python black.pex --output black.xarYou can also create a XAR from a directory, and tell XAR which executable to
run once it starts.> mkdir myxar
> echo -n "#\!/bin/sh\nshift\necho \$@" > myxar/echo
> chmod +x myxar/echo
> make_xar --raw myxar --raw-executable echo --output echo
> ./echo hello world
hello worldxarexec_fusewill execute the executable it is given using the XAR path as the
first argument, and will forward the XARs arguments after.Running the Circle CI tests locallyFirst you need to install docker (and possible docker-machine), as it is how it
runs the the code. Then you need toinstallthecirclecicli, and runcircleci buildIf you change.circleci/config.ymlyou should validate it before committingcircleci config validateContributingSee the CONTRIBUTING file for how to help out.LicenseXAR is BSD-licensed.