Package: treeshaker
treeshakerA tree-shaking tool for Python.What it doesIf you give treeshaker a root environment (specified by arequirements.txtfile), a list of target packagestarget_packages(in this casemypkgandotherpkg), and a target module (in this casemypkg.target), treeshaker will
create an output directory that containsthe subset of modules among all the modules in all thetarget_packagesthat are needed to import the target module, copied as .py files directly
into the root of the output directory anda newrequirements.txtfile containing the subset of the rootrequirements.txtlines needed to import all the modules in (1)An end user can then open the folder,pip install -r requirements.txt, and
import the target module, without installing or obtaining the complete code for
any of thetarget_packages.Quick startInstall treeshaker:$ pip install treeshakerDefine your root environment in arequirements.txtfile and ensure the
listed packages are available in your current Python environment. If you
don't have arequirements.txtfile yet, you can run$ pip install <package that provides target module>
$ pip freeze > requirements.txtIf you already have arequirements.txtfile, you can instead run$ pip install -r requirements.txtCreate a configuration file (see theexample config filefor reference), specifying the location of your rootrequirements.txtfile
as well as the details of what targets should be built and how.Run treeshaker:$ treeshakerDemogit clone https://github.com/sclabs/treeshaker.gitcd treeshakervirtualenv venvsource venv/bin/activatepip install . ./examples/mypkg ./examples/otherpkg ./examples/externaldep ./examples/otherdeptreeshaker --config examples/treeshaker.cfgTo clean up after the demo run$ python examples/clean.pyImportant featuresBuilding multiple targets versus building one targetTo build one specific target, run$ treeshaker --target <target_module_name>treeshaker will find the configuration file section[target:<target_module>](which must exist) to determine theoutdir(a required key in every[target:<target_module>]section) and all other parameters for the build.To build a list of targets, add the desired targets to the[targets]section
(as keys with no values) and run$ treeshakerModule renamingThe copied modules will be renamed if necessary to avoid name conflicts.If one copied module ismypkg.depand another isotherpkg.dep, these will be
renamed todep_mypkg.pyanddep_otherpkg.py, respectively.Import rewritingDuring module renaming, treeshaker will attempt to rewrite import statements in
the copied modules as necessary. The logic for doing this is quite rudimentary
and is only expected to succeed when the import is of the formimport mypkg.mymodorfrom mypkg.mymod import not_a_moduleIn these cases, whenmypkg.mymodis copied, it will be renamed and a simple
find-and-replace will "fix" the import statement.The following form of import is expected to fail:from mypkg import mymodRelative imports are generally also expected to fail, though they may work when
importing a sibling module whose name has no conflicts.Configuration file inheritanceIf a key is missing from a[target:<target_module>]section, treeshaker will
fall back to looking for the key in the[target]section.Theexample configuration filedefines many keys in
the[target]section, but all of these can be overridden on a per-target basis
by adding the key to the corresponding[target:<target_module>]section.<outdir>interpolationYou can use the special string<outdir>in the[target]section or any[target:<target_module>]section and it will be replaced with the actual
outdir at runtime.Automatic documentationFunctions and/or classes listed in afunctionskey andpython-firecomponents listed in afire_componentskey in the configuration file will be
documented in<outdir>/README.md, using their docstring or fire help page,
respectively.You can also manually write a README section that will be included in<outdir>/README.mdby specifying a filename in areadmekey. If this file
does not exist on the disk, it will not be included.Custom configuration file nameTo run treeshaker using a specific configuration file, run$ treeshaker --config path/to/customconfig.cfgHow does it work?Dependency graph constructionPython providesmodulefinderin the
standard library, but it is not quite powerful of flexible enough to accomplish
what we want. Instead, we use themodulegraphlibrary, which is both performant (thanks to the usefulexcludesargument) and
flexible (it creates a dependency graph that we can analyze in detail to
classify imported modules).Matching PyPI package names to root module namesHow can treeshaker determine that if you import a module fromsklearn, the
correspondingrequirements.txtline is e.g.,scikit-learn==0.22.1?pipreqshas compiled a mapping of PyPI names
to root module names for many PyPI packages; we use this list to perform this
matching. We considered adding a dependency on pipreqs, but we decided to vendor
it ourselves to avoid an additional seven dependent package installs (we only
use the single file that contains the mapping).For consistency, we choose to use the actual module name (the one used in
imports and the one that reflects the location of the package on-disk after
install) to specify thetarget_packages.CaveatsWhile treeshaking is a cool concept, there's rarely any real need to perform
treeshaking on Python code. This library was created for a highly specific use
case: automatically minimizing the amount of code revealed from a monolithic
private codebase when providing code for one specific module (or more
specifically, the task that module performs).In a monolithic library design, it's possible to have a very large number of
modules and a large number of tasks that the library can accomplish. It might be
better to apply the "single responsibility principle" and design a set of
libraries that each do one task, but what happens when the monolith already
exists and a redesign is not feasible? In most situations, there is no harm in
distributing the entire library even if the person requesting the code wants to
accomplish just one task. However, when the library code is private, it may be
desirable to minimize the total amount of code "exposed" or "revealed" to the
requestor.