Package: parsesetup
This package returns information from a python packagessetup.pyfile,
without installing the package or trusting the code inside the
setup.py.What’s this for?This is really an experiment in large scale processing of setup.py files.
The information provided in these files lists the project description, author and
other metadata like package dependencies.Unfortunately there isn’t any easy way of programmatically accessing this information
for an arbitrary setup.py script. Each setup.py script is a full python program,
and the data can be arbitrary python objects passed as arguments to a function.
This package aims to provide a way for python programs to retrieve all this metadata
programmatically, without doing things like installing the program or trusting
the setup.py file to not have malicious side effects.I’ve verified it by running on every Python repository on GitHub that has more
than 500 stars (about 2500 repos). Every package that can be
installed on a clean python installation can be parsed by this code, and additionally a bunch of
packages that can’t be installed for different reasons can also have their setup contents
retrieved here.How does this work?Since setup.py files are Python files, the arguments to the setup call can be arbitrary
Python expressions. This means that the only reliable way of getting these arguments is by
evaluating the setup.py file using a Python interpreter.To return the arguments passed to setuptools.setup call from a setup.py file, I’m temporarily
monkey patching the setuptools.setup function to collect its arguments - and then
using exec to execute the setup.py file:setup_args=[None]# patch setup functions to just keep track of arguments passed to themdefpatched_setup(**kwargs):setup_args[0]=kwargssetuptools.setup=distutils.core=patched_setupexec(open(setup_py_filename).read(),{"__name__":"__main__","__builtins__":__builtins__,"__file__":setup_py_filename})Globals are explicitly set up to match what most scripts expect, including__name__ == "__main__"guards. Likewise there are some special cases with setting up the python path, current directory
etc that are taken care of by the full code.Sandboxing with DockerRunning exec on an untrusted python file is a bad idea. As an example, somesetup.py scriptsdo interesting things like mess around
with your root git config - but the potential harm could be much much worse than that.To prevent harmful side-effects, this package runs by default in a sandboxed Docker container. In addition to security benefits,
this also lets us to cleanly fall back to using a Python2.7 Docker image in the case where the syntax is invalid for Python3.Running in a docker container can be disabled by setting atrusted=Trueflag when calling. Also note that its probably worth
configuring Docker to usegVisorto provide some extra piece of mind when parsing untrusted code.Handling Missing DependenciesA common pattern for setup.py files is to import the uninstalled module to look up things like version strings.
While this works, it can have the side effect of importing the modules dependencies before they are installed.As an example,tensorlayerimports some metadata from its root module - which in turn imports tensorflow, which hasn’t been installed yet in the docker
image.To hack around this problem, this code has the option of hooking into Python’s import handling to prevent ImportError’s
from surfacing when running.The idea is to provide a module importer tosys.meta_paththat always finds a module if the existing
resolution fails:classMockModuleImporter(object):deffind_module(self,fullname,path=None):returnselfdefload_module(self,name):mock=MockModule(name)sys.modules[name]=mockreturnmock# This hooks into Pythons' import mechanism, meaning that any# module that fails to import will be replaced with a MockModule# objectsys.meta_path.append([MockModuleImporter()])TheMockModuleinherits fromtypes.Modulesand just returns a Mock object object
with the common magic methods definedclassMockModule(types.ModuleType):def__getattr__(self,name,*args,**kwargs):returnMock()def__call__(self,*args,**kwargs):returnMock()classMock(object):def__getattr__(self,*args,**kwargs):returnself__call__=__getitem__=__setitem__=__add__=__getattr__...etc...This prevents a sizeable number of errors, and doesn’t seem to affect the output noticeably.
This behaviour can be disabled by settingmock_imports=False.UsageThis code can be installed via pip:pipinstallparsesetupTo runimportparsesetup# parses the setup.py file, returning arguments as a dictsetup_args=parsesetup.parse(path_to_setup_py)# Parses a single package without using docker (dangerous!)setup_args=parsesetup.parse(path_to_setup_py,trusted=True)# Parses multiple packages in a single docker container. All packages# need to share a common directory root for this to workwithparsesetup.DockerSetupParser(ROOT_PATH)asparser:setup_a=parser.parse(path_to_setup_py_a)setup_b=parser.parse(path_to_setup_py_a)FeaturesProgrammatically lets you inspect information contained in setup.py filesHandles both python2.7 and 3.6 scriptsHooks into setuptools, distutils.core and numpy.distutils.core setupsRuns untrusted setup.py files in a docker containerReads files with a __name__ == “__main__” guardReleased under the MIT License