Package: buildout.recipe.isolation
This buildout recipe’s sole purpose is to isolate distribution packages and
there dependencies. This recipe was originally developed to be used in the
packaging of Zope2 for the Debian operating system. The recipe can additionally
be used to populate a Python enviroment (or virtual environment). This can be
handy in situations where buildout is required to build the application, but
the developer/system administrator wants to use the application in a typical
Python environment.Isolation of distributionsThebuildout.recipe.isolationrecipe can be used to isolate distributions and
their dependencies in a single directory. The recipe takes a number of options:distsA list of distributions to isolate given as one or more setuptools
requirement strings. Each requirements string should be given on a
separate line. The default is to use the part name as the distribution.exclude-dists (optional)A list of distributions that should be excluded from the isolation.dists-location (optional)A directory location where the isolated distributions should be put.
This option
defaults to a location in the buildout parts directory under the section
name where the recipe is being used.scripts-location (optional)A directory location where the distribution scripts should be isolated.
This option defaults to a location in the buildout parts directory under
the part (or section) name followed by ‘-scripts’ (e.g. for a part
namedisolatedthe default scripts directory name would beisolated-scripts).pth-location (optional)A directory location where a.pthfile will be created for this isolation.
This option defaults to a location in the buildout parts directory under
the part (or section) name followed by ‘-pth’ (e.g. for a part namedisolated, the default pth directory name would beisolated-pth).The final name of the pth file will be the part name with a.pthextension. To reference the resulting.pthfile, use thepth-file-locationoptions.pth-file-location (reference only)A location where the.pthfile lives. The resulting.pthfile is used
during the script generation process to provide a list of distributions
that are isolated somewhere else on the filesystem.NoteThis option can only be referenced. Later versions of this recipe
may provide a means to change the pth filename.extra-pth (optional)A list of.pthfiles to include as part of the script initialization.This option resolves dependency issues caused by dependency isolation.
For instance, if you are usingexclude-distsand those distributions
that are being exluded are required to run a script, you probably want
to include the.pthfile with locations to those dependencies.exclude-own-pth (optional)A boolean option, that when set will exclude the in context part’s generated.pthfile from inclusion in scripts. This option is closely tied to
pth-file-location and extra-pth. This option is false by default.The reason this option has been included is because the locations in the.pthfile main already be included in the python path via the.pthfile’s location in site-packages.executable (optional)The location of the Python executable. By default this issys.executable.The executable specified is not executed in the recipe. The location is
used as the shebang line during the scripts generation.stage-locally (optional)A boolean option to specify whether we should stage the resources or
put them in there final destination. If this option is true, the values
specified fordist-location,script-locationandpth-locationare
used to generate the resources, but the resources are placed in
the default parts locations. This option is handy for staged installation.Recipe deliverablesA directory that contains a specified distribution(s) package and its
dependency package(s).A directory that contains a.pthfile, which lists the absolute path
for each package in the isolation context.A directory that contains the scripts that have been generated from the
distribution(s) package and its dependency packages.How it worksWe have a sample buildout.  Let’s update it’s configuration file to
install the demo package.>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = buildout.recipe.isolation
... dists = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... """ % dict(server=link_server))In this example, we limited ourselves to revisions before 0.3. We also
specified where to find distributions using the find-links option.In order to control the distribution test data, we decided to use buildout’s
testing index, shown below:>>> print get(link_server),
<html><body>
<a href="bigdemo-0.1-py2.3.egg">bigdemo-0.1-pyN.N.egg</a><br>
<a href="demo-0.1-py2.3.egg">demo-0.1-py2.3.egg</a><br>
<a href="demo-0.2-py2.3.egg">demo-0.2-py2.3.egg</a><br>
<a href="demo-0.3-py2.3.egg">demo-0.3-py2.3.egg</a><br>
<a href="demo-0.4c1-py2.3.egg">demo-0.4c1-py2.3.egg</a><br>
<a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br>
<a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br>
<a href="demoneeded-1.2c1.zip">demoneeded-1.2c1.zip</a><br>
<a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br>
<a href="index/">index/</a><br>
<a href="other-1.0-py2.3.egg">other-1.0-py2.3.egg</a><br>
</body></html>We will be using this index through the testing structure and further
explaining the relationships before each of these distributions.Let’s run the buildout:>>> import os
>>> print system(buildout), #doctest: +ELLIPSIS
Installing demo.
Getting distribution for 'demo<0.3'.
Got demo 0.2.
Getting distribution for 'demoneeded'.
install_dir ...
Got demoneeded 1.2c1.
demo: Copying demo to the destination directory.
demo: Copying demoneeded to the destination directory.
demo: Generated script '/sample-buildout/parts/demo-scripts/demo'.Now, if we look at the buildout parts directory for the isolation:>>> ls(sample_buildout, 'parts/demo')
-  demo-0.2-py2.3.egg
d  demoneeded-1.2c1-py2.3.eggThese distributions have been entered into a.pthfile as well. This file
is not directly useful to the buildout, but has it’s place in a Python
environment. The contents of the.pthfile will be the absolute path for each
of the distributions that have been installed into the isolation. Let’s have
a look:>>> cat(sample_buildout, 'parts/demo-pth', 'demo.pth')
/sample-buildout/parts/demo/demo-0.2-py2.6.egg
/sample-buildout/parts/demo/demoneeded-1.2c1-py2.6.eggBy default the name of the.pthfiles will be the name of the buildout
section, which in this case is demo. You can change the location of the.pthfile using thepth-file-locationoption.NoteWhen using thepth-file-locationoption, the directory that the.pthfile will reside, must exist prior to running the buildout.
If directory  does not exist, anIOErrorwill be raised and the
buildout will fail.Dependency exclusionLet’s now try a buildout with a slightly larger example that we can use to
illustrate the exclude dependencies from a certain isolation.Let’s create a new buildout configuration based on the previous one. This
configuration is setup to isolate the bigdemo distribution and its
dependencies, but exclude the demoneeded dependency.>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts =
...     demoneeded
...     demo
... find-links = %(server)s
... index = %(server)s/index
...
... [demoneeded]
... recipe = buildout.recipe.isolation
... dists = demoneeded
...
... [demo]
... recipe = buildout.recipe.isolation
... dists = bigdemo
... exclude-dists = ${demoneeded:dists}
... """ % dict(server=link_server))>>> print system(buildout), #doctest: +ELLIPSIS
Uninstalling demo.
Installing demoneeded.
demoneeded: Copying demoneeded to the destination directory.
Installing demo.
Getting distribution for 'bigdemo'.
Got bigdemo 0.1.
Getting distribution for 'demo'.
Got demo 0.4c1.
demo: Copying demo to the destination directory.
demo: Copying bigdemo to the destination directory.
demo: Generated script '/sample-buildout/parts/demo-scripts/demo'.Check the isolated results:>>> ls(sample_buildout, 'parts/demo')
-  bigdemo-0.1-py2.6.egg
-  demo-0.4c1-py2.6.egg
>>> ls(sample_buildout, 'parts/demoneeded')
d  demoneeded-1.2c1-py2.6.eggScript generationSome distributions supply command-line scripts with there packages. Buildout
typically generates these scripts for us, because it needs to supply the built
packages to to script. It does this by injecting the distribution locations
into the Python system path. In some cases we do not want to inject anything
into the Python system path, because we may have deposited the generated .pth
file in a virtual environment’s site-packages directory. While in other cases,
we might want to supply our .pth file as a mean for import resolution. Let’s
take a closer look at both cases.For the general case, we will likely want to supply our .pth file to the
script. Additionally, we will probably want to supply any .pth files that
dependent isolations may have generated. Here is an example.>>> import sys
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts =
...     demoneeded
...     demo
... find-links = %(server)s
... index = %(server)s/index
...
... [demoneeded]
... recipe = buildout.recipe.isolation
... dists = demoneeded
...
... [demo]
... recipe = buildout.recipe.isolation
... dists = bigdemo
... exclude-dists = ${demoneeded:dists}
... extra-pth = ${demoneeded:pth-file-location}
... executable = %(python)s
... """ % dict(server=link_server, python=sys.executable))
>>> print system(buildout), #doctest: +ELLIPSIS
Uninstalling demo.
Uninstalling demoneeded.
Installing demoneeded.
demoneeded: Copying demoneeded to the destination directory.
Installing demo.
demo: Copying demo to the destination directory.
demo: Copying bigdemo to the destination directory.
demo: Generated script '/sample-buildout/parts/demo-scripts/demo'.The resulting script should have two .pth files in it. The demo.pth file has
been defined and generated from the recipe in context. The demoneeded.pth file
was generated by the demoneeded section and pulled in using the extra-pth
recipe option.>>> if sys.platform == 'win32':
...    script_name = 'demo-script.py'
... else:
...    script_name = 'demo'
>>> script_dir = 'parts/demo-scripts'
>>> f = open(os.path.join(sample_buildout, script_dir, script_name))
>>> shebang = f.readline().strip()
>>> if shebang[:3] == '#!"' and shebang[-1] == '"':
...     shebang = '#!'+shebang[3:-1]
>>> shebang == '#!' + os.path.realpath(sys.executable)
True
>>> print f.read(), # doctest: +NORMALIZE_WHITESPACE
<BLANKLINE>
import sys
def pth_injector(pth_file):
    path_file = open(pth_file, 'r')
    sys.path[0:0] = [line
        for line in path_file.read().split('\n')
        if line is not None]
<BLANKLINE>
pth_files = ['/sample-buildout/parts/demo-pth/demo.pth', '/sample-buildout/parts/demoneeded-pth/demoneeded.pth']
for pth in pth_files:
    pth_injector(pth)
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
if __name__ == '__main__':
    eggrecipedemo.main()
>>> f.close()The second case is where we have deposited the .pth files into a virtual
environment. Let’s setup afakevirtual environment structure inside the
buildout structure for demonstration sake.>>> virtenv = os.path.join(sample_buildout, 'virtenv')
>>> mkdir(virtenv)
>>> mkdir(virtenv, 'bin')
>>> mkdir(virtenv, 'lib')
>>> mkdir(virtenv, 'lib', 'python2.6')
>>> mkdir(virtenv, 'lib', 'python2.6', 'site-packages')
>>> site_pkgs = os.path.join(virtenv, 'lib', 'python2.6', 'site-packages')All we really need for the purpose of this demonstration is the site-packages
directory.>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts =
...     demoneeded
...     demo
... find-links = %(server)s
... index = %(server)s/index
...
... [demoneeded]
... recipe = buildout.recipe.isolation
... dists = demoneeded
... pth-file-location = %(site_pkgs)s
...
... [demo]
... recipe = buildout.recipe.isolation
... dists = bigdemo
... exclude-dists = ${demoneeded:dists}
... pth-file-location = %(site_pkgs)s
... exclude-own-pth = trUE
... python = %(python)s
... """ % dict(server=link_server, python=sys.executable,
...     site_pkgs=site_pkgs))
>>> print system(buildout), #doctest: +ELLIPSIS
Uninstalling demo.
Uninstalling demoneeded.
Installing demoneeded.
demoneeded: Copying demoneeded to the destination directory.
Installing demo.
demo: Copying demo to the destination directory.
demo: Copying bigdemo to the destination directory.
demo: Generated script '/sample-buildout/parts/demo-scripts/demo'.Now if we print out the demo script, we’ll find no mention of the .pth files.>>> f = open(os.path.join(sample_buildout, script_dir, script_name))
>>> shebang = f.readline().strip()
>>> if shebang[:3] == '#!"' and shebang[-1] == '"':
...     shebang = '#!'+shebang[3:-1]
>>> shebang == '#!' + os.path.realpath(sys.executable)
True
>>> print f.read(), # doctest: +NORMALIZE_WHITESPACE
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
if __name__ == '__main__':
    eggrecipedemo.main()
>>> f.close()Why does this work? If we were to use the virtual environments Python
executable, it would load the site-packages directory and any .pth files in
it. This would in turn load the modules we built using the buildout.NoteWe aren’t actually using the virtual environments Python executable
in this test case, but it is a simple matter of changing the executable
value in the system_python section of this buildout.cfg.Staging the isolationIn some situations it is handy to build the packages locally before
transfering these resources to a final destination. To do this we stage the
isolation process with thestage-locallyoption.This option will allow you to set thedists-location,scripts-locationandpth-file-locationas final destinations, but place the results in their
default build location. The default build location, if you recall, is in the
buildout’s parts directory.NoteThe following example isn’t necessarily useful beyond the test that
it satisfies. If you’re trying to figure out how to use the staging parts
of this recipe and run into issues or parts you don’t understand, please
feel free to contact the author (see the package metadata for the address).Let’s have a look at how this works by creating similar buildout to those about
execept now we are setting thestage-locallyoption totrue:>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts =
...     demo
... find-links = %(server)s
... index = %(server)s/index
...
... [demo]
... recipe = buildout.recipe.isolation
... dists = bigdemo
... dists-location = %(site_pkgs)s
... scripts-location = %(bin_dir)s
... pth-location = %(site_pkgs)s
... executable = %(python)s
... stage-locally = true
... """ % dict(server=link_server,
...            bin_dir=os.path.join(virtenv, 'bin'),
...            python=os.path.join(virtenv, 'bin', 'python'),
...            site_pkgs=site_pkgs))
>>> print system(buildout), #doctest: +ELLIPSIS
Uninstalling demo.
Uninstalling demoneeded.
Installing demo.
demo: Copying demo to the staging directory.
demo: Copying demoneeded to the staging directory.
demo: Copying bigdemo to the staging directory.
demo: Can't find the executable on the filesystem. Perhaps this setup is not destine to be used on this machine. So we are using the given executable value /sample-buildout/virtenv/bin/python as is.
demo: Generated script '/sample-buildout/parts/demo-scripts/demo'.To verify that things have been staged, let’s have a closer look at the demo
script to verify everything went as planned. For one, we expect the script
to be in the parts directory:>>> parts_dir = os.path.join(sample_buildout, 'parts')
>>> demo_script = os.path.join(parts_dir, 'demo-scripts', 'demo')
>>> os.path.exists(demo_script)
True
>>> cat(demo_script)
#!/sample-buildout/virtenv/bin/python
<BLANKLINE>
import sys
def pth_injector(pth_file):
    path_file = open(pth_file, 'r')
    sys.path[0:0] = [line
        for line in path_file.read().split('\n')
        if line is not None]
<BLANKLINE>
pth_files = ['/sample-buildout/virtenv/lib/python2.6/site-packages/demo.pth']
for pth in pth_files:
    pth_injector(pth)
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
if __name__ == '__main__':
    eggrecipedemo.main()And also we want to check that the pth locations are correct and that the pth
itself is in the staging area with parts:>>> demo_pth = os.path.join(parts_dir, 'demo-pth', 'demo.pth')
>>> cat(demo_pth)
/sample-buildout/virtenv/lib/python2.6/site-packages/demo-0.4c1-py2.1.egg
/sample-buildout/virtenv/lib/python2.6/site-packages/demoneeded-1.2c1-py2.1.egg
/sample-buildout/virtenv/lib/python2.6/site-packages/bigdemo-0.1-py2.1.eggIssues and helpIf you have issues or need assistance, file a bug report onWebLion’s project
siteor contact us viaIRCor email.