Package: minitage.recipe.egg
ContentsIntroductionMakina Corpus sponsored softwareminitage.recipe.eggAbstractSpecific optionsPatchesDetailled documentationInstalling eggs from index or find links, the classical way to install python packagesStatic distribution dev+static urlsOffline and newest modesInstalling eggs with a patchInstalling eggs with custom build_ext_optionsDistribute SupportCHANGELOG1.107 (2013-06-26)1.106 (2013-06-26)1.105 (2013-06-22)1.104 (2013-06-02)1.103 (2013-06-02)1.102 (2013-06-02)1.101 (2013-06-02)1.100 (2013-06-02)1.99 (2013-04-02)1.97 (2013-03-29)1.96 (2013-02-23)1.94 (2013-02-19)1.93 (2013-02-13)1.92 (2012-08-28)1.91 (2012-06-08)1.90 (2012-06-08)1.89 (2012-06-08)1.881.821.811.76 - 1.771.751.73 - 1.74:1.721.711.621.381.371.361.35IntroductionThe egg has those entry point:egg: install python eggs / packages ‘setuptoolisables’The reasons why i have rewrite yet another buildout recipes builder are:Support for downloading stuffDo not rely on easy_install dependency systemSupport on the fly patchs for eggs and other distribution.Support multiple hooks at each stage of the build system.Robust offline modeWe like pypi, but offer a mode to scan for eggs without need to check
the index,Support malformed or not indexed distributions.
In other terms, we provide an url, and the recipe builds it, that’s all.Support automaticly minitage dependencies and rpath linking.You can browse the code on minitage’s following resources:http://git.minitage.org/git/minitage/eggs/minitage.recipe.egg/http://www.minitage.org/trac/browser/minitage/eggs/minitage.recipe.eggYou can migrate your buldouts without any effort with buildout.minitagificator:http://pypi.python.org/pypi/buildout.minitagificatorMakina Corpus sponsored softwarePlanet Makina CorpusContact usminitage.recipe.eggAbstractThis recipe intends to install eggs and python softwareIts heavilly inspired by zc.recipe.eggs* and try to completly replace it whereas be API compatbile.You can use it in conjunction with the buildout.minitagificator extension which monkey patch zc.buildout to use minitage recipes.The recipe has a robust offline mode.What we can do that zc.recipe.egg wouldnt do, either at all or not in the way we want to:Don’t rely on easy_install to detect and install dependencies, that can lead to versions inccompatibilitiesHandles and preserve eggs extra dependenciesApply specific patches for eggs based on their name and them generate a specific egg with a specific version, burried in the buildout via the “versions”.Make the minitage environnent comes into the environment when building if any, making compilation steps easy if you have declared and build the neccessary dependencies.Be able to install unindexed at all stuff, just by precising url to install, that can be even an automatic checkout from any repository.You have hooks to play with the recipe, if it doesnt fit exactly to your need, you can hook for a specific egg at any point of the build.Check md5 on indexes which append md5 fragments on urls, to verify package integrityIf you need scripts generation, just use the minitage.recipe:scripts recipe, it’s a specialized recipe of this one. Its use is similar, with just a bunch more options.Specific optionsPlease look for options at :http://pypi.python.org/pypi/minitage.recipe.common#options-shared-by-all-the-recipesurlsSee the shared options for more information on how to set them.
This is how to specify a distrbituion with is not indexed on pypi and where find-links dance can not work.
This is also how to specify to install something from svn:urls = http://foo.tld/my_super_egg|svn|666 # checkout and install this egg from svn at revision 666The directory bit can be used to set the subdirectory to cd to find the setup.py
eg:urls = http://foo.tld/my_super_egg|svn|666|subdirectory # checkout and install this egg from svn at revision 666 and will cd to subdirectory to do the setup.py danceeggsA list of egg requirements to install without the version specs bit.:Plone
lxmlEGGNAME-no-activate / EGGNAME-VERSION-no-activate:
in static distribution mode, force not to activate the distribution (just build)EGGNAME-force-rebuild / EGGNAME-VERSION-force-rebuild:
in static distribution mode, force to rebuild the egg:urls=http://pil.tar.gz
PIL-force-rebuild=trueEGGNAME-patch-optionspatch binary to use for this eggEGGNAME-patch-binaryOptions to give to the patch program when applying patches for this eggEGGNAME-patchesSpecific patchs for an egg name to apply at install time:Django-patches = ${buildout:directory}/foo.patchEGGNAME-UNAME-patchesSame as previous, but will just occurs on this UNAME specifc OS (linux|freebsd|darwin)
Specific patchs for an egg name to apply at install time:Django-linux-patches = ${buildout:directory}/foo.patchEGGNAME-VERSION-patchespatches apply only on this specific versionEGGNAME-VERSION-UNAME-patchespatches apply only on this specific versionversionsDefault to buildout:versions. versions part to use to pin the version of the installed eggs.
It defaults to buildout’s oneindexCustom eggs index (not pypi/simple). It defaults to buildout’s oneinstall-previous-versioninstall previous version of an egg (recursivly on errors)find-linksadditionnal links vhere we can find eggs. It defaults to buildout’s oneextra-pathsExtra paths to include in a generated script or at build time.relative-pathsIf set to true, then egg paths will be generated relative to the script path.
This allows a buildout to be moved without breaking egg paths.
This option can be set in either the script section or in the buildout section.Specifying the python to use, two ways:pythonThe name of a section to get the Python executable from. If not specified, then
the buildout python option is used. The Python executable is found in the
executable option of the named section. It defaults to buildout’s oneexecutablepath to the python executable to use.hooksA hook is in the form /path/to/hook:CALLABLE:myhook=${buildout:directory}/toto.py:fooWhere we have toto.py:def foo(options, buildout):
    return 'Hourray'The complete possible hooks list:post-download-hookhook executed after each downloadpost-checkout-hookhook executed after each checkoutEGGNAME-pre-setup-hookhook executed before running the setup.py danceEGGNAME-post-setup-hookhook executed after running the setup.py danceBDIST EGGS OPTIONS:You can have global and per distributions bdist_ext options.(EGGNAME-)defineA comma-separated list of names of C preprocessor variables to define.(EGGNAME-)undefA comman separated list of names of C preprocessor variables to undefine.(EGGNAME-)link-objectsThe name of an link object to link against. Due to limitations in distutils and desprite the option name, only a single link object can be specified.(EGGNAME-)debugCompile/link with debugging information(EGGNAME-)forceForcibly build everything (ignore file timestamps)(EGGNAME-)compilerSpecify the compiler type(EGGNAME-)swigThe path to the swig executable(EGGNAME-)swig-cppMake SWIG create C++ files (default is C)(EGGNAME-)swig-optsList of SWIG command line options(EGGNAME-)bdistext-OPTIONNAMEwill add an entry in the setup.cfg’s bdist_egg like:OPTIONNAME = fooPatchesWhen you use patches for an egg, his version will becomeDjango-1.0-final -> Django-1.0-final-ZMinitagePatched-$PatchesNamesComputation$This name have some Z* inside to make some precedence on other eggs at the same version. (see setuptools naming scheme)After that the egg is created, the buildout is backed up and patched to point to this versionThus, you can have in your common egg cache, this egg for your specific project, and the classical one for others.
This can be interessant, for example, for the zope RelStorage patch to apply on ZODB code.Detailled documentationInstalling eggs from index or find links, the classical way to install python packagesWe need to specify a find-links entry to make the recipe find our ‘foo’ egg as it is not on pypi
As we want to show the update capability of the recipe, we will first install the oldest foo version.>>> globals().update(layer['globs'])Let’s create a buildout configuration file and a basic egg not published on pypi:>>> makedist()
>>> makedist(version="2.0")
>>> makedist(name="foo3", version="2.0")
>>> makedist(name="foo4", version="2.0")
>>> makedist(name="foo4", version="3.0")
>>> data = """
... [versions]
... foo=1.0
... [buildout]
... eggs-directory =${buildout:directory}/eggs
... download-cache=${buildout:directory}
... index=%(index)s
... parts =
...     part
... [part]
... recipe=minitage.recipe.egg
... find-links=%(index)s
... eggs=foo
... """% bsettings
>>> touch('buildout.cfg', data=data)
>>> clean()
>>> sh('bin/buildout -vvvvv install part')
bin/...
Installing part.
minitage.recipe: Installing python egg(s)...
minitage.recipe: Downloading http://...:.../foo-1.0.tar.gz in .../eggs/foo-1.0.tar.gz
minitage.recipe: Unpacking in ...
Processing foo-1.0.tar.gz...
minitage.recipe: Installed foo 1.0 (.../eggs/foo-1.0-p....egg)...Static distribution dev+static urlsYou can also install directly from urls.
We ll use it to check the already present distribution files in the cache.>>> bsettings["foo3url"] = "%sfoo3-2.0.tar.gz" % (bsettings["index"])
>>> bsettings["foo4url"] = "%sfoo4-2.0.tar.gz" % (bsettings["index"])
>>> bsettings["foo43url"] = "%sfoo4-3.0.tar.gz" % (bsettings["index"])
>>> data = """
... [versions]
... mr.developer=0.15
... [buildout]
... eggs-directory =${buildout:directory}/eggs
... versions = versions
... download-cache=${buildout:directory}
... parts =
...     part
... index=%(index)s
... [part]
... recipe = minitage.recipe.egg
... eggs=mr.developer
... [a]
... recipe=minitage.recipe.egg
... urls=
...      %(foo3url)s
...      %(foo4url)s
... """ % bsettings
>>> touch('buildout.cfg', data=data)
>>> sh('bin/buildout -vvvvvv install a') #doctest +REPORT_NDIFF
b...
Installing a...
minitage.recipe: Download archive from http://.../foo...-2.0.tar.gz...
minitage.recipe: Installed foo...
minitage.recipe: Pinning custom egg version in buildout, trying to write the configuration
minitage.recipe: CREATING buildout backup in ...
...As we are installing from an url, we must pin the version to be sure to use this egg,
even if we have some other similar egg on index or find-links.
See the versions pinned to use your downloaded stuff>>> cat('buildout.cfg') # doctest: +REPORT_NDIFF
<BLANKLINE>
[versions]
mr.developer=0.15
foo... = 2.0
foo... = 2.0
[buildout]
eggs-directory =${buildout:directory}/eggs
versions = versions
download-cache=${buildout:directory}
parts =
    part
index...
[part]
recipe = minitage.recipe.egg
eggs=mr.developer
[a]
recipe=minitage.recipe.egg
urls=
.../foo3-2.0.tar.gz
.../foo4-2.0.tar.gz...If we try to install a newer version, via an url, it will work, even if the version is pinned.:>>> data = """
... [versions]
... mr.developer=0.15
... [buildout]
... versions = versions
... download-cache=${buildout:directory}
... eggs-directory =${buildout:directory}/eggs
... parts =
...     part
... index=%(index)s
... [part]
... recipe = minitage.recipe.egg
... eggs=mr.developer
... [a]
... recipe=minitage.recipe.egg
... urls=
...     %(foo43url)s
... """ % bsettings
>>> touch('buildout.cfg', data=data)
>>> sh('bin/buildout -vvvvv install a')
b...
>>> cat('buildout.cfg') # doctest: +REPORT_NDIFF
<BLANKLINE>
[versions]...
foo4 = 3.0...If we try to rebuild the egg, we cannot, as the same egg is already built.
This is to prevent rebuilding triggered by buildout each time we launch it,
and also to delete already good versions present in the cache.:>>> sh('bin/buildout -ovvvvv install a')
b...
minitage.recipe: If you want to rebuild, please do 'rm -rf .../eggs/foo...Offline and newest modesWe have ways to make buildout not download the latest versions found
on indexes and be very conservative on what we already got local.Let’s create a buildout configuration file and a basic egg not published on pypi, and install it:>>> globals().update(layer['globs'])
>>> makedist()
>>> makedist(version='2.0')
>>> data = """
... [versions]
... foo=1.0
... [buildout]
... eggs-directory =${buildout:directory}/eggs
... download-cache=${buildout:directory}
... index=%(index)s
... parts =
...     part
... [part]
... recipe=minitage.recipe.egg
... find-links=%(index)s
... eggs=foo
... """%bsettings
>>> touch('buildout.cfg', data=data)
>>> clean()
>>> sh('bin/buildout -vvvvv install part')
bin/...
Installing part.
minitage.recipe: Installing python egg(s)...
minitage.recipe: Downloading http://...:.../foo-1.0.tar.gz in .../minitage/eggs/foo-1.0.tar.gz
minitage.recipe: Unpacking in ...
Processing foo-1.0.tar.gz...
minitage.recipe: Installed foo 1.0 (.../eggs/foo-1.0-p....egg)...Well, now, we are 1.0.
Removing the version bit, but choosing to be non newest will make
buildout not to install the new foo-2.0 version.:>>> data = """\
... [versions]
... [buildout]
... download-cache=${buildout:directory}
... eggs-directory =${buildout:directory}/eggs
... parts =
...     part
... versions = versions
... index=%(index)s
... [t]
... recipe=minitage.recipe.egg
... find-links=%(index)s
... eggs=foo
... """%bsettings
>>> touch('buildout.cfg', data=data)
>>> sh('bin/buildout -Nvvvvv install t')
b...
minitage.recipe: Installing python egg(s).
minitage.recipe: Picked: foo = 1.0
minitage.recipe: All egg dependencies seem to be installed!...Idem in offline mode.:>>> sh('bin/buildout -ovvvvv install t')
b...
minitage.recipe: Picked: foo = 1.0
minitage.recipe: All egg dependencies seem to be installed!...But then, going online/newest will trigger the installation of the 2.0 egg.:>>> sh('bin/buildout -nvvvvv install t')
b...
minitage.recipe: Picked: foo = 2.0
minitage.recipe: All egg dependencies seem to be installed!...File urls work in offline mode.:>>> data = """
... [buildout]
... download-cache=${buildout:directory}
... eggs-directory =${buildout:directory}/eggs
... index=%(index)s
... parts =
...     part
... [part]
... recipe=minitage.recipe.egg
... urls=file://${buildout:directory}/foo-2.0.tar.gz
... """ % bsettings
>>> clean()
>>> touch('buildout.cfg', data=data)
>>> sh('bin/buildout -vvvvv install part')
b...
>>> [egg for egg in os.listdir('eggs') if 'foo' in egg]
['foo-2.0-py...egg']
>>> sh('bin/buildout -o install part')
b...
>>> [egg for egg in os.listdir('eggs') if 'foo' in egg]
['foo-2.0-py...egg']Installing eggs with a patchWe need to specify a find-links entry to make the recipe find our ‘foo’ egg as it is not on pypi
As we want to show the update capability of the recipe, we will first install the oldest foo version.Let’s create a buildout installation:>>> globals().update(layer['globs'])
>>> makedist()
>>> makedist(version='2.0')
>>> touch('patch', data="""
... --- foo.old/toto.py     2013-03-30 20:42:04.575613999 +0100
... +++ foo/toto.py 2013-03-30 20:42:10.603721734 +0100
... @@ -1,3 +1,7 @@
...
...  def f():
...      print "foo"
... +
... +
... +
... +
...
... """)Patching is easy, just put your patches in YouEgg-patches.>>> data = """
... [versions]
... foo=1.0
... [buildout]
... download-cache=${buildout:directory}
... eggs-directory =${buildout:directory}/eggs
... parts = part
... index=%(index)s
... [part]
... recipe=minitage.recipe.egg
... find-links=%(index)s
... foo-patches = ${buildout:directory}/patch
... eggs=foo
... """%bsettings
>>> touch('buildout.cfg', data=data)
>>> sh('bin/buildout install part')
b...
can't find file to patch at input line 4
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|
|--- foo.old/toto.py     2013-03-30 20:42:04.575613999 +0100
|+++ foo/toto.py 2013-03-30 20:42:10.603721734 +0100
--------------------------
No file to patch.  Skipping patch.
...
minitage.recipe: Message was:
    ('Failed', 'patch -t -Np0 < ...patch')...Oups, the patch level ! .>>> data = """\
... [versions]
... foo=1.0
... [buildout]
... download-cache=${buildout:directory}
... eggs-directory =${buildout:directory}/eggs
... parts = part
... index=%(index)s
... [part]
... recipe=minitage.recipe.egg
... find-links=%(index)s
... eggs=foo
... foo-patches = ${buildout:directory}/patch
... foo-patch-options = -Np1
... """%bsettings
>>> touch('buildout.cfg', data=data)
>>> sh('bin/buildout -vvvvv install part')
b...
minitage.recipe: Running patch -t -Np1 < .../patch
patching file toto.py...
minitage.recipe: Installed foo 1.0-ZMinitagePatched-Patch...Now that we have it, try to resintall.>>> sh('bin/buildout -vvvvv install part')
bin/buildout...
minitage.recipe: We have the distribution that satisfies 'foo==1.0-ZMinitagePatched-Patch'.
minitage.recipe: Pinning custom egg version in buildout, trying to write the configuration
minitage.recipe: Version already pinned, nothing has been wroten...In all cases our buildout is patched.>>> cat('buildout.cfg')
[versions]
foo...=...1.0-ZMinitagePatched-Patch...Installing eggs with custom build_ext_options>>> globals().update(layer['globs'])
>>> makedist(version='2.0', setup="""
...
... import os
... if os.path.exists('setup.cfg'):
...     print 'setup.cfg:'
...     print '---------------'
...     print open('setup.cfg').read()
... """)Just add the bdist_ext options you want to the buildout>>> data = """
... [versions]
... [buildout]
... download-cache=${buildout:directory}
... eggs-directory =${buildout:directory}/eggs
... parts =
...     part
... index=%(index)s
... [part]
... recipe=minitage.recipe.egg
... find-links=%(index)s
... define=Foo=1,Bar=2
... bdistext-foo = bar
... bdistext-bar = titi
... foo-bdistext-foo = tutu
... foo-swig-cpp = toto
... eggs=foo
... """% bsettings
>>> touch('buildout.cfg', data=data)
>>> sh('bin/buildout -vvvvv install part')
b...
minitage.recipe: Using bdist_ext option: define = Foo=1,Bar=2
minitage.recipe: Using bdist_ext option: bar = titi
minitage.recipe: Using bdist_ext option: foo = bar...
minitage.recipe: foo: Using bdist_ext option: swig-cpp = toto
minitage.recipe: foo: Using bdist_ext option: foo = tutu...
setup.cfg:
---------------
[egg_info]
tag_build =
tag_date = 0
tag_svn_revision = 0
<BLANKLINE>
[build_ext]
foo = tutu
bar = titi
swig-cpp = toto
define = Foo=1,Bar=2...
setup.cfg:
---------------
[egg_info]
tag_build =
tag_date = 0
tag_svn_revision = 0
<BLANKLINE>
[build_ext]
foo = tutu
bar = titi
swig-cpp = toto
define = Foo=1,Bar=2...
minitage.recipe: All egg dependencies seem to be installed!...Distribute Support>>> globals().update(layer['globs'])If we are in distribute mode, it would significate that distribute is installed and is available on sys.path.
If so, we detect that and we will just install distribute rather than setuptools if any requirement wants setuptools.
And even if the version is fixed.>>> data = """
... [buildout]
... versions = versions
... eggs-directory =${buildout:directory}/eggs
... index=%(index)s
... parts = part
...         s
... [part]
... recipe=minitage.recipe.egg
... eggs = virtualenv
... [s]
... recipe=minitage.recipe.scripts
... eggs = virtualenv
... [versions]
... """ % bsettings
>>> touch('buildout.cfg', data=data)Running buildout with distribute to see that it’s picked.>>> sh('bin/buildout -vvvvvv install part s')
bin/buildout...
minitage.recipe: Using distribute!...CHANGELOG1.107 (2013-06-26)fix previous patch1.106 (2013-06-26)try to install at first with a clean pythonpath (only setuptools)1.105 (2013-06-22)handle last setuptools1.104 (2013-06-02)rebuild force mode for static distributions1.103 (2013-06-02)again, fix1.102 (2013-06-02)again, fix1.101 (2013-06-02)fix race condition in install from urls1.100 (2013-06-02)fix install from urls1.99 (2013-04-02)QA release (pep8, coverage, new test layer & contineous integration)1.97 (2013-03-29)Support for looking distribution inside default cache directoryOffline bugfix, load correctly source dist from all download caches1.96 (2013-02-23)avoid circular error (buildout2 again) [kiorky]1.94 (2013-02-19)buildout2 compat again1.93 (2013-02-13)buildout2 compat1.92 (2012-08-28)fix extras handling in some border cases1.91 (2012-06-08)less verbose again1.90 (2012-06-08)less verbose1.89 (2012-06-08)fix some dependency tree bugs [kiorky]1.88compatibility with buildout.dumppickledversion1.82more compatible with allow-picked-version == False1.81handle comments in versions bloc.1.76 - 1.77Some distribute/setuptools fixesfix testsbetter error output1.75better handle of download problems1.73 - 1.74:Fix for cross platform buildsIt allows lot of things on OSX(mostly run buildout with a flat compiled pythonand use a targeted python compiled for i386)1.72fix to avoid conflicterror when you fix versions in buildout.cfg and somerone fix another in setup.py, it just display a big warning1.71better support for distribute, again.1.62specific dir changepath fixedmodify how environ is computedugly bugfix when your egg cache is not on the same partition as thebuildout as it make cross device references errors when renaming from
sandboxzipped egg bugfixmake sdist take precedence over eggs when you are patching on the fly.bugfix for compatibility with python2.4 and urlparse(tuple)add bdist_ext options support like zc.recipe.egg:custommake recipe cooler with distribution filename identifierssplit tests in multiple files1.38decorator helper1.37fix exception mappings when a patch fails1.36fix tests develop eggs1.35splitted out from minitage.recipe