Package: keas.build
A set of tools for managing multi-package software deployments with buildout.keas.buildManage large multi-package projects with easekeas.buildis a command line tool for quickly creating new
eggs and buildout configurations for an application that has multiple
interdependant packages.  As an example, consider a web application
that helps you manage your twitter followers, called Twollo (short
for twitter followers), you may have several different python packages
which handle different aspects of your application:twollo.integration- a package for managing integration with
twittertwollo.web- a web front end for thetwollo.integrationpackagetwollo.rest- a rest API for working withtwollo.integrationtwollo.utils- a package filled with random utilities and
stuff.Usingkeas.buildyou will be able to manage the lifecycle of these
python packages and any number of deployment configurations
coherently.  Specifically,keas.buildwill:Allow you to define a Project, which is a collection of
interdependent eggs that generally get released together.Automatically create new egg releases of each package when necessary.Upload new eggs to a private egg repository.Generate versioned buildout configuration files that combine the
eggs properly.Upload the buildout configuration files to a private configuration server.Upload dependent buildout configuration files to a private configuration
server (by checking the extends= chain)InstallationInstalling a released version of keas.buildYou can useeasy_installto get the latest release:$ easy_install keas.buildInstalling a developer version of keas.buildCheckout the code:$ svn checkout svn://svn.zope.org/repos/main/keas.build/trunk keas.build
$ cd keas.buildRun the bootstrap script and buildout:$ python bootstrap.py
$ ./bin/buildoutRunning thebuild-packagescriptOnce installation is complete, you should be able to run thebuild-packagescript. (Found in./bin/build-packagewith developer
installations):$ build-package
Usage: build-package [options]

Options:
  -h, --help            show this help message and exit
  -c FILE, --config-file=FILE
                        The file containing the configuration of the project.
  -q, --quiet           When specified, no messages are displayed.
  -v, --verbose         When specified, debug information is created.
  -d, --use-defaults    When specified, no user input is required and the defaults are used.
  -o, --offline-mode    When set, no server commands are executed.
  -n, --next-version    When set, the system guesses the next version to generate.
  -b BRANCH, --use-branch=BRANCH
                        When specified, this branch will be always used.
  -i BRANCHES --independent-branches=BRANCH1 BRANCH2,
                        When specified, the system guesses the next version from all this branches.
  --no-upload           When set, the generated configuration files are not uploaded.
  --no-branch-update    When set, the branch is not updated with a new version after a release is created.Getting StartedAssumptionsFirst we need to start with some assumptions that keas.build has about
the way packages are laid out. When developing multiple packages in
tandem, it often makes sense to have your subversion repository laid
out like so:SVNROOT/packages/
    branches/
        Twollo-0.x/
            ...
        Twollo-1.x/
            twollo.integration/
            twollo.rest/
            twollo.utils/
            twollo.web/
    tags/
        twollo.integration-1.2/
        twollo.integration-1.3/
        twollo.web-1.7/
        twollo.web-1.8/
        twollo.web-1.9/
        ...
    trunk/
        twollo.integration/
        twollo.rest/
        twollo.utils/
        twollo.web/The important thing to note is that each package doesnothave its
own branches/ tags/ trunk/ directories, but rather there is just one
set of the entire “project.”Defining a Project configuration fileBefore you can really do anything with thebuild-packagescript,
you have to define a configuration file. Project configuration files
use the INI[1]file format.  Every project configuration file
must have a[build]section.  The project configuration file for
the Twollo would look something like this:# Twollo.cfg

[build]
name = Twollo  #this has nothing to do with the twollo package namespace
version = +
template = Twollo-Release-Template.cfg
package-index = https://build.twollo.com/eggs/
package-index-username = someuser
package-index-password = somepass
buildout-server = https://build.twollo.com/buildouts/
buildout-server-username = someuser
buildout-server-password = somepass
svn-repos = https://svn.twollo.com/svn/packages/
svn-repos-username = somesvnuser
svn-repos-password = somepass
packages = twollo.integration
           twollo.web
           twollo.rest
           twollo.utilsLet’s go over each of the settings in thebuildsection ofTwollo.cfg.name- This is the name of the project. It can be anything you
want and has nothing to do with the packages that make up the
project.  The name will be part of the generation buildout
configuration files.version- This is the version to use when making a new release
of the Project.  The version number becomes part of the filename for
the generated buildout configuration files.Using+as the version will simply increment the version
number of the project from the versions that have already been
released.template- This is a base buildout configuration file to use for
all deployments.  When a new Project release is created, the[versions]section will automatically be updated with the
correct versions of each of thetwollow.*packages.  More on
this later.tag-layout- Choose fromflatorsubfolderflatTags will be created in svn as /tags/package-version
This is the default setting.subfolderTags will be created in svn as /tags/package/versionupload-type- Choose frominternalorsetup.pyinternalUpload packages to a WebDAV enabled web server using the
below credetials. This is the default setting.
(actually does apython setup.py sdistand uploads the result)setup.pyExecutespython setup.py sdist register upload,
does nothing else as this command should take care of the upload.package-index- The url to a WebDAV[2]enabled web
server where generated eggs for each of thetwollow.*packages
should be uploaded. Used for upload only ifupload-typeisinternal.
Also used to check/get existing versions of packages.package-index-username- The username for accessing the WebDAV
serverpackage-index-password- The password for accessing the WebDAV
serverbuildout-upload-type- Choose fromwebdav,localormypypiwebdavUpload generated buildout files to the url specified bybuildout-serverwith the WebDAV protocol.localJust generate buildout files, don’t upload them.
Ifbuildout-serveris given buildout files will be copied to that
folder.mypypiUpload generated buildout files to the url specified bybuildout-server. The url should point to the mypypi upload page.
(Something likehttp://yourhost/++projects++/)buildout-server- The url to a WebDAV enabled web
server where generated buildout files should be uploaded.
Ifbuildout-upload-typeislocalthis is a path on the local
filesystem. Buildout files wil be copied to this folder.
If not given, the process stops after releasing the packages.buildout-server-username- The username for accessing the WebDAV
serverbuildout-server-password- The password for accessing the WebDAV
serversvn-repos- The url for the subversion repository where all the
source code lives, including release tags.svn-repos-username- The username for the url repository.
Use the command line option--force-svnauthto force all svn operations
to use this credential.
Otherwise cached authentication will be used.svn-repos-password- The password for the url repository.hash-config-files- Add hashes based on file content to dependent config
filenames.packages- a list of packages that are part of the project.
These are the packages that live in the svn repository and that
should be released in conjunction with each other.Defining a Release TemplateAs we saw in the previous section,Twollo.cfgrefers to a file
calledTwollo-Release-Template.cfg.  This is just a base buildout
configuration.  Additional to this, we can also define different configuration
data as define in stage and production section. Such sections can get used in
a product deployment as additional (shared) variables.  For the Twollo project,
it might look like this:# Twollo-Release-Template.cfg

[buildout]
extends = http://download.zope.org/zope3.4/3.4.0/versions.cfg
parts = test
find-links = https://build.twollo.com/eggs/

[test]
recipe = zc.recipe.testrunner
eggs = twollo.web
       twollo.integration
       twollo.utils
       twollo.rest

[twollo-app]
recipe = zc.zope3recipes:app
servers = zserver
site.zcml = <include package="twollo.web" file="app.zcml" />
eggs = twollo.web

[zope3]
location =

[stage]
memcached = 127.0.0.1:11211

[production]
memcached = 10.0.0.1:11211When a new release of the Twollo project is made, a[versions]section will be added to this configuration file with all the
correcttwollow.*versions pinned down.Defining Multiple Deployment ConfigurationsEach time you release a Project, you may want to generate different
buildout configuration files for all the different deployment
environments you might have.  For example, you may have three
different environments: Development, QA, and Production. These are
called variants.  Each environment may need to have the application
run on different ports, at different log levels, or have other small
differences.We can easily generate additional configuration variants by adding
addtional sections to theTwollo.cfgfile:# Twollo.cfg

[Development]
template = Twollo-Instance-Template.cfg
vars = stage
port = 9080
logdir = /opt/twollo/dev/logs
install-dir = /opt/twollo/dev
loglevel = debug
cache-size = 1000

[QA]
template = Twollo-Instance-Template.cfg
vars = stage
port = 9082
logdir = /opt/twollo/qa/logs
install-dir = /opt/twollo/qa
loglevel = info
cache-size = 1000

[Production]
template = Twollo-Instance-Template.cfg
vars = production
port = 8080
logdir = /var/log/twollo
install-dir = /opt/twollo/
loglevel = warn
cache-size = 200000We can then have a singleTwollo-Instance-Template.cfgfile that
uses python’s built in string templating to access the variables we
set inTwollo.cfg.  For the Twollo project, it might look like this:# Twollo-Instance-Template.cfg

[buildout]
parts += twollo
directory = %(install-dir)s

[database]
recipe = zc.recipe.filestorage

[twollo]
recipe = zc.zope3recipes:instance
application = twollo-app
zope.conf =

  <product-config memcached>
    memcached %(memcached)s
  </product-config>

  <zodb>
    cache-size %(cache-size)s
    <filestorage>
      path ${database:path}
    </filestorage>
  </zodb>

  <server>
    type WSGI-HTTP
    address %(port)s
  </server>

  <eventlog>
    level %(loglevel)s
    <logfile>
      formatter zope.exceptions.log.Formatter
      path %(logdir)s/twollo.log
    </logfile>
  </eventlog>

  <accesslog>
    <logfile>
      level info
      path %(logdir)s/twollo-access.log
    </logfile>
  </accesslog>As you can see, the Twollo.cfg uses additional vars (stage, production) which
makes it very simple to define a lagrge amount of shared attributes in a
release template and use them in the instance template. Note, a side effect
from the python config parser is, that you will inherit arguments define in a
duplicated section define in an template loaded via (buildout) extends.Releasing a ProjectOnce you have created all the necessary configuration files, you can
make your first project release.  This is where thebuild-packagescript comes in to play.  The first time you run thebuild-packagescript, the only option you need to pass in will be the configuration
file.Thebuild-packagescript will prompt you for version information
about each of the packages it will be releasing as part of the
Twollo.cfg project.  Your first interaction with the script might look
like this:$ build-package -c Twollo.cfg --quiet
Version for `twollo.integration` : 1.0.0
The release twollo.integration-1.0.0 does not exist.
Do you want to create it? yes/no [yes]: yes
Version for `twollo.rest` : 1.0.0
The release twollo.rest-1.0.0 does not exist.
Do you want to create it? yes/no [yes]: yes
Version for `twollo.utils` : 1.0.0
The release twollo.utils-1.0.0 does not exist.
Do you want to create it? yes/no [yes]: yes
Version for `twollo.web` : 1.0.0
The release twollo.web-1.0.0 does not exist.
Do you want to create it? yes/no [yes]: yesThe next time you make a release, you can set the-nflag forbuild-packageto automatically guess the next version that should
be released.  It does this by first looking for all the release tags
of a given package and finding the last changed revision for the trunk
of a given package.  If any code for the given package was changed
since the last time it was released, it will bump the most minor
version number automatically.  If no change has occured, it will
choose the latest existing release.You can also use the-dflag to makebuild-packagenot prompt
you before creating a new release.If you need to make a new release from a particular branch, you can
use the-boption.  For example, if bug fixes have been made to the
Twollo-1.x branch, we can create a new release using code from this
branch like this:$ build-package -c Twollo.cfg -nb Twollo-1.xWhen the new package versions are calculated, they will be versioned
along the 1.x line, even if you have since created 2.x releases, by
analyzing the name of the branch.Caveat when using-nand-don a branch with a branchname ending
with the version number is that you’ll need to have the package versions
matching the branch version.
E.g. having a branch:branches/twollo-1.9will suppose packages liketwollow.web-1.9.xandtwollow.utils-1.9.xand so on.
You should be aware of this also when releasing packages from the trunk.
Most probably you’ll drive development on the trunk and branch out for a
stable. In this case package versions on the branch should be kept inline.Installing a Released Projectkeas.buildalso comes with a very simple installation script that
can be used to quickly install any variant of the released project:$ install --help
Usage: install [options]

Options:
  -h, --help            show this help message and exit
  -u URL, --url=URL     The base URL at which the releases can be found.
  -p PROJECT, --project=PROJECT
                        The name of the project to be installed.
  -V VARIANT, --variant=VARIANT
                        The variant of the project to be installed.
  -v VERSION, --version=VERSION
                        The version of the project to be installed.
  -l, --latest          When specified, the latest version will be chosen.
  --username=USER       The username needed to access the site.
  --password=PASSWORD   The password needed to access the site.
  -b PATH, --buildout-path=PATH
                        The path to the buildout executable.
  --quiet               When specified, no messages are displayed.
  --verbose             When specified, debug information is created.For example, to install the latest QA version of theTwolloproject,
you would run:$ install -uhttps://build.twollo.com/buildouts/-p Twollo -V QA –latestCreating Helper ScriptsSometimes it can be a pain to remember what all the command line
options are that you need to pass for building your project.
Fortunately, it is really easy to create helper scripts that just set
some defaults for you.For example, to create a build-twollo script, you would add the
following to a buildout configuration file:[build-twollo]
recipe = zc.recipe.egg
eggs = keas.build
scripts = build=build-twollo
initialization =
    sys.argv[1:1] = ['-c', 'Twollo.cfg']As another example, you could create an install-twollo-dev script that
automatically installs the latest development release:[install-twollo-dev]
recipe = zc.recipe.egg
eggs = keas.build
scripts = install=install-twollo-dev
initialization =
    sys.argv[1:1] = ['-u', 'http://build.twollo.com/buildouts/',
                     ' --username', 'someuser', '--password', 'somepass',
                     '-p', 'Twollo', '-V', 'Development', '--latest']The possibilities are endless!Footnotes[1]Seehttp://en.wikipedia.org/wiki/INI_file[2]Seehttp://en.wikipedia.org/wiki/WebDAVCHANGES0.4.1 (2013-11-28)bugfix: prevent generating bad buildout option key/value seperators. Generate+=instead of+ =used for e.g. buildout parts options. Otherwise it
fails since buildout 2.00.4.0 (2013-11-24)feature: implemented a new argumentvarsin deployment template section. The
newvarsargument can get used for define a release template section. If
defined, the build process will apply all arguments from this section as they
would be a a part of the project section. This allows to inherit a large
amount of arguments in a project definition instead of define them in
each project. This is usefull if you have a stage and many production variants
and the instance uses config data like a database setup and you need to
share them. For such a usecase you can simply define a stage and production
section and define all shared arguments and use the rightvarssection in
your product deployment setup. See the memcached sample in index.txtNote, this new product deployment templatevarsargument could be a problem
for your existing setup if you use a vars argument with a value similar then
a section name defined in your section template.0.3.1 (2013-09-25)Improvement: addeddefault-package-versioncommand line option
to avoid interactive questions on new packages0.3.0 (2012-12-27)feature: added -i –independent-branches option. This option will force
to check if the last release was made from the same branch we are
releasing from. This is required if you develop a new software generation
in a branch which is independent from the trunk. Previous version of
keas.build where only able to handle branch releases as bug fix releases
and didn’t make sure that we don’t mix trunk and branch releases. Now with
the -i option we force that all released packages will be made or reused
based on the current trunk or branch (-b trunk,branch)added more logging infos for find or skip next version which makes is simpler
to see what’s going on0.2.2 (2011-08-29)Improvement: addcurrent-datetime,current-date,current-timevariables0.2.1 (2011-04-07)Fix RawConfigParser usage, it destroys options values by making them
all lowercase.0.2.0 (2011-04-06)Bump versions to ZTK 1.1Improvement: add SVN repo infos to the project config file.
Yah, I know this can be checked anytime, but adding this saves a lot of time.Improvement: add optionhash-config-files0.1.8 (2010-05-11)Fix: Do NOTregisterwithpython setup.pyFix: bump setuptools and zc.buildout versionsFix: 0.1.7 tar.gz was broken0.1.7 (2010-04-26)BUGFIX: the dependent config file gathering killed the version pins
in the master config fileImprovement: PYPI-like simple index support when checking package versionsImprovement: Check dependent configs, upload all to the server.Improvement: Add--force-versionoption.Improvement: Add version to svn log comment. That makes life easier (at least
with TortoiseSVN)Improvement: Add--force-svnauthoption.Improvement: Add--directoryoption toinstall.0.1.6 (2009-11-2)Improvement: When determining released versions of a branch, support
branch names ending with .x, as in MyProject-0.3.x0.1.5 (2009-10-16)Improvement: add username and password on install to the URL that buildout
gets for the .cfg file
Let’s hope that buildout does not leave that password around.Improvement(?) or fix: remove twisted dependency that was used in deploy
for SSHImprovement: removed dependency on lxml.  Now we just use pythons
built-in xml library.Improvement: addedbuildout-upload-typeoption.
See doc for further infoBug Fix: multiline template option values crashed on parsingBug Fix: re did not like non-text parameters coming from BeautifulSoupImprovement: added the--timeoutoption to installImprovement: added mypypi buildout file upload supportBug Fix: revert to checkout of the complete source tree0.1.4 (2009-10-01)Bug Fix: installer script would crash when -u option was missing a
trailing /Bug Fix: installer script would crash when no variants were found.Improvement: Building a package will no longer checkout the entire
branch just to update the setup.py file with the new version
number.  Instead only the top level directory is checked out.Bug Fix: project build script would crash when variant
configuration was missing information required by its template. Now
a helpful error message is printed and no files will be uploaded
until all have been created successfully.Bug Fix: Typing Ctrl+c on the command line no longer causes a
KeyboardInterrupt traceback to be spit out.Bug Fix: when running the installer, if the buildout command prompts
the user for input, the installer will no longer eat up the prompt.0.1.3 (2009-09-30)Initial public release.0.1.1 (internal)Bug Fix: If a project variant is specified for which no release have
been made, the build script now exits nicely with a human readable
error message.0.1.0 (internal)Initial release.