Package: anybox.buildbot.openerp
ContentsIntroductionMaster setupBuildoutsManifest file formatSlave setupBuildbot slave softwareSystem build dependenciesRegistration and slave capabilitiesPostgreSQL requirements and capability declarationCustom buildsCapability custom environment mappingsTweaks, optimization and trapsContributeUnit testsImprovementsChanges0.9 (2015-05-15)0.8.10.70.60.50.4.40.4.30.4.20.4.1Introductionanybox.buildbot.openerpaims to be a turnkey buildbot master setup
for a bunch of buildout-based OpenERP installations (seeanybox.recipe.openerp).It is able to run buildouts against the several postgreSQL versions
that can be found in attached slaves.Having a new OpenERP generic or custom installation buildbotted
against all the slaves attached to the
master is just a matter of copying the corresponding buildout in thebuildoutssubdirectory of the master and referencing it inbuildouts/MANIFEST.cfg.An interesting practice for buildbotting of in-house custom projects
is to put this subdirectory itself under version control with your
preferred VCS, and let the developpers push on it.It is designed not to be too intrusive to buildbot itself, so that
buildbot users can tweak their configuration in the normal buildbot
way, and even add more builds, possibly not even related to
OpenERP.The real-time scheduling works by polling the remote VCS systems
(currently for Bazaar and Mercurial only). There is a basic URL
rewritting capability to ease make this polling efficient.Master setupThese steps are for a first setup.Install this package in a virtualenv. This will install buildbot as
well.Create a master in the standard way (seebuildbotcreate-master--help).If you are creating a new buildbot master, the filemaster.cfg.sampleincluded within this package should work out of the box. Just rename itmaster.cfgand put it in the master directory.If you are extending an existing buildbot master, add these lines inmaster.cfgright after the definition ofBuildMasterConfig:from anybox.buildbot.openerp import configure_from_buildouts
configure_from_buildouts(basedir, BuildmasterConfig)Copy thebuildoutsdirectory included in the source
distribution in the master or make your own (checkbuildouts/MANIFEST.cfgfor an example on how to do
that). In previous step, one can actually provide explicit
locations for buildouts directories.Put aslaves.cfgfile in the master directory. See the includedslaves.cfg.samplefor instructions.BuildoutsThe buildouts to install and test are stored in thebuildoutsdirectory; they must be declared with appropriated options in thebuildouts/MANIFEST.cfg. The ones included with this package
are run by <http://buildbot.anybox.fr>_.Alternatively, one can specify several manifest files, to aggregate from
several sources.http://buildbot.anybox.frdemonstrates this by running:the buildouts included in this packagethe buildouts shipping withanybox.recipe.openerp. These actually play the role of integration tests for the recipe itself.other combinations of OpenERP versions and community addons that are of interest for Anybox.Manifest file formatIn this manifest file, each section corresponds to a buildout (or at
least aBuildFactoryobject).
Options are:buildout = TYPE SPECIFICATION,
where TYPE can bestandaloneor indicate a VCS (currentlyhgonly is supported).
For standalone buildouts, SPECIFICATION is a path from the buildmaster
directory.
For VCSes, SPECIFICATION takes the form URL BRANCH PATH,
where PATH is the path from a clone of URL on branch BRANCH to the
wished buildout configuration. This allows to use configuration
files withextendsand to track the buildout configuration
itself, and to reduce duplication. Buildouts from VCSes are always
updated to the head of the prescribed branch, independently of the
changes detected by the buildmaster.watch = LINES: a list of VCS locations to watch for changes (all
occurrences of this buildout will be rebuilt/retested if any change
in them). If you use a VCS buildout type, you need to register it here also
to build if the buildout itself has changed in the remote VCS.build-for = LINES: a list of software combinations that this
buildout should be run against. Takes the form of a software name
(currently “postgresql” only) and a version requirement (see
included example and docstrings inanybox.buildout.openerp.versionfor format). See also “slave
capabilities” below.build_requires: build will happen only on slaves meeting the requirements
(see also “slaves capabilities” below)
Some known use-cases:dependencies on additional software or services (LibreOffice server, postgis, functional testing frameworks)access to private source code repositoriesnetwork topology conditions, such as quick access to real-life database
dumps.db_template: the template the database will be built with. Intended
for preload of PostgreSQL extensions, such as postgis, but can be
used for testing data as well. Should be paired with a conventional
requirement expressing that the template exists and can be used.bootstrap options: any option of the formbootstrap-foowill
give rise to a command-line option--foowith the same value
for the bootstrap. Example:bootstrap-version = 2.1.0Exceptions: some options, such as--eggsor-ccan be passed this
way. They are managed internally by the configurator. The error
message will tell you.The--versionoption ofbootstrap.pyis mean to require azc.buildoutversion, thebootstrap.pyscript may itself be
more or less recent. You may specify the major version ofbootstrap.pyitself in the following way:bootstrap-type = v2..warning :: currently,bootstrap-typedefaults tov1. If itdoes not match the reality, the buildwill fail, because
command-line options have changed a lot betweenv1andv2.Slave setupWe strongly recommend that you install and run the buildslave with its
own dedicated POSIX user, e.g.:sudo adduser --system buildslave
sudo -su buildslave
cd(the--systemoption forbids direct logins by setting the default
shell to/bin/false, seeman adduser)Buildbot slave softwareFor slave software itself, just follow the official buildbot way of doing:virtualenv buildslaveenv
buildslaveenv/bin/pip install buildbot-slave
bin/buildslave create-slave --helpSystem build dependenciesThe slave host system must have all build dependencies
for the available buildouts to run. Indeed, the required python eggs may have
to be installed from pypi, and this can trigger some compilations. In
turn, these usually require build utilities (gcc, make, etc),
libraries and headers.There arepackages for debian-based systemsthat install all needed dependencies for OpenERP buildouts.Registration and slave capabilitiesHave your slave registered to the master admin, specifying the
available versions of PostgreSQL (e.g, 8.4, 9.0), and other
capabilities if there are special builds that make use of them.
See “PostgreSQL requirements” below for details about Postgresql
capability properties.The best is to provide aslaves.cfgfragment (seeslaves.cfg.samplefor syntax and
supported options).Capabilities are defined as aslaves.cfgoption, with one line per
capability and version pair. Each line ends with additionalcapability properties:[my-slave]
capability = postgresql 8.4
             postgresql 9.1 port=5433
             private-bzr+ssh-access
             selenium-server 2.3Capabilities are used forfiltering: running builds only on those that can take them (seebuild-requiresoption)slave-local conditions: applying parameters that depend on the
slave (here the port for PostgreSQL 9.1) through build properties
and environment variables. Everything is already tuned by
default for thepostgresqlcapability, but an advanced user can
register environment variables mappings inmaster.cfgfor other
capabilities.demultiplication: this is thebuild-foroption ofMANIFEST.cfg.The example above demonstrates how to use that to indicate access to
some private repositories, assuming that the master’sMANIFEST.cfgdeclares the builds that need this access:build-requires=private-bzr+ssh-accessIn some cases, it’s meaningful to further restrict a buildslave to run
only those builds that really need it. This is useful for rare or
expensive resources. Sampleslave.cfgextract for that:[mybuildslave]
build-only-if-requires=seleniumPostgreSQL requirements and capability declarationYou must of course provide one or several working PostgreSQL
installation (clusters). These are described ascapabilitiesin the
configuration file that makes the master know about your slave and how
to run builds on it.The default values assumes a standard PostgreSQL cluster on the
same system as the slave, with a PostgreSQL user having the same name
as the POSIX user running the slave, having database creation rights.
Assuming the slave POSIX user isbuildslave, just do:sudo -u postgres createuser --createdb --no-createrole \
     --no-superuser buildslaveAlternatively, you can provide host, port, and password (seeslaves.cfgfile to see how to express in the master configuration).WARNING: currently, setting user/password is not
supported. Only Unix-socket domains will work (see below).The default blank value for host on Debian-based distributions will make the
slave connect to the PostgreSQL cluster through a Unix-domain socket, ie, the
user name is the same as the POSIX user running the slave. Default
PostgreSQL configurations allow such connections without a password (identauthentication method inpg_hba.conf).To useidentauthentication on secondary or custom compiled
clusters, we provide additional capability properties:Thebinandlibshould point to the executable and library
directories of the cluster. Otherwise, the build could be run with a
wrong version of the client libraries.Ifunix_socket_directoryis set inpostgresql.conf, then
provide it as thehostcapability property. Otherwise, thepsqlexecutable and the client libraries use the same defaults
as the server, providedbinandlibare correct (see above).youmustprovide the port number if not the default 5432, because
the port identifies the cluster uniquely, even for Unix-domain socketsExamples:# Default cluster of a secondary PostgreSQL from Debian & Ubuntu
capability postgresql 9.1 port=5433

# Compiled PostgreSQL with --prefix=/opt/postgresql,
# port set to 5434 and unix_socket_directory unset in postgresql.conf
capability postgresql 9.2devel bin=/opt/postgresql/bin lib=/opt/postgresql/lib port=5434

# If unix_socket_directory is set to /opt/postgresql/run, add this:
# ... host=/opt/postgresql/runCustom buildsThere is a hook to replace the steps that run after the buildout (test
run, then log analysis) by custom ones. This is an advanced option, meant
for users that are aware of the internals ofanybox.buildbot.openerp, and notably of the properties that it
sets and uses.In the master configuration file, register a callable that
returns a list of buildbot steps. Instead of callingconfigure_from_buildouts, follow this example:from anybox.buildbot.openerp.configurator import BuildoutsConfigurator
configurator = BuildoutsConfigurator(basedir)
configurator.post_buildout_steps['mycase'] = mycase_callable
configurator.populate(BuildmasterConfig)wheremycase_callableis typically a function having the same
signature as thepost_buildout_steps_standardmethod ofBuildoutsConfigurator.
This means in particular that it can read the options dict, hence
react to its own options.Then, report themycasename inMANIFEST.cfg, in the sections
for the relevant buildouts:[mybuildout]
post-buildout-steps = mycase
...The standard build is given by thestandardkey. You can actually
chain them by specifying several such keys (one per line) in the
configuration option. Here’s a real-life example:[mybuildout]
post-buildout-steps = static-analysis
                      standard
                      docCurrently,standardis the only builtin set of post buildout steps.TODO: provide more builtin sets of post buildout steps ; refactor the
doc in two sections, the first listing them and explaining how to use
them in conf, the second explaining how to register custom ones. The
first doc would not require internal knowledge of buildbot oranybox.buildbot.openerp.Capability custom environment mappingsAs explained above, the capability system is able to set environment
variables depending on the selected buildlsave and capability
version. Of course, this is useful if the tests themselves make use
directly or indirectly of them.The environment mappings are preset forpostgresqlonly, here’s how to do
register some for another capability, frommaster.cfg. Again,
this goes by splitting througth instantiation of a configurator object
instead of theconfigure_from_buildoutshelper function:abo_conf = BuildoutsConfigurator(basedir)
abo_conf.add_capability_environ(
    'rabbitmq',
    dict(version_prop='rabbitmq_version',
         environ={'RMQ_BASE_URI': '%(cap(base_uri):-)s'),
                  'RMQ_BINARY': '%(cap(binary):-)s'),
                  'AMQP_CTL_SUDO': '%(cap(sudo):-TRUE)s'),
      }))

abo_conf.populate(BuildmasterConfig)Now withrabbitmqcapability defined this way on slaves:rabbitmq 2.8.4 base_uri=amqp://guest:guest@localhost:5672/ binary=rabbitmqctl sudo=TrueThis will setupRMQ_BASE_URI,RMQ_BINARYandAMQP_CTL_SUDOto these values.The values, in theenvironsub-dict areWithPropertiesstatement, with their entire expressivity ; just notice thecap(option_name)added syntax to refer to properties corresponding
to capability options.Tweaks, optimization and trapseggs and openerp downloads are shared on a per-slave basis. A lock
system prevents concurrency in buildout runs.Windows slaves are currently unsupported : some steps use ‘/’
separators in arguments.Donotstart the slave while its virtualenv is “activated”; also take
care that the bin/ directory of the virtualenvmust notbe on the
POSIX user default PATH. Many build steps are not designed for that,
and would miss some dependencies. This is notably the case for the
buildout step.If you want to add virtualenv based build factories, such as the
ones found inhttp://buildbot.anybox.fr(notably this distribution),
make sure that the default system python has virtualenv >=1.5. Prior
versions have hardcoded file names in /tmp, that lead to permission
errors in case virtualenv is run again with a different system user
(meaning that any invocation of virtualenv outside the slave will
break subsequent builds in the slave that need it). In particular,
note that in Debian 6.0 (Squeeze), python-virtualenv is currently
1.4.9, and is absent from squeeze-backports. You’ll have to set it
up manually (install python-pip first).ContributeAuthor:Georges Racinet (Anybox)Contributors:Stéphane Bidoul (Acsone)The primary branch is on the launchpad:Code repository and bug tracker:https://launchpad.net/anybox.buildbot.openerpPyPI page:http://pypi.python.org/pypi/anybox.buildbot.openerpPlease branch on the launchpad or contact the authors to report any bug or ask
for a new feature.Unit testsTo run unit tests for this package:pip install nose
python setup.py nosetestsCurrently,python setup.py testtries and install nose and run thenose.collectortest suite but fails in tearDown.ImprovementsSee the includedTODO.txtfile and the project on launchpad:http://launchpad.net/anybox.buildbot.openerpChanges0.9 (2015-05-15)Git support (buildbout repo + watch) (several issues on launchpad)launchpad #1201138: simple inheritance systemlaunchpad #1201175: auto watch for VCS buildout itself
(not what it references)added a ‘static-analysis’ postbuildout subfactorylaunchpad #1201099: introduce subfactories for cleaner pluggability(post buildout steps, etc)launchpad #1196310: provide a “post buildout steps” for functional testinglaunchpad #1196308: provide a “post buildout steps” for nose testinglaunchpad #1198702: bootstrap options in MANIFESTlaunchpad #1142994: url rewrites for vcs pollinglaunchpad #1154673: treeStableTimer scheduler parameter now
configurable on a per-buildout basislaunchpad #1281136: subfactory for sphinx doc compilation & uploadlaunchpad #1281137: subfactories for packaging of hg versioned buildouts0.8.1launchpad #1130838: build-only-if-requires buildslave optionUsing the uniform test launcher script provided by anybox.recipe.openerp 1.2launchpad #1086066: detecting unittest2 failures and errorslaunchpad #1086392: resilience wrt missing remote mercurial
branches by retrying one branch after the otherpost download steps for alternative presentation to buildout and
tests (allow for packaging and testing the packaged)hgtag buildout source to read from a tag expressed in propertiesquality: flake8 compliance0.7launchpad #999069: Test run parts of build factories are now customizable.launchpad #1040070: can read several manifest fileslaunchpad #1050842: now standalone buildouts paths are relative to manifest
directory.db_template buildout option.launchpad #999066: Utility script to find a free port in a rangeignore divergences in bzr branch pulls (notably for mirrors)0.6launchpad #1008985: Now buildouts can be retrieved directly from
VCSes (currently Mercurial only).launchpad #1004844: dispatching of PostgreSQL versions by
capability allows to build within a single slave against several of
them.launchpad #999116: filtering of slaves for a given build factory
(buildout) by capability.launchpad #1004916: slaves max_builds and notify_on_missingparameters now taken into account0.5using vcs-clear-retry option of OpenERP recipelaunchpad #994524: Configuration option “build-for” allows to
specify PosgreSQL version rangeslaunchpad #998829: New build-category option in MANIFEST.cfg0.4.4List of addons to install now can be specified per build factory0.4.3Documentation improvements0.4.2Documentation improvements0.4.1Initial release on pypi