Package: autobot
buildbot automationWhat is autobot?autobot is a continuous integration solution built as a front-end to
buildbot. We have a lot of software.  We’re talented, so
usually it doesn’t break.  But we’re not infalliable.  Our robot ally,
autobot, is there to test things for us.  Let’s meet autobot!Installing autobotautobot may be installed using the install script:curl http://k0s.org/hg/autobot/raw-file/tip/INSTALL.sh | bashThis will create a virtualenv and install autobot for development
($VIRTUAL_ENV/src/autobot).  You can also use the script as
instructions and install it in the same manner as any other python package.Setting up a buildmaster and slaveOnce you have autobot installed and the virtualenv activated, you’ll
want to create a buildmaster and a buildslave.You can create a master-slave pair by runningcreate-autobotafter
activating the virtualenv. This is mostly useful for autobot
development. The scriptscreate-autobot-masterandcreate-autobot-slaveare also available and are more useful for
production, as normally you will not want to keep the master and the
slave on the same machine except for development and testing.  The
scripts will prompt you for a factory or you can specify one or more
from the commandline using--factoryor-f. The factories are
fromautobot.projectsand its subdirectories.  You can list the
available factories withcreate-autobot--list-factories(orcreate-autobot-master--list-factories).Runningcreate-autobot--helpwill give the variables you can set
when it makes a new bot for you (create-autobot-masterandcreate-autobot-slavealso have a--help, the variables increate-autobotbeing a superset of these).If you don’t specify a variable, the default will be used to create
your particular bot.  You can change these in the resultant
configuration later.Using autobotThe buildmaster and buildslave are started withbuildbot start masterandbuildslave stop slavefrom the bot’s
directories. Respectivestopcommands also exist.  If you used thecreate-autobotcommand the generated bot will have arestart_buildbot.pyscript that will stop and start both the
master and slave and (if debug is set) remove the log as well.The generatedmaster.cfgfile reads values from amaster.inifile in the same directory.  The master.ini contains a number of
different sections:[:master:]contains the top level configuration for the master
as well as default settings for slavessections like[slave:slavename]indicate a slave of nameslavenameall other sections are construction parameters for factoriesThe format of the.inifile is detailed in the subsequent
section.  You may change which .ini file is used by editing themaster.cfgfile, or, if for whatever reason you don’t want to use
an .ini file as a driver you can construct the appropriate
configuration therein yourself.An overview of the build status is detailed at the waterfall display,
by default athttp://localhost:8010/waterfall. To force a build,
click on the desired builder and there will be a force build button
towards the bottom.It is important to remember that continuous integration is a safety
net, not a first line of defense.  In other words, continue to check
your codebeforesending it to autobot..ini file formatAs stated above, amaster.inifile contains three different types
of sections:[:master:]contains the top level configuration for the master
as well as default settings for slavessections like[slave:slavename]indicate a slave of nameslavenameother sections are construction parameters for factoriesWhat goes in each of these?:master:The master section contains parameters appropriate to the
buildmaster:slaveport: which port to listen for the slaves onhtmlport: which port to serve the waterfall display onpollInterval: how often to poll the repositories, in seconds [DEFAULT: 60]treeStable: how long to wait before the tree settles down before
building [DEFAULT: 60]The other defaults may be seen by runningcreate-autobot-master--help.Other parameters given in the[:master:]section are used as the
baseline defaults for each slave.  They may be overridden in eachslave:sectionslave:parameters for each slave:password: each slavemusthave a password.  Unless there’s a
reason to have a different password per slave, its probably better
to put this in the:master:sectionfactories: space-separated list of factories to run on that slave.
If the special value*is used, all factories will be run.  You
can view the factories available withcreate-autobot--list-factories. The factory name corresponds to
the directory (or module) name inautobot.projects.  If
every slave is meant to run the same factories, you can put this in
the:master:sectionos: the operating system of the slave. Should be one oflinux,win, ormac(though see TODO about future use of MozInfo
making this obselescent). You probablyshouldn’thave a default
key for this in the:master:section (though autobot won’t try
to stop you!).factory sections:  All other sections are used to build the factories.
Each parameter in a factory section is used as a constructor
(__init__) keyword argument when they are created in themaster.cfg. So a factory section like:[foo]
bar = fleem
baz = another parameterwill invoke the foo factory (lets say it maps toMyFooFactory) on
each slave like:MyFooFactory(**dict(bar='fleem', baz='another parameter'))In addition, if a factory has a special key,platform, the slave
will pass its platform information when instantiating a factory.
Currently, this is a dict with a single key,os, but more may be
added as needed.  As noted in the TODO below, ideally this would be
deprecated entirely by MozInfo but such is the interim solution.Sourcesautobot.process.factory:SourceFactoryis an abstract base class
for specifying sources.  Sources live as a member on the instance, named
(oddly)sources.  This is a dict with the key being the source
type and the value being the source to use.  Source types are
currently “git” and “hg”, though this is easily extensible.Internally, an individual source is stored as a 2-tuple with the first
item being the URL of the source and the second item being the
branch.  If the branch isNoneor not specified, this is assumed
to equate tomasterfor git ordefaultfor hg.  However, you
may also use a string in the form ofURL#branch. In addition, you
may specify a whitespace separated source, whichSourceFactorywill split into a list of sources.The generatedmaster.cfgpolls for changes on the given source,
then usesbuildbot.schedulers.filter:Changefilterin conjunction
with aSchedulerto trigger the appropriate builds.  This is done
byGitPollerandHgPollerinautobot.changes.poller. Again, more pollers can be added as
needed or the provided buildbot change sources may be utilized.While, in general, the canonical sources should be specified at the
class level, if an appropriate argument (e.g.hgorgit) is passed
in to theSourceFactoryconstructor, this will override that
source type.  This is useful for testing changes on non-canonical URLs
and/or branches.Adding a New ProjectOccassionally, you’ll need to add a new project to test.  You can add
a__init__.pyfile to a directory underautobot.projectsand,
if you have a factory therein, autobot will find it and add it to the
list of factories it knows about.There is also a script,create-autobot-project, that can create
this stub for you:Usage: create-autobot-project [options] project <output>projectis the name of the project andoutput, if specified,
is the path of the file to create.  If output is not specified, it
will find its way to a directory named for the project underautobot.projects.Several factories (BuildFactorydescendents) are inautobot.processto make building a new project easier:SourceFactory: as described above, this processes the sources
and gives a method (checkout()) for downloading them.master.cfglooks in each factory and gleems its needed
repository from itssourcesattribute (if any) in the internal
storage mechanism ofSourceFactoryand sets up schedulers
accordingly.VirtualenvFactory: inherits fromSourceFactory. creates a
python virtualenv and provides build properties%(virtualenv)s,%(python)s, and%(scripts)s(location ofbinon unix orScriptson windows with respect to the virtualenv.  It also has
afindScript()method which accepts the unix-style script name
(no extension) and returns the path to the script on the platform.PythonSourceFactory: inherits fromVirtualenvFactory. It
treatsall(for now, can be changed) sources as sources of python
packages.  In addition to checking them out in%(virtualenv)s/src, it will also install them (using%(python)s setup.py install)See theautobot.process.factoryfile for more details.  These
classes are intended as mix-ins (not my favorite pattern, but it seems
to basically be what buildbot wants you to do).  More work needs to be
done to provide for the breadth of use-cases, but what exists now is
considered a “good start” versus a “final form”.  More factories may
be added per necessity.Is your autobot being feisty?Let me know! I’d like to make autobot a solution that works for all
stake-holders, and if you’re reading this, that means you!k0scist@gmail.comTODONo software of any size is ever finished.  Here are a few things I
would like to add:email notificationssingular checkout of repos on slaves: the slaves should have a
singular master repo that is checked out once for each repo URL,
branch pair.  It is then updated as the slaves need and (e.g.)
cloned from there.  This should effectively minimize fetch time.