Package: ursabot
Ursa Labs' buildbot configuration for Apache ArrowUrsabot is a continous integration framework based on thebuildbotframework. The primary focus of ursabot is to
execute various builds benchmark and packaging tasks forApache Arrowhoweverursabotcan be used for arbitrary projects.Notable featuresa standalone project abstraction to make the project configurations module
and reusable, and a less verbose master configuration supporting multiple
projectslocally reproducible builds via command line interfaceattachable interactive shells to the docker workers in case of build failureslocal source mounting for docker workersdeclerative builder configuration and a docker builder which makes it easier
to work with docker latent workersextended github hook to drive buildbot via github commentsclick based comment parserimproved change filter to filter changes based on build propertiesreimplemented github reporters: status-, comment- and review reporterseasily extensible formatter classes to use with the reimplemented reporterssteps implemented based on new-style ShellCommand stepa token rotator to use multiple github tokens with github servicesa docker image tool to maintain and build hierachical docker imagescommand line interface and additional utilitiesDriving UrsabotAllowing PR reviewers to request additional checks on demand within the review
process makes it easier for us to apply extra scrutiny at review time while
also conserving CI bandwidth by using human expertise to know which checks are
needed.via CommentsUrsabot receives github events through a webhook. It listens on pull request
comments mentioning @ursabot. It follows the semantics of a command line
interface, to see the available commands add a comment on the pull request:@ursabot --help.The @ursabot GitHub user will respond orreactthat it has
started a build for you. The command parser is implemented incommands.py.Currently available commands:@ursabot build: Triggers all the ursabot tests. These tests are run
automatically, but this is a convinient way to force a re-build.@ursabot benchmark: Triggers C++ benchmarks and sends back the results as
a github comment and highlights the regressions.@ursabot crossbow test cpp-python: Triggers thecpp-pythontest group
defined intest.ymland responds with a URL pointing to
submitted crossbow branches at the github UI showing the build statuses.@ursabot crossbow package -g wheel -g conda: Triggers thewheelandcondacrossbow packaging groups defined intasks.yml.@ursabot crossbow package wheel-win-cp35m wheel-win-cp36m: Triggers only
two tasks passed explicitly.Note that the commands won't trigger any builds if the commit message contains
a skip pattern, like[skip ci]or[ci skip]. In order to drive ursabot
the user must have either 'OWNER', 'MEMBER' or 'CONTRIBUTORroles.via the Web UIYou can also initiate a build for a specific architecture/configuration in thebuildbot UI. Navigate toBuilds > Builders,
select a builder, and clickBuild apache/arrowbuttin at the top right. This
triggers the force schedulers where you can specify a branch and/or commit to
build. In the future specialized builders will have different fields to provide
the neccessary information.via CLIBuildbot supports submitting local patches directly to the cluster and
triggering specific builders. TheTryScheduleris a really handy way to test
local changes without polluting the git history:buildbottry\--connect=pb\--master=...\--username=...\--passwd=...\--get-builder-namesIf someone wants to use this feature then please raise an issue, because it
requires custom credentials.Install ursabot and the CLIRunning it locally helps with the development and testing new feature and/or
debugging issues without touching the production instance.Installation requires at least Python 3.6:cd/path/to/ursabot
pipinstall-e.Now theursabotcommand is available which looks for amaster.cfgfile in
the current directory.master.cfgcan be passed explicitly via the--configoption:ursabot-cpath/to/master.cfgDescribe the loaded master configuration:ursabotdescDescribe the loaded project configuration:ursabotprojectdesc# for master configs with a single projectursabotproject-parrowdesc# for master configs with multiple projectsHow to validate the configurationsThecheckconfigcommand runs sanity checks and various validations on the
master configuration. Most of the time ischeckconfigpasses then the master
can be run successfully (unless there are some variables only available at
runtime).ursabotcheckconfigursabotcommand loadsmaster.cfgfrom the current directory by default, but--configargument can be passed to explicitly define a configuration file.ursabot-carrow/master.cfgcheckconfigThe top-levelmaster.cfgcontains the production configuration for
ci.ursalabs.org so it requires additional dependencies likepass.
To installpass:whichapt&&sudo-Haptinstall-V-ypass
whichbrew&&brewinstallpassRun a local instance of UrsabotAfter installation master's database must be initialized:ursabot-vupgrade-masterStart/stop/restart the master:ursabot-vstart|stop|restartDefine the configuration environment (prod|test) and start the service:exportURSABOT_ENV=test# this is the defaultbuildbotrestartursabot
tail-fursabot/twisted.logThen openhttp://localhost:8100in the browser.Commands for local reproducibilityBuilders can be run locally without the web interface using theursabot project buildcommand.TestingAMD64 Conda C++builder on master:ursabotprojectbuild'AMD64 Conda C++'TestingAMD64 Conda C++builder with github pull request number 140:ursabotprojectbuild-pr140'AMD64 Conda C++'TestingAMD64 Conda C++with local repository:ursabotprojectbuild-s~/Workspace/arrow:.'AMD64 Conda C++'Where~/Workspace/arrowis the path of the local Arrow repository and.is the destination directory under the worker's build directory (in this case:/buildbot/AMD64_Conda_C__/.)Passing multiple buildbot properties for the build:ursabotprojectbuild-pprop=value-pmyprop=myvalue'AMD64 Conda C++'Attach on failureUrsabot supports debugging failed builds with attach attaching ordinary shells
to the still running workers - where the build has previously failed.Use the--attach-on-failureor-aflags.ursabotprojectbuild--attach-on-failure`AMD64CondaC++`Configuring UrsabotThe buildmaster configuration happens in themaster.cfgfiles. Originally
buildbot loads the dictionary calledBuildmasterConfig, but to make it more
flexible and moduler ursabot introduces theProjectConfigandMasterConfigabstractions.ProjectConfigcontains all the relevant information for testing a project
like Apache Arrow or Ursabot itself.ProjectConfigcan be run alone, it must
be passed to aMasterConfigobject which provides a thin abstraction over
the original buildbotBuildmasterConfig. OneMasterConfigcancontain multipleProjectConfigobjects.Including other project configurationsmakes it possible
to maintain the project relevant settings within the projects' repositories
instead of a decoupled one dedicated for the buildmaster.Adding a new build(er)sThe closest abstraction to the traditional yaml based CI configs in ursabot are
the Builders. In the simplest case a builder is defined by a sequence of steps
which are executed as shell commands on the worker.
The following example builder presumes, thatapt-getandgitis available
on the worker.frombuildbot.pluginsimportutil,workerfromursabot.stepsimportShellCommandfromursabot.buildersimportBuilderfromursabot.schedulersimportAnyBranchSchedulerrepo='https://github.com/example/repo'classTestBuilder(Builder):tags=['example-build','arbitrary-tag']steps=[GitHub(name='Clone the test repository',repourl=repo,mode='full'),ShellCommand(name='Install dependencies',command=['apt-get','install','-y'],args=['my','packages']),ShellCommand(name='Execute tests',command=['my-custom-test-runner',util.Property('test-selector')])]# in the master.cfglocal_worker=worker.LocalWorker('my-local-worker')simple_builder=TestBuilder(workers=[local_worker],properties={'test-selector':'all'})scheduler=AnyBranchScheduler(name='my-scheduler-name',builders=[simple_builder])project=ProjectConfig([name='example/repo',repo='https://github.com/example/repo'workers=[local_worker],builders=[simple_builder],schedulers=[scheduler]])master=MasterConfig(title='TestConfig',projects=[project])TheDockerBuilderprovides more flexibility, faster builds and better worker
isolation, Ursabot usesDockerBuildersextensively.fromursabot.dockerimportDockerImagefromursabot.buildersimportDockerBuilderfromursabot.workersimportDockerLatentWorkerminiconda=DockerImage('conda',base='continuumio/miniconda3',arch='amd64',os='debian-9')classTestDockerBuilder(DockerBuilder):tags=['build-within-docker-container']steps=[# checkout the source codeGitHub(args0),# execute arbitrary commandsShellCommand(args1),ShellCommand(args2),# ...]docker_worker=DockerLatentWorker(name='my-docker-worker'arch='amd64'password=None,max_builds=2)# instantiates builders based on the available workers, the Builder's# images and the workers are matched based on their architecturedocker_builders=TestDockerBuilder.combine_with(workers=[docker_worker],images=[miniconda])scheduler=AnyBranchScheduler(name='my-scheduler-name',builders=docker_builders)project=ProjectConfig([name='example/repo',repo='https://github.com/example/repo'images=[miniconda],workers=[docker_worker],builders=docker_builders,schedulers=[scheduler]])master=MasterConfig(title='TestConfig',projects=[project])Define docker imagesArrow supports multiple platforms, has a wide variety of features thus a lot of
dependencies. Installing them in each build would be time and resource
consuming, so ursabot ships docker images for reusability.There is a small docker utility inursabot.dockermodule to define
hierachical images. It uses a DSL implemented in python instead of plain
Dockerfiles. A small example to demonstrate it:fromursabot.dockerimportDockerImage,ImageCollectionfromursabot.dockerimportRUN,ENV,CMD,ADD,apt,condaminiconda=DockerImage(name='conda',base='continuumio/miniconda3',arch='amd64',os='debian-9')pandas=DockerImage(name='pandas',base=miniconda,steps=[RUN(conda('pandas'))])pyarrow=DockerImage(name='pyarrow',base=miniconda,steps=[RUN(conda('pyarrow'))])images=ImageCollection([miniconda,pandas,pyarrow])# create a docker image for each of the previous ones running jupyter notebookjupyter_steps=[RUN(conda('jupyter')),CMD(['jupyter','notebook','--ip','0.0.0.0','--no-browser','--allow-root'])]images.extend(DockerImage(name=image.name,base=image,tag='jupyter',steps=jupyter_steps)forimageinimages])# build all of the images in topological orderimages.build()# filter the imagesprint(images.filter(name='pyarrow',tag='jupyter'))Try running jupyter with pyarrow pre-installed:dockerrun-p8888:8888amd64-debian-9-pyarrow:jupyterUrsabot has a CLI interface to build the docker images:ursabotdockerbuild--helpTo list Arrow C++amd64condacppimages:ursabot--verbosedocker--archamd64--variantconda--namecpplistAdditional filtering:ursabotdocker--archamd64list
ursabotdocker--archamd64--variantcondalist
ursabotdocker--archamd64--variantconda--namecpplist
ursabotdocker--archamd64--variantconda--namecpp--tagworkerlist
ursabotdocker--archamd64--variantconda--namecpp--osdebian-9listTo build and push Arrow C++amd64condacppimages:ursabot--verbosedocker--archamd64--variantconda--namecppbuild--pushTo build and push allarm64v8alpineimages:ursabot--verbose\docker--docker-hosttcp://arm-machine:2375--archarm64v8--osalpine-3.9\build--pushDeveloping UrsabotBuildbot doesn't distribute its testing suite with binary wheels, so in order
to run the unit tests buildbot must be installed from source:pipinstall--no-binarybuildbot-e.
pytest-vursabotPre-commit hooksInstallpre-committhen to setup the githooksrunpre-commit install.Adding new workers to ci.ursalabs.orgAdding docker latent workers requires a worker entry in theworkers.yamlconfiguration.
Name, architecture and a docker host (accessable by the buildmaster) are
required, see an example inworkers.yaml.
Adding non-docker workers are also possible, but must register them in themaster.cfg.Possible further improvementsThese have been discussed and would be valuable, but they are definitely
"nice to haves" and should be deferred until the primary goals are met.Database for storing benchmark resultsCentral station for hosting the build artifactsDashboard showing build health across all platforms and configurationsMore closely Ursabot related:Multi-master setup for scalingSetup WAMP/Crossbar to restart the buildmaster without cancelling the running
buildsWindows containers and workers (docker in virtualized nodes)