Add layout file validation.

Based on voluptuous library.

Basic validation should catch typos, missing or extra attributes.
Can be expanded to do more serious validation (ie, specifying
a comment in a trigger should require the event be comment-added).

Adds a command line option to validate a named layout file and
exit.

(Also add dist/ to .gitignore.)

Change-Id: Ia864ebde1765141d4d1a52bc77033689b6210e81
Reviewed-on: https://review.openstack.org/19443
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Make report message introduction configurable.

By setting parameters on a pipeline, you can change the default
"Build successful/failed" introductory messages left on Gerrit.

Change-Id: Ie05c8c87a47994faced4ee67fda4ab4e64ea0d20
Reviewed-on: https://review.openstack.org/19677
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
sphinx is now stricter

The documentation files often have minor typo or badly formatted
commands which makes sphinx emits a warning. To make sure we always
catch them via the Jenkins build, this patch makes sphinx to convert
warning to errors thus aborting the build, simply add to pass the -W
option to sphinx-build.

This patch also fix some warnings:
* There is no source/_static dir so disable html_static_path
* In launchers.rst, File reference name should be before the title

Test plan:

$ make clean; make html
rm -rf build/*
sphinx-build -b html -d build/doctrees  -W source build/html
Making output directory...
Running Sphinx v1.1.3
loading pickled environment... not yet created
loading intersphinx inventory from
http://docs.python.org/2.7/objects.inv...
building [html]: targets for 5 source files that are out of date
updating environment: 5 added, 0 changed, 0 removed
reading sources... [100%] zuul
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] zuul
writing additional files... genindex search
copying static files... done
dumping search index... done
dumping object inventory... done
build succeeded.

Build finished. The HTML pages are in build/html.
$

Change-Id: Id907cc2c9aeccd077593b2fa1b78b220a159ed71
Reviewed-on: https://review.openstack.org/16316
Reviewed-by: Paul Belanger <paul.belanger@polybeacon.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Add layout validation env to tox.ini.

Change-Id: I7e3bd8eed585133ee217a1e462b9833df494be21
Reviewed-on: https://review.openstack.org/19718
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Report pipeline stats.

 * Number of changes processed.
 * Time each change spent in pipeline.
 * Current number of changes in pipeline.

All of those per-pipeline, and per-pipeline, per-project.

Change-Id: I609ff33ab7c26fc43d323f8a0bbb61f93382af90
Reviewed-on: https://review.openstack.org/19764
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
representation for Ref objects

The ref objects were missing a python __repr__, ended up with a not so
useful trace in the debug.log. This creates a meaningful representation
for human consumption:

	<Ref 0x7f51f848ffd0 creates refs/tags/v2.0.0 on 4eb656fe>
	<Ref 0x7f51f8491990 refs/tags/v2.0.0 updated 4eb656f..2260abb>
	<Ref 0x7f51f848f610 deletes refs/tags/v2.0.0 from 2260abb>

(sha1 above have been made shorter than the actual output)

Change-Id: If3b60494f1984a04e9ffada38f7e3b99f2a8ba0e
Reviewed-on: https://review.openstack.org/19925
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Add basic status HTML/JS.

A working example that can be customized for site use.

Change-Id: I6e6fdf245a55a767aa7e86486b6c47dae221f47b
Reviewed-on: https://review.openstack.org/19989
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Update zuul doc to cover SIGUSR1 behavior.

Change-Id: I6dcd87a59d836dc22b67a2bee7c764e0b603a809
Reviewed-on: https://review.openstack.org/21543
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
Update URL for openstack zuul example

Change-Id: Iae253977eafaff1a7cbdcaf1b873388487c4e36e
Signed-off-by: Paul Belanger <paul.belanger@polybeacon.com>
Reviewed-on: https://review.openstack.org/21730
Reviewed-by: Elizabeth Krumbach <lyz@princessleia.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
Dequeue changes when new patchset created.

When a new patchset is created for a change that is in a pipeline,
cancel running builds and dequeue that change (and possibly
dependent changes that can no longer merge).

Make this optional (and document the option).

Fixes bug 1022643.

Change-Id: I8f591956cf86645443e4b6075b8cdfc95a939e4f
Reviewed-on: https://review.openstack.org/20948
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Dequeue changes if they have no hope of merging.

In the DependentPipeline remove changes if they have absolutely no
chance of merging (all possible merge combinations fail). This allows
zuul to remove changes as soon as possible when it is known that they
cannot merge and continue testing the remaining changes.

Fixes bug 1123329

Change-Id: I97af03cca997541f6a8b3fb519d2a44825dbd937
Reviewed-on: https://review.openstack.org/21723
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
isUpdateOf check field presence before comparison.

isUpdateOf is passed an "other" changeish that may be a Ref which does
not have number and patchset fields which are compared against in
Change's isUpdateOf method. Check that fields exist before comparisons
are made.

Change-Id: Ifb5a36dcb0377f496e6b0707ae27810b96f2c752
Reviewed-on: https://review.openstack.org/22159
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Revert "Dequeue changes if they have no hope of merging."

This reverts commit dd26978d5da1e37f75e7fc38201c0d5427417d72.

The performance of mergability checking is poor and starves zuul from
processing other things. revert the change until we can come up with a
better performing solution to this problem.

Change-Id: I744031159572236f61b3fca45bb22492135ebcb2
Reviewed-on: https://review.openstack.org/22251
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: James E. Blair <corvus@inaugust.com>
Make canceling a queue item a POST request.

Needed by Jenkins 1.502.

(Also don't print the results of error messages so they don't
clutter stdout, but leave them there as a debug option
comments because they are very handy when debugging
Jenkins API issues.)

Change-Id: I2e1dcc1ac2344b1eb9f66292b6fd30900cb45e6d
Reviewed-on: https://review.openstack.org/22579
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Update Voluptuous requirement to version 0.6.

Voluptuous has a version 0.6 now which deprecates the any, match, and
required predicates and replaces them with Any, Match, and Required.
Update the minimum required version of Voluptuous and stop using the
deprecated predicates.

Change-Id: I554a6fda2c622a0b63f9247b1eae5eb494af225b
Reviewed-on: https://review.openstack.org/22167
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Test that Zuul refs end up in the correct repos.

Change-Id: I5a0040e6cf1076dfb21e355b7c021fdfc1914ca5
Reviewed-on: https://review.openstack.org/22738
Reviewed-by: Anita Kuno <akuno@lavabit.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Correct two links.

The github project has changed from `openstack-ci` to `openstack-infra`,
but two links in the documentation do not reflect this change.

Change-Id: I9359d1071f33719da39fdefd14deea728b9dbfff
Reviewed-on: https://review.openstack.org/23033
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Remember merges to save time.

Previously Zuul had to merge the entire change tree for each change
in the DependentPipeline. Remember previous merges to save time and
only merge changes against their project and branch.

Co-Authored-By: James E. Blair <jeblair@openstack.org>

Change-Id: I8105ac3aa4e4697e90e8d7fac1e54c5614f79daf
Reviewed-on: https://review.openstack.org/22504
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Restart jobs behind a removed change.

Even though we're removing from (potentially) the middle of the
queue, restart jobs behind this change since the situation has
changed.

Change-Id: Iab44898bda07d991a3d4c53715bf7a04ff58c9d1
Reviewed-on: https://review.openstack.org/23038
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Make trigger/result queue lengths available to status page.

Change-Id: I79fc7d1df0612fbe87e9b61499111c2d08d78dd0
Reviewed-on: https://review.openstack.org/23046
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins