Convert README to reStructuredText

Change-Id: Iceb9049640aa3e0ff5c030fa5b93225e1e8ff887
Signed-off-by: Paul Belanger <paul.belanger@polybeacon.com>
Reviewed-on: https://review.openstack.org/26381
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
fix WebOb dependency

tools/pip-requires list WebOb in lower case which confuse pip freeze and
gives a false positive:

Before patch:

 $ pip freeze -r tools/pip-requires 2>&1|grep -i webob
 Requirement file contains webob, but that package is not installed
 WebOb==1.1.1
 $

The second line comes from another packages which list 'WebOb' as a
dependency.

After patch:

 $ pip freeze -r tools/pip-requires 2>&1|grep -i webob
 WebOb==1.1.1
 $

Change-Id: I73d01c734c25fc65022de146264e5b2c3cc2e67f
Reviewed-on: https://review.openstack.org/26786
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Test statsd.

Add a fake statsd listener, and actually listen on a random udp
port for statsd messages.  Configure STATSD env vars to use it
before each test.

Add a test that exercises all of the statsd methods that we use
explicitly.

Add checks of the statsd output to the basic "run jobs" test case
to make sure that we're covering some of the statsd code in the
scheduler.

Change-Id: I3361c4e334155dda413bf343e6f1c6f4e277a3ff
Reviewed-on: https://review.openstack.org/27306
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
doc: sync usage with recent changes

The documentation was missing recent optional arguments such as
--version.

Change-Id: Iad111d2f5c14c54d1495f2ba837627e0ea42acb0
Reviewed-on: https://review.openstack.org/27172
Reviewed-by: Jesse Keating <jesse.keating@rackspace.com>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
soften statsd version requirement to allow v2

We used to pin statsd to version 1.0.0.  James E. Blair has written a
few tests that now let us allow the current v2.0.1 version.  To prevent
unexpected upgrades, we disallow v3.0.0.

Zuul scheduler only do very simple statsd operations. Testing with
statsd 2.0.1:

I have set up a basic listener:

 listener$ netcat -l 127.0.0.1 -p 666 --udp

Then wrote the python script:

  import extras
  statsd = extras.try_import('statsd.statsd')

  statsd.incr('zuul.counter')
  statsd.timing('zuul.timer', 1234)
  statsd.timing('zuul.timer', 1)
  statsd.gaug:('zuul.gauge', 1234)

Running it as:
  STATSD_HOST='127.0.0.1' STATSD_PORT=666 python statsd-test.py

The listener gets:

  zuul.counter:1|czuul.timer:1234|mszuul.timer:1|mszuul.gauge:1234|g

Sounds legit enough.

Fix bug #1169599 'Zuul should depends on statsd 2 or later'

Change-Id: I41b797d606d55eeda300e673b05c2ae4442ce029
Reviewed-on: https://review.openstack.org/27008
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Add option to report elapsed build times.

The new format will be:
  "http://logs.example.com/6/1/gate/project2-merge/2 : SUCCESS in 00:00:00"

And the option is enabled by default.

Change-Id: Ib50c4948ea0a7b552d46a0e72ecb6c1a9609a771
Reviewed-on: https://review.openstack.org/27570
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
Fix config test interaction with report_times.

In a config test, the scheduler is created without a config object.
Handle that case gracefully in the pipeline constructor.

Add a test that runs the configtest.

Change-Id: Id59b3194aff7b5347fa04fe1cc29b1069d2a3f7f
Reviewed-on: https://review.openstack.org/27585
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
project templating system

On setup where Zuul ends up triggering hundreds of projects, you end up
having projects using roughly the same pipeline/jobs.  Whenever one want
to add a job in all the similiar project, he has to edit each project
one by one.

To save some precious time, this patch introduces the concept of project
templates.  It lets you define a set of pipeline and attached jobs
though the job names can be passed parameters defined on a per project
basis.  Thus, updating similiar projects is all about editing a single
template.

A basic example is provided in the documentation.

The voluptuous schema has been updated. It does check whether all
parameters are properly passed to a template but does NOT check whether
the resulting job name exist.

The parameter expansion in templates is borrowed from Jenkins Job
Builder (deep_format function). It has been tweaked to also expand
dictionary keys.

Layout test plan:

  $ nosetests -m layout  --nocapture
  Test layout file validation ...
  <...>
  bad_template1.yaml
     required key not provided @
  data['projects'][0]['template']['project']
  bad_template2.yaml
     extra keys not allowed @
  data['projects'][0]['template']['extraparam']
  good_template1.yaml
  ok
  <...>
  $

A basic test hasbeen added to verify whether a project-template properly
triggers its tests:

  $ nosetests --nocapture \
  tests/test_scheduler.py:testScheduler.test_job_from_templates_launched
  Test whether a job generated via a template can be launched ... ok

  ----------------------------------------------------------------------
  Ran 1 test in 0.863s

  OK
 $

Change-Id: Ib82e4719331c204de87fbb4b20c198842b7e32f4
Reviewed-on: https://review.openstack.org/21881
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Zuul status page: Redesign and fix bugs

status.html:
- Added eventqueue-length status (was already present in production
  but not committed here yet).
- HTML5 markup.
- Remove unused jquery-visibility.min.js and jquery-graphite.js.

status.js:
- Clean up (various js best practices and consistent coding style)
- Use empty() instead of html('').
- Use text() for text instead of html().
  html() will trigger the parser where text will simply create
  a text node with the string literal, much faster, safer and
  semantically correct.
- Fix implied global variable leak 'result'.
- Fix reference error that crashes/freezes the page
  Property data.trigger_event_queue and data.trigger_event_queue
  can be undefined, in which case data.trigger_event_queue.length
  causes an uncaught TypeError to be thrown.
- Use a closure instead of polluting global scope.
- Rewrite object oriented.
- Added 'demo' feature for easy local testing.

Downstream commits at
https://gerrit.wikimedia.org/r/#/q/project:integration/docroot+topic:zuul-js+branch:master+owner:Krinkle+is:merged,n,z

Change-Id: Iddd4e2787f2e2eb27bf428f733fbb8b4a9d162d5
Reviewed-on: https://review.openstack.org/26416
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
In FakeStatsd, use select() rather than poll()

The poll() system call is not available on all systems (like OS X, for
example), causing the test suite to fail. This change replaces poll()
with select(), which is more portable. With this change the test suite passes
under OS X.

Change-Id: I3ad716b127c8781d8dbf870b0f0c0e1162f264d1
Reviewed-on: https://review.openstack.org/28126
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
report message even if no action is specified

Whenever a pipeline does not provide any action, the job message would
not be reported which is a bit annoying.  This patch makes it so we skip
reporting when there is no action and no message.

My use case is:

A `postmerge` pipeline that simply react to the `change-merged` events,
since the change is closed, there is no point in voting on it.  Jobs
then uses the `success-message` and `failure-message` to finely tune
what is reported back to the user.

Configuration exemple

  pipelines:
   - name: postmerge
     manager: IndependentPipelineManager
     trigger:
      - event: change-merged

  projects:
   - name: operations/config
     postmerge:
      - operations-config-deploy

  jobs:
   - name: operations-config-deploy
     branch: ^production$
     success-message: 'Your configuration has been deployed by Jenkins.'
     failure-message: 'Configuration NOT APPLIED. You broke wikipedia.'

Before this patch, the debug would be:

------------8<---------------8<---------------8<-----------------------
INFO zuul.IndependentPipelineManager: Reporting
change <Change 0x7f67a0d61f50 61980,1>, action: None

DEBUG zuul.Gerrit: Report change <Change
0x7f67a0d61f50 61980,1>, action None, message: Build succeeded.

- https://jenkins-url-goes-there.example.com/
  : Your configuration has been deployed by Jenkins.

DEBUG zuul.Gerrit: No action specified; not reporting

DEBUG zuul.IndependentPipelineManager: Removing reported
------------8<---------------8<---------------8<-----------------------

Change-Id: I9ab4d47a8cfc1ae66870e8ac55e8f78dfc29da3d
Reviewed-on: https://review.openstack.org/28045
Reviewed-by: Jesse Keating <jesse.keating@rackspace.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Drop Jenkins.getBestBuildURL() method

The ability to configure success and failure URL patterns (cf.
'success-pattern' and 'failure-pattern') obsoletes the need to
guess-by-fetching an appropriate link for the build status, which can be
extremely expensive. (Wikimedia's Zuul instance makes three HTTP requests per
invocation -- 'testReport', which 302s to 'testReport/', which 404s, and then
'consoleFull', which often runs to hundreds of kilobytes.)

Also corrects a small typo in README.rst.

Change-Id: Ib222f544c98253152a5e787ec0cdf28fa2d80cf6
Reviewed-on: https://review.openstack.org/28128
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
make report times a little more user friendly

report times as 1h 22m 08s

only report hours if the job runs longer than an hour

only report minutes if the job runs longer than a minute

don't zero pad whatever the leading time unit is, making it a
little easier to scan and see differences.

Change-Id: Ibb58be233fdef1bbdf4e90a83731d43eb0be47f1
Reviewed-on: https://review.openstack.org/28143
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: matthew-wagoner <matthew.wagoner@hp.com>
Reviewed-by: Antoine Musso <hashar@free.fr>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Ignore pep8 E123.

From the pep8 spec:

  The closing brace/bracket/parenthesis on multi-line constructs
  may either line up under the last item of the list, as in:
  ...
  or it may be lined up under the first character of the line that
  starts the multi-line construct, as in:
  ...

Bizzarly, the pep8 tool ignores the first option.

Change-Id: Ic0e8f8ad964881276d213ffd50deb75a537c250b
Reviewed-on: https://review.openstack.org/28609
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
fix message report when no action is given

031e23f allowed the Gerrit trigger to report a message even when no
action has been given (for the context of postmerge builds where the
change cant be altered).  I have noticed the Gerrit lib would stack
trace because it is missing an action (actually None).  This patch skip
the action expansion in Gerrit command line whenever there is no action.

Related stacktrace:

  INFO zuul.IndependentPipelineManager: Reporting
   change <Change 0x7fb72099d190 62446,3>, action: None
  DEBUG zuul.Gerrit: Report change <Change 0x7fb72099d190 62446,3>,
   action None, message: Build succeeded.
  ERROR zuul.IndependentPipelineManager: Exception while reporting:
  Traceback (most recent call last):
  File "zuul-5b84b5c-py2.7.egg/zuul/scheduler.py",
    line 883, in reportChange
      ret = self.sched.trigger.report(change, report, action)
  File "zuul-5b84b5c-py2.7.egg/zuul/trigger/gerrit.py",
    line 142, in report message, action)
  File "zuul-5b84b5c-py2.7.egg/zuul/lib/gerrit.py",
    line 119, in review
      for k, v in action.items():
  AttributeError: 'NoneType' object has no attribute 'items'

Change-Id: I51b66025eb6e1d074d79cf77ac4085ba3ffde77e
Reviewed-on: https://review.openstack.org/28395
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Sync from oslo-incubator.

Change-Id: I3a7c6bdfae9f2273cf8059c322d648cda07d830d
Reviewed-on: https://review.openstack.org/28619
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
Switch the launcher to Gearman.

Remove the Jenkins launcher and add a new Gearman launcher (designed
to be compatible with Jenkins) in its place.

See the documentation for how to set up the Gearman Plugin for
Jenkins.

Change-Id: Ie7224396271d7375f4ea42eebb57f883bc291738