Enforce ref only for gerrit events that supply a ref

Invalid layout could be used and cause zuul to crash. If a ref
was specified for an event that did not supply a ref,
(e.g. patchset-created) then zuul would crash trying to match
the regex supplied in layout.yaml to event.ref, which would be
None. This results in all jobs failing, so this patch
makes sure that only the ref-updated gerrit event can have
a ref pattern specified.

Change-Id: I297e38f3749166ce54a946f6cbcf708744f20fa1
Remove docutils pin

This was added to work around bug 1091333.  A new sphinx has been
released and so this is no longer necessary.

Also update sphinx requirement with global requirements.

Change-Id: I9107884d964757a9154bdd1feb4ea6df4e9df43e
Switched documentation to point to storyboard.

We've migrated this project to storyboard, I'm updating the documentation links.

Change-Id: I32ca3ee67ba78cd8f2daf1247488264647e44617
Pin GitPython version to prevent test failure

Change-Id: I3cadf46e3b5d9264040d1a24535fa77b276f9236
Add gerrit owner information to status.json

gerrit.query does return the owner information, we can save that
in the Change object and serialize it to status.json

Change-Id: Ib505ef8703bc60058d2775f522fcc3d02910b05f
implement gerrit pagination in simplequery

gerrit defaults to a limit of 500 changes returned. Nova regularly has
more than 500 open reviews. This means that some operations aren't
seeing all the open reviews, and causing some oddness (like merge
check).

Implement implicit paging.

In looking at this code I also believe I found a double -1 slicing
error which would mean that we would miss some reviews being listed.

Change-Id: I00d47c05e4a740b61c690510593fd29eaef49679
Merge "Enforce ref only for gerrit events that supply a ref"
Merge "implement gerrit pagination in simplequery"
Fix KeyError: 'sortKey'

The data returned by the gerrit query are in the line format:

  <json for change 1>
  <json for change 2>
  ...
  <json for change N>
  <json of query stats>
  <blank line>

The current [:-1] dropped the blank line, and previously the Gerrit
trigger would then drop the query stats with a [:-1].  That has
since been removed.  However, it needs to be dropped before the
simpleQuery processor performs the sortKey lookup.

Instead, drop any lines which don't include 'sortKey', which means
they aren't valid data lines (they are either stats lines, the blank,
or some other cruft).

Change-Id: I2b8ffdc70af436b34e7cee431ec4820c6e2d91c3
Merge "Remove docutils pin"
Merger: ensure_cloned() now looks for '.git'

merger.Repo() would not clone a directory if the target path already
exists, even though it might not have been cloned.  Adjust the test to
look whether '.git' exist (which can be either a directory or a file in
the case of submodules).

I came accross this issue when using Zuul cloner to clone two
repositories, one having the other as a submodule.

Given two repositories:
- mediawiki/extension
- mediawiki (having 'mediawiki/extension' has a submodule to /extension)

We run: `zuul-cloner mediawiki mediawiki/extension`

It first clone the mediawiki repository with a placeholder directory for
the submodule. The directory structure is:

    .
    ├── .git/
    └── mediawiki/
	└── extension/

When cloning mediawiki/extension, ensure_cloned() would skip it since
the directory exists.  Looking for subpath '.git', we trick it in
cloning the repository resulting in:

    .
    ├── .git/
    │   └── modules/
    │       └── extension/
    └── mediawiki/
	└── extension/
	    └── .git   # note: a file, not a directory

My use case is cloning a mediawiki/core branch which has several
extensions registered as submodules that I am not going to initialize.

Change-Id: I4fe67a1db31adc2e23e8baf57528fce76f282986
Update merge status after merge:merge is submitted

When preparing a reference, we set the merge state to PENDING before
having emitted the merge:merge.  If any exception occurs when submitting
the merge:merge job, the buildset is left PENDING and is never retried
because prepareRef early exit in such case.

Move the merge_state change after the job has been submitted. An
exception would let the state as is (ie NEW) and thus indicate it should
be retried.

prepareRef happens before processing jobs and we do not have a way to
pause the scheduler.  So I overrided the merger.submitJob() to throw an
exception, the scheduler keep processing until the event fail.  From
there we can assert the merge is still pending (BuildSet.NEW) then
resume the submitJob() method to clear the queue properly.

I ran the test with the change, and it fails properly (there is still a
change stuck in queue).

Closes-Bug: #1358517
Change-Id: I4d91a15aaae878ed231d50ab5f4f7a65f0d0e830
cloner: support for ref-updated parameters

When a job is triggered by a `ref-update` event, zuul-cloner complains
about missing parameters:

    zuul-cloner: error: Some Zuul parameters are not properly set:
	zuul_branch
	zuul_change
	zuul_patchset
    Define them either via environment variables or using options above.

The reason is that change and ref have slightly different parameters
(see zuul/launcher/gearman.py).

Split the list of expected Zuul parameters in three set, the commons one
being parameters shared by both change and ref events. An OrderedDict()
is used since the structure is reused in the argparse help message.

The parameter validation will reject a command line mixing parameters
from both change and ref since it does not make sense.

When parameter from one of the set has been set, hint about the missing
parameters for THAT set which would dramatically help. Else point to
--help which has all the details.

Downstream bug: https://phabricator.wikimedia.org/T76003

Change-Id: Ic920c2c8fe7d4159a4094ce8af5f07962f22a4af
wmf: soften requirements

WebOb and paramiko have a minimal version which is enforced by OpenStack
but is most probably not needed.

ordereddict is for python 2.6, the feature is included in python 2.7.
soften PrettyTable. Precise only has 0.5, only impacts Zuul client.

Change-Id: I1aa1e3ae496d68d25dd64e8b21d2057c473342f4
Merge branch 'dependencies' into master

Adjust requirements.txt to point to the tarballs of:
- PrettyTable
- six
- Babel
- pytz

Change-Id: Ie1bc27fbf8959f8cf0ac9cc4fee1de4b19344d8f
Merge https://review.openstack.org/#/c/123437/1

5f6643 Merger: ensure_cloned() now looks for '.git'

Zuul cloner: fails on extension jobs against a wmf branch
Bug: 71133

Change-Id: Ic3c5aa2bf338ab4dd117ddd03011f7f8c0534080
Merge https://review.openstack.org/#/c/128921/3

86587d7 Update merge status after merge:merge is submitted

Zuul prepareRef does not handle failure to connect to Gearman
Bug: 72113
Merge https://review.openstack.org/#/c/137675/1

079917d cloner: support for ref-updated parameters

Conflicts:
    tests/test_merger_repo.py

Bug: T76003
Change-Id: Ifad1854f05bad975186d7af6654cb1fd91a7bc5e
cloner: set zuul_branch on ref-update events

The `ref-update` events do not contains a ZUUL_BRANCH which is needed to
fallback to an appropriate branch.  Given a release branch `2.x` and a
tag 2.1, we would want to checkout the `2.x` branches of the
repositories.

We can find the branch a tag belong to by using `git branch -r
--contains` on a clone of the project that had the reference updated,
thus set ZUUL_BRANCH to match the fallback system expectation.

To do so, we need to first clone all repositories and on the repo that
had the ref update find the branch that contains the ref. Then we can
apply the Zuul crafted references and fallback to the zuul_branch.

We would then be able to add zuul-cloner support for ref-updates events
( https://review.openstack.org/#/c/137675/ ).

* Move update() and prune() from prepareRepo() to cloneUpstream() so the
  first stage is properly up-to-date.
* Whenever zuul_branch is missing, find the branch containing the ref.
* Rename prepareRepo() to applyZuulState(). It no more clones anything.
* Fix a debug message mentionning provided in branch instead of the
  indicated_branch

Change-Id: I01bddc932f4b7b775460526ece02e6d26ec6fde3
cloner: support for ref-updated parameters

When a job is triggered by a `ref-update` event, zuul-cloner complains
about missing parameters:

    zuul-cloner: error: Some Zuul parameters are not properly set:
	zuul_branch
	zuul_change
	zuul_patchset
    Define them either via environment variables or using options above.

The reason is that change and ref have slightly different parameters
(see zuul/launcher/gearman.py).

Split the list of expected Zuul parameters in three set, the commons one
being parameters shared by both change and ref events. An OrderedDict()
is used since the structure is reused in the argparse help message.

The parameter validation will reject a command line mixing parameters
from both change and ref since it does not make sense.

When parameter from one of the set has been set, hint about the missing
parameters for THAT set which would dramatically help. Else point to
--help which has all the details.

Downstream bug: https://phabricator.wikimedia.org/T76003

Change-Id: Ic920c2c8fe7d4159a4094ce8af5f07962f22a4af