Add precedence to merge jobs

When creating a merge job, give it the precedence of the associated
pipeline.

Change-Id: I96c6a942a08f603ae7cce442427ae171d7e76d78
add support for getting status of individual changes

This expands the rest API for zuul for selecting a portion of the zuul
data.

The supported urls are:
 - /status: return a complex data structure that represents the entire
   queue / pipeline structure of the system
 - /status.json (backwards compatibility): same as /status
 - /status/change/X,Y: return status just for gerrit change X,Y

In the individual status case the changes are returned as a simple
array, and not in the pipeline structure.

Tests are added to demonstrate this working, as well as ensure 404 is
correctly returned when invalid urls are provided.

Co-Authored-By: Joshua Hesketh <joshua.hesketh@rackspace.com>

Change-Id: Ib8d80530cc99c222226f73046c17ab0bbf6e080b
Fix docu after config repo split

Use new location for OpenStack's layout.yaml file.
Change all occcurences of github in the file to use git.openstack.org

Change-Id: I416fe33563e0df32b8d175b5694243550ee9e63a
Merge "add support for getting status of individual changes"
wmf: soften requirements

WebOb and paramiko have a minimal version which is enforced by OpenStack
but is most probably not needed.

ordereddict is for python 2.6, the feature is included in python 2.7.
soften PrettyTable. Precise only has 0.5, only impacts Zuul client.

Change-Id: I1aa1e3ae496d68d25dd64e8b21d2057c473342f4
Merger: ensure_cloned() now looks for '.git'

merger.Repo() would not clone a directory if the target path already
exists, even though it might not have been cloned.  Adjust the test to
look whether '.git' exist (which can be either a directory or a file in
the case of submodules).

I came accross this issue when using Zuul cloner to clone two
repositories, one having the other as a submodule.

Given two repositories:
- mediawiki/extension
- mediawiki (having 'mediawiki/extension' has a submodule to /extension)

We run: `zuul-cloner mediawiki mediawiki/extension`

It first clone the mediawiki repository with a placeholder directory for
the submodule. The directory structure is:

    .
    ├── .git/
    └── mediawiki/
	└── extension/

When cloning mediawiki/extension, ensure_cloned() would skip it since
the directory exists.  Looking for subpath '.git', we trick it in
cloning the repository resulting in:

    .
    ├── .git/
    │   └── modules/
    │       └── extension/
    └── mediawiki/
	└── extension/
	    └── .git   # note: a file, not a directory

My use case is cloning a mediawiki/core branch which has several
extensions registered as submodules that I am not going to initialize.

Change-Id: I4fe67a1db31adc2e23e8baf57528fce76f282986
Merge branch 'dependencies' into master

Adjust requirements.txt to point to the tarballs of:
- PrettyTable
- six
- Babel
- pytz

Change-Id: Ie1bc27fbf8959f8cf0ac9cc4fee1de4b19344d8f
Merge https://review.openstack.org/#/c/123437/1

5f6643 Merger: ensure_cloned() now looks for '.git'

Zuul cloner: fails on extension jobs against a wmf branch
Bug: 71133

Change-Id: Ic3c5aa2bf338ab4dd117ddd03011f7f8c0534080
Update merge status after merge:merge is submitted

When preparing a reference, we set the merge state to PENDING before
having emitted the merge:merge.  If any exception occurs when submitting
the merge:merge job, the buildset is left PENDING and is never retried
because prepareRef early exit in such case.

Move the merge_state change after the job has been submitted. An
exception would let the state as is (ie NEW) and thus indicate it should
be retried.

prepareRef happens before processing jobs and we do not have a way to
pause the scheduler.  So I overrided the merger.submitJob() to throw an
exception, the scheduler keep processing until the event fail.  From
there we can assert the merge is still pending (BuildSet.NEW) then
resume the submitJob() method to clear the queue properly.

I ran the test with the change, and it fails properly (there is still a
change stuck in queue).

Closes-Bug: #1358517
Change-Id: I4d91a15aaae878ed231d50ab5f4f7a65f0d0e830
Merge https://review.openstack.org/#/c/128921/3

86587d7 Update merge status after merge:merge is submitted

Zuul prepareRef does not handle failure to connect to Gearman
Bug: 72113