Add javelin.log to logstash

Dependant on I787c08506d9a4d12081b5b5b16d752d5147f8e72 to collect the
javelin logs.

Since javelin2 uses tempest code to do the testing it generates useful
logs. As javelin2 grows to cover more we will most likely run into
issues that need debugging.
Add these logs to logstash to help us do analysis on javelin
results.

Change-Id: Ic3b2e33177bdc65fe294ddddd581a93de6c1cb60
Add single_use_slave coverage to the apply tests

Currently the single_use_slave class is not tested by the apply tests
because there is no code path from a long-living puppet node to this
class, so create a dummy node definition for the purpose of adding a
path to this class in the apply tests.

Change-Id: I4ff4f1e42107b6df7ee7e4c845faa37d0f97ea65
Handle log processing subprocess cleanup better

We are leaking file descriptors in our log worker processes because we
are are not catch all possible errors leaving some actions left behind
to do. More aggressively catch errors so that all cleanup happens

Change-Id: I7a73a36c6fc42d4eba636cf36c8cfffcea48a318
Add ability to change the slave gerrit user

Some slaves may want to connect to gerrit as a different person.
Rather than hardcoding jenkins@openstack.org - parameterize it.

Change-Id: Iafeb76f1bec7a0ba1a33b0c9c5b74e2b88d13889
Document typical response time for third-party-requests is one week

In order to avoid complaints from impatient requesters, document that
requests typically take a week.

Change-Id: I8e755f89882836f77698377fd233ce3532b95736
Run bandersnatch under file locking and a timeout

* modules/openstack_project/manifests/pypi_mirror.pp: Bandersnatch
occasionally seems to livelock in a select poll loop at teardown.

https://bitbucket.org/pypa/bandersnatch/issue/52

As such this blocks subsequent runs of our wrapper script, filling
the log with useless tracebacks every 5 minutes while leaving the
mirror perpetually stale until someone comes along and notices. Work
around this unfortunate situation by setting a half hour timeout on
the process, and add a file lock to the invocation so that it will
not try to run more instances of the wrapper script until the
current one terminates.

Change-Id: I055b623801af3063d22b081eb9c9ef3634e5e5b5
Add syslog tag to os-collect-config logs

We now have consistent os-collect-config log formats for both
fedora and ubuntu tripleo ci jobs(from systemd journal and syslog
respectively), see I27ea3d465670277ef1ddf3d1b3b9d52df4162807 and
Ib8c1fc39d56b2b9c6d8e9b64a868def619aa2f1f. Tagging them as syslog
will ensure they are handled by the logstash syslog grok filter,
they currently don't match anything which results in messages
being indexed with a value of "%{logmessage}".

Change-Id: Iec5ffcb08e5b45fb01db25e14c943f000ca97a83
Closes-Bug: #1350121
Improve jeepyb.rst content

ACLs are stored in organization/project-name, fix the description for
this.
Use project-name-release as this is the current OpenStack infra
convention.

Change-Id: Id792594ad49120d9d9dcbaeae22ead6beb0864b2
Remove d-i-b tools

These are ported into the new repo with change
I087a2396581bf4415c9676f70b4c739a62c84b23

Change-Id: I9378cef3555f29857a6869a273b3904c636bb7b0
Ignore extra-atcs lines which look like comments

Change-Id: I8054f4582ad93a5dd31a2d5de288250780ebbf3f
Allow a git ref to pin the governance repo

Change-Id: I230c22fb83447b336eda14a7f10cf33e52e3d321
Stop statusbot joining #openstack-packaging

Change-Id: Ia1b8aafc0fab77d1061251a860b5218bd2994f6b
Partial-Bug: #1360324
Added deferred worker process to storyboard

StoryBoard now supports deferred processors. This enables them
for storyboard.openstack.org.

Change-Id: I7b4e32f94db66f0e0b399d13b6a6a04d7c53639c
Remove projects_alphabetized from tox.ini

Since the split to project-config, there is no longer a need to
run 'tox' for projects_alphabetized, since those resources no
longer exist. Also removing the shell scripts that were running
the tests.

Change-Id: Ib7a762d872da9c5d945206807c405e7452b49638
Remove jjb from tox.ini

Since the split to project-config, there is no longer a need to
test this environment in tox.

Change-Id: I9a3866126506a9784a38f2c859bac6b5774660b7
Remove irc testenv from tox.ini

Change-Id: I1b998ba9cfeb8744b9801c4968afae872ebcfeb3
testenv:irc is obsolete since the project-config split
Remove undefined upstream environment from tox.ini

Commit number 3fa3be7edef356fe42ed3c6adc21590604634665 initially
added the upstream testenv, but it was merged to the 'projects'
testenv a long time ago. Which is now obsolete, because of the
split to project-config.

Change-Id: I2d011af7df2ad9d14d2db67fa78c4dc6e4e4cbb6
Clean up bashate failures

Fix all of the bashate failures.  Also, bashate will install its negative
tests in its virtualenv, so we need to generate a list of files to pass to
bashate that excludes .tox/

Change-Id: I55559bf6137f705aec9a7f277928ad8d4cadc2ca
refactor of hideci for readability and modularity

This is a refactors the hideci code into much smaller functions,
making the whole thing a bit more readable for fixing this going
forward. This was done in order to add a couple of new features. The
smaller functions should also make this code much easier to review.

After the refactor we now do a top down parsing of all comments in a
review, classifying them as 'is_ci' and 'is_trusted_ci' so that we can
more easily compute the results we want going forward.

In the process a few additional features were added.

1) CI results are only shown for the latest patch set, even if no CI
system has reported on it. This is accomplished via
ci_latest_patchset.

2) If CI system is in merge conflict, that's displayed in the results
box.

3) Test results records are lifted in total to the test results
display panel, and each set of test results includes the date that we
got it from. This means that for patches that get 'rechecked' a lot,
you'll see the whole history of test results on current patch. You
will also be able to quickly realize that the current test results are
older than the last recheck comment.

The UX for part 3 is different, and may need tweaking going forward,
however the merge of results from different runs was creating a lot of
confusion to reviewers, so this errors on the side of more clarity.

Because of the largely inconvenient workflow for development (the fact
that you basically need to copy / paste function definitions into
chrome dev console) this isn't broken up as smaller patches. This
should, hopefully, make future patches much easier on this code
though.

Change-Id: I3f128a449756fd6988f3810863cd1d49c476b0dd
Ensure redis is on slaves (and running)

Quite a few projects are/would like to use redis and
being able to have their tests use the slave provided
servers would be a great addition to ensure functionality
is working as expected (when mocking just doesn't cut
it).

Change-Id: I6081b6cfe1c6393c7c6f2d028c313da759220a72