Provide swift credentials without requiring a connection

To generate the hmac details for the swift FormPost middleware
you do not need to actually connect to swift if you know the
URL key and storage path.

Change the configuration to reflect this and only connect if
required.

Change-Id: Iad5008ca2707ef5310b172fe3b6844e51bee2372
Make swiftclient an optional dependency

I can not use the switclient on my installation and need a way to
prevent the swiftclient import entirely.

The only usage of the swiftclient module is to establish a connection
which only happen if there is a [swift] section in the configuration.

Moving the import from the top of the file next to its only usage would
make sure my installs never has to get swiftclient installed.

Change-Id: Ie2ef84f86f9f57796a85758147ba87d8b22c3e55
Report correctly when dequeuing dependent changes

The change in commit b7179777f4df9d0832abb12c2fb055ab60b4499d
(change I98eb8b8817bbda57efc2ef5bfcc2a5076fe8f4fd) erroneously
altered the logic in formatReport.  The "dequeued_needing_change"
flag was only checked if the merger encountered a merge conflict.
So, a change whose parent was dequeued would only be reported as
having been dequeued if it had a merge conflict.  In the (frequent)
case that a change is dequeued because its parent failed tests or
otherwise failed to merge, but has no merge conflicts itself, the
flag was not checked and the usual path of displaying job results
was taken.  In our configuration, it is not possible to format
those job results (they are all SKIPPED), so Zuul would log an
exception and not report.

This change restores the original hierarchy of report types:

 1. Dequeued because it depends on a change that failed to merge.
 2. Merge conflict.
 3. Job report.

It also moves the bulk of the job report formatting to its own
method so that the logic around the report types is more clear,
and reduces redundancy introduced by the report footer option.
Finally, it updates a test to verify that the correct report types
are issued.

Story: #147
Change-Id: If1fe3af8b4d0896a9e43d8163bc2690663df0637
Co-Authored-By: James E. Blair <jeblair@hp.com>
Add pipeline source

A TriggerEvent may originate from a trigger that does not represent
the canonical location of the project source.  For instance, the
timer trigger strangely depends on the gerrit trigger to actually
handle Git operations behind the scenes.  Instead, make an explicit
association between pipelines and their source triggers so that
their event trigger does not need to have that implicit association.

This is a step toward having pipelines support multiple triggers
(they already support multiple reporters).

Change-Id: Ie80ffde411fe40fddfc4496b7adb0004f660c48c
Merge "Report correctly when dequeuing dependent changes"
add support for draft-published event

Gerrit 2.4 did not provide a stream event for draft patchsets therefore it was not
supported in zuul either.  We've just upgraded to Gerrit 2.8 which does provide a
new draft-published stream event.  This change will make zuul aware of the draft-published
event.

Change-Id: Ica363b4672cfce375d64463636d84e92c2949fd6
Partial-Bug: #1255166
Merge "Provide swift credentials without requiring a connection"
Merge "Make swiftclient an optional dependency"
Merge "Add pipeline source"
Add a Zuul trigger

This adds the ability for a pipelite to have multiple triggers.

This also adds a "Zuul" trigger which is used to generate trigger
events based on internal actions Zuul has taken.

It supports two event types:

 * parent-change-enqueued: This can be used so that other pipelines
   can enqueue children of parents that are enqueued in a different
   pipeline.  Specifically, this lets OpenStack enqueue changes in
   check when their parents are enqueued in gate (which may be
   necessary because of our clean check rules).

   This could be used to replace the internal logic that enqueues
   children in dependent pipelines (moving that into explicit
   configuration instead).

   One can also imagine a future 'change-enqueued' event so that a
   pipeline could react directly to a change in another.

 * project-change-merged:  This can be used to trigger changes on all
   open changes for a project when a change is merged to that project.

   Specifically, this lets us perform light-weight merge checks on all
   open changes whenever a change is merged.

Change-Id: I2a67699dbed92a6b9c143a77795cb126f1f4dd57
Merge "Add a Zuul trigger"
Update gerrit change attributes even if merged

Function updateChange short-cuts dependency checking for merged changes, but
some attributes that need to be updated were after the short-cut return.
This resulted in zuul saying status is MERGED, but then not matching
Changeish filter for status MERGED.

Closes-Bug: 1356662
Change-Id: I954b2716a5af75a959d4129ba88d7dae0750d2a5
Fix queries for Zuul trigger

Against a live gerrit, we observed that sometimes gerrit may not
return data for a change.  In the case of the zuul trigger's
project-change-merged event, it's best to just ignore errors
for those changes and proceed.

Also, the gerrit trigger was attempting to parse the timing
results from the query.  The test did not catch this because it
did not supply them in the mocked method.  Correct this as well.

Also, double check that the query used was the one expected in
the test.

Change-Id: I792127d29f67f53a419eb94e9e0afb83b6e1bcb2
Stop removing changes from the cache

The zuul trigger ends up querying a large number of changes on
each merge, most of which are not in any pipelines.  To avoid
needlessly querying gerrit for data that do not change, keep
all changes that Zuul ever sees in the cache.

We could consider removing changes that are both not in pipelines
and closed (ie, keep all open changes), but we still end up querying
a number of merged changes each time due to Zuul following
dependencies of open changes.

Change-Id: Ie78df9aa43ec5ac35bdea79dcdafdfdd41d51d5b
Make smtp test more robust

It may have ended up with changes still in the pipeline because
it uses the timer trigger.

Change-Id: Ieb46d7b5b7020e62e7d3baa704e3f75f9f87dca1
Merge "Update gerrit change attributes even if merged"
Maintain the trigger cache after reconfiguring

A previous change removed the trigger cache cleanup.  When Zuul
is live-reconfigured, it creates new Project objects and
attaches them to all changes currently in pipelines.  Because
the cache contained changes outside of pipelines, those changes
were not being updated with new Project objects.

This restores the maintainCache method and uses it during the
reconfigure event to clear the cache of all changes not currently
in pipelines.  This corrects the error introduced by the previous
change as well as gives Zuul an opportunity to release unused
memory (as now the cache will at least be emptied on each
reconfiguration).

The project-change-merged test is updated to explicitly test this
situation.

Change-Id: I67736fca08f2e14ab733bd9f143820da19839ef9
Merge "Make smtp test more robust"
BuildSet now has a representation

I had Zuul Scheduler report warning messages such as:

zuul.Scheduler: Build set <zuul.model.BuildSet object at 0x1a9a310> is
not current

Which is not very meaningful.  This adds a __repr__ to BuildSet which
shows up:
- the QueueItem representation
- number of builds in the set
- merge state

The merge states are internally stored using integer constants.  I have
duplicated the information in a dict to lookup the int as a meaningful
string.  That is not pretty :(

Change-Id: I0e087e4cb3f4c1da4ef40d0805793f737af1765e
Fix zuul javascript cookie load

The default filter value for the status page was not being set correctly.
As such the filter would incorrectly remove all results. When no cookie
is set, make sure the filter is blank.

Change-Id: I4868db8b357373d1fd94dc6ffc7528692b75dd8d