Import/export acceptance tests

This commit adds two new acceptance tests: one for testing an
export from legacy storeconfigs, and one for testing the new
puppetdb import/export functionality directly.  The test of the
new import/export functionality is passing, but the legacy
storeconfigs is currently failing because PuppetDB can't handle
a catalog submission where there are no edges present in the
catalog.  After discussing this with Nick, we've decided that
the best solution is to alter the legacy export tool to generate
edges for the resources, rather than modifying PuppetDB to support
catalogs that don't contain edges.

This commit also adds some utility methods for recursive hash
diffing (required for comparing the exported JSON catalogs), and
a small fix to add the `rake package:bootstrap` target to the
setup steps if you are trying to run the acceptance tests locally.
Fixups from pull request feedback
Extract garbage collection tasks to their own functions

This simplifies the GC loop, and lets us refactor those functions
independently, or break them out.
Add error handling to garbage collection tasks

This helps to make them more standalone, to more easily refactor the way
they're managed and invoked.
Use at-at to schedule garbage collection tasks

This relieves us from having to manage the lifecycle of these tasks
ourselves. It also makes them more flexible, by decoupling each task
from the others and making them properly asynchronous.

                  ________
              _,.-Y  |  |  Y-._
          .-~"   ||  |  |  |   "-.
          I" ""=="|" !""! "|"[]""|     _____
          L__  [] |..------|:   _[----I" .-{"-.
         I___|  ..| l______|l_ [__L]_[I_/r(=}=-P
        [L______L_[________]______j~  '-=c_]/=-^
         \_I_j.--.\==I|I==_/.--L_]
           [_((==)[`-----"](==)j
              I--I"~~"""~~"I--I
              |[]|         |[]|
              l__j         l__j
              |!!|         |!!|
              |..|         |..|
              ([])         ([])
              ]--[         ]--[
              [_L]         [_L]  -Row
             /|..|\       /|..|\
            `=}--{='     `=}--{='
           .-^--r-^-.   .-^--r-^-.
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                __               __
         ____ _/ /_       ____ _/ /_
        / __ `/ __/______/ __ `/ __/
       / /_/ / /_ /_____/ /_/ / /_
       \__,_/\__/       \__,_/\__/
Convert TTLs to seconds before passing them to garbage collect functions

There's no need to do this every time. May as well simplify those
functions a bit.
Use Period objects for representing time settings

This lets us basically avoid having to track whether these have been
parsed/munged and what their units are, because we can just check that
they're Periods. It also lets us simplify a bit of code turning them
into seconds unnecessarily.
Add automatic compression of the dead-letter office

Prior to this, there was no self-management of the DLO. If messages went
into it, they were never removed or dealt with unless by the user. This
meant that a user with a lot of failures, who didn't know about the DLO,
could have their storage eaten up the messages (which were effectively
useless, since they didn't know about them). Now the DLO will be
periodically compressed (per the `dlo-compression-threshold` setting).
Any messages older than the threshold will be compressed into a .tgz
file. This runs on startup and periodically according to the
`gc-interval`, but will only produce one .tgz file per threshold period.
Any other messages will have their compression deferred until the next
period begins.
Fix use/require in command test
Specify encoding when creating tarballs

This was failing with the error "This archives contains unclosed
entries.", because we were calling `count` on a string to get the number
of bytes, without specifying the encoding to use. Now, encoding is a
required parameter for `add-entry` and `tar`, and is used to determine
the length of the data to be written.
Add metrics for DLO size and compression

This adds global and command-specific metrics for the dead-letter
office, including number of messages, number of archives, space usage on
disk, number of failures, and compression time. Overall values for
compression time, space usage, and number of messages are exposed via
the Dashboard.
Include Stage[main] and edges in storeconfigs export

PuppetDB expects well-formed catalogs to always have some edges.
Specifically, every resource is at least contained in Stage[main]. So in
order for our exported data to be accepted, it also needs to contain
edges. We add a (non-exported) Stage[main] resource and mark every
resource as being contained by it.
Fix post condition in export tool
Don't export nil file/line, or any aliases, from storeconfigs

This more accurately matches the format output by the PuppetDB export
command.
Change `pmap` back to `map` in export tool

Because clojure's `pmap` attempts to evaluate lazy sequences in
chunks of 32, the existing code for the export tool could cause
us to run out of Postgres connections because of too many
simultaneous catalog requests.  This commit simply changes the
code back to using a regular `map`, which is slower but safer
for now.  I may submit another pull request with another implementation
of parallelization if I can put together something simple without
spending too much time on it, but for now, it's best for us to
just make sure that we're safe and stable since performance isn't
a hugely pressing concern for this tool.
Add middleware to log incoming request URIs at DEBUG level

This commit adds a simple middleware function that will log all
incoming HTTP request URIs at DEBUG level.  This allows you to
add a line to your log4j.properties that will give you visibility
into all of the requests that are coming in.
Compare resource tags as sets in export acceptance tests

These are technically order-independent, but were being compared as if
they were ordered, which is incorrect.
Merge pull request #468 from nicklewis/storeconfigs-migration

Add import/export tests
Merge pull request #466 from cprice-puppet/bug/master/excessive-database-connections-during-export

Bug/master/excessive database connections during export
Don't use a redundant file-filter for compressible-files

The messages function already only returns files with the right file
extension (eg. the messages), so we need only filter by the one property
we care about: compressibility.