Streaming resource queries

Previously, we'd load all rows from a resource query into RAM, then do a
bunch of sorting and aggregation to transform them into a format that
clients expect. That has obvious problems involving RAM usage for large
result sets. Furthermore, this does all the work for querying
up-front...if a client disconnects, the query continues to tax the
database until it completes. And lastly, we'd have to wait until all the
query results have been paged into RAM before we could send anything to
the client.

This patchset adds streaming capabilities to resource queries. This is
accomplished through a few key additions:

1) HTTP functions for streaming a response. Ring can handle an
InputStream as a response object, and it will pass that to Jetty who
just reads from the stream and sends output to the client using a
Chunked Transfer encoding. I've created a macro that will wrap the body,
binding a Writer object to local scope. A PipedInputStream is then
hooked up to that Writer, and is returned as the Ring response. This
way, the body can stream content to the Writer object and Ring takes
care of the rest. No muss, no fuss!

2) Functions for streaming a JSON response. While Jackson does streaming
output, my testing seemed to indicate that it still needs to buffer up
into RAM the next object to be streamed. That doesn't really help us for
very large JSON arrays. I've written a simple function that streams an
array one object at a time, thus reducing resident memory usage.

3) Functions for streaming a JDBC query. We can't use the stock
clojure.java.jdbc querying code, as that doesn't open up a proper
database cursor. So I've written a function that will do the proper
initialization of a database cursor, and invoke a callback on the
resultset.  By using a callback-based approach instead of a lazy-sequence
of rows, we can more precisely control the lifecycle of the cursor. To
wit, if the callback fails for whatever reason, we can simply cancel the
query.

Taken together, we can now issue a query for, say, all resources we know
about. As soon as the database can produce the first complete resource,
the client can receive it. And if the client disconnects prematurely,
the query is cancelled, sparing the database from doing too much work.
This is all done asynchronously and with low memory usage, though at the
expense of an additional thread (for the PipedInputStream).
Add a table containing full parameter sets

Much of the slowness of resource queries comes from the format of the
resultset. We get back one row per parameter, and we need to collapse
multiple rows into single resource objects that we can emit to the
client.

It's much faster and tidier to just have a separate table that contains
a json-ified version of all a resource's parameters. That way, there's
no need to collapse multiple rows at all, or do any kind of ORDER BY to
ensure that the rows are properly sequenced.

In testing, this appears to speed up queries between 2x-3x...the
improvement is much better the larger the resultset.

Also, the parameter cache table can easily be extended to serve as an
additional set of constraints on resources and parameters. Previously,
there was no foreign key relationship between the two tables, as they
were linked in a many-to-many way. The cache table, though, has a single
row for each resource. This allows for foreign keys that point into that
table, thus improving the overall consistency of our data.

Some minor changes needed to be made to make this happen. We need to
make sure that all resources have an entry in the cache table, even
those with no parameters (thus making parameters a nullable column). We
then need to add the foreign keys to catalog_resources and
resource_params. We then need to slightly change insertion order when
catalogs are added...we need ot make sure that entries are added to the
cache table first, prior to catalog_resources and resource_params (to
satisfy foreign key constraints).

Finally, we use a higher schema migration number, to allow for other
branches.
Remove resource-query-limit

As we now can stream resource queries containing zillions of rows
without blowing out RAM, there's no longer a need for this safeguard.
Remove unneeded transaction
Add test cases for params cache migration
Merge pull request #612 from grimradical/streaming2

Streaming resource queries (2)
Fix broken test case on PG
Merge pull request #617 from grimradical/test-fix

Fix broken test case on PG
(#21520) Provide option for soft failure during terminus writes when PuppetDB is unavailable

This patch adds a new option 'soft_write_failure' for puppetdb.conf to change
the terminus behaviour so that if a command or write fails, instead of
throwing an exception and causing the agent to stop it will simply log an
error to the puppet master log.

Signed-off-by: Ken Barber <ken@bob.sh>