Use bulk query to determine resource existence

Previously, we used a SELECT for each resource we wanted to store to determine
if it already exists in the database. If a catalog has 1000 resources, this
results in 1000 queries. It's much more efficient to use a single query to
determine the existence of a set of resources in one shot.

Signed-off-by: Deepak Giridharagopal <deepak@puppetlabs.com>
Cache computation of resource hashes

The hash computation is a pure, referentially transparent function. Therefore,
we can easily memoize the function to return a previously computed result when
encountering an argument list we've already seen.

For catalogs with many resources, this provides a meaningful performance
improvement. In my testing with a catalog using 1000 resources, this drops
hash-computation time during add-catalog! from ~200ms to ~20ms.

Initial sizing of the cache is based on the "medium" site persona, which is
said to have ~40k unique resources in the population.

Signed-off-by: Deepak Giridharagopal <deepak@puppetlabs.com>
Add "valset" function that returns a set of values from a map

Signed-off-by: Deepak Giridharagopal <deepak@puppetlabs.com>
Simplify logic inside add-resources!

We were creating intermediate variables that weren't actually necessary for the
computation; we just need a set of all the resource hashes.

Signed-off-by: Deepak Giridharagopal <deepak@puppetlabs.com>
Remove unused :use statment

Signed-off-by: Deepak Giridharagopal <deepak@puppetlabs.com>
Merge pull request #92 from grimradical/perf-improvements

Use bulk query to determine resource existence
Merge pull request #93 from grimradical/resource-hash-caching

Cache computation of resource hashes
The number of command-processing threads should be end-user configurable

We'll default to half the number of CPUs in the system, and allow the user to
override this via configuration file directives.

Signed-off-by: Deepak Giridharagopal <deepak@puppetlabs.com>
Merge pull request #95 from grimradical/configurable-threads

The number of command-processing threads should be end-user configurable
Release 1.0.0-b1

Signed-off-by: Deepak Giridharagopal <deepak@puppetlabs.com>
Flip back to 1.0.0-SNAPSHOT

Signed-off-by: Deepak Giridharagopal <deepak@puppetlabs.com>
Remove the vestigial populatemq CLI

It's been obsolete for quite some time.

Signed-off-by: Deepak Giridharagopal <deepak@puppetlabs.com>
(#13439) refactor spec_helper to provide compatibility with both puppet 2.7
and master
Display usage information when no subcommand is given

To assemble the list of eligible subcommands, we scan the classpath to find all
namespaces under ".cli". When displaying usage information, we find descriptions
for each subcommand by looking for a `cli-description` var in their namespace.

Signed-off-by: Deepak Giridharagopal <deepak@puppetlabs.com>
Extract fixtures and examples

Many of our tests were using the exact same fixtures for db/mq/app, but
there was no code being shared between them. These now live as proper
functions in `com.puppetlabs.cmdb.fixtures`, so all that is needed is to
specify, for example, `(use-fixtures :each with-test-db with-test-mq)`.

Similarly, there were example catalogs being defined directly in tests
that used them, when they are generally useful. These are now available
in `com.puppetlabs.cmdb.examples`.
Store receipt timestamp for catalogs

This allows us to make smart decisions about storing catalogs with
respect to retries. If the catalog in the database is more recent than
the catalog we are being asked to store (which is most likely to happen
if the current command has been retried, and another catalog was stored
in the meantime), we will simply ignore the command.
Add with-transacted-connection macro, use that everywhere

With postgresql, you can never truly operate outside of a transaction (which is
a good thing); You get one automatically when you open a new connection. If we
issue a malformed query, and if the query isn't inside of a (sql/transaction)
block, we never explicitly call "rollback" upon an exception (which you need to
do to "clear" the aborted transaction). From that point on, that database
connection is in a bad state; the database will reject any subsequent commands
until a "rollback" is issued (which won't happen).

Theoretically, the connection pool will eventually discover that the connection
is poisoned and will kill it off. While that's a nice safeguard, it doesn't
happen immediately and in the meantime, all queries over a poisoned connection
will fail. We can proactively prevent this by properly enclosing all database
activities within explicit transaction blocks.

This patchset adds a new macro, `with-transacted-connection`. It's exactly like
`with-connection`, except that it automatically executes enclosed forms within
an explicit transaction.

Signed-off-by: Deepak Giridharagopal <deepak@puppetlabs.com>
Merge pull request #97 from grimradical/cli-cleanup

Cli cleanup
Merge pull request #96 from cprice-puppet/bug/master/13439-spec-compatibility

(#13439) refactor spec_helper to provide compatibility with both puppet 2.7 and master
with-transacted-connection should actually invoke the supplied body

Signed-off-by: Deepak Giridharagopal <deepak@puppetlabs.com>