[BZ 1009945] squash commit of jsanda/aggregation branch
commit 1 - Initial commit for new Aggregator class

This is a first pass at computing multiple aggregates concurrently.

Conflicts:
	modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java

commit 2 - update 6 hr index after inserting 1 hr data

commit 3 -initial support for generating 6 hr and 24 hr data

Conflicts:
	modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java

commit 4 - delete 1 hr index entries when aggregation of time slice data is finished

commit 5 - handle scenarios for when there is no 1 hr and/or 6 hr data

This commit also includes some major test changes. AggregationTests is a more
thorough set (of not yet complete) tests to provide better coverage.

commit 6 - finishing test for 24 hour data

Conflicts:
	modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java

commit 7 - adding initial test for failure scenario

This test is for when we fail to fetch the raw data index.

commit 8 - This is a big refactoring of the initial implementation of Aggregator.java

The overall design is the same in terms of the fan out approach for doing the
calculations, but some changes were necessary after an extensive performance
analysis. Initially the aggregation tasks were to granular. Tasks were
submitted to the thread pool to process a single schedule. Schedules are now
processed in batches of 250. I did a lot of performance testing with various
batch sizes, and 250 seems optimal both in terms of execution time as well as
memory consumption. The other major change involves throttling. The initial
implementation had no throttling in place which immediately caused read and/or
write timeouts. Then I put throttling in place using RateLimiters, but there
was separate throttling that is used in MetricsServer for inserting raw data.
The same throttling is now used by both MetricsServer and Aggregator. There
are actually separate rate limiters for reads and for writes. Both are
configurable as well.

Conflicts:
	modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java

commit 9 - adding back support for aggregating 6 hour data

commit 10 - fix logic for determining if aggregation has finished

We check the respective counters to see if raw, 1 hr, and 6 hr data aggregation
has finished. Prior to checking the counters though, we need to make sure that
they have been initialized. They do not get initialized until the respective
index entries arrive. I have refactored the logic into a new method,
isAggregationFinished. It first waits for the arrival of all the index entries
and then checks the counters.

commit 11 - set counters for remaining data when we fail to get index entries for 1 and 6 hr data

commit 12 - refactoring some duplicate code and fixing when tasks are scheduled

Aggregation tasks for 1 and 6 hour data were getting schedule too early. They
cannot get scheduled until both their respective index entries have arrived and
aggregation for the previous buckets has completed.

commit 13 - make connection pool sizes and rate limits configurable

Connection pools per host and rate limits are configurable via system
properties. While the connection pool sizes are only set at start up, I plan to
make the rate limits self-tuning.

This commit also adds the missing method body to kick off 6 hour data
aggregation in Aggregate1HourData.java.

Conflicts:
	modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java

commit 14 - add support for running async aggregation in simulator

commit 15 - eliminate a couple possible race condition when processing index entries

commit 16 - cleaning up logging and adding some javadocs

Also moving all aggregation related classes into the org.rhq.server.metrics.aggregation
package. All classes except Aggregator now have package-level access.

commit 17 - updating test with package refactoring

commit 18 - turn on async aggregation by default
[BZ 1042663] upgrade datastax C* to 1.0.5
Upgrade to d3 version version 3.3.13.
Plugin validation for mysql plugin was failing because the Class.forName() statement was invoked from the constructor of the component. I moved this code to the method that actually opens the connection. This is the same strategy we use with our postgres plugin.
Bug 968361 - Improve database plugin design to support connection pooling

This changeset introduces a new API for database plugins and deprecates the previous one. Compatibility with the previous API will be maintained until next major version of RHQ.

The 'rhq-database-plugin' was based on org.rhq.plugins.database.DatabaseComponent interface which encouraged plugin authors to share a single JDBC connection across database components. This was wrong for various reasons (connection leaks, concurrent JDBC calls... etc).

The new API introduces three important classes:
* org.rhq.plugins.database.PooledConnectionProvider
* org.rhq.plugins.database.BasePooledConnectionProvider
* org.rhq.plugins.database.ConnectionPoolingSupport

BasePooledConnectionProvider is a base implementation of a PooledConnectionProvider. Plugin authors should create a concrete implementation of BasePooledConnectionProvider which overrides the #getDriverClass() method. This is important if a database plugin embeds a JDBC driver: the database-specific driver class must be loaded by the child plugin classloader.

ConnectionPoolingSupport helps to manage the compatibility with the old API. It's a contract that all new database resource components should obey to. It declares the following methods:
* #supportsConnectionPooling()
* #getPooledConnectionProvider()
Results of calls to #supportsConnectionPooling() #getPooledConnectionProvider() must be consistent. In practice, a top level server database component should be able to create a PooledConnectionProvider instance, and child servers and services should indicate they support connection pooling only if their parent component does.

The 'rhq-database-plugin' embeds the BoneCP library (JDBC connection pooling) and its dependencies (Google's Guava). Child plugins will have all the classes accessible as soon as they have this node in their plugin descriptor:
===
<depends plugin="Database" useClasses="true"/>
===

This changeset includes the necessary changes to support connection pooling in the Oracle, Postgres and MySQL plugins.

Thanks to Elias Ross for contributing the original patch from which this changeset is derived.
Add BoneCP to Eclipse classpath file
Fix return codes of the rhqctl command, rebased to the 4.10 master.
Fix the return codes for rhq-server.sh status and clean commands.
BZ 994250 - finish the merging/peer review for patches submitted so that rhqctl returns proper exit codes. Note that this completes the merge of the two submitted patches - see prior two commits to this one. This third commit fixes some problems with the original patches: 1. Some scripts/files are not found in bin/internal of the distro, but rather are in bin/ - so we need to avoid using getBinDir() in those cases 2. Fix the code to conform to code conventions - DEATH TO TABS! 3. Remove a constant that got resurrected (ControlCommand.RHQ_STORAGE_BASEDIR_PROP is no longer needed) 4. A couple other minor things
i'll assume this was a mistake to import a JUnit annotation, and it should have been testng
[1023451] [perf] Large retained heap during inventory report merge (causing OOMs)
Note - this is related work, not necessarily a solution to this problem.
One factor in the memory consumption of mergeInventoryReport is that we
return the entire resource hierarchy for the platform.  These hierarchy
is comprised of lightweight (ResourceSyncInfo) objects, but still, for
a large inventory this can be large. Moreover, the agent must receive the
same structure into memory.

This commit aims to reduce memory consumption at the expense of "chunking"
the hierarchy by top-level-server.  The general flow is that mergeInventoryReport
now returns the new PlatformSyncInfo, which is just the platform sync info and
the list of top level servers.  The agent then performs the sync in pieces, calling back to the server for each
top level server's ResourceSyncInfo.

notes:
- New abstract SyncInfo pulled down from ResourceSyncInfo and new superclass
  PlatformSyncInfo. These are both now used in the sync process.
- ResourceSyncInfo made more lightweight by replacing parentResource with
  parentId.
- Had to change test mocks to handle the new strategy
  - add mock support for getResourceSyncInfo service
  - enhance AbstractIgnoreTypesInventoryManagerBaseTest to actually assign
    resource ids to the simulated server inventory
- Fix a few PC itest issues
  - increase waitForInventory max time, it wasn't long enough
  - add missing waitForInventory in a couple of places
Plugin validation for mysql plugin was failing because the Class.forName() statement was invoked from the constructor of the component. I moved this code to the method that actually opens the connection. This is the same strategy we use with our postgres plugin.
Bug 968361 - Improve database plugin design to support connection pooling

This changeset introduces a new API for database plugins and deprecates the previous one. Compatibility with the previous API will be maintained until next major version of RHQ.

The 'rhq-database-plugin' was based on org.rhq.plugins.database.DatabaseComponent interface which encouraged plugin authors to share a single JDBC connection across database components. This was wrong for various reasons (connection leaks, concurrent JDBC calls... etc).

The new API introduces three important classes:
* org.rhq.plugins.database.PooledConnectionProvider
* org.rhq.plugins.database.BasePooledConnectionProvider
* org.rhq.plugins.database.ConnectionPoolingSupport

BasePooledConnectionProvider is a base implementation of a PooledConnectionProvider. Plugin authors should create a concrete implementation of BasePooledConnectionProvider which overrides the #getDriverClass() method. This is important if a database plugin embeds a JDBC driver: the database-specific driver class must be loaded by the child plugin classloader.

ConnectionPoolingSupport helps to manage the compatibility with the old API. It's a contract that all new database resource components should obey to. It declares the following methods:
* #supportsConnectionPooling()
* #getPooledConnectionProvider()
Results of calls to #supportsConnectionPooling() #getPooledConnectionProvider() must be consistent. In practice, a top level server database component should be able to create a PooledConnectionProvider instance, and child servers and services should indicate they support connection pooling only if their parent component does.

The 'rhq-database-plugin' embeds the BoneCP library (JDBC connection pooling) and its dependencies (Google's Guava). Child plugins will have all the classes accessible as soon as they have this node in their plugin descriptor:
===
<depends plugin="Database" useClasses="true"/>
===

This changeset includes the necessary changes to support connection pooling in the Oracle, Postgres and MySQL plugins.

Thanks to Elias Ross for contributing the original patch from which this changeset is derived.
Fix return codes of the rhqctl command, rebased to the 4.10 master.
BZ 994250 - finish the merging/peer review for patches submitted so that rhqctl returns proper exit codes. Note that this completes the merge of the two submitted patches - see prior two commits to this one. This third commit fixes some problems with the original patches: 1. Some scripts/files are not found in bin/internal of the distro, but rather are in bin/ - so we need to avoid using getBinDir() in those cases 2. Fix the code to conform to code conventions - DEATH TO TABS! 3. Remove a constant that got resurrected (ControlCommand.RHQ_STORAGE_BASEDIR_PROP is no longer needed) 4. A couple other minor things
[1023451] [perf] Large retained heap during inventory report merge (causing OOMs)
Note - this is related work, not necessarily a solution to this problem.
This commit builds on the "chunking" work introduced earlier. This work
replaces the tree structure, previously used to pass sync info between
agent and server, with flat collections.  This allows us to replace the
costly Hibernate-based tree building approach with more comprehensive queries
that reduce the number of DB round trips dramatically.
Notes:
- This commit fixed an issue with the previous work regarding the handling
  of top level services.
- The ResourceSyncInfo class is now even lighter weight, parent-child info is
  removed.
- PlatformSyncInfo is now a POJO.
- The handling of unknown resources (agent side) changed significantly
  because it had depended on the previous tree structure.
- mocks again had to be updated
Add a little more logging to get a better start/end of a sync.
Plugin validation for mysql plugin was failing because the Class.forName() statement was invoked from the constructor of the component. I moved this code to the method that actually opens the connection. This is the same strategy we use with our postgres plugin.
Bug 968361 - Improve database plugin design to support connection pooling

This changeset introduces a new API for database plugins and deprecates the previous one. Compatibility with the previous API will be maintained until next major version of RHQ.

The 'rhq-database-plugin' was based on org.rhq.plugins.database.DatabaseComponent interface which encouraged plugin authors to share a single JDBC connection across database components. This was wrong for various reasons (connection leaks, concurrent JDBC calls... etc).

The new API introduces three important classes:
* org.rhq.plugins.database.PooledConnectionProvider
* org.rhq.plugins.database.BasePooledConnectionProvider
* org.rhq.plugins.database.ConnectionPoolingSupport

BasePooledConnectionProvider is a base implementation of a PooledConnectionProvider. Plugin authors should create a concrete implementation of BasePooledConnectionProvider which overrides the #getDriverClass() method. This is important if a database plugin embeds a JDBC driver: the database-specific driver class must be loaded by the child plugin classloader.

ConnectionPoolingSupport helps to manage the compatibility with the old API. It's a contract that all new database resource components should obey to. It declares the following methods:
* #supportsConnectionPooling()
* #getPooledConnectionProvider()
Results of calls to #supportsConnectionPooling() #getPooledConnectionProvider() must be consistent. In practice, a top level server database component should be able to create a PooledConnectionProvider instance, and child servers and services should indicate they support connection pooling only if their parent component does.

The 'rhq-database-plugin' embeds the BoneCP library (JDBC connection pooling) and its dependencies (Google's Guava). Child plugins will have all the classes accessible as soon as they have this node in their plugin descriptor:
===
<depends plugin="Database" useClasses="true"/>
===

This changeset includes the necessary changes to support connection pooling in the Oracle, Postgres and MySQL plugins.

Thanks to Elias Ross for contributing the original patch from which this changeset is derived.
Fix return codes of the rhqctl command, rebased to the 4.10 master.