AdamBien/lightfish — pull request #11 by rveldpau (2013-02-26 21:40:43+00:00)
Title: Improved multi instance support

This pull request reflects 3 main changes
1. Staggered Data Collection
2. Parallel Data Collection fixes
3. Fault Tolerances

### Staggered Data Collection

This pull request improves multiple instance support by staggering the requests rather than trying to pull all data at once. The following diagram shows how the new data collection works for multiple instances.
![data-collection-timeline](https://f.cloud.github.com/assets/1345647/198436/7cfd8d88-805b-11e2-8e92-ce4decf91878.png)

The diagram assumes the interval is set to two seconds, each time the interval is reached data collection for a new instance is started. Once a collection has been started for all server instances the system waits for all the data collection to complete. Once all the data collections are complete they are combined into an overall snapshot and sent on their merry way.

### Parallel Data Collection fixes

When I switched from the Fork/Join to Asynchronous EJBs I introduced a few bugs/inefficiencies. The main two were the reprocessing of already processed data points and starting extra actions. The first bug caused the application to fail persisting any Snapshots because of a duplicate key for the Applications. The second used extra resources for no purpose because of some extra code left hanging around from refactoring. I also added the max threads per instance option to the advanced configuration page.

### Fault Tolerances

When contacting the Glassfish server for statistics, every once in a while (more common with parallel data collection enabled) the Glassfish server will fail to respond properly. Instead of failing in these instances the system can now retry these failed calls, up to a maximum number of attempts. This option is configurable on the advanced configurations page.