Added third_party MD5s and scripts for Apache Commons, Curator and Lucene
Updated Makefile.am with latest classes and third party jars
I tried out Hist4j as a new histogram class but it doesn't handle rolling data, as per Tsuna's notes in the code. StackOverflow recommended using the Apache Commons math class DescriptiveStatistics() as it implements percentiles as well as min/max/mean and has a rolling window. It's working nicely so far and doesn't seem to be using too many resources.
Re-enabled the latency callback in IncomingDataPoints.java
TEMP hack in JSON to allow non-numeric numbers such as NaN which is coming in from Collectd
TEMP hack in TSDB to set the path to the Lucene index depending on OS, I will break that out into a config variable later
Added a put latency callback to TSDB.addPointInternal() (why is this used and not IncomingDataPoints.addPointInternal()?)
TEMP hack in CollectdJSON.java to reformat "nan" as "NaN" so Jackson would recognize it
Renamed MetaData to MetaDataCache
Created MetaData as an abstract class that GeneralMeta and TimeseriesMeta implement, just to make things a little neater and reduce shared/redundant code. It will be easier in the future if we need to add more fields to all metadata types
Added .copy(), .equals() and .parseJSON() methods to the GeneralMeta and TimeSeriesMeta classes
Added code to SearchQuery.java to set the query string to an empty string if it was null
Disabled query == null/empty checks in Searcher
Modified Searcher.search() so that if the query is empty or null, then it will return ALL timeseries documents by getting everything with a "created" field
Modified StatsCollector to handle the DescriptiveStatistics() class properly.
Also modified StatsCollector to get rid of unused "record" methods
Modified the final StatsCollector.record() overload to put the tags before the final number so it didn't collide with previous overloads
Modified the final StatsCollector.record() method to take a number. I want to be able to put floating point values in the stats.
Modified TsdbStore to accept a Config class on construction (and the implementors of course)
Added the option of using Zookeeper row locks in TsdbStoreHBase.java instead of the Hbase locks. The code is working nicely but I've disabled it for now. See UID manager notes below.
Updated TSDMain to load all TSUID hashes before starting Netty
Updated all tools to handle the Config constructor
Commented out everything in GroupRPC till I can sort it with Lucene
Removed the MapRPC class since I'm not using maps any more
Modified TimeseriesUID locks to timeout after 100 ms and dump a stacktrace when that happens (need to clean up logging here)
Removed mapping code from TimeseriesUID
Added TimeseriesUID.loadAllHashes() to pre-load all of the TSUID hash codes
Added TimeseriesUID.collectStats()
Added a new_tsuids counter to TimeseriesUID to track how many new TSUIDs have been detected since startup
Removed the UniqueIdMap.java since Lucene will handle mapping and searching
Removed map functions from UniqueId.java
Fixed getOrCreateId() UID collisions. NOTE: When creating multiple UIDs very quickly, HBase was returning the same, unincremented UID from a .get() call on occasion, despite the row being locked with Zookeeper or HBase row locks. This may be because of region issues and the incremented put not propagating or maybe a caching issue. I added a local lock around the increment code, but that didn't solve it. What fixed the issue was adding a local counter that incremented on each UID increment request. Before writing back to HBase, we compare against the local counter and if there is a difference, we use the higher value. That takes care of secondary TSDs incrementing the UID (in most cases, though we can still run into the collisions) and HBase cache issues. In the future, I should investigate using Zookeeper atomic counters or HBase atomics for distributed collision fixes.