OpenTSDB/asynchbase — pull request #90 by rajeshal (2014-07-01 12:39:03+00:00)
Title: Force HBaceClient to search region in meta table instead of getting from cache, during NSR condition

There is an open issue, https://github.com/OpenTSDB/asynchbase/issues/65, which causes the hbase client get stuck during region splits. Somehow hbase send wrong region name or client reads wrong region name. Once it happens during heavy writes, it will not be give up or get the right region instead of this wrong entry.

The probe created to rediscover the region checks the region info from cached region first like other RPCs and this prevents invalidating the wrong region immediately.

The fix here forces the HBaseClient to hit hbase meta data always, for such probes, to get the region info. With this fix we never seen TSDBs stuck NSREing during region splits. Tested this by asking HBASE to split region via hbase shell, which created an NSRE spike and the TSDBs was able recover from this very quickly.