forcedotcom/phoenix — pull request #439 by jyates (2013-09-22 02:16:02+00:00)
Title: Parallelizing writes to the index tables.

This is the last, likely, major source of 'slowness' in indexing. It actually slows the existing tests down a bit in the default implementation as we start with no threads and have to spin some up just for each index batch, creating more overhead. However, for more sustained write-load (especially to a consistent set of index tables), we should see dramatic speedup (and we can roll it back/fix it, if not :)).

We basically add a thread pool to which we submit write tasks for each index table
in the batch. Each batch operates independently and either returns success or throws
an exception. If any write throws an exception, we will need to replay that entire batch
(at least in the current implementation), so we attempt to stop all pending index writes
as soon as possible.

Also added a CachingHTableFactory. Its a very simple caching algorithm - it just uses
the JVM GC to release HTables when they are not used. This lets us save the effort of
setting up and tearing down the table for each batch of index writes, potentially a huge gain
over time.