Minor cleanup in preparation of making socket checkouts along non-blocking code paths asynchronous.

src/java/voldemort/store/nonblockingstore/ThreadPoolBasedNonblockingStoreImpl.java
- fix missing ns to ms conversion

src/java/voldemort/store/socket/clientrequest/ClientRequestExecutor.java
src/java/voldemort/store/socket/clientrequest/ClientRequestExecutorFactory.java
- remove unnecessary selectionKey from checkTimeout interface

src/java/voldemort/utils/pool/KeyedResourcePool.java
- removed extraneous maxCreateAttempts
- attemptGrow returns true if pool grew
- refactored checkoutOrCreateResource
  - renamed to attemptCheckout
  - attemptCheckout is non blocking
  - attemptCheckout does not check timeouts
- refactored checkout
  - removed unreachable code (only one attempt could ever be made so attempts and maxCreateAttempts are extraneous.
  - made it clearer that exceptions control flow through try block
* even though code looks a lot different, I believe exact
  functionality is preserved except for one thing: the
  possibility of a single additional non-blocking get on the
  pool (resources) when attemptGrow returns true.
Adding a SlowStorageEngine to permit end-to-end testing with slow
servers in a cluster.

src/java/voldemort/store/memory/SlowStorageConfiguration.java
src/java/voldemort/store/memory/SlowStorageEngine.java
- The SlowStorageEngine/Configuration is inspired by the
  SlowStore used in other unit tests. The SlowStorageEngine
  produces delays on a per-operation-type basis. Delays can
  either be concurrent or queued: concurrent delays can overlap
  in time, queued delays occur in serial.

src/java/voldemort/server/VoldemortConfig.java
- Added config options for SlowStorageEngine

test/unit/voldemort/store/memory/SlowStorageEngineTest.java
- Unit test to confirm queued/concurrent delay behavior of a single SlowStorageEngine
he current nonblocking API can actually block if a server slows
down. The server slowdown can result in a socket checkout
blocking because all sockets are exhausted and there are a queue
of threads doing a blockingGet to get the socket. Adding an
integration test, E2ENonblockingCheckoutTest, that demonstrates
this undesirable behavior. Once a fix is in place, this
integration test may become a unit test for the correct behavior.

test/integration/voldemort/nonblocking/E2ENonblockingCheckoutTest.java
- Demonstrates the blocking behavior of the current socket
  checkout code. The integration test sets up a local voldemort
  instance with three servers, one of which is slow. Puts are
  done by two threads to a specific key that uses a fast node as
  the master, then the slow node, followed by the other fast node
  for the parallel puts. The second thread to attempt parallel
  operations blocks upon the first returning the checkout
  socket. This is observed in the console output.

test/common/voldemort/ServerTestUtils.java
- Added another startVoldemortServer method that takes a VoldemortConfig

src/java/voldemort/store/socket/SocketStore.java
- minor fix to logger.debug print out so .toString is not called on null object
Code clean up and refactoring in preparation of adding truly nonblocking requests.

src/java/voldemort/store/socket/SocketStore.java
- Moved logic from request async into ClientRequestExecutorPool::submitAsync
- Also moved private class NonblockingStoreCallbackClientRequest into ClientRequestExecutorPool

src/java/voldemort/store/socket/clientrequest/ClientRequestExecutorPool.java
- added private class AsyncRequestContext to start parceling up request context
- added submitAsync and submitAsyncRequest methods. These currently do exactly what was done by SocketStore:requestAsync
- I.e., I do not believe I have changed the behavior of any of this code. I just moved it to where I need to change it.

src/java/voldemort/utils/pool/KeyedResourcePool.java
- made some private methods protected with the expectation of
  extending this class with a subclass that can queue up async
  requests for resources
- changed many variable names to be more consistent in naming/usage
- refactored attemptCheckout to be simpler; added attemptCheckoutGrowCheckout to parcel up old behavior of attemptCheckout
- Do not believe I changed any behavior with these code changes.
- added getResourcePoolForExistingKey method
  - Throws IllegalArgumentException whenever a non existing pool is requested
  - changed all bare resourcePoolMap.get(key) accesses in other methods to use this method
  - May have changed behavior in that NPEs due to pool==null will now be IllegalArgumentExceptions in some methods
- Documented that some getFooCount methods are approximate in face of concurrency via comments in the code
Additional refactoring and code prep before making connection checkout async.

src/java/voldemort/store/socket/clientrequest/ClientRequestExecutorPool.java

Substantial refactoring and preparation of
ClientRequestExecutorPool to actually do an asynchronous
checkout. I believe I have essentially preserved prior
behavior. I.e., checkout of destination still blocks. But, the
pattern now looks a lot more like what is needed to do the
checkout asyncrhnously and then callback with the checkedout
resource. The exact exception strings likely changed during this
refactoring (but control flow should be the same).

src/java/voldemort/utils/pool/QueuedKeyedResourcePool.java

Skeleton of new class QueuedKeyedResourcePool that extends
KeyedResourcePool.
Initial commit of asynchronous checkouts.

src/java/log4j.properties
- print out more concise timestamp (don't print the date)
- print thread at end of message

src/java/voldemort/utils/pool/KeyedResourcePool.java
- reverted all protected data members back to private
- refactored attemptGrow to do the size check to determine if it
  is worth trying to attempt to grow. This made the method more
  useful to subclasses and cleaned up the local member that
  called it.
- added an internalClose method that returns whether or not the
  caller is "the one thread" responsible for closing everything
  down

src/java/voldemort/utils/pool/QueuedKeyedResourcePool.java
- first complete implementation of this class
- Four TODOs left in the code for the sake of interim code review

test/integration/voldemort/nonblocking/E2ENonblockingCheckoutTest.java
- minor tweaks to the test
Clean up of KeyedResourcePool and significant hardening of the unit test.

src/java/voldemort/utils/pool/KeyedResourcePool.java
- Documented the invariants (or lack thereof) guaranteed by this class.
- Documented this classes expectations of its users.
- Moved attemptGrow into the inner Pool class.
- Got rid of the attemptCheckoutGrowCheckout method. ;)

src/java/voldemort/utils/pool/QueuedKeyedResourcePool.java
- tweaked to match revised attemptGrow interface

test/unit/voldemort/utils/pool/KeyedResourcePoolTest.java
- Added a bunch of 'negative' tests. I.e., they demonstrate
  non-desirable behavior of current KeyedResourcePool.
- Added a contention test that has many threads checkout,
  possibly invalidate, and then checkin resources for some key.
Added unit test for QueuedKeyedResourcePool. Minor clean up of
QueuedKeyedResourcePool and of KeyedResourcePoolTest. Still one
outstanding TODO in QueuedKeyedResourcePool wrt semantics of
close(K key) method.
Made ant target 'junit-test' produce a report like all the other
junit ant targets. The report is useful because for a test class
with many sub-tests, the report clearly explains which sub-tests
failed. Unlike other ant junit targets, this target also produce
a plain text report. The plain text report is useful for running
a single test in a loop until it fails.

Clarified the help message for the 'junit-test' ant target.
Change SlowStorageEngineTest to avoid JUnit assertions in worker
threads. Also removed JUnit4 style annotations (@Test) since this
is actually a Junit3 style test (because it extends other Junit3
style tests that extend TestCase).
Wrapped tests that hang because of my changes with timeouts. This
is necessary for me to debug these tests and easily run all the
other tests. (By 'hang', I mean get into a state where clients
run forever complaining that there are not enough servers.)
Wrapping with timeouts is done both at the Junit
level (i.e., "@Test(timeout = ...")  and at the ant level (i.e.,
for an entire unit test).

Switched the tests I was touching to use Junit4 idiom rather than
Junit3 idiom. I.e., removed the 'extends TestCase' from the class
declaration.

Upped the memory allowed for all junit tets in ant from 1024m to
2048m since I ran into ant out of memory errors.
Addressed most feedback from reviews by refactoring:
- Made ResourceRequest a first class entity rather than a nested interface
- Refactored TimeoutConfig to tease apart an OpTimeMap which may be more generally useful.
- Renamed slow storage configs in Voldemort config with to make their testing nature more clear.
- Dropped OperationDelays object from SlowStorage in favor of OpTimeMap
Moved SlowStorageEngine into tests/integration.
Fix the problems with the RebalanceTest when using the revised
KeyedResourcePool. The test setup worked before the changes
because the KeyedResourcePool did not aquire resources until
full. With the new behavior, there is some mismatch between pool
sizes and numbers of threads. Increasing the number of admin
threads fixes the issue. There may be a better way of fixing this
problem.
Addressed most of the feedback from the code review.
- Renamed many variables, methods & classes
- Addressed most of the TODOs in my changes based on the feedback
src/java/voldemort/store/routed/action/PerformParallelPutRequests.java
- clean up handling of ObsoleteVersionException to do what the
  client said should be done. This will stop
  ObsoleteVersionExceptions from being escalated to
  InsufficientOperationalNodes exceptions.

test/integration/voldemort/performance/benchmark/Benchmark.java
- Add connections/node to configuration of benchmark tool
- Set client timeouts to recommended values

src/java/voldemort/store/socket/clientrequest/ClientRequestExecutor.java
- refactored a synchronized method to isolate the bare minimum
  steps that need to be synchronized. This allows local.complete
  to be called outside of 'synchronized' which avoids deadlocking
  nio selector threads.
Implementations of various async vs sync queueing policies for socket checkout. Two commented out policies are included in this commit.

src/java/voldemort/store/socket/clientrequest/ClientRequestExecutorPool.java
- a few TODOs to be investigated before completing work on async checkouts

/src/java/voldemort/utils/pool/AsyncResourceRequest.java
- a couple helper methods for implementing/debugging queueing policies

src/java/voldemort/utils/pool/KeyedResourcePool.java
- refactor to clean up checkin method
- TODOs for further code cleanup
- cleaned up all methods for tracking stats, added stats tracking of length of synchronous queue
- various aspects of (commented out) socket checkout queuing policies

src/java/voldemort/utils/pool/QueuedKeyedResourcePool.java
- fixes to async socket checkout
- various aspects of (commented out) socket checkout queuing policies
- TODOs for further code cleanup
- cleaned up stats tracking for async queue length

*Test.java
- minor tweaks/cleanup
Removed the commented out implementations of distinct policies for async socket checkout.
Added Jmx interfaces for all queue stats we now track. Updated ClientSocketStatsTest as well.

Added a big TODO expressing concern over how statistics are tracked with suggestions for improvements.
Minor cleanup --- changed some todos to documentation and comments.