Async store stress test added and run new async store against functioknal tests
Make sure store enqueing and expected state updates happen atomically

* This mimics what the cache would do, which is acquiring the lock on
the key and them modify the cache store. This allows us to be able to
verify the final state of the underlying cache store.
AsyncStore: fix deadlock

Apparently AbstractQueuedSynchronizer cannot handle threads that block for
different reasons. A failed AQS.acquire requires that some other enqueued
thread is eligible for execution. If all enqueued threads refuse to run
(e.g. due to buffer full), we have a deadlock.

Use separate AQS objects for the the different blocking conditions, i.e.
buffer full, buffer empty and the read-write lock to fix this.

Note: the same functionality could be implemented using higher level
synchronization objects (e.g. two Semaphores and a ReentrantReadWriteLock),
with some overhead (i.e. we don't need reentrancy) and slighlty twisted
semantics (i.e. ReentrantReadWriteLock allows multiple readers, we need
multiple writers).

Signed-off-by: Karsten Blees <blees@dcon.de>
AsyncStore: add a unit test that exposes some of the problems in 4bb904aa

Signed-off-by: Karsten Blees <blees@dcon.de>
AsyncStore: more tests that expose problems in 4bb904aa

"ISPN-1174 Fix async cache store consistency issues with some operations"
completely breaks eviction and the LIRS strategy.

All these tests pass with disabled AsyncStore (USE_ASYNC_STORE = false).

Signed-off-by: Karsten Blees <blees@dcon.de>
AsyncStore: fix unit tests

Many AsyncStore tests flush the AsyncStore (stop() + start()) and wait for
the changes to appear in the back-end CacheStore. This is not how a
CacheStore is supposed to work (at least, Infinispan's Cache implementation
doesn't use it that way). Changes made via CacheStore.store/remove/clear
should be immediately visible via CacheStore.load(), even if they haven't
been flushed to the back-end store yet.

Fix the unit test by removing surplus stop/start calls and wait loops.

Additionally, add four end-to-end tests to verify that updates and removes
work correctly in Caches using the AsyncStore.

Also test if the CacheStore blocks if the configured modificationQueueSize
is reached (to prevent OutOfMemoryError if entries are added faster than
the back-end store can handle them).

Signed-off-by: Karsten Blees <blees@dcon.de>
Compare two async stores and minor fix to test
AsyncStoreStressTest: make the test profiler-friendly

Add a main function and provide meaningful worker thread names to simplify
executing the stress test in a profiler.

Signed-off-by: Karsten Blees <blees@dcon.de>
AsyncStoreStressTest: fix statistics calculation

The '*/s' statistics printed by the test are actually operations per second
_per thread_ (i.e. more threads result in smaller numbers). Factor out the
thread count to get comparable results.

Signed-off-by: Karsten Blees <blees@dcon.de>
Apply Karten's stats fix to map stress test too
AsyncStoreStressTest: fix bottlenecks in the test classes

Put performance is affected by additional debug log output. Make sure all
worker threads log with trace level, so that logging affects them all in
the same way.

Most of the execution time is spent in DummyInMemoryCacheStore.record, due
to busy waiting trying to update statistics. Use AtomicInteger to get rid
of Thread.sleep(200).

Signed-off-by: Karsten Blees <blees@dcon.de>
Remove debug log calls and transform them into trace
AsyncStore2: allow the modificationQueueSize feature to be switched off

The modificationQueueSize feature doesn't work properly in the original
AsyncStore. Allow the feature to be switched off in the new AsyncStore
implementation (by setting modificationQueueSize == 0) to get meaningful
performance measurements.

With modificationQueueSize == 0, it is no longer possible to calculate the
batch size of AsyncStoreProcessor threads in advance. Rewrite the thread
setup code to distribute entries more evenly across available threads.

Signed-off-by: Karsten Blees <blees@dcon.de>
Revert async store related changes introduced with ISPN-1174

* Fix remaining unit tests to work with new async store.