infinispan/infinispan — pull request #1418 by anistor (2012-10-23 23:24:49+00:00)
Title: ISPN-2373 State transfer does not end because some segments are erroneously reported as unreceived

Jira: https://issues.jboss.org/browse/ISPN-2373

This was a really tough one to debug. I added more logging and a bunch of paranoid checks to be able to tell what's happening. The issue is finally solved by increasing the number of threads in async exec pool during testing but besides this I think it's worth integrating the other changes.

* Increased number of threads in async exec pool during tests. This is now a constant that can be easily modified in TestCacheManagerFactory.
* Do not accept state in StateConsumerImpl unless we have an InboundTransferTask waiting for it
* StateConsumerImpl.addTransfers : moved check for already active transfers inside the loop to prevent creating duplicate transfers
* Rename StateTransferLock.transactionDataReceived/topologyInstalled to notifyTransactionDataReceived/notifyTopologyInstalled
* Better exception logging for commands submitted to async executor
* Better trace logging
* OutboundTransferTask sends StateResponseCommands synchronously in order to ensure they are received in the same order and the receiver can properly detect termination by looking for isLast flag
* StateTransferManagerImpl.stop must countdown initialStateTransferComplete latch to ensure that a cache that stops before it completes the initial state transfer does not keep the client waiting indefinitely in waitForInitialStateTransferToComplete()