hazelcast/hazelcast — pull request #4056 by pveentjer (2014-11-11 06:13:39+00:00)
Title: BasicInvocation reduced locking and removal of scheduler for backups

This commit has 2 important changes:
- removal of unwanted locking in the BasicInvocation. The previous implementation contains quite a lot of locking. Most of the locking has been removed.
- removal of the scheduler to deal with backup timeout.

It is now done using a periodic scan over the pending invocations. The problem with using a scheduler is that tasks are being stored, even though the whole operation including its backups could have completed a long time ago. In case of a 5 second timeout it isn't that big of an issue, but if this time is increased than you can get an OOME. For the back pressure we need to have a larger timeout to be able to provide back pressure and we immediately ran into OOME problems.

Apart from not running into an OOME, it gave quite a nice performance improvement for backup aware operations because dealing with the scheduled executor was quite an expensive operation. In our tests we saw a 35% throughput improvement.