amplab/tachyon — pull request #574 by ShaoshanLiu (2014-12-16 22:45:03+00:00)
Title: FIX ISSUE TACHYON-251: race condition between RemoteBlockInStream close() and doneReCache()

https://tachyon.atlassian.net/browse/TACHYON-251?filter=-1

there is a race condition in RemoteBlockInStream, between method close() and DoneReCache(). When close is called, currently it would cancel BlockOutStream if block out stream is not yet done writing to cache, leading to remote blocks not able to be cached.

The root cause of this problem is the lack of synchronization between close() and DoneReCache(). In this change, we added synchronization mechanisms between this two methods such that:
1. when close() is called, it waits for recache to be done with a timeout of 5 seconds
2. if recache is not done when timeout expires, it would fall back to the original logic to cancel BlockOutStream.
3. otherwise, we make sure that remote blocks are properly cached.

Performance:

Before Change:
Query Time taken: 160.786 seconds
Query Time taken: 131.347 seconds

After Change:
Query Time taken: 5.9 seconds
Query Time taken: 5.891 seconds

Test Done Locally:
it has passed all unit tests locally with "mvn install"