FIX ISSUE TACHYON-251: race condition between RemoteBlockInStream close() and doneReCache()

https://tachyon.atlassian.net/browse/TACHYON-251?filter=-1

there is a race condition in RemoteBlockInStream, between methods close() and DoneReCache().  When close() is called, currently it would cancel BlockOutStream if block out stream is not yet done writing to cache, leading to remote blocks not able to be cached. 

The root cause of this problem is the lack of synchronization between close() and DoneReCache().  In this change, we added synchronization mechanisms between this two methods such that:
1. when close() is called, it waits for recache to be done with a timeout of 5 seconds
2. if recache is not done when timeout expires, it would fall back to the original logic to cancel BlockOutStream.
3. otherwise, we make sure that remote blocks are properly cached.


Performance:

Before Change:
Query Time taken: 160.786 seconds
Query Time taken: 131.347 seconds


After Change:
Query Time taken: 5.9 seconds
Query Time taken: 5.891 seconds


Test Done Locally:
it has passed all unit tests locally with "mvn install"

Cool, thx!

On Thursday, December 18, 2014, Shaoshan Liu <notifications@github.com>
wrote:

> In core/src/main/java/tachyon/client/RemoteBlockInStream.java
> <https://github.com/amplab/tachyon/pull/574#discussion-diff-22084343>:
>
> > @@ -72,6 +72,21 @@
> >    private Object mUFSConf = null;
> >
> >    /**
> > +  * The object for synchronization to avoid race conditions between close() and doneRecache()
> > +  */
> > +  private Object mFlagDoneRecache = null;
>
> well, it saves a little bit of memory when mRecache is not used. But sure,
> I will change it to final as you suggested.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/amplab/tachyon/pull/574/files#r22084343>.
>