Repository: DataBiosphere/toil

Issue:
readFileStream deadlocks when not read through completely
I think I was aware of this at one point, but using readFileStream without reading all the way to EOF will totally deadlock the worker process on anything but the file jobStore. This is pretty unintuitive (surely this is one of the only use cases for streaming the file?). It may be especially confusing to new users who aren't used to toil deadlocks.

Unfortunately, it's not easy to fix. The writing thread will be blocked on write, so there's no way to wake it up and tell the writing thread to quit. Python threads can't (safely) be killed from their parents. The thread *should* have dropped the GIL while blocked on the write, so we might barely get away with killing its PID. But I'd suggest adjusting the ReadableThread/WritableThread classes to use processes rather than threads. They already use pipes, so very little would need to change. There would be a bit of added overhead from the extra fork(), but surely that's worth getting rid of some of these GIL nightmares.
readFileStream deadlocks when not read through completely
I think I was aware of this at one point, but using readFileStream without reading all the way to EOF will totally deadlock the worker process on anything but the file jobStore. This is pretty unintuitive (surely this is one of the only use cases for streaming the file?). It may be especially confusing to new users who aren't used to toil deadlocks.

Unfortunately, it's not easy to fix. The writing thread will be blocked on write, so there's no way to wake it up and tell the writing thread to quit. Python threads can't (safely) be killed from their parents. The thread *should* have dropped the GIL while blocked on the write, so we might barely get away with killing its PID. But I'd suggest adjusting the ReadableThread/WritableThread classes to use processes rather than threads. They already use pipes, so very little would need to change. There would be a bit of added overhead from the extra fork(), but surely that's worth getting rid of some of these GIL nightmares.


Discussion:

Setting thread.daemon to True may solve the deadlock problems:

        self.__thread = threading.Thread(target=self.__reader)
        self.__thread.daemon = True

 http://bugs.python.org/issue21822
 http://code.activestate.com/recipes/496735-workaround-for-missed-sigint-in-multithreaded-prog/


Joel Armstrong <notifications@github.com> writes:
> I think I was aware of this at one point, but using readFileStream without
> reading all the way to EOF will totally deadlock the worker process on anything
> but the file jobStore. This is pretty unintuitive (surely this is one of the
> only use cases for streaming the file?). It may be especially confusing to new
> users who aren't used to toil deadlocks.
> 
> Unfortunately, it's not easy to fix. The writing thread will be blocked on
> write, so there's no way to wake it up and tell the writing thread to quit.
> Python threads can't (safely) be killed from their parents. The thread should
> have dropped the GIL while blocked on the write, so we might barely get away
> with killing its PID. But I'd suggest adjusting the ReadableThread/
> WritableThread classes to use processes rather than threads. They already use
> pipes, so very little would need to change. There would be a bit of added
> overhead from the extra fork(), but surely that's worth getting rid of some of
> these GIL nightmares.
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub, or mute the thread.*
> 

I use thread.daemon all the time, it's really useful, but I don't think it'll solve this problem. The writing thread is stuck on write(), the other thread is stuck on thread.join(). We could possibly send SIGINT to the writing thread, and rely on that to cause an exception, but my Unix-fu is lacking, I don't know if that would always interrupt write(). The man page says:
```
       EINTR  The  call  was interrupted by a signal before any data was writ‐
              ten; see signal(7).
```
so does that imply that write blocks the signal handler from running if it's already written something but is blocked on writing more?

We could attempt to be sane and use timeouts in the writing thread, but we often just delegate to boto or shutil.copyfileobj which don't support timeouts.

Does the read thread close the stream before join()?  this would
cause the write to fail.





Joel Armstrong <notifications@github.com> writes:
> I use thread.daemon all the time, it's really useful, but I don't think it'll
> solve this problem. The writing thread is stuck on write(), the other thread is
> stuck on thread.join(). We could possibly send SIGINT to the writing thread,
> and rely on that to cause an exception, but my Unix-fu is lacking, I don't know
> if that would always interrupt write(). The man page says:
> 
>        EINTR  The  call  was interrupted by a signal before any data was writ‐
>               ten; see signal(7).
> 
> so does that imply that write blocks the signal handler from running if it's
> already written something but is blocked on writing more?
> 
> We could attempt to be sane and use timeouts in the writing thread, but we
> often just delegate to boto or shutil.copyfileobj which don't support timeouts.
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or mute the thread.*
> 

d'oh, that's brilliant! I'll try that.

Setting thread.daemon to True may solve the deadlock problems:

        self.__thread = threading.Thread(target=self.__reader)
        self.__thread.daemon = True

 http://bugs.python.org/issue21822
 http://code.activestate.com/recipes/496735-workaround-for-missed-sigint-in-multithreaded-prog/


Joel Armstrong <notifications@github.com> writes:
> I think I was aware of this at one point, but using readFileStream without
> reading all the way to EOF will totally deadlock the worker process on anything
> but the file jobStore. This is pretty unintuitive (surely this is one of the
> only use cases for streaming the file?). It may be especially confusing to new
> users who aren't used to toil deadlocks.
> 
> Unfortunately, it's not easy to fix. The writing thread will be blocked on
> write, so there's no way to wake it up and tell the writing thread to quit.
> Python threads can't (safely) be killed from their parents. The thread should
> have dropped the GIL while blocked on the write, so we might barely get away
> with killing its PID. But I'd suggest adjusting the ReadableThread/
> WritableThread classes to use processes rather than threads. They already use
> pipes, so very little would need to change. There would be a bit of added
> overhead from the extra fork(), but surely that's worth getting rid of some of
> these GIL nightmares.
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub, or mute the thread.*
> 

I use thread.daemon all the time, it's really useful, but I don't think it'll solve this problem. The writing thread is stuck on write(), the other thread is stuck on thread.join(). We could possibly send SIGINT to the writing thread, and rely on that to cause an exception, but my Unix-fu is lacking, I don't know if that would always interrupt write(). The man page says:
```
       EINTR  The  call  was interrupted by a signal before any data was writ‐
              ten; see signal(7).
```
so does that imply that write blocks the signal handler from running if it's already written something but is blocked on writing more?

We could attempt to be sane and use timeouts in the writing thread, but we often just delegate to boto or shutil.copyfileobj which don't support timeouts.

Does the read thread close the stream before join()?  this would
cause the write to fail.





Joel Armstrong <notifications@github.com> writes:
> I use thread.daemon all the time, it's really useful, but I don't think it'll
> solve this problem. The writing thread is stuck on write(), the other thread is
> stuck on thread.join(). We could possibly send SIGINT to the writing thread,
> and rely on that to cause an exception, but my Unix-fu is lacking, I don't know
> if that would always interrupt write(). The man page says:
> 
>        EINTR  The  call  was interrupted by a signal before any data was writ‐
>               ten; see signal(7).
> 
> so does that imply that write blocks the signal handler from running if it's
> already written something but is blocked on writing more?
> 
> We could attempt to be sane and use timeouts in the writing thread, but we
> often just delegate to boto or shutil.copyfileobj which don't support timeouts.
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or mute the thread.*
> 

d'oh, that's brilliant! I'll try that.

Resolving patch:
diff --git a/src/toil/jobStores/utils.py b/src/toil/jobStores/utils.py
--- a/src/toil/jobStores/utils.py
+++ b/src/toil/jobStores/utils.py
@@ -1,5 +1,6 @@
 import logging
 import os
+import errno
 from abc import ABCMeta
 from abc import abstractmethod
 
@@ -105,10 +106,14 @@ def __exit__(self, exc_type, exc_val, exc_tb):
             self.writable.close()
             # Closeing the writable end will send EOF to the readable and cause the reader thread
             # to finish.
+            if self.thread is not None:
+                # reraises any exception that was raised in the thread
+                self.thread.join()
+        except:
             if exc_type is None:
-                if self.thread is not None:
-                    # reraises any exception that was raised in the thread
-                    self.thread.join()
+                # Only raise the child exception if there wasn't
+                # already an exception in the main thread
+                raise
         finally:
             # The responsibility for closing the readable end is generally that of the reader
             # thread. To cover the small window before the reader takes over we also close it here.
@@ -119,13 +124,6 @@ def __exit__(self, exc_type, exc_val, exc_tb):
                 os.close(readable_fh)
 
 
-# FIXME: Unfortunately these two classes are almost an exact mirror image of each other.
-# Basically, read and write are swapped. The only asymmetry lies in how shutdown is handled. I
-# tried generalizing but the code becomes inscrutable. Until I (or someone else) has a better
-# idea how to solve this, I think its better to have code that is readable at the expense of
-# duplication.
-
-
 class ReadablePipe(object):
     """
     An object-oriented wrapper for os.pipe. Clients should subclass it, implement
@@ -201,10 +199,14 @@ def writeTo(self, writable):
         raise NotImplementedError()
 
     def _writer(self):
-        with os.fdopen(self.writable_fh, 'w') as writable:
-            # FIXME: another race here, causing a redundant attempt to close in the main thread
-            self.writable_fh = None  # signal to parent thread that we've taken over
-            self.writeTo(writable)
+        try:
+            with os.fdopen(self.writable_fh, 'w') as writable:
+                self.writeTo(writable)
+        except IOError as e:
+            # The other side of the pipe may have been closed by the
+            # reading thread, which is OK.
+            if e.errno != errno.EPIPE:
+                raise
 
     def __init__(self):
         super(ReadablePipe, self).__init__()
@@ -220,17 +222,16 @@ def __enter__(self):
         return self.readable
 
     def __exit__(self, exc_type, exc_val, exc_tb):
+        # Close the read end of the pipe. The writing thread may
+        # still be writing to the other end, but this will wake it up
+        # if that's the case.
+        self.readable.close()
         try:
+            if self.thread is not None:
+                # reraises any exception that was raised in the thread
+                self.thread.join()
+        except:
             if exc_type is None:
-                if self.thread is not None:
-                    # reraises any exception that was raised in the thread
-                    self.thread.join()
-        finally:
-            self.readable.close()
-            # The responsibility for closing the writable end is generally that of the writer
-            # thread. To cover the small window before the writer takes over we also close it here.
-            writable_fh = self.writable_fh
-            if writable_fh is not None:
-                # FIXME: This is still racy. The writer thread could close it now, and someone
-                # else may immediately open a new file, reusing the file handle.
-                os.close(writable_fh)
+                # Only raise the child exception if there wasn't
+                # already an exception in the main thread
+                raise


Test patch:
diff --git a/src/toil/test/jobStores/jobStoreTest.py b/src/toil/test/jobStores/jobStoreTest.py
--- a/src/toil/test/jobStores/jobStoreTest.py
+++ b/src/toil/test/jobStores/jobStoreTest.py
@@ -14,6 +14,7 @@
 
 from __future__ import absolute_import
 
+import pytest
 import SocketServer
 import hashlib
 import logging
@@ -713,6 +714,25 @@ def testManyJobs(self):
             # on the jobs iterator for certain cloud providers
             self.assertTrue(len(allJobs) <= 3001)
 
+        # NB: the 'thread' method seems to be needed here to actually
+        # ensure the timeout is raised, probably because the only
+        # "live" thread doesn't hold the GIL.
+        @pytest.mark.timeout(45, method='thread')
+        def testPartialReadFromStream(self):
+            """Test whether readFileStream will deadlock on a partial read."""
+            job = self.master.create(self.arbitraryJob)
+            with self.master.writeFileStream(job.jobStoreID) as (f, fileID):
+                # Write enough data to make sure the writer thread
+                # will get blocked on the write. Technically anything
+                # greater than the pipe buffer size plus the libc
+                # buffer size (64K + 4K(?))  should trigger this bug,
+                # but this gives us a lot of extra room just to be
+                # sure.
+                f.write('a' * 300000)
+            with self.master.readFileStream(fileID) as f:
+                self.assertEquals(f.read(1), "a")
+            # If it times out here, there's a deadlock
+
         @abstractmethod
         def _corruptJobStore(self):
             """