Repository: certbot/certbot

Issue:
I/O possible
I often (but not always) get the `I/O possible` error. It's been happening for a while now, long before boulder branch got included.

```
DEBUG:root:Sending POST data: {"header": {"alg": "RS256", "jwk": {"e": "AQAB", "kty": "RSA", "n": "rkxqHTiL6PtN8FIQt9MblqmLi0Hry4NrWjQQdwnQ8vwRb16RKjxLF5O4J6Xkv4pjeAsVJLJi5qpiLFz2uQn8m4PiQI6fY0lyo3ag2EXAYUaJsZRM92pcGWTEJYzWo55JwSzmHwxnaG9f3qp06IFz_xQ3qVm7g05iNDDYdOs-t1Mxz4fIrQ3brYfbFg80Xv_OhGvPGI4Ntm2f7VqEgGlTlRRuXl9-6b1-oictlekbATkIxojPZfB-ZbPCBcHViKJ9Mw0rD8bJbDCdl-I_yXeuNfAjLrmpjF7MuorKoJ7dK08otDzkebOIJP7qQ3BbyQd0uG7BMwwAh5BJwytmkcuVbw"}}, "payload": "eyJpZGVudGlmaWVyIjogeyJ0eXBlIjogImRucyIsICJ2YWx1ZSI6ICJmb28uY29tIn19", "signature": "XMJxL7idrIsh2ieJlElc7H8b57S6gsBDpC84DkL3FvDf5Quhb0wQRdsqWAmEnEXC8gF3TIEzGI4W2nTInylU2ca2H_rcJhhpTMObKmaZ1rX0Hqx8QFiDVoeu7XNVGQA81RV_KNlo0ej4YiusRvJr7VRK2uFB9Wn7kM6KFj3la_-kEoYTAmaRBcryzzcPlXLbKa5aeUtZT08eJ6fi_7f6J3WuseWpnyjiDv1wJt_HftsLJ00ix3EEygkuJJtRCbf9clAuuJfRTPpdDc42TuNF3wrlFlItSqwFMn-Pw2X4jMJGI2kGf_tmVaTVSL5Wfru3ZWj-LnXn3MAbA89aP1dr7A"}
DEBUG:root:Received response <Response [201]>: {"identifier":{"type":"dns","value":"foo.com"},"key":{"e":"AQAB","kty":"RSA","n":"rkxqHTiL6PtN8FIQt9MblqmLi0Hry4NrWjQQdwnQ8vwRb16RKjxLF5O4J6Xkv4pjeAsVJLJi5qpiLFz2uQn8m4PiQI6fY0lyo3ag2EXAYUaJsZRM92pcGWTEJYzWo55JwSzmHwxnaG9f3qp06IFz_xQ3qVm7g05iNDDYdOs-t1Mxz4fIrQ3brYfbFg80Xv_OhGvPGI4Ntm2f7VqEgGlTlRRuXl9-6b1-oictlekbATkIxojPZfB-ZbPCBcHViKJ9Mw0rD8bJbDCdl-I_yXeuNfAjLrmpjF7MuorKoJ7dK08otDzkebOIJP7qQ3BbyQd0uG7BMwwAh5BJwytmkcuVbw"},"status":"pending","expires":"0001-01-01T00:00:00Z","challenges":[{"type":"simpleHttps","status":"pending","uri":"https://www.letsencrypt-demo.org/acme/authz/5QAI1_s1ZU71HpghUM0RATsaRrZAQpuT4yMxTHXfKDA?challenge=0","token":"mNJS7oiNFH2GfVAsPc9Nr763Xld1PAZLmQNe0dhkSeQ"},{"type":"dvsni","status":"pending","uri":"https://www.letsencrypt-demo.org/acme/authz/5QAI1_s1ZU71HpghUM0RATsaRrZAQpuT4yMxTHXfKDA?challenge=1","r":"mxgVtg9L7imjzQOGTjCbziygn388rZAtFZXWrjdzCn0","nonce":"f7ca5861765fb80b03415942139383da"}],"combinations":[[0],[1]]}
INFO:root:Performing the following challenges:
INFO:root:  DVSNI challenge for foo.com.
I/O possible
(venv)root@le:~/lets-encrypt-preview# echo $?
157
(venv)root@le:~/lets-encrypt-preview# ps auxxww | grep letse
root     15423  0.0  7.6 121884 38696 pts/1    S    14:45   0:00 /root/lets-encrypt-preview/venv/bin/python2 /root/lets-encrypt-preview/venv/bin/letsencrypt -vvvv -ted foo.com -m admin@foo.com auth
```

I/O possible
I often (but not always) get the `I/O possible` error. It's been happening for a while now, long before boulder branch got included.

```
DEBUG:root:Sending POST data: {"header": {"alg": "RS256", "jwk": {"e": "AQAB", "kty": "RSA", "n": "rkxqHTiL6PtN8FIQt9MblqmLi0Hry4NrWjQQdwnQ8vwRb16RKjxLF5O4J6Xkv4pjeAsVJLJi5qpiLFz2uQn8m4PiQI6fY0lyo3ag2EXAYUaJsZRM92pcGWTEJYzWo55JwSzmHwxnaG9f3qp06IFz_xQ3qVm7g05iNDDYdOs-t1Mxz4fIrQ3brYfbFg80Xv_OhGvPGI4Ntm2f7VqEgGlTlRRuXl9-6b1-oictlekbATkIxojPZfB-ZbPCBcHViKJ9Mw0rD8bJbDCdl-I_yXeuNfAjLrmpjF7MuorKoJ7dK08otDzkebOIJP7qQ3BbyQd0uG7BMwwAh5BJwytmkcuVbw"}}, "payload": "eyJpZGVudGlmaWVyIjogeyJ0eXBlIjogImRucyIsICJ2YWx1ZSI6ICJmb28uY29tIn19", "signature": "XMJxL7idrIsh2ieJlElc7H8b57S6gsBDpC84DkL3FvDf5Quhb0wQRdsqWAmEnEXC8gF3TIEzGI4W2nTInylU2ca2H_rcJhhpTMObKmaZ1rX0Hqx8QFiDVoeu7XNVGQA81RV_KNlo0ej4YiusRvJr7VRK2uFB9Wn7kM6KFj3la_-kEoYTAmaRBcryzzcPlXLbKa5aeUtZT08eJ6fi_7f6J3WuseWpnyjiDv1wJt_HftsLJ00ix3EEygkuJJtRCbf9clAuuJfRTPpdDc42TuNF3wrlFlItSqwFMn-Pw2X4jMJGI2kGf_tmVaTVSL5Wfru3ZWj-LnXn3MAbA89aP1dr7A"}
DEBUG:root:Received response <Response [201]>: {"identifier":{"type":"dns","value":"foo.com"},"key":{"e":"AQAB","kty":"RSA","n":"rkxqHTiL6PtN8FIQt9MblqmLi0Hry4NrWjQQdwnQ8vwRb16RKjxLF5O4J6Xkv4pjeAsVJLJi5qpiLFz2uQn8m4PiQI6fY0lyo3ag2EXAYUaJsZRM92pcGWTEJYzWo55JwSzmHwxnaG9f3qp06IFz_xQ3qVm7g05iNDDYdOs-t1Mxz4fIrQ3brYfbFg80Xv_OhGvPGI4Ntm2f7VqEgGlTlRRuXl9-6b1-oictlekbATkIxojPZfB-ZbPCBcHViKJ9Mw0rD8bJbDCdl-I_yXeuNfAjLrmpjF7MuorKoJ7dK08otDzkebOIJP7qQ3BbyQd0uG7BMwwAh5BJwytmkcuVbw"},"status":"pending","expires":"0001-01-01T00:00:00Z","challenges":[{"type":"simpleHttps","status":"pending","uri":"https://www.letsencrypt-demo.org/acme/authz/5QAI1_s1ZU71HpghUM0RATsaRrZAQpuT4yMxTHXfKDA?challenge=0","token":"mNJS7oiNFH2GfVAsPc9Nr763Xld1PAZLmQNe0dhkSeQ"},{"type":"dvsni","status":"pending","uri":"https://www.letsencrypt-demo.org/acme/authz/5QAI1_s1ZU71HpghUM0RATsaRrZAQpuT4yMxTHXfKDA?challenge=1","r":"mxgVtg9L7imjzQOGTjCbziygn388rZAtFZXWrjdzCn0","nonce":"f7ca5861765fb80b03415942139383da"}],"combinations":[[0],[1]]}
INFO:root:Performing the following challenges:
INFO:root:  DVSNI challenge for foo.com.
I/O possible
(venv)root@le:~/lets-encrypt-preview# echo $?
157
(venv)root@le:~/lets-encrypt-preview# ps auxxww | grep letse
root     15423  0.0  7.6 121884 38696 pts/1    S    14:45   0:00 /root/lets-encrypt-preview/venv/bin/python2 /root/lets-encrypt-preview/venv/bin/letsencrypt -vvvv -ted foo.com -m admin@foo.com auth
```



Discussion:
Ah, that was a run for `letsencrypt -vvvv -ted foo.com -m admin@foo.com auth`, standalone authenticator.

That sounds like a result of this [SIGIO from the standalone child to its parent](https://github.com/letsencrypt/lets-encrypt-preview/blob/master/letsencrypt/plugins/standalone/authenticator.py#L221) ("I/O possible" is the string description of the SIGIO signal).  But the fact that it's crashing is a bit odd, because the signal should be caught and handled [here](https://github.com/letsencrypt/lets-encrypt-preview/blob/master/letsencrypt/plugins/standalone/authenticator.py#L155).

@schoen is this a race condition, or something else?

@pde it's a race condition: the parent's first action is to set the signal handler, while the child signals the parent immediately after binding a port, which could nonetheless take a negligible amount of time.

I guess the correct solution would be to have the original process set the signal handler before forking, and then the child can revert the signal handler (which isn't applicable to it). There's no obvious way for the child to check whether or not the parent is ready to receive the signal, unless we want to start using other IPC primitives.

Ah, that was a run for `letsencrypt -vvvv -ted foo.com -m admin@foo.com auth`, standalone authenticator.

That sounds like a result of this [SIGIO from the standalone child to its parent](https://github.com/letsencrypt/lets-encrypt-preview/blob/master/letsencrypt/plugins/standalone/authenticator.py#L221) ("I/O possible" is the string description of the SIGIO signal).  But the fact that it's crashing is a bit odd, because the signal should be caught and handled [here](https://github.com/letsencrypt/lets-encrypt-preview/blob/master/letsencrypt/plugins/standalone/authenticator.py#L155).

@schoen is this a race condition, or something else?

@pde it's a race condition: the parent's first action is to set the signal handler, while the child signals the parent immediately after binding a port, which could nonetheless take a negligible amount of time.

I guess the correct solution would be to have the original process set the signal handler before forking, and then the child can revert the signal handler (which isn't applicable to it). There's no obvious way for the child to check whether or not the parent is ready to receive the signal, unless we want to start using other IPC primitives.


Resolving patch:
diff --git a/letsencrypt/plugins/standalone/authenticator.py b/letsencrypt/plugins/standalone/authenticator.py
--- a/letsencrypt/plugins/standalone/authenticator.py
+++ b/letsencrypt/plugins/standalone/authenticator.py
@@ -152,9 +152,6 @@ def do_parent_process(self, port, delay_amount=5):
         :rtype: bool
 
         """
-        signal.signal(signal.SIGIO, self.client_signal_handler)
-        signal.signal(signal.SIGUSR1, self.client_signal_handler)
-        signal.signal(signal.SIGUSR2, self.client_signal_handler)
 
         display = zope.component.getUtility(interfaces.IDisplay)
 
@@ -259,6 +256,16 @@ def start_listener(self, port, key):
         :rtype: bool
 
         """
+        # In order to avoid a race condition, we set the signal handler
+        # that will be needed by the parent process now, and undo this
+        # action if we turn out to be the child process.  (This needs
+        # to happen before the fork because the child will send one of
+        # these signals to the parent almost immediately after the
+        # fork, and the parent must already be ready to receive it.)
+        signal.signal(signal.SIGIO, self.client_signal_handler)
+        signal.signal(signal.SIGUSR1, self.client_signal_handler)
+        signal.signal(signal.SIGUSR2, self.client_signal_handler)
+
         fork_result = os.fork()
         Crypto.Random.atfork()
         if fork_result:
@@ -269,6 +276,12 @@ def start_listener(self, port, key):
             return self.do_parent_process(port)
         else:
             # CHILD process (the TCP listener subprocess)
+            # Undo the parent's signal handler settings, which aren't
+            # applicable to us.
+            signal.signal(signal.SIGIO, signal.SIG_DFL)
+            signal.signal(signal.SIGUSR1, signal.SIG_DFL)
+            signal.signal(signal.SIGUSR2, signal.SIG_DFL)
+
             self.child_pid = os.getpid()
             # do_child_process() is normally not expected to return but
             # should terminate via sys.exit().


Test patch:
diff --git a/letsencrypt/plugins/standalone/tests/authenticator_test.py b/letsencrypt/plugins/standalone/tests/authenticator_test.py
--- a/letsencrypt/plugins/standalone/tests/authenticator_test.py
+++ b/letsencrypt/plugins/standalone/tests/authenticator_test.py
@@ -404,47 +404,39 @@ def setUp(self):
             StandaloneAuthenticator
         self.authenticator = StandaloneAuthenticator(config=None, name=None)
 
-    @mock.patch("letsencrypt.plugins.standalone.authenticator.signal.signal")
     @mock.patch("letsencrypt.plugins.standalone.authenticator."
                 "zope.component.getUtility")
-    def test_do_parent_process_ok(self, mock_get_utility, mock_signal):
+    def test_do_parent_process_ok(self, mock_get_utility):
         self.authenticator.subproc_state = "ready"
         result = self.authenticator.do_parent_process(1717)
         self.assertTrue(result)
         self.assertEqual(mock_get_utility.call_count, 1)
-        self.assertEqual(mock_signal.call_count, 3)
 
-    @mock.patch("letsencrypt.plugins.standalone.authenticator.signal.signal")
     @mock.patch("letsencrypt.plugins.standalone.authenticator."
                 "zope.component.getUtility")
-    def test_do_parent_process_inuse(self, mock_get_utility, mock_signal):
+    def test_do_parent_process_inuse(self, mock_get_utility):
         self.authenticator.subproc_state = "inuse"
         result = self.authenticator.do_parent_process(1717)
         self.assertFalse(result)
         self.assertEqual(mock_get_utility.call_count, 1)
-        self.assertEqual(mock_signal.call_count, 3)
 
-    @mock.patch("letsencrypt.plugins.standalone.authenticator.signal.signal")
     @mock.patch("letsencrypt.plugins.standalone.authenticator."
                 "zope.component.getUtility")
-    def test_do_parent_process_cantbind(self, mock_get_utility, mock_signal):
+    def test_do_parent_process_cantbind(self, mock_get_utility):
         self.authenticator.subproc_state = "cantbind"
         result = self.authenticator.do_parent_process(1717)
         self.assertFalse(result)
         self.assertEqual(mock_get_utility.call_count, 1)
-        self.assertEqual(mock_signal.call_count, 3)
 
-    @mock.patch("letsencrypt.plugins.standalone.authenticator.signal.signal")
     @mock.patch("letsencrypt.plugins.standalone.authenticator."
                 "zope.component.getUtility")
-    def test_do_parent_process_timeout(self, mock_get_utility, mock_signal):
+    def test_do_parent_process_timeout(self, mock_get_utility):
         # Normally times out in 5 seconds and returns False.  We can
         # now set delay_amount to a lower value so that it times out
         # faster than it would under normal use.
         result = self.authenticator.do_parent_process(1717, delay_amount=1)
         self.assertFalse(result)
         self.assertEqual(mock_get_utility.call_count, 1)
-        self.assertEqual(mock_signal.call_count, 3)
 
 
 class DoChildProcessTest(unittest.TestCase):