Repository: urllib3/urllib3

Issue:
Interrupted system call while profiling with plop
Using Python 2.7.12, requests 2.19.0, urllib3 1.23 and using [plop](https://github.com/bdarnell/plop) for profiling, I'm intermittently hitting this stack trace in long-running code:
```
  File "/home/bmerry/work/sdp/env/local/lib/python2.7/site-packages/requests/sessions.py", line 525, in get
    return self.request('GET', url, **kwargs)
  File "/home/bmerry/work/sdp/env/local/lib/python2.7/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/bmerry/work/sdp/env/local/lib/python2.7/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/home/bmerry/work/sdp/git/katdal/katdal/chunkstore_s3.py", line 56, in send
    return super(_TimeoutHTTPAdapter, self).send(request, stream, timeout, *args, **kwargs)
  File "/home/bmerry/work/sdp/env/local/lib/python2.7/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/home/bmerry/work/sdp/env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 588, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/home/bmerry/work/sdp/env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 239, in _get_conn
    if conn and is_connection_dropped(conn):
  File "/home/bmerry/work/sdp/env/local/lib/python2.7/site-packages/urllib3/util/connection.py", line 23, in is_connection_dropped
    return wait_for_read(sock, timeout=0.0)
  File "/home/bmerry/work/sdp/env/local/lib/python2.7/site-packages/urllib3/util/wait.py", line 146, in wait_for_read
    return wait_for_socket(sock, read=True, timeout=timeout)
  File "/home/bmerry/work/sdp/env/local/lib/python2.7/site-packages/urllib3/util/wait.py", line 107, in poll_wait_for_socket
    return bool(_retry_on_intr(do_poll, timeout))
  File "/home/bmerry/work/sdp/env/local/lib/python2.7/site-packages/urllib3/util/wait.py", line 47, in _retry_on_intr
    return fn(timeout)
  File "/home/bmerry/work/sdp/env/local/lib/python2.7/site-packages/urllib3/util/wait.py", line 105, in do_poll
    return poll_obj.poll(t)
select.error: (4, 'Interrupted system call')
Profiling timer expired
```

Looking at the implementation of `_retry_on_intr` for older Pythons, it has this special case:
```python
        if timeout is not None and timeout <= 0:
            return fn(timeout)
```
which in turn seems to apply to the call stack above (see is_connection_dropped, which passes a timeout of 0.0). So apparently there are cases where poll can fail with EINTR even with a zero timeout. FWIW, I'm running Ubuntu 16.04 and Linux 4.4.0-116-generic. I'll try commenting out that fast path and doing some testing overnight to confirm that that is the problem.

I don't yet had a minimal reproducible example, but I'll work on it (my first attempt of just banging on some URL in a loop hasn't worked). I wanted to file this before I forgot.


Discussion:
I thought our select code handled `EINTR`, I know there's been changes to that code recently.
> I thought our select code handled EINTR

It does... except in the case of a zero timeout, where the code I quoted above bypasses the EINTR handling. Here's the [context](https://github.com/urllib3/urllib3/blob/6be637268931e100b0d2d15cecd5524323fd4f36/urllib3/util/wait.py#L46-L47) of that code. I presume it's special-cased to avoid the cost of calling `monotonic()`, but the same effect could probably be achieved by setting `deadline` to 0 (or -inf) if timeout is zero.

I got around to running the same code with those two lines commented out, and can confirm that it fixed the problem. If I get time I'll see if I can write a minimal example, but it may turn out to be sensitive to the access patterns/timings on the particular internal HTTP server I'm hitting. I'll also look at doing a PR with the fix suggested above.
Ok, I've managed to reproduce the problem. My first attempt was run against a Python SimpleHTTPServer, which only implements HTTP 1.0 and hence there is no keep-alive and no connection reuse. When run against an HTTP 1.1 server (I used the local CUPS server on my machine just because that's what was running) this crashes within seconds using urllib3 1.23:

```python
#!/usr/bin/env python
import plop.collector
import requests


collector = plop.collector.Collector(interval=0.0001)
collector.start()
session = requests.Session()
while True:
    session.get('http://localhost:631/', timeout=3)
```

If I get time I'll write a PR to fix it.

Resolving patch:
diff --git a/urllib3/util/wait.py b/urllib3/util/wait.py
--- a/urllib3/util/wait.py
+++ b/urllib3/util/wait.py
@@ -43,9 +43,6 @@ def _retry_on_intr(fn, timeout):
 else:
     # Old and broken Pythons.
     def _retry_on_intr(fn, timeout):
-        if timeout is not None and timeout <= 0:
-            return fn(timeout)
-
         if timeout is None:
             deadline = float("inf")
         else:
@@ -117,7 +114,7 @@ def _have_working_poll():
     # from libraries like eventlet/greenlet.
     try:
         poll_obj = select.poll()
-        poll_obj.poll(0)
+        _retry_on_intr(poll_obj.poll, 0)
     except (AttributeError, OSError):
         return False
     else:


Test patch:
diff --git a/test/test_wait.py b/test/test_wait.py
--- a/test/test_wait.py
+++ b/test/test_wait.py
@@ -137,6 +137,40 @@ def handler(sig, frame):
     assert interrupt_count[0] > 0
 
 
+@pytest.mark.skipif(
+    not hasattr(signal, "setitimer"),
+    reason="need setitimer() support"
+)
+@pytest.mark.parametrize("wfs", variants)
+def test_eintr_zero_timeout(wfs, spair):
+    a, b = spair
+    interrupt_count = [0]
+
+    def handler(sig, frame):
+        assert sig == signal.SIGALRM
+        interrupt_count[0] += 1
+
+    old_handler = signal.signal(signal.SIGALRM, handler)
+    try:
+        assert not wfs(a, read=True, timeout=0)
+        try:
+            # Start delivering SIGALRM 1000 times per second,
+            # to trigger race conditions such as
+            # https://github.com/urllib3/urllib3/issues/1396.
+            signal.setitimer(signal.ITIMER_REAL, 0.001, 0.001)
+            # Hammer the system call for a while to trigger the
+            # race.
+            for i in range(100000):
+                wfs(a, read=True, timeout=0)
+        finally:
+            # Stop delivering SIGALRM
+            signal.setitimer(signal.ITIMER_REAL, 0)
+    finally:
+        signal.signal(signal.SIGALRM, old_handler)
+
+    assert interrupt_count[0] > 0
+
+
 @pytest.mark.skipif(
     not hasattr(signal, "setitimer"),
     reason="need setitimer() support"