Repository: minrk/wurlitzer

Issue:
wurlitzer.pipes hang if output from C is very large
<!-- Thank you for contributing. These HTML commments will not render in the issue, but you can delete them once you've read them if you prefer! -->

### Bug description
The following code works for most C function calls, but hangs when the C-side tries to output thousands of lines. Unfortunately, I am not able to quantify the number of kB at which it hangs:

```
            with pipes() as (stdout, stderr):
                result = c_function(args)
            lines = (it.strip() for it in stdout.readlines())
            for line in lines:
                print(line)
```

Naturally, when the `c_function` is called without the context manager, all lines are printed to stdout and application completes normally.

I have also tried changing the arguments to pipes, and I've tried sys_pipes(), etc.

<!-- Use this section to clearly and concisely describe the bug. -->

#### Expected behaviour
<!-- Tell us what you thought would happen. -->

I know that pipes filling up can cause this, so I suspect that the stdout buffer is exceeding 64k. If you have any suggestions as to how to increase that size, I'm open to that as well; however adding that as an argument to `pipes` would be ideal.

#### Actual behaviour
<!-- Tell us what actually happens. -->

### How to reproduce
<!-- Use this section to describe the steps that a user would take to experience this bug. -->

Try creating a C function that outputs many kB of code and determine at which point it hangs.

### Your personal set up
<!-- Tell us a little about the system you're using. -->

 - OS: linux, SL7
 <!-- [e.g. linux, OSX] -->
 - Version: python3.8
 <!-- e.g. jupyterhub --version. --->
 - Configuration:
 <!-- Be careful not to share any sensitive information. --->


Discussion:

Patch:
diff --git a/wurlitzer.py b/wurlitzer.py
index 9a48e19..d786c89 100644
--- a/wurlitzer.py
+++ b/wurlitzer.py
@@ -32,8 +32,9 @@ from queue import Queue
 try:
     from fcntl import F_SETPIPE_SZ
 except ImportError:
-    # ref: linux fcntl.h
+    # ref: linux uapi/linux/fcntl.h
     F_SETPIPE_SZ = 1024 + 7
+    F_GETPIPE_SZ = 1024 + 8
 
 libc = ctypes.CDLL(None)
 
@@ -200,17 +201,16 @@ class Wurlitzer:
         self._save_fds[name] = save_fd
 
         pipe_out, pipe_in = os.pipe()
-        dup2(pipe_in, real_fd)
-        os.close(pipe_in)
-        self._real_fds[name] = real_fd
-
         # set max pipe buffer size (linux only)
         if self._bufsize:
             try:
                 fcntl(pipe_in, F_SETPIPE_SZ, self._bufsize)
             except OSError:
-                # ignore failure to set pipe size
-                pass
+                warnings.warn("Failed to set pipe buffer size", RuntimeWarning)
+
+        dup2(pipe_in, real_fd)
+        os.close(pipe_in)
+        self._real_fds[name] = real_fd
 
         # make pipe_out non-blocking
         flags = fcntl(pipe_out, F_GETFL)


Test patch:
diff --git a/test.py b/test.py
index 42ee5d3..7ca4c89 100644
--- a/test.py
+++ b/test.py
@@ -4,10 +4,14 @@ from __future__ import print_function
 import io
 import os
 import platform
+import sys
 import time
+from fcntl import fcntl
 from tempfile import TemporaryFile
 from unittest import mock
 
+import pytest
+
 import wurlitzer
 from wurlitzer import (
     PIPE,
@@ -158,3 +162,18 @@ def test_pipe_max_size():
         assert 65535 <= max_pipe_size <= 1024 * 1024
     else:
         assert max_pipe_size is None
+
+
+@pytest.mark.skipif(
+    wurlitzer._get_max_pipe_size() is None, reason="requires _get_max_pipe_size"
+)
+def test_bufsize():
+    default_bufsize = wurlitzer._get_max_pipe_size()
+    with wurlitzer.pipes() as (stdout, stderr):
+        assert fcntl(sys.__stdout__, wurlitzer.F_GETPIPE_SZ) == default_bufsize
+        assert fcntl(sys.__stderr__, wurlitzer.F_GETPIPE_SZ) == default_bufsize
+
+    bufsize = 32768  # seems to only accept powers of two?
+    with wurlitzer.pipes(bufsize=bufsize) as (stdout, stderr):
+        assert fcntl(sys.__stdout__, wurlitzer.F_GETPIPE_SZ) == bufsize
+        assert fcntl(sys.__stderr__, wurlitzer.F_GETPIPE_SZ) == bufsize