Repository: Pyomo/pyomo

Issue:
Error in solver tee output on Windows
## Summary

When I an solving a pyomo problem and use the "tee=True" option, and I scroll up in the CMD window on Windows to see some output displayed above, I often get the error below.

```sh
Number of objective function evaluations             = 30
Number of objective gradient evaluations             = 6
Number of equality constraint evaluations            = 31
Number of inequality constraint evaluations          = 0
Number of equality constraint Jacobian evaluations   = 6
Number of inequality constraint Jacobian evaluations = 0
Number of Lagrangian Hessian evaluations             = 5
Total CPU secs in IPOPT (w/o function evaluations)   =      0.668
Total CPU secs in NLP function evaluations           =     25.060

EXIT: Optimal Solution Found.
    solver.solve(m, tee=True)
  File "c:\users\eslickj\git\pyomo\pyomo\opt\base\solvers.py", line 578, in solve
    _status = self._apply_solver()
  File "c:\users\eslickj\git\pyomo\pyomo\opt\solver\shellcmd.py", line 246, in _apply_solver
    self._rc, self._log = self._execute_command(self._command)
  File "c:\users\eslickj\git\pyomo\pyomo\opt\solver\shellcmd.py", line 324, in _execute_command
    t.STDERR.flush()
  File "c:\users\eslickj\git\pyomo\pyomo\common\tee.py", line 390, in __exit__
    self.close(et is not None)
  File "c:\users\eslickj\git\pyomo\pyomo\common\tee.py", line 378, in close
    raise RuntimeError(
RuntimeError: TeeStream: deadlock observed joining reader threads
```

### Steps to reproduce the issue

I'm not sure this happens 100% of the time, but you just need to run a pyomo problem on windows in a cmd window and be scrolling up when the solver completes.

It's not with a specific script.  It happens with with any problem I run.  Should be reproducible with any problem using ipopt and "tee=True."   

### Error Message

```
  File "c:\users\eslickj\git\pyomo\pyomo\common\tee.py", line 378, in close
    raise RuntimeError(
RuntimeError: TeeStream: deadlock observed joining reader threads
```

### Information on your system

Pyomo version: Pyomo 6.0.2 (devel {string_args}) (CPython 3.9.4 on Windows 10)
Python version:  3.9.4
Operating system: Windows 10
How Pyomo was installed (PyPI, conda, source): source (pip with -e option)
Solver (if applicable): Ipopt





Discussion:
@eslickj, can you try editing the following line in `pyomo/common/tee.py`:
```diff
diff --git a/pyomo/common/tee.py b/pyomo/common/tee.py
index 170850a..b16feba 100644
--- a/pyomo/common/tee.py
+++ b/pyomo/common/tee.py
@@ -370,7 +370,7 @@ class TeeStream(object):
             if not self._threads:
                 break
             _poll *= 2
-            if _poll >= 2:  # bail after a total of 2 seconds * #threads
+            if _poll >= 10:  # bail after a total of ~10 seconds * #threads
                 if in_exception:
                     # We are already processing an exception: no reason
                     # to trigger another
```
@jsiirola yes.  I will try that.  I just observed if I drag the scroll bar thing back to the end, the error message only shows up when I let go of the mouse button.  I think when I grab the scroll bar with the mouse, output to the terminal must stop and be cached until I let go.  I wonder if that is causing some conflict here.
Is this happening at the command prompt, or within an IDE (Jupyter / Spyder / PyCharm / IPython / etc)?
The command prompt (CMD.exe).  I don't think it happens in iPython or Jupyter.  The change you suggested seems to make it harder to reproduce, but if I grab the scroll bar long enough it still happens.  Now that I played with it more, I'm pretty sure it has some thing to do with the the way the scroll bar works in CMD.  I'm going to try powershell, and see if it works the same.
Yeah, happens in powershell too.
So the problem arises when the subprocess has exited, but Python is still waiting for the threads that handled the subprocess stdout/stderr to exit (we close the pipe that was sending data to the threads, and then wait for the threads to notice and shut down).  Holding on to the scroll bar appears to block the thread from seeing that the stream is closed.

I generally worry about "utility" methods that lead to deadlock, so had the system bail if it started taking "too long."  -- but of course, "too long" is in the eye of the beholder.

What do you think is the right behavior?  Should we just deadlock? (that seems bad).  Alternatively we could just log an error instead of raising an exception.  The output streams could be left in a messy state, but the code *could* continue -- although if the next step was to open / delete / rename the file that the stream handler was writing too, that could lead to a different set of fatal errors in Windows.
Unfortunately I seem to run into this a lot.  Due to the way IDAES models work, I find myself scrolling up to see in previous solves worked right or to check on some output.  Then the whole script fails and I have to start over.  That's just to say I think it is important to fix since it can be very frustrating for users.  As for how to fix it, I don't really know.  It doesn't sound like there is a really good solution.  I'll put some more thought into this.



Resolving patch:
diff --git a/pyomo/common/tee.py b/pyomo/common/tee.py
--- a/pyomo/common/tee.py
+++ b/pyomo/common/tee.py
@@ -23,8 +23,18 @@
 import time
 from io import StringIO
 
-_mswindows = sys.platform.startswith('win')
 _poll_interval = 0.0001
+_poll_rampup_limit = 0.099
+# reader polling: number of timeouts with no data before increasing the
+# polling interval
+_poll_rampup = 10
+# polling timeout when waiting to close threads.  This will bail on
+# closing threast after a minimum of 13.1 seconds and a worst case of
+# ~(13.1 * #threads) seconds
+_poll_timeout = 3  # 15 rounds: 0.0001 * 2**15 == 3.2768
+_poll_timeout_deadlock = 200  # 21 rounds: 0.0001 * 2**21 == 209.7152
+
+_mswindows = sys.platform.startswith('win')
 try:
     if _mswindows:
         from msvcrt import get_osfhandle
@@ -129,10 +139,10 @@ def __exit__(self, t, v, traceback):
         # or else the new file that points to the original (duplicated)
         # file descriptor
         if self.target_file is not None:
-            setattr(sys, self.std, self.original_file)
             self.target_file.flush()
             self.target_file.close()
             self.target_file = None
+            setattr(sys, self.std, self.original_file)
         # Restore stdout's FD (implicitly closing the FD we opened)
         os.dup2(self.original_fd, self.fd, inheritable=bool(self.std))
         # Close the temporary FD
@@ -176,8 +186,8 @@ def __enter__(self):
 
     def __exit__(self, et, ev, tb):
         if self.fd_redirect is not None:
-            self.fd_redirect[0].__exit__(et, ev, tb)
             self.fd_redirect[1].__exit__(et, ev, tb)
+            self.fd_redirect[0].__exit__(et, ev, tb)
             self.fd_redirect = None
         FAIL = self.tee.STDOUT is not sys.stdout
         self.tee.__exit__(et, ev, tb)
@@ -248,7 +258,6 @@ def close(self):
             self.write_pipe = None
 
     def finalize(self, ostreams):
-        self.close()
         self.decodeIncomingBuffer()
         if ostreams:
             # Turn off buffering for the final write
@@ -327,6 +336,7 @@ def __init__(self, *ostreams, encoding=None):
         self._stdout = None
         self._stderr = None
         self._handles = []
+        self._active_handles = []
         self._threads = []
 
     @property
@@ -345,12 +355,22 @@ def open(self, mode='w', buffering=-1, encoding=None, newline=None):
         if encoding is None:
             encoding = self.encoding
         handle = _StreamHandle(mode, buffering, encoding, newline)
+        # Note that is it VERY important to close file handles in the
+        # same thread that opens it.  If you don't you can see deadlocks
+        # and a peculiar error ("libgcc_s.so.1 must be installed for
+        # pthread_cancel to work"; see https://bugs.python.org/issue18748)
+        #
+        # To accomplish this, we will keep two handle lists: one is the
+        # set of "active" handles that the (merged reader) thread is
+        # using, and the other the list of all handles so the original
+        # thread can close them after the reader thread terminates.
         if handle.buffering:
-            self._handles.append(handle)
+            self._active_handles.append(handle)
         else:
             # Unbuffered handles should appear earlier in the list so
             # that they get processed first
-            self._handles.insert(0, handle)
+            self._active_handles.insert(0, handle)
+        self._handles.append(handle)
         self._start(handle)
         return handle.write_file
 
@@ -370,16 +390,24 @@ def close(self, in_exception=False):
             if not self._threads:
                 break
             _poll *= 2
-            if _poll >= 2:  # bail after a total of 2 seconds * #threads
+            if _poll_timeout <= _poll < 2*_poll_timeout:
                 if in_exception:
                     # We are already processing an exception: no reason
-                    # to trigger another
+                    # to trigger another, nor to deadlock for an extended time
                     break
+                logger.warning(
+                    "Significant delay observed waiting to join reader "
+                    "threads, possible output stream deadlock")
+            elif _poll >= _poll_timeout_deadlock:
                 raise RuntimeError(
                     "TeeStream: deadlock observed joining reader threads")
 
+        for h in list(self._handles):
+            h.finalize(self.ostreams)
+
         self._threads.clear()
         self._handles.clear()
+        self._active_handles.clear()
         self._stdout = None
         self._stderr = None
 
@@ -391,7 +419,9 @@ def __exit__(self, et, ev, tb):
 
     def __del__(self):
         # Implement __del__ to guarantee that file descriptors are closed
-        self.close()
+        # ... but only if we are not called by the GC in the handler thread
+        if threading.current_thread() not in self._threads:
+            self.close()
 
     def _start(self, handle):
         if not _peek_available:
@@ -413,7 +443,6 @@ def _streamReader(self, handle):
         while True:
             new_data = os.read(handle.read_pipe, io.DEFAULT_BUFFER_SIZE)
             if not new_data:
-                handle.finalize(self.ostreams)
                 break
             handle.decoder_buffer += new_data
 
@@ -427,18 +456,25 @@ def _streamReader(self, handle):
 
     def _mergedReader(self):
         noop = []
-        handles = self._handles
+        handles = self._active_handles
         _poll = _poll_interval
-        _fast_poll_ct = 10
+        _fast_poll_ct = _poll_rampup
         new_data = '' # something not None
         while handles:
             if new_data is None:
+                # For performance reasons, we use very aggressive
+                # polling at the beginning (_poll_interval) and then
+                # ramp up to a much more modest polling interval
+                # (_poll_rampup_limit) as the process runs and the
+                # frequency of new data appearing on the pipe slows
                 if _fast_poll_ct:
                     _fast_poll_ct -= 1
                     if not _fast_poll_ct:
                         _poll *= 10
-                        if _poll < 0.095:
-                            _fast_poll_ct = 10
+                        if _poll < _poll_rampup_limit:
+                            # reset the counter (to potentially increase
+                            # the polling interval again)
+                            _fast_poll_ct = _poll_rampup
             else:
                 new_data = None
             if _mswindows:
@@ -451,7 +487,6 @@ def _mergedReader(self):
                             handle.decoder_buffer += new_data
                             break
                     except:
-                        handle.finalize(self.ostreams)
                         handles.remove(handle)
                         new_data = None
                 if new_data is None:
@@ -476,7 +511,6 @@ def _mergedReader(self):
                 handle = ready_handles[0]
                 new_data = os.read(handle.read_pipe, io.DEFAULT_BUFFER_SIZE)
                 if not new_data:
-                    handle.finalize(self.ostreams)
                     handles.remove(handle)
                     continue
                 handle.decoder_buffer += new_data
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -133,6 +133,7 @@ def run(self):
 
     def _print_deps(self, deplist):
         implementation_name = sys.implementation.name
+        platform_system = platform.system()
         python_version = '.'.join(platform.python_version_tuple()[:2])
         for entry in deplist:
             dep, _, condition = (_.strip() for _ in entry.partition(';'))
@@ -209,6 +210,11 @@ def _print_deps(self, deplist):
             'sympy',     # differentiation
             'xlrd',      # dataportals
             'z3-solver', # community_detection
+            #
+            # subprocess output is merged more reliably if
+            # 'PeekNamedPipe' is available from pywin32
+            'pywin32; platform_system=="Windows"',
+            #
             # The following optional dependencies are difficult to
             # install on PyPy (binary wheels are not available), so we
             # will only "require" them on other (CPython) platforms:


Test patch:
diff --git a/pyomo/common/tests/test_tee.py b/pyomo/common/tests/test_tee.py
--- a/pyomo/common/tests/test_tee.py
+++ b/pyomo/common/tests/test_tee.py
@@ -53,7 +53,7 @@ def test_merge_out_and_err(self):
             # flush() and short pause should help
             t.STDOUT.write("Hello\nWorld")
             t.STDOUT.flush()
-            time.sleep(tee._poll_interval*10)
+            time.sleep(tee._poll_interval*100)
             t.STDERR.write("interrupting\ncow")
             t.STDERR.flush()
             # For determinism, it is important that the STDERR message
@@ -168,6 +168,29 @@ def test_capture_output_stack_error(self):
         finally:
             sys.stdout, sys.stderr = old
 
+    def test_deadlock(self):
+        class MockStream(object):
+            def write(self, data):
+                time.sleep(0.2)
+
+        _save = tee._poll_timeout, tee._poll_timeout_deadlock
+        tee._poll_timeout = tee._poll_interval * 2**5  # 0.0032
+        tee._poll_timeout_deadlock = tee._poll_interval * 2**7 # 0.0128
+
+        try:
+            with LoggingIntercept() as LOG, self.assertRaisesRegex(
+                    RuntimeError, 'deadlock'):
+                with tee.TeeStream(MockStream()) as t:
+                    err = t.STDERR
+                    err.write('*')
+            self.assertEqual(
+                'Significant delay observed waiting to join reader '
+                'threads, possible output stream deadlock\n',
+                LOG.getvalue()
+            )
+        finally:
+            tee._poll_timeout, tee._poll_timeout_deadlock = _save
+
 class TestFileDescriptor(unittest.TestCase):
     def setUp(self):
         self.out = sys.stdout
@@ -176,6 +199,7 @@ def setUp(self):
     def tearDown(self):
         sys.stdout = self.out
         os.dup2(self.out_fd, 1)
+        os.close(self.out_fd)
 
     def _generate_output(self, redirector):
         with redirector:
@@ -289,6 +313,9 @@ def test_caputure_output_fd(self):
 
         self.assertEqual(OUT.getvalue(), "to_stdout_1\nto_fd1_1\n")
         with os.fdopen(r, 'r') as FILE:
-            os.close(w)
             os.close(1)
+            os.close(w)
             self.assertEqual(FILE.read(), "to_stdout_2\nto_fd1_2\n")
+
+if __name__ == '__main__':
+    unittest.main()
diff --git a/pyomo/environ/tests/test_environ.py b/pyomo/environ/tests/test_environ.py
--- a/pyomo/environ/tests/test_environ.py
+++ b/pyomo/environ/tests/test_environ.py
@@ -137,6 +137,8 @@ def test_tpl_import_time(self):
             'tempfile',
             'textwrap',
             'typing',
+            'win32file',
+            'win32pipe',
         }
         # Non-standard-library TPLs that Pyomo will load unconditionally
         ref.add('ply')
diff --git a/pyomo/opt/tests/solver/test_shellcmd.py b/pyomo/opt/tests/solver/test_shellcmd.py
--- a/pyomo/opt/tests/solver/test_shellcmd.py
+++ b/pyomo/opt/tests/solver/test_shellcmd.py
@@ -15,15 +15,24 @@
 
 import pyomo.common.unittest as unittest
 from pyomo.common.errors import ApplicationError
+from pyomo.common.fileutils import this_file_dir
 
 from pyomo.opt.base import UnknownSolver
 from pyomo.opt.base.solvers import SolverFactory
 from pyomo.opt.solver import SystemCallSolver
 
-thisdir = os.path.dirname(os.path.abspath(__file__))
+thisdir = os.path.normpath(this_file_dir())
 exedirname = "exe_dir"
 exedir = os.path.join(thisdir, exedirname)
-exedir_user = exedir.replace(os.path.expanduser("~"), "~", 1)
+user_home = os.path.normpath(os.path.expanduser("~"))
+user_home_pos = exedir.find(user_home)
+exedir_user = exedir
+if user_home_pos >= 0:
+    _test_user_exedir = os.path.join(
+        user_home, exedir[user_home_pos + len(user_home) + 1:])
+    if os.path.samefile(exedir, _test_user_exedir):
+        exedir_user = os.path.join(
+            "~", exedir[user_home_pos + len(user_home) + 1:])
 
 notexe_nopath  = "file_not_executable"
 notexe_abspath = os.path.join(exedir, notexe_nopath)
@@ -225,8 +234,10 @@ def test_set_executable_isexe_abspath_user(self):
         with SystemCallSolver(type='test') as opt:
             self.assertEqual(id(opt._user_executable), id(None))
             opt.set_executable(isexe_abspath_user)
-            self.assertEqual(opt._user_executable, isexe_abspath)
-            self.assertEqual(opt.executable(), isexe_abspath)
+            self.assertTrue(os.path.samefile(
+                opt._user_executable, isexe_abspath))
+            self.assertTrue(os.path.samefile(
+                opt.executable(), isexe_abspath))
             opt._user_executable = None
             opt.set_executable(isexe_abspath_user, validate=False)
             self.assertEqual(opt._user_executable, isexe_abspath_user)
@@ -281,8 +292,13 @@ def test_executable_isexe_abspath_user(self):
             with SolverFactory(name, executable=isexe_abspath_user) as opt:
                 if isinstance(opt, UnknownSolver):
                     continue
-                self.assertEqual(opt._user_executable, isexe_abspath)
-                self.assertEqual(opt.executable(), isexe_abspath)
+                # Note the user path could be different from our
+                # computed path due to symlinks.  Check that the
+                # executable is the same file.
+                self.assertTrue(os.path.samefile(
+                    opt._user_executable, isexe_abspath))
+                self.assertTrue(os.path.samefile(
+                    opt.executable(), isexe_abspath))
 
 if __name__ == "__main__":
     unittest.main()