Repository: Pyomo/pyomo

Issue:
Capture Highs output when solving
## Summary

Highs logs are not entirely captured. As discussed [on the Highs bug tracker](https://github.com/ERGO-Code/HiGHS/issues/1670), it can result in error messages like the following:
```ERROR: Output stream (<_io.TextIOWrapper name='' mode='w' encoding='utf-8'>) closed before all output was written to it. The following was left in the output buffer: 'ERROR: Output stream (<_io.TextIOWrapper name='' mode='w'\nencoding='utf-8'>)```. The expected behaviour is that no error message and no log line is written.

Logs in Highs may be written by any function, including `addRows`. I think the reason for this issue is that they are not covered by `TeeStream`. This is related to #3003.

### Steps to reproduce the issue

Run the hangs.py file from [this repository](https://github.com/SolverMax/Random/tree/main/HiGHS-testing/Hangs) on Windows

### Error Message

Multiple lines like
```ERROR: Output stream (<_io.TextIOWrapper name='' mode='w' encoding='utf-8'>) closed before all output was written to it. The following was left in the output buffer: 'ERROR: Output stream (<_io.TextIOWrapper name='' mode='w'\nencoding='utf-8'>) ```

### Information on your system

Pyomo version: 6.7
Python version: 3.11
Operating system: Windows
How Pyomo was installed (PyPI, conda, source): pip
Solver (if applicable): Highspy


### Additional information

<!-- If you have any additional information, please list it here. -->


<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question (how to use Pyomo, what it can do, etc.), please refer to our [online documentation](https://github.com/Pyomo/pyomo#getting-help) on ways to get help. -->




Discussion:
I have also encountered this issue after setting `logging.getLogger("pyomo").set_level(logging.INFO)`, in addition to seeing the log messages with varying levels of nesting, as in:

```
Running HiGHS 1.11.0 (git hash: 364c83a): Copyright (c) 2025 HiGHS under MIT licence terms
RUN!
MIP  has 372 rows; 348 cols; 1207 nonzeros; 28 integer variables (28 binary)
INFO: RUN!
Coefficient ranges:
  Matrix [3e-05, 1e+03]
  Cost   [1e+00, 1e+00]
  Bound  [1e+00, 1e+03]
  RHS    [4e-01, 1e+01]
INFO: MIP  has 372 rows; 348 cols; 1207 nonzeros; 28 integer variables (28
binary)
INFO: INFO: RUN!
Presolving model
INFO: Coefficient ranges:
INFO: Matrix [3e-05, 1e+03]
INFO: Cost   [1e+00, 1e+00]
243 rows, 224 cols, 896 nonzeros  0s
INFO: Bound  [1e+00, 1e+03]
175 rows, 168 cols, 748 nonzeros  0s
INFO: RHS    [4e-01, 1e+01]
INFO: INFO: MIP  has 372 rows; 348 cols; 1207 nonzeros; 28 integer variables
(28
INFO: binary)
INFO: INFO: INFO: RUN!
INFO: Presolving model
INFO: INFO: Coefficient ranges:
INFO: INFO: Matrix [3e-05, 1e+03]
INFO: INFO: Cost   [1e+00, 1e+00]
INFO: 243 rows, 224 cols, 896 nonzeros  0s
INFO: INFO: Bound  [1e+00, 1e+03]
INFO: 175 rows, 168 cols, 748 nonzeros  0s
124 rows, 119 cols, 535 nonzeros  0s
INFO: INFO: RHS    [4e-01, 1e+01]
INFO: INFO: INFO: MIP  has 372 rows; 348 cols; 1207 nonzeros; 28 integer
variables
INFO: (28
INFO: INFO: binary)
INFO: INFO: INFO: INFO: RUN!
```

@rschwarz, it looks like you are running the Pyomo development branch (`Pyomo/main`), but before at least #3613.  Can you update your Pyomo installation and re-try?  I have been trying (unsuccessfully) to reproduce the issue reported here using the current codebase and was about to proposing to close this issue as resolved.
I was using:
```
highspy=1.11.0
pyomo==6.9.2
python==3.12.3
```

But I can try with a development version of Pyomo, and try to reproduce on a smaller example (but not today).
So I've `git clone`d the `pyomo` repo (`main` at a8d0bfaad87b7514e45653593d6e36babfb72638) and tried my setup on a smaller example.

If I set `logging.getLogger('pyomo').setLevel(logging.INFO)` I get an error on every run. It doesn't matter whether I use `tee=True` or `tee=False` in the call to `solve`. But, the behaviour seems non-deterministic, in the sense that I get varying amount of output on reruns.

I've added the script and two example outputs to this Gist: https://gist.github.com/rschwarz/5b6b5a79c2077c16ba0cc7f31f61821b

----

```
Using Python 3.12.3 environment at /home/robert/src/pyomo/env
Package Version
------- ----------
highspy 1.11.0
numpy   2.3.0
ply     3.11
pyomo   6.9.3.dev0
```




Patch:
diff --git a/doc/OnlineDocs/_templates/recursive-base.rst b/doc/OnlineDocs/_templates/recursive-base.rst
index 6a4e827fbe2..df588d45fcc 100644
--- a/doc/OnlineDocs/_templates/recursive-base.rst
+++ b/doc/OnlineDocs/_templates/recursive-base.rst
@@ -7,6 +7,11 @@
    # import everything from the module containing this class so that
    # doctests for the class docstrings see the correct environment
    from {{ module }} import *
+   try:
+       from {{ module }} import _autosummary_doctest_setup
+       _autosummary_doctest_setup()
+   except ImportError:
+       pass
 
 .. currentmodule:: {{ module }}
 
diff --git a/doc/OnlineDocs/_templates/recursive-class.rst b/doc/OnlineDocs/_templates/recursive-class.rst
index 0691af291aa..7265c1b42ba 100644
--- a/doc/OnlineDocs/_templates/recursive-class.rst
+++ b/doc/OnlineDocs/_templates/recursive-class.rst
@@ -7,6 +7,11 @@
    # import everything from the module containing this class so that
    # doctests for the class docstrings see the correct environment
    from {{ module }} import *
+   try:
+       from {{ module }} import _autosummary_doctest_setup
+       _autosummary_doctest_setup()
+   except ImportError:
+       pass
 
 .. currentmodule:: {{ module }}
 
diff --git a/doc/OnlineDocs/_templates/recursive-enum.rst b/doc/OnlineDocs/_templates/recursive-enum.rst
index 966cb178b78..77c25a678c9 100644
--- a/doc/OnlineDocs/_templates/recursive-enum.rst
+++ b/doc/OnlineDocs/_templates/recursive-enum.rst
@@ -7,6 +7,11 @@
    # import everything from the module containing this class so that
    # doctests for the class docstrings see the correct environment
    from {{ module }} import *
+   try:
+       from {{ module }} import _autosummary_doctest_setup
+       _autosummary_doctest_setup()
+   except ImportError:
+       pass
 
 .. currentmodule:: {{ module }}
 
diff --git a/doc/OnlineDocs/_templates/recursive-module.rst b/doc/OnlineDocs/_templates/recursive-module.rst
index dc69e33731d..90ba5bd2c85 100644
--- a/doc/OnlineDocs/_templates/recursive-module.rst
+++ b/doc/OnlineDocs/_templates/recursive-module.rst
@@ -5,6 +5,19 @@ Library Reference
 {{ name | escape | underline}}
 {% endif %}
 
+.. testsetup:: *
+
+   # import everything from the module containing this class so that
+   # doctests for the class docstrings see the correct environment
+   from {{ module }} import *
+   try:
+       from {{ module }} import _autosummary_doctest_setup
+       _autosummary_doctest_setup()
+   except ImportError:
+       pass
+
+.. currentmodule:: {{ module }}
+
 .. automodule:: {{ fullname }}
    :undoc-members:
 
diff --git a/doc/OnlineDocs/errors.rst b/doc/OnlineDocs/errors.rst
index 162c2e10257..5b27abe6fbf 100644
--- a/doc/OnlineDocs/errors.rst
+++ b/doc/OnlineDocs/errors.rst
@@ -8,10 +8,13 @@ Common Warnings/Errors
    for backwards compatibility, DO NOT recycle old ID (no longer used)
    numbers.
 
-.. doctest::
-   :hide:
+.. testsetup::
 
-   >>> import pyomo.environ as pyo
+   import pyomo.environ as pyo
+   # Ensure that all logged messages are sent to stdout
+   # (so they show up in the doctest output and can be tested)
+   import pyomo.common.log as _log
+   _log.pyomo_handler.__class__ = _log.StdoutHandler
 
 .. py:currentmodule:: pyomo.environ
 
diff --git a/doc/OnlineDocs/explanation/developer_utils/deprecation.rst b/doc/OnlineDocs/explanation/developer_utils/deprecation.rst
index 7fc5ec2b0ff..5f450561669 100644
--- a/doc/OnlineDocs/explanation/developer_utils/deprecation.rst
+++ b/doc/OnlineDocs/explanation/developer_utils/deprecation.rst
@@ -9,14 +9,18 @@ Deprecation
 -----------
 
 We offer a set of tools to help with deprecation in
-``pyomo.common.deprecation``.
+:py:mod:`pyomo.common.deprecation`.
 
 By policy, when deprecating or moving an existing capability, one of the
 following utilities should be leveraged.  Each has a required
 ``version`` argument that should be set to current development version (e.g.,
 ``"6.6.2.dev0"``).  This version will be updated to the next actual
 release as part of the Pyomo release process.  The current development version
-can be found by running ``pyomo --version`` on your local fork/branch.
+can be found by running
+
+   ``pyomo --version``
+
+on your local fork/branch.
 
 .. currentmodule:: pyomo.common.deprecation
 
@@ -24,25 +28,10 @@ can be found by running ``pyomo --version`` on your local fork/branch.
 
    deprecated
    deprecation_warning
-   relocated_module
+   moved_module
    relocated_module_attribute
    RenamedClass
 
-.. autodecorator:: pyomo.common.deprecation.deprecated
-   :noindex:
-
-.. autofunction:: pyomo.common.deprecation.deprecation_warning
-   :noindex:
-
-.. autofunction:: pyomo.common.deprecation.relocated_module
-   :noindex:
-
-.. autofunction:: pyomo.common.deprecation.relocated_module_attribute
-   :noindex:
-
-.. autoclass:: pyomo.common.deprecation.RenamedClass
-   :noindex:
-
 
 Removal
 -------
@@ -52,10 +41,8 @@ warning, pending extenuating circumstances. The functionality should
 be deprecated, following the information above.
 
 If the functionality is documented in the most recent
-edition of [`Pyomo - Optimization Modeling in Python`_], it may not be removed
-until the next major version release.
-
-.. _Pyomo - Optimization Modeling in Python: https://doi.org/10.1007/978-3-030-68928-5
+edition of :ref:`Pyomo - Optimization Modeling in Python <pyomobookiii>`,
+it may not be removed until the next major version release.
 
 For other functionality, it is preferred that ample time is given
 before removing the functionality. At minimum, significant functionality
diff --git a/doc/OnlineDocs/explanation/solvers/pynumero/tutorial.linear_solver_interfaces.rst b/doc/OnlineDocs/explanation/solvers/pynumero/tutorial.linear_solver_interfaces.rst
index 02ffe761778..9fff497b6a3 100644
--- a/doc/OnlineDocs/explanation/solvers/pynumero/tutorial.linear_solver_interfaces.rst
+++ b/doc/OnlineDocs/explanation/solvers/pynumero/tutorial.linear_solver_interfaces.rst
@@ -42,7 +42,7 @@ Interface to MA27
    >>> status = solver.do_numeric_factorization(A)
    >>> x, status = solver.do_back_solve(rhs)
    >>> np.max(np.abs(A*x - rhs)) <= 1e-15
-   True
+   np.True_
 
 
 Interface to MUMPS
diff --git a/doc/OnlineDocs/reference/bibliography.rst b/doc/OnlineDocs/reference/bibliography.rst
index 4a4ea37f3e5..4d91f09701a 100644
--- a/doc/OnlineDocs/reference/bibliography.rst
+++ b/doc/OnlineDocs/reference/bibliography.rst
@@ -29,6 +29,10 @@ These publications describe various Pyomo capabilitites or subpackages:
    Vol. 67. Springer. 2021.  DOI `10.1007/978-3-030-68928-5
    <https://doi.org/10.1007/978-3-030-68928-5>`_
 
+..
+   NOTE: When adding a new edition of the Pyomo Book, search the codebase
+   both for citation references "[PyomoBookIII]" and references "pyomobookiii"
+
 .. [PyomoDAE-paper] Bethany Nicholson, John D. Siirola, Jean-Paul Watson,
    Victor M. Zavala, and Lorenz T. Biegler. "pyomo.dae: a modeling and
    automatic discretization framework for optimization with differential
@@ -126,7 +130,7 @@ Bibliography
 
 .. [RB01] W. C. Rooney and L. T. Biegler. "Design for model parameter
    uncertainty using nonlinear confidence regions", *AIChE Journal*,
-   47(8). 2001.
+   47(8). 2001.  DOI `10.1002/aic.690470811 <https://doi.org/10.1002/aic.690470811>`_
 
 .. [RG94] R. Raman and I. E. Grossmann. "Modelling and computational
    techniques for logic based integer programming", *Computers and
diff --git a/pyomo/common/dependencies.py b/pyomo/common/dependencies.py
index 61fbb645e3c..a2cf06fbb85 100644
--- a/pyomo/common/dependencies.py
+++ b/pyomo/common/dependencies.py
@@ -856,7 +856,6 @@ def declare_deferred_modules_as_importable(globals_dict):
        ...     'scipy', callback=_finalize_scipy,
        ...     deferred_submodules=['stats', 'sparse', 'spatial', 'integrate'])
        >>> declare_deferred_modules_as_importable(globals())
-       WARNING: DEPRECATED: ...
 
     Which enables users to use:
 
diff --git a/pyomo/common/deprecation.py b/pyomo/common/deprecation.py
index 1246de84b8c..4f34043d713 100644
--- a/pyomo/common/deprecation.py
+++ b/pyomo/common/deprecation.py
@@ -36,6 +36,18 @@
 _doc_flag = '.. deprecated::'
 
 
+def _autosummary_doctest_setup():
+    """This function gets called to setup the doctest environment before
+    running any autosummary doctests in this module.
+    """
+
+    # Ensure that all logged messages are sent to stdout
+    # (so they show up in the doctest output and can be tested)
+    import pyomo.common.log as _log
+
+    _log.pyomo_handler.__class__ = _log.StdoutHandler
+
+
 def default_deprecation_msg(obj, user_msg, version, remove_in):
     """Generate the default deprecation message.
 
@@ -157,7 +169,7 @@ def _find_calling_frame(module_offset):
 def deprecation_warning(
     msg, logger=None, version=None, remove_in=None, calling_frame=None
 ):
-    """Standardized formatter for deprecation warnings
+    """Standardized function for formatting and emitting deprecation warnings.
 
     This is a standardized routine for formatting deprecation warnings
     so that things look consistent and "nice".
diff --git a/pyomo/common/log.py b/pyomo/common/log.py
index b8275290839..2a48afe13d8 100644
--- a/pyomo/common/log.py
+++ b/pyomo/common/log.py
@@ -208,13 +208,25 @@ def format(self, record):
 class StdoutHandler(logging.StreamHandler):
     """A logging handler that emits to the current value of sys.stdout"""
 
+    def __init__(self):
+        super().__init__()
+        self.stream = None
+
     def flush(self):
-        self.stream = sys.stdout
-        super(StdoutHandler, self).flush()
+        try:
+            orig = self.stream
+            self.stream = sys.stdout
+            super(StdoutHandler, self).flush()
+        finally:
+            self.stream = orig
 
     def emit(self, record):
-        self.stream = sys.stdout
-        super(StdoutHandler, self).emit(record)
+        try:
+            orig = self.stream
+            self.stream = sys.stdout
+            super(StdoutHandler, self).emit(record)
+        finally:
+            self.stream = orig
 
 
 class Preformatted(object):
@@ -248,7 +260,7 @@ def filter(self, record):
 # debugging.  It has been updated to suppress output if any handlers
 # have been defined at the root level.
 pyomo_logger = logging.getLogger('pyomo')
-pyomo_handler = StdoutHandler()
+pyomo_handler = logging.StreamHandler(sys.stdout)
 pyomo_formatter = LegacyPyomoFormatter(
     base=PYOMO_ROOT_DIR, verbosity=lambda: pyomo_logger.isEnabledFor(logging.DEBUG)
 )
@@ -445,23 +457,25 @@ class _StreamRedirector(object):
     def __init__(self, handler, fd):
         self.handler = handler
         self.fd = fd
+        self.local_fd = None
         self.orig_stream = None
 
     def __enter__(self):
+        assert self.local_fd is None
         self.orig_stream = self.handler.stream
         # Note: ideally, we would use closefd=True and let Python handle
         # closing the local file descriptor that we are about to create.
         # However, it appears that closefd is ignored on Windows (see
         # #3587), so we will just handle it explicitly ourselves.
+        self.local_fd = os.dup(self.fd)
         self.handler.stream = os.fdopen(
-            os.dup(self.fd), mode="w", closefd=False
+            self.local_fd, mode="a", closefd=False
         ).__enter__()
 
     def __exit__(self, et, ev, tb):
         try:
-            fd = self.handler.stream.fileno()
             self.handler.stream.__exit__(et, ev, tb)
-            os.close(fd)
+            os.close(self.local_fd)
         finally:
             self.handler.stream = self.orig_stream
 
@@ -469,22 +483,24 @@ def __exit__(self, et, ev, tb):
 class _LastResortRedirector(object):
     def __init__(self, fd):
         self.fd = fd
+        self.local_fd = None
         self.orig_stream = None
 
     def __enter__(self):
+        assert self.local_fd is None
         self.orig = logging.lastResort
         # Note: ideally, we would use closefd=True and let Python handle
         # closing the local file descriptor that we are about to create.
         # However, it appears that closefd is ignored on Windows (see
         # #3587), so we will just handle it explicitly ourselves.
+        self.local_fd = os.dup(self.fd)
         logging.lastResort = logging.StreamHandler(
-            os.fdopen(os.dup(self.fd), mode="w", closefd=False).__enter__()
+            os.fdopen(self.local_fd, mode="a", closefd=False).__enter__()
         )
 
     def __exit__(self, et, ev, tb):
         try:
-            fd = logging.lastResort.stream.fileno()
             logging.lastResort.stream.close()
-            os.close(fd)
+            os.close(self.local_fd)
         finally:
             logging.lastResort = self.orig
diff --git a/pyomo/common/tee.py b/pyomo/common/tee.py
index 276c1bf95e6..70070bd894d 100644
--- a/pyomo/common/tee.py
+++ b/pyomo/common/tee.py
@@ -169,7 +169,7 @@ def __enter__(self):
 
         if self.synchronize and self.std:
             # Cause Python's stdout to point to our new file
-            self.target_file = os.fdopen(self.fd, 'w', closefd=False)
+            self.target_file = os.fdopen(self.fd, 'a', closefd=False)
             setattr(sys, self.std, self.target_file)
 
         return self
@@ -349,7 +349,7 @@ def _enter_impl(self):
                 log_stream = self._enter_context(
                     os.fdopen(
                         self._enter_context(_fd_closer(os.dup(old_fd[1] or 2))),
-                        mode="w",
+                        mode="a",
                         closefd=False,
                     )
                 )
@@ -358,7 +358,7 @@ def _enter_impl(self):
             self._enter_context(LoggingIntercept(log_stream, logger=logger, level=None))
 
             if isinstance(self.output, str):
-                self.output_stream = self._enter_context(open(self.output, 'w'))
+                self.output_stream = self._enter_context(open(self.output, 'a'))
             elif self.output is None:
                 self.output_stream = io.StringIO()
             else:
@@ -415,7 +415,7 @@ def _enter_impl(self):
                                     _fd_closer(os.dup(fd_redirect[fd].original_fd)),
                                     prior_to=self.tee,
                                 ),
-                                mode="w",
+                                mode="a",
                                 closefd=False,
                             ),
                             prior_to=self.tee,
@@ -673,7 +673,7 @@ def STDERR(self):
             self._stderr = self.open(buffering=b)
         return self._stderr
 
-    def open(self, mode='w', buffering=-1, encoding=None, newline=None):
+    def open(self, mode='a', buffering=-1, encoding=None, newline=None):
         if encoding is None:
             encoding = self.encoding
         handle = _StreamHandle(mode, buffering, encoding, newline)


Test patch:
diff --git a/pyomo/common/tests/test_log.py b/pyomo/common/tests/test_log.py
index 43e85af1308..650bfd60306 100644
--- a/pyomo/common/tests/test_log.py
+++ b/pyomo/common/tests/test_log.py
@@ -19,6 +19,7 @@
 
 import logging
 import os
+import sys
 from inspect import currentframe, getframeinfo
 from io import StringIO
 
@@ -30,6 +31,7 @@
     LogHandler,
     LogStream,
     Preformatted,
+    StdoutHandler,
     WrappingFormatter,
     pyomo_formatter,
 )
@@ -619,3 +621,52 @@ def test_propagate(self):
             self.assertEqual(logger.level, 40)
         finally:
             logger.setLevel(30)
+
+
+class TestStdoutHandler(unittest.TestCase):
+    def setUp(self):
+        self.orig = sys.stdout
+
+    def tearDown(self):
+        sys.stdout = self.orig
+
+    def test_emit(self):
+        handler = StdoutHandler()
+        self.assertIsNone(handler.stream)
+
+        sys.stdout = StringIO()
+        record = logging.LogRecord(
+            name="test",
+            level=logging.WARNING,
+            pathname="/path/to/file.py",
+            lineno=42,
+            msg="Test msg",
+            args=(),
+            exc_info=None,
+        )
+        handler.emit(record)
+        handler.flush()
+
+        self.assertEqual(sys.stdout.getvalue(), "Test msg\n")
+        self.assertIsNone(handler.stream)
+
+    def test_handler(self):
+        logger = logging.getLogger(__name__)
+        propagate, level = logger.propagate, logger.level
+        handler = StdoutHandler()
+        try:
+            logger.addHandler(handler)
+            logger.propagate = False
+            sys.stdout = StringIO()
+
+            logger.setLevel(logging.WARNING)
+            logger.info("Test1")
+            self.assertEqual(sys.stdout.getvalue(), "")
+
+            logger.setLevel(logging.INFO)
+            logger.info("Test2")
+            self.assertEqual(sys.stdout.getvalue(), "Test2\n")
+        finally:
+            logger.removeHandler(handler)
+            logger.propagate = propagate
+            logger.setLevel(level)