Repository: Pyomo/pyomo

Issue:
ERROR: Output stream closed before all output was written to it.
## Summary

<!-- Explain, in a clear and concise way, the command you ran and the result you were trying to achieve.
Example: "I ran `pyomo solve` to optimize a model and ..." -->

### Steps to reproduce the issue

<!-- Please include the command run and/or provide an example script that demonstrates the problem. -->

```console
$ command1 [options]
$ command2 [options]
...
```

```
# example.py
import pyomo.environ
...
```

### Error Message
ERROR: Output stream closed before all output was written to it. The following
    was left in the output buffer:
Actually I really don't know whether this comes from pyomo or gurobi.
<!-- If Pyomo reported an error, provide the error message. PLEASE INCLUDE THE ENTIRE STACK TRACE. -->

<!-- If it did not report an error but the output appears incorrect, provide the incorrect output. If there was no error message and no output but the result is incorrect, describe how it does not match what you expect. -->
```console
$ # Output message here, including entire stack trace, if available
```

### Information on your system

Pyomo version: 6.0
Python version: 3.7.9
Operating system: Ubuntu 20.04.2 LTS (GNU/Linux 5.8.0-50-generic x86_64)
How Pyomo was installed (PyPI, conda, source): PyPI
Solver (if applicable): Gurobi


### Additional information

<!-- If you have any additional information, please list it here. -->
+ This situation happened today, and nothing wrong before, sorry I really don't know how to reproduce it
+ I updated all my packages using pipdate yesterday
+ Nothing changed after I reinstall gurobi
<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question (how to use Pyomo, what it can do, etc.), please refer to our [online documentation](https://github.com/Pyomo/pyomo#getting-help) on ways to get help. -->

![image](https://user-images.githubusercontent.com/55777837/120187307-34d87680-c247-11eb-97da-31edb914ebcf.png)

Guys, I encountered this problem today when I'm solving a milp problem with pyomo. I specificied my solver as gurobi and linked it with excutable path. But once gurobi logging starts to output this error will occur once and once again blinding me from real log I want to check.

I tried to reinstall gurobi on server but nothing changed. To behonest, I'm not even sure of which part should be responsible for this error: pyomo or gurobi. And I searched both websites with keywords like title states, nothing comes in visual. Although my model is still being solving by gurobi, the logging of ERROR is really annoying and noisy.

Does anyone have ever met this? Any solution?


Discussion:
Just upgraded to pyomo 6.0 and using Ipopt as a solver, seeing this message as well.
I am getting a full log from Pyomo, even though its reporting I'm not. Are you seeing the same @Betristor? If that's the case, you can use `logging.addFilter` to suppress this particular message, the handler is "pyomo.common.tee".
The error (or at least the fix) is going to be in Pyomo.  Pyomo 6.0 removed the dependency on pyutilib, and reimplemented certain core functionality natively in Pyomo.  One part of that is how we handle "`tee`ing" solver output.  The error you are seeing is because output coming from the solver hasn't been fully flushed before the `TeeStream` handler closes (it comes from `pyomo/common/tee.py:200`).

We spent a lot of time exercising this locally before releasing Pyomo and haven't seen this behavior, so I am guessing a bit here.  The first thing I would like to try is to edit `pyomo/opt/solver/shellcmd.py` with the following patch:
```diff
diff --git a/pyomo/opt/solver/shellcmd.py b/pyomo/opt/solver/shellcmd.py
index 2c0e244..5bda59e 100644
--- a/pyomo/opt/solver/shellcmd.py
+++ b/pyomo/opt/solver/shellcmd.py
@@ -320,6 +320,8 @@ class SystemCallSolver(OptSolver):
                     timeout=timeout,
                     universal_newlines=True,
                 )
+                t.STDOUT.flush()
+                t.STDERR.flush()
 
             rc = results.returncode
             log = ostreams[0].getvalue()
```

You may also consider explicitly uninstalling pyutilib (`pip uninstall pyutilib`)

To aid in further debugging, can you include more of your driver script?  How are you calling `solve()` (i.e., with output redirection, with `tee=True`, etc.)?
I tried to apply @jsiirola's patch but am still seeing the the logging error message. The error message appears before the patch is hit. Here is the call we are using:

    solver.solve(m, tee=True, symbolic_solver_labels=True)

where m is our ConcreteModel.

I will see if I can create a reproducible example I can share.
@rhshadrach, can you also apply the following patch and post the new error:
```diff
diff --git a/pyomo/common/tee.py b/pyomo/common/tee.py
index 3922361..b351999 100644
--- a/pyomo/common/tee.py
+++ b/pyomo/common/tee.py
@@ -197,9 +197,10 @@ class _StreamHandle(object):
                 stream.flush()
             if written != len(ostring):
                 logger.error(
-                    "Output stream closed before all output was "
+                    "Output stream (%s) closed before all output was "
                     "written to it. The following was left in "
-                    "the output buffer:\n\t%r" % (ostring[written:],))
+                    "the output buffer:\n\t%r" % (
+                        stream, ostring[written:],))
 
 
 class TeeStream(object):
```

(I am interested in which output stream -- the `StringIO` buffer or `sys.stdout` -- is being prematurely closed)
The only time I see the error message is when running via jupyter notebook. Running the code below in the shell (e.g. `python pyomo_mwe.py`) does not produce the error message.

<details>
<summary>Reproducible example</summary>

```
import pyomo
from pyomo.opt import SolverFactory
import pyomo.environ as pe

print(f'Pyomo version: {pyomo.__version__}')
m = pe.ConcreteModel()
m.v = pe.Var(bounds=(0, 1))
m.obj = pe.Objective(rule=lambda m: m.v)

solver = SolverFactory('ipopt')
_ = solver.solve(
    m, tee=True, symbolic_solver_labels=True
)
```

</details>

<details>
<summary>Output</summary>

```
Pyomo version: 6.0
Ipopt 3.12.13: 

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit http://projects.coin-or.org/Ipopt
******************************************************************************

This is Ipopt version 3.12.13, running with linear solver ma27.

Number of nonzeros in equality constraint Jacobian...:        0
Number of nonzeros in inequality constraint Jacobian.:        0
Number of nonzeros in Lagrangian Hessian.............:        0

Total number of variables............................:        1
                     variables with only lower bounds:        0
                variables with lower and upper bounds:        1
                     variables with only upper bounds:        0
Total number of equality constraints.................:        0
Total number of inequality constraints...............:        0
        inequality constraints with only lower bounds:        0
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:        0

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0  9.9999900e-03 0.00e+00 1.00e+00  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0
ERROR: Output stream closed before all output was written to it. The following
    was left in the output buffer:
    	'Ipopt 3.12.13: \n\n*****************************************************
    	*************************\nThis program contains Ipopt, a library for
    	large-scale nonlinear optimization.\n Ipopt is released as open source
    	code under the Eclipse Public License (EPL).\n         For more
    	information visit http://projects.coin-or.org/Ipopt\n********************
    	**********************************************************\n\nThis is
    	Ipopt version 3.12.13, running with linear solver ma27.\n\nNumber of
    	nonzeros in equality constraint Jacobian...:        0\nNumber of nonzeros
    	in inequality constraint Jacobian.:        0\nNumber of nonzeros in
    	Lagrangian Hessian.............:        0\n\nTotal number of
    	variables............................:        1\n
    	variables with only lower bounds:        0\n                variables
    	with lower and upper bounds:        1\n                     variables
    	with only upper bounds:        0\nTotal number of equality
    	constraints.................:        0\nTotal number of inequality
    	constraints...............:        0\n        inequality constraints with
    	only lower bounds:        0\n   inequality constraints with lower and
    	upper bounds:        0\n        inequality constraints with only upper
    	bounds:        0\n\niter    objective    inf_pr   inf_du lg(mu)  ||d||
    	lg(rg) alpha_du alpha_pr  ls\n   0  9.9999900e-03 0.00e+00 1.00e+00  -1.0
    	0.00e+00    -  0.00e+00 0.00e+00   0\n'
   1  1.9699990e-02 0.00e+00 1.11e-16  -1.7 9.70e-03    -  1.00e+00 1.00e+00f  1
   2  7.2798901e-04 0.00e+00 3.47e-18  -3.8 1.90e-02    -  9.86e-01 1.00e+00f  1
   3  1.8368497e-06 0.00e+00 8.79e-17  -5.7 7.26e-04    -  1.00e+00 1.00e+00f  1
   4 -7.4907020e-09 0.00e+00 3.84e-17  -8.6 1.84e-06    -  1.00e+00 1.00e+00f  1

Number of Iterations....: 4

                                   (scaled)                 (unscaled)
Objective...............:  -7.4907019811010820e-09   -7.4907019811010820e-09
Dual infeasibility......:   3.8430731849874861e-17    3.8430731849874861e-17
Constraint violation....:   0.0000000000000000e+00    0.0000000000000000e+00
Complementarity.........:   2.5092980251784595e-09    2.5092980251784595e-09
Overall NLP error.......:   2.5092980251784595e-09    2.5092980251784595e-09


Number of objective function evaluations             = 5
Number of objective gradient evaluations             = 5
Number of equality constraint evaluations            = 0
Number of inequality constraint evaluations          = 0
Number of equality constraint Jacobian evaluations   = 0
Number of inequality constraint Jacobian evaluations = 0
Number of Lagrangian Hessian evaluations             = 4
Total CPU secs in IPOPT (w/o function evaluations)   =      0.001
Total CPU secs in NLP function evaluations           =      0.000

EXIT: Optimal Solution Found.
ERROR: Output stream closed before all output was written to it. The following
    was left in the output buffer:
    	'   1  1.9699990e-02 0.00e+00 1.11e-16  -1.7 9.70e-03    -  1.00e+00
    	1.00e+00f  1\n   2  7.2798901e-04 0.00e+00 3.47e-18  -3.8 1.90e-02    -
    	9.86e-01 1.00e+00f  1\n   3  1.8368497e-06 0.00e+00 8.79e-17  -5.7
    	7.26e-04    -  1.00e+00 1.00e+00f  1\n   4 -7.4907020e-09 0.00e+00
    	3.84e-17  -8.6 1.84e-06    -  1.00e+00 1.00e+00f  1\n\nNumber of
    	Iterations....: 4\n\n                                   (scaled)
    	(unscaled)\nObjective...............:  -7.4907019811010820e-09
    	-7.4907019811010820e-09\nDual infeasibility......:
    	3.8430731849874861e-17    3.8430731849874861e-17\nConstraint
    	violation....:   0.0000000000000000e+00
    	0.0000000000000000e+00\nComplementarity.........:
    	2.5092980251784595e-09    2.5092980251784595e-09\nOverall NLP
    	error.......:   2.5092980251784595e-09
    	2.5092980251784595e-09\n\n\nNumber of objective function evaluations
    	= 5\nNumber of objective gradient evaluations             = 5\nNumber of
    	equality constraint evaluations            = 0\nNumber of inequality
    	constraint evaluations          = 0\nNumber of equality constraint
    	Jacobian evaluations   = 0\nNumber of inequality constraint Jacobian
    	evaluations = 0\nNumber of Lagrangian Hessian evaluations             =
    	4\nTotal CPU secs in IPOPT (w/o function evaluations)   =
    	0.001\nTotal CPU secs in NLP function evaluations           =
    	0.000\n\nEXIT: Optimal Solution Found.\n'
ERROR: Output stream closed before all output was written to it. The following
    was left in the output buffer:
    	'\x08\x08\x08\x08\x08\x08\x08\x08\x08\x08\x08\x08\x08\x08\x08'
```

</details>

Applying the patch from https://github.com/Pyomo/pyomo/issues/1997#issuecomment-853132718, I get:

    ERROR: Output stream (<ipykernel.iostream.OutStream object at 0x7fdd25c1f460>)
    closed

I replaced `stream` in the patch with `stream.name` and got:

    ERROR: Output stream (stdout) closed


This is making a lot more sense.  The IPython kernel is doing something very similar to what we are doing to manage redirection of stdout/stderr to buffers.  I suspect that there is something where the IPython buffer is filling up and that thread is not getting a chance to empty it.
OK.  I think I understand what is going on.  IPython's `OutStream.write()` ([source](https://github.com/ipython/ipykernel/blob/fda115626dd434f49a4d8f8e55080d5fd2c2e2b9/ipykernel/iostream.py#L479)) does not adhere to the Python `TextIOBase`  [API](https://docs.python.org/3/library/io.html#io.TextIOBase.write): the API calls for `write()` to return the number of characters written, but IPython's `OutStream.write()` just returns `None`.  

The following patch should get around this bug, but the real solution will be to get IPython to fix their implementation.
```diff
diff --git a/pyomo/common/tee.py b/pyomo/common/tee.py
index 3922361..248c3b9 100644
--- a/pyomo/common/tee.py
+++ b/pyomo/common/tee.py
@@ -195,11 +195,12 @@ class _StreamHandle(object):
                 written = 0
             if written and not self.buffering:
                 stream.flush()
-            if written != len(ostring):
+            if written is not None and written != len(ostring):
                 logger.error(
-                    "Output stream closed before all output was "
+                    "Output stream (%s) closed before all output was "
                     "written to it. The following was left in "
-                    "the output buffer:\n\t%r" % (ostring[written:],))
+                    "the output buffer:\n\t%r" % (
+                        stream, ostring[written:],))
 
 
 class TeeStream(object):
```

Resolving patch:
diff --git a/pyomo/common/tee.py b/pyomo/common/tee.py
--- a/pyomo/common/tee.py
+++ b/pyomo/common/tee.py
@@ -195,11 +195,16 @@ def writeOutputBuffer(self, ostreams):
                 written = 0
             if written and not self.buffering:
                 stream.flush()
-            if written != len(ostring):
+            # Note: some derived file-like objects fail to return the
+            # number of characters written (and implicitly return None).
+            # If we get None, we will just assume that everything was
+            # fine (as opposed to tossing the incomplete write error).
+            if written is not None and written != len(ostring):
                 logger.error(
-                    "Output stream closed before all output was "
+                    "Output stream (%s) closed before all output was "
                     "written to it. The following was left in "
-                    "the output buffer:\n\t%r" % (ostring[written:],))
+                    "the output buffer:\n\t%r" % (
+                        stream, ostring[written:],))
 
 
 class TeeStream(object):
diff --git a/pyomo/opt/solver/shellcmd.py b/pyomo/opt/solver/shellcmd.py
--- a/pyomo/opt/solver/shellcmd.py
+++ b/pyomo/opt/solver/shellcmd.py
@@ -320,6 +320,8 @@ def _execute_command(self,command):
                     timeout=timeout,
                     universal_newlines=True,
                 )
+                t.STDOUT.flush()
+                t.STDERR.flush()
 
             rc = results.returncode
             log = ostreams[0].getvalue()


Test patch:
diff --git a/pyomo/common/tests/test_tee.py b/pyomo/common/tests/test_tee.py
--- a/pyomo/common/tests/test_tee.py
+++ b/pyomo/common/tests/test_tee.py
@@ -118,10 +118,10 @@ def test_decoder_and_buffer_errors(self):
             with tee.TeeStream(out) as t:
                 out.close()
                 t.STDOUT.write("hi\n")
-        self.assertEqual(
+        self.assertRegex(
             log.getvalue(),
-            "Output stream closed before all output was written to it. "
-            "The following was left in the output buffer:\n\t'hi\\n'\n"
+            r"^Output stream \(<.*?>\) closed before all output was written "
+            r"to it. The following was left in the output buffer:\n\t'hi\\n'\n$"
         )
 
     def test_capture_output(self):