Repository: Pyomo/pyomo

Issue:
Windows: CPLEX direct, solver_io="direct" throws error when tee=True and keepfiles=False
Below is an example in which the error occurs. The problem is that `tee=True` seems to reserve the file, which is why `pyutilib` cannot remove the temporary files. The code is followed by the console output.

Setting `keepfiles=False` will 'solve' this issue, but this is not very asthetic. `results = opt.solve(m, tee=True, keepfiles=False)`.

```python
from pyomo.environ import *
from pyomo.opt import SolverFactory

# define the model
n_steps = 100
m = ConcreteModel()
m.STEPS = Set(initialize=range(n_steps))
m.target = Param(m.STEPS, initialize=lambda m, s: 1.0)
m.Build = Var()
m.Operate = Var(m.STEPS)
m.Meet_Target = Constraint(m.STEPS, rule=lambda m, s: m.Operate[s] >= m.target[s])
m.Operate_Limit = Constraint(m.STEPS, rule=lambda m, s: m.Operate[s] <= m.Build)
m.MinCost = Objective(
    rule=lambda m: n_steps * m.Build + sum(1.0 * m.Operate[s] for s in m.STEPS), 
    sense=minimize
)

# solve the model
opt = SolverFactory("cplex", solver_io="direct")
results = opt.solve(m, tee=True, keepfiles=False)
m.solutions.load_from(results)
```

```
CPXPARAM_Read_DataCheck                          1
CPXPARAM_Read_APIEncoding                        "UTF-8"
CPXPARAM_MIP_Strategy_CallbackReducedLP          0
Tried aggregator 1 time.
LP Presolve eliminated 200 rows and 101 columns.
All rows and columns eliminated.
Presolve time = 0.00 sec. (0.05 ticks)
Traceback (most recent call last):
  File "C:\Anaconda3\envs\python35\lib\site-packages\pyutilib\component\config\tempfiles.py", line 171, in pop
    os.remove(filename)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Jasper\\AppData\\Local\\Temp\\tmp770of057.log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "error.py", line 20, in <module>
    results = opt.solve(m, tee=True, keepfiles=False)
  File "C:\Anaconda3\envs\python35\lib\site-packages\pyomo\solvers\plugins\solvers\direct_solver.py", line 149, in solve
    result = self._postsolve()
  File "C:\Anaconda3\envs\python35\lib\site-packages\pyomo\solvers\plugins\solvers\cplex_direct.py", line 639, in _postsolve
    pyutilib.services.TempfileManager.pop(remove=not self._keepfiles)
  File "C:\Anaconda3\envs\python35\lib\site-packages\pyutilib\component\config\tempfiles.py", line 179, in pop
    os.remove(filename)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Jasper\\AppData\\Local\\Temp\\tmp770of057.log'
```
Thank you for your great work!


Discussion:
This script works fine on my mac with both keepfiles=True and keepfiles=False. Unfortunately it will be some time before I can test this on a Windows machine. 

What version of cplex are you using?

Given your email about the reduced_costs bug, I assume you are using the master branch of pyomo (as opposed to a specific release)?

Michael

> On Dec 18, 2017, at 6:37 AM, Jasper <notifications@github.com> wrote:
> 
> Below is an example in which the error occurs. The problem is that tee=True seems to reserve the file, which is why pyutilib cannot remove the temporary files. The code is followed by the console output.
> 
> Setting keepfiles=False will solve this issue, but this is not very asthetic. results = opt.solve(m, tee=True, keepfiles=False).
> 
> from pyomo.environ import *
> from pyomo.opt import SolverFactory
> 
> # define the model
> n_steps = 100
> m = ConcreteModel()
> m.STEPS = Set(initialize=range(n_steps))
> m.target = Param(m.STEPS, initialize=lambda m, s: 1.0)
> m.Build = Var()
> m.Operate = Var(m.STEPS)
> m.Meet_Target = Constraint(m.STEPS, rule=lambda m, s: m.Operate[s] >= m.target[s])
> m.Operate_Limit = Constraint(m.STEPS, rule=lambda m, s: m.Operate[s] <= m.Build)
> m.MinCost = Objective(
>     rule=lambda m: n_steps * m.Build + sum(1.0 * m.Operate[s] for s in m.STEPS), 
>     sense=minimize
> )
> 
> # solve the model
> opt = SolverFactory("cplex", solver_io="direct")
> results = opt.solve(m, tee=True, keepfiles=False)
> m.solutions.load_from(results)
> CPXPARAM_Read_DataCheck                          1
> CPXPARAM_Read_APIEncoding                        "UTF-8"
> CPXPARAM_MIP_Strategy_CallbackReducedLP          0
> Tried aggregator 1 time.
> LP Presolve eliminated 200 rows and 101 columns.
> All rows and columns eliminated.
> Presolve time = 0.00 sec. (0.05 ticks)
> Traceback (most recent call last):
>   File "C:\Anaconda3\envs\python35\lib\site-packages\pyutilib\component\config\tempfiles.py", line 171, in pop
>     os.remove(filename)
> PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Jasper\\AppData\\Local\\Temp\\tmp770of057.log'
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "error.py", line 20, in <module>
>     results = opt.solve(m, tee=True, keepfiles=False)
>   File "C:\Anaconda3\envs\python35\lib\site-packages\pyomo\solvers\plugins\solvers\direct_solver.py", line 149, in solve
>     result = self._postsolve()
>   File "C:\Anaconda3\envs\python35\lib\site-packages\pyomo\solvers\plugins\solvers\cplex_direct.py", line 639, in _postsolve
>     pyutilib.services.TempfileManager.pop(remove=not self._keepfiles)
>   File "C:\Anaconda3\envs\python35\lib\site-packages\pyutilib\component\config\tempfiles.py", line 179, in pop
>     os.remove(filename)
> PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Jasper\\AppData\\Local\\Temp\\tmp770of057.log'
> Thank you for your great work!
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub <https://github.com/Pyomo/pyomo/issues/285>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATdOLlX_B6CtVVHAVvTcJZSklHRPPbNVks5tBmqVgaJpZM4RFesV>.
> 


Hi Michael,

I'm using CPLEX 12.7.1.0 (academic release). It is correct that I'm working on the master branch.
Note that the bug is not critical, as the user can choose to remove the temporary files can be removed manually anyway.

Thanks,
Jasper
I am having the same issue. Any updates on this?
On the note, this happens not only in CPLEX direct, but also in CPLEX persistent. I am having the same issue on Windows.

Resolving patch:
diff --git a/pyomo/solvers/plugins/solvers/cplex_direct.py b/pyomo/solvers/plugins/solvers/cplex_direct.py
--- a/pyomo/solvers/plugins/solvers/cplex_direct.py
+++ b/pyomo/solvers/plugins/solvers/cplex_direct.py
@@ -10,6 +10,7 @@
 
 import logging
 import re
+import six
 import sys
 import pyomo.common
 from pyutilib.misc import Bunch
@@ -159,18 +160,27 @@ def _apply_solver(self):
             for block in self._pyomo_model.block_data_objects(descend_into=True, active=True):
                 for var in block.component_data_objects(ctype=pyomo.core.base.var.Var, descend_into=False, active=True, sort=False):
                     var.stale = True
-        _log_file = self._log_file
-        if self.version() >= (12, 10):
-            _log_file = open(self._log_file, 'w')
+        # In recent versions of CPLEX it is helpful to manually open the
+        # log file and then explicitly close it after CPLEX is finished.
+        # This ensures that the file is closed (and unlocked) on Windows
+        # before the TempfileManager (or user) attempts to delete the
+        # log file.  Passing in an opened file object is supported at
+        # least as far back as CPLEX 12.5.1 [the oldest version
+        # supported by IBM as of 1 Oct 2020]
+        if self.version() >= (12, 5, 1) \
+           and isinstance(self._log_file, six.string_types):
+            _log_file = (open(self._log_file, 'a'),)
+            _close_log_file = True
+        else:
+            _log_file = (self._log_file,)
+            _close_log_file = False
+        if self._tee:
+            def _process_stream(arg):
+                sys.stdout.write(arg)
+                return arg
+            _log_file += (_process_stream,)
         try:
-            if self._tee:
-                def _process_stream(arg):
-                    sys.stdout.write(arg)
-                    return arg
-                self._solver_model.set_results_stream(_log_file, _process_stream)
-            else:
-                self._solver_model.set_results_stream(_log_file)
-            
+            self._solver_model.set_results_stream(*_log_file)
             if self._keepfiles:
                 print("Solver log file: "+self._log_file)
             
@@ -240,8 +250,9 @@ def _process_stream(arg):
             t1 = time.time()
             self._wallclock_time = t1 - t0
         finally:
-            if self.version() >= (12, 10):
-                _log_file.close()
+            self._solver_model.set_results_stream(None)
+            if _close_log_file:
+                _log_file[0].close()
 
         # FIXME: can we get a return code indicating if CPLEX had a significant failure?
         return Bunch(rc=None, log=None)


Test patch: