Repository: Pyomo/pyomo

Issue:
Can't run solve() inside a ThreadPool with GLPK
## Summary

From https://stackoverflow.com/questions/73064859/pyomo-how-do-i-run-multiple-solve-concurrently-in-parallel-on-the-same-model:

> I'm using opt = pyo.SolverFactory('glpk'), and I'd like to call opt.solve(), multiple times, inside a map function of a ThreadPool, but it gives error caused by race conditions.
>
> 
> It doesn't work even if I instanciate opt inside the map function.
> 
> The errors I get are random, but in many cases they refer to a log_file which can't be opened.
> 
> I'm even cloning the model for each map function call, to avoid race conditions on it.
> 
> I can't use MPI, I need to use a ThreadPool (which parallelizes only I/O ops, like waiting for the solver to end the task, which is good enough for me)

### Steps to reproduce the issue

```python
import pyomo.environ as pyo
from pyomo.opt import SolverFactory
from multiprocessing.dummy import Pool as ThreadPool

model = pyo.ConcreteModel()
model.nVars = pyo.Param(initialize=4)
model.N = pyo.RangeSet(model.nVars)
model.x = pyo.Var(model.N, within=pyo.Binary)
model.obj = pyo.Objective(expr=pyo.summation(model.x))
model.cuts = pyo.ConstraintList()

def test(_):
    opt = SolverFactory('glpk')
    opt.solve(model) 

    # Iterate, adding a cut to exclude the previously found solution
    for i in range(5):
        expr = 0
        for j in model.x:
            if pyo.value(model.x[j]) < 0.5:
                expr += model.x[j]
            else:
                expr += (1 - model.x[j])
        model.cuts.add( expr >= 1 )
        results = opt.solve(model)
        return results

tp = ThreadPool(4)
results = tp.map(test, range(5))
print(results)
```
or:
```python
import pyomo.environ as pyo
from pyomo.opt import SolverFactory
from multiprocessing.dummy import Pool as ThreadPool

model = pyo.ConcreteModel()
model.nVars = pyo.Param(initialize=4)
model.N = pyo.RangeSet(model.nVars)
model.x = pyo.Var(model.N, within=pyo.Binary)
model.obj = pyo.Objective(expr=pyo.summation(model.x))
model.cuts = pyo.ConstraintList()

def test(model):
    opt = SolverFactory('glpk')
    opt.solve(model) 

    # Iterate, adding a cut to exclude the previously found solution
    for i in range(5):
        expr = 0
        for j in model.x:
            if pyo.value(model.x[j]) < 0.5:
                expr += model.x[j]
            else:
                expr += (1 - model.x[j])
        model.cuts.add( expr >= 1 )
        results = opt.solve(model)
        return results

tp = ThreadPool(4)
results = tp.map(test, [model.clone() for i in range(4)])
print(results)
```

### Error Message

```console
WARNING: Empty constraint block written in LP format - solver may error
WARNING: Empty constraint block written in LP format - solver may error
WARNING: Empty constraint block written in LP format - solver may error
WARNING: Empty constraint block written in LP format - solver may error
Traceback (most recent call last):
  File "c:\Users\_\Desktop\problem.py", line 29, in <module>
    results = tp.map(test, [model.clone() for i in range(4)])
  File "D:\Programs\Python310\lib\multiprocessing\pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "D:\Programs\Python310\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
  File "D:\Programs\Python310\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "D:\Programs\Python310\lib\multiprocessing\pool.py", line 48, in mapstar
    return list(map(*args))
  File "c:\Users\_\Desktop\problem.py", line 14, in test
    opt.solve(model)
  File "D:\Programs\Python310\lib\site-packages\pyomo\opt\base\solvers.py", line 602, in solve
    result = self._postsolve()
  File "D:\Programs\Python310\lib\site-packages\pyomo\opt\solver\shellcmd.py", line 281, in _postsolve
    results = self.process_output(self._rc)
  File "D:\Programs\Python310\lib\site-packages\pyomo\opt\solver\shellcmd.py", line 354, in process_output
    results = self.process_logfile()
  File "D:\Programs\Python310\lib\site-packages\pyomo\solvers\plugins\solvers\GLPK.py", line 213, in process_logfile
    with open(self._log_file, 'r') as output:
OSError: [WinError 6] The handle is invalid
Exception ignored in: <function TeeStream.__del__ at 0x00000237C62AE830>
Traceback (most recent call last):
  File "D:\Programs\Python310\lib\site-packages\pyomo\common\tee.py", line 425, in __del__
    self.close()
  File "D:\Programs\Python310\lib\site-packages\pyomo\common\tee.py", line 407, in close
    h.finalize(self.ostreams)
  File "D:\Programs\Python310\lib\site-packages\pyomo\common\tee.py", line 267, in finalize
    os.close(self.read_pipe)
OSError: [Errno 9] Bad file descriptor
```

### Information on your system

Pyomo version: 6.4.1
Python version: 3.10.5
Operating system: Windows
How Pyomo was installed (PyPI, conda, source): PyPI
Solver (if applicable): GLPK


### Additional information

I posted on the forum too, but none responded:
https://groups.google.com/g/pyomo-forum/c/CovfenBC0wg
Problem solving in multithreading
Hi, I am trying to solve a stochastic problem. The problem for 24 steps and in each step I solve it for 450 different instances.
Now it is possible to solve the problem serially, but that takes a lot of time, so i wanted to solve those 450 instances parallely. 
I approached this in 2 ways:
1) using solver_manager and multiple pyro mip servers. Here I just have to queue the insances and the solver manager would assign the problems to pyro mip servers. But the pyro mip servers occupy a lot of memory and retain it even after the final result is computed. 
2) using opt solver.
a) I can solve using the opt solver directly, but it is slow. 
b) using multi threading. I tried to create a threadPoolExecutor with 5 threads and then solve those 450 instances in different threads. But just shifting to multithreading causes the pyomo to throw errors. 

Code for this approach - 
`
def optimize():

    optsolver = SolverFactory(self.solver_name)
    value = self.initialize_all_values()
    for timestep in range(24):
        instance_list = self.create_instance_list(value)
        futures = {}
        with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
            for instance_object in instance_list:
                futures = {executor.submit(self.thread_solver, instance_object["instance"], optsolver, timestep)}
            for future in concurrent.futures.as_completed(futures):
                try:
                    result = future.result()
                    value.update(result)
                except Exception as e:
                    self.logger.error(e)
`
`
def thread_solver(instance, optsolver, timestep):

    result = optsolver.solve(instance)
    if (result.solver.status == SolverStatus.ok) and (
            result.solver.termination_condition == TerminationCondition.optimal):
        instance.solutions.load_from(result)\
        my_dict = {}
        for v in instance.component_objects(Var, active=True):
            varobject = getattr(instance, str(v))
            var_list = []
            try:
                for index in varobject:
                    var_list.append(varobject[index].value)
                my_dict[str(v)] = var_list
            except Exception as e:
                self.logger.error("error reading result " + str(e))
        value = {key: my_dict[k][0]}
        return value
    elif result.solver.termination_condition == TerminationCondition.infeasible:
        self.logger.info("Termination condition is infeasible")
        return {}
    else:
        self.logger.info("Nothing fits")
        return {}
`
I get the following errors/outputs:
1) Nothing fits
2) Solver failed to locate input problem file: /usr/src/app/temp/pyomo/tmpn23yhmz_.pyomo.lp
I have set the pyomo temp file location to "/usr/src/app/temp/pyomo"

The same code runs perfectly fine without multithreading. 
Any reason for the pyomo to not solve the problem?


Discussion:
After some investigation, I found out that the problem is related to the fact that each solve() call uses the same variable to store the log file name, because there is only one singleton TempFileManager instance.

This issue could be solved by avoiding using one singleton for all the temporary files, allowing to pass a tempfilemanager instance either to solve or solverfactory, making them efectively independent (i.e. thread safe).

Thanks for the bug report and the follow-up.  The root of this is that the TempfileManager was originally written to be used as a stack (which is inherently not thread safe in the way you want to use it).  We have updated it so that it can be used in threaded environments.  Unfortunately, the "stack" usage is hard coded into the older solver interfaces, and a design decision from the very beginning of Pyomo (2008) makes it very difficult to replace the "stack" usage of the TempfileManager with the thread-safe explicit management of the temp file context.  This is on the radar to fix, but it will take some time.

An alternative (more expedient) solution may be to extend the APPSI interface to support GLPK.  @michaelbynum?
I thought it would just be a matter of removing the global instance (the singleton) and replacing broken references with new references to a local instance, optionally passed to the constructor of the solver factory maybe
Each factory would then use its own stack I think. Maybe I didn't understand something. If that's the case, I'm sorry
The problem is that for the older solver interfaces the Solver doesn't directly create / manage the problem writer (there is a sort of "meet in the middle" infrastructure - dating back to the Coopr project when the solver interfaces were not explicitly tied to the Pyomo modeling environment).  This makes it very challenging for the solver to inform the writer of which TempfileManager instance / context that it should use.  Newer solvers (e.g., APPSI) explicitly handle the problem writers, so things could work closer to what you suggest.
Another problem would be that there is no APPSI for GLPK
> Another problem would be that there is no APPSI for GLPK

There will be, but it will take some time.
Alright, thanks.

Today I came up with a monkeypatch idea for the problem.
What do you think about this code? I've put it at the bottom of [tempfiles.py](https://github.com/Pyomo/pyomo/blob/main/pyomo/common/tempfiles.py) locally.
I tested it on my machine (Windows, python 3.10.5) and seems to work in my use case.

```python
from threading import get_ident
class MultiThreadTempFileManager():
    def __init__(self):
        self.tfm_dict = {}

    def __getattr__(self, attr):
        id = get_ident()
        if id not in self.tfm_dict:
            self.tfm_dict[id] = self.__new_manager(id)
        return getattr(self.tfm_dict[id], attr)

    def __new_manager(self, id):
        t = TempfileManagerClass()
        t.tempdir = os.path.join(tempfile.gettempdir(), 'pyomo_' + str(id))
        if os.path.isdir(t.tempdir):
            shutil.rmtree(t.tempdir)
        elif os.path.isfile(t.tempdir):
            os.remove(t.tempdir)
        os.makedirs(t.tempdir, exist_ok=False)
        return t

# The global Pyomo TempfileManager instance
TempfileManager = MultiThreadTempFileManager()
```


If you approve, I can open a PR.


[Edit]
the solution can probably be improved to ensure that tempdir is deleted afterwards.
@ldeluigi: Thank you!  I think this approach has promise, and a PR would be great.

Some immediate thoughts:
- I like the idea of making the `TempfileManager` "thread-specific" (we probably want to also consider this in other contexts where we have global stacks, e.g., `PauseGC`)
- I don't think that each thread-specific `TempfileManager` needs to create their own directory.  As each `TempfileManager` will maintain a list of the tempfiles that it creates (and it uses the underlying system tools for ensuring that tempfiles don't collide), there shouldn't be an issue with collisions or cleanup.
- When we create new `TempfileManager` instances (e.g., for new threads), we should copy over / inherit the `tempdir` from the main thread.  Alternatively, `_resolve_tempdir()` should look to the main thread as part of the search path [I think I like this approach better]
- The `MultiThreadedTempfileManager` should have an API to access the manager associated with the "main" thread.  
I'll work on this in the future, thanks for the feedback!
Issues on the Pyomo GitHub page are meant for bug-reports and feature requests. If you have modeling or implementation questions please ask them on the Pyomo forum or on StackOverflow.
Hi all,
I believe this is a Pyomo bug or feature needed to be added to Pyomo because in multi-threading scenarios, if we want to solve multiple models in parallel.
Parallel calls of
`results = solver.solve(model)`
generate an error: 
**Solver failed to locate input problem file: /tmp/tmpvk4cpr9c.pyomo.lp**
I am experiencing the same issue, I can solve multiple models using multiprocessing successfully. However, I'd like to solve them using multithreading to avoid the serialization overhead. However, I get a similar error as @yifanguan. My pyomo version is 5.7.3.

Resolving patch:
diff --git a/pyomo/common/gc_manager.py b/pyomo/common/gc_manager.py
--- a/pyomo/common/gc_manager.py
+++ b/pyomo/common/gc_manager.py
@@ -17,6 +17,13 @@
 #  ___________________________________________________________________________
 
 import gc
+from pyomo.common.multithread import MultiThreadWrapper
+
+class __PauseGCCompanion(object):
+    def __init__(self):
+        self._stack_depth = 0
+
+PauseGCCompanion: __PauseGCCompanion = MultiThreadWrapper(__PauseGCCompanion)
 
 # PauseGC is a class for clean, scoped management of the Python
 # garbage collector.  To disable the GC for the duration of a
@@ -32,10 +39,7 @@
 # safe to nest instances of PauseGC That is, you don't have to worry
 # if an outer function/method has its own instance of PauseGC.
 class PauseGC(object):
-
     __slots__ = ("reenable_gc", "stack_pointer")
-    _stack_depth = 0
-
     def __init__(self):
         self.stack_pointer = None
         self.reenable_gc = None
@@ -44,8 +48,8 @@ def __enter__(self):
         if self.stack_pointer:
             raise RuntimeError(
                 "Entering PauseGC context manager that was already entered.")
-        PauseGC._stack_depth += 1
-        self.stack_pointer = PauseGC._stack_depth
+        PauseGCCompanion._stack_depth += 1
+        self.stack_pointer = PauseGCCompanion._stack_depth
         self.reenable_gc = gc.isenabled()
         if self.reenable_gc:
             gc.disable()
@@ -57,14 +61,14 @@ def __exit__(self, type, value, traceback):
     def close(self):
         if not self.stack_pointer:
             return
-        if self._stack_depth:
-            if self._stack_depth != self.stack_pointer:
+        if PauseGCCompanion._stack_depth:
+            if PauseGCCompanion._stack_depth != self.stack_pointer:
                 raise RuntimeError(
                     "Exiting PauseGC context manager out of order: there "
                     "are other active PauseGC context managers that were "
                     "entered after this context manager and have not yet "
                     "been exited.")
-            PauseGC._stack_depth -= 1
+            PauseGCCompanion._stack_depth -= 1
             self.stack_pointer = None
         if self.reenable_gc:
             gc.enable()
diff --git a/pyomo/common/multithread.py b/pyomo/common/multithread.py
new file mode 100644
--- /dev/null
+++ b/pyomo/common/multithread.py
@@ -0,0 +1,70 @@
+from collections import defaultdict
+from threading import get_ident, main_thread
+
+
+class MultiThreadWrapper():
+    """A python object proxy that wraps different instances for each thread.
+
+    This is useful for handling thread-safe access to singleton objects without
+    having to refactor the code that depends on them.
+
+    Note that instances of the wrapped object are reused if two threads share the same
+    identifier, because identifiers could be reused and are unique only for any given moment.
+    See [get_ident()](https://docs.python.org/3/library/threading.html#threading.get_ident) for more information.
+    """
+
+    __slots__ = '_mtdict'
+
+    def __init__(self, base):
+        object.__setattr__(self, '_mtdict', defaultdict(base))
+
+    def __getattr__(self, attr):
+        return getattr(self._mtdict[get_ident()], attr)
+    
+    def __setattr__(self, attr, value):
+        setattr(self._mtdict[get_ident()], attr, value)
+    
+    def __delattr__(self, attr):
+        delattr(self._mtdict[get_ident()], attr)
+    
+    def __enter__(self):
+        return self._mtdict[get_ident()].__enter__()
+    
+    def __exit__(self, exc_type, exc_value, traceback):
+        return self._mtdict[get_ident()].__exit__(exc_type, exc_value, traceback)
+    
+    def __dir__(self):
+        return list(object.__dir__(self)) + list(self._mtdict[get_ident()].__dir__())
+    
+    def __str__(self):
+        return self._mtdict[get_ident()].__str__()
+
+    def __new__(cls, wrapped):
+        return super().__new__(type('MultiThreadMeta' + wrapped.__name__, (cls,), {'__doc__': wrapped.__doc__}))
+
+
+class MultiThreadWrapperWithMain(MultiThreadWrapper):
+    """An extension of `MultiThreadWrapper` that exposes the wrapped instance
+    corresponding to the [main_thread()](https://docs.python.org/3/library/threading.html#threading.main_thread)
+    under the `.main_thread` field.
+
+    This is useful for a falling back to a main instance when needed, but results
+    in race conditions if used improperly.
+    """
+
+    def __init__(self, base):
+        super().__init__(base)
+
+    def __getattr__(self, attr):
+        if attr == 'main_thread':
+            return self._mtdict[main_thread().ident]
+        return super().__getattr__(attr)
+
+    def __setattr__(self, attr, value):
+        if attr == 'main_thread':
+            raise ValueError('Setting `main_thread` attribute is not allowed')
+        else:
+            super().__setattr__(attr, value)
+    
+    def __dir__(self):
+        return super().__dir__() + ['main_thread']
diff --git a/pyomo/common/tempfiles.py b/pyomo/common/tempfiles.py
--- a/pyomo/common/tempfiles.py
+++ b/pyomo/common/tempfiles.py
@@ -24,6 +24,7 @@
 import weakref
 from pyomo.common.deprecation import deprecated, deprecation_warning
 from pyomo.common.errors import TempfileContextError
+from pyomo.common.multithread import MultiThreadWrapperWithMain
 try:
     from pyutilib.component.config.tempfiles import (
         TempfileManager as pyutilib_mngr
@@ -428,6 +429,8 @@ def _resolve_tempdir(self, dir=None):
             return self.tempdir
         elif self.manager().tempdir is not None:
             return self.manager().tempdir
+        elif TempfileManager.main_thread.tempdir is not None:
+            return TempfileManager.main_thread.tempdir
         elif pyutilib_mngr is not None and pyutilib_mngr.tempdir is not None:
             deprecation_warning(
                 "The use of the PyUtilib TempfileManager.tempdir "
@@ -467,5 +470,6 @@ def _remove_filesystem_object(self, name):
             name,
             ignore_errors=not deletion_errors_are_fatal)
 
+
 # The global Pyomo TempfileManager instance
-TempfileManager = TempfileManagerClass()
+TempfileManager: TempfileManagerClass = MultiThreadWrapperWithMain(TempfileManagerClass)


Test patch:
diff --git a/pyomo/common/tests/test_multithread.py b/pyomo/common/tests/test_multithread.py
new file mode 100644
--- /dev/null
+++ b/pyomo/common/tests/test_multithread.py
@@ -0,0 +1,118 @@
+import threading
+import pyomo.common.unittest as unittest
+from pyomo.common.multithread import *
+from threading import Thread
+from pyomo.opt.base.solvers import check_available_solvers
+
+class Dummy():
+    """asdfg"""
+    def __init__(self):
+        self.number = 1
+        self.rnd = threading.get_ident()
+
+    def __str__(self):
+        return 'asd'
+
+
+class TestMultithreading(unittest.TestCase):
+
+    def test_wrapper_docs(self):
+        sut = MultiThreadWrapper(Dummy)
+        self.assertEqual(sut.__class__.__doc__, Dummy.__doc__)
+
+    def test_wrapper_field(self):
+        sut = MultiThreadWrapper(Dummy)
+        self.assertEqual(sut.number, 1)
+
+    def test_independent_writes(self):
+        sut = MultiThreadWrapper(Dummy)
+        sut.number = 2
+        def thread_func():
+            self.assertEqual(sut.number, 1)
+        t = Thread(target=thread_func)
+        t.start()
+        t.join()
+
+    def test_independent_writes2(self):
+        sut = MultiThreadWrapper(Dummy)
+        def thread_func():
+            sut.number = 2
+        t = Thread(target=thread_func)
+        t.start()
+        t.join()
+        self.assertEqual(sut.number, 1)
+
+    def test_independent_del(self):
+        sut = MultiThreadWrapper(Dummy)
+        del sut.number
+        def thread_func():
+            self.assertEqual(sut.number, 1)
+        t = Thread(target=thread_func)
+        t.start()
+        t.join()
+
+    def test_independent_del2(self):
+        sut = MultiThreadWrapper(Dummy)
+        def thread_func():
+            del sut.number
+        t = Thread(target=thread_func)
+        t.start()
+        t.join()
+        self.assertEqual(sut.number, 1)
+
+    def test_special_methods(self):
+        sut = MultiThreadWrapper(Dummy)
+        self.assertTrue(set(Dummy().__dir__()).issubset(set(sut.__dir__())))
+        self.assertEqual(str(sut), str(Dummy()))
+
+    def test_main(self):
+        sut = MultiThreadWrapperWithMain(Dummy)
+        self.assertIs(sut.main_thread.rnd, sut.rnd)
+        sut.number = 5
+        def thread_func():
+            self.assertEqual(sut.number, 1)
+            self.assertIsNot(sut.main_thread.rnd, sut.rnd)
+            del sut.number
+        t = Thread(target=thread_func)
+        t.start()
+        t.join()
+        self.assertEqual(sut.number, 5)
+    
+    @unittest.skipIf(len(check_available_solvers('glpk')) < 1, "glpk solver not available")
+    def test_solve(self):
+        # Based on the minimal example in https://github.com/Pyomo/pyomo/issues/2475
+        import pyomo.environ as pyo
+        from pyomo.opt import SolverFactory
+        from multiprocessing.dummy import Pool as ThreadPool
+
+        model = pyo.ConcreteModel()
+        model.nVars = pyo.Param(initialize=4)
+        model.N = pyo.RangeSet(model.nVars)
+        model.x = pyo.Var(model.N, within=pyo.Binary)
+        model.obj = pyo.Objective(expr=pyo.summation(model.x))
+        model.cuts = pyo.ConstraintList()
+
+        def test(model):
+            opt = SolverFactory('glpk')
+            opt.solve(model) 
+
+            # Iterate, adding a cut to exclude the previously found solution
+            for _ in range(5):
+                expr = 0
+                for j in model.x:
+                    if pyo.value(model.x[j]) < 0.5:
+                        expr += model.x[j]
+                    else:
+                        expr += (1 - model.x[j])
+                model.cuts.add( expr >= 1 )
+                results = opt.solve(model)
+            return results, [v for v in model.x]
+
+        tp = ThreadPool(4)
+        results = tp.map(test, [model.clone() for i in range(4)])
+        tp.close()
+        for result, _ in results:
+            self.assertEqual(result.solver.termination_condition, pyo.TerminationCondition.optimal)
+        results = list(results)
+        for _, values in results[1:]:
+            self.assertEqual(values, results[0][1])