Repository: FlexMeasures/flexmeasures (language: python)

Issue:
HiGHs solver in Python3.8
Under 3.8, the HiGHs solver runs into a "maximum recursion depth exceeded" error.

That's why #796 switched back to Cbc for testing.

Maybe we can find out why 3.8 has this problem and enable 3.8 installations to use HiGHS solver.


Discussion / hints:
Thank for reporting this behavior!

I could reproduce the issue and I found the following:
- The Pyomo and HiGHS versions are the same in all three python versions.
- The test is failing in the creation of the constraint  `device_energy_bounds[0,113]` to `device_energy_bounds[0,187]`. This happens when adding a target SOC in the test  `test_trigger_and_get_schedule`, in the second case [message1 - Test charging station].
- Also, I found that the schedule only fails when being called through the API, this is super weird. Deselecting it makes the test suit pass... (`pytest -k 'not test_sensor_schedules'`)

That's what I got so far.



Fix patch:
diff --git a/documentation/configuration.rst b/documentation/configuration.rst
index dc97881df..d17314f4a 100644
--- a/documentation/configuration.rst
+++ b/documentation/configuration.rst
@@ -59,7 +59,7 @@ The command to run the scheduling solver. This is the executable command which F
 We have tested FlexMeasures with `HiGHS <https://highs.dev/>`_ and `Cbc <https://coin-or.github.io/Cbc/intro>`_.
 Note that you need to install the solver, read more at :ref:`installing-a-solver`.
 
-Default: ``"appsi_highs"``  (in unit testing, we use ``cbc`` as it runs well for us in Python3.8)
+Default: ``"appsi_highs"``
 
 
 
diff --git a/documentation/host/deployment.rst b/documentation/host/deployment.rst
index f25494a68..f10153006 100644
--- a/documentation/host/deployment.rst
+++ b/documentation/host/deployment.rst
@@ -56,7 +56,7 @@ Solvers are used through `Pyomo <http://www.pyomo.org>`_\ , so in principle supp
 They need to be installed in addition to FlexMeasures. Here is advice on how to install the two solvers we test internally:
 
 
-.. note:: We default to HiGHS, as it seems more powerful, but during unit tests we currently run Cbc, as it works for us on Python3.8
+.. note:: We default to HiGHS, as it seems more powerful
 
 
 HiGHS can be installed using pip:
diff --git a/flexmeasures/utils/config_defaults.py b/flexmeasures/utils/config_defaults.py
index b20415a0b..346eea8d6 100644
--- a/flexmeasures/utils/config_defaults.py
+++ b/flexmeasures/utils/config_defaults.py
@@ -195,9 +195,6 @@ class TestingConfig(Config):
     SECURITY_HASHING_SCHEMES: list[str] = ["hex_md5"]
     SECURITY_DEPRECATED_HASHING_SCHEMES: list[str] = []
     FLEXMEASURES_MODE: str = "test"
-    FLEXMEASURES_LP_SOLVER: str = (
-        "cbc"  # this solver is currently the one we know is working in Python3.8
-    )
     FLEXMEASURES_PLANNING_HORIZON: timedelta = timedelta(
         hours=2 * 24
     )  # if more than 2 days, consider setting up more days of price data for tests


Test patch:
diff --git a/flexmeasures/data/models/planning/tests/conftest.py b/flexmeasures/data/models/planning/tests/conftest.py
index 5510778db..0f9d50fc0 100644
--- a/flexmeasures/data/models/planning/tests/conftest.py
+++ b/flexmeasures/data/models/planning/tests/conftest.py
@@ -11,6 +11,21 @@
 from flexmeasures.data.models.time_series import Sensor, TimedBelief
 
 
+@pytest.fixture(params=["appsi_highs", "cbc"])
+def app_with_each_solver(app, request):
+    """Set up the app config to run with different solvers.
+
+    A test that uses this fixtures runs all of its test cases with HiGHS and then again with Cbc.
+    """
+    original_solver = app.config["FLEXMEASURES_LP_SOLVER"]
+    app.config["FLEXMEASURES_LP_SOLVER"] = request.param
+
+    yield app
+
+    # Restore original config setting for the solver
+    app.config["FLEXMEASURES_LP_SOLVER"] = original_solver
+
+
 @pytest.fixture(scope="module", autouse=True)
 def setup_planning_test_data(db, add_market_prices, add_charging_station_assets):
     """
diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py
index 4091ded70..898b96570 100644
--- a/flexmeasures/data/models/planning/tests/test_solver.py
+++ b/flexmeasures/data/models/planning/tests/test_solver.py
@@ -992,8 +992,7 @@ def get_sensors_from_db(battery_assets):
     return epex_da, battery
 
 
-@pytest.mark.parametrize("solver", ["appsi_highs", "cbc"])
-def test_numerical_errors(app, setup_planning_test_data, solver):
+def test_numerical_errors(app_with_each_solver, setup_planning_test_data):
     """Test that a soc-target = soc-max can exceed this value due to numerical errors in the operations
     to compute the device constraint DataFrame.
     In the case of HiGHS, the tiny difference creates an infeasible constraint.
@@ -1017,9 +1016,6 @@ def test_numerical_errors(app, setup_planning_test_data, solver):
         start + 2 * duration_until_next_target
     ).isoformat()
 
-    # select which solver to use
-    app.config["FLEXMEASURES_LP_SOLVER"] = solver
-
     scheduler = StorageScheduler(
         charging_station,
         start,