Repository: coiled/benchmarks (language: python)

Issue:
TPC-H PySpark: ValueError: You passed `region` as a kwarg to Cluster(...), and included region_name in the backend_options dict. Only one of those should be specified.
```
ERROR tests/tpch/test_pyspark.py::test_query_1 - ValueError: You passed `region` as a kwarg to Cluster(...), and included region_name in the backend_options dict. Only one of those should be specified.
ERROR tests/tpch/test_pyspark.py::test_query_2 - ValueError: You passed `region` as a kwarg to Cluster(...), and included region_name in the backend_options dict. Only one of those should be specified.
ERROR tests/tpch/test_pyspark.py::test_query_3 - ValueError: You passed `region` as a kwarg to Cluster(...), and included region_name in the backend_options dict. Only one of those should be specified.
ERROR tests/tpch/test_pyspark.py::test_query_4 - ValueError: You passed `region` as a kwarg to Cluster(...), and included region_name in the backend_options dict. Only one of those should be specified.
ERROR tests/tpch/test_pyspark.py::test_query_5 - ValueError: You passed `region` as a kwarg to Cluster(...), and included region_name in the backend_options dict. Only one of those should be specified.
ERROR tests/tpch/test_pyspark.py::test_query_6 - ValueError: You passed `region` as a kwarg to Cluster(...), and included region_name in the backend_options dict. Only one of those should be specified.
ERROR tests/tpch/test_pyspark.py::test_query_7 - ValueError: You passed `region` as a kwarg to Cluster(...), and included region_name in the backend_options dict. Only one of those should be specified.
```

https://github.com/coiled/benchmarks/actions/runs/6661165578/job/18103591355#step:11:3241

#1153


Discussion / hints:

Fix patch:
diff --git a/AB_environments/AB_baseline.conda.yaml b/AB_environments/AB_baseline.conda.yaml
index 0baa1d60ae..ccce0c9d28 100644
--- a/AB_environments/AB_baseline.conda.yaml
+++ b/AB_environments/AB_baseline.conda.yaml
@@ -44,7 +44,10 @@ dependencies:
   - gilknocker ==0.4.1
   - openssl >1.1.0g
   - pyspark ==3.4.1
-  - openjdk ==20.0.2
+  - openjdk ~=11.0
+  - grpcio ==1.58.1
+  - grpcio-status ==1.57.0
+  - protobuf ==4.24.3
   - python-duckdb ==0.9.1
   - polars >=0.19.3,<=0.19.8  # 0.19.3 on Windows, 0.19.8 on Linux and MacOS
   # End copy-paste
diff --git a/AB_environments/AB_sample.conda.yaml b/AB_environments/AB_sample.conda.yaml
index fc8cca8af1..95f3426b29 100644
--- a/AB_environments/AB_sample.conda.yaml
+++ b/AB_environments/AB_sample.conda.yaml
@@ -49,7 +49,10 @@ dependencies:
   - gilknocker ==0.4.1
   - openssl >1.1.0g
   - pyspark ==3.4.1
-  - openjdk ==20.0.2
+  - openjdk ~=11.0
+  - grpcio ==1.58.1
+  - grpcio-status ==1.57.0
+  - protobuf ==4.24.3
   - python-duckdb ==0.9.1
   - polars >=0.19.3,<=0.19.8  # 0.19.3 on Windows, 0.19.8 on Linux and MacOS
   # End copy-paste
diff --git a/ci/environment.yml b/ci/environment.yml
index 18975fbf49..6cbe00a3ae 100644
--- a/ci/environment.yml
+++ b/ci/environment.yml
@@ -39,7 +39,10 @@ dependencies:
   - gilknocker ==0.4.1
   - openssl >1.1.0g
   - pyspark ==3.4.1
-  - openjdk ==20.0.2
+  - openjdk ~=11.0
+  - grpcio ==1.58.1
+  - grpcio-status ==1.57.0
+  - protobuf ==4.24.3
   - python-duckdb ==0.9.1
   - polars >=0.19.3,<=0.19.10  # 0.19.3 on Windows, 0.19.10 on Linux and MacOS
   - altair


Test patch:
diff --git a/tests/tpch/conftest.py b/tests/tpch/conftest.py
index 3442878354..bb70c2d0c4 100644
--- a/tests/tpch/conftest.py
+++ b/tests/tpch/conftest.py
@@ -377,7 +377,13 @@ def run(module_run, restart, benchmark_time, warm_start, make_chart):
 
 
 @pytest.fixture(scope="session")
-def make_chart(name, tmp_path_factory, local, scale):
+def make_chart(request, name, tmp_path_factory, local, scale):
+    if not request.config.getoption("--benchmark"):
+        # Won't create the sqlite DB, and thus won't be able
+        # to read test run information
+        yield
+        return
+
     root_tmp_dir = tmp_path_factory.getbasetemp().parent
     lock = filelock.FileLock(root_tmp_dir / "tpch.lock")