Repository: coiled/benchmarks (language: python)

Issue:
Update TPCH Infrastructure
This does a few things:

1.  Generate performance reports for every query if you add the `--performance-report` flag
2. Generate altair comparison charts
3. Change the infrastructure size up and down based on scale
4. Use Arrow filesystem (although I'm not sure how big a deal this is yet)
5. Improve charts to use consistent coloring, and drop ibis for pandas


Discussion / hints:
Check out this pull request on&nbsp; <a href="https://app.reviewnb.com/coiled/benchmarks/pull/1134"><img align="absmiddle"  alt="ReviewNB" height="28" class="BotMessageButtonImage" src="https://raw.githubusercontent.com/ReviewNB/support/master/images/button_reviewnb.png"/></a> 

 See visual diffs & provide feedback on Jupyter Notebooks. 

---

 <i>Powered by <a href='https://www.reviewnb.com/?utm_source=gh'>ReviewNB</a></i>
Merging tomorrow morning my time if there are no comments
@mrockling arrow file system doesn't work with dask-expr main. I am guessing that you are still using my hacky branch that avoid the error? Should we invest effort into making this work or roll back for now?
@phofl why not just merge your fixes in dask-expr?
or how much work would that be if you say this is "hacky"?
My fix removes the filesystem from our hashing logic, resulting in ugly patches in a few places. This was never meant to get merged, rather to try it out.

We have to implement a tokenize logic for Arrow Filesystems. This might be trivial, I don't know that
@phofl: What branch are you talking about here? There's https://github.com/dask/distributed/issues/8301, for which I'd need the TPC-H tests to work to dig deeper into it.
https://github.com/phofl/dask-expr/tree/testing
> @mrocklin arrow file system doesn't work with dask-expr main. I am guessing that you are still using my hacky branch that avoid the error? Should we invest effort into making this work or roll back for now?

I think I'm no longer using arrow filesystem anyway (although I would like to)

> We have to implement a tokenize logic for Arrow Filesystems

We don't need to do anything smart here I think.  It's ok for us to be dumb I think?  We might not get good reuse, but that's not a big deal.  Or maybe I don't understand the situation.  Is there somewhere I can learn more?
@hendrikmakait you can apply this diff (I'll fix this shortly)

```diff
diff --git a/tests/tpch/conftest.py b/tests/tpch/conftest.py
index d174a78..07a9237 100644
--- a/tests/tpch/conftest.py
+++ b/tests/tpch/conftest.py
@@ -291,6 +291,7 @@ def fs(local):
     if local:
         return None
     else:
+        return None
         import boto3
         from pyarrow.fs import S3FileSystem
```

Fix patch:
diff --git a/AB_environments/AB_baseline.conda.yaml b/AB_environments/AB_baseline.conda.yaml
index 0baa1d60ae..32db749370 100644
--- a/AB_environments/AB_baseline.conda.yaml
+++ b/AB_environments/AB_baseline.conda.yaml
@@ -46,7 +46,7 @@ dependencies:
   - pyspark ==3.4.1
   - openjdk ==20.0.2
   - python-duckdb ==0.9.1
-  - polars >=0.19.3,<=0.19.8  # 0.19.3 on Windows, 0.19.8 on Linux and MacOS
+  - polars >=0.19.3,<=0.19.10  # 0.19.3 on Windows, 0.19.10 on Linux and MacOS
   # End copy-paste
 
   - pip:
diff --git a/AB_environments/AB_sample.conda.yaml b/AB_environments/AB_sample.conda.yaml
index fc8cca8af1..980386b31d 100644
--- a/AB_environments/AB_sample.conda.yaml
+++ b/AB_environments/AB_sample.conda.yaml
@@ -51,7 +51,7 @@ dependencies:
   - pyspark ==3.4.1
   - openjdk ==20.0.2
   - python-duckdb ==0.9.1
-  - polars >=0.19.3,<=0.19.8  # 0.19.3 on Windows, 0.19.8 on Linux and MacOS
+  - polars >=0.19.3,<=0.19.10  # 0.19.3 on Windows, 0.19.10 on Linux and MacOS
   # End copy-paste
 
   - pip:
diff --git a/ci/environment.yml b/ci/environment.yml
index 18975fbf49..d857d26a15 100644
--- a/ci/environment.yml
+++ b/ci/environment.yml
@@ -1,6 +1,13 @@
 channels:
   - conda-forge
 dependencies:
+########################################################
+# PLEASE READ:
+# When modifying this file, you must also align to match
+# - AB_environments/AB_baseline.conda.yaml
+# - AB_environments/AB_sample.conda.yaml
+########################################################
+
   - python >=3.9
   - pip
   - coiled >=0.2.54
@@ -42,4 +49,10 @@ dependencies:
   - openjdk ==20.0.2
   - python-duckdb ==0.9.1
   - polars >=0.19.3,<=0.19.10  # 0.19.3 on Windows, 0.19.10 on Linux and MacOS
-  - altair
+
+########################################################
+# PLEASE READ:
+# When modifying this file, you must also align to match
+# - AB_environments/AB_baseline.conda.yaml
+# - AB_environments/AB_sample.conda.yaml
+########################################################


Test patch:
diff --git a/ci/environment-test.yml b/ci/environment-test.yml
index 50b61778b5..8f393499a3 100644
--- a/ci/environment-test.yml
+++ b/ci/environment-test.yml
@@ -4,12 +4,13 @@ channels:
   - conda-forge
 dependencies:
   # Testing dependencies
+  - alembic
+  - altair
+  - conda
+  - filelock
   - jinja2
   - packaging
   - pytest
   - pytest-timeout
   - pytest-xdist
   - pyyaml
-  - alembic
-  - filelock
-  - conda