Repository: CoffeaTeam/af-images
Pull request #79: Coffea 2025.5.0.rc1 image
Author: login: oshadura
Created: 2025-05-13T10:07:29Z | Merged: 2025-05-14T08:18:46Z



Diff:
diff --git a/.github/workflows/gh-ci.yaml b/.github/workflows/gh-ci.yaml
index 00320b2..135f8ce 100644
--- a/.github/workflows/gh-ci.yaml
+++ b/.github/workflows/gh-ci.yaml
@@ -17,7 +17,7 @@ env:
   python_latest: "3.12"
   python_latestv0: "3.10"
   # For coffea 2024.x.x we have conda release, github CI bot will detect new version and open PR with changes
-  release: "2025.3.0"
+  release: "2025.5.0.rc1"
   # For coffea 0.7.23 we dont have conda release, please update it manually, as well in coffea-base/environment.yaml
   releasev0: "0.7.26"
 
diff --git a/coffea-dask/environment-aarch64.yaml b/coffea-dask/environment-aarch64.yaml
index fe856d8..7596d35 100644
--- a/coffea-dask/environment-aarch64.yaml
+++ b/coffea-dask/environment-aarch64.yaml
@@ -53,14 +53,16 @@ dependencies:
   - pytorch
   ##- torch-scatter  # no aarch64 support
   - pip
-  - coffea=2025.3.0
-  - rucio-clients
     # pyg
   ##- pyg # no aarch64 support
   ##- pytorch-cluster # no aarch64 support
   ##- pytorch-sparse # no aarch64 support
   ##- pytorch-spline-conv # no aarch64 support
+  #- coffea=2025.3.0
+  #- coffea=2025.3.0
+  - rucio-clients
   - pip:
+    - coffea==2025.5.0.rc1
     - fastjet # to be added to conda-forge: https://github.com/scikit-hep/fastjet/issues/133
     - tritonclient[all]
     - ai-edge-litert-nightly # ai-edge-litert as replacement of tflite is still not available for python
diff --git a/coffea-dask/environment-noml.yaml b/coffea-dask/environment-noml.yaml
index 833f0f7..92b1dfc 100644
--- a/coffea-dask/environment-noml.yaml
+++ b/coffea-dask/environment-noml.yaml
@@ -47,8 +47,9 @@ dependencies:
   - vector
   - hist
   - pip
-  - coffea=2025.3.0
+  #- coffea=2025.3.0
   - rucio-clients
   - pip:
+    - coffea==2025.5.0.rc1
     - fastjet # to be added to conda-forge: https://github.com/scikit-hep/fastjet/issues/133
     - fsspec-xrootd
diff --git a/coffea-dask/environment.yaml b/coffea-dask/environment.yaml
index 9bac5c6..942148b 100644
--- a/coffea-dask/environment.yaml
+++ b/coffea-dask/environment.yaml
@@ -53,14 +53,15 @@ dependencies:
   - pytorch
   - torch-scatter
   - pip
-  - coffea=2025.3.0
-  - rucio-clients
     # pyg
   - pyg
   - pytorch-cluster
   - pytorch-sparse
   - pytorch-spline-conv
+  #- coffea=2025.3.0
+  - rucio-clients
   - pip:
+    - coffea==2025.5.0.rc1
     - fastjet # to be added to conda-forge: https://github.com/scikit-hep/fastjet/issues/133
     - tritonclient[all]
     - ai-edge-litert-nightly # ai-edge-litert as replacement of tflite is still not available for python
diff --git a/tests/test_dimuon.py b/tests/test_dimuon.py
index 68e153b..4e72451 100644
--- a/tests/test_dimuon.py
+++ b/tests/test_dimuon.py
@@ -1,8 +1,3 @@
-try:
-    import hist.dask as hda
-except ImportError:
-    import hist as hda
-
 import dask
 import awkward as ak
 
@@ -13,14 +8,18 @@
 from dask.distributed import Client
 import pytest
 
-fileset = "https://github.com/CoffeaTeam/coffea/raw/master/tests/samples/nano_dimuon.root"
+fileset = (
+    "https://github.com/CoffeaTeam/coffea/raw/master/tests/samples/nano_dimuon.root"
+)
+
 
 class MyProcessor(processor.ProcessorABC):
-    def __init__(self):
-        pass
+    def __init__(self, mode):
+        assert mode in ["eager", "virtual", "dask"]
+        self._mode = mode
 
     def process(self, events):
-        dataset = events.metadata['dataset']
+        dataset = events.metadata["dataset"]
         muons = ak.zip(
             {
                 "pt": events.Muon_pt,
@@ -33,10 +32,14 @@ def process(self, events):
             behavior=candidate.behavior,
         )
 
+        if self._mode == "dask":
+            from hist.dask import Hist as hist_class
+        else:
+            from hist import Hist as hist_class
+
         h_mass = (
-            hda.Hist.new
-            .StrCat(["opposite", "same"], name="sign")
-            .Log(1000, 0.2, 200., name="mass", label="$m_{\mu\mu}$ [GeV]")
+            hist_class.new.StrCat(["opposite", "same"], name="sign")
+            .Log(1000, 0.2, 200.0, name="mass", label="$m_{\mu\mu}$ [GeV]")
             .Int64()
         )
 
@@ -59,47 +62,57 @@ def process(self, events):
     def postprocess(self, accumulator):
         pass
 
+
 @pytest.mark.v0
 def test_processor_dimu_massv0():
-    client = Client()
-    executor = processor.DaskExecutor(client=client)
-    run = processor.Runner(executor=executor,
-                           schema=BaseSchema)
-    out = run({"dimuon": [fileset]},
-              treename="Events",
-              processor_instance=MyProcessor())
-    print(out)
-    assert out["dimuon"]["entries"] == 40
+    with Client() as client:
+        executor = processor.DaskExecutor(client=client)
+        run = processor.Runner(
+            executor=executor,
+            schema=BaseSchema,
+            chunksize=20,
+        )
+        out = run(
+            {"dimuon": [fileset]},
+            treename="Events",
+            processor_instance=MyProcessor("virtual"),
+        )
+        print(out)
+        assert out["dimuon"]["entries"] == 40
+
 
 @pytest.mark.calver
 def test_dimu_masscalver():
-    with Client():
-        events = NanoEventsFactory.from_root(
-            {fileset: "Events"},
-            metadata={"dataset": "DoubleMuon"},
-            schemaclass=BaseSchema
-            ).events()
-        p = MyProcessor()
-        out = p.process(events)
-        (computed,) = dask.compute(out)
-        assert computed["DoubleMuon"]["entries"] == 40
-    #from coffea.dataset_tools import (
-    #    apply_to_fileset,
-    #    max_chunks,
-    #    preprocess,
-    #)
-    #Still not sure where it is used (?)
-    #client = Client()
-    #dataset_runnable = preprocess(
-    #    fileset,
-    #    align_clusters=False,
-    #    files_per_batch=10,
-    #    skip_bad_files=True,
-    #    save_form=False,
-    #)
-    #to_compute = apply_to_fileset(
-    #            MyProcessor(),
-    #            max_chunks(dataset_runnable, 300),
-    #            schemaclass=BaseSchema,
-    #        )
-    #(out,) = dask.compute(to_compute)
+    from coffea.dataset_tools import apply_to_fileset, preprocess
+
+    with Client() as client:
+        executor = processor.DaskExecutor(client=client)
+        run = processor.Runner(
+            executor=executor,
+            schema=BaseSchema,
+            chunksize=20,
+        )
+        out = run(
+            {"DoubleMuon": {"files": {fileset: "Events"}}},
+            processor_instance=MyProcessor("virtual"),
+        )
+        print(out)
+        assert out["DoubleMuon"]["entries"] == 40
+
+        dataset_runnable, dataet_updated = preprocess(
+            {"DoubleMuon": {"files": {fileset: "Events"}}},
+            step_size=20,
+            align_clusters=False,
+            files_per_batch=1,
+            skip_bad_files=True,
+            save_form=False,
+            scheduler=client,
+        )
+        to_compute = apply_to_fileset(
+            MyProcessor("dask"),
+            dataset_runnable,
+            schemaclass=BaseSchema,
+        )
+        (out,) = dask.compute(to_compute)
+        print(out)
+        assert out["DoubleMuon"]["DoubleMuon"]["entries"] == 40


Reviews:
user: login: alexander-held
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-13T18:40:13Z
user: login: ikrommyd
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T18:44:40Z
user: login: ikrommyd
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T18:45:09Z
user: login: alexander-held
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-13T18:46:12Z
user: login: ikrommyd
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T18:51:36Z
user: login: ikrommyd
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T19:04:49Z

Review comments:
diff_hunk: @@ -33,8 +29,13 @@ def process(self, events):
             behavior=candidate.behavior,
         )
 
+        if self._mode == "dask":
+            from hist.dask import Hist as hist_class
path: tests/test_dimuon.py
user: login: alexander-held
body: Is the new coffea RC already changing defaults to use virtual arrays or why does the old setup break?
created_at: 2025-05-13T18:40:13Z
author_association: NONE
diff_hunk: @@ -33,8 +29,13 @@ def process(self, events):
             behavior=candidate.behavior,
         )
 
+        if self._mode == "dask":
+            from hist.dask import Hist as hist_class
path: tests/test_dimuon.py
user: login: ikrommyd
body: yes it is :)
created_at: 2025-05-13T18:44:40Z
author_association: COLLABORATOR
diff_hunk: @@ -33,8 +29,13 @@ def process(self, events):
             behavior=candidate.behavior,
         )
 
+        if self._mode == "dask":
+            from hist.dask import Hist as hist_class
path: tests/test_dimuon.py
user: login: ikrommyd
body: Also brings back executors so I'm adding a test for that in calver as well.
created_at: 2025-05-13T18:45:09Z
author_association: COLLABORATOR
diff_hunk: @@ -33,8 +29,13 @@ def process(self, events):
             behavior=candidate.behavior,
         )
 
+        if self._mode == "dask":
+            from hist.dask import Hist as hist_class
path: tests/test_dimuon.py
user: login: alexander-held
body: I would suggest making this clear in the release notes then, this is not evident from this for me https://github.com/scikit-hep/coffea/releases/tag/v2025.5.0.rc0.
created_at: 2025-05-13T18:46:11Z
author_association: NONE
diff_hunk: @@ -33,8 +29,13 @@ def process(self, events):
             behavior=candidate.behavior,
         )
 
+        if self._mode == "dask":
+            from hist.dask import Hist as hist_class
path: tests/test_dimuon.py
user: login: ikrommyd
body: Yup I agree. It was only a pre-release though that's why. For old/dask behavior all you gotta do is specify `mode="dask"` in nanoevents `from_root`.
created_at: 2025-05-13T18:51:36Z
author_association: COLLABORATOR
diff_hunk: @@ -33,8 +29,13 @@ def process(self, events):
             behavior=candidate.behavior,
         )
 
+        if self._mode == "dask":
+            from hist.dask import Hist as hist_class
path: tests/test_dimuon.py
user: login: ikrommyd
body: I added a comment in https://github.com/scikit-hep/coffea/releases/tag/v2025.5.0.rc0
created_at: 2025-05-13T19:04:49Z
author_association: COLLABORATOR

PR comments:
user: login: ikrommyd
created_at: 2025-05-13T18:57:37Z
body: @oshadura I think the wrong version is being installed, the test fails because it can't find `DaskExecutor` while it exists in the rc
author_association: COLLABORATOR
user: login: ikrommyd
created_at: 2025-05-13T18:59:30Z
body: Fresh environment:
```
 uv venv --python 3.11 --seed
Using CPython 3.11.11 interpreter at: /opt/homebrew/opt/python@3.11/bin/python3.11
Creating virtual environment with seed packages at: .venv
 + pip==25.1.1
 + setuptools==80.4.0
 + wheel==0.45.1
Activate with: source .venv/bin/activate

  iason  …/coffea    master ?    v3.9.6   13:58 
 source .venv/bin/activate

  iason  …/coffea    master ?    v3.11.11   13:58 
 uv pip install coffea==2025.5.0rc1
Resolved 71 packages in 763ms
Prepared 21 packages in 3.93s
Installed 71 packages in 249ms
 + aiohappyeyeballs==2.6.1
 + aiohttp==3.11.18
 + aiosignal==1.3.2
 + annotated-types==0.7.0
 + attrs==25.3.0
 + awkward==2.8.2
 + awkward-cpp==45
 + boost-histogram==1.5.2
 + cachetools==5.5.2
 + certifi==2025.4.26
 + charset-normalizer==3.4.2
 + click==8.2.0
 + cloudpickle==3.1.1
 + coffea==2025.5.0rc1
 + contourpy==1.3.2
 + correctionlib==2.7.0
 + cramjam==2.10.0
 + cycler==0.12.1
 + dask==2025.3.0
 + dask-awkward==2025.5.0
 + dask-histogram==2025.2.0
 + fonttools==4.58.0
 + frozenlist==1.6.0
 + fsspec==2025.3.2
 + fsspec-xrootd==0.5.1
 + hist==2.8.1
 + histoprint==2.6.0
 + idna==3.10
 + importlib-metadata==8.7.0
 + kiwisolver==1.4.8
 + llvmlite==0.44.0
 + locket==1.0.0
 + lz4==4.4.4
 + markdown-it-py==3.0.0
 + matplotlib==3.10.3
 + mdurl==0.1.2
 + mplhep==0.3.59
 + mplhep-data==0.0.4
 + multidict==6.4.3
 + numba==0.61.2
 + numpy==2.2.5
 + packaging==25.0
 + pandas==2.2.3
 + partd==1.4.2
 + pillow==11.2.1
 + propcache==0.3.1
 + pyarrow==20.0.0
 + pydantic==2.11.4
 + pydantic-core==2.33.2
 + pygments==2.19.1
 + pyparsing==3.2.3
 + python-dateutil==2.9.0.post0
 + pytz==2025.2
 + pyyaml==6.0.2
 + requests==2.32.3
 + rich==14.0.0
 + scipy==1.15.3
 + six==1.17.0
 + toml==0.10.2
 + toolz==1.0.0
 + tqdm==4.67.1
 + typing-extensions==4.13.2
 + typing-inspection==0.4.0
 + tzdata==2025.2
 + uhi==0.5.0
 + uproot==5.6.1
 + urllib3==2.4.0
 + vector==1.6.2
 + xxhash==3.5.0
 + yarl==1.20.0
 + zipp==3.21.0

  iason  …/coffea    master ?    v3.11.11   13:58 
 python
Python 3.11.11 (main, Dec  3 2024, 17:20:40) [Clang 16.0.0 (clang-1600.0.26.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from coffea import processor
/Users/iason/Dropbox/work/coffea_dev/coffea/.venv/lib/python3.11/site-packages/coffea/nanoevents/schemas/edm4hep.py:7: FutureWarning: In version 2025.1.0 (target date: 2024-12-31 11:59:59-06:00), this will be an error.
To raise these warnings as errors (and get stack traces to find out where they're called), run
    import warnings
    warnings.filterwarnings("error", module="coffea.*")
after the first `import coffea` or use `@pytest.mark.filterwarnings("error:::coffea.*")` in pytest.
Issue: coffea.nanoevents.methods.vector will be removed and replaced with scikit-hep vector. Nanoevents schemas internal to coffea will be migrated. Otherwise please consider using that package!.
  from coffea.nanoevents.methods import vector
>>> processor.DaskExecutor
<class 'coffea.processor.executor.DaskExecutor'>
```
author_association: COLLABORATOR
user: login: ikrommyd
created_at: 2025-05-13T19:07:02Z
body: I actually see `coffea=2025.3.0=pyh29332c3_0` in the log
author_association: COLLABORATOR
user: login: oshadura
created_at: 2025-05-13T19:15:03Z
body: Yes, I only tried with `noml` image, just to see the results of tests.

On Tue, May 13, 2025, 21:07 Iason Krommydas ***@***.***>
wrote:

> *ikrommyd* left a comment (CoffeaTeam/af-images#79)
> <https://github.com/CoffeaTeam/af-images/pull/79#issuecomment-2877652839>
>
> I actually see coffea=2025.3.0=pyh29332c3_0 in the log
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/CoffeaTeam/af-images/pull/79#issuecomment-2877652839>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABVQAREFH3O5AOCKJMG6EZL26I7GXAVCNFSM6AAAAAB5AFOUMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNZXGY2TEOBTHE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

author_association: MEMBER
user: login: ikrommyd
created_at: 2025-05-13T19:16:43Z
body: @oshadura Please check if you're happy with how I modified the tests as well. I made the calver test to use the `apply_to_fileset` api too, added executors test for calver as well and did a `ruff format`.
author_association: COLLABORATOR