Repository: dask/dask
Pull request #11863 (2025-04-02 14:48:52)

Faster tokenization of pd.RangeIndex
Closes https://github.com/dask/dask/issues/11862

cc @dcherian 

github-actions[bot]: ## Unit Test Results
_See [test report](https://dask.github.io/dask/test_report.html) for an extended history of previous test failures. This is useful for diagnosing flaky tests._

      9 files  ±0        9 suites  ±0   3h 30m 20s ⏱️ + 1m 40s
 18 002 tests +1   16 786 ✅  - 1   1 214 💤 ±0   2 ❌ + 2 
161 092 runs  +9  148 977 ✅ +1  12 105 💤  - 2  10 ❌ +10 

For more details on these failures, see [this check](https://github.com/dask/dask/runs/39857823050).

Results for commit fb22cc17. ± Comparison against base commit d9e07f52.

[test-results]:data:application/gzip;base64,H4sIAIpZ7WcC/1WMwQ6DIBAFf8Vw7oFdFZb+TKMICalKg3Ay/feiEUyPMy9vdmbdbDb2bNSjYVtyscKUwhCdXzMCCuRZ5TEeMxDnWPi1Ja0PKSSJW77d53xCV50d3Jzd/TQh+JDN0Q5pPdMCuMKLa7ojJWWVNc374q40lNJfWvtlcTEDsyOi1iA7JQh0Cz2RHewEiJLsqNqpb4kMGPb9AaRgelMZAQAA


Patch:
diff --git a/dask/tokenize.py b/dask/tokenize.py
index e1b434c41..fb3b81aa1 100644
--- a/dask/tokenize.py
+++ b/dask/tokenize.py
@@ -274,10 +274,14 @@ def _normalize_dataclass(obj):
 def register_pandas():
     import pandas as pd
 
+    @normalize_token.register(pd.RangeIndex)
+    def normalize_range_index(x):
+        return type(x), x.start, x.stop, x.step, x.dtype, x.name
+
     @normalize_token.register(pd.Index)
     def normalize_index(ind):
         values = ind.array
-        return [ind.name, normalize_token(values)]
+        return type(ind), ind.name, normalize_token(values)
 
     @normalize_token.register(pd.MultiIndex)
     def normalize_index(ind):
diff --git a/docs/source/support.rst b/docs/source/support.rst
index 1d4e586e8..588950d66 100644
--- a/docs/source/support.rst
+++ b/docs/source/support.rst
@@ -99,4 +99,4 @@ following organizations (listed in alphabetical order):
 
 -   `Anaconda <https://www.anaconda.com/products/professional-services>`_
 -  `Coiled <https://coiled.io?utm_source=dask-docs&utm_medium=support>`_
--   `Quansight <https://www.quansight.com/open-source-support>`_
+-   `Quansight <https://www.quansight.com/>`_


Test patch:
diff --git a/dask/tests/test_tokenize.py b/dask/tests/test_tokenize.py
index 6e1c165e8..51c6750fa 100644
--- a/dask/tests/test_tokenize.py
+++ b/dask/tests/test_tokenize.py
@@ -1481,3 +1481,43 @@ def test_tokenize_nested_sequence_thread_safe():
         assert len({f.result() for f in futures}) == 1
         futures = [pool.submit(normalize_token, nested_dict) for _ in range(1000)]
         assert len({f.result() for f in futures}) == 1
+
+
+def test_tokenize_range_index():
+    np = pytest.importorskip("numpy")
+    pd = pytest.importorskip("pandas")
+    a = pd.RangeIndex(0, 10)
+    b = pd.RangeIndex(0, 10)
+    c = pd.RangeIndex(0, 20)
+    assert check_tokenize(a) == check_tokenize(b)
+    assert check_tokenize(a) != check_tokenize(c)
+
+    d = pd.RangeIndex(0, 10, step=2)
+    e = pd.RangeIndex(0, 10, step=3)
+    assert check_tokenize(d) != check_tokenize(e)
+
+    large = pd.RangeIndex(0, 1000000)
+    array_token = normalize_token(np.zeros(10))
+    assert isinstance(array_token[0], str)
+    # hex_buffer_output_len
+    # It's difficult to assert what actually goes into the tokens
+    # Therefore we just check if there is a string of the appropriate length in
+    # the tokens below This check is to verify the assumption and to avoid hard
+    # coding a length. This seems to not be stable accross python versions or
+    # platforms but there is a minimal length
+    hex_hash_len = len(array_token[0])
+    assert hex_hash_len >= 32
+
+    def assert_no_hashes(tokens):
+        if isinstance(tokens, (list, tuple)):
+            for token in tokens:
+                assert_no_hashes(token)
+        elif isinstance(tokens, dict):
+            for key, value in tokens.items():
+                assert_no_hashes(key)
+                assert_no_hashes(value)
+        elif isinstance(tokens, str):
+            if len(tokens) == hex_hash_len:
+                raise AssertionError("found a hash")
+
+    assert_no_hashes(normalize_token(large))