Repository: dask/distributed

Issue:
msgpack 1.0.6 breaks Client.profile on Windows
These two tests have been failing on Windows since msgpack 1.0.6 was released on conda:
```
  File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\msgpack\fallback.py", line 805, in _pack
    return self._pack_map_pairs(len(obj), obj.items(), nest_limit - 1)
  File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\msgpack\fallback.py", line 909, in _pack_map_pairs
    self._pack(v, nest_limit - 1)
  File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\msgpack\fallback.py", line 805, in _pack
    return self._pack_map_pairs(len(obj), obj.items(), nest_limit - 1)
  File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\msgpack\fallback.py", line 908, in _pack_map_pairs
    self._pack(k, nest_limit - 1)
  File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\msgpack\fallback.py", line 713, in _pack
    raise ValueError("recursion limit exceeded")
ValueError: recursion limit exceeded
```

![image](https://github.com/dask/distributed/assets/6213168/35313b12-176d-439f-94fc-51cb2f2be50b)
![image](https://github.com/dask/distributed/assets/6213168/ea747644-15ba-4726-9419-335cc50bb3b1)



Discussion:
- XREF https://github.com/dask/distributed/pull/3455
The issue with `test_profile_nested_sizeof` is that
- sizeof() raises RecursionError on all platforms at 430 nested calls
- msgpack.dumps() 1.0.5 raises ValueError on all platforms at 512 nested calls
- msgpack.dumps() 1.0.6 raises ValueError on Windows at 256 nested calls and at 512 nested calls on MacOSX and Linux

`test_profile_nested_sizeof` was hitting a sweet spot where msgpack was still working but sizeof wasn't.
`test_profile_server` is more serious - this is not just a test problem. It shows that msgpack 1.0.6 simply does not have enough recursion depth on Windows to work together with profile.
```python
import sys
import msgpack

for i in range(200, 5000):
    root = outer = {}
    inner = {}

    for _ in range(i):
        outer["children"] = inner
        outer, inner = inner, {}

    try:
        msgpack.dumps(root)
    except ValueError:
        print(
            f"msgpack {msgpack.__version__} on Python "
            f"{sys.version.split()[0]}/{sys.platform} raises at {i}"
        )
        break
```
Output:
```
msgpack 1.0.5 on Python 3.9.18/linux raises at 512
msgpack 1.0.5 on Python 3.10.12/linux raises at 512
msgpack 1.0.5 on Python 3.11.5/linux raises at 512

msgpack 1.0.5 on Python 3.9.18/win32 raises at 512
msgpack 1.0.5 on Python 3.10.12/win32 raises at 512
msgpack 1.0.5 on Python 3.11.5/win32 raises at 512

msgpack 1.0.6 on Python 3.9.18/linux raises at 512
msgpack 1.0.6 on Python 3.10.12/linux raises at 512
msgpack 1.0.6 on Python 3.11.5/linux raises at 512

msgpack 1.0.6 on Python 3.9.18/win32 raises at 256
msgpack 1.0.6 on Python 3.10.12/win32 raises at 256
msgpack 1.0.6 on Python 3.11.5/win32 raises at 256
```
Upstream: https://github.com/msgpack/msgpack-python/issues/566

Resolving patch:
diff --git a/distributed/comm/utils.py b/distributed/comm/utils.py
--- a/distributed/comm/utils.py
+++ b/distributed/comm/utils.py
@@ -1,14 +1,13 @@
 from __future__ import annotations
 
 import logging
-import math
 import socket
 
 import dask
 from dask.utils import parse_bytes
 
 from distributed import protocol
-from distributed.sizeof import safe_sizeof as sizeof
+from distributed.sizeof import safe_sizeof
 from distributed.utils import get_ip, get_ipv6, nbytes, offload
 
 logger = logging.getLogger(__name__)
@@ -59,17 +58,16 @@ def _to_frames():
             raise
 
     if OFFLOAD_THRESHOLD and allow_offload:
-        try:
-            msg_size = sizeof(msg)
-        except RecursionError:
-            msg_size = math.inf
-    else:
-        msg_size = 0
-
-    if allow_offload and OFFLOAD_THRESHOLD and msg_size > OFFLOAD_THRESHOLD:
-        return await offload(_to_frames)
-    else:
-        return _to_frames()
+        # dask.sizeof.sizeof() starts raising RecursionError at ~140 recursion depth,
+        # whereas msgpack can go on for quite a bit longer, until 512 (sometimes 256,
+        # depending on compilation flags). The default default_size of
+        # distributed.sizeof.safe_sizeof() is 1MB, which is less than the
+        # OFFLOAD_THRESHOLD.
+        msg_size = safe_sizeof(msg, default_size=-1)
+        if msg_size == -1 or msg_size > OFFLOAD_THRESHOLD:
+            return await offload(_to_frames)
+
+    return _to_frames()
 
 
 async def from_frames(frames, deserialize=True, deserializers=None, allow_offload=True):


Test patch:
diff --git a/distributed/protocol/tests/test_serialize.py b/distributed/protocol/tests/test_serialize.py
--- a/distributed/protocol/tests/test_serialize.py
+++ b/distributed/protocol/tests/test_serialize.py
@@ -446,9 +446,15 @@ def dumps(f):
 
 
 @gen_test()
-async def test_profile_nested_sizeof():
-    # https://github.com/dask/distributed/issues/1674
-    n = 500
+@pytest.mark.parametrize("n", range(100, 600, 50))
+async def test_deeply_nested_structures(n):
+    """sizeof() raises RecursionError at ~140 recursion depth.
+    msgpack doesn't raise until 512 (sometimes 256 depending on compile options).
+    These thresholds change substantially between python versions, msgpack versions, and
+    platforms.
+
+    Test that when sizeof() starts failing, things keep working until msgpack fails.
+    """
     original = outer = {}
     inner = {}
 
@@ -456,8 +462,11 @@ async def test_profile_nested_sizeof():
         outer["children"] = inner
         outer, inner = inner, {}
 
-    msg = {"data": original}
-    frames = await to_frames(msg)
+    try:
+        await to_frames(original)
+    except ValueError as e:
+        # msgpack failed
+        assert "recursion limit exceeded" in str(e)
 
 
 def test_different_compression_families():