Repository: dask/dask

Issue:
`dask_key_name`s with hyphens can get altered
**What happened**: If one uses `bind` with `delayed(... dask_key_name)`, the result of `dask.distributed.get_worker().get_current_task()` will not necessarily be the `dask_key_name` provided by the user. I noticed the problem when using ISO 8601 formatted dates as part of the `dask_key_name`. 

**What you expected to happen**: I would expect `dask.distributed.get_worker().get_current_task()` to return the `dask_key_name` provided by the user.

**Minimal Complete Verifiable Example**:

I would have expected to see `child-2000-01-01` rather than `child-9431b3ddf5f3dcd214bf0d080e3c0dee`.

```python

$ cat unexpected_task_key.py 
import dask
import dask.distributed
import dask.graph_manipulation

def f():
    dask.distributed.print(dask.distributed.get_worker().get_current_task())

if __name__ == '__main__':
    cluster = dask.distributed.LocalCluster(n_workers=2, threads_per_worker=8)
    client = dask.distributed.Client(cluster)
    parent = dask.delayed(f)(dask_key_name="parent-1999-12-31")
    child = dask.delayed(f)(dask_key_name="child-2000-01-01")
    child = dask.graph_manipulation.bind(child, parents)
    result = child.compute()

$ venv/bin/python unexpected_task_key.py 
parent-1999-12-31
parent-1999-12-31
child-9431b3ddf5f3dcd214bf0d080e3c0dee
child-9431b3ddf5f3dcd214bf0d080e3c0dee

$ venv/bin/pip list | grep dask         
dask              2022.7.1

$ venv/bin/python --version    
Python 3.8.9

$ sw_vers
ProductName:	macOS
ProductVersion:	11.6.7
BuildVersion:	20G630
```

**Anything else we need to know?**: I suspect the problem is in either `dask.base.key_split` or `dask.base.clone_key`.

**Environment**:

- Dask version: `2022.7.1`
- Python version: `3.8.9`
- Operating System: macOS 11.6.7 
- Install method (conda, pip, source): `pip`



Discussion:
@andrewbarisser Welcome! I think this might be intentional because [the docstring for `bind` says](https://docs.dask.org/en/stable/graph_manipulation.html#dask.graph_manipulation.bind):

> All keys of children will be regenerated, up to and excluding the keys of omit.

@ncclementi what do you think?
Thanks for pointing out the docstring. Are there rules for how one should expect keys to be regenerated? Based on https://github.com/dask/dask/issues/8783, I would think that at least `child-2000-01-01` would be part of the new key. It seems that hyphens play a special role in the key names but that isn't obvious from the documentation. 

Also (and this is probably a much bigger issue), it isn't clear why one would need to change the key in the first place. The documentation makes it sound like the point of `bind` is to add edges from the `parents` to the `children` in the underlying task graph -- it doesn't make sense that it would create vertices with new attributes.
@andrewbarisser Ah, I see what you mean.

I think you're absolutely right that the root cause is `key_split`. However, `key_split` seems to be intentionally trimming any numbers after hyphens. I agree that we can document this better.

https://github.com/dask/dask/blob/e61405cb5d345e73f1952ee3d50708566b5263d1/dask/utils.py#L1804-L1830

@crusaderky Do you have thoughts on this? Especially about the broader question around why `bind` needs to regenerate the keys of `children`?

Resolving patch:
diff --git a/dask/graph_manipulation.py b/dask/graph_manipulation.py
--- a/dask/graph_manipulation.py
+++ b/dask/graph_manipulation.py
@@ -271,10 +271,12 @@ def bind(
     -------
     Same as ``children``
         Dask collection or structure of dask collection equivalent to ``children``,
-        which compute to the same values. All keys of ``children`` will be regenerated,
-        up to and excluding the keys of ``omit``. Nodes immediately above ``omit``, or
+        which compute to the same values. All nodes of ``children`` will be regenerated,
+        up to and excluding the nodes of ``omit``. Nodes immediately above ``omit``, or
         the leaf nodes if the collections in ``omit`` are not found, are prevented from
         computing until all collections in ``parents`` have been fully computed.
+        The keys of the regenerated nodes will be different from the original ones, so
+        that they can be used within the same graph.
     """
     if seed is None:
         seed = uuid.uuid4().bytes
@@ -429,6 +431,17 @@ def clone(*collections, omit=None, seed: Hashable = None, assume_layers: bool =
      ('add-5', 0): (<function operator.add>, ('add-4', 0), 1),
      ('add-5', 1): (<function operator.add>, ('add-4', 1), 1)}
 
+    The typical usage pattern for clone() is the following:
+
+    >>> x = cheap_computation_with_large_output()  # doctest: +SKIP
+    >>> y = expensive_and_long_computation(x)  # doctest: +SKIP
+    >>> z = wrap_up(clone(x), y)  # doctest: +SKIP
+
+    In the above code, the chunks of x will be forgotten as soon as they are consumed by
+    the chunks of y, and then they'll be regenerated from scratch at the very end of the
+    computation. Without clone(), x would only be computed once and then kept in memory
+    throughout the whole computation of y, needlessly consuming memory.
+
     Parameters
     ----------
     collections
@@ -446,6 +459,8 @@ def clone(*collections, omit=None, seed: Hashable = None, assume_layers: bool =
         Dask collections of the same type as the inputs, which compute to the same
         value, or nested structures equivalent to the inputs, where the original
         collections have been replaced.
+        The keys of the regenerated nodes in the new collections will be different from
+        the original ones, so that they can be used within the same graph.
     """
     out = bind(
         collections, parents=None, omit=omit, seed=seed, assume_layers=assume_layers
@@ -489,6 +504,8 @@ def wait_on(
         Dask collection of the same type as the input, which computes to the same value,
         or a nested structure equivalent to the input where the original collections
         have been replaced.
+        The keys of the regenerated nodes of the new collections will be different from
+        the original ones, so that they can be used within the same graph.
     """
     blocker = checkpoint(*collections, split_every=split_every)
 


Test patch: