Repository: dask/distributed

Issue:
Dask substitutes a string argument with a `Future.result()` 
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**Describe the issue**:

The `main_task` should receive 2 arguments: a string and a dictionary. Instead, it receives 2 dictionaries if a future's key equals the string argument.

**Minimal Complete Verifiable Example**:

```python
from dask.distributed import LocalCluster, Client
from typing import Dict

def main_task(data_name: str, data: Dict):
    return isinstance(data_name, dict) and isinstance(data, dict) and data_name == data

def get_data(data_name: str) -> Dict:
    return {"value": data_name}

cluster = LocalCluster(n_workers=3)
client = Client(cluster.scheduler_address)
data_name = "test_data_name"
f1 = client.submit(
    get_data,
    data_name=data_name,
    key=data_name,
)
f2 = client.submit(
    main_task,
    data_name=data_name,
    data=f1,
)
# The `result` is `True` which is wrong. `data_name` and `data` should have different types and values
result = f2.result()
assert result is True
```

**Anything else we need to know?**:

**Environment**:

- Dask version: 2022.9.2, 2023.1.1
- Python version: 3.9.16
- Operating System: ubuntu:22.04, macos:13.2.1
- Install method (conda, pip, source): pip 



Discussion:
Thanks for the report. I think what you are describing can be avoided by not setting the `key` of `f1` yourself.

This indeed causes some confusion in our internal logic since we are walking the graph searching for occurrences of `key`. Typically `key` is something truly unique, e.g. by using a UUID. I admit this is a bit of a leaky abstraction but this is unfortunately something we will not be able to fix in the short to mid term.

Concretely, I suggest to either not provide an explicit key to `f1` or generate it uniquely. Definitely it should be a different value as the value passed to `data_name`.

e.g. 

```python
import uuid
f1 = client.submit(
    get_data,
    data_name=data_name,
    # Or use time(), a hash, etc. or just not provide the key yourself
    # and we'll take care of this for you
    key=data_name + uuid.uuid4().hex, 
)
```
Thank you for your quick reply. Using different values for a `key` and `data_name` solved my issue. I haven't found documentation for the internal resolution logic. Could this be documented, so other users avoid this issue? Thank you.
> Could this be documented, so other users avoid this issue? Thank you.

No objections. Are you interested in preparing a PR for this?
> > Could this be documented, so other users avoid this issue? Thank you.
> 
> No objections. Are you interested in preparing a PR for this?

Sure. I found a few places where in my opinion this additional information belongs
- https://github.com/dask/distributed/blob/main/distributed/client.py#L1811 - a link to the docs describing an issue
- https://github.com/dask/dask/blob/main/docs/source/futures.rst - an additional paragraph or a warning describing an issue
- https://github.com/dask/dask-examples/blob/main/futures.ipynb - an additional paragraph with an example of code I opened this PR with

Please let me know what you think. Thank you.
> https://github.com/dask/distributed/blob/main/distributed/client.py#L1811 - a link to the docs describing an issue

I think this is most applicable. I'm a bit hesitant of adding this to the examples or high level documentation.

This is not necessarily an intended feature and I hope we can get rid of it (see also https://github.com/dask/dask/issues/9969). I would suggest to add a note/warning to Client.submit, Client.map and Client.scatter.

Thank you. I'm planning to open a PR by the end of this week.

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -1843,6 +1843,15 @@ def submit(
         --------
         >>> c = client.submit(add, a, b)  # doctest: +SKIP
 
+        Notes
+        -----
+        The current implementation of a task graph resolution searches for occurrences of ``key``
+        and replaces it with a corresponding ``Future`` result. That can lead to unwanted
+        substitution of strings passed as arguments to a task if these strings match some ``key``
+        that already exists on a cluster. To avoid these situations it is required to use unique
+        values if a ``key`` is set manually.
+        See https://github.com/dask/dask/issues/9969 to track progress on resolving this issue.
+
         Returns
         -------
         Future
@@ -1984,6 +1993,15 @@ def map(
         --------
         >>> L = client.map(func, sequence)  # doctest: +SKIP
 
+        Notes
+        -----
+        The current implementation of a task graph resolution searches for occurrences of ``key``
+        and replaces it with a corresponding ``Future`` result. That can lead to unwanted
+        substitution of strings passed as arguments to a task if these strings match some ``key``
+        that already exists on a cluster. To avoid these situations it is required to use unique
+        values if a ``key`` is set manually.
+        See https://github.com/dask/dask/issues/9969 to track progress on resolving this issue.
+
         Returns
         -------
         List, iterator, or Queue of futures, depending on the type of the
@@ -2501,6 +2519,16 @@ def scatter(
         >>> data = c.scatter(data, broadcast=True)  # doctest: +SKIP
         >>> res = [c.submit(func, data, i) for i in range(100)]
 
+        Notes
+        -----
+        Scattering a dictionary uses ``dict`` keys to create ``Future`` keys.
+        The current implementation of a task graph resolution searches for occurrences of ``key``
+        and replaces it with a corresponding ``Future`` result. That can lead to unwanted
+        substitution of strings passed as arguments to a task if these strings match some ``key``
+        that already exists on a cluster. To avoid these situations it is required to use unique
+        values if a ``key`` is set manually.
+        See https://github.com/dask/dask/issues/9969 to track progress on resolving this issue.
+
         See Also
         --------
         Client.gather : Gather data back to local process


Test patch: