Repository: dask/distributed

Issue:
Error removing task group
Believe this is related to #3322: running on master since pulling in that PR, I'm suddenly seeing a lot of these errors:
```
distributed.scheduler - ERROR - Error transitioning '_plot_hourly_scatterplots-e923799215afb4e224b3d5672cdf8be3' from 'processing' to 'memory'
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/distributed/scheduler.py", line 4658, in transition
    ts.prefix.groups.remove(tg)
ValueError: list.remove(x): x not in list
distributed.core - ERROR - list.remove(x): x not in list
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/distributed/core.py", line 473, in handle_stream
    handler(**merge(extra, msg))
  File "/usr/local/lib/python3.7/site-packages/distributed/scheduler.py", line 2600, in handle_task_finished
    r = self.stimulus_task_finished(key=key, worker=worker, **msg)
  File "/usr/local/lib/python3.7/site-packages/distributed/scheduler.py", line 2015, in stimulus_task_finished
    recommendations = self.transition(key, "memory", worker=worker, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/distributed/scheduler.py", line 4658, in transition
    ts.prefix.groups.remove(tg)
ValueError: list.remove(x): x not in list
```
Played around a bit with trying to simply reproduce but no luck so far; some possible culprits are:
- many of these tasks are stolen
- some workers could be starting/stopping during processing
- I believe the tasks that are failing are being `fire_and_forget`ten 

@jrbourbeau anything stand out to you...? if it's possible for `ts.group.states` and `ts.prefix.groups` to be inconsistent then maybe there should be another `if` or `try/pass` here regardless?
fire_and_forget scheduler error in 2.11.0
Hi,

Discovered what looks like a bug with fire_and_forget in the scheduler in 2.11.0. The error doesn't exist in 2.10.1. 

**Repro**

Running a 1-worker cluster like:
```
docker run -it --network host daskdev/dask:2.11.0 dask-scheduler
docker run -it --network host daskdev/dask:2.11.0 dask-worker localhost:8786
```

I run a python shell:
```
virtualenv .venv
source .venv/bin/activate

pip install dask distributed
python
```

Using the small dask.distributed example:
```
import time
import random
import dask.distributed
import logging

 
def inc(x):
    logging.warning('blah: x')
    time.sleep(random.random())
    return x + 1


def dec(x):
    logging.warning('blah: y')
    time.sleep(random.random())
    return x - 1


def add(x, y):
    logging.warning('blah: z')
    time.sleep(random.random())
    return x + y


def repro():
    with dask.config.set({"distributed.comm.compression": "lz4"}):
        client = dask.distributed.Client("tcp://localhost:8786")
        f = client.get({
            'x': (inc, 1),
            'y': (dec, 2),
            'z': (add, 'x', 'y')
        }, 'z', sync=False)
        dask.distributed.fire_and_forget(f)


repro()
```

I get this error in the scheduler:
```
distributed.scheduler - INFO - Plugin failed with exception
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 4645, in transition
    plugin.transition(key, start, finish2, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/distributed/diagnostics/graph_layout.py", line 93, in transition
    self.visible_updates.append((self.index[key], "False"))
KeyError: 'z'
distributed.scheduler - ERROR - Error transitioning 'z' from 'processing' to 'memory'
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 4655, in transition
    ts.prefix.groups.remove(tg)
ValueError: list.remove(x): x not in list
distributed.core - ERROR - list.remove(x): x not in list
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/distributed/core.py", line 473, in handle_stream
    handler(**merge(extra, msg))
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 2597, in handle_task_finished
    r = self.stimulus_task_finished(key=key, worker=worker, **msg)
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 2015, in stimulus_task_finished
    recommendations = self.transition(key, "memory", worker=worker, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 4655, in transition
    ts.prefix.groups.remove(tg)
ValueError: list.remove(x): x not in list
distributed.scheduler - INFO - Remove worker <Worker 'tcp://127.0.0.1:36953', name: tcp://127.0.0.1:36953, memory: 0, processing: 0>
distributed.core - INFO - Removing comms to tcp://127.0.0.1:36953
distributed.scheduler - INFO - Lost all workers
distributed.utils - ERROR - list.remove(x): x not in list
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/distributed/utils.py", line 665, in log_errors
    yield
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 1739, in add_worker
    await self.handle_worker(comm=comm, worker=address)
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 2694, in handle_worker
    await self.handle_stream(comm=comm, extra={"worker": worker})
  File "/opt/conda/lib/python3.7/site-packages/distributed/core.py", line 473, in handle_stream
    handler(**merge(extra, msg))
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 2597, in handle_task_finished
    r = self.stimulus_task_finished(key=key, worker=worker, **msg)
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 2015, in stimulus_task_finished
    recommendations = self.transition(key, "memory", worker=worker, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 4655, in transition
    ts.prefix.groups.remove(tg)
ValueError: list.remove(x): x not in list
distributed.core - ERROR - list.remove(x): x not in list
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/distributed/core.py", line 412, in handle_comm
    result = await result
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 1739, in add_worker
    await self.handle_worker(comm=comm, worker=address)
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 2694, in handle_worker
    await self.handle_stream(comm=comm, extra={"worker": worker})
  File "/opt/conda/lib/python3.7/site-packages/distributed/core.py", line 473, in handle_stream
    handler(**merge(extra, msg))
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 2597, in handle_task_finished
    r = self.stimulus_task_finished(key=key, worker=worker, **msg)
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 2015, in stimulus_task_finished
    recommendations = self.transition(key, "memory", worker=worker, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 4655, in transition
    ts.prefix.groups.remove(tg)
ValueError: list.remove(x): x not in list
distributed.scheduler - INFO - Register worker <Worker 'tcp://127.0.0.1:36953', name: tcp://127.0.0.1:36953, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:36953
distributed.core - INFO - Starting established connection
```

All of the functions complete on the worker (I see the output printed). However, once the exception is thrown, the next call to `repro` does not get executed. A third call to `repro` does get executed and again throws the exception.

Running the exact same code against 2.10.1 scheduler and worker runs without the exception and behaves as expected.


Thanks,

Mark


Discussion:
Thanks for reporting this @bnaul! Your usage of the dev version of `dask` and `distributed` has been valuable to the project on several occasions

The failure definitely looks related to #3322. After a quick glance at some relevant portions of the codebase, I don't see where `ts.group.states` and `ts.prefix.groups` would get out of sync. I'll take a deeper look tomorrow. If you're able to provide a reproducer, that would be awesome 
Glad to hear it 🙂

I haven't been able to reproduce so far but I do suspect stealing is related; it does seem like https://github.com/dask/distributed/pull/3322/files#diff-6a3108098afcddf88b38652ae1e383b5R4611-R4612 would definitely fail in that way if it were run twice, and couldn't that be a possible race condition involved with stealing? For now I'm trying to work around this with
```
                 if not any(tg.states.get(s) for s in ALL_TASK_STATES):
-                    ts.prefix.groups.remove(tg)
-                    del self.task_groups[tg.name]
+                    try:
+                        ts.prefix.groups.remove(tg)
+                        del self.task_groups[tg.name]
+                    except (KeyError, ValueError):
+                        pass
```
any reason the above might be undesirable?
Just checking in, does anyone have any thoughts about this bug? I am pretty confident it's going to be a problem for people once #3322 is released so it seems important to fix it before then; not affecting me personally bc I am already using the workaround I posted above but it would be nice to move back off our fork 🙂 

cc @jrbourbeau @TomAugspurger 
Thanks for following up @bnaul! Do you know why work stealing would cause that section of code to run multiple times? A `try`/`except` catch seems reasonable if it's difficult to avoid, e.g. a race condition with work stealing, but I just want to make sure we're fixing the problem if possible and not just working around a side effect
Unfortunately all I really have are guesses; work stealing is the only source of a race condition that comes to mind...if it helps I could rerun my workflow w/o work stealing and see if that avoids the issue, but I don't know that I'll be able to come up with a proper reproducible example since the only times I've seen it have been pretty big jobs.
Rerunning without work stealing would be great if you don't mind 
@jrbourbeau managed to isolate the issue! it is related to fire-and-forget + work stealing:
```
import distributed
from distributed.utils_test import slowadd
c = distributed.Client()
x = c.scatter(list(range(100)), workers=list(c.scheduler_info()['workers'].keys())[0])
# same error does not appear w/o fire-and-forget
distributed.fire_and_forget([c.submit(slowadd, i, x[i]) for i in range(len(x))])


distributed.core - ERROR - list.remove(x): x not in list
Traceback (most recent call last):
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/core.py", line 412, in handle_comm
    result = await result
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/scheduler.py", line 1739, in add_worker
    await self.handle_worker(comm=comm, worker=address)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/scheduler.py", line 2694, in handle_worker
    await self.handle_stream(comm=comm, extra={"worker": worker})
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/core.py", line 473, in handle_stream
    handler(**merge(extra, msg))
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/scheduler.py", line 2597, in handle_task_finished
    r = self.stimulus_task_finished(key=key, worker=worker, **msg)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/scheduler.py", line 2015, in stimulus_task_finished
    recommendations = self.transition(key, "memory", worker=worker, **kwargs)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/scheduler.py", line 4655, in transition
    ts.prefix.groups.remove(tg)
ValueError: list.remove(x): x not in list
```
this is happening for me in the new 2.11.0; the workaround I posted above fixes the issue, not sure if there's a better way of handling this though.
Just checking in, does the example above reproduce the issue for others? Happy to PR our workaround but I figure there's probably a better way to properly avoid this that @jrbourbeau or @mrocklin could come up with 🙂
As a disclaimer, most of the people who could handle this issue are
currently as the Dask summit (we're missing you @bnaul) so it may be that
nothing happens here for a few days.

On Thu, Feb 27, 2020 at 8:41 AM Brett Naul <notifications@github.com> wrote:

> Just checking in, does the example above reproduce the issue for others?
> Happy to PR our workaround but I figure there's probably a better way to
> properly avoid this that @jrbourbeau <https://github.com/jrbourbeau> or
> @mrocklin <https://github.com/mrocklin> could come up with 🙂
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/3465?email_source=notifications&email_token=AACKZTAR567WHBFUO3J6YE3RE7ULLA5CNFSM4KSWZJQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENFBH2Y#issuecomment-592057323>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTFFQ6ARZ2M3I75WYHDRE7ULLANCNFSM4KSWZJQA>
> .
>

Thanks for the report. I'm not able to reproduce on distributed master (though I'm just using local processes, not docker. Is docker necessary?). Can you try with master?
Thanks for opening up an issue @mgh35! I think this is a duplicate of #3465. I'm working on a fix for it now

Resolving patch:
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -4649,7 +4649,7 @@ def transition(self, key, finish, *args, **kwargs):
                 if ts.state == "forgotten":
                     del self.tasks[ts.key]
 
-            if ts.state == "forgotten":
+            if ts.state == "forgotten" and ts.group.name in self.task_groups:
                 # Remove TaskGroup if all tasks are in the forgotten state
                 tg = ts.group
                 if not any(tg.states.get(s) for s in ALL_TASK_STATES):


Test patch:
diff --git a/distributed/tests/test_scheduler.py b/distributed/tests/test_scheduler.py
--- a/distributed/tests/test_scheduler.py
+++ b/distributed/tests/test_scheduler.py
@@ -1837,6 +1837,17 @@ async def test_task_unique_groups(c, s, a, b):
     assert s.task_prefixes["sum"].states["memory"] == 2
 
 
+@gen_cluster(client=True)
+async def test_task_group_on_fire_and_forget(c, s, a, b):
+    # Regression test for https://github.com/dask/distributed/issues/3465
+    with captured_logger("distributed.scheduler") as logs:
+        x = await c.scatter(list(range(10)))
+        fire_and_forget([c.submit(slowadd, i, x[i]) for i in range(len(x))])
+        await asyncio.sleep(1)
+
+    assert "Error transitioning" not in logs.getvalue()
+
+
 class BrokenComm(Comm):
     peer_address = None
     local_address = None