Repository: dask/distributed

Issue:
Issue with dask.distributed.performance-report when used twice
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:
If client.distributed.performance_report is invoked twice in a jupyter notebook, the user encounters a key error very similar to one that got fixed here https://github.com/dask/distributed/issues/3858. Seems like a very similar issue in how values are fetched from the map in the function task_stream.py.bisect().

**What you expected to happen**:

**Minimal Complete Verifiable Example**:
Steps to replicate 
Run the following lines of code in a python notebook(.ipynb) twice. After the first time, restart the notebook

```python

from dask.distributed import Client, performance_report
from datetime import datetime

def get_square(num):
	return num**2

if __name__ == "__main__":
    service_port = os.environ['DASK_SCHEDULER_SERVICE_PORT'] 
    service_host = os.environ['DASK_SCHEDULER_SERVICE_HOST']

    client = Client(address=f'{service_host}:{service_port}', direct_to_workers=True) // Connect to the remote dask cluster

    with performance_report(filename=f"/mnt/results/dask-report_{str(datetime.now())}.html"):
    	dask_map = client.map(get_square, [1,2,3,4,5])
    	print(client.gather(dask_map))
```

**Anything else we need to know?**:
Error reported 
```python

/opt/conda/lib/python3.8/site-packages/distributed/scheduler.py in get_task_stream()
   6570         self.add_plugin(TaskStreamPlugin, idempotent=True)
   6571         tsp = [p for p in self.plugins if isinstance(p, TaskStreamPlugin)][0]
-> 6572         return tsp.collect(start=start, stop=stop, count=count)
   6573 
   6574     def start_task_metadata(self, comm=None, name=None):

/opt/conda/lib/python3.8/site-packages/distributed/diagnostics/task_stream.py in collect()
     55             start = time() - parse_timedelta(start)
     56         if start is not None:
---> 57             start = bisect(start, 0, len(self.buffer))
     58 
     59         if isinstance(stop, str):

/opt/conda/lib/python3.8/site-packages/distributed/diagnostics/task_stream.py in bisect()
     44             mid = (left + right) // 2
     45             value = max(
---> 46                 startstop["stop"] for startstop in self.buffer[mid]["startstops"]
     47             )
     48 

KeyError: 'startstops'

```
**Environment**:

- Dask version:dask==2021.06.0 
	dask distributed==2021.06.0
- Python version: 3.8
- Operating System:
- Install method (conda, pip, source):



Discussion:
Thanks for raising an issue @chittshota. Unfortunately I'm not able to reproduce with the 2021.06.0 release or the latest dev version of `dask` and `distributed`.

I created a local cluster by running `$ dask-scheduler` in one terminal tab and `dask-worker localhost:8786` in another tab. I then ran 

```python
from dask.distributed import Client, performance_report

def get_square(num):
    return num**2

client = Client("tcp://localhost:8786", direct_to_workers=True)

with performance_report():
    dask_map = client.map(get_square, [1,2,3,4,5])
    print(client.gather(dask_map))
```

in a JupyterLab notebook twice (restarting the kernel between runs) and everything run successfully -- I didn't encounter the `KeyError` you're seeing.

You might try going through the steps I outlined above to see if that replicated the `KeyError` on your system 

Resolving patch:
diff --git a/distributed/diagnostics/task_stream.py b/distributed/diagnostics/task_stream.py
--- a/distributed/diagnostics/task_stream.py
+++ b/distributed/diagnostics/task_stream.py
@@ -33,6 +33,9 @@ def transition(self, key, start, finish, *args, **kwargs):
         if start == "processing":
             if key not in self.scheduler.tasks:
                 return
+            if not kwargs.get("startstops"):
+                # Other methods require `kwargs` to have a non-empty list of `startstops`
+                return
             kwargs["key"] = key
             if finish == "memory" or finish == "erred":
                 self.buffer.append(kwargs)


Test patch:
diff --git a/distributed/diagnostics/tests/test_task_stream.py b/distributed/diagnostics/tests/test_task_stream.py
--- a/distributed/diagnostics/tests/test_task_stream.py
+++ b/distributed/diagnostics/tests/test_task_stream.py
@@ -83,6 +83,29 @@ async def test_collect(c, s, a, b):
     assert tasks.collect(start=start, count=3) == list(tasks.buffer)[:3]
 
 
+@gen_cluster(client=True)
+async def test_no_startstops(c, s, a, b):
+    tasks = TaskStreamPlugin(s)
+    s.add_plugin(tasks)
+    # just to create the key on the scheduler
+    future = c.submit(inc, 1)
+    await wait(future)
+    assert len(tasks.buffer) == 1
+
+    tasks.transition(future.key, "processing", "erred")
+    # Transition was not recorded because it didn't contain `startstops`
+    assert len(tasks.buffer) == 1
+
+    tasks.transition(future.key, "processing", "erred", startstops=[])
+    # Transition was not recorded because `startstops` was empty
+    assert len(tasks.buffer) == 1
+
+    tasks.transition(
+        future.key, "processing", "erred", startstops=[dict(start=time(), stop=time())]
+    )
+    assert len(tasks.buffer) == 2
+
+
 @gen_cluster(client=True)
 async def test_client(c, s, a, b):
     L = await c.get_task_stream()