Repository: dask/distributed

Issue:
Worker does not execute task after cancelling future and using resources
The minimal working example provided below simulates a long-running task that is cancelled at some point. As mentioned in this [SO post](https://stackoverflow.com/questions/49203128/how-do-i-stop-a-running-task-in-dask), I expect the function itself to keep on running, i.e. keeping the memory block allocated and printing its size.

However, even when waiting until the function itself is ended (time.wait(50)), I cannot run the function a second time on the same worker. It seems that the reason is that the worker is still marked as having all [resources](https://distributed.readthedocs.io/en/latest/resources.html) consumed (although bokeh reports the resource NOT to be consumed). This can be verified by increasing the resources from 1 to 2. In that case, the function is run.

Other scenarios where the function is successfully executed include:
- The task is not cancelled, but successfully finished.
- The worker is stopped using ```lc.stop_worker(w)``` and restarted using ```w_new = lc.start_worker(ncores=1, memory_limit=0, resources={"computer": 1})```

In the beginning, I thought it has something to do with using [Variables](https://distributed.readthedocs.io/en/latest/api.html#distributed.Variable). However, the problem persists when omitting Variables entirely (as seen in the example below).

```
from distributed.client import Client
from distributed import LocalCluster
from dask import delayed
import time
import numpy as np

def long_running_task(a, b):
    """
    Some long-running task
    """
    print(a)
    # allocate some data to see that actually something is happening
    l = np.zeros((25000, 25000))
    for i in range(30):
        print(l.nbytes)
        time.sleep(1)
    print(b)

lc = LocalCluster(processes=False, n_workers=1, threads_per_worker=1, scheduler_port=8786,
                          diagnostics_port=8787)
lc.start_worker(ncores=1, memory_limit=0, resources={"computer": 1})

# give the cluster enough time to get up running
time.sleep(10)

# connect to the cluster
c = Client(lc)

lt = delayed(long_running_task)("hi", "ho")
# specify that this computation needs one unit of abstract computer resource
# thus, no resources anymore for that worker
future = c.compute(lt, resources={lt.key:{"computer":1}})

time.sleep(5)

for i in range(3):
    print("Status is {} and done is {} before cancelling".format(future.status, future.done()))
    time.sleep(1)
    
# cancel future
# after cancelling, we expect the function to keep on running as mentioned in
# https://stackoverflow.com/questions/49203128/how-do-i-stop-a-running-task-in-dask
# Thus, the long-running function keeps on printing stuff
# However, the worker does not have it anymore (c.has_what())
c.cancel(future, force=True)
print("cancelled")
for i in range(50):
    print("Status is {} and done is {} after cancelling".format(future.status, future.done()))
    print(c.has_what())
    time.sleep(1)

time.sleep(10)

c.close()

##############################################
# Trying to do another iteration
c_new = Client(lc)
lt = delayed(long_running_task)("he", "ha")
# At this point, I would expect the worker to be free again in terms of
# abstract resources and thus print stuff again (from the long-running function).
# In fact, looking at the worker pages in the bokeh server,
# consumed resources are 0, total=1. However, state does not transition from
# processing to released.
# On the other hand, when specifying 2 units of computer (instead of 1), the
# task is successfully executed.
# The task is also successfully executed when NOT cancelling and just letting 
# the function finish until the future status chanages to finished.
future = c_new.compute(lt, resources={lt.key:{"computer":1}})


for i in range(10):
    print("Status is {} and done is {} before cancelling".format(future.status, future.done()))
    print(c_new.has_what())
    time.sleep(1)
    
time.sleep(5)
```
**Actual Output:**
![bug_report](https://user-images.githubusercontent.com/9250590/42560995-631472b8-84f8-11e8-89d3-468073b107c2.png)

**Expected output:**
Printing _5000000000_ in the end **as well**

**System:**
Win7 Enterprise, Service Pack 1, 64bit

**Python (conda env export -p <<install_path>> -v --json -f <<file>>):**
- name:
- channels:
  - aaronzs
  - conda-forge
  - defaults
- dependencies:
  - tensorflow=1.4.0=py36he4e0f4f_0
  - tensorflow-tensorboard=0.4.0rc3=py36h1eb756b_0
  - bzip2=1.0.6=vc14_1
  - freetype=2.7=vc14_1
  - icu=58.2=vc14_0
  - itsdangerous=0.24=py36_1
  - jpeg=9b=vc14_2
  - libiconv=1.15=vc14_0
  - libpng=1.6.34=vc14_0
  - libprotobuf=3.5.2=vc14_0
  - libtiff=4.0.7=vc14_1
  - libxml2=2.9.8=vc14_0
  - libxslt=1.1.32=vc14_0
  - marshmallow=2.13.5=py36_0
  - matplotlib=1.5.3=np113py36_8
  - multiprocess=0.70.5=py36_0
  - openssl=1.0.2o=vc14_0
  - pathos=0.2.1=py36_1
  - pillow=4.3.0=py36_1
  - pox=0.2.3=py36_0
  - ppft=1.6.4.7.1=py36_0
  - protobuf=3.5.2=py36_vc14_0
  - pydensecrf=1.0rc3=py36_0
  - pyside=1.2.4=py36_vc14_8
  - qt=4.8.7=vc14_7
  - tk=8.6.7=vc14_0
  - yaml=0.1.7=vc14_0
  - zlib=1.2.11=vc14_0
  - attrs=18.1.0=py36_0
  - bleach=1.5.0=py36_0
  - bokeh=0.12.16=py36_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py36_0
  - click=6.7=py36hec8c647_0
  - cloudpickle=0.5.3=py36_0
  - colorama=0.3.9=py36h029ae33_0
  - cycler=0.10.0=py36h009560c_0
  - cython=0.28.2=py36hfa6e2cd_0
  - cytoolz=0.9.0.1=py36hfa6e2cd_0
  - dask=0.17.4=py36_0
  - dask-core=0.17.4=py36_0
  - decorator=4.3.0=py36_0
  - dill=0.2.7.1=py36hf552773_0
  - distributed=1.21.8=py36_0
  - flask=1.0.2=py36_1
  - heapdict=1.0.0=py36_2
  - html5lib=0.9999999=py36_0
  - icc_rt=2017.0.4=h97af966_0
  - imageio=2.3.0=py36_0
  - intel-openmp=2018.0.0=8
  - jinja2=2.10=py36h292fed1_0
  - joblib=0.11=py36_0
  - locket=0.2.0=py36hfed976d_1
  - markdown=2.6.11=py36_0
  - markupsafe=1.0=py36h0e26971_1
  - mkl=2018.0.2=1
  - mkl_fft=1.0.1=py36h452e1ab_0
  - mkl_random=1.0.1=py36h9258bd6_0
  - more-itertools=4.1.0=py36_0
  - msgpack-python=0.5.6=py36he980bc4_0
  - networkx=2.1=py36_0
  - numpy=1.13.3=py36h5c71026_4
  - olefile=0.45.1=py36_0
  - packaging=17.1=py36_0
  - pandas=0.22.0=py36h6538335_0
  - partd=0.3.8=py36hc8e763b_0
  - pip=10.0.1=py36_0
  - pluggy=0.6.0=py36hc7daf1e_0
  - psutil=5.4.5=py36hfa6e2cd_0
  - py=1.5.3=py36_0
  - pyparsing=2.2.0=py36h785a196_1
  - pyreadline=2.1=py36h094d952_1
  - pytest=3.5.1=py36_0
  - python=3.6.3=h3b118a2_4
  - python-dateutil=2.7.3=py36_0
  - pytz=2018.4=py36_0
  - pywavelets=0.5.2=py36hc649158_0
  - pyyaml=3.12=py36h1d1928f_1
  - scikit-image=0.13.1=py36hfa6e2cd_1
  - scikit-learn=0.19.1=py36h53aea1b_0
  - scipy=1.1.0=py36h672f292_0
  - setuptools=39.1.0=py36_0
  - sip=4.18=py36_0
  - six=1.11.0=py36h4db2310_1
  - sortedcontainers=1.5.10=py36_0
  - sqlalchemy=1.2.7=py36ha85dd04_0
  - tblib=1.3.2=py36h30f5020_0
  - toolz=0.9.0=py36_0
  - tornado=5.0.2=py36_0
  - vc=14=h0510ff6_3
  - vs2015_runtime=14.0.25123=3
  - waitress=1.1.0=py36h7181276_0
  - werkzeug=0.14.1=py36_0
  - wheel=0.31.1=py36_0
  - wincertstore=0.2=py36h7fe50ca_0
  - zict=0.1.3=py36h2d8e73e_0
  - pip:
    - enum34==1.1.6

Any help/hints/workarounds are highly appreciated.

Please also let me know if you need any further information. I am more than happy to help identifying the problem.

Best regards
Felix



Discussion:
Related to the comment above: Is there any way to determine whether the actual function is still running? The status of the future seems to be entirely invariant w.r.t. whether the computation has finished or not. This is most likely related to this [SO post](https://stackoverflow.com/questions/49203128/how-do-i-stop-a-running-task-in-dask), right?

Resolving patch:
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -2033,6 +2033,9 @@ def release_key(self, key, cause=None, reason=None, report=True):
                 self.executing.remove(key)
 
             if key in self.resource_restrictions:
+                if state == 'executing':
+                    for resource, quantity in self.resource_restrictions[key].items():
+                        self.available_resources[resource] += quantity
                 del self.resource_restrictions[key]
 
             if report and state in PROCESSING:  # not finished


Test patch:
diff --git a/distributed/tests/test_scheduler.py b/distributed/tests/test_scheduler.py
--- a/distributed/tests/test_scheduler.py
+++ b/distributed/tests/test_scheduler.py
@@ -6,6 +6,7 @@
 import json
 from operator import add, mul
 import sys
+from time import sleep
 
 import dask
 from dask import delayed
@@ -1340,3 +1341,22 @@ def test_closing_scheduler_closes_workers(s, a, b):
     while a.status != 'closed' or b.status != 'closed':
         yield gen.sleep(0.01)
         assert time() < start + 2
+
+
+@gen_cluster(client=True, ncores=[('127.0.0.1', 1)],
+             worker_kwargs={'resources': {'A': 1}})
+def test_resources_reset_after_cancelled_task(c, s, w):
+    future = c.submit(sleep, 0.2, resources={'A': 1})
+
+    while not w.executing:
+        yield gen.sleep(0.01)
+
+    yield future.cancel()
+
+    while w.executing:
+        yield gen.sleep(0.01)
+
+    assert not s.workers[w.address].used_resources['A']
+    assert w.available_resources == {'A': 1}
+
+    yield c.submit(inc, 1, resources={'A': 1})