Fix bug noticed on codereview

The solution is, when cleaning up, to _not_ clean up tasks which are not yet
done, one way or another (success, failure or cancellation).