Fix broken logic

The logic for handling task removal by reload was not merged in
correctly.