cluster: don't assert if worker has no handles

As it says.
Do we need the else block here?