cluster: don't assert if worker has no handles

As it says.