Use the originally discovered actors to terminate and join

This is to prevent a race while actors are shutting down