Use the originally discovered actors to terminate and join

This is to prevent a race while actors are shutting down
Ignore Threads which are being used for a task