Don't remove new runners in old jobs.

Fixes #91