Add map protection to maintainWorkers

Since terminate can mutate the map, it's unsafe to be accessing it at
the same time. This defers the terminations until we're done with the
map.