Don't remove elements from a map during iteration.

Doing so relies on implementation details which might change.

See #3911 / SI-8774.

Review by @lrytz; /cc @alextg-github