Fix access of iterators after they have been invalidated

Iterators get invalidated after elements of std::vector/set are
deleted. Avoid this by remembering which elements need to be
deleted and do it after iterating over the data structure.