Fix edge case when List/EnumSet iterator is serialized right after call to remove