GroupManager resolves groups with removed entities lazily by removing them from the group upon request of entities in the group.