Delete groups when they do not exist in the perm_spec.