Remove fields that are attached via, vsite_attach_fields_to_group()

#1974