validation, duplicate ids removed