moved duplicate threshold check to a validation function