add dupe tests, and dupe checking code