de-dupe as part of sanitizing