Added dedup funciton and a test for it