Added duplicate removal code, need to test