Put candidates into a set as we collect them

Removes the need to reduplicate after the fact.