Let's just do this right now so that there are no gotchas later.

Why do I have to argue for this?
HashSet is already a wrapped Map. This is more performant.
This looks a bit uglier but this is exactly the kind of thing that
an identity map is used for.