What about using a [`MutableSet`](http://static.rust-lang.org/doc/master/std/container/trait.MutableSet.html) here?