On remove, we need to delete the id from the set, else it can never be
added again