Don't return true for sets which have been removed