To keep the api consistent, this should return an immutable Set.