Make optimization more deterministic

This was discovered in another PR, but I want to merge it separatly to make sure the change to OrderedSet() are well done.