Optimizations

This change does not change the hash code for elements in this set, it's simply a performance optimization. If you want to discuss this I feel it is best opened as a new issue.