Refactored the way sets are managed internally

They now use this hash value which fixes a couple of issues. Still need
to tweak and fix some minor things, but this is a working commit.