core: refactoring the way sets work internally v2

This is a refactored solution for PR #616. Functionally this is still the same change, but it’s implemented a lot cleaner with less code and less changes to existing parts of TF.
There is test coverage for this I'm guessing? So we can safely try to refactor this later?