core: refactoring the way sets work internally v2

This is a refactored solution for PR #616. Functionally this is still the same change, but it’s implemented a lot cleaner with less code and less changes to existing parts of TF.