Refactor reference set to have map implementation
Decode now removes from reference set