Remove items from sets when the set data changes.
Don't trigger unneeded property change notifications.
Keep the set objects around so the observers keep working.