Using a set seems unnecessarily expensive. Removing an observer will probably be a fairly rare operation – an array might be better.