Used a weakref.WeakSet just to stay on the safe side against memory leaks