Fix race condition in TraitChangeNotifyWrapper.listener_deleted