Fix race condition when removing a traits listener.

Fixes #56.