Remove and reattache listeners on attached and detached callbacks

Fixes #1 Key listener leaks, should probably use attached and detached