Remove keyboard observers more carefully (fixes #763)