I think this removes EVERY listener for that key.  You actually want to find the KeyListener here and then remove just your client from his set of listeners.