Also we should remove the keyListener map if the client is null so we don't do extra work.