Fix keyboard buffer handling when we're not in a term_read