Fix focus change on keyboard up.

The scroll to bottom was also moving the focus to the bottom most
element, which is not what was desired.