Fix position being reset when scrolling with directional keys.