Improve panning edge detection

If, for example, you are panning, and the horizontal boundary is
reached, but you can still pan vertically, keep going vertically.