Only check grabbable area of the relevant border (i.e. the one we are panning away from)