Don't pan if you're not clicking on the canvas