Do not start a pan when clicking the sky, but continue to pan when terrain was clicked and mouse moves over the sky.