Only move the camera when there has been both a mouse down and a mouse up.