Take mouse down offset into account for better user experience.