Take mouse down offset into account for better user experience.

Fix #23