Don't actually start the zoom until a drag is detected, but do save the coordinates from the initial mouse press. This can do two things, one is allow clicks without zooming, and on devices without absolute precision like touchscreens, prevents accidental "drags" when it was just a press.