Optimize distance computations for multi-to-multi and single-to-multi

Check distance against 0 only when the minimum distance is updated