Fix repeated calls of check_pairwise and type casting in pairwise_distances_argmin_min

Fixes https://github.com/scikit-learn/scikit-learn/issues/3807

I am able to bring the time down from 313s as seen in this gist (https://gist.github.com/MechCoder/8342cec755d167cc7cad) to 249s as seen in this gist (https://gist.github.com/MechCoder/6c29efb33917c5d97857)