It seems like either `sklearn.metrics.pairwise.pairwise_distances` or the distance functions handle this. I removed the check and added a test case.