I think this is handled differently in sklearn.metrics:

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/metrics.py#L1484