Repository: scikit-learn/scikit-learn
Pull request: [MRG+1] Adds multilabels permutation tests to metrics/test_common
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
Fixes #12309 for multilabel and multioutput


#### What does this implement/fix? Explain your changes.
Adds test to check that multilabel and multioutput metrics are invariant under label permutations. 


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

Problem statement:
add multiclass and multilabel metric tests for label permutations
Multiclass and multilabel metrics should be invariant to labeling the columns.
That should be tested in ``metrics/test_common.py``.
I haven't checked if there's a test for standard multiclass, but there's none for thresholded metrics.

Hints from issue discussion:

Commit message: RFC: Removes unneeded code

Patch to review:
diff --git a/sklearn/metrics/tests/test_common.py b/sklearn/metrics/tests/test_common.py
index 7c393f81ce10f..6111c785bb5c5 100644
--- a/sklearn/metrics/tests/test_common.py
+++ b/sklearn/metrics/tests/test_common.py
@@ -3,6 +3,7 @@
 from functools import partial
 from itertools import product
 from itertools import chain
+from itertools import permutations
 
 import numpy as np
 import scipy.sparse as sp
@@ -17,6 +18,7 @@
 from sklearn.utils import shuffle
 
 from sklearn.utils.testing import assert_allclose
+from sklearn.utils.testing import assert_almost_equal
 from sklearn.utils.testing import assert_array_equal
 from sklearn.utils.testing import assert_array_less
 from sklearn.utils.testing import assert_equal
@@ -1201,3 +1203,53 @@ def test_no_averaging_labels():
             score_labels = metric(y_true, y_pred, labels=labels, average=None)
             score = metric(y_true, y_pred, average=None)
             assert_array_equal(score_labels, score[inverse_labels])
+
+
+@pytest.mark.parametrize(
+    'name', MULTILABELS_METRICS - NOT_SYMMETRIC_METRICS)
+def test_multilabel_label_permutations_invariance(name):
+    random_state = check_random_state(0)
+    n_samples, n_classes = 20, 4
+
+    y_true = random_state.randint(0, 2, size=(n_samples, n_classes))
+    y_score = random_state.randint(0, 2, size=(n_samples, n_classes))
+
+    metric = ALL_METRICS[name]
+    score = metric(y_true, y_score)
+
+    for perm in permutations(range(n_classes), n_classes):
+        inv_perm = np.zeros(n_classes, dtype=int)
+        inv_perm[list(perm)] = np.arange(n_classes)
+
+        y_score_perm = y_score[:, inv_perm]
+        y_true_perm = y_true[:, inv_perm]
+
+        current_score = metric(y_true_perm, y_score_perm)
+        assert_almost_equal(score, current_score)
+
+
+@pytest.mark.parametrize(
+    'name', THRESHOLDED_MULTILABEL_METRICS | MULTIOUTPUT_METRICS)
+def test_thresholded_multilabel_multioutput_permutations_invariance(name):
+    random_state = check_random_state(0)
+    n_samples, n_classes = 20, 4
+    y_true = random_state.randint(0, 2, size=(n_samples, n_classes))
+    y_score = random_state.normal(size=y_true.shape)
+
+    # Makes sure all samples have multiple classes. This works around errors
+    # when running metrics where average="sample"
+    y_true[y_true.sum(1) == 4, 0] = 0
+    y_true[y_true.sum(1) == 0, 0] = 1
+
+    metric = ALL_METRICS[name]
+    score = metric(y_true, y_score)
+
+    for perm in permutations(range(n_classes), n_classes):
+        inv_perm = np.zeros(n_classes, dtype=int)
+        inv_perm[list(perm)] = np.arange(n_classes)
+
+        y_score_perm = y_score[:, inv_perm]
+        y_true_perm = y_true[:, inv_perm]
+
+        current_score = metric(y_true_perm, y_score_perm)
+        assert_almost_equal(score, current_score)


Code review comments:
diff_hunk: @@ -1201,3 +1203,53 @@ def test_no_averaging_labels():
             score_labels = metric(y_true, y_pred, labels=labels, average=None)
             score = metric(y_true, y_pred, average=None)
             assert_array_equal(score_labels, score[inverse_labels])
+
+
+@pytest.mark.parametrize(
+    'name', MULTILABELS_METRICS - NOT_SYMMETRIC_METRICS)
+def test_multilabel_label_permutations_invariance(name):
+    random_state = check_random_state(0)
+    n_samples, n_classes = 20, 4
+
+    y_true = random_state.randint(0, 2, size=(n_samples, n_classes))
+    y_score = random_state.randint(0, 2, size=(n_samples, n_classes))
+
+    metric = ALL_METRICS[name]
+    score = metric(y_true, y_score)
+
+    for perm in permutations(range(n_classes), n_classes):
+        inv_perm = np.zeros(n_classes, dtype=int)
path: sklearn/metrics/tests/test_common.py
text: @user1:
Awesome! Any reason you first do the `inv_perm` instead of directly using the permutations? Would it change the test at all if you're going through all permutations?

@author:
Good point! This PR was updated to use the permutations directly.