Repository: scikit-learn-contrib/imbalanced-learn (Python)

Issue:
geometric_mean_score inside classification_report_imbalanced is wrong?
Hey,

I was getting different reported values for geometric mean score when using the geometric_mean_score and was curious why. Looks like there are a few mixed up parameters in classification_report_imbalanced.

geometric_mean_score wants y_true before y_pred:
```
imblearn.metrics.geometric_mean_score(y_true, y_pred, labels=None, pos_label=1, average='multiclass', sample_weight=None, correction=0.0)
````

However, inside classification_report_imbalanced:
```
geo = geometric_mean_score(
            y_pred,
            y_true,
            labels=labels,
            average=None,
            sample_weight=None)
```

So to for index balanced geometric mean:
```
# Index balanced accuracy
    iba_gmean = make_index_balanced_accuracy(
        alpha=alpha, squared=True)(geometric_mean_score)
    iba = iba_gmean(
        y_pred,
        y_true,
        labels=labels,
        average=None,
        sample_weight=sample_weight)
```

On average, the results are quite different:

```
PRE     ++ REC     ++ SPE     ++ F1      ++ GEO     ++ IBA
  0.496 ##   0.485 ##   0.904 ##   0.462 ##   0.637 ##   0.407
```

```
                   pre       rec       spe        f1       geo       iba       sup
...
avg / total       0.50      0.48      0.90      0.46      0.68      0.45      2800
```


After changing param order I was able to get consistent results 

Versions:
```
>>> ('Python', '2.7.10 (v2.7.10:15c95b7d81dc, May 23 2015, 09:33:12) \n[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]')
>>> ('NumPy', '1.13.3')
>>> ('SciPy', '1.0.0')
>>> ('Scikit-Learn', '0.19.0')
>>> ('Imbalanced-Learn', '0.3.1')
```


Discussion:
Can you provide a stand-alone script to reproduce the problem ? Please read https://stackoverflow.com/help/mcve. I am confused with your bug report.

Are you sure that the same `average` was used.
`classification_report_imbalanced` calls `geometric_mean_score` with y_pred in place of y_true and vice versa. The output is different when these two parameters have switched place. In the documentation (and looking at how geometic_mean_score is implemented), y_true is the first parameter and y_pred the second. 

`classification_report_imbalanced` should call `geometric_mean_score` with y_truth as first argument and y_pred as second argument. Because it yields different results otherwise, and the API specifies that order.

Edit: Seems like the parameter order miss match is not the main issue to why I'm getting inconsistent results (was not able reproduce the same problem in isolation). I'll report back when I know more. Thanks!
Wait! This might confirm that the order of y_truth and y_pred has to be correct:

```
import numpy as np
from imblearn.metrics import geometric_mean_score
from sklearn.utils.multiclass import unique_labels


y_pred = np.random.randint(25, size=2000)
y_true = np.random.randint(25, size=2000)
labels = unique_labels(y_true, y_pred)

geo_pt = geometric_mean_score(y_pred, y_true, labels=labels, average=None, sample_weight=None)
geo_tp = geometric_mean_score(y_true, y_pred, labels=labels, average=None, sample_weight=None)

print geo_pt
print geo_tp

print np.average(geo_pt)
print np.average(geo_tp)
```

```
→ python script.py
[ 0.19193913  0.26226263  0.22981553  0.11717751  0.11349908  0.23257081
  0.16934315  0.22731255  0.23107233  0.17646221  0.16535631  0.14142755
  0.1565899   0.27222751  0.19445269  0.1824145   0.14627426  0.2151913
  0.11958237  0.17762588  0.25085307  0.23654274  0.11802481  0.11393099
  0.3001162 ]
[ 0.18524655  0.27342537  0.24588147  0.11263222  0.10280463  0.21830842
  0.15739718  0.24537299  0.22099909  0.17927076  0.15064496  0.15616334
  0.14944024  0.28286225  0.18534714  0.20246446  0.1616976   0.22034215
  0.10851487  0.18349999  0.28022071  0.24939906  0.11266257  0.11177659
  0.26637954]
0.189682600417
0.190510166464
```
> classification_report_imbalanced calls geometric_mean_score with y_pred in place of y_true and vice versa. The output is different when these two parameters have switched place. In the documentation (and looking at how geometic_mean_score is implemented), y_true is the first parameter and y_pred the second.

Oh I see thanks for reporting. You are right, those 2 lines need to be exchanged:

https://github.com/scikit-learn-contrib/imbalanced-learn/blob/cab9fcf984d5038341ce2974b205595d6a14b5a7/imblearn/metrics/classification.py#L869-L871

https://github.com/scikit-learn-contrib/imbalanced-learn/blob/cab9fcf984d5038341ce2974b205595d6a14b5a7/imblearn/metrics/classification.py#L878-L880

Do you wish to make a PR to solve the issue.
Great! I'll make a PR



Patch:
diff --git a/doc/whats_new/v0.0.4.rst b/doc/whats_new/v0.0.4.rst
index 468f0deb3..67b012ae1 100644
--- a/doc/whats_new/v0.0.4.rst
+++ b/doc/whats_new/v0.0.4.rst
@@ -18,6 +18,10 @@ Enhancement
 Bug fixes
 .........
 
+- Fix bug in :func:`metrics.classification_report_imbalanced` for which
+  `y_pred` and `y_true` where inversed. :issue:`394` by :user:`Ole Silvig
+  <klizter>.`
+
 - Fix bug in ADASYN to consider only samples from the current class when
   generating new samples. :issue:`354` by :user:`Guillaume Lemaitre
   <glemaitre>`.
diff --git a/imblearn/metrics/classification.py b/imblearn/metrics/classification.py
index 46b32cbbf..ff8994c28 100644
--- a/imblearn/metrics/classification.py
+++ b/imblearn/metrics/classification.py
@@ -815,14 +815,14 @@ def classification_report_imbalanced(y_true,
                        pre       rec       spe        f1       geo       iba\
        sup
     <BLANKLINE>
-        class 0       0.50      1.00      0.75      0.67      0.71      0.48\
+        class 0       0.50      1.00      0.75      0.67      0.87      0.77\
          1
         class 1       0.00      0.00      0.75      0.00      0.00      0.00\
          1
-        class 2       1.00      0.67      1.00      0.80      0.82      0.69\
+        class 2       1.00      0.67      1.00      0.80      0.82      0.64\
          3
     <BLANKLINE>
-    avg / total       0.70      0.60      0.90      0.61      0.63      0.51\
+    avg / total       0.70      0.60      0.90      0.61      0.66      0.54\
          5
     <BLANKLINE>
 
@@ -867,8 +867,8 @@ class 2       1.00      0.67      1.00      0.80      0.82      0.69\
         sample_weight=sample_weight)
     # Geometric mean
     geo_mean = geometric_mean_score(
-        y_pred,
         y_true,
+        y_pred,
         labels=labels,
         average=None,
         sample_weight=sample_weight)
@@ -876,8 +876,8 @@ class 2       1.00      0.67      1.00      0.80      0.82      0.69\
     iba_gmean = make_index_balanced_accuracy(
         alpha=alpha, squared=True)(geometric_mean_score)
     iba = iba_gmean(
-        y_pred,
         y_true,
+        y_pred,
         labels=labels,
         average=None,
         sample_weight=sample_weight)


Test patch:
diff --git a/imblearn/metrics/tests/test_classification.py b/imblearn/metrics/tests/test_classification.py
index 76ac30c48..c37f955f1 100644
--- a/imblearn/metrics/tests/test_classification.py
+++ b/imblearn/metrics/tests/test_classification.py
@@ -1,3 +1,5 @@
+# coding: utf-8
+
 """Testing the metric for classification with imbalanced dataset"""
 # Authors: Guillaume Lemaitre <g.lemaitre58@gmail.com>
 #          Christos Aridas
@@ -221,7 +223,6 @@ def test_geometric_mean_multiclass():
 
     y_true = [0, 0, 0, 0]
     y_pred = [1, 1, 1, 1]
-    print(geometric_mean_score(y_true, y_pred))
     assert_allclose(geometric_mean_score(y_true, y_pred), 0.0, rtol=R_TOL)
 
     cor = 0.001
@@ -316,9 +317,9 @@ def test_classification_report_imbalanced_multiclass():
 
     # print classification report with class names
     expected_report = ('pre rec spe f1 geo iba sup setosa 0.83 0.79 0.92 '
-                       '0.81 0.86 0.74 24 versicolor 0.33 0.10 0.86 0.15 '
-                       '0.44 0.19 31 virginica 0.42 0.90 0.55 0.57 0.63 '
-                       '0.37 20 avg / total 0.51 0.53 0.80 0.47 0.62 0.41 75')
+                       '0.81 0.85 0.72 24 versicolor 0.33 0.10 0.86 0.15 '
+                       '0.29 0.08 31 virginica 0.42 0.90 0.55 0.57 0.70 '
+                       '0.51 20 avg / total 0.51 0.53 0.80 0.47 0.58 0.40 75')
 
     report = classification_report_imbalanced(
         y_true,
@@ -328,9 +329,9 @@ def test_classification_report_imbalanced_multiclass():
     assert _format_report(report) == expected_report
     # print classification report with label detection
     expected_report = ('pre rec spe f1 geo iba sup 0 0.83 0.79 0.92 0.81 '
-                       '0.86 0.74 24 1 0.33 0.10 0.86 0.15 0.44 0.19 31 2 '
-                       '0.42 0.90 0.55 0.57 0.63 0.37 20 avg / total 0.51 '
-                       '0.53 0.80 0.47 0.62 0.41 75')
+                       '0.85 0.72 24 1 0.33 0.10 0.86 0.15 0.29 0.08 31 '
+                       '2 0.42 0.90 0.55 0.57 0.70 0.51 20 avg / total '
+                       '0.51 0.53 0.80 0.47 0.58 0.40 75')
 
     report = classification_report_imbalanced(y_true, y_pred)
     assert _format_report(report) == expected_report
@@ -342,11 +343,11 @@ def test_classification_report_imbalanced_multiclass_with_digits():
 
     # print classification report with class names
     expected_report = ('pre rec spe f1 geo iba sup setosa 0.82609 0.79167 '
-                       '0.92157 0.80851 0.86409 0.74085 24 versicolor '
-                       '0.33333 0.09677 0.86364 0.15000 0.43809 0.18727 31 '
-                       'virginica 0.41860 0.90000 0.54545 0.57143 0.62645 '
-                       '0.37208 20 avg / total 0.51375 0.53333 0.79733 '
-                       '0.47310 0.62464 0.41370 75')
+                       '0.92157 0.80851 0.85415 0.72010 24 versicolor '
+                       '0.33333 0.09677 0.86364 0.15000 0.28910 0.07717 '
+                       '31 virginica 0.41860 0.90000 0.54545 0.57143 0.70065 '
+                       '0.50831 20 avg / total 0.51375 0.53333 0.79733 '
+                       '0.47310 0.57966 0.39788 75')
     report = classification_report_imbalanced(
         y_true,
         y_pred,
@@ -356,9 +357,9 @@ def test_classification_report_imbalanced_multiclass_with_digits():
     assert _format_report(report) == expected_report
     # print classification report with label detection
     expected_report = ('pre rec spe f1 geo iba sup 0 0.83 0.79 0.92 0.81 '
-                       '0.86 0.74 24 1 0.33 0.10 0.86 0.15 0.44 0.19 31 2 '
-                       '0.42 0.90 0.55 0.57 0.63 0.37 20 avg / total 0.51 '
-                       '0.53 0.80 0.47 0.62 0.41 75')
+                       '0.85 0.72 24 1 0.33 0.10 0.86 0.15 0.29 0.08 31 '
+                       '2 0.42 0.90 0.55 0.57 0.70 0.51 20 avg / total 0.51 '
+                       '0.53 0.80 0.47 0.58 0.40 75')
     report = classification_report_imbalanced(y_true, y_pred)
     assert _format_report(report) == expected_report
 
@@ -369,17 +370,17 @@ def test_classification_report_imbalanced_multiclass_with_string_label():
     y_true = np.array(["blue", "green", "red"])[y_true]
     y_pred = np.array(["blue", "green", "red"])[y_pred]
 
-    expected_report = ('pre rec spe f1 geo iba sup blue 0.83 0.79 0.92 '
-                       '0.81 0.86 0.74 24 green 0.33 0.10 0.86 0.15 0.44 '
-                       '0.19 31 red 0.42 0.90 0.55 0.57 0.63 0.37 20 '
-                       'avg / total 0.51 0.53 0.80 0.47 0.62 0.41 75')
+    expected_report = ('pre rec spe f1 geo iba sup blue 0.83 0.79 0.92 0.81 '
+                       '0.85 0.72 24 green 0.33 0.10 0.86 0.15 0.29 0.08 31 '
+                       'red 0.42 0.90 0.55 0.57 0.70 0.51 20 avg / total '
+                       '0.51 0.53 0.80 0.47 0.58 0.40 75')
     report = classification_report_imbalanced(y_true, y_pred)
     assert _format_report(report) == expected_report
 
-    expected_report = ('pre rec spe f1 geo iba sup a 0.83 0.79 0.92 0.81 '
-                       '0.86 0.74 24 b 0.33 0.10 0.86 0.15 0.44 0.19 31 '
-                       'c 0.42 0.90 0.55 0.57 0.63 0.37 20 avg / total '
-                       '0.51 0.53 0.80 0.47 0.62 0.41 75')
+    expected_report = ('pre rec spe f1 geo iba sup a 0.83 0.79 0.92 0.81 0.85 '
+                       '0.72 24 b 0.33 0.10 0.86 0.15 0.29 0.08 31 c 0.42 '
+                       '0.90 0.55 0.57 0.70 0.51 20 avg / total 0.51 0.53 '
+                       '0.80 0.47 0.58 0.40 75')
     report = classification_report_imbalanced(
         y_true, y_pred, target_names=["a", "b", "c"])
     assert _format_report(report) == expected_report
@@ -392,10 +393,10 @@ def test_classification_report_imbalanced_multiclass_with_unicode_label():
     y_true = labels[y_true]
     y_pred = labels[y_pred]
 
-    expected_report = (u'pre rec spe f1 geo iba sup blue\xa2 0.83 0.79 '
-                       u'0.92 0.81 0.86 0.74 24 green\xa2 0.33 0.10 0.86 '
-                       u'0.15 0.44 0.19 31 red\xa2 0.42 0.90 0.55 0.57 0.63 '
-                       u'0.37 20 avg / total 0.51 0.53 0.80 0.47 0.62 0.41 75')
+    expected_report = (u'pre rec spe f1 geo iba sup blue¢ 0.83 0.79 0.92 0.81 '
+                       u'0.85 0.72 24 green¢ 0.33 0.10 0.86 0.15 0.29 0.08 31 '
+                       u'red¢ 0.42 0.90 0.55 0.57 0.70 0.51 20 avg / total '
+                       u'0.51 0.53 0.80 0.47 0.58 0.40 75')
     if np_version[:3] < (1, 7, 0):
         with raises(RuntimeError, match="NumPy < 1.7.0"):
             classification_report_imbalanced(y_true, y_pred)
@@ -412,9 +413,9 @@ def test_classification_report_imbalanced_multiclass_with_long_string_label():
     y_pred = labels[y_pred]
 
     expected_report = ('pre rec spe f1 geo iba sup blue 0.83 0.79 0.92 0.81 '
-                       '0.86 0.74 24 greengreengreengreengreen 0.33 0.10 '
-                       '0.86 0.15 0.44 0.19 31 red 0.42 0.90 0.55 0.57 0.63 '
-                       '0.37 20 avg / total 0.51 0.53 0.80 0.47 0.62 0.41 75')
+                       '0.85 0.72 24 greengreengreengreengreen 0.33 0.10 '
+                       '0.86 0.15 0.29 0.08 31 red 0.42 0.90 0.55 0.57 0.70 '
+                       '0.51 20 avg / total 0.51 0.53 0.80 0.47 0.58 0.40 75')
 
     report = classification_report_imbalanced(y_true, y_pred)
     assert _format_report(report) == expected_report