Repository: scikit-learn/scikit-learn
Pull request: [MRG] sklearn.metrics.normalized_mutual_info_score for continuous data does not return a warning #17892
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes [#17892](https://github.com/scikit-learn/scikit-learn/issues/17892)

#### What does this implement/fix? Explain your changes.
Fixes an an error where normalized_mutual_info_score for continuous data does not return a warning for continous values

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
using sklearn.metrics.normalized_mutual_info_score for continuous data does not return a warning
I have been informed that using sklearn.metrics.normalized_mutual_info_score to estimate a normalized mutual information score on continuous data should have returned at least a warning that this module is for use only on discrete data; and error on floats.

an example of test data:
```
noise=np.random.rand(500)
wavelength=np.linspace(0.01,1,500)*1e-6

# testing the normalized MI score between wavelength and noise - should return near zero for totally unrelated data sets
constant_normalized_mi=normalized_mutual_info_score(wavelength,noise)
```
output:
1.0

Hints from issue discussion:
I think the ``check_clustering`` function should call ``type_of_target``. We're doing that in the classification metrics. I think this has come up before, I think there's something in the issue tracker about that already.
I can take this. 
I've implemented the change for `check_clustering`  to raise an error for a float. In the example that @bfiranski  can you confirm whether the function should produce an error or a near zero value. 
#17938 
can you please update me on this.

Commit message: Implement warning in check clustering

Patch to review:
diff --git a/sklearn/metrics/cluster/_supervised.py b/sklearn/metrics/cluster/_supervised.py
index 703cfd86fe72e..565d25b3450dc 100644
--- a/sklearn/metrics/cluster/_supervised.py
+++ b/sklearn/metrics/cluster/_supervised.py
@@ -15,6 +15,7 @@
 # License: BSD 3 clause
 
 
+import warnings
 from math import log
 
 import numpy as np
@@ -22,9 +23,10 @@
 from scipy.special import comb
 
 from ._expected_mutual_info_fast import expected_mutual_information
-from ...utils.validation import check_array, check_consistent_length
-from ...utils.validation import _deprecate_positional_args
 from ...utils.fixes import _astype_copy_false
+from ...utils.multiclass import type_of_target
+from ...utils.validation import _deprecate_positional_args
+from ...utils.validation import check_array, check_consistent_length
 
 
 def _comb2(n):
@@ -47,10 +49,18 @@ def check_clusterings(labels_true, labels_pred):
     labels_true = check_array(
         labels_true, ensure_2d=False, ensure_min_samples=0, dtype=None,
     )
+
     labels_pred = check_array(
         labels_pred, ensure_2d=False, ensure_min_samples=0, dtype=None,
     )
 
+    type_label = type_of_target(labels_true)
+    type_pred = type_of_target(labels_pred)
+
+    if type_pred or type_pred == 'continous':
+        warnings.warn(UserWarning('Expects discrete values received {} '
+                                  'for label, and {} for target'
+                                  .format(type_label, type_label)))
     # input checks
     if labels_true.ndim != 1:
         raise ValueError(
@@ -856,6 +866,7 @@ def normalized_mutual_info_score(labels_true, labels_pred, *,
     labels_true, labels_pred = check_clusterings(labels_true, labels_pred)
     classes = np.unique(labels_true)
     clusters = np.unique(labels_pred)
+
     # Special limit cases: no clustering since the data is not split.
     # This is a perfect match hence return 1.0.
     if (classes.shape[0] == clusters.shape[0] == 1 or
diff --git a/sklearn/metrics/cluster/tests/test_supervised.py b/sklearn/metrics/cluster/tests/test_supervised.py
index a0168ae5ac7a8..8c1f4a6203a06 100644
--- a/sklearn/metrics/cluster/tests/test_supervised.py
+++ b/sklearn/metrics/cluster/tests/test_supervised.py
@@ -348,3 +348,12 @@ def test_fowlkes_mallows_score_properties():
 def test_mutual_info_score_positive_constant_label(labels_true, labels_pred):
     # non-regression test for #16355
     assert mutual_info_score(labels_true, labels_pred) >= 0
+
+
+def test_check_clustering_error():
+    # Test warning message for continuous values
+    noise = np.random.rand(500)
+    wavelength = np.linspace(0.01, 1, 500) * 1e-6
+
+    with pytest.warns(UserWarning):
+        normalized_mutual_info_score(wavelength, noise)


Code review comments:
diff_hunk: @@ -348,3 +348,12 @@ def test_fowlkes_mallows_score_properties():
 def test_mutual_info_score_positive_constant_label(labels_true, labels_pred):
     # non-regression test for #16355
     assert mutual_info_score(labels_true, labels_pred) >= 0
+
+
+def test_check_clustering_error():
+    # Test warning message for continuous values
+    noise = np.random.rand(500)
+    wavelength = np.linspace(0.01, 1, 500) * 1e-6
+
+    with pytest.warns(UserWarning):
+        normalized_mutual_info_score(wavelength, noise)
path: sklearn/metrics/cluster/tests/test_supervised.py
text: @user1:
The name of this test is `test_check_clustering_error` but we are checking `normalized_mutual_info_score`.

@author:
`check_clusterings` is called in `normalized_mutual_info_score` and i wrote to the test to ensure that the fix the original pull request.

diff_hunk: @@ -47,10 +49,18 @@ def check_clusterings(labels_true, labels_pred):
     labels_true = check_array(
         labels_true, ensure_2d=False, ensure_min_samples=0, dtype=None,
     )
+
     labels_pred = check_array(
         labels_pred, ensure_2d=False, ensure_min_samples=0, dtype=None,
     )
 
+    type_label = type_of_target(labels_true)
+    type_pred = type_of_target(labels_pred)
+
+    if type_pred or type_pred == 'continous':
path: sklearn/metrics/cluster/_supervised.py
text: @user1:
```suggestion
    if 'continous' in (type_pred, type_pred):
```

diff_hunk: @@ -348,3 +348,12 @@ def test_fowlkes_mallows_score_properties():
 def test_mutual_info_score_positive_constant_label(labels_true, labels_pred):
     # non-regression test for #16355
     assert mutual_info_score(labels_true, labels_pred) >= 0
+
+
+def test_check_clustering_error():
+    # Test warning message for continuous values
+    noise = np.random.rand(500)
+    wavelength = np.linspace(0.01, 1, 500) * 1e-6
+
+    with pytest.warns(UserWarning):
path: sklearn/metrics/cluster/tests/test_supervised.py
text: @user1:
Check for the message explicitly:

```suggestion
    msg = "..."
    with pytest.warns(UserWarning, match=msg):
```

diff_hunk: @@ -47,10 +49,18 @@ def check_clusterings(labels_true, labels_pred):
     labels_true = check_array(
         labels_true, ensure_2d=False, ensure_min_samples=0, dtype=None,
     )
+
     labels_pred = check_array(
         labels_pred, ensure_2d=False, ensure_min_samples=0, dtype=None,
     )
 
+    type_label = type_of_target(labels_true)
+    type_pred = type_of_target(labels_pred)
+
+    if type_pred or type_pred == 'continous':
+        warnings.warn(UserWarning('Expects discrete values received {} '
+                                  'for label, and {} for target'
+                                  .format(type_label, type_label)))
path: sklearn/metrics/cluster/_supervised.py
text: @user1:
We can use f-strings here:

```suggestion
        warnings.warn(f'Expects discrete values received {type_label} '
                      f'for label, and {type_label} for target',
                      UserWarning)
```

diff_hunk: @@ -348,3 +348,12 @@ def test_fowlkes_mallows_score_properties():
 def test_mutual_info_score_positive_constant_label(labels_true, labels_pred):
     # non-regression test for #16355
     assert mutual_info_score(labels_true, labels_pred) >= 0
+
+
+def test_check_clustering_error():
+    # Test warning message for continuous values
+    noise = np.random.rand(500)
path: sklearn/metrics/cluster/tests/test_supervised.py
text: @user1:
```suggestion
    rng = np.RandomState(42)
    noise = rng.rand(500)
```