Repository: scikit-learn/scikit-learn
Pull request: FIX Fix adjusted_mutual_info_score numerical issue
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
fixes #30950

#### What does this implement/fix? Explain your changes.
- Make clipping on the values inside `adjusted_mutual_info_score` symmetric, i.e. both denominator and nominator are now clipped;
- Add shortcut to calculations, to return 0 when there is no split in the data;
- Add non-regression test.

#### Any other comments?

I'm not sure this is the best way forward

Problem statement:
Potential Problem in the Computation of Adjusted Mutual Info Score
### Describe the bug

It seems to me that for clusters of size 2 and 4, the AMI yields unexpected results of 0 instead of 1, if all items belong to different clusters. 



### Steps/Code to Reproduce

Sample Code to Reproduce:

```python
>>> from sklearn.metrics.cluster import adjusted_mutual_info_score
>>> print(adjusted_mutual_info_score([1, 2], [3, 4])
0.0
>>> print(adjusted_mutual_info_score([1, 2, 3, 4], [5, 6, 7, 8])
0.0
>>> print(adjusted_mutual_info_score([1, 2, 3, 4, 5], [6, 7, 8, 9, 10])
1.0
```

### Expected Results

As the clusters are identical in all cases, I'd expect the result to be 1.0 in all cases. This happens with version 1.6.1.

### Actual Results

So we have the strange behavior that the code outputs for lists containing different labels with 2 items and with 4 items the value 0, while we deal with identical partitions. I tested until 1000 items, it only occurs with 2 and 4.

### Versions

```shell
System:
    python: 3.13.1 (main, Dec  4 2024, 18:05:56) [GCC 14.2.1 20240910]
executable: /home/mattis/.envs/lexi/bin/python
   machine: Linux-6.12.10-arch1-1-x86_64-with-glibc2.41

Python dependencies:
      sklearn: 1.6.1
          pip: 24.3.1
   setuptools: 75.8.0
        numpy: 2.2.2
        scipy: 1.15.2
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libscipy_openblas
       filepath: /home/mattis/.envs/lexi/lib/python3.13/site-packages/numpy.libs/libscipy_openblas64_-6bb31eeb.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libscipy_openblas
       filepath: /home/mattis/.envs/lexi/lib/python3.13/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 20
         prefix: libgomp
       filepath: /home/mattis/.envs/lexi/lib/python3.13/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

Hints from issue discussion:
Weird, it seems like a real bug, although probably not that important in practice? I guess someone needs to investigate in more details?
Yes, it occurs only if you stubbornly try to determine how these methods react on different clusters. That's what I did. But it leaves some general suspicion to the method, right? Specifically the fact that it occurs in only those two cases (as far as I have investigated it).
Hello,

first of all, I see that the bug is reproducible even with the same clusterings:

```
>>> from sklearn.metrics.cluster import adjusted_mutual_info_score

>>> print(adjusted_mutual_info_score([0, 1], [0, 1]))
0.0
>>> print(adjusted_mutual_info_score([1, 2, 3], [1, 2, 3]))
1.0
>>> print(adjusted_mutual_info_score([1, 2, 3, 4], [1, 2, 3, 4]))
0.0
>>> print(adjusted_mutual_info_score([1, 2, 3, 4, 5], [1, 2, 3, 4, 5]))
1.0
```

This is part of the source code of `adjust_mutual_info_score`. 

```
    contingency = contingency_matrix(labels_true, labels_pred, sparse=True)

    # Calculate the MI for the two clusterings
    mi = mutual_info_score(labels_true, labels_pred, contingency=contingency)

    # Calculate the expected value for the mutual information
    emi = expected_mutual_information(contingency, n_samples)

    # Calculate entropy for each labeling
    h_true, h_pred = entropy(labels_true), entropy(labels_pred)
    normalizer = _generalized_average(h_true, h_pred, average_method)
    denominator = normalizer - emi

    # Avoid 0.0 / 0.0 when expectation equals maximum, i.e. a perfect match.
    # normalizer should always be >= emi, but because of floating-point
    # representation, sometimes emi is slightly larger. Correct this
    # by preserving the sign.
    if denominator < 0:
        denominator = min(denominator, -np.finfo("float64").eps)
    else:
        denominator = max(denominator, np.finfo("float64").eps)
    ami = (mi - emi) / denominator
    return float(ami)
```

It seems that the issue arises from the fact that `mi` is exactly equal to `emi` for the cases of 2 and 4 labels, so eventually `ami=0`. 

- for `adjusted_mutual_info_score([0, 1], [0, 1])` , `mi = emi =0.6931471805599453`
- for `adjusted_mutual_info_score([1, 2, 3], [1, 2, 3])` , `mi =1.0986122886681096`  and `emi =1.0986122886681098`


Perhaps this is a numerical issue as for the case of size 3 `mi` and `emi` differ only at the last digit.

Commit message: Merge branch 'main' into adj-mi-bugfix

Patch to review:
diff --git a/doc/whats_new/upcoming_changes/sklearn.metrics/31065.fix.rst b/doc/whats_new/upcoming_changes/sklearn.metrics/31065.fix.rst
new file mode 100644
index 0000000000000..a73f6e36c342a
--- /dev/null
+++ b/doc/whats_new/upcoming_changes/sklearn.metrics/31065.fix.rst
@@ -0,0 +1,3 @@
+- :func:`metrics.adjusted_mutual_info_score` will return correct
+  values when number of classes and samples is low.
+  By :user:`Hleb Levitski <glevv>`
diff --git a/sklearn/metrics/cluster/_supervised.py b/sklearn/metrics/cluster/_supervised.py
index 0f56513abca8e..0f9417b4acaf3 100644
--- a/sklearn/metrics/cluster/_supervised.py
+++ b/sklearn/metrics/cluster/_supervised.py
@@ -1033,12 +1033,16 @@ def adjusted_mutual_info_score(
         or classes.shape[0] == clusters.shape[0] == 0
     ):
         return 1.0
+    # if there is only one class or one cluster return 0.0.
+    elif classes.shape[0] == 1 or clusters.shape[0] == 1:
+        return 0.0
 
     contingency = contingency_matrix(labels_true, labels_pred, sparse=True)
     # Calculate the MI for the two clusterings
     mi = mutual_info_score(labels_true, labels_pred, contingency=contingency)
     # Calculate the expected value for the mutual information
     emi = expected_mutual_information(contingency, n_samples)
+    nominator = mi - emi
     # Calculate entropy for each labeling
     h_true, h_pred = entropy(labels_true), entropy(labels_pred)
     normalizer = _generalized_average(h_true, h_pred, average_method)
@@ -1051,8 +1055,12 @@ def adjusted_mutual_info_score(
         denominator = min(denominator, -np.finfo("float64").eps)
     else:
         denominator = max(denominator, np.finfo("float64").eps)
-    ami = (mi - emi) / denominator
-    return float(ami)
+    # We repeat the same process for the nominator to make transformation symmetric.
+    if nominator < 0:
+        nominator = min(nominator, -np.finfo("float64").eps)
+    else:
+        nominator = max(nominator, np.finfo("float64").eps)
+    return float(nominator / denominator)
 
 
 @validate_params(
diff --git a/sklearn/metrics/cluster/tests/test_supervised.py b/sklearn/metrics/cluster/tests/test_supervised.py
index 1d04255633da2..42e2639bd79eb 100644
--- a/sklearn/metrics/cluster/tests/test_supervised.py
+++ b/sklearn/metrics/cluster/tests/test_supervised.py
@@ -477,6 +477,18 @@ def test_adjusted_rand_score_overflow():
         adjusted_rand_score(y_true, y_pred)
 
 
+@pytest.mark.parametrize("inputs", [[0, 1], [0, 1, 2, 3]])
+@pytest.mark.parametrize("average_method", ["min", "arithmetic", "geometric", "max"])
+def test_adjusted_mi_score_low_class_count(inputs, average_method):
+    """Check that 2 and 4 samples and classes will return correct values
+    Non-regression test for:
+    https://github.com/scikit-learn/scikit-learn/issues/30950
+    """
+    assert (
+        adjusted_mutual_info_score(inputs, inputs, average_method=average_method) == 1.0
+    )
+
+
 @pytest.mark.parametrize("average_method", ["min", "arithmetic", "geometric", "max"])
 def test_normalized_mutual_info_score_bounded(average_method):
     """Check that nmi returns a score between 0 (included) and 1 (excluded


Code review comments:
diff_hunk: @@ -1033,12 +1033,16 @@ def adjusted_mutual_info_score(
         or classes.shape[0] == clusters.shape[0] == 0
     ):
         return 1.0
+    # if there is only one class or one cluster return 0.0.
+    elif classes.shape[0] == 1 or clusters.shape[0] == 1:
+        return 0.0
 
     contingency = contingency_matrix(labels_true, labels_pred, sparse=True)
     # Calculate the MI for the two clusterings
     mi = mutual_info_score(labels_true, labels_pred, contingency=contingency)
     # Calculate the expected value for the mutual information
     emi = expected_mutual_information(contingency, n_samples)
+    nominator = mi - emi
path: sklearn/metrics/cluster/_supervised.py
text: @user1:
Can you use `numerator` instead of `nominator`?

Not a native English speaker, but I never heard of `nominator` before and `numerator` rings a bell. This seems to match this [StackExchange answer](https://math.stackexchange.com/a/159082).

@author:
It was a mistake on my part, sorry, will fix it