Repository: scikit-learn/scikit-learn
Pull request: FIX Ensure correct sklearn.metrics.coverage_error error message for 1D array
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #23368 

#### What does this implement/fix? Explain your changes.
* Change the 2 calls `check_array(., ensure_2d=False)` at the beginning of coverage_error to `ensure_2d=True`.
* Add "if metric == coverage_error then reshape y_true to 2d" in `test_regression_thresholded_inf_nan_input` in metrics common test to fix failed tests.
* Add coverage_error specific test to check correct error message is raised when passing a 1d array as input.

#### Any other comments?
Not sure if adding the "if else" in the common test is elegant. It might adds more cognitive complexity. Hoping for any comment on it.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
sklearn.metrics.coverage_error wrong error message for 1D array
### Describe the bug

Let y_true and y_score be numpy arrays of shape (K,),
when you try to run the "sklearn.metrics.coverage_error" as explained in the documentation
it returns "binary type not supported" error, but this is not the case at all, the metric works with,
binary, the problem is it expects matrices of shape (1, K) if there is only one sample,
Please either fix the function so that it can work with 1D array or please fix the error message.

### Steps/Code to Reproduce

```python
coverage = np.mean([coverage_error(discrete_labels[i], discrete_scores[i]) - 1 for i in range(N)])
```
### Expected Results

No error or, can not use 1D array error

### Actual Results

```python-traceback
   raise ValueError("{0} format is not supported".format(y_type))
ValueError: binary format is not supported
```

### Versions

```shell
>>> import sklearn; sklearn.show_versions()

System:
    python: 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:20:46)  [GCC 9.4.0]
executable: /home/delivahit/anaconda3/envs/bugra/bin/python
   machine: Linux-5.4.0-107-generic-x86_64-with-glibc2.27

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 1.0.2
        numpy: 1.21.2
        scipy: 1.7.2
       Cython: None
       pandas: 1.3.4
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 3.0.0

Built with OpenMP: True
```

Hints from issue discussion:
> Please either fix the function so that it can work with 1D array or please fix the error message.

I agree we should fix the error messages. I think it comes from the fact that the 2 calls `check_array(., ensure_2d=False)` at the beginning of `coverage_error` should be changes to `ensure_2d=True` instead. Some tests might need to be updated accordingly and we would need a new test to check that the right error message is raised when passing a 1d array as input.

Commit message: Modify 1D error message test

Patch to review:
diff --git a/sklearn/metrics/_ranking.py b/sklearn/metrics/_ranking.py
index 7f64f479ed275..9e746f167381f 100644
--- a/sklearn/metrics/_ranking.py
+++ b/sklearn/metrics/_ranking.py
@@ -1155,8 +1155,8 @@ def coverage_error(y_true, y_score, *, sample_weight=None):
            handbook (pp. 667-685). Springer US.
 
     """
-    y_true = check_array(y_true, ensure_2d=False)
-    y_score = check_array(y_score, ensure_2d=False)
+    y_true = check_array(y_true, ensure_2d=True)
+    y_score = check_array(y_score, ensure_2d=True)
     check_consistent_length(y_true, y_score, sample_weight)
 
     y_type = type_of_target(y_true, input_name="y_true")
diff --git a/sklearn/metrics/tests/test_common.py b/sklearn/metrics/tests/test_common.py
index 1e627f9f86676..3492ce144d82e 100644
--- a/sklearn/metrics/tests/test_common.py
+++ b/sklearn/metrics/tests/test_common.py
@@ -910,6 +910,9 @@ def test_thresholded_invariance_string_vs_numbers_labels(name):
 )
 @pytest.mark.parametrize("y_true, y_score", invalids_nan_inf)
 def test_regression_thresholded_inf_nan_input(metric, y_true, y_score):
+    if metric == coverage_error:
+        y_true = [y_true]
+        y_score = [y_score]
     with pytest.raises(ValueError, match=r"contains (NaN|infinity)"):
         metric(y_true, y_score)
 
diff --git a/sklearn/metrics/tests/test_ranking.py b/sklearn/metrics/tests/test_ranking.py
index 7d2338337b83d..b667a967753cb 100644
--- a/sklearn/metrics/tests/test_ranking.py
+++ b/sklearn/metrics/tests/test_ranking.py
@@ -1569,6 +1569,13 @@ def test_coverage_tie_handling():
     assert_almost_equal(coverage_error([[1, 1, 1]], [[0.25, 0.5, 0.5]]), 3)
 
 
+def test_coverage_1d_error_message():
+    with pytest.raises(ValueError, match=r"Expected 2D array, got 1D array instead"):
+        coverage_error([1, 0, 1], [0.25, 0.5, 0.5])
+        coverage_error([1, 0, 1], [[0.25, 0.5, 0.5]])
+        coverage_error([[1, 0, 1]], [0.25, 0.5, 0.5])
+
+
 def test_label_ranking_loss():
     assert_almost_equal(label_ranking_loss([[0, 1]], [[0.25, 0.75]]), 0)
     assert_almost_equal(label_ranking_loss([[0, 1]], [[0.75, 0.25]]), 1)


Code review comments:
diff_hunk: @@ -1569,6 +1569,13 @@ def test_coverage_tie_handling():
     assert_almost_equal(coverage_error([[1, 1, 1]], [[0.25, 0.5, 0.5]]), 3)
 
 
+def test_coverage_1d_error_message():
+    with pytest.raises(ValueError, match=r"Expected 2D array, got 1D array instead"):
+        coverage_error([1, 0, 1], [0.25, 0.5, 0.5])
path: sklearn/metrics/tests/test_ranking.py
text: @user1:
This test would only test the first line, since it will fail right away and the other lines would not run.

I suggest using a `pytest.mark.parametrize` to parametrize over the different inputs.

@author:
Thanks for the catch! I used `pytest.mark.parametrize` to test different 1d array input combinations.
The test should raise ValueError if one of the input is 1d array.