Talip7/scikit-learn-issues-multilabel huggingface dataset:
NDCG in case of abscence of relevant items

### Describe the bug

In `sklearn.metrics._ndcg_sample_scores`, there is a counterintuitive handling of the case where all true relevances are equal to zero for some samples. In this case, DCG = 0, IDCG = 0, and the whole NDCG is not defined. In `sklearn` implementation it is defined as 0 and included in the averaged NDCG calculation.  The least leads to strange effects, like `ndcg_score(y,y) != 1`; moreover, it affects the metric value in non-trivial cases too.

In the original 2002 paper where NDCG is proposed, it is not stated how to handle such situations, but it is clearly mentioned that 
```
The (D)CG vectors for each IR technique can be normalized by dividing them
by the corresponding ideal (D)CG vectors, component by component. In this way,
for any vector position, the normalized value 1 represents ideal performance,
and values in the range [0, 1) the share of ideal performance cumulated by each
technique.
```
meaning that NDCG(y,y) must always be 1. 


I suggest excluding observations without relevant items and/or throwing a warning.

### Steps/Code to Reproduce

```
>>> from sklearn.metrics import ndcg_score
>>> y = np.array([[1.0, 0.0, 1.0], [0.0, 0.0, 0.0]])
>>> ndcg_score(y, y)
```


### Expected Results

1.

### Actual Results

0.5

### Versions

```shell
This code was not changed in 1.5, so I guess for newer versions the issue also is actual.



System:
    python: 3.11.8 (main, Feb 12 2024, 14:50:05) [GCC 13.2.1 20230801]
executable: /usr/bin/python3
   machine: Linux-6.6.19-1-MANJARO-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.3.1
          pip: 24.0
   setuptools: 69.0.3
        numpy: 1.26.4
        scipy: 1.10.1
       Cython: 3.0.9
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/arabella/.local/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/arabella/.local/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/arabella/.local/lib/python3.11/site-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8
```

COMMENT:
> the ndcg score in this case is 0 / 0

It looks a case where we want to introduce the parameter `zero_division` where `"warn"` (returning `np.nan` with a warning) is the default and it can be change to 0 or 1 depending of the application. For instance, this is what we are doing for the thresholded classification metric where it makes sense (https://github.com/scikit-learn/scikit-learn/issues/29048).

Labels: Bug, help wanted
NDCG in case of abscence of relevant items

### Describe the bug

In `sklearn.metrics._ndcg_sample_scores`, there is a counterintuitive handling of the case where all true relevances are equal to zero for some samples. In this case, DCG = 0, IDCG = 0, and the whole NDCG is not defined. In `sklearn` implementation it is defined as 0 and included in the averaged NDCG calculation.  The least leads to strange effects, like `ndcg_score(y,y) != 1`; moreover, it affects the metric value in non-trivial cases too.

In the original 2002 paper where NDCG is proposed, it is not stated how to handle such situations, but it is clearly mentioned that 
```
The (D)CG vectors for each IR technique can be normalized by dividing them
by the corresponding ideal (D)CG vectors, component by component. In this way,
for any vector position, the normalized value 1 represents ideal performance,
and values in the range [0, 1) the share of ideal performance cumulated by each
technique.
```
meaning that NDCG(y,y) must always be 1. 


I suggest excluding observations without relevant items and/or throwing a warning.

### Steps/Code to Reproduce

```
>>> from sklearn.metrics import ndcg_score
>>> y = np.array([[1.0, 0.0, 1.0], [0.0, 0.0, 0.0]])
>>> ndcg_score(y, y)
```


### Expected Results

1.

### Actual Results

0.5

### Versions

```shell
This code was not changed in 1.5, so I guess for newer versions the issue also is actual.



System:
    python: 3.11.8 (main, Feb 12 2024, 14:50:05) [GCC 13.2.1 20230801]
executable: /usr/bin/python3
   machine: Linux-6.6.19-1-MANJARO-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.3.1
          pip: 24.0
   setuptools: 69.0.3
        numpy: 1.26.4
        scipy: 1.10.1
       Cython: 3.0.9
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/arabella/.local/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/arabella/.local/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/arabella/.local/lib/python3.11/site-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8
```

COMMENT:
I would like to work on this.

Labels: Bug, help wanted
NDCG in case of abscence of relevant items

### Describe the bug

In `sklearn.metrics._ndcg_sample_scores`, there is a counterintuitive handling of the case where all true relevances are equal to zero for some samples. In this case, DCG = 0, IDCG = 0, and the whole NDCG is not defined. In `sklearn` implementation it is defined as 0 and included in the averaged NDCG calculation.  The least leads to strange effects, like `ndcg_score(y,y) != 1`; moreover, it affects the metric value in non-trivial cases too.

In the original 2002 paper where NDCG is proposed, it is not stated how to handle such situations, but it is clearly mentioned that 
```
The (D)CG vectors for each IR technique can be normalized by dividing them
by the corresponding ideal (D)CG vectors, component by component. In this way,
for any vector position, the normalized value 1 represents ideal performance,
and values in the range [0, 1) the share of ideal performance cumulated by each
technique.
```
meaning that NDCG(y,y) must always be 1. 


I suggest excluding observations without relevant items and/or throwing a warning.

### Steps/Code to Reproduce

```
>>> from sklearn.metrics import ndcg_score
>>> y = np.array([[1.0, 0.0, 1.0], [0.0, 0.0, 0.0]])
>>> ndcg_score(y, y)
```


### Expected Results

1.

### Actual Results

0.5

### Versions

```shell
This code was not changed in 1.5, so I guess for newer versions the issue also is actual.



System:
    python: 3.11.8 (main, Feb 12 2024, 14:50:05) [GCC 13.2.1 20230801]
executable: /usr/bin/python3
   machine: Linux-6.6.19-1-MANJARO-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.3.1
          pip: 24.0
   setuptools: 69.0.3
        numpy: 1.26.4
        scipy: 1.10.1
       Cython: 3.0.9
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/arabella/.local/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/arabella/.local/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/arabella/.local/lib/python3.11/site-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8
```

COMMENT:
This PR better wait for https://github.com/scikit-learn/scikit-learn/issues/29048#issuecomment-2460062084 to be decided and implemented to take that change into account.

Labels: Bug, help wanted
Handle all-zeros cases for multioutput metrics

### Describe the workflow you want to enable

For multioutput problems, all-zero label columns (or in general constant label columns) can sometimes happen, for example when using cross-validation. Most metrics (e.g. precision, recall, F1, AUPRC/average recall) return 0.0 with a warning, AUROC throws an error.

This is particularly common in chemoinformatics, where we have many heavily imbalanced multioutput problems. Getting all-zeros column or two among 50-100 targets almost always happens. Currently, I write wrappers in each project, manually looping through columns and ignoring those with constant targets, which is 1) code duplication 2) clearly a missing feature from the built-in metrics.

### Describe your proposed solution

add an option for the multioutput case for metrics whether to ignore the label columns with constant value, e.g. `constant_value: str = "use"`, with possible values "use" or "ignore". If all columns are constant, throw an error. If only some are constant, ignore them and aggregate the value only from those columns that had at least 2 classes.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think the API we have settled on is to pass `labels` like we do in ['roc_auc_score'](https://scikit-learn.org/dev/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score).

A PR to handle that in other metrics would be welcome.

Labels: New Feature, Needs Decision
Handle all-zeros cases for multioutput metrics

### Describe the workflow you want to enable

For multioutput problems, all-zero label columns (or in general constant label columns) can sometimes happen, for example when using cross-validation. Most metrics (e.g. precision, recall, F1, AUPRC/average recall) return 0.0 with a warning, AUROC throws an error.

This is particularly common in chemoinformatics, where we have many heavily imbalanced multioutput problems. Getting all-zeros column or two among 50-100 targets almost always happens. Currently, I write wrappers in each project, manually looping through columns and ignoring those with constant targets, which is 1) code duplication 2) clearly a missing feature from the built-in metrics.

### Describe your proposed solution

add an option for the multioutput case for metrics whether to ignore the label columns with constant value, e.g. `constant_value: str = "use"`, with possible values "use" or "ignore". If all columns are constant, throw an error. If only some are constant, ignore them and aggregate the value only from those columns that had at least 2 classes.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@adrinjalali how does using `labels` solve this problem for AUROC? For example:
```
import numpy as np
from sklearn.metrics import roc_auc_score

true = np.array([
    [0, 0],
    [0, 1],
    [0, 1]
])
pred = np.array([
    [0, 1],
    [0, 1],
    [1, 1]
])

roc_auc_score(true, pred, labels=[0, 1])
```
This will still throw an error. What I propose in this issue is:
```
val = roc_auc_score(true, pred, constant_value="ignore")
# val is calculated ignoring first column, i.e. as roc_auc_score([0, 1, 1], [1, 1, 1])
```

Labels: New Feature, Needs Decision
Handle all-zeros cases for multioutput metrics

### Describe the workflow you want to enable

For multioutput problems, all-zero label columns (or in general constant label columns) can sometimes happen, for example when using cross-validation. Most metrics (e.g. precision, recall, F1, AUPRC/average recall) return 0.0 with a warning, AUROC throws an error.

This is particularly common in chemoinformatics, where we have many heavily imbalanced multioutput problems. Getting all-zeros column or two among 50-100 targets almost always happens. Currently, I write wrappers in each project, manually looping through columns and ignoring those with constant targets, which is 1) code duplication 2) clearly a missing feature from the built-in metrics.

### Describe your proposed solution

add an option for the multioutput case for metrics whether to ignore the label columns with constant value, e.g. `constant_value: str = "use"`, with possible values "use" or "ignore". If all columns are constant, throw an error. If only some are constant, ignore them and aggregate the value only from those columns that had at least 2 classes.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I'm surprised by your comment in https://github.com/scikit-fingerprints/scikit-fingerprints/issues/210

> Scikit-learn refused to add support for missing values, and seems confused about all-zero cases, so we need to implement this here.

I see the misunderstanding on my side, but we didn't refuse (we might, but that hasn't happened yet).

But regardless, we really don't appreciate that kind of language and have a 0 tolerance for such behavior.

I'm not sure if ignoring the first column in your example would be the best path forward, in cases where the problem becomes ill-defined.

@ogrisel @lorentzenchr @adam2392 WDYT?

Labels: New Feature, Needs Decision
Handle all-zeros cases for multioutput metrics

### Describe the workflow you want to enable

For multioutput problems, all-zero label columns (or in general constant label columns) can sometimes happen, for example when using cross-validation. Most metrics (e.g. precision, recall, F1, AUPRC/average recall) return 0.0 with a warning, AUROC throws an error.

This is particularly common in chemoinformatics, where we have many heavily imbalanced multioutput problems. Getting all-zeros column or two among 50-100 targets almost always happens. Currently, I write wrappers in each project, manually looping through columns and ignoring those with constant targets, which is 1) code duplication 2) clearly a missing feature from the built-in metrics.

### Describe your proposed solution

add an option for the multioutput case for metrics whether to ignore the label columns with constant value, e.g. `constant_value: str = "use"`, with possible values "use" or "ignore". If all columns are constant, throw an error. If only some are constant, ignore them and aggregate the value only from those columns that had at least 2 classes.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Sorry, this was normal communication for me. I didn't mean this negatively, just as a regular statement. I can edit and word this differently, if you like.

Ignoring the column is the standard practice for molecular benchmarks. This may not be the best fit for scikit-learn, but is a correct  and expected behavior in multioutput classification in chemoinformatics.

Labels: New Feature, Needs Decision
Handle all-zeros cases for multioutput metrics

### Describe the workflow you want to enable

For multioutput problems, all-zero label columns (or in general constant label columns) can sometimes happen, for example when using cross-validation. Most metrics (e.g. precision, recall, F1, AUPRC/average recall) return 0.0 with a warning, AUROC throws an error.

This is particularly common in chemoinformatics, where we have many heavily imbalanced multioutput problems. Getting all-zeros column or two among 50-100 targets almost always happens. Currently, I write wrappers in each project, manually looping through columns and ignoring those with constant targets, which is 1) code duplication 2) clearly a missing feature from the built-in metrics.

### Describe your proposed solution

add an option for the multioutput case for metrics whether to ignore the label columns with constant value, e.g. `constant_value: str = "use"`, with possible values "use" or "ignore". If all columns are constant, throw an error. If only some are constant, ignore them and aggregate the value only from those columns that had at least 2 classes.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
This seems like an edge case, so idk if it warrants a brand new kwarg, so I wonder if it's possible to define a custom scorer:

```py
# something like this pseudocode
def roc_auc_scorer_molecular_dynamics(y, y_pred, **kwargs):
      nondegen_output = []
      for ioutput in range(y.shape[1]):
            if len(np.unique(y[:, ioutput])) > 1:
                   nondegen_output.append(ioutput)

      # return roc_auc score without that output
      return roc_auc_score(y[:, nondegen_output], y_pred[:, nondegen_output], **kwargs)
``` 

? In addition, I wonder if this warrants an example in the documentation using a simple molecular benchmark dataset?

Labels: New Feature, Needs Decision
Handle all-zeros cases for multioutput metrics

### Describe the workflow you want to enable

For multioutput problems, all-zero label columns (or in general constant label columns) can sometimes happen, for example when using cross-validation. Most metrics (e.g. precision, recall, F1, AUPRC/average recall) return 0.0 with a warning, AUROC throws an error.

This is particularly common in chemoinformatics, where we have many heavily imbalanced multioutput problems. Getting all-zeros column or two among 50-100 targets almost always happens. Currently, I write wrappers in each project, manually looping through columns and ignoring those with constant targets, which is 1) code duplication 2) clearly a missing feature from the built-in metrics.

### Describe your proposed solution

add an option for the multioutput case for metrics whether to ignore the label columns with constant value, e.g. `constant_value: str = "use"`, with possible values "use" or "ignore". If all columns are constant, throw an error. If only some are constant, ignore them and aggregate the value only from those columns that had at least 2 classes.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Where I agree with @j-adamczyk , is that in multioutput, it seems like it'd be quite easy in a CV test to get in cases where a column is all zero since the target can be sparse. So I would be happy if we could include some sort of a fix in sklearn. My doubt is, I'm not sure if ignoring them is the right thing to do. We can at least raise a warning, but that doesn't really help in practice where one wants to run cv benchmarks.

Labels: New Feature, Needs Decision
Handle all-zeros cases for multioutput metrics

### Describe the workflow you want to enable

For multioutput problems, all-zero label columns (or in general constant label columns) can sometimes happen, for example when using cross-validation. Most metrics (e.g. precision, recall, F1, AUPRC/average recall) return 0.0 with a warning, AUROC throws an error.

This is particularly common in chemoinformatics, where we have many heavily imbalanced multioutput problems. Getting all-zeros column or two among 50-100 targets almost always happens. Currently, I write wrappers in each project, manually looping through columns and ignoring those with constant targets, which is 1) code duplication 2) clearly a missing feature from the built-in metrics.

### Describe your proposed solution

add an option for the multioutput case for metrics whether to ignore the label columns with constant value, e.g. `constant_value: str = "use"`, with possible values "use" or "ignore". If all columns are constant, throw an error. If only some are constant, ignore them and aggregate the value only from those columns that had at least 2 classes.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
the `constant_value` can be an optional ***kwargs* argument which is not required in most cases but uses for edge cases like molecular dynamics.

Labels: New Feature, Needs Decision
Handle all-zeros cases for multioutput metrics

### Describe the workflow you want to enable

For multioutput problems, all-zero label columns (or in general constant label columns) can sometimes happen, for example when using cross-validation. Most metrics (e.g. precision, recall, F1, AUPRC/average recall) return 0.0 with a warning, AUROC throws an error.

This is particularly common in chemoinformatics, where we have many heavily imbalanced multioutput problems. Getting all-zeros column or two among 50-100 targets almost always happens. Currently, I write wrappers in each project, manually looping through columns and ignoring those with constant targets, which is 1) code duplication 2) clearly a missing feature from the built-in metrics.

### Describe your proposed solution

add an option for the multioutput case for metrics whether to ignore the label columns with constant value, e.g. `constant_value: str = "use"`, with possible values "use" or "ignore". If all columns are constant, throw an error. If only some are constant, ignore them and aggregate the value only from those columns that had at least 2 classes.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I don't mind introducing the optional arg. But we'd need others' opinion here as well, to find a consensus.

Labels: New Feature, Needs Decision
Handle all-zeros cases for multioutput metrics

### Describe the workflow you want to enable

For multioutput problems, all-zero label columns (or in general constant label columns) can sometimes happen, for example when using cross-validation. Most metrics (e.g. precision, recall, F1, AUPRC/average recall) return 0.0 with a warning, AUROC throws an error.

This is particularly common in chemoinformatics, where we have many heavily imbalanced multioutput problems. Getting all-zeros column or two among 50-100 targets almost always happens. Currently, I write wrappers in each project, manually looping through columns and ignoring those with constant targets, which is 1) code duplication 2) clearly a missing feature from the built-in metrics.

### Describe your proposed solution

add an option for the multioutput case for metrics whether to ignore the label columns with constant value, e.g. `constant_value: str = "use"`, with possible values "use" or "ignore". If all columns are constant, throw an error. If only some are constant, ignore them and aggregate the value only from those columns that had at least 2 classes.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@adrinjalali I think you should remove the "help wanted" tag since this issue still needs decision

Labels: New Feature, Needs Decision