Talip7/scikit-learn-issues-multilabel huggingface dataset:
Make `zero_division` parameter consistent in the different metric

This is an issue to report the step to actually take over the work of @marctorsoc in https://github.com/scikit-learn/scikit-learn/pull/23183 and split the PR into smaller one to facilitate the review process.

The intend is to make the `zero_division` parameter consistent across different metrics in scikit-learn. In this regards, we have the following TODO list:

- [x] Introduce the `zero_division` parameter to the `accuracy_score` function when `y_true` and `y_pred` are empty.
    - https://github.com/scikit-learn/scikit-learn/pull/29213
- [x] Introduce the `zero_division` parameter to the `class_likelihood_ratios` and remove `raise_warning`.
    - https://github.com/scikit-learn/scikit-learn/pull/31331
- [x] Introduce the `zero_division` parameter to the `cohen_kappa_score` function
  - https://github.com/scikit-learn/scikit-learn/pull/29210
- [x] Introduce the `zero_division` parameter to the `matthew_corr_coeff` function
  - #23183
  - #28509
- [ ] <del>Open a PR to make sure the empty input lead to `np.nan` in `classification_report` function.</del> `classification_report` should raise an error instead: see https://github.com/scikit-learn/scikit-learn/issues/29048#issuecomment-2857931743

All those items have been addressed in #23183 and can be extracted in individual PRs. The changelog presenting the changes should acknowledge @marctorsoc.

In addition, we should investigate #27047 and check if we should add the `zero_division` parameter to the `precision_recall_curve` and `roc_curve` as well. This might add two additional items to the list above.

COMMENT:
No it can be there at the same time as we deprecate away `zero_division`. I'm just saying we need to make sure, when people don't give any values to these args, they still still a deprecation warning telling them that the default return value changes in case of a division by zero.

Labels: Enhancement
Make `zero_division` parameter consistent in the different metric

This is an issue to report the step to actually take over the work of @marctorsoc in https://github.com/scikit-learn/scikit-learn/pull/23183 and split the PR into smaller one to facilitate the review process.

The intend is to make the `zero_division` parameter consistent across different metrics in scikit-learn. In this regards, we have the following TODO list:

- [x] Introduce the `zero_division` parameter to the `accuracy_score` function when `y_true` and `y_pred` are empty.
    - https://github.com/scikit-learn/scikit-learn/pull/29213
- [x] Introduce the `zero_division` parameter to the `class_likelihood_ratios` and remove `raise_warning`.
    - https://github.com/scikit-learn/scikit-learn/pull/31331
- [x] Introduce the `zero_division` parameter to the `cohen_kappa_score` function
  - https://github.com/scikit-learn/scikit-learn/pull/29210
- [x] Introduce the `zero_division` parameter to the `matthew_corr_coeff` function
  - #23183
  - #28509
- [ ] <del>Open a PR to make sure the empty input lead to `np.nan` in `classification_report` function.</del> `classification_report` should raise an error instead: see https://github.com/scikit-learn/scikit-learn/issues/29048#issuecomment-2857931743

All those items have been addressed in #23183 and can be extracted in individual PRs. The changelog presenting the changes should acknowledge @marctorsoc.

In addition, we should investigate #27047 and check if we should add the `zero_division` parameter to the `precision_recall_curve` and `roc_curve` as well. This might add two additional items to the list above.

COMMENT:
Yes, makes sense. Thank you, @adrinjalali.

Labels: Enhancement
Make `zero_division` parameter consistent in the different metric

This is an issue to report the step to actually take over the work of @marctorsoc in https://github.com/scikit-learn/scikit-learn/pull/23183 and split the PR into smaller one to facilitate the review process.

The intend is to make the `zero_division` parameter consistent across different metrics in scikit-learn. In this regards, we have the following TODO list:

- [x] Introduce the `zero_division` parameter to the `accuracy_score` function when `y_true` and `y_pred` are empty.
    - https://github.com/scikit-learn/scikit-learn/pull/29213
- [x] Introduce the `zero_division` parameter to the `class_likelihood_ratios` and remove `raise_warning`.
    - https://github.com/scikit-learn/scikit-learn/pull/31331
- [x] Introduce the `zero_division` parameter to the `cohen_kappa_score` function
  - https://github.com/scikit-learn/scikit-learn/pull/29210
- [x] Introduce the `zero_division` parameter to the `matthew_corr_coeff` function
  - #23183
  - #28509
- [ ] <del>Open a PR to make sure the empty input lead to `np.nan` in `classification_report` function.</del> `classification_report` should raise an error instead: see https://github.com/scikit-learn/scikit-learn/issues/29048#issuecomment-2857931743

All those items have been addressed in #23183 and can be extracted in individual PRs. The changelog presenting the changes should acknowledge @marctorsoc.

In addition, we should investigate #27047 and check if we should add the `zero_division` parameter to the `precision_recall_curve` and `roc_curve` as well. This might add two additional items to the list above.

COMMENT:
> I am wondering what the default argument for replace_undefined_by would be? I would think:
    - `np.nan` for metrics that do not aggregate

Oh no, I have just seen that we have set the default to the lowest score in the past PRs. I have misremembered this.
But which is better? `np.nan` or the lowest score?

Labels: Enhancement
Make `zero_division` parameter consistent in the different metric

This is an issue to report the step to actually take over the work of @marctorsoc in https://github.com/scikit-learn/scikit-learn/pull/23183 and split the PR into smaller one to facilitate the review process.

The intend is to make the `zero_division` parameter consistent across different metrics in scikit-learn. In this regards, we have the following TODO list:

- [x] Introduce the `zero_division` parameter to the `accuracy_score` function when `y_true` and `y_pred` are empty.
    - https://github.com/scikit-learn/scikit-learn/pull/29213
- [x] Introduce the `zero_division` parameter to the `class_likelihood_ratios` and remove `raise_warning`.
    - https://github.com/scikit-learn/scikit-learn/pull/31331
- [x] Introduce the `zero_division` parameter to the `cohen_kappa_score` function
  - https://github.com/scikit-learn/scikit-learn/pull/29210
- [x] Introduce the `zero_division` parameter to the `matthew_corr_coeff` function
  - #23183
  - #28509
- [ ] <del>Open a PR to make sure the empty input lead to `np.nan` in `classification_report` function.</del> `classification_report` should raise an error instead: see https://github.com/scikit-learn/scikit-learn/issues/29048#issuecomment-2857931743

All those items have been addressed in #23183 and can be extracted in individual PRs. The changelog presenting the changes should acknowledge @marctorsoc.

In addition, we should investigate #27047 and check if we should add the `zero_division` parameter to the `precision_recall_curve` and `roc_curve` as well. This might add two additional items to the list above.

COMMENT:
> np.nan or the lowest score?

That's a good question. I don't know.

Labels: Enhancement
Make `zero_division` parameter consistent in the different metric

This is an issue to report the step to actually take over the work of @marctorsoc in https://github.com/scikit-learn/scikit-learn/pull/23183 and split the PR into smaller one to facilitate the review process.

The intend is to make the `zero_division` parameter consistent across different metrics in scikit-learn. In this regards, we have the following TODO list:

- [x] Introduce the `zero_division` parameter to the `accuracy_score` function when `y_true` and `y_pred` are empty.
    - https://github.com/scikit-learn/scikit-learn/pull/29213
- [x] Introduce the `zero_division` parameter to the `class_likelihood_ratios` and remove `raise_warning`.
    - https://github.com/scikit-learn/scikit-learn/pull/31331
- [x] Introduce the `zero_division` parameter to the `cohen_kappa_score` function
  - https://github.com/scikit-learn/scikit-learn/pull/29210
- [x] Introduce the `zero_division` parameter to the `matthew_corr_coeff` function
  - #23183
  - #28509
- [ ] <del>Open a PR to make sure the empty input lead to `np.nan` in `classification_report` function.</del> `classification_report` should raise an error instead: see https://github.com/scikit-learn/scikit-learn/issues/29048#issuecomment-2857931743

All those items have been addressed in #23183 and can be extracted in individual PRs. The changelog presenting the changes should acknowledge @marctorsoc.

In addition, we should investigate #27047 and check if we should add the `zero_division` parameter to the `precision_recall_curve` and `roc_curve` as well. This might add two additional items to the list above.

COMMENT:
> But which is better? np.nan or the lowest score?

Here I am re-phrasing this question, giving a bit more details and links to find out about this quickly: https://github.com/scikit-learn/scikit-learn/pull/29288#issuecomment-2478698279

Labels: Enhancement
MAINT define a single time _estimator_has and refactor code

From past discussion, I realized that we are defining the same `_estimator_has` in several places while it does exactly the same job and has the same semantic.

I think we should do a bit of cleaning by moving this function into a submodule in `sklearn.utils`. I would probably keep this function private for the moment even thought it could be useful for developers of third-party libraries.

Maybe @StefanieSenger would be interested in leading the effort since you should be familiar with this function after working on #28167?

COMMENT:
Oh yes, that's a good effort and a nice project to work on. Thank you, @glemaitre, I will open a PR for this.

Labels: Enhancement
MAINT define a single time _estimator_has and refactor code

From past discussion, I realized that we are defining the same `_estimator_has` in several places while it does exactly the same job and has the same semantic.

I think we should do a bit of cleaning by moving this function into a submodule in `sklearn.utils`. I would probably keep this function private for the moment even thought it could be useful for developers of third-party libraries.

Maybe @StefanieSenger would be interested in leading the effort since you should be familiar with this function after working on #28167?

COMMENT:
@StefanieSenger can you please review - https://github.com/scikit-learn/scikit-learn/pull/29319?

Labels: Enhancement
⚠️ CI failed on Linux_Nightly_PyPy.pypy3 (last failure: Jun 03, 2024) ⚠️

**CI is still failing on [Linux_Nightly_PyPy.pypy3](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=67132&view=logs&j=0b16f832-29d6-5b92-1c23-eb006f606a66)** (Jun 03, 2024)
Unable to find junit file. Please see link for details.

COMMENT:
This was temporarily fixed by https://github.com/scikit-learn/scikit-learn/pull/28894 but then the PyPy lock file was updated again without testing PyPy in the PR I guess.

Root cause seems to be https://github.com/scikit-learn/scikit-learn/pull/28869#issuecomment-2068832492 together with a dependency update.

Labels: Bug
⚠️ CI failed on Linux_Nightly_PyPy.pypy3 (last failure: Jun 03, 2024) ⚠️

**CI is still failing on [Linux_Nightly_PyPy.pypy3](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=67132&view=logs&j=0b16f832-29d6-5b92-1c23-eb006f606a66)** (Jun 03, 2024)
Unable to find junit file. Please see link for details.

COMMENT:
Closing since PyPy support has been dropped officially #29128

Labels: Bug
Revamp the developer documentation when it comes to roll scikit-learn compatible estimator

I find the documentation helping at writing a scikit-learn estimator a bit oldish: https://scikit-learn.org/dev/developers/develop.html

I think that we could revamp the documentation with a new look. Probably, we would like to mention what are the minimum implementation required and then go into details in the additional feature given by the mixin that we added overtime.

Finally, it should be the place where we provide some documentation regarding the developer tools and manage the expectation regarding the deprecation cycle for those.

COMMENT:
A bunch of the work for this issue was merged with #30253

It can certainly be further improved though.

Labels: Documentation, Developer API
Improve `FunctionTransformer` diagram representation

### Describe the workflow you want to enable

Currently, using multiple `FunctionTransformers` in a pipeline leads to an uninformative view:

```python
import pandas as pd
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer

df = pd.DataFrame([[1,2,3], [4,5,6]], columns=['one','two','three']) # sample data
def a(df): return df+1 # 1st transformer
def b(df): return df*10 # 2nd transformer

make_pipeline(FunctionTransformer(a), FunctionTransformer(b))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/5570380/21382d09-82ad-4e14-8091-6e14ab9989e8)

I would like to see the name of the function being used in the visual blocks


### Describe your proposed solution

I would like to see something like this:

![image](https://github.com/scikit-learn/scikit-learn/assets/5570380/8c67e04a-6328-4afd-9ebc-a0e1da29ca57)

(or perhaps `Function(<name of function>)`  or `<name of function>()` or `FunctionTransformer_<name of function>`) 
 
A sample implementation might be look like this:

```python
from sklearn.preprocessing import FunctionTransformer
from sklearn.utils._estimator_html_repr import _VisualBlock
from functools import partial

class PrettyFunctionTransformer(FunctionTransformer):
    def _sk_visual_block_(self):
        return _VisualBlock(
            "single",
            self,
            names=self.func.func.__name__ if isinstance(self.func, partial) else self.func.__name__,
            name_details=str(self),
        )
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Some of your suggestions are available when clicking on the transformers:

![image](https://github.com/scikit-learn/scikit-learn/assets/7454015/fae015af-29b6-4ebd-9083-441942db829b)

I don't know if we should treat specifically `FunctionTransformer` since this is a really generic transformer and extract out the information to display it at the first level.

Labels: New Feature
Improve `FunctionTransformer` diagram representation

### Describe the workflow you want to enable

Currently, using multiple `FunctionTransformers` in a pipeline leads to an uninformative view:

```python
import pandas as pd
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer

df = pd.DataFrame([[1,2,3], [4,5,6]], columns=['one','two','three']) # sample data
def a(df): return df+1 # 1st transformer
def b(df): return df*10 # 2nd transformer

make_pipeline(FunctionTransformer(a), FunctionTransformer(b))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/5570380/21382d09-82ad-4e14-8091-6e14ab9989e8)

I would like to see the name of the function being used in the visual blocks


### Describe your proposed solution

I would like to see something like this:

![image](https://github.com/scikit-learn/scikit-learn/assets/5570380/8c67e04a-6328-4afd-9ebc-a0e1da29ca57)

(or perhaps `Function(<name of function>)`  or `<name of function>()` or `FunctionTransformer_<name of function>`) 
 
A sample implementation might be look like this:

```python
from sklearn.preprocessing import FunctionTransformer
from sklearn.utils._estimator_html_repr import _VisualBlock
from functools import partial

class PrettyFunctionTransformer(FunctionTransformer):
    def _sk_visual_block_(self):
        return _VisualBlock(
            "single",
            self,
            names=self.func.func.__name__ if isinstance(self.func, partial) else self.func.__name__,
            name_details=str(self),
        )
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
The use case I envision is defining a scikit-learn pipeline for feature engineering. Feature engineering should be done on the training data, but also at inference time. If you add it to the model pipeline, you get 1) easier deployment (no preprocessing) and 2) safer pipelines, as feature engineering would be applied on each split separately. If you use the `memory` argument to cache the feature engineering pipeline, you also don't get the downside of repeating the same computations.

These two pipelines are identical, but the visualization on the right is much clearer:

<table border="0">
 <tr>
    <td><img src="https://github.com/scikit-learn/scikit-learn/assets/5570380/1e74d210-7385-47f0-af6e-b1533ad61ca2"></td>
    <td><img src="https://github.com/scikit-learn/scikit-learn/assets/5570380/8f1926c7-cee2-4e8e-b530-5a71756a8ef5"></td>
 </tr>
</table>

Labels: New Feature
Improve `FunctionTransformer` diagram representation

### Describe the workflow you want to enable

Currently, using multiple `FunctionTransformers` in a pipeline leads to an uninformative view:

```python
import pandas as pd
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer

df = pd.DataFrame([[1,2,3], [4,5,6]], columns=['one','two','three']) # sample data
def a(df): return df+1 # 1st transformer
def b(df): return df*10 # 2nd transformer

make_pipeline(FunctionTransformer(a), FunctionTransformer(b))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/5570380/21382d09-82ad-4e14-8091-6e14ab9989e8)

I would like to see the name of the function being used in the visual blocks


### Describe your proposed solution

I would like to see something like this:

![image](https://github.com/scikit-learn/scikit-learn/assets/5570380/8c67e04a-6328-4afd-9ebc-a0e1da29ca57)

(or perhaps `Function(<name of function>)`  or `<name of function>()` or `FunctionTransformer_<name of function>`) 
 
A sample implementation might be look like this:

```python
from sklearn.preprocessing import FunctionTransformer
from sklearn.utils._estimator_html_repr import _VisualBlock
from functools import partial

class PrettyFunctionTransformer(FunctionTransformer):
    def _sk_visual_block_(self):
        return _VisualBlock(
            "single",
            self,
            names=self.func.func.__name__ if isinstance(self.func, partial) else self.func.__name__,
            name_details=str(self),
        )
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Convinced, we need to work the details but definitely this is better. I still think we should have the info that this is a `FunctionTransformer` in some way.

Labels: New Feature
Improve `FunctionTransformer` diagram representation

### Describe the workflow you want to enable

Currently, using multiple `FunctionTransformers` in a pipeline leads to an uninformative view:

```python
import pandas as pd
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer

df = pd.DataFrame([[1,2,3], [4,5,6]], columns=['one','two','three']) # sample data
def a(df): return df+1 # 1st transformer
def b(df): return df*10 # 2nd transformer

make_pipeline(FunctionTransformer(a), FunctionTransformer(b))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/5570380/21382d09-82ad-4e14-8091-6e14ab9989e8)

I would like to see the name of the function being used in the visual blocks


### Describe your proposed solution

I would like to see something like this:

![image](https://github.com/scikit-learn/scikit-learn/assets/5570380/8c67e04a-6328-4afd-9ebc-a0e1da29ca57)

(or perhaps `Function(<name of function>)`  or `<name of function>()` or `FunctionTransformer_<name of function>`) 
 
A sample implementation might be look like this:

```python
from sklearn.preprocessing import FunctionTransformer
from sklearn.utils._estimator_html_repr import _VisualBlock
from functools import partial

class PrettyFunctionTransformer(FunctionTransformer):
    def _sk_visual_block_(self):
        return _VisualBlock(
            "single",
            self,
            names=self.func.func.__name__ if isinstance(self.func, partial) else self.func.__name__,
            name_details=str(self),
        )
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
| Lazier way | Harder way |
| :--------: | :--------: |
| ![image](https://github.com/scikit-learn/scikit-learn/assets/108576690/be55f6c2-dd15-459c-a7d8-2e9cc956e5f4) | ![image](https://github.com/scikit-learn/scikit-learn/assets/108576690/9e2bb4d1-34f9-4cf1-92de-14278f5fd23f) |
| Just @timvink's implementation plus including the class name, wrapping in an inline-block and setting `white-space: pre-wrap`. Directly fits into the framework. | Maybe look a bit better? But this requires altering the structure a bit. In particular, adding a parameter `caption` to the visual blocks (default `None`) and render in the HTML. |

Labels: New Feature
Improve `FunctionTransformer` diagram representation

### Describe the workflow you want to enable

Currently, using multiple `FunctionTransformers` in a pipeline leads to an uninformative view:

```python
import pandas as pd
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer

df = pd.DataFrame([[1,2,3], [4,5,6]], columns=['one','two','three']) # sample data
def a(df): return df+1 # 1st transformer
def b(df): return df*10 # 2nd transformer

make_pipeline(FunctionTransformer(a), FunctionTransformer(b))
```

![image](https://github.com/scikit-learn/scikit-learn/assets/5570380/21382d09-82ad-4e14-8091-6e14ab9989e8)

I would like to see the name of the function being used in the visual blocks


### Describe your proposed solution

I would like to see something like this:

![image](https://github.com/scikit-learn/scikit-learn/assets/5570380/8c67e04a-6328-4afd-9ebc-a0e1da29ca57)

(or perhaps `Function(<name of function>)`  or `<name of function>()` or `FunctionTransformer_<name of function>`) 
 
A sample implementation might be look like this:

```python
from sklearn.preprocessing import FunctionTransformer
from sklearn.utils._estimator_html_repr import _VisualBlock
from functools import partial

class PrettyFunctionTransformer(FunctionTransformer):
    def _sk_visual_block_(self):
        return _VisualBlock(
            "single",
            self,
            names=self.func.func.__name__ if isinstance(self.func, partial) else self.func.__name__,
            name_details=str(self),
        )
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think that I better the harder way (unfortunately :)).

Labels: New Feature