Talip7/scikit-learn-issues-multilabel huggingface dataset:
GridSearchCV scoring behavior on weighted metrics, such as Balanced Accuracy

### Describe the workflow you want to enable

As a user who specify weighted metrics/scoring such as balanced accuracy, I would like the scoring `GridSearchCV` uses for optimising and `best_score_` to be weighted as well. Based on current documentation, I think it is currently simply a mean, so I do not aware if this has been tackled/asked before.

I am thinking about this since my data is imbalanced, and saw [this](https://stackoverflow.com/questions/47572191/scikit-learn-scoring-in-gridsearchcv) post in SO too.

The mean seems to work fine when `StratifiedKFold` with minority class is multiple of `K`. If not (where the number minority class in each fold can be different) the scoring could be different with the 'combined scoring'. However, this is not as severe as doing this in `LeaveOneOut`.

### Describe your proposed solution

When the user specify built-in weighted scoring, record the validation results for all folds and calculate scoring from combined validation result.

This can be default behavior or an option to turn this on/off. However, I couldn't see why people who specify weighted metrics want to turn this off.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Closing as duplicate of https://github.com/scikit-learn/scikit-learn/issues/8158

Labels: New Feature, module:model_selection, Needs Decision - Include Feature
GridSearchCV scoring behavior on weighted metrics, such as Balanced Accuracy

### Describe the workflow you want to enable

As a user who specify weighted metrics/scoring such as balanced accuracy, I would like the scoring `GridSearchCV` uses for optimising and `best_score_` to be weighted as well. Based on current documentation, I think it is currently simply a mean, so I do not aware if this has been tackled/asked before.

I am thinking about this since my data is imbalanced, and saw [this](https://stackoverflow.com/questions/47572191/scikit-learn-scoring-in-gridsearchcv) post in SO too.

The mean seems to work fine when `StratifiedKFold` with minority class is multiple of `K`. If not (where the number minority class in each fold can be different) the scoring could be different with the 'combined scoring'. However, this is not as severe as doing this in `LeaveOneOut`.

### Describe your proposed solution

When the user specify built-in weighted scoring, record the validation results for all folds and calculate scoring from combined validation result.

This can be default behavior or an option to turn this on/off. However, I couldn't see why people who specify weighted metrics want to turn this off.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@adrinjalali Thank you. Is there any ETA for that?

Also, is there any workaround for the current version? I think I would need to create a child class of `GridSearchCV`, but I am lost as to which function I should override to get the predictions (instead of scores) of the inner cross-validation and apply the scoring function to them once (need to access the ground truth/`y`).

Labels: New Feature, module:model_selection, Needs Decision - Include Feature
GridSearchCV scoring behavior on weighted metrics, such as Balanced Accuracy

### Describe the workflow you want to enable

As a user who specify weighted metrics/scoring such as balanced accuracy, I would like the scoring `GridSearchCV` uses for optimising and `best_score_` to be weighted as well. Based on current documentation, I think it is currently simply a mean, so I do not aware if this has been tackled/asked before.

I am thinking about this since my data is imbalanced, and saw [this](https://stackoverflow.com/questions/47572191/scikit-learn-scoring-in-gridsearchcv) post in SO too.

The mean seems to work fine when `StratifiedKFold` with minority class is multiple of `K`. If not (where the number minority class in each fold can be different) the scoring could be different with the 'combined scoring'. However, this is not as severe as doing this in `LeaveOneOut`.

### Describe your proposed solution

When the user specify built-in weighted scoring, record the validation results for all folds and calculate scoring from combined validation result.

This can be default behavior or an option to turn this on/off. However, I couldn't see why people who specify weighted metrics want to turn this off.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
We're planning to have that by the release for October/November (and earlier in our nightly builds).

In the meantime, I'm not sure if it's still working, but I wrote this answer a while ago: https://stackoverflow.com/a/49598597/2536294

Labels: New Feature, module:model_selection, Needs Decision - Include Feature
Add expected calibration error (ECE) functionality to sklearn.

### Describe the workflow you want to enable

Measuring calibration error in deep learning is a big issue now-a-days. However, we do not find any suitable package available to measure the calibration error. I would like to add a function that can measure calibration error with adaptive bin size as described here: https://openaccess.thecvf.com/content_CVPRW_2019/papers/Uncertainty%20and%20Robustness%20in%20Deep%20Visual%20Learning/Nixon_Measuring_Calibration_in_Deep_Learning_CVPRW_2019_paper.pdf.  Note that there are many ECE algorithms that measures the calibration error with fixed bin size that is major problem for inhomogeneous data distribution.

### Describe your proposed solution

We already have a tentative implementation here: https://github.com/neurodata/kdg/blob/cc88886819676020402f19ec8b224ed5aafdc440/kdg/utils.py#L8. We would like to adapt the code for sklearn so that people can use it.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thank you for opening the issue. I am closing this issue because it is a duplicate of https://github.com/scikit-learn/scikit-learn/issues/18268. You are welcome to continue the discussion there.

Labels: New Feature, Needs Triage
Cache final transformer in pipeline with memory setting

### Describe the bug

When setting the `memory` parameter of a transformer `Pipeline` (i.e., one whose last step is a transformer), the final transformer is not cached.

Discovered at https://stackoverflow.com/q/71812869/10495893.

### Steps/Code to Reproduce

```
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
import time

class Test(BaseEstimator, TransformerMixin):
    def __init__(self, col):
        self.col = col

    def fit(self, X, y=None):
        print(self.col)
        return self

    def transform(self, X, y=None):
        for t in range(5):
            # just to slow it down / check caching.
            print(".")
            time.sleep(1)
        #print(self.col)
        return X

pipline = Pipeline(
    [
        ("test", Test(col="this_column")),
        ("test2", Test(col="that_column"))
    ],
    memory="tmp/cache",
)

pipline.fit(None)
pipline.fit(None)
pipline.fit(None)
```

### Expected Results

```
this_column
.
.
.
.
.
that_column
```

### Actual Results

```
this_column
.
.
.
.
.
that_column
that_column
that_column
```

### Versions

```shell
System:
    python: 3.7.13 (default, Mar 16 2022, 17:37:17)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.4.1
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.2.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
The original use case for caching is for the final step to be a classifier or regressor, where all previous steps are cached. As for this issue, the final step is a transformer, so technically it should be cached according to pipeline's docstring:

> Used to cache the fitted transformers of the pipeline.

If we strictly follow this, then we would have caching behavior depending on what the final step is, which can be confusing semantically. The alternative solution is to update the docs and say that we only cache `steps[:-1]`.

What do you think @jnothman ?

Labels: Bug, module:pipeline
Cache final transformer in pipeline with memory setting

### Describe the bug

When setting the `memory` parameter of a transformer `Pipeline` (i.e., one whose last step is a transformer), the final transformer is not cached.

Discovered at https://stackoverflow.com/q/71812869/10495893.

### Steps/Code to Reproduce

```
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
import time

class Test(BaseEstimator, TransformerMixin):
    def __init__(self, col):
        self.col = col

    def fit(self, X, y=None):
        print(self.col)
        return self

    def transform(self, X, y=None):
        for t in range(5):
            # just to slow it down / check caching.
            print(".")
            time.sleep(1)
        #print(self.col)
        return X

pipline = Pipeline(
    [
        ("test", Test(col="this_column")),
        ("test2", Test(col="that_column"))
    ],
    memory="tmp/cache",
)

pipline.fit(None)
pipline.fit(None)
pipline.fit(None)
```

### Expected Results

```
this_column
.
.
.
.
.
that_column
```

### Actual Results

```
this_column
.
.
.
.
.
that_column
that_column
that_column
```

### Versions

```shell
System:
    python: 3.7.13 (default, Mar 16 2022, 17:37:17)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.4.1
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.2.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
We can first update the documentation to make it clear that we cache up to the last step (not included).

We can later think of an enhancement where we can cache all transformers of a pipeline.

Labels: Bug, module:pipeline
Cache final transformer in pipeline with memory setting

### Describe the bug

When setting the `memory` parameter of a transformer `Pipeline` (i.e., one whose last step is a transformer), the final transformer is not cached.

Discovered at https://stackoverflow.com/q/71812869/10495893.

### Steps/Code to Reproduce

```
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
import time

class Test(BaseEstimator, TransformerMixin):
    def __init__(self, col):
        self.col = col

    def fit(self, X, y=None):
        print(self.col)
        return self

    def transform(self, X, y=None):
        for t in range(5):
            # just to slow it down / check caching.
            print(".")
            time.sleep(1)
        #print(self.col)
        return X

pipline = Pipeline(
    [
        ("test", Test(col="this_column")),
        ("test2", Test(col="that_column"))
    ],
    memory="tmp/cache",
)

pipline.fit(None)
pipline.fit(None)
pipline.fit(None)
```

### Expected Results

```
this_column
.
.
.
.
.
that_column
```

### Actual Results

```
this_column
.
.
.
.
.
that_column
that_column
that_column
```

### Versions

```shell
System:
    python: 3.7.13 (default, Mar 16 2022, 17:37:17)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.4.1
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.2.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
Discussion with @glemaitre suggests that if the last step of a pipeline is a transformer, then fit/transform/fit_transform methods of Pipeline class should also cache the fit/transform/fit_transform call of the last step.

Labels: Bug, module:pipeline
Cache final transformer in pipeline with memory setting

### Describe the bug

When setting the `memory` parameter of a transformer `Pipeline` (i.e., one whose last step is a transformer), the final transformer is not cached.

Discovered at https://stackoverflow.com/q/71812869/10495893.

### Steps/Code to Reproduce

```
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
import time

class Test(BaseEstimator, TransformerMixin):
    def __init__(self, col):
        self.col = col

    def fit(self, X, y=None):
        print(self.col)
        return self

    def transform(self, X, y=None):
        for t in range(5):
            # just to slow it down / check caching.
            print(".")
            time.sleep(1)
        #print(self.col)
        return X

pipline = Pipeline(
    [
        ("test", Test(col="this_column")),
        ("test2", Test(col="that_column"))
    ],
    memory="tmp/cache",
)

pipline.fit(None)
pipline.fit(None)
pipline.fit(None)
```

### Expected Results

```
this_column
.
.
.
.
.
that_column
```

### Actual Results

```
this_column
.
.
.
.
.
that_column
that_column
that_column
```

### Versions

```shell
System:
    python: 3.7.13 (default, Mar 16 2022, 17:37:17)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.4.1
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.2.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
@glemaitre suggested that detecting transformers is most robust with `isinstance(step, TransformerMixin)` as opposed to using ducktyping checks with `getattr(step, "fit_transform") or getattr(step, "transform")`

Labels: Bug, module:pipeline
Cache final transformer in pipeline with memory setting

### Describe the bug

When setting the `memory` parameter of a transformer `Pipeline` (i.e., one whose last step is a transformer), the final transformer is not cached.

Discovered at https://stackoverflow.com/q/71812869/10495893.

### Steps/Code to Reproduce

```
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
import time

class Test(BaseEstimator, TransformerMixin):
    def __init__(self, col):
        self.col = col

    def fit(self, X, y=None):
        print(self.col)
        return self

    def transform(self, X, y=None):
        for t in range(5):
            # just to slow it down / check caching.
            print(".")
            time.sleep(1)
        #print(self.col)
        return X

pipline = Pipeline(
    [
        ("test", Test(col="this_column")),
        ("test2", Test(col="that_column"))
    ],
    memory="tmp/cache",
)

pipline.fit(None)
pipline.fit(None)
pipline.fit(None)
```

### Expected Results

```
this_column
.
.
.
.
.
that_column
```

### Actual Results

```
this_column
.
.
.
.
.
that_column
that_column
that_column
```

### Versions

```shell
System:
    python: 3.7.13 (default, Mar 16 2022, 17:37:17)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.4.1
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.2.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
I just realized that caching is currently also supported by Pipeline.fit_predict and it is not supported by Pipeline.transform. So I prepare the first solution for fit and fit_transform only.

Labels: Bug, module:pipeline
Cache final transformer in pipeline with memory setting

### Describe the bug

When setting the `memory` parameter of a transformer `Pipeline` (i.e., one whose last step is a transformer), the final transformer is not cached.

Discovered at https://stackoverflow.com/q/71812869/10495893.

### Steps/Code to Reproduce

```
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
import time

class Test(BaseEstimator, TransformerMixin):
    def __init__(self, col):
        self.col = col

    def fit(self, X, y=None):
        print(self.col)
        return self

    def transform(self, X, y=None):
        for t in range(5):
            # just to slow it down / check caching.
            print(".")
            time.sleep(1)
        #print(self.col)
        return X

pipline = Pipeline(
    [
        ("test", Test(col="this_column")),
        ("test2", Test(col="that_column"))
    ],
    memory="tmp/cache",
)

pipline.fit(None)
pipline.fit(None)
pipline.fit(None)
```

### Expected Results

```
this_column
.
.
.
.
.
that_column
```

### Actual Results

```
this_column
.
.
.
.
.
that_column
that_column
that_column
```

### Versions

```shell
System:
    python: 3.7.13 (default, Mar 16 2022, 17:37:17)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.4.1
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.2.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
This Draft PR illustrates the idea before implementing test code: https://github.com/scikit-learn/scikit-learn/pull/26008/files

Labels: Bug, module:pipeline
Cache final transformer in pipeline with memory setting

### Describe the bug

When setting the `memory` parameter of a transformer `Pipeline` (i.e., one whose last step is a transformer), the final transformer is not cached.

Discovered at https://stackoverflow.com/q/71812869/10495893.

### Steps/Code to Reproduce

```
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
import time

class Test(BaseEstimator, TransformerMixin):
    def __init__(self, col):
        self.col = col

    def fit(self, X, y=None):
        print(self.col)
        return self

    def transform(self, X, y=None):
        for t in range(5):
            # just to slow it down / check caching.
            print(".")
            time.sleep(1)
        #print(self.col)
        return X

pipline = Pipeline(
    [
        ("test", Test(col="this_column")),
        ("test2", Test(col="that_column"))
    ],
    memory="tmp/cache",
)

pipline.fit(None)
pipline.fit(None)
pipline.fit(None)
```

### Expected Results

```
this_column
.
.
.
.
.
that_column
```

### Actual Results

```
this_column
.
.
.
.
.
that_column
that_column
that_column
```

### Versions

```shell
System:
    python: 3.7.13 (default, Mar 16 2022, 17:37:17)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.4.1
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.2.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
For `Pipeline.transform()` I suggest a bigger change:
1.  the Pipeline object should remember whether `Pipeline.fit()` was called
2. `Pipeline.transform()` throws an error if `Pipeline.fit()` was not called and only executes step.transform() of the last step because all previous steps have been fit-transformed within `Pipeline.fit()`

Labels: Bug, module:pipeline
Cache final transformer in pipeline with memory setting

### Describe the bug

When setting the `memory` parameter of a transformer `Pipeline` (i.e., one whose last step is a transformer), the final transformer is not cached.

Discovered at https://stackoverflow.com/q/71812869/10495893.

### Steps/Code to Reproduce

```
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
import time

class Test(BaseEstimator, TransformerMixin):
    def __init__(self, col):
        self.col = col

    def fit(self, X, y=None):
        print(self.col)
        return self

    def transform(self, X, y=None):
        for t in range(5):
            # just to slow it down / check caching.
            print(".")
            time.sleep(1)
        #print(self.col)
        return X

pipline = Pipeline(
    [
        ("test", Test(col="this_column")),
        ("test2", Test(col="that_column"))
    ],
    memory="tmp/cache",
)

pipline.fit(None)
pipline.fit(None)
pipline.fit(None)
```

### Expected Results

```
this_column
.
.
.
.
.
that_column
```

### Actual Results

```
this_column
.
.
.
.
.
that_column
that_column
that_column
```

### Versions

```shell
System:
    python: 3.7.13 (default, Mar 16 2022, 17:37:17)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.4.1
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.2.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
The main problem with caching the last transformer step in `Pipeline.transform()` is due to the fact that in `Pipeline.fit_transform()` we only cache `_fit_transform_one` as a combined operation. One conclusion could be that we might always cache `fit()` and `transform()` separately for the last transformer.

Labels: Bug, module:pipeline
Cache final transformer in pipeline with memory setting

### Describe the bug

When setting the `memory` parameter of a transformer `Pipeline` (i.e., one whose last step is a transformer), the final transformer is not cached.

Discovered at https://stackoverflow.com/q/71812869/10495893.

### Steps/Code to Reproduce

```
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
import time

class Test(BaseEstimator, TransformerMixin):
    def __init__(self, col):
        self.col = col

    def fit(self, X, y=None):
        print(self.col)
        return self

    def transform(self, X, y=None):
        for t in range(5):
            # just to slow it down / check caching.
            print(".")
            time.sleep(1)
        #print(self.col)
        return X

pipline = Pipeline(
    [
        ("test", Test(col="this_column")),
        ("test2", Test(col="that_column"))
    ],
    memory="tmp/cache",
)

pipline.fit(None)
pipline.fit(None)
pipline.fit(None)
```

### Expected Results

```
this_column
.
.
.
.
.
that_column
```

### Actual Results

```
this_column
.
.
.
.
.
that_column
that_column
that_column
```

### Versions

```shell
System:
    python: 3.7.13 (default, Mar 16 2022, 17:37:17)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.4.1
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.2.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
In case you say that we are generally on the right track, here, I would add test cases to the draft PR.

Labels: Bug, module:pipeline