Talip7/scikit-learn-issues-multilabel huggingface dataset:
ValueError when a key present in class_weight is not found in labels

### Describe the bug

While using `sklearn.ensemble.RandomForestClassifier` with a custom dictionary specified for `class_weight` parameter, a `ValueError` is raised when a key (label) present in `class_weight` is not found in any of the labels. I got this error while using `TimeSeriesSplit`. Specifically, if a split is such that training data doesn't have a label specified in `class_weight`, a `ValueError` will be raised. This stops users from evaluating their model since `class_weight` cannot be changed on a go after the model has been created. 

Is there any reason this check is being done? There's no error when we don't specify class_weight. The code raising this error can be found here ([link](https://github.com/scikit-learn/scikit-learn/blob/5900e2660f4293b52ea57bf830e7bff312779036/sklearn/utils/class_weight.py#L8)).



### Steps/Code to Reproduce

```
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(class_weight={1:10, 0:1})
rfc.fit([[0, 0, 1], [1, 0, 1]], [0, 0])
```



### Expected Results

No errors raised.

### Actual Results

`ValueError: Class label 1 not present.`


### Versions

```shell
System:
    python: 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)]
executable: c:\python3.9.2\python.exe
   machine: Windows-10-10.0.19041-SP0

Python dependencies:
          pip: 21.3.1
   setuptools: 58.3.0
      sklearn: 0.24.1
        numpy: 1.21.3
        scipy: 1.6.1
       Cython: 0.29.24
       pandas: 1.2.3
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```

COMMENT:
I think the error message was designed to prevent user error:

```python
from sklearn.tree import DecisionTreeClassifier

# Misspelled "dog" in `class_weight`
tree = DecisionTreeClassifier(class_weight={"dogs": 2, "cat": 1})
tree.fit([[1, 2, 3], [1, 3, 2]], ["dog", "cat"])
```

Currently, `class_weight` is defining all the classes and we are enforcing that all classes appear in `y`.

We can switch it around and let `y` define all the classes. In other words, we ignore class weights that is not in the `y`, but still error if `y` has a class that is not in `class_weight`.

We have to be careful about changing `compute_sample_weight`'s behavior, because it is a public function.

Labels: Bug, Moderate, module:tree
ValueError when a key present in class_weight is not found in labels

### Describe the bug

While using `sklearn.ensemble.RandomForestClassifier` with a custom dictionary specified for `class_weight` parameter, a `ValueError` is raised when a key (label) present in `class_weight` is not found in any of the labels. I got this error while using `TimeSeriesSplit`. Specifically, if a split is such that training data doesn't have a label specified in `class_weight`, a `ValueError` will be raised. This stops users from evaluating their model since `class_weight` cannot be changed on a go after the model has been created. 

Is there any reason this check is being done? There's no error when we don't specify class_weight. The code raising this error can be found here ([link](https://github.com/scikit-learn/scikit-learn/blob/5900e2660f4293b52ea57bf830e7bff312779036/sklearn/utils/class_weight.py#L8)).



### Steps/Code to Reproduce

```
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(class_weight={1:10, 0:1})
rfc.fit([[0, 0, 1], [1, 0, 1]], [0, 0])
```



### Expected Results

No errors raised.

### Actual Results

`ValueError: Class label 1 not present.`


### Versions

```shell
System:
    python: 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)]
executable: c:\python3.9.2\python.exe
   machine: Windows-10-10.0.19041-SP0

Python dependencies:
          pip: 21.3.1
   setuptools: 58.3.0
      sklearn: 0.24.1
        numpy: 1.21.3
        scipy: 1.6.1
       Cython: 0.29.24
       pandas: 1.2.3
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```

COMMENT:
> I think the error message was designed to prevent user error:

To this end, could we potentially raise a warning to alert the user in such a case? That way they can determine whether it's the specific behaviour they expect or not.

> We can switch it around and let `y` define all the classes. In other words, we ignore class weights that is not in the `y`, but still error if `y` has a class that is not in `class_weight`.

I'm a fan of this.

Labels: Bug, Moderate, module:tree
ValueError when a key present in class_weight is not found in labels

### Describe the bug

While using `sklearn.ensemble.RandomForestClassifier` with a custom dictionary specified for `class_weight` parameter, a `ValueError` is raised when a key (label) present in `class_weight` is not found in any of the labels. I got this error while using `TimeSeriesSplit`. Specifically, if a split is such that training data doesn't have a label specified in `class_weight`, a `ValueError` will be raised. This stops users from evaluating their model since `class_weight` cannot be changed on a go after the model has been created. 

Is there any reason this check is being done? There's no error when we don't specify class_weight. The code raising this error can be found here ([link](https://github.com/scikit-learn/scikit-learn/blob/5900e2660f4293b52ea57bf830e7bff312779036/sklearn/utils/class_weight.py#L8)).



### Steps/Code to Reproduce

```
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(class_weight={1:10, 0:1})
rfc.fit([[0, 0, 1], [1, 0, 1]], [0, 0])
```



### Expected Results

No errors raised.

### Actual Results

`ValueError: Class label 1 not present.`


### Versions

```shell
System:
    python: 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)]
executable: c:\python3.9.2\python.exe
   machine: Windows-10-10.0.19041-SP0

Python dependencies:
          pip: 21.3.1
   setuptools: 58.3.0
      sklearn: 0.24.1
        numpy: 1.21.3
        scipy: 1.6.1
       Cython: 0.29.24
       pandas: 1.2.3
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```

COMMENT:
@thomasjpfan @Micky774 If anyone is not working on this at the moment could I take this up?

Labels: Bug, Moderate, module:tree
ValueError when a key present in class_weight is not found in labels

### Describe the bug

While using `sklearn.ensemble.RandomForestClassifier` with a custom dictionary specified for `class_weight` parameter, a `ValueError` is raised when a key (label) present in `class_weight` is not found in any of the labels. I got this error while using `TimeSeriesSplit`. Specifically, if a split is such that training data doesn't have a label specified in `class_weight`, a `ValueError` will be raised. This stops users from evaluating their model since `class_weight` cannot be changed on a go after the model has been created. 

Is there any reason this check is being done? There's no error when we don't specify class_weight. The code raising this error can be found here ([link](https://github.com/scikit-learn/scikit-learn/blob/5900e2660f4293b52ea57bf830e7bff312779036/sklearn/utils/class_weight.py#L8)).



### Steps/Code to Reproduce

```
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(class_weight={1:10, 0:1})
rfc.fit([[0, 0, 1], [1, 0, 1]], [0, 0])
```



### Expected Results

No errors raised.

### Actual Results

`ValueError: Class label 1 not present.`


### Versions

```shell
System:
    python: 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)]
executable: c:\python3.9.2\python.exe
   machine: Windows-10-10.0.19041-SP0

Python dependencies:
          pip: 21.3.1
   setuptools: 58.3.0
      sklearn: 0.24.1
        numpy: 1.21.3
        scipy: 1.6.1
       Cython: 0.29.24
       pandas: 1.2.3
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```

COMMENT:
@ss-is-master-chief You are welcome to open a PR. Note that the PR will require a non-regression test for this issue and another test for `compute_sample_weight` as we are adjusting it's behavior.

Labels: Bug, Moderate, module:tree
ValueError when a key present in class_weight is not found in labels

### Describe the bug

While using `sklearn.ensemble.RandomForestClassifier` with a custom dictionary specified for `class_weight` parameter, a `ValueError` is raised when a key (label) present in `class_weight` is not found in any of the labels. I got this error while using `TimeSeriesSplit`. Specifically, if a split is such that training data doesn't have a label specified in `class_weight`, a `ValueError` will be raised. This stops users from evaluating their model since `class_weight` cannot be changed on a go after the model has been created. 

Is there any reason this check is being done? There's no error when we don't specify class_weight. The code raising this error can be found here ([link](https://github.com/scikit-learn/scikit-learn/blob/5900e2660f4293b52ea57bf830e7bff312779036/sklearn/utils/class_weight.py#L8)).



### Steps/Code to Reproduce

```
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(class_weight={1:10, 0:1})
rfc.fit([[0, 0, 1], [1, 0, 1]], [0, 0])
```



### Expected Results

No errors raised.

### Actual Results

`ValueError: Class label 1 not present.`


### Versions

```shell
System:
    python: 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)]
executable: c:\python3.9.2\python.exe
   machine: Windows-10-10.0.19041-SP0

Python dependencies:
          pip: 21.3.1
   setuptools: 58.3.0
      sklearn: 0.24.1
        numpy: 1.21.3
        scipy: 1.6.1
       Cython: 0.29.24
       pandas: 1.2.3
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```

COMMENT:
Sorry for opening another PR. I was just holding onto my code for a couple nights as I was struggling with opening a PR (I was using a private fork...). 

> We can switch it around and let y define all the classes. In other words, we ignore class weights that is not in the y, but still error if y has a class that is not in class_weight.

I followed this in my PR, but I used `classes` as a source of truth as I understand it shouldn't have duplicates like `y`.

Labels: Bug, Moderate, module:tree
ValueError when a key present in class_weight is not found in labels

### Describe the bug

While using `sklearn.ensemble.RandomForestClassifier` with a custom dictionary specified for `class_weight` parameter, a `ValueError` is raised when a key (label) present in `class_weight` is not found in any of the labels. I got this error while using `TimeSeriesSplit`. Specifically, if a split is such that training data doesn't have a label specified in `class_weight`, a `ValueError` will be raised. This stops users from evaluating their model since `class_weight` cannot be changed on a go after the model has been created. 

Is there any reason this check is being done? There's no error when we don't specify class_weight. The code raising this error can be found here ([link](https://github.com/scikit-learn/scikit-learn/blob/5900e2660f4293b52ea57bf830e7bff312779036/sklearn/utils/class_weight.py#L8)).



### Steps/Code to Reproduce

```
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(class_weight={1:10, 0:1})
rfc.fit([[0, 0, 1], [1, 0, 1]], [0, 0])
```



### Expected Results

No errors raised.

### Actual Results

`ValueError: Class label 1 not present.`


### Versions

```shell
System:
    python: 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)]
executable: c:\python3.9.2\python.exe
   machine: Windows-10-10.0.19041-SP0

Python dependencies:
          pip: 21.3.1
   setuptools: 58.3.0
      sklearn: 0.24.1
        numpy: 1.21.3
        scipy: 1.6.1
       Cython: 0.29.24
       pandas: 1.2.3
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```

COMMENT:
> I followed this in my PR, but I used classes as a source of truth as I understand it shouldn't have duplicates like y.

Sorry if I was not clear. Functionally `compute_sample_weight` should use `classes`.

(In the end, `classes` is defined by `y`, so `y` is still indirectly the "source of truth")

Labels: Bug, Moderate, module:tree
Pooled variance for LDA estimators

### Describe the workflow you want to enable

We use LDA a lot in ERP-BCI research. So far we use our custom implementation of LDA as it makes use of pooled variance, i.e., subtract class-wise means from `X` and use all data to calculate the within class covariance matrix. In practice, this often leads to better performance, especially when classes are unbalanced and only few data is available.

### Describe your proposed solution

If there is interest, I could push a PR that enables this behaviour for the `eigen` and `lsqr` solvers by using a new parameter, e.g., `use_pooled_cov=False` which is passed to `_class_cov` and if it is true, first subtract class means and then estimate the covariance on all data.
I am not aware if there is an equivalent implementation for the `svd` solver.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I'm not a core maintainer, but this does sound interesting! Fair warning though, you'll likely need to provide benchmarks to verify that it is indeed an improvement, at least in certain cases. The standards for inclusion are fairly high due to the induced overhead and long-term maintenance costs.

Overall, aside from a core implementation, you'd also benefit from having:
1. Benchmarks which demonstrate at efficacy in at least certain situations (e.g. datasets with exceptionally high intra-class variance)
2. Testing, to confirm behavior -- this would also include potentially extending existing tests to cover your novel contribution
3. Associated documentation in both docstrings and an entry in the user guide

Of course if there are other interested parties, it could be a shared load, and if you have any questions on how to proceed I can also offer some help!

Labels: New Feature, Needs Decision - Include Feature
Pooled variance for LDA estimators

### Describe the workflow you want to enable

We use LDA a lot in ERP-BCI research. So far we use our custom implementation of LDA as it makes use of pooled variance, i.e., subtract class-wise means from `X` and use all data to calculate the within class covariance matrix. In practice, this often leads to better performance, especially when classes are unbalanced and only few data is available.

### Describe your proposed solution

If there is interest, I could push a PR that enables this behaviour for the `eigen` and `lsqr` solvers by using a new parameter, e.g., `use_pooled_cov=False` which is passed to `_class_cov` and if it is true, first subtract class means and then estimate the covariance on all data.
I am not aware if there is an equivalent implementation for the `svd` solver.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thank you for your input! I understand the need for documentation benchmarks etc. Currently I can see better performance for our spatiotemporal eeg data, but the data is, even for only a single subject, 650MB, so I will try to find generated data that has similar properties.

I will try to wrap up a draft PR in the next few days where I propose data / benchmarks. I think the actual implementation is fairly straightforward, at least for `lsqr` and `eigen`.

Labels: New Feature, Needs Decision - Include Feature
Deprecate the parameter `**kwds` in function `def paired_distances` since it is unused

### References
#22380 

Deprecate the parameter `**kwds` in function `def paired_distances` since it is unused:  
`def paired_distances(X, Y, *, metric="euclidean", **kwds):`

_Originally posted by @reshamas in https://github.com/scikit-learn/scikit-learn/issues/22380#issuecomment-1031844726_

COMMENT:
Why is `kwargs` not passed to `metric` when callable?

Labels: API, Needs Decision, module:metrics
Deprecate the parameter `**kwds` in function `def paired_distances` since it is unused

### References
#22380 

Deprecate the parameter `**kwds` in function `def paired_distances` since it is unused:  
`def paired_distances(X, Y, *, metric="euclidean", **kwds):`

_Originally posted by @reshamas in https://github.com/scikit-learn/scikit-learn/issues/22380#issuecomment-1031844726_

COMMENT:
> Why is `kwargs` not passed to `metric` when callable?

I guess it stays consistent w/ the other paired metrics, since they each only take `(X,Y)` as input, w/ no `**kwargs`. In general if it's a callable, I imagine it'll be sufficient to define any potential `**kwargs` in the actual body of the callable, or even use `functools.partial` if needed. 

If it really is the case that the user is expected to pass in a callable that specifically does not _rely_ on `**kwargs` being passed, then `**kwds` is truly unused, but idk what the original intention of the design was.

Labels: API, Needs Decision, module:metrics
Deprecate the parameter `**kwds` in function `def paired_distances` since it is unused

### References
#22380 

Deprecate the parameter `**kwds` in function `def paired_distances` since it is unused:  
`def paired_distances(X, Y, *, metric="euclidean", **kwds):`

_Originally posted by @reshamas in https://github.com/scikit-learn/scikit-learn/issues/22380#issuecomment-1031844726_

COMMENT:
During the triage meeting, we decided to not deprecated `**kwd` as it may be useful for metrics with kwrags such as `minkowski` in the future.

For now, we can document that it is unused for the current list of supported metrics.

Labels: API, Needs Decision, module:metrics
Deprecate the parameter `**kwds` in function `def paired_distances` since it is unused

### References
#22380 

Deprecate the parameter `**kwds` in function `def paired_distances` since it is unused:  
`def paired_distances(X, Y, *, metric="euclidean", **kwds):`

_Originally posted by @reshamas in https://github.com/scikit-learn/scikit-learn/issues/22380#issuecomment-1031844726_

COMMENT:
We can also fix the code and actually pass it to the callable metric

Labels: API, Needs Decision, module:metrics