Talip7/scikit-learn-issues-multilabel huggingface dataset:
Generalized Scores for Multi-Class Imbalanced Classification

### Describe the workflow you want to enable

We wish to extend Matthew coefficient and F1 to multi- class problems.

### Describe your proposed solution

We already generated a pypi package
https://pypi.org/project/matthew-Coef-MultiClass/  which supports this. 

We created a unique function that follows the signature pf metrics._classification.py  current solution for the binary case
I will be glad to add this solution 

### Describe alternatives you've considered, if relevant

The following functions performs the required  (as an alternative to use our own package):

```python
class AVERAG_TYPE(Enum):
    MATTHEW_GEN = 2
    F1_GEN = 3

def generalized_matthew(y_true,y_pred,labels,ave_type=AVERAG_TYPE.MATTHEW_GEN.value):

    # A utitliy function which normalizes the confusion and perfroms the relevant average
    def norm_confusion_mat(conf_mat, average, dimension_size):
        m1 = conf_mat / np.sum(conf_mat, axis=1)
        m2 = conf_mat / np.sum(conf_mat, axis=0)

        return [[average([m1[i, j], m2[j][i]])
                 for i in range(dimension_size)] for j in range(dimension_size)]

    # A utitliy function that perfroms the generaized score of F1 . it calculates the hamonic average
    # over the trace of the normalized confusion matrix
    def gen_f1_scalar_op(h_conf_mat):
        l_mat = len(h_conf_mat)
        return st.mstats.hmean([h_conf_mat[i][i] for i in range(l_mat)])

    dimension_size =len(labels)
    conf_mat=confusion_matrix(y_true=y_true,y_pred=y_pred ,labels=labels)
    if ave_type == AVERAG_TYPE.MATTHEW_GEN.value:
        av_func =st.mstats.gmean
        scalar_op =np.linalg.det
    elif ave_type ==AVERAG_TYPE.F1_GEN.value:
        av_func = st.mstats.hmean
        scalar_op =gen_f1_scalar_op

    G_mat= norm_confusion_mat(conf_mat, av_func, dimension_size)
    return scalar_op(G_mat)
```


### Additional context

I wish to stress that there is a sound mathematical work behind this generalization which is soon to be published . The paper will be gladly provided by either myself or Uri Itai

COMMENT:
> Do you mean this is not enough https://scholar.google.com/scholar?hl=en&as_sdt=0,5&q=geometric+mean

The proposed method is not just the geometric mean though, but rather the actual implementation and generalization of MCC/F1. There are reputable [MCC generalizations](https://www.sciencedirect.com/science/article/pii/S1476927104000799), and perhaps F1 as well though I don't know any off the top of my head. However, you're proposing a distinct mathematical approach that needs to be specifically validated.

> and indeed this one gave us  a hint how to move forward. I tjust didn prove
what we did and waiting for publishment .

As mentioned prior, scikit-learn is _very strict_ with regards to what methods we may adopt. Publishing isn't even necessarily enough, since you need e.g. 200 citations.

Labels: New Feature, Needs Decision - Include Feature
Generalized Scores for Multi-Class Imbalanced Classification

### Describe the workflow you want to enable

We wish to extend Matthew coefficient and F1 to multi- class problems.

### Describe your proposed solution

We already generated a pypi package
https://pypi.org/project/matthew-Coef-MultiClass/  which supports this. 

We created a unique function that follows the signature pf metrics._classification.py  current solution for the binary case
I will be glad to add this solution 

### Describe alternatives you've considered, if relevant

The following functions performs the required  (as an alternative to use our own package):

```python
class AVERAG_TYPE(Enum):
    MATTHEW_GEN = 2
    F1_GEN = 3

def generalized_matthew(y_true,y_pred,labels,ave_type=AVERAG_TYPE.MATTHEW_GEN.value):

    # A utitliy function which normalizes the confusion and perfroms the relevant average
    def norm_confusion_mat(conf_mat, average, dimension_size):
        m1 = conf_mat / np.sum(conf_mat, axis=1)
        m2 = conf_mat / np.sum(conf_mat, axis=0)

        return [[average([m1[i, j], m2[j][i]])
                 for i in range(dimension_size)] for j in range(dimension_size)]

    # A utitliy function that perfroms the generaized score of F1 . it calculates the hamonic average
    # over the trace of the normalized confusion matrix
    def gen_f1_scalar_op(h_conf_mat):
        l_mat = len(h_conf_mat)
        return st.mstats.hmean([h_conf_mat[i][i] for i in range(l_mat)])

    dimension_size =len(labels)
    conf_mat=confusion_matrix(y_true=y_true,y_pred=y_pred ,labels=labels)
    if ave_type == AVERAG_TYPE.MATTHEW_GEN.value:
        av_func =st.mstats.gmean
        scalar_op =np.linalg.det
    elif ave_type ==AVERAG_TYPE.F1_GEN.value:
        av_func = st.mstats.hmean
        scalar_op =gen_f1_scalar_op

    G_mat= norm_confusion_mat(conf_mat, av_func, dimension_size)
    return scalar_op(G_mat)
```


### Additional context

I wish to stress that there is a sound mathematical work behind this generalization which is soon to be published . The paper will be gladly provided by either myself or Uri Itai

COMMENT:
It means that when it is officially published it's a "go" from your end too?

On Mon, Aug 1, 2022 at 4:44 PM Thomas J. Fan ***@***.***>
wrote:

> I was referring to your comment about this working not being published:
>
> I wish to stress that there is a sound mathematical work behind this
> generalization which is soon to be published . The paper will be gladly
> provided by either myself or Uri Itai
>
> Papers only geometric mean is not enough. We require well-established
> published references that showcases the proposed method for computing the
> Matthew coefficient on multi-class problems.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/scikit-learn/scikit-learn/issues/24008#issuecomment-1201225124>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB7W2Z5S5D6R2K5LGEZX2NLVW7ICDANCNFSM54VCHWLA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

Labels: New Feature, Needs Decision - Include Feature
Generalized Scores for Multi-Class Imbalanced Classification

### Describe the workflow you want to enable

We wish to extend Matthew coefficient and F1 to multi- class problems.

### Describe your proposed solution

We already generated a pypi package
https://pypi.org/project/matthew-Coef-MultiClass/  which supports this. 

We created a unique function that follows the signature pf metrics._classification.py  current solution for the binary case
I will be glad to add this solution 

### Describe alternatives you've considered, if relevant

The following functions performs the required  (as an alternative to use our own package):

```python
class AVERAG_TYPE(Enum):
    MATTHEW_GEN = 2
    F1_GEN = 3

def generalized_matthew(y_true,y_pred,labels,ave_type=AVERAG_TYPE.MATTHEW_GEN.value):

    # A utitliy function which normalizes the confusion and perfroms the relevant average
    def norm_confusion_mat(conf_mat, average, dimension_size):
        m1 = conf_mat / np.sum(conf_mat, axis=1)
        m2 = conf_mat / np.sum(conf_mat, axis=0)

        return [[average([m1[i, j], m2[j][i]])
                 for i in range(dimension_size)] for j in range(dimension_size)]

    # A utitliy function that perfroms the generaized score of F1 . it calculates the hamonic average
    # over the trace of the normalized confusion matrix
    def gen_f1_scalar_op(h_conf_mat):
        l_mat = len(h_conf_mat)
        return st.mstats.hmean([h_conf_mat[i][i] for i in range(l_mat)])

    dimension_size =len(labels)
    conf_mat=confusion_matrix(y_true=y_true,y_pred=y_pred ,labels=labels)
    if ave_type == AVERAG_TYPE.MATTHEW_GEN.value:
        av_func =st.mstats.gmean
        scalar_op =np.linalg.det
    elif ave_type ==AVERAG_TYPE.F1_GEN.value:
        av_func = st.mstats.hmean
        scalar_op =gen_f1_scalar_op

    G_mat= norm_confusion_mat(conf_mat, av_func, dimension_size)
    return scalar_op(G_mat)
```


### Additional context

I wish to stress that there is a sound mathematical work behind this generalization which is soon to be published . The paper will be gladly provided by either myself or Uri Itai

COMMENT:
> It means that when it is officially published it's a "go" from your end too?

According to our [inclusion criterion](https://scikit-learn.org/dev/faq.html#what-are-the-inclusion-criteria-for-new-algorithms), the paper needs at least 3 years since publication, 200+ citations, and wide use.

Labels: New Feature, Needs Decision - Include Feature
Generalized Scores for Multi-Class Imbalanced Classification

### Describe the workflow you want to enable

We wish to extend Matthew coefficient and F1 to multi- class problems.

### Describe your proposed solution

We already generated a pypi package
https://pypi.org/project/matthew-Coef-MultiClass/  which supports this. 

We created a unique function that follows the signature pf metrics._classification.py  current solution for the binary case
I will be glad to add this solution 

### Describe alternatives you've considered, if relevant

The following functions performs the required  (as an alternative to use our own package):

```python
class AVERAG_TYPE(Enum):
    MATTHEW_GEN = 2
    F1_GEN = 3

def generalized_matthew(y_true,y_pred,labels,ave_type=AVERAG_TYPE.MATTHEW_GEN.value):

    # A utitliy function which normalizes the confusion and perfroms the relevant average
    def norm_confusion_mat(conf_mat, average, dimension_size):
        m1 = conf_mat / np.sum(conf_mat, axis=1)
        m2 = conf_mat / np.sum(conf_mat, axis=0)

        return [[average([m1[i, j], m2[j][i]])
                 for i in range(dimension_size)] for j in range(dimension_size)]

    # A utitliy function that perfroms the generaized score of F1 . it calculates the hamonic average
    # over the trace of the normalized confusion matrix
    def gen_f1_scalar_op(h_conf_mat):
        l_mat = len(h_conf_mat)
        return st.mstats.hmean([h_conf_mat[i][i] for i in range(l_mat)])

    dimension_size =len(labels)
    conf_mat=confusion_matrix(y_true=y_true,y_pred=y_pred ,labels=labels)
    if ave_type == AVERAG_TYPE.MATTHEW_GEN.value:
        av_func =st.mstats.gmean
        scalar_op =np.linalg.det
    elif ave_type ==AVERAG_TYPE.F1_GEN.value:
        av_func = st.mstats.hmean
        scalar_op =gen_f1_scalar_op

    G_mat= norm_confusion_mat(conf_mat, av_func, dimension_size)
    return scalar_op(G_mat)
```


### Additional context

I wish to stress that there is a sound mathematical work behind this generalization which is soon to be published . The paper will be gladly provided by either myself or Uri Itai

COMMENT:
Closing for now since this does not meet inclusion criteria.

Labels: New Feature, Needs Decision - Include Feature
KBinsDiscretizer crashes when strategy='kmeans'

### Describe the bug

Kernel is Jupyter notebook crashes when `strategy='kmeans'` is used with `KBinsDiscretizer`.

Versions:
3.8.12 (default, Oct 12 2021, 06:23:56) 
[Clang 10.0.0 ]
Python 3.8.12
numpy:  1.19.2
sklearn:  1.0.1

### Steps/Code to Reproduce

```py
# library
import numpy as np
from sklearn import preprocessing

# Feature array
x = np.array([
    [1, 2, 3, 4],
    [-1, -2, -3, -4],
    [4, 3, 2, 1],
    [0, 1, 0, 1]
])

# Fit and transform
est = preprocessing.KBinsDiscretizer(n_bins=[2, 2, 2, 2], encode='ordinal', strategy='kmeans')
est.fit_transform(x)

```

### Expected Results

No error

### Actual Results

Kernel crashes

### Versions

```shell
System:
    python: 3.8.12 (default, Oct 12 2021, 06:23:56)  [Clang 10.0.0 ]
executable: /opt/anaconda3/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 1.0.1
        numpy: 1.19.2
        scipy: 1.6.2
       Cython: 0.29.24
       pandas: 1.4.2
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
Weird, I get this error when running your snippet:
```
ValueError: n_bins must be a scalar or array of shape (n_features,).
```

Labels: Bug, Needs Triage
KBinsDiscretizer crashes when strategy='kmeans'

### Describe the bug

Kernel is Jupyter notebook crashes when `strategy='kmeans'` is used with `KBinsDiscretizer`.

Versions:
3.8.12 (default, Oct 12 2021, 06:23:56) 
[Clang 10.0.0 ]
Python 3.8.12
numpy:  1.19.2
sklearn:  1.0.1

### Steps/Code to Reproduce

```py
# library
import numpy as np
from sklearn import preprocessing

# Feature array
x = np.array([
    [1, 2, 3, 4],
    [-1, -2, -3, -4],
    [4, 3, 2, 1],
    [0, 1, 0, 1]
])

# Fit and transform
est = preprocessing.KBinsDiscretizer(n_bins=[2, 2, 2, 2], encode='ordinal', strategy='kmeans')
est.fit_transform(x)

```

### Expected Results

No error

### Actual Results

Kernel crashes

### Versions

```shell
System:
    python: 3.8.12 (default, Oct 12 2021, 06:23:56)  [Clang 10.0.0 ]
executable: /opt/anaconda3/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 1.0.1
        numpy: 1.19.2
        scipy: 1.6.2
       Cython: 0.29.24
       pandas: 1.4.2
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
In google colab it also returns an error:
`ValueError: n_bins must be a scalar or array of shape (n_features,).`

Python 3.7.13
scikit-learn (1.0.2)
joblib (1.1.0)
numpy (1.21.6)
threadpoolctl (3.1.0)
scipy (1.7.3)

Labels: Bug, Needs Triage
KBinsDiscretizer crashes when strategy='kmeans'

### Describe the bug

Kernel is Jupyter notebook crashes when `strategy='kmeans'` is used with `KBinsDiscretizer`.

Versions:
3.8.12 (default, Oct 12 2021, 06:23:56) 
[Clang 10.0.0 ]
Python 3.8.12
numpy:  1.19.2
sklearn:  1.0.1

### Steps/Code to Reproduce

```py
# library
import numpy as np
from sklearn import preprocessing

# Feature array
x = np.array([
    [1, 2, 3, 4],
    [-1, -2, -3, -4],
    [4, 3, 2, 1],
    [0, 1, 0, 1]
])

# Fit and transform
est = preprocessing.KBinsDiscretizer(n_bins=[2, 2, 2, 2], encode='ordinal', strategy='kmeans')
est.fit_transform(x)

```

### Expected Results

No error

### Actual Results

Kernel crashes

### Versions

```shell
System:
    python: 3.8.12 (default, Oct 12 2021, 06:23:56)  [Clang 10.0.0 ]
executable: /opt/anaconda3/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 1.0.1
        numpy: 1.19.2
        scipy: 1.6.2
       Cython: 0.29.24
       pandas: 1.4.2
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
Sorry updated the  line of code to `n_bins=[2, 2, 2, 2]`.  This should resolve the error for array shape.

Labels: Bug, Needs Triage
KBinsDiscretizer crashes when strategy='kmeans'

### Describe the bug

Kernel is Jupyter notebook crashes when `strategy='kmeans'` is used with `KBinsDiscretizer`.

Versions:
3.8.12 (default, Oct 12 2021, 06:23:56) 
[Clang 10.0.0 ]
Python 3.8.12
numpy:  1.19.2
sklearn:  1.0.1

### Steps/Code to Reproduce

```py
# library
import numpy as np
from sklearn import preprocessing

# Feature array
x = np.array([
    [1, 2, 3, 4],
    [-1, -2, -3, -4],
    [4, 3, 2, 1],
    [0, 1, 0, 1]
])

# Fit and transform
est = preprocessing.KBinsDiscretizer(n_bins=[2, 2, 2, 2], encode='ordinal', strategy='kmeans')
est.fit_transform(x)

```

### Expected Results

No error

### Actual Results

Kernel crashes

### Versions

```shell
System:
    python: 3.8.12 (default, Oct 12 2021, 06:23:56)  [Clang 10.0.0 ]
executable: /opt/anaconda3/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 1.0.1
        numpy: 1.19.2
        scipy: 1.6.2
       Cython: 0.29.24
       pandas: 1.4.2
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
Your snippet runs fine for me (note: I remove the `.toarray` from your original snippet guessing that it is not related to the issue here).

I am going to wild-guess this may be OSX-dependent and I'll let people with more knowledge weigh in. This may be similar to https://github.com/scikit-learn/scikit-learn/issues/23574 actually which reports a OSX segmentation fault with KMeans on a similar machine as yours.

Labels: Bug, Needs Triage
KBinsDiscretizer crashes when strategy='kmeans'

### Describe the bug

Kernel is Jupyter notebook crashes when `strategy='kmeans'` is used with `KBinsDiscretizer`.

Versions:
3.8.12 (default, Oct 12 2021, 06:23:56) 
[Clang 10.0.0 ]
Python 3.8.12
numpy:  1.19.2
sklearn:  1.0.1

### Steps/Code to Reproduce

```py
# library
import numpy as np
from sklearn import preprocessing

# Feature array
x = np.array([
    [1, 2, 3, 4],
    [-1, -2, -3, -4],
    [4, 3, 2, 1],
    [0, 1, 0, 1]
])

# Fit and transform
est = preprocessing.KBinsDiscretizer(n_bins=[2, 2, 2, 2], encode='ordinal', strategy='kmeans')
est.fit_transform(x)

```

### Expected Results

No error

### Actual Results

Kernel crashes

### Versions

```shell
System:
    python: 3.8.12 (default, Oct 12 2021, 06:23:56)  [Clang 10.0.0 ]
executable: /opt/anaconda3/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 1.0.1
        numpy: 1.19.2
        scipy: 1.6.2
       Cython: 0.29.24
       pandas: 1.4.2
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
@nileshingle for future reference, posting the output of `conda list` may be useful.

It could be useful as well if you manage to run your snippet as script and attach a debugger by following https://github.com/scikit-learn/scikit-learn/issues/23574#issuecomment-1152379936 (this may need to be adapted a bit according to https://github.com/scikit-learn/scikit-learn/issues/23574#issuecomment-1152620466)

Labels: Bug, Needs Triage
KBinsDiscretizer crashes when strategy='kmeans'

### Describe the bug

Kernel is Jupyter notebook crashes when `strategy='kmeans'` is used with `KBinsDiscretizer`.

Versions:
3.8.12 (default, Oct 12 2021, 06:23:56) 
[Clang 10.0.0 ]
Python 3.8.12
numpy:  1.19.2
sklearn:  1.0.1

### Steps/Code to Reproduce

```py
# library
import numpy as np
from sklearn import preprocessing

# Feature array
x = np.array([
    [1, 2, 3, 4],
    [-1, -2, -3, -4],
    [4, 3, 2, 1],
    [0, 1, 0, 1]
])

# Fit and transform
est = preprocessing.KBinsDiscretizer(n_bins=[2, 2, 2, 2], encode='ordinal', strategy='kmeans')
est.fit_transform(x)

```

### Expected Results

No error

### Actual Results

Kernel crashes

### Versions

```shell
System:
    python: 3.8.12 (default, Oct 12 2021, 06:23:56)  [Clang 10.0.0 ]
executable: /opt/anaconda3/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 1.0.1
        numpy: 1.19.2
        scipy: 1.6.2
       Cython: 0.29.24
       pandas: 1.4.2
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
Closing. Thank you.

Labels: Bug, Needs Triage
Remove `sign_flip` from FastICA

From https://github.com/scikit-learn/scikit-learn/pull/23935#discussion_r922842213, we may want to remove `sign_flip` from `FastICA`:

https://github.com/scikit-learn/scikit-learn/blob/f0af898de5bfa93916b659438b4cf0b86fffd974/sklearn/decomposition/_fastica.py#L264


This parameter was not released yet, so we can remove it without any deprecation.

COMMENT:
I just made a PR.

Labels: Blocker, module:decomposition
Add option to RocCurveDisplay to display the average of different length ROC curves

### Describe the workflow you want to enable

When using k-fold cross-validation the resulting ROC curves can vary in length if there are a different number of positive and/or negative samples in each fold. I would like to add an option to sklearn.metrics.RocCurveDisplay to display the average of different length ROC curves.

Section 8 of [An introduction to ROC analysis](https://ccrma.stanford.edu/workshops/mir2009/references/ROCintro.pdf) by Fawcett describes the method.

### Describe your proposed solution

```python
def average_roc_curves(roc_curves_list,):
    """
    This function takes the average of different length ROC curves returned from 
    sklearn.metrics.roc_curve. The function expects a list of fpr, tpr, and 
    thresholds. The function subsamples from these lists and returns an 
    average_fpr, average_tpr, and average_thresholds.

    Parameters
    ----------
    roc_curves_list : list
        A list of fpr, tpr, and thresholds
    Returns
    -------
    average_fpr : list
        A list of averaged false positive rates
    average_tpr : list
        A list of averaged true positive rates
    average_thresholds : list
        A list of averaged thresholds
    """
    kfolds = len(roc_curves_list)
    min_length = np.min(list(len(roc_curves_list[k][2]) for k in range(kfolds)))
    shortened_roc_curves_list = list()
    for k in range(kfolds):
        fpr, tpr, thresholds = roc_curves_list[k]
        indicies = list(i for i in range(len(thresholds)))
        selected_indicies = np.sort(np.random.choice(indicies, min_length, replace=False))
        shortened_roc_curves_list.append([fpr[selected_indicies], tpr[selected_indicies], thresholds[selected_indicies]])
    average_fpr, average_tpr, average_thresholds = np.mean(shortened_roc_curves_list, axis=0)
    return average_fpr, average_tpr, average_thresholds
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Every feature we include has a [maintenance cost](http://scikit-learn.org/dev/faq.html#why-are-you-so-selective-on-what-algorithms-you-include-in-scikit-learn). Our maintainers are mostly volunteers. For a new feature to be included, we need evidence that it is often useful and, ideally, [well-established](http://scikit-learn.org/dev/faq.html#what-are-the-inclusion-criteria-for-new-algorithms) in the literature or in practice.

Can you provide a reference on this method for averaging ROC curves?

Labels: New Feature, module:metrics, Needs Decision - Include Feature