Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add the baseline corrected accuracy score for (multi-class) classification to sklearn.metrics

### Describe the workflow you want to enable

Would it be possible to add a new score to `sklearn.metrics`, namely the baseline corrected accuracy score (BCAS) ([DOI:10.5281/zenodo.15262049](https://doi.org/10.5281/zenodo.15262049)). The proposed metric quantifies the model improvement w.r.t. the baseline, and represents a direct evaluation of classifier performance. See the proposed code below, which is label agnostic, and is suitable for both binary and multi-class classification.

### Describe your proposed solution

```
import numpy as np

def BCAS(y_true, y_pred):
    """Baseline corrected accuracy score (BCAS).

    Parameters
    ----------
    y_true : Ground truth (correct) labels.

    y_pred : Predicted labels.

    Returns
    -------
    score : float
    """
    label, count = np.unique(y_true, return_counts=True)
    most_frequent_class = label[np.argmax(count)]
    y_baseline = np.full(len(y_true), most_frequent_class)
    as_baseline = np.mean(y_true == y_baseline)
    as_predicted = np.mean(y_true == y_pred)
    return (as_predicted - as_baseline)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thanks for your suggestion. However, new feature in scikit-learn are subject to the following inclusion criterion:

- https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms

As far as I can see, what you propose is not standard enough in the ML community to be included as such the project's code base.

Labels: New Feature
SVC Sigmoid sometimes ROC AUC from predict_proba & decision_function are each other's inverse

### Describe the bug

Uncertain if this is a bug or counter-intuitive expected behavior.

Under certain circumstances the ROC AUC calculated for `SVC` with the `sigmoid` kernel will not agree depending on if you use `predict_proba` or `decision_function`. In fact, they will be nearly `1-other_method_auc`.

This was noticed when comparing ROC AUC calculated using `roc_auc_score` with predictions from `predict_proba(X)[:, 1]` to using the scorer from `get_scorer('roc_auc')` which appears to be calling `roc_auc_score` with scores from `decision_function`. 

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, get_scorer
from sklearn.model_selection import train_test_split

n_samples = 100
n_features = 100
random_state = 123
rng = np.random.default_rng(random_state)

X = rng.normal(loc=0.0, scale=1.0, size=(n_samples, n_features))
y = rng.integers(0, 2, size=n_samples)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=random_state)

svc_params = {
    "kernel": "sigmoid",
    "probability": True,
    "random_state":random_state,
}   
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('svc', SVC(**svc_params))
])  
pipeline.fit(X_train, y_train)
y_proba = pipeline.predict_proba(X_test)[:, 1]
y_dec = pipeline.decision_function(X_test)
roc_auc_proba = roc_auc_score(y_test, y_proba)
roc_auc_dec = roc_auc_score(y_test, y_dec)
auc_scorer = get_scorer('roc_auc')
scorer_auc = auc_scorer(pipeline, X_test, y_test)

print(f"AUC (roc_auc_score from predict_proba) = {roc_auc_proba:.4f}")
print(f"AUC (roc_auc_score from decision_function) = {roc_auc_dec:.4f}")
print(f"AUC (get_scorer) = {scorer_auc:.4f}")
```

### Expected Results

The measures of ROC AUC agree

### Actual Results

```shell
AUC (roc_auc_score from predict_proba) = 0.5833
AUC (roc_auc_score from decision_function) = 0.4295
AUC (get_scorer) = 0.4295
```

### Versions

```shell
System:
    python: 3.11.5

Python dependencies:
      sklearn: 1.7.dev0
          pip: 25.0.1
   setuptools: 65.5.0
        numpy: 1.26.4
        scipy: 1.15.2
       Cython: 3.0.12
       pandas: 2.2.3
   matplotlib: 3.10.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
cc @snath-xoc @antoinebaker

Labels: Bug, Needs Investigation
SVC Sigmoid sometimes ROC AUC from predict_proba & decision_function are each other's inverse

### Describe the bug

Uncertain if this is a bug or counter-intuitive expected behavior.

Under certain circumstances the ROC AUC calculated for `SVC` with the `sigmoid` kernel will not agree depending on if you use `predict_proba` or `decision_function`. In fact, they will be nearly `1-other_method_auc`.

This was noticed when comparing ROC AUC calculated using `roc_auc_score` with predictions from `predict_proba(X)[:, 1]` to using the scorer from `get_scorer('roc_auc')` which appears to be calling `roc_auc_score` with scores from `decision_function`. 

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, get_scorer
from sklearn.model_selection import train_test_split

n_samples = 100
n_features = 100
random_state = 123
rng = np.random.default_rng(random_state)

X = rng.normal(loc=0.0, scale=1.0, size=(n_samples, n_features))
y = rng.integers(0, 2, size=n_samples)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=random_state)

svc_params = {
    "kernel": "sigmoid",
    "probability": True,
    "random_state":random_state,
}   
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('svc', SVC(**svc_params))
])  
pipeline.fit(X_train, y_train)
y_proba = pipeline.predict_proba(X_test)[:, 1]
y_dec = pipeline.decision_function(X_test)
roc_auc_proba = roc_auc_score(y_test, y_proba)
roc_auc_dec = roc_auc_score(y_test, y_dec)
auc_scorer = get_scorer('roc_auc')
scorer_auc = auc_scorer(pipeline, X_test, y_test)

print(f"AUC (roc_auc_score from predict_proba) = {roc_auc_proba:.4f}")
print(f"AUC (roc_auc_score from decision_function) = {roc_auc_dec:.4f}")
print(f"AUC (get_scorer) = {scorer_auc:.4f}")
```

### Expected Results

The measures of ROC AUC agree

### Actual Results

```shell
AUC (roc_auc_score from predict_proba) = 0.5833
AUC (roc_auc_score from decision_function) = 0.4295
AUC (get_scorer) = 0.4295
```

### Versions

```shell
System:
    python: 3.11.5

Python dependencies:
      sklearn: 1.7.dev0
          pip: 25.0.1
   setuptools: 65.5.0
        numpy: 1.26.4
        scipy: 1.15.2
       Cython: 3.0.12
       pandas: 2.2.3
   matplotlib: 3.10.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
I was able to get the same results on my side as well, thank you @arhall0. I agree, this behaviour may be expected and possibly we should deprecate the ```probability=True``` option. This would also save a lot of confusion when using sample weighting.

Labels: Bug, Needs Investigation
SVC Sigmoid sometimes ROC AUC from predict_proba & decision_function are each other's inverse

### Describe the bug

Uncertain if this is a bug or counter-intuitive expected behavior.

Under certain circumstances the ROC AUC calculated for `SVC` with the `sigmoid` kernel will not agree depending on if you use `predict_proba` or `decision_function`. In fact, they will be nearly `1-other_method_auc`.

This was noticed when comparing ROC AUC calculated using `roc_auc_score` with predictions from `predict_proba(X)[:, 1]` to using the scorer from `get_scorer('roc_auc')` which appears to be calling `roc_auc_score` with scores from `decision_function`. 

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, get_scorer
from sklearn.model_selection import train_test_split

n_samples = 100
n_features = 100
random_state = 123
rng = np.random.default_rng(random_state)

X = rng.normal(loc=0.0, scale=1.0, size=(n_samples, n_features))
y = rng.integers(0, 2, size=n_samples)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=random_state)

svc_params = {
    "kernel": "sigmoid",
    "probability": True,
    "random_state":random_state,
}   
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('svc', SVC(**svc_params))
])  
pipeline.fit(X_train, y_train)
y_proba = pipeline.predict_proba(X_test)[:, 1]
y_dec = pipeline.decision_function(X_test)
roc_auc_proba = roc_auc_score(y_test, y_proba)
roc_auc_dec = roc_auc_score(y_test, y_dec)
auc_scorer = get_scorer('roc_auc')
scorer_auc = auc_scorer(pipeline, X_test, y_test)

print(f"AUC (roc_auc_score from predict_proba) = {roc_auc_proba:.4f}")
print(f"AUC (roc_auc_score from decision_function) = {roc_auc_dec:.4f}")
print(f"AUC (get_scorer) = {scorer_auc:.4f}")
```

### Expected Results

The measures of ROC AUC agree

### Actual Results

```shell
AUC (roc_auc_score from predict_proba) = 0.5833
AUC (roc_auc_score from decision_function) = 0.4295
AUC (get_scorer) = 0.4295
```

### Versions

```shell
System:
    python: 3.11.5

Python dependencies:
      sklearn: 1.7.dev0
          pip: 25.0.1
   setuptools: 65.5.0
        numpy: 1.26.4
        scipy: 1.15.2
       Cython: 3.0.12
       pandas: 2.2.3
   matplotlib: 3.10.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
Hi! 👋

I’d like to help investigate and potentially work on a fix or documentation clarification for this issue. I've reproduced the problem and am looking into how `SVC(probability=True)` performs Platt-scaling internally versus `CalibratedClassifierCV`.

I’ll share my findings shortly and would appreciate any suggestions or directions you might have in the meantime.

Thanks!

Labels: Bug, Needs Investigation
Add Categorical Feature Support to `IterativeImputer`

### Describe the workflow you want to enable

I want to impute missing values in categorical columns using a similar approach to `IterativeImputer`, which currently works only for continuous data. Specifically, I want to enable the following workflow:

- Identify and handle categorical columns in the dataset
- Use classifier models (e.g., RandomForestClassifier) to impute missing values in categorical columns based on other features
- Integrate with existing pipelines seamlessly, without needing to separate and impute categorical columns manually

### Describe your proposed solution

Extend the current `IterativeImputer` class (or create a new class, such as `IterativeCategoricalImputer`) to handle categorical data:

- Detect categorical columns automatically (e.g., using `dtype='object'` or `category`) or accept them via a `categorical_features` parameter
- Encode the categorical variables using an internal encoder (e.g., `LabelEncoder`)
- Use a classifier model (e.g., `RandomForestClassifier`) instead of a regression model for those columns
- Predict only the missing values, then inverse transform the predictions back to the original categories

This would enable more robust and automatic preprocessing for datasets that have numeric and categorical features.

### Describe alternatives you've considered, if relevant

- Manually encoding categorical variables, using a classifier-based imputation strategy outside of `IterativeImputer`
- Using other libraries like `autoimpute` or `fancyimpute`, which support mixed-type imputation but lack full integration with Scikit-learn pipelines
- Creating separate imputation steps for categorical and numeric features and merging them later, which adds complexity and can introduce data leakage risks

None of these are as clean or pipeline-friendly as a built-in solution.


### Additional context

This feature would make `IterativeImputer` more powerful and suitable for real-world datasets that include both numeric and categorical variables with missing values.

I'm currently working with datasets like this in my own data science projects and would be interested in contributing a prototype or proof-of-concept if this idea is accepted by the maintainers.

COMMENT:
Maybe @GaelVaroquaux has some opinions to share on this.

Labels: New Feature
Add Categorical Feature Support to `IterativeImputer`

### Describe the workflow you want to enable

I want to impute missing values in categorical columns using a similar approach to `IterativeImputer`, which currently works only for continuous data. Specifically, I want to enable the following workflow:

- Identify and handle categorical columns in the dataset
- Use classifier models (e.g., RandomForestClassifier) to impute missing values in categorical columns based on other features
- Integrate with existing pipelines seamlessly, without needing to separate and impute categorical columns manually

### Describe your proposed solution

Extend the current `IterativeImputer` class (or create a new class, such as `IterativeCategoricalImputer`) to handle categorical data:

- Detect categorical columns automatically (e.g., using `dtype='object'` or `category`) or accept them via a `categorical_features` parameter
- Encode the categorical variables using an internal encoder (e.g., `LabelEncoder`)
- Use a classifier model (e.g., `RandomForestClassifier`) instead of a regression model for those columns
- Predict only the missing values, then inverse transform the predictions back to the original categories

This would enable more robust and automatic preprocessing for datasets that have numeric and categorical features.

### Describe alternatives you've considered, if relevant

- Manually encoding categorical variables, using a classifier-based imputation strategy outside of `IterativeImputer`
- Using other libraries like `autoimpute` or `fancyimpute`, which support mixed-type imputation but lack full integration with Scikit-learn pipelines
- Creating separate imputation steps for categorical and numeric features and merging them later, which adds complexity and can introduce data leakage risks

None of these are as clean or pipeline-friendly as a built-in solution.


### Additional context

This feature would make `IterativeImputer` more powerful and suitable for real-world datasets that include both numeric and categorical variables with missing values.

I'm currently working with datasets like this in my own data science projects and would be interested in contributing a prototype or proof-of-concept if this idea is accepted by the maintainers.

COMMENT:
Hello,

I’d love to help implement thisl feature.

Would it be alright if I take this on? Thanks!

Labels: New Feature
Add Categorical Feature Support to `IterativeImputer`

### Describe the workflow you want to enable

I want to impute missing values in categorical columns using a similar approach to `IterativeImputer`, which currently works only for continuous data. Specifically, I want to enable the following workflow:

- Identify and handle categorical columns in the dataset
- Use classifier models (e.g., RandomForestClassifier) to impute missing values in categorical columns based on other features
- Integrate with existing pipelines seamlessly, without needing to separate and impute categorical columns manually

### Describe your proposed solution

Extend the current `IterativeImputer` class (or create a new class, such as `IterativeCategoricalImputer`) to handle categorical data:

- Detect categorical columns automatically (e.g., using `dtype='object'` or `category`) or accept them via a `categorical_features` parameter
- Encode the categorical variables using an internal encoder (e.g., `LabelEncoder`)
- Use a classifier model (e.g., `RandomForestClassifier`) instead of a regression model for those columns
- Predict only the missing values, then inverse transform the predictions back to the original categories

This would enable more robust and automatic preprocessing for datasets that have numeric and categorical features.

### Describe alternatives you've considered, if relevant

- Manually encoding categorical variables, using a classifier-based imputation strategy outside of `IterativeImputer`
- Using other libraries like `autoimpute` or `fancyimpute`, which support mixed-type imputation but lack full integration with Scikit-learn pipelines
- Creating separate imputation steps for categorical and numeric features and merging them later, which adds complexity and can introduce data leakage risks

None of these are as clean or pipeline-friendly as a built-in solution.


### Additional context

This feature would make `IterativeImputer` more powerful and suitable for real-world datasets that include both numeric and categorical variables with missing values.

I'm currently working with datasets like this in my own data science projects and would be interested in contributing a prototype or proof-of-concept if this idea is accepted by the maintainers.

COMMENT:
@ogrisel

Labels: New Feature
Add P4 classification metric

### Describe the workflow you want to enable

Hi, while working on a classification problem I found out there is no dedicated function to compute the P4 metric implemented in sklearn. As a reminder, P4 metrics is a binary classification metric that is commonly seen as an extension of the f_beta metrics because it takes into account all four True Positive, False Positive, True Negative and False Negative values, and because is it symmetrical unlike the f_beta metrics.

P4 is defined as follows : P4 = 4 / ( 1/precision + 1/recall + 1/specificity + 1/NPV )

Wikipedia page right [here](https://en.wikipedia.org/wiki/P4-metric)

Medium article right [there](https://medium.com/@thomas.vidori/better-than-the-f1-score-discover-the-p-4-score-903242e9545b)


### Describe your proposed solution

My idea was to create a function `p4_support` similar to `precision_recall_fscore_support`. Since it is a binary metric, multiclass and multi-label inputs would be managed with `multilabel_confusion_matrix` so the arguments for `average` would be `'macro', 'samples', 'weighted', 'binary', None`.
I would compute all necessaries values such as 1/precision, 1/recall, 1/specificity and 1/NPV using `_prf_divide`. If any of these four ratios are zero divisions, then P4 would also return the zero division argument. Indeed, for example if precision is null, then 1/precision is +inf and the whole denominator of the P4 is +inf which make P4 = 0 (Btw, this behavior is a reason why it is harder to achieve a high P4 score than f_score since all four ratios need to be 1 to have a P4 equals to 1.). The function would return the tuple (p4_value, support)

A second function `p4_score` which would be the one actually used by users would return only the first element of the previously described `p4_support` function.

### Describe alternatives you've considered, if relevant

Extras : 

Since specificity and NVP are computed anyway, the `p4_support` function could return the tuple (specificity, NVP, p4_score, support) and then be called `specificity_nvp_p4_support`. It would then also be possible to add `specificity` and `NVP` functions as well using the same scheme as precision or recall. 

Responding to #21000 issue, P4 could be added in the `classification_report` function and would be a good summary of all TP, FP, TN, FN values and their combinations. 

### Additional context

I have checked that this feature is not already in the issues or pull requests.

COMMENT:
I propose to close this feature request as "not planned" for now. If people disagree with what I wrote above, please feel free to upvote this issue and comment below to extend the analysis and we can consider reopening once the inclusion criteria are met.

Labels: New Feature
Issues with pairwise_distances(metric='euclidean') when used on the output of UMAP

### Describe the bug

When using pairwise_distances with metric='euclidean' on the output of some data from a UMAP, a `RuntimeWarning: divide by zero encountered in matmul ret = a @ b` is raised. This warning is not raised if you just use pairwise_distances on some normally distributed values of the same dimension, it specifically happens when used on the output of UMAP. The warning is not raised if calling `scipy.pdist` on the same data. The warning doesn't come up with any other metric (other than euclidean family e.g. nan_euclidean etc)

### Steps/Code to Reproduce

```python
import numpy as np
import umap.umap_ as umap 
from sklearn.metrics import pairwise_distances

np.random.seed(42)
arr = np.random.normal(size = (300, 10))
reducer = umap.UMAP()
embedding = reducer.fit_transform(arr)

# this line produces the warning 
dist_mat = pairwise_distances(embedding, metric = 'euclidean')

# no warning produced by this code
synthetic = np.random.normal(size = (300, 2))
dist_mat_synth = pairwise_distances(synthetic, metric = 'euclidean')
```

### Expected Results

Would expect to see no RuntimeWarning (FutureWarning is expected)

### Actual Results

``` 

[.../site-packages/sklearn/utils/deprecation.py:151]: FutureWarning: 'force_all_finite' was renamed to 'ensure_all_finite' in 1.6 and will be removed in 1.8.
  warnings.warn(
[.../site-packages/sklearn/utils/extmath.py:203] RuntimeWarning: divide by zero encountered in matmul
  ret = a @ b
[.../site-packages/sklearn/utils/extmath.py:203]: RuntimeWarning: overflow encountered in matmul
  ret = a @ b
[.../site-packages/sklearn/utils/extmath.py:203]: RuntimeWarning: invalid value encountered in matmul
  ret = a @ b

```

### Versions

```shell
System:
    python: 3.10.16 (main, Feb 25 2025, 09:29:51) [Clang 16.0.0 (clang-1600.0.26.6)]
executable: .../bin/python
   machine: macOS-15.4-arm64-arm-64bit

Python dependencies:
      sklearn: 1.6.1
          pip: 25.0.1
   setuptools: 65.5.0
        numpy: 2.1.3
        scipy: 1.15.1
       Cython: None
       pandas: 2.2.3
   matplotlib: 3.10.0
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 14
         prefix: libomp
       filepath: .../site-packages/sklearn/.dylibs/libomp.dylib
        version: None

       user_api: openmp
   internal_api: openmp
    num_threads: 14
         prefix: libomp
       filepath: /opt/homebrew/Cellar/libomp/19.1.7/lib/libomp.dylib
        version: None
```

COMMENT:
I can work on this issue

Labels: Bug, Needs Investigation