Talip7/scikit-learn-issues-multilabel huggingface dataset:
TunedThreasholdClassiffierCV not understanding `func(y_pred, y_true, ...)` as a valid `scoring`

This code

```py
from sklearn.model_selection import TunedThresholdClassifierCV
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
import sklearn
import numpy as np

sklearn.set_config(enable_metadata_routing=True)

def my_metric(y_true, y_pred, sample_weight=None):
    assert sample_weight is not None
    return np.mean(y_pred)

X, y = make_classification(random_state=0)
sample_weight = np.random.rand(len(y))

est = TunedThresholdClassifierCV(LogisticRegression(), cv=2, scoring=my_metric)
est.fit(X, y, sample_weight=sample_weight)
```

gives this:

```py
Traceback (most recent call last):
  File "/tmp/2.py", line 17, in <module>
    est.fit(X, y, sample_weight=sample_weight)
  File "/path/to/scikit-learn/sklearn/base.py", line 1366, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/scikit-learn/sklearn/model_selection/_classification_threshold.py", line 129, in fit
    self._fit(X, y, **params)
  File "/path/to/scikit-learn/sklearn/model_selection/_classification_threshold.py", line 742, in _fit
    routed_params = process_routing(self, "fit", **params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/scikit-learn/sklearn/utils/_metadata_requests.py", line 1636, in process_routing
    request_routing = get_routing_for_object(_obj)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/scikit-learn/sklearn/utils/_metadata_requests.py", line 1197, in get_routing_for_object
    return deepcopy(obj.get_metadata_routing())
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/scikit-learn/sklearn/model_selection/_classification_threshold.py", line 871, in get_metadata_routing
    scorer=self._get_curve_scorer(),
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/scikit-learn/sklearn/model_selection/_classification_threshold.py", line 880, in _get_curve_scorer
    curve_scorer = _CurveScorer.from_scorer(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/scikit-learn/sklearn/metrics/_scorer.py", line 1108, in from_scorer
    score_func=scorer._score_func,
               ^^^^^^^^^^^^^^^^^^
AttributeError: 'function' object has no attribute '_score_func'
```

cc @glemaitre

COMMENT:
Closing this as it's expected. The `check_scoring` is a different issue.

Labels: Bug
`SVC(probability=True)`  is not thread-safe

This was discovered while running:

```
pytest -v --parallel-threads=4 --iterations=2 sklearn/svm/tests/test_sparse.py
```

before including the fix pushed to #30041 under https://github.com/scikit-learn/scikit-learn/pull/30041/commits/bce2b4eb7d5ab49cf758f98c667e86243883d1de.

I suspect the problem is that the built-in Platt scaling implementation of the vendored C++ code base of libsvm that uses a singleton pseudo random generator. Therefore, seeding the shared RNG state from competing threads prevents getting reproducible results and hence the test failure.

COMMENT:
We could spend time in crafting a minimal reproducer and trying to craft a fix, but since the Platt scaling implementation of `SVC` is also problematic for at least two other reasons (#16298, #31222), I would rather just deprecate the `probability=True` option of that estimator.

Labels: Bug, Moderate
`SVC(probability=True)`  is not thread-safe

This was discovered while running:

```
pytest -v --parallel-threads=4 --iterations=2 sklearn/svm/tests/test_sparse.py
```

before including the fix pushed to #30041 under https://github.com/scikit-learn/scikit-learn/pull/30041/commits/bce2b4eb7d5ab49cf758f98c667e86243883d1de.

I suspect the problem is that the built-in Platt scaling implementation of the vendored C++ code base of libsvm that uses a singleton pseudo random generator. Therefore, seeding the shared RNG state from competing threads prevents getting reproducible results and hence the test failure.

COMMENT:
Is the modern correct way to get the same feature via a `CalibratedClassifierCV`? Does that take around the same time as `probability=True`? I'm wondering if `probability=True` is helpful enough that it's worth keeping?

Labels: Bug, Moderate
`SVC(probability=True)`  is not thread-safe

This was discovered while running:

```
pytest -v --parallel-threads=4 --iterations=2 sklearn/svm/tests/test_sparse.py
```

before including the fix pushed to #30041 under https://github.com/scikit-learn/scikit-learn/pull/30041/commits/bce2b4eb7d5ab49cf758f98c667e86243883d1de.

I suspect the problem is that the built-in Platt scaling implementation of the vendored C++ code base of libsvm that uses a singleton pseudo random generator. Therefore, seeding the shared RNG state from competing threads prevents getting reproducible results and hence the test failure.

COMMENT:
Hi I’m interested in contributing and would like to help with this thread-safety issue.  
I can try creating a minimal reproducible example or help with testing fixes.  
Please let me know how I can assist! Thanks

Labels: Bug, Moderate
`SVC(probability=True)`  is not thread-safe

This was discovered while running:

```
pytest -v --parallel-threads=4 --iterations=2 sklearn/svm/tests/test_sparse.py
```

before including the fix pushed to #30041 under https://github.com/scikit-learn/scikit-learn/pull/30041/commits/bce2b4eb7d5ab49cf758f98c667e86243883d1de.

I suspect the problem is that the built-in Platt scaling implementation of the vendored C++ code base of libsvm that uses a singleton pseudo random generator. Therefore, seeding the shared RNG state from competing threads prevents getting reproducible results and hence the test failure.

COMMENT:
They should be equivalent (up to different RNG / CV splits). +1 for deprecating `SVC(probability=True)` with a message that points to `CalibratedClassifierCV` instead.

Labels: Bug, Moderate
`SVC(probability=True)`  is not thread-safe

This was discovered while running:

```
pytest -v --parallel-threads=4 --iterations=2 sklearn/svm/tests/test_sparse.py
```

before including the fix pushed to #30041 under https://github.com/scikit-learn/scikit-learn/pull/30041/commits/bce2b4eb7d5ab49cf758f98c667e86243883d1de.

I suspect the problem is that the built-in Platt scaling implementation of the vendored C++ code base of libsvm that uses a singleton pseudo random generator. Therefore, seeding the shared RNG state from competing threads prevents getting reproducible results and hence the test failure.

COMMENT:
Hi, I am interested in solving this :). Could I be assigned?

Labels: Bug, Moderate
`SVC(probability=True)`  is not thread-safe

This was discovered while running:

```
pytest -v --parallel-threads=4 --iterations=2 sklearn/svm/tests/test_sparse.py
```

before including the fix pushed to #30041 under https://github.com/scikit-learn/scikit-learn/pull/30041/commits/bce2b4eb7d5ab49cf758f98c667e86243883d1de.

I suspect the problem is that the built-in Platt scaling implementation of the vendored C++ code base of libsvm that uses a singleton pseudo random generator. Therefore, seeding the shared RNG state from competing threads prevents getting reproducible results and hence the test failure.

COMMENT:
Use this to run ok?? 

```
platform linux -- Python 3.11.11, pytest-8.4.1, pluggy-1.6.0
rootdir: /mnt/workspace/scikit-learn
configfile: pyproject.toml
testpaths: sklearn
plugins: hydra-core-1.3.2, anyio-4.10.0, typeguard-2.13.3, cov-6.2.1, parallel-0.1.1, xdist-3.8.0
```

Labels: Bug, Moderate
`SVC(probability=True)`  is not thread-safe

This was discovered while running:

```
pytest -v --parallel-threads=4 --iterations=2 sklearn/svm/tests/test_sparse.py
```

before including the fix pushed to #30041 under https://github.com/scikit-learn/scikit-learn/pull/30041/commits/bce2b4eb7d5ab49cf758f98c667e86243883d1de.

I suspect the problem is that the built-in Platt scaling implementation of the vendored C++ code base of libsvm that uses a singleton pseudo random generator. Therefore, seeding the shared RNG state from competing threads prevents getting reproducible results and hence the test failure.

COMMENT:
Thanks everyone for your interest in contributing to scikit-learn, unfortunately this is not an easy issue to work on.

I would suggest you look at our [contributing doc](https://scikit-learn.org/dev/developers/contributing.html) and in particular the section [Issues for new contributors](https://scikit-learn.org/dev/developers/contributing.html#new-contributors).

Labels: Bug, Moderate
`SVC(probability=True)`  is not thread-safe

This was discovered while running:

```
pytest -v --parallel-threads=4 --iterations=2 sklearn/svm/tests/test_sparse.py
```

before including the fix pushed to #30041 under https://github.com/scikit-learn/scikit-learn/pull/30041/commits/bce2b4eb7d5ab49cf758f98c667e86243883d1de.

I suspect the problem is that the built-in Platt scaling implementation of the vendored C++ code base of libsvm that uses a singleton pseudo random generator. Therefore, seeding the shared RNG state from competing threads prevents getting reproducible results and hence the test failure.

COMMENT:
Hi, I’m interested in this issue and would like to work on it. Could you please assign it to me? Thank you!

Labels: Bug, Moderate
Proposal to Contribute Uncertainty Quantification via Aleatoric/Epistemic Decomposition to scikit-learn

### Describe the workflow you want to enable

Hi,

While ensemble methods like RandomForestRegressor are widely used, scikit-learn currently lacks native support for estimating and exposing predictive uncertainty—an increasingly essential feature in many applied domains such as healthcare, scientific modeling, and decision support systems.

### Describe your proposed solution


I propose adding functionality to expose both:

    Aleatoric uncertainty (data-driven),
    Epistemic uncertainty (model-driven).


Importantly, this is not just a concept—I have already implemented this wrapper as part of my ongoing PhD research. The approach is detailed in a preprint available here:

http://dx.doi.org/10.22541/au.175373261.14525669/v1 . 

The implementation is functional, tested, and used in geophysical mapping described in the paper.

This contribution builds on established research by Mohammad Hossein Shaker and Eyke Hüllermeier in uncertainty estimation for Random Forest Classification, and I have extended those principles to Random Forest Regression.

The approach is detailed in this article available here:

http://dx.doi.org/10.1007/978-3-030-44584-3_35

Thanks

### Describe alternatives you've considered, if relevant





### Additional context

_No response_

COMMENT:
Generally speaking, we have `predict_proba` and `predict_log_proba` as the API to indicate uncertainty, and appreciate well established implementations for those methods in our library. However, we try to avoid very new research since they're not yet established in the community.

If you can provide us with a nice overview of the literature in this area, and what could be considered as well established, we're happy to include them. We can even consider multiple methods of uncertainty estimation tuned by an estimator hyper-parameter.

I think @glemaitre or @adam2392 might have a better overview of the existing literature.

Labels: New Feature, Needs Decision - Include Feature
Proposal to Contribute Uncertainty Quantification via Aleatoric/Epistemic Decomposition to scikit-learn

### Describe the workflow you want to enable

Hi,

While ensemble methods like RandomForestRegressor are widely used, scikit-learn currently lacks native support for estimating and exposing predictive uncertainty—an increasingly essential feature in many applied domains such as healthcare, scientific modeling, and decision support systems.

### Describe your proposed solution


I propose adding functionality to expose both:

    Aleatoric uncertainty (data-driven),
    Epistemic uncertainty (model-driven).


Importantly, this is not just a concept—I have already implemented this wrapper as part of my ongoing PhD research. The approach is detailed in a preprint available here:

http://dx.doi.org/10.22541/au.175373261.14525669/v1 . 

The implementation is functional, tested, and used in geophysical mapping described in the paper.

This contribution builds on established research by Mohammad Hossein Shaker and Eyke Hüllermeier in uncertainty estimation for Random Forest Classification, and I have extended those principles to Random Forest Regression.

The approach is detailed in this article available here:

http://dx.doi.org/10.1007/978-3-030-44584-3_35

Thanks

### Describe alternatives you've considered, if relevant





### Additional context

_No response_

COMMENT:
Hi,

You're right that predict_proba and predict_log_proba are standard APIs for expressing predictive uncertainty in classification tasks. These are implemented in classifiers like RandomForestClassifier, but not in regressors such as RandomForestRegressor.

Uncertainty Decomposition in Classification
In ensemble classifiers like random forests, predictive uncertainty can be interpreted using entropy — which reflects the confidence in class probability predictions (as returned by predict_proba).

Uncertainty can be broken down into:

Aleatoric Uncertainty (data noise): Comes from ambiguity in the data itself, such as overlapping or noisy labels.

Epistemic Uncertainty (model uncertainty): Arises from limited knowledge or insufficient data, and is reducible with more training data.

From an ensemble perspective:

Each tree produces a probability distribution based on the training labels in its leaves.

The entropy of each tree’s prediction (intra-tree entropy) reflects aleatoric uncertainty.

The entropy of the average class probability across all trees (i.e., predict_proba) reflects total uncertainty.

The difference between the two captures epistemic uncertainty, indicating how much the trees disagree.



Uncertainty in Regression
For regression, the law of total variance applies:

Aleatoric Uncertainty: Variance within each tree’s leaves (data noise)

Epistemic Uncertainty: Variance across tree predictions (model uncertainty)

Total Uncertainty: Sum of both components

These decomposistions are grounded in classical theory (entropy for classification, variance for regression) and align with standard ensemble behavior, avoiding reliance on newer, less-established Bayesian or deep learning techniques.

Thanks.

Labels: New Feature, Needs Decision - Include Feature
Faster algorithm for KMeans

### Describe the workflow you want to enable

Dear community and developers, 

I think [this work](https://arxiv.org/abs/2308.09701) might be interesting to the scikit-community.  In this work, we discuss 2 classical algorithms for an sampling-based version of k-means, which return an epsilon-approximation of the centroids (which is user-determined). 

I was wondering if this could be an interesting addition to your (great) library, as it shows practical advantages already on small datasets.

### Describe your proposed solution

Algorithm 1 of  [this work](https://arxiv.org/abs/2308.09701) can result in a faster k-mean algorithm. 

I implemented the algorithm, which can be found [here](
https://github.com/Scinawa/do-you-know-what-q-means). However, as it is just a proof of concept, is not ready to be merged in scikit-learn. 



### Describe alternatives you've considered, if relevant

There are other fast coreset-based algorithms, which are much more complicated to implement, and are practically slower than our algorithm. 

### Additional context

_No response_

COMMENT:
Interesting work. However, we tend to avoid including new research / work in the library, since otherwise we wouldn't be able to afford maintaining the library.

I also didn't see any actual time comparisons between our implementations and yours in the paper or on your repo. The actual cost of running these algorithms has a lot to do with the optimisations included in the implementations as well as what kind of machine / how many cores is included in that machine.

Therefore, I don't think we can include the work as is, and closing the issue. But pinging @jeremiedbb in case he things differently.

Labels: New Feature, Needs Triage
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@lucascolley I was trying this out for TemperatureScaling and within that we make use of `scipy.optimize.minimize_scalar` which currently is not compatible with the Array API. 
https://github.com/scikit-learn/scikit-learn/blob/332ab38d9bdf20ae58d394cb7542b1b5174c13b3/sklearn/calibration.py#L1131-L1148
Do we have any alternative that we can use or should we consider adding an Array API compatible version of `minimize_scalar` in the utils?

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@mdhaber have you thought about array API support for constrained minimisation? Any opinions on where to start?

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
gh-32107 shows the existing test-suite passing while naïvely replacing `minimize_scalar` with `bracket_minimum` and `find_minimum` (benchmarks may be a different matter).

As documented at https://scipy.github.io/devdocs/dev/api-dev/array_api_modules_tables/optimize_elementwise.html, these functions have CuPy and PyTorch support, so this may be an interesting starting point for you @OmarManzoor @lucyleeow !

I haven't investigated performance for NumPy at all, but this may at least get you by as a fallback for non-NumPy arrays, while you wait for some more ticks to materialise in https://scipy.github.io/devdocs/dev/api-dev/array_api_modules_tables/optimize.html.

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@OmarManzoor, [<code>find_minimum</code>](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.elementwise.find_minimum.html) and [<code>bracket_minimum</code>](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.elementwise.bracket_minimum.html) support the Array API. I've been thinking about this as well.

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@virchan see my recent comment :)

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@lucascolley Thank you for the details on how to use `bracket_minimum` and `find_minimum`.

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> [@OmarManzoor](https://github.com/OmarManzoor), [`find_minimum`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.elementwise.find_minimum.html) and [`bracket_minimum`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.elementwise.bracket_minimum.html) support the Array API. I've been thinking about this as well.

Thank you.

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@OmarManzoor are you wanting to work on this?

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@lucyleeow I have done some work and experimentation with regards to TemperatureScaling. I am looking to create a PR to share my findings.

Labels: New Feature, help wanted, Hard, module:calibration, Array API