Talip7/scikit-learn-issues-multilabel huggingface dataset:
from sklearn.datasets import make_regression FileNotFoundError

### Describe the bug

When running examples/application/plot_prediction_latency.py a FileNotFoundError occurs as there is no file named make_regression in datasets dir. 
I have cloned the scikit-learn repo and installed it using ```pip install -e .``` 
Completely unable to  ```import scikit_learn ``` or ```sklearn ``` albeit it showing up when ```pip list -> scikit-learn    1.7.dev0    /Users/user/scikit-learn ```



### Steps/Code to Reproduce

from sklearn.datasets import make_regression

### Expected Results

No error is thrown 

### Actual Results

Exception has occurred: FileNotFoundError
[Errno 2] No such file or directory: '/private/var/folders/0q/80gytspx42v3rtlkkq_h59jw0000gn/T/pip-build-env-53amsfeb/normal/bin/ninja'

### Versions

```shell
scikit-learn    1.7.dev0
```

COMMENT:
According to https://scikit-learn.org/dev/developers/advanced_installation.html#building-from-source, you should do:

```
pip install --editable . \
   --verbose --no-build-isolation \
   --config-settings editable-verbose=true
```

I think this is a limitation of using build isolation with meson editable install. It installs in a temporary environment but this environment does not exist after `pip` returns is finished and you get the confusing `FileNotFoundError` aobut `ninja` not being found with a path that looks like a temporary pip folder.

Labels: Bug, Needs Triage
Add checking if tree criterion/splitter are classes

### Describe the workflow you want to enable

In the process of creating custom splitters, criterions & models that inherit from the respective _scikit-learn_ classes, a very convenient (albeit currently impossible) solution is to add the splitter & criterion classes as parameters to the model constructor. The currently supported parameter types are strings (referencing splitters & criterions that are already in _scikit-learn_) or objects. Because the splitters & criterions depend on parameters from the fitting function, there is a need for class support in the process of parameter parsing.

### Describe your proposed solution

Checking if the splitter/criterion is a class and constructing it accordingly. A code solution is available [here](https://github.com/gilramot/scikit-learn/commit/ed1b4f3920f6f1aa073c620abd29046cc12a1214).

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
It would help if you can explain your motivation behind this and maybe share some pseudo-code. For example, is this to write a research article to compare other criterion/splitters, is it because you have found a particular criterion/splitter was useful in practice and missing from scikit-learn.

If your motivation is more the research article kind, it feels like as a work-around would be to add your own classes into `sklearn.tree._classes.CRITERIA_CLF/CRITERIA_REG/SPLITTERS` dictionaries.

Something to keep in mind: we have to balance extensibility vs maintainability of the library. My experience is when you add extensibility mostly because you can, it is harder to follow the code, harder to test, and more likely to introduce subtle bugs.

Labels: New Feature, Needs Info
Add checking if tree criterion/splitter are classes

### Describe the workflow you want to enable

In the process of creating custom splitters, criterions & models that inherit from the respective _scikit-learn_ classes, a very convenient (albeit currently impossible) solution is to add the splitter & criterion classes as parameters to the model constructor. The currently supported parameter types are strings (referencing splitters & criterions that are already in _scikit-learn_) or objects. Because the splitters & criterions depend on parameters from the fitting function, there is a need for class support in the process of parameter parsing.

### Describe your proposed solution

Checking if the splitter/criterion is a class and constructing it accordingly. A code solution is available [here](https://github.com/gilramot/scikit-learn/commit/ed1b4f3920f6f1aa073c620abd29046cc12a1214).

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Closing because of lack of response.

Labels: New Feature, Needs Info
Multiple thresholds in FixedThresholdClassifier

### Describe the workflow you want to enable

Currently FixedThresholdClassifier only allows for a unique threshold as a float. It would be nicer to be also able to accept a list of floats and that multiple classes would be produced accordingly. 

### Describe your proposed solution

Typically, default behaviour would be to cut the scores into bins and labels the bins accordingly from 0 to n, where n is the number of thresholds. Additional options could be welcome (providing a list of labels, option to starts counting at 1 for our non technical friends ... etc.)

### Describe alternatives you've considered, if relevant

Current solutions is to cut (pd.cut) outputs myself, outside of the sklearn pipeline. 

### Additional context

There are industries where instances are expected to be binned in different risk classes. See for exemple the rating grades in the IRB Framework. See [EBA Guidelines on PD estimation](https://extranet.eba.europa.eu/sites/default/documents/files/documents/10180/2192133/f5a2e068-dc4b-4a0e-a10f-378b517ac19c/Guidelines%20on%20PD%20and%20LGD%20estimation%20%28EBA-GL-2017-16%29_EN.pdf?retry=1) 5.2.4 Rating philosophy Art. 66 "Institutions should choose an appropriate philosophy underlying the assignment of obligors or exposures to grades or pools (‘rating philosophy’) [...]"

COMMENT:
Hi @lcrmorin, `TunedTresholdClassifier` and `FixedTresholdClassifier` were designed for binary classification and I don't think that there's a plan to support multilabel classification for now. You might be able make it work using a `OneVsRestClassifer` by decomposing your problem into n binary problems and then set a different treshold for each of these problems. Ping @glemaitre who might be able to give more details.

Labels: New Feature
Multiple thresholds in FixedThresholdClassifier

### Describe the workflow you want to enable

Currently FixedThresholdClassifier only allows for a unique threshold as a float. It would be nicer to be also able to accept a list of floats and that multiple classes would be produced accordingly. 

### Describe your proposed solution

Typically, default behaviour would be to cut the scores into bins and labels the bins accordingly from 0 to n, where n is the number of thresholds. Additional options could be welcome (providing a list of labels, option to starts counting at 1 for our non technical friends ... etc.)

### Describe alternatives you've considered, if relevant

Current solutions is to cut (pd.cut) outputs myself, outside of the sklearn pipeline. 

### Additional context

There are industries where instances are expected to be binned in different risk classes. See for exemple the rating grades in the IRB Framework. See [EBA Guidelines on PD estimation](https://extranet.eba.europa.eu/sites/default/documents/files/documents/10180/2192133/f5a2e068-dc4b-4a0e-a10f-378b517ac19c/Guidelines%20on%20PD%20and%20LGD%20estimation%20%28EBA-GL-2017-16%29_EN.pdf?retry=1) 5.2.4 Rating philosophy Art. 66 "Institutions should choose an appropriate philosophy underlying the assignment of obligors or exposures to grades or pools (‘rating philosophy’) [...]"

COMMENT:
It's not about Multiclass classification but about binning output of a binary classification (or regression). Typically, for real life applications we don't want 0 or 1 outputs. Sometimes we want probabilities but often, notably for action / validation we want risk classes. For exemple: 
- below 1%: 'Low risk'
- 1%-5%: 'Medium risk'
- above 5%: 'High risk'

Typically in that case we have two thresholds: .01 and .05, and three labels. My proposal is to accept the list [.01,.05] as thresholds and the list ['Low Risk','Medium Risk','High Risk'] as labels so that the model outputs the binned probability.

Labels: New Feature
Multiple thresholds in FixedThresholdClassifier

### Describe the workflow you want to enable

Currently FixedThresholdClassifier only allows for a unique threshold as a float. It would be nicer to be also able to accept a list of floats and that multiple classes would be produced accordingly. 

### Describe your proposed solution

Typically, default behaviour would be to cut the scores into bins and labels the bins accordingly from 0 to n, where n is the number of thresholds. Additional options could be welcome (providing a list of labels, option to starts counting at 1 for our non technical friends ... etc.)

### Describe alternatives you've considered, if relevant

Current solutions is to cut (pd.cut) outputs myself, outside of the sklearn pipeline. 

### Additional context

There are industries where instances are expected to be binned in different risk classes. See for exemple the rating grades in the IRB Framework. See [EBA Guidelines on PD estimation](https://extranet.eba.europa.eu/sites/default/documents/files/documents/10180/2192133/f5a2e068-dc4b-4a0e-a10f-378b517ac19c/Guidelines%20on%20PD%20and%20LGD%20estimation%20%28EBA-GL-2017-16%29_EN.pdf?retry=1) 5.2.4 Rating philosophy Art. 66 "Institutions should choose an appropriate philosophy underlying the assignment of obligors or exposures to grades or pools (‘rating philosophy’) [...]"

COMMENT:
Ah thanks for the clarification. This is something that we just discussed with @glemaitre. In that case we are outside of the scope of the `predict` method. We're not trying to predict a class but rather take a decision based on the predicted probabilities. So this is not really doable with the `FixedThresholdClassifier` since it just rewrites the `predict` method.

The cleaner solution I'd see is to extend our meta-estimators and scorers to accept arbitrary response methods beyond `predict`, `predict_proba`, ... but that's a big API change that would require more discussion from the maintainers.

cc/ @ogrisel who might also be interested by this discussion.

Labels: New Feature
Multiple thresholds in FixedThresholdClassifier

### Describe the workflow you want to enable

Currently FixedThresholdClassifier only allows for a unique threshold as a float. It would be nicer to be also able to accept a list of floats and that multiple classes would be produced accordingly. 

### Describe your proposed solution

Typically, default behaviour would be to cut the scores into bins and labels the bins accordingly from 0 to n, where n is the number of thresholds. Additional options could be welcome (providing a list of labels, option to starts counting at 1 for our non technical friends ... etc.)

### Describe alternatives you've considered, if relevant

Current solutions is to cut (pd.cut) outputs myself, outside of the sklearn pipeline. 

### Additional context

There are industries where instances are expected to be binned in different risk classes. See for exemple the rating grades in the IRB Framework. See [EBA Guidelines on PD estimation](https://extranet.eba.europa.eu/sites/default/documents/files/documents/10180/2192133/f5a2e068-dc4b-4a0e-a10f-378b517ac19c/Guidelines%20on%20PD%20and%20LGD%20estimation%20%28EBA-GL-2017-16%29_EN.pdf?retry=1) 5.2.4 Rating philosophy Art. 66 "Institutions should choose an appropriate philosophy underlying the assignment of obligors or exposures to grades or pools (‘rating philosophy’) [...]"

COMMENT:
May be I am wrong but it seems that _threshold_scores_to_class_labels could be rewritten with numpy's digitise to accept arbitrary list of thresholds / labels. Though I am not sure that having custom outcomes won't break anything down the line.

Labels: New Feature
duck typed estimators fail in check_estimator

### Describe the bug

I believe these 5 lines, which check for specific types:

https://github.com/scikit-learn/scikit-learn/blob/76ae0a539a0e87145c9f6fedcd7033494082fa17/sklearn/utils/estimator_checks.py#L4439-L4443

breaks the documentation in https://scikit-learn.org/stable/developers/develop.html#rolling-your-own-estimator

Where it says "We tend to use “duck typing” instead of checking for isinstance, which means it’s technically possible to implement estimator without inheriting from scikit-learn classes."

Since \_\_sklearn\_tags\_\_ appears to now be a requirement, and if those specific Tag classes are required to be returned from \_\_sklearn\_tags\_\_, then it is no longer possible to implement scikit-learn estimators through duck typing.  I believe either the tests should be changed, or the documentation updated.  I would prefer the tests to change.

### Steps/Code to Reproduce

see above

### Expected Results

see above

### Actual Results

see above

### Versions

```shell
1.6.0
```

COMMENT:
cc @adrinjalali

Labels: Bug
duck typed estimators fail in check_estimator

### Describe the bug

I believe these 5 lines, which check for specific types:

https://github.com/scikit-learn/scikit-learn/blob/76ae0a539a0e87145c9f6fedcd7033494082fa17/sklearn/utils/estimator_checks.py#L4439-L4443

breaks the documentation in https://scikit-learn.org/stable/developers/develop.html#rolling-your-own-estimator

Where it says "We tend to use “duck typing” instead of checking for isinstance, which means it’s technically possible to implement estimator without inheriting from scikit-learn classes."

Since \_\_sklearn\_tags\_\_ appears to now be a requirement, and if those specific Tag classes are required to be returned from \_\_sklearn\_tags\_\_, then it is no longer possible to implement scikit-learn estimators through duck typing.  I believe either the tests should be changed, or the documentation updated.  I would prefer the tests to change.

### Steps/Code to Reproduce

see above

### Expected Results

see above

### Actual Results

see above

### Versions

```shell
1.6.0
```

COMMENT:
We say it's "technically" possible, but not recommended really. We even warn if you don't inherit from `BaseEstimator`. At this point there is so much packed in our mixin classes and especially `BaseEstimator`, that it's highly unlikely that one can easily stay compatible with sklearn w/o inheriting from them.

We also use tags in our codebase, and not just the tests. So if you'd like to have your own tags for some reason and not inherit from sklearn tags while doing that (which I don't recommend), you can still do, and you can skip this test when calling `check_estimator`.

Labels: Bug
duck typed estimators fail in check_estimator

### Describe the bug

I believe these 5 lines, which check for specific types:

https://github.com/scikit-learn/scikit-learn/blob/76ae0a539a0e87145c9f6fedcd7033494082fa17/sklearn/utils/estimator_checks.py#L4439-L4443

breaks the documentation in https://scikit-learn.org/stable/developers/develop.html#rolling-your-own-estimator

Where it says "We tend to use “duck typing” instead of checking for isinstance, which means it’s technically possible to implement estimator without inheriting from scikit-learn classes."

Since \_\_sklearn\_tags\_\_ appears to now be a requirement, and if those specific Tag classes are required to be returned from \_\_sklearn\_tags\_\_, then it is no longer possible to implement scikit-learn estimators through duck typing.  I believe either the tests should be changed, or the documentation updated.  I would prefer the tests to change.

### Steps/Code to Reproduce

see above

### Expected Results

see above

### Actual Results

see above

### Versions

```shell
1.6.0
```

COMMENT:
@paulbkoch I see the way that you implemented compatibility in `interpretml`: https://github.com/interpretml/interpret/commit/e0369a70a3bb704261d82e2d44c38603bbc4e9c2

I would like to understand more the trade-off there: is there a reason not to inherit from the tags provided by scikit-learn that allow you to pass the checks?

One potential reason that I see is to support older scikit-learn versions where the new tags would not be available. For this case, we look at providing [`sklearn-compat`](https://sklearn-compat.readthedocs.io/en/latest/#upgrading-to-scikit-learn-16) where the "new" tags would be available and make it possible to support old scikit-learn version.

But I'm interested to know more to see if we overlook some aspects.

Labels: Bug
duck typed estimators fail in check_estimator

### Describe the bug

I believe these 5 lines, which check for specific types:

https://github.com/scikit-learn/scikit-learn/blob/76ae0a539a0e87145c9f6fedcd7033494082fa17/sklearn/utils/estimator_checks.py#L4439-L4443

breaks the documentation in https://scikit-learn.org/stable/developers/develop.html#rolling-your-own-estimator

Where it says "We tend to use “duck typing” instead of checking for isinstance, which means it’s technically possible to implement estimator without inheriting from scikit-learn classes."

Since \_\_sklearn\_tags\_\_ appears to now be a requirement, and if those specific Tag classes are required to be returned from \_\_sklearn\_tags\_\_, then it is no longer possible to implement scikit-learn estimators through duck typing.  I believe either the tests should be changed, or the documentation updated.  I would prefer the tests to change.

### Steps/Code to Reproduce

see above

### Expected Results

see above

### Actual Results

see above

### Versions

```shell
1.6.0
```

COMMENT:
Just checked what LGBM does, and they do the same:

https://github.com/microsoft/LightGBM/blob/ee0131c554844084d6b4eb7fbdf76c069fc40ad2/python-package/lightgbm/compat.py#L103

https://github.com/microsoft/LightGBM/blob/ee0131c554844084d6b4eb7fbdf76c069fc40ad2/python-package/lightgbm/compat.py#L118

Labels: Bug
duck typed estimators fail in check_estimator

### Describe the bug

I believe these 5 lines, which check for specific types:

https://github.com/scikit-learn/scikit-learn/blob/76ae0a539a0e87145c9f6fedcd7033494082fa17/sklearn/utils/estimator_checks.py#L4439-L4443

breaks the documentation in https://scikit-learn.org/stable/developers/develop.html#rolling-your-own-estimator

Where it says "We tend to use “duck typing” instead of checking for isinstance, which means it’s technically possible to implement estimator without inheriting from scikit-learn classes."

Since \_\_sklearn\_tags\_\_ appears to now be a requirement, and if those specific Tag classes are required to be returned from \_\_sklearn\_tags\_\_, then it is no longer possible to implement scikit-learn estimators through duck typing.  I believe either the tests should be changed, or the documentation updated.  I would prefer the tests to change.

### Steps/Code to Reproduce

see above

### Expected Results

see above

### Actual Results

see above

### Versions

```shell
1.6.0
```

COMMENT:
The approach of "use sklearn base classes if available" works because when sklearn is installed and the estimator is checked against sklearn with `check_estimator`, the inheritance is done right and everything works as expected. However, when sklearn is not installed, those packages are not really sklearn compatible and many features are missing, which you might say is okay cause they don't claim they support those features.

Basically, you can think of it as "I support extra features if sklearn is installed, but I don't force you to install sklearn and you can choose if you want those features or not".

We have packed a lot of code in our base classes and mixins, and they're subject to change as we develop the library. You can still have a scikit-learn-like estimator w/o depending on our base classes, but unless you keep copying the new features in your estimators from our developments in our base classes, you'd fall behind.

Labels: Bug
duck typed estimators fail in check_estimator

### Describe the bug

I believe these 5 lines, which check for specific types:

https://github.com/scikit-learn/scikit-learn/blob/76ae0a539a0e87145c9f6fedcd7033494082fa17/sklearn/utils/estimator_checks.py#L4439-L4443

breaks the documentation in https://scikit-learn.org/stable/developers/develop.html#rolling-your-own-estimator

Where it says "We tend to use “duck typing” instead of checking for isinstance, which means it’s technically possible to implement estimator without inheriting from scikit-learn classes."

Since \_\_sklearn\_tags\_\_ appears to now be a requirement, and if those specific Tag classes are required to be returned from \_\_sklearn\_tags\_\_, then it is no longer possible to implement scikit-learn estimators through duck typing.  I believe either the tests should be changed, or the documentation updated.  I would prefer the tests to change.

### Steps/Code to Reproduce

see above

### Expected Results

see above

### Actual Results

see above

### Versions

```shell
1.6.0
```

COMMENT:
As examples of things we've added in the past couple of releases that you would get for "free" if you inherit from the right classes are:

- metadata routing for consumers: if you have a simple estimator (as opposed to a meta-estimator), metadata routing works out of the box for fit arguments that are explicitly in the fit signature
- users can set the transform output type, and if you use the right functions to set the input feature names, etc, your transformer would output polars or pandas dataframes on a `.transform` call
- html representation of estimators in jupyter notebooks simply works

All of these would be missing if you don't inherit from our classes, or you'd need to implement them yourself.

Labels: Bug
duck typed estimators fail in check_estimator

### Describe the bug

I believe these 5 lines, which check for specific types:

https://github.com/scikit-learn/scikit-learn/blob/76ae0a539a0e87145c9f6fedcd7033494082fa17/sklearn/utils/estimator_checks.py#L4439-L4443

breaks the documentation in https://scikit-learn.org/stable/developers/develop.html#rolling-your-own-estimator

Where it says "We tend to use “duck typing” instead of checking for isinstance, which means it’s technically possible to implement estimator without inheriting from scikit-learn classes."

Since \_\_sklearn\_tags\_\_ appears to now be a requirement, and if those specific Tag classes are required to be returned from \_\_sklearn\_tags\_\_, then it is no longer possible to implement scikit-learn estimators through duck typing.  I believe either the tests should be changed, or the documentation updated.  I would prefer the tests to change.

### Steps/Code to Reproduce

see above

### Expected Results

see above

### Actual Results

see above

### Versions

```shell
1.6.0
```

COMMENT:
For most of those things, I'm fine if they are missing when scikit-learn is not installed.  If you want all the bells and whistles, then install scikit.  I imagine having a scikit-learn-less environment is really only important in production settings where someone wants to strip dependencies down to the minimum.  In that case they probably don't need html representations in jupyter.  Tags strike me though as more important to support even when scikit-learn is not installed, which is why I duplicated the class structure.  Explainable Boosting Machine models have their own additional tags that I'd like to add like "privacy" for differentially private EBMs, and it would be nice to slot that into a common interface that is available even without scikit-learn.  It will definitely be a higher maintenance burden to have these shadow tag classes and keep them updated, but I think this is the best available solution now.  Happy to receive other suggestions.

Labels: Bug