Talip7/scikit-learn-issues-multilabel huggingface dataset:
FEA Add missing-value support to sparse splitter in RandomForest and ExtraTrees

### Summary
While missing-value support for decision trees have been added recently, they only work when encoded in a dense array. Since `RandomForest*` and `ExtraTrees*` both support sparse `X`, if a user encodes `np.nan` inside sparse `X`, it should still work.

### Solution
Add missing-value logic in `SparsePartitioner` in `_parititoner.pyx`, `BestSparseSplitter` and `RandomSparseSplitter` in `_splitter.pyx`.

The logic is the same as in the dense case, but just has to handle the fact that `X` is now sparse CSC array format.

### Misc.
              FYI https://github.com/scikit-learn/scikit-learn/pull/27966 will introduce native support for missing values in the `ExtraTree*` models (i.e. random splitter). 

One thing I noticed though as I went through the PR is that the current codebase still does not support missing values in the sparse splitter. I think this might be pretty easy to add, but should we re-open this issue technically?

Xref: https://github.com/scikit-learn/scikit-learn/issues/5870#issuecomment-1166581736

_Originally posted by @adam2392 in https://github.com/scikit-learn/scikit-learn/issues/5870#issuecomment-2212688552_

COMMENT:
@adam2392 I reviewed the current `SparsePartitioner` implementation earlier today and had a rough plan for the changes I wanted to propose. I was planning on writing them up and sharing them with you by tonight. However, if @vitorpohlenz is already working on this, I can look for another issue to contribute to.

I have some free time this week and was hoping to make a couple of contributions :)
Also, yes, I’ve worked with Cython before.

Thanks!

Labels: help wanted, module:tree, cython
FEA Add missing-value support to sparse splitter in RandomForest and ExtraTrees

### Summary
While missing-value support for decision trees have been added recently, they only work when encoded in a dense array. Since `RandomForest*` and `ExtraTrees*` both support sparse `X`, if a user encodes `np.nan` inside sparse `X`, it should still work.

### Solution
Add missing-value logic in `SparsePartitioner` in `_parititoner.pyx`, `BestSparseSplitter` and `RandomSparseSplitter` in `_splitter.pyx`.

The logic is the same as in the dense case, but just has to handle the fact that `X` is now sparse CSC array format.

### Misc.
              FYI https://github.com/scikit-learn/scikit-learn/pull/27966 will introduce native support for missing values in the `ExtraTree*` models (i.e. random splitter). 

One thing I noticed though as I went through the PR is that the current codebase still does not support missing values in the sparse splitter. I think this might be pretty easy to add, but should we re-open this issue technically?

Xref: https://github.com/scikit-learn/scikit-learn/issues/5870#issuecomment-1166581736

_Originally posted by @adam2392 in https://github.com/scikit-learn/scikit-learn/issues/5870#issuecomment-2212688552_

COMMENT:
Hi everyone,

In that case @adam2392 and @rusdes. If you think if you think you can finish it quickly I would not be upset about steping back, and finding another Issue.

________________________________
From: Rushil Desai ***@***.***>
Sent: Wednesday, June 11, 2025 7:02:57 PM
To: scikit-learn/scikit-learn ***@***.***>
Cc: Vitor Pohlenz ***@***.***>; Mention ***@***.***>
Subject: Re: [scikit-learn/scikit-learn] FEA Add missing-value support to sparse splitter in RandomForest and ExtraTrees (Issue #29542)

[https://avatars.githubusercontent.com/u/40771799?s=20&v=4]rusdes left a comment (scikit-learn/scikit-learn#29542)<https://github.com/scikit-learn/scikit-learn/issues/29542#issuecomment-2964405957>

@adam2392<https://github.com/adam2392> I reviewed the current SparsePartitioner implementation earlier today and had a rough plan for the changes I wanted to propose. I was planning on writing them up and sharing them with you by tonight. However, if @vitorpohlenz<https://github.com/vitorpohlenz> is already working on this, I can look for another issue to contribute to.

I have some free time this week and was hoping to make a couple of contributions :)
Also, yes, I’ve worked with Cython before.

Thanks!

—
Reply to this email directly, view it on GitHub<https://github.com/scikit-learn/scikit-learn/issues/29542#issuecomment-2964405957>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKLIL6H6ZUQVA5D5MITTTU33DCRRDAVCNFSM6AAAAAB4BHOCAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNRUGQYDKOJVG4>.
You are receiving this because you were mentioned.Message ID: ***@***.***>

Labels: help wanted, module:tree, cython
FEA Add missing-value support to sparse splitter in RandomForest and ExtraTrees

### Summary
While missing-value support for decision trees have been added recently, they only work when encoded in a dense array. Since `RandomForest*` and `ExtraTrees*` both support sparse `X`, if a user encodes `np.nan` inside sparse `X`, it should still work.

### Solution
Add missing-value logic in `SparsePartitioner` in `_parititoner.pyx`, `BestSparseSplitter` and `RandomSparseSplitter` in `_splitter.pyx`.

The logic is the same as in the dense case, but just has to handle the fact that `X` is now sparse CSC array format.

### Misc.
              FYI https://github.com/scikit-learn/scikit-learn/pull/27966 will introduce native support for missing values in the `ExtraTree*` models (i.e. random splitter). 

One thing I noticed though as I went through the PR is that the current codebase still does not support missing values in the sparse splitter. I think this might be pretty easy to add, but should we re-open this issue technically?

Xref: https://github.com/scikit-learn/scikit-learn/issues/5870#issuecomment-1166581736

_Originally posted by @adam2392 in https://github.com/scikit-learn/scikit-learn/issues/5870#issuecomment-2212688552_

COMMENT:
Hi @adam2392, thank you for your advice, and sorry for taking so long to answer!
Based on the conversation, I have some suggestions on how to tackle this Issue.
> As a first step, I would look at this PR: [#27966](https://github.com/scikit-learn/scikit-learn/pull/27966), review how Cython works. Then what's needed is for one to:
> 
> 1. understand how sparse arrays work
> 2. understand how missing-values are split in the sklearn RF algo
> 3. apply that understanding to the sparse splitter logic
> 
> Feel free to propose/iterate on this plan accordingly!

The main question that comes to my mind is whether we should consider `NAs` as "important numbers" or not. From this question, I thought in two strategies:

A. Convert `NAs` as zeros (non-relevant) and keep the splitting the same way. In my mind, this is a good way to handle MCAR data. This is also some kind of "lazy/easy approach", because just the relevant (non-zero and not NA) data will be used in the splits.

B. Consider `NAs` as "important" and then choose between the two approaches mentioned in your PR: [#27966](https://github.com/scikit-learn/scikit-learn/pull/27966):
  B1. Splitting `NAs` randomly to the left/right 
  B2. Sending all `NAs` to left/right and all the other data to the other side

So I would like to know your opinion about this. I my naive opinion, Strategy A and B1 are quite similar, maybe equivalent. But is more of a wild guess from my side.

PS: `NA` stands for "Not Available" = Missing Data

Feel free to say if I completely misunderstood the Issue as well.

Labels: help wanted, module:tree, cython
FEA Add missing-value support to sparse splitter in RandomForest and ExtraTrees

### Summary
While missing-value support for decision trees have been added recently, they only work when encoded in a dense array. Since `RandomForest*` and `ExtraTrees*` both support sparse `X`, if a user encodes `np.nan` inside sparse `X`, it should still work.

### Solution
Add missing-value logic in `SparsePartitioner` in `_parititoner.pyx`, `BestSparseSplitter` and `RandomSparseSplitter` in `_splitter.pyx`.

The logic is the same as in the dense case, but just has to handle the fact that `X` is now sparse CSC array format.

### Misc.
              FYI https://github.com/scikit-learn/scikit-learn/pull/27966 will introduce native support for missing values in the `ExtraTree*` models (i.e. random splitter). 

One thing I noticed though as I went through the PR is that the current codebase still does not support missing values in the sparse splitter. I think this might be pretty easy to add, but should we re-open this issue technically?

Xref: https://github.com/scikit-learn/scikit-learn/issues/5870#issuecomment-1166581736

_Originally posted by @adam2392 in https://github.com/scikit-learn/scikit-learn/issues/5870#issuecomment-2212688552_

COMMENT:
I believe #27966 only applies for extra trees, and chooses one of the options in B1 randomly.

random forests evaluates B and B2, as shown in the `_splitter.pyx` code

Labels: help wanted, module:tree, cython
Tag for identifying capability to handle non-numeric data in input

### Describe the workflow you want to enable

I want to be able to find out whether an estimator supports non-numeric features in the input data passed to it in fit/transform. Example : `OneHotEncoder`, `LabelEncoder` supports this while `StandardScaler` does not.

### Describe your proposed solution

Setting appropriate tags for identifying the capability of the estimator.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I am trying to extend support for categorical features in `sktime` where I am now facing an issue with an sklearn adapter where the categorical support depends on the sklearn transformer passed to the adapter. Having some way to inspect the transformer and find out whether it can take categorical inputs would be useful.

COMMENT:
Hi @Abhay-Lejith, it' possible that the estimator tags contain the information you're looking for: https://scikit-learn.org/stable/developers/develop.html#estimator-tags. Currently we use them for testing purpose but I can imagine that it could be used for discovery.

Labels: New Feature
Tag for identifying capability to handle non-numeric data in input

### Describe the workflow you want to enable

I want to be able to find out whether an estimator supports non-numeric features in the input data passed to it in fit/transform. Example : `OneHotEncoder`, `LabelEncoder` supports this while `StandardScaler` does not.

### Describe your proposed solution

Setting appropriate tags for identifying the capability of the estimator.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I am trying to extend support for categorical features in `sktime` where I am now facing an issue with an sklearn adapter where the categorical support depends on the sklearn transformer passed to the adapter. Having some way to inspect the transformer and find out whether it can take categorical inputs would be useful.

COMMENT:
@jeremiedbb thanks for the quick reply. I somehow didn't notice when I went though the docs earlier that the `X_types` tag also has 'categorical' as a possible value. 
However I just noticed that for LabelEncoder it is only `1dlabels` and not `categorical` . The label encoder does take categorical inputs as well, so is there any reason for why it has been omitted?

Labels: New Feature
Tag for identifying capability to handle non-numeric data in input

### Describe the workflow you want to enable

I want to be able to find out whether an estimator supports non-numeric features in the input data passed to it in fit/transform. Example : `OneHotEncoder`, `LabelEncoder` supports this while `StandardScaler` does not.

### Describe your proposed solution

Setting appropriate tags for identifying the capability of the estimator.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I am trying to extend support for categorical features in `sktime` where I am now facing an issue with an sklearn adapter where the categorical support depends on the sklearn transformer passed to the adapter. Having some way to inspect the transformer and find out whether it can take categorical inputs would be useful.

COMMENT:
``LabelEncoder`` is meant to encode the target y. Maybe what you're looking for is the [``OrdinalEncoder``](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OrdinalEncoder.html) ? This one has ``categorical``.

Labels: New Feature
Tag for identifying capability to handle non-numeric data in input

### Describe the workflow you want to enable

I want to be able to find out whether an estimator supports non-numeric features in the input data passed to it in fit/transform. Example : `OneHotEncoder`, `LabelEncoder` supports this while `StandardScaler` does not.

### Describe your proposed solution

Setting appropriate tags for identifying the capability of the estimator.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I am trying to extend support for categorical features in `sktime` where I am now facing an issue with an sklearn adapter where the categorical support depends on the sklearn transformer passed to the adapter. Having some way to inspect the transformer and find out whether it can take categorical inputs would be useful.

COMMENT:
Okay yes, that should do too. Thanks a lot for the help.

Labels: New Feature
Add FN and FP weight parameter in MCC

### Describe the workflow you want to enable

Introducing a weight parameter for false negatives (FN) and false positives (FP) in Matthews Correlation Coefficient (MCC) would enhance the metric’s flexibility and applicability, particularly in contexts where the costs of different types of errors vary significantly. By allowing the impact of FN and FP to be adjusted, MCC can be tailored to better reflect the specific goals of the application. For example, in medical problems, minimizing false negatives might be crucial to avoid missing a critical condition, while in other contexts, reducing false positives might be more important to prevent unnecessary interventions. 

### Describe your proposed solution

Original:
```
C = confusion_matrix(y_true, y_pred, sample_weight=sample_weight)
t_sum = C.sum(axis=1, dtype=np.float64)
p_sum = C.sum(axis=0, dtype=np.float64)
n_correct = np.trace(C, dtype=np.float64)
n_samples = p_sum.sum()

cov_ytyp = n_correct * n_samples - np.dot(t_sum, p_sum)
cov_ypyp = n_samples**2 - np.dot(p_sum, p_sum)
cov_ytyt = n_samples**2 - np.dot(t_sum, t_sum)

if cov_ypyp * cov_ytyt == 0:
    return 0.0
else:
    return cov_ytyp / np.sqrt(cov_ytyt * cov_ypyp)
```

Proposed:
```
C = confusion_matrix(y_true, y_pred, sample_weight=sample_weight)
t_sum = C.sum(axis=1, dtype=np.float64)
p_sum = C.sum(axis=0, dtype=np.float64)
n_correct = np.trace(C, dtype=np.float64)
n_samples = p_sum.sum()

weighted_t_sum = t_sum * fn_weight
weighted_p_sum = p_sum * fp_weight
    
cov_ytyp = n_correct * n_samples - np.dot(weighted_t_sum, weighted_p_sum)
cov_ypyp = n_samples**2 - np.dot(weighted_p_sum, weighted_p_sum)
cov_ytyt = n_samples**2 - np.dot(weighted_t_sum, weighted_t_sum)

if cov_ypyp * cov_ytyt == 0:
    return 0.0
else:
    return cov_ytyp / np.sqrt(cov_ytyt * cov_ypyp)
```

### Describe alternatives you've considered, if relevant

- Instead of two parameters, one single parameter can be used for weighting, similarly to beta in fbeta_score.

### Additional context

_No response_

COMMENT:
I think all these cases are best dealt with using custom scorers, such as the one presented here: https://scikit-learn.org/dev/auto_examples/model_selection/plot_cost_sensitive_learning.html#addressing-the-problem-with-a-business-metric

And therefore I'd say this is out of scope since defining those custom scorers is pretty easy.

Also ping @glemaitre in case he feels differently.

Labels: New Feature, Needs Triage
Add FN and FP weight parameter in MCC

### Describe the workflow you want to enable

Introducing a weight parameter for false negatives (FN) and false positives (FP) in Matthews Correlation Coefficient (MCC) would enhance the metric’s flexibility and applicability, particularly in contexts where the costs of different types of errors vary significantly. By allowing the impact of FN and FP to be adjusted, MCC can be tailored to better reflect the specific goals of the application. For example, in medical problems, minimizing false negatives might be crucial to avoid missing a critical condition, while in other contexts, reducing false positives might be more important to prevent unnecessary interventions. 

### Describe your proposed solution

Original:
```
C = confusion_matrix(y_true, y_pred, sample_weight=sample_weight)
t_sum = C.sum(axis=1, dtype=np.float64)
p_sum = C.sum(axis=0, dtype=np.float64)
n_correct = np.trace(C, dtype=np.float64)
n_samples = p_sum.sum()

cov_ytyp = n_correct * n_samples - np.dot(t_sum, p_sum)
cov_ypyp = n_samples**2 - np.dot(p_sum, p_sum)
cov_ytyt = n_samples**2 - np.dot(t_sum, t_sum)

if cov_ypyp * cov_ytyt == 0:
    return 0.0
else:
    return cov_ytyp / np.sqrt(cov_ytyt * cov_ypyp)
```

Proposed:
```
C = confusion_matrix(y_true, y_pred, sample_weight=sample_weight)
t_sum = C.sum(axis=1, dtype=np.float64)
p_sum = C.sum(axis=0, dtype=np.float64)
n_correct = np.trace(C, dtype=np.float64)
n_samples = p_sum.sum()

weighted_t_sum = t_sum * fn_weight
weighted_p_sum = p_sum * fp_weight
    
cov_ytyp = n_correct * n_samples - np.dot(weighted_t_sum, weighted_p_sum)
cov_ypyp = n_samples**2 - np.dot(weighted_p_sum, weighted_p_sum)
cov_ytyt = n_samples**2 - np.dot(weighted_t_sum, weighted_t_sum)

if cov_ypyp * cov_ytyt == 0:
    return 0.0
else:
    return cov_ytyp / np.sqrt(cov_ytyt * cov_ypyp)
```

### Describe alternatives you've considered, if relevant

- Instead of two parameters, one single parameter can be used for weighting, similarly to beta in fbeta_score.

### Additional context

_No response_

COMMENT:
Indeed, I agree with @adrinjalali. I think this is better to stick to the default definition. Then, we should make a good job at documented how to make your own metric but I think that we already have some documentation.

Labels: New Feature, Needs Triage
RFE results are inconsistent between machines with ties in feature importances at threshold

### Describe the bug

RFE uses np.argsort on the feature_importances from the estimator, this is not repeatable across machines. This only matters when there are ties in the feature importances that overlap with the threshold. For example if the importances are [0, 2, 0, 1] and it needs to eliminate 1 feature, it may not choose the same feature to eliminate on all machines.

### Steps/Code to Reproduce

It is difficult to write code that will show this as it requires multiple machines to test; however, the following test could be added to the _rfe unit tests to check if the sort is stable, though the sort could be consistent but not stable, so this isn't enough to prove an issue.
```
def test_rfe_ties_around_threshold():
    X, y = make_classification(n_features=47, random_state=0)
    clf = MockClassifier() # mock classifier returns constant feature_importances
    rfe = RFE(estimator=clf, n_features_to_select=4, step=2)
    rfe.fit(X, y)

    assert_array_equal(rfe.support_ ,np.array([*[False]*43, *[True]*4]))
```

### Expected Results

Expected to get the same results on all machines, being stable is not necessary though it is an easy way to enforce testable consistency.

### Actual Results

Current sort does not result in a stable sort and fails the previously provided unit test.

### Versions

```shell
main branch
```


This issue is most likely to occur with zero importance features, especially at the initial steps of RFE.

COMMENT:
See also: #29495 and #29496 for discussion on the `np.argsort` inconsistency between machines.

Labels: Bug
RFE results are inconsistent between machines with ties in feature importances at threshold

### Describe the bug

RFE uses np.argsort on the feature_importances from the estimator, this is not repeatable across machines. This only matters when there are ties in the feature importances that overlap with the threshold. For example if the importances are [0, 2, 0, 1] and it needs to eliminate 1 feature, it may not choose the same feature to eliminate on all machines.

### Steps/Code to Reproduce

It is difficult to write code that will show this as it requires multiple machines to test; however, the following test could be added to the _rfe unit tests to check if the sort is stable, though the sort could be consistent but not stable, so this isn't enough to prove an issue.
```
def test_rfe_ties_around_threshold():
    X, y = make_classification(n_features=47, random_state=0)
    clf = MockClassifier() # mock classifier returns constant feature_importances
    rfe = RFE(estimator=clf, n_features_to_select=4, step=2)
    rfe.fit(X, y)

    assert_array_equal(rfe.support_ ,np.array([*[False]*43, *[True]*4]))
```

### Expected Results

Expected to get the same results on all machines, being stable is not necessary though it is an easy way to enforce testable consistency.

### Actual Results

Current sort does not result in a stable sort and fails the previously provided unit test.

### Versions

```shell
main branch
```


This issue is most likely to occur with zero importance features, especially at the initial steps of RFE.

COMMENT:
Also, adding this means we take a hit on the performance cause we're moving from `quicksort` to `mergesort`. So I wonder if it's worth it.

Labels: Bug
RFE results are inconsistent between machines with ties in feature importances at threshold

### Describe the bug

RFE uses np.argsort on the feature_importances from the estimator, this is not repeatable across machines. This only matters when there are ties in the feature importances that overlap with the threshold. For example if the importances are [0, 2, 0, 1] and it needs to eliminate 1 feature, it may not choose the same feature to eliminate on all machines.

### Steps/Code to Reproduce

It is difficult to write code that will show this as it requires multiple machines to test; however, the following test could be added to the _rfe unit tests to check if the sort is stable, though the sort could be consistent but not stable, so this isn't enough to prove an issue.
```
def test_rfe_ties_around_threshold():
    X, y = make_classification(n_features=47, random_state=0)
    clf = MockClassifier() # mock classifier returns constant feature_importances
    rfe = RFE(estimator=clf, n_features_to_select=4, step=2)
    rfe.fit(X, y)

    assert_array_equal(rfe.support_ ,np.array([*[False]*43, *[True]*4]))
```

### Expected Results

Expected to get the same results on all machines, being stable is not necessary though it is an easy way to enforce testable consistency.

### Actual Results

Current sort does not result in a stable sort and fails the previously provided unit test.

### Versions

```shell
main branch
```


This issue is most likely to occur with zero importance features, especially at the initial steps of RFE.

COMMENT:
> Also, adding this means we take a hit on the performance cause we're moving from `quicksort` to `mergesort`. So I wonder if it's worth it.

At the scale of the number of input features I'm not sure how big the difference is, so my suggestion may just take even more time.

But could unstable sort run, and if there are ties at the threshold, then it could switch to stable sort, and pssoibly even just stable sort the tied region?

Worst case, with lots of big ties, this is obviously much worse. But in the common case with no ties in the relevant region, it may be better?

Labels: Bug