Talip7/scikit-learn-issues-multilabel huggingface dataset:
PERF `PairwiseDistancesReductions` subsequent work

`PairwiseDistancesReductions` have been introduced as a hierarchy of Cython classes to implement back-ends of some scikit-learn algorithms.

Initial work was listed in #22587.

:bulb: See [this presentation](https://docs.google.com/presentation/d/1RwX_P9lnsb9_YRZ0cA88l3VoEYhiMndQYoKLLF_0Dv0/edit?usp=sharing) for a better understanding of the design of `PairwiseDistancesReductions`.

:bulb: **Before working on improving performance, one must profile the current execution of algorithms to see if there are any substantial benefits.**

Subsequent possible work includes:

 - [ ] #26983 
 - [ ] #25097
 - [ ] Support `"precomputed"` distances
 - Implement back-end for other `Estimators` or `Transformers`, in particular, introduce a specialised back-end for which would remove the costly sequential portion around the current call to `kneighbors` or `radius_neighbors` for (non-complete list):
 	- [ ] computing the MST from the data matrix (`mst_from_data_matrix`) see https://github.com/scikit-learn/scikit-learn/pull/25656#pullrequestreview-1326267314
 	- [ ] computing the MST linkage ([`mst_linkage_core`](https://github.com/scikit-learn/scikit-learn/blob/2ab1d81e167512719b6272d4ddf286bba5d3d232/sklearn/cluster/_hierarchical_fast.pyx#L428-L506))
 	- [x] `LocalOutlierFactor` ([no further optimization possible](https://github.com/scikit-learn/scikit-learn/pull/26316))
 	- [ ] `KNeighbors*.predict*`
 	   - [x] #24076 by @Micky774
 	   - [ ] #28219 by @OmarManzoor
 	   - [ ] Implement multi-output support (WIP [here](https://github.com/Micky774/scikit-learn/pull/3))
 	   - [ ] Support `"distance"` weighting
 	- [ ] `RadiusNeighbors*.predict*`
 	   - [x] #26828
 	   - [ ] Implement Euclidean specialization
 	   - [ ] Implement multi-output support
 	   - [ ] Support `"distance"` weighting
 	- some part of `KMeans`
 - [ ] Force the use of `PairwiseDistancesReductions` for F-contiguous arrays
   - use a scikit-learn configuration entry to accept and convert F-contiguous arrays to C-contiguous arrays
 - [ ] Release the GIL before calling `parallel_on_{X|Y}` instead of releasing it in those methods
    - :bulb: I once have tried and I don't think this is possible.
 - [ ] Integrate `mimalloc` to have proper memory allocation for multi-threaded implementations:
    - :bulb: Changing the default implementation of `malloc(1)` might come with unexpected changes and might break the whole ecosystem. Also maintaining it might be costly.
 - [ ] Tests improvements ideas:
   - https://github.com/scikit-learn/scikit-learn/pull/27281#issuecomment-1704334881
   - parametrize old tests for public API backed by `PairwiseDistancesReductions`?
   - systematically test public API on combinations of sparse and dense datasets?
 - [ ] Extend this design for a new back-end for [`pairwise_kernels`](https://github.com/scikit-learn/scikit-learn/blob/7d1c31892d34e6456c467c3d538ecb0ec4e01c35/sklearn/metrics/pairwise.py#L2116)-based interfaces

COMMENT:
@Micky774 and @OmarManzoor: feel free to add your latest contributions to the description of this issue. :)

Labels: Performance, cython, Meta-issue
IterativeImputer - keep_empty_features parameter contains a bug

### Describe the bug

By making the "keep_empty_features" parameter to be True, then I found the behavior of the iterative imputation is the same as the simple imputation, which is a bug. By looking at the code, I found line 633 of _iterative.py "mask_missing_values[:, valid_mask] = True," which makes all features containing the missing value to be True, that missing it masks all data to be missing. 

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

X = np.random.rand(100,100)
X[:, 2:5] = np.nan
imp = IterativeImputer(
    max_iter=21,
    skip_complete=True,
    keep_empty_features=True,
    random_state=21
)
X_imp = imp.fit_transform(X)
```

### Expected Results

Expected keep_emtpy_features to only make columns that are all missing to be contained. 

### Actual Results

Still, it masks all features to be missing, and the behavior of iterative imputer is the same as simple imputer - initial imputation.

### Versions

```shell
Python dependencies:
      sklearn: 1.2.1
          pip: 23.0.1
   setuptools: 67.4.0
        numpy: 1.24.2
        scipy: 1.10.0
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.6.3
       joblib: 1.2.0
threadpoolctl: 3.1.0
```

COMMENT:
It seems that the ```IterativeImputer``` is not behaving as expected when the ```keep_empty_features``` parameter is set to True. According to the documentation, this parameter should only keep columns that are entirely missing (all NaNs) during imputation. However, it seems that all features are being masked to be missing.

I suggest you try setting the initial_strategy parameter to "mean" or "median" instead of the default "mean" to see if it resolves the issue. This should ensure that the initial imputation is not simply filling in ```NaN``` values with the mean or median, which can result in masking all features as missing.
```
X = np.random.rand(100,100)
X[:, 2:5] = np.nan
imp = IterativeImputer(
    max_iter=21,
    skip_complete=True,
    keep_empty_features=True,
    random_state=21,
    initial_strategy='median')  # or 'mean'
X_imp = imp.fit_transform(X)
```
If this not resolves the issue then.


You can modify the ```_iterative.py``` file in the ```sklearn/impute directory``` of your local installation of ```scikit-learn```. Specifically, you can replace the following line of code in the ```IterativeImputer``` class:

```
mask_missing_values[:, valid_mask] = True
```

with the following code:

```
mask_missing_values[:, valid_mask] = np.logical_or(
    mask_missing_values[:, valid_mask], np.all(np.isnan(X[:, valid_mask]), axis=0))
```

This change should correctly keep only the empty features when ```keep_empty_features``` is set to ```True```.

Labels: Bug, Needs Triage
IterativeImputer - keep_empty_features parameter contains a bug

### Describe the bug

By making the "keep_empty_features" parameter to be True, then I found the behavior of the iterative imputation is the same as the simple imputation, which is a bug. By looking at the code, I found line 633 of _iterative.py "mask_missing_values[:, valid_mask] = True," which makes all features containing the missing value to be True, that missing it masks all data to be missing. 

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

X = np.random.rand(100,100)
X[:, 2:5] = np.nan
imp = IterativeImputer(
    max_iter=21,
    skip_complete=True,
    keep_empty_features=True,
    random_state=21
)
X_imp = imp.fit_transform(X)
```

### Expected Results

Expected keep_emtpy_features to only make columns that are all missing to be contained. 

### Actual Results

Still, it masks all features to be missing, and the behavior of iterative imputer is the same as simple imputer - initial imputation.

### Versions

```shell
Python dependencies:
      sklearn: 1.2.1
          pip: 23.0.1
   setuptools: 67.4.0
        numpy: 1.24.2
        scipy: 1.10.0
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.6.3
       joblib: 1.2.0
threadpoolctl: 3.1.0
```

COMMENT:
Seems like the behavior is as expected, and the output is as expected in your case. Closing as doesn't seem to be a bug.

Happy to reopen if you provide expected values and comparing with actual values.

Labels: Bug, Needs Triage
IterativeImputer - keep_empty_features parameter contains a bug

### Describe the bug

By making the "keep_empty_features" parameter to be True, then I found the behavior of the iterative imputation is the same as the simple imputation, which is a bug. By looking at the code, I found line 633 of _iterative.py "mask_missing_values[:, valid_mask] = True," which makes all features containing the missing value to be True, that missing it masks all data to be missing. 

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

X = np.random.rand(100,100)
X[:, 2:5] = np.nan
imp = IterativeImputer(
    max_iter=21,
    skip_complete=True,
    keep_empty_features=True,
    random_state=21
)
X_imp = imp.fit_transform(X)
```

### Expected Results

Expected keep_emtpy_features to only make columns that are all missing to be contained. 

### Actual Results

Still, it masks all features to be missing, and the behavior of iterative imputer is the same as simple imputer - initial imputation.

### Versions

```shell
Python dependencies:
      sklearn: 1.2.1
          pip: 23.0.1
   setuptools: 67.4.0
        numpy: 1.24.2
        scipy: 1.10.0
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.6.3
       joblib: 1.2.0
threadpoolctl: 3.1.0
```

COMMENT:
Hi @adrinjalali 

Can you check #29375

Labels: Bug, Needs Triage
Appears that the checking of deprecated attributes in turn causing issues

![image](https://user-images.githubusercontent.com/68793903/225709709-5f69e6c3-15e2-42f2-bdff-67dfe86bad55.png)

##########
environment info
--------------
Microsoft Windows [Version 10.0.19045.2604]
(c) Microsoft Corporation. All rights reserved.

C:\WINDOWS\system32>pip show scikit-learn
Name: scikit-learn
Version: 1.2.2
Summary: A set of python modules for machine learning and data mining
Home-page: http://scikit-learn.org
Author:
Author-email:
License: new BSD
Location: c:\program files\python3\lib\site-packages
Requires: joblib, numpy, scipy, threadpoolctl
Required-by: ffn, lazypredict, lightgbm, shap

C:\WINDOWS\system32>python --version
Python 3.10.8

C:\WINDOWS\system32>

COMMENT:
Hi @tqiu592, can you provide a minimal reproducible example ?

Labels: Needs Triage
Appears that the checking of deprecated attributes in turn causing issues

![image](https://user-images.githubusercontent.com/68793903/225709709-5f69e6c3-15e2-42f2-bdff-67dfe86bad55.png)

##########
environment info
--------------
Microsoft Windows [Version 10.0.19045.2604]
(c) Microsoft Corporation. All rights reserved.

C:\WINDOWS\system32>pip show scikit-learn
Name: scikit-learn
Version: 1.2.2
Summary: A set of python modules for machine learning and data mining
Home-page: http://scikit-learn.org
Author:
Author-email:
License: new BSD
Location: c:\program files\python3\lib\site-packages
Requires: joblib, numpy, scipy, threadpoolctl
Required-by: ffn, lazypredict, lightgbm, shap

C:\WINDOWS\system32>python --version
Python 3.10.8

C:\WINDOWS\system32>

COMMENT:
I think https://github.com/scikit-learn/scikit-learn/pull/25668 might be related/relevant context.

Labels: Needs Triage
Appears that the checking of deprecated attributes in turn causing issues

![image](https://user-images.githubusercontent.com/68793903/225709709-5f69e6c3-15e2-42f2-bdff-67dfe86bad55.png)

##########
environment info
--------------
Microsoft Windows [Version 10.0.19045.2604]
(c) Microsoft Corporation. All rights reserved.

C:\WINDOWS\system32>pip show scikit-learn
Name: scikit-learn
Version: 1.2.2
Summary: A set of python modules for machine learning and data mining
Home-page: http://scikit-learn.org
Author:
Author-email:
License: new BSD
Location: c:\program files\python3\lib\site-packages
Requires: joblib, numpy, scipy, threadpoolctl
Required-by: ffn, lazypredict, lightgbm, shap

C:\WINDOWS\system32>python --version
Python 3.10.8

C:\WINDOWS\system32>

COMMENT:
> Hi @tqiu592, can you provide a minimal reproducible example ?

Whoops - my bad. I rushed to revert the scikit-learn version back to 1.1.x, and the codes no longer complaining. So I might not be able to generate the same error message as in the screenshot above.

But essentially this is an compatibility issue I guess, which arose when the RandomForestRegressor class calling the new base file in 1.2.2 version is attempting to wrap up the old attributes "self.estimator_" vs new attribute "self.estimator", the checking code on line 156 actually using the self.estimator, which doesn't exist in the old version of RFRegressor (it only has self.estimator_ ). hence the error message above.

Labels: Needs Triage
Appears that the checking of deprecated attributes in turn causing issues

![image](https://user-images.githubusercontent.com/68793903/225709709-5f69e6c3-15e2-42f2-bdff-67dfe86bad55.png)

##########
environment info
--------------
Microsoft Windows [Version 10.0.19045.2604]
(c) Microsoft Corporation. All rights reserved.

C:\WINDOWS\system32>pip show scikit-learn
Name: scikit-learn
Version: 1.2.2
Summary: A set of python modules for machine learning and data mining
Home-page: http://scikit-learn.org
Author:
Author-email:
License: new BSD
Location: c:\program files\python3\lib\site-packages
Requires: joblib, numpy, scipy, threadpoolctl
Required-by: ffn, lazypredict, lightgbm, shap

C:\WINDOWS\system32>python --version
Python 3.10.8

C:\WINDOWS\system32>

COMMENT:
> I think #25668 might be related/relevant context.

Yes, that looks super relevant, but maybe the fixes didn't go in the v1.2.2 release? (Or perhaps, it hasn't fully address the issue yet?)

Labels: Needs Triage
Appears that the checking of deprecated attributes in turn causing issues

![image](https://user-images.githubusercontent.com/68793903/225709709-5f69e6c3-15e2-42f2-bdff-67dfe86bad55.png)

##########
environment info
--------------
Microsoft Windows [Version 10.0.19045.2604]
(c) Microsoft Corporation. All rights reserved.

C:\WINDOWS\system32>pip show scikit-learn
Name: scikit-learn
Version: 1.2.2
Summary: A set of python modules for machine learning and data mining
Home-page: http://scikit-learn.org
Author:
Author-email:
License: new BSD
Location: c:\program files\python3\lib\site-packages
Requires: joblib, numpy, scipy, threadpoolctl
Required-by: ffn, lazypredict, lightgbm, shap

C:\WINDOWS\system32>python --version
Python 3.10.8

C:\WINDOWS\system32>

COMMENT:
Closing for lack of a reproducible. Happy to re-open if a reproducible is provided.

Labels: Needs Triage
make_classification: allow shuffling of samples only

### Describe the workflow you want to enable

At present, `make_classification` allows shuffling both the samples and the features.  I would like to create a sample dataset with some categorical features, and I can do that by thresholding some of the features.  I have set some of the features to be informative and some useless, so I must ensure that the features are not shuffled so I know which ones are which before doing the thresholding.  But I would still like to shuffle the samples.

### Describe your proposed solution

I think the simplest solution is to allow the `shuffle` parameter to take four different values: `True`, `False` (behaving as at present), `samples` (shuffle the samples only), `features` (shuffle the features only).

An alternative would be introducing a new `shuffle_samples` parameter, taking the values `None` (default: ignore this parameter), `True`, `False` (overriding the `shuffle` parameter).  But I think this alternative is more confusing.

### Describe alternatives you've considered, if relevant

The alternative, which is what I am currently doing, is to manually shuffle the samples myself.

### Additional context

_No response_

COMMENT:
Hi @juliangilbey I think there's a simple enough solution that doesn't require changing the API:
```py
from sklearn.datasets import make_classification
from sklearn.utils import shuffle
X, y = make_classification(shuffle=False)
X, y = shuflle(X, y)
```
I'm not strongly opposed to changing the API and in this regard, ``shuffle="samples"/"features"`` is the best alternative imo. Let's see what other maintainers think.

Labels: New Feature, Needs Decision - Include Feature
make_classification: allow shuffling of samples only

### Describe the workflow you want to enable

At present, `make_classification` allows shuffling both the samples and the features.  I would like to create a sample dataset with some categorical features, and I can do that by thresholding some of the features.  I have set some of the features to be informative and some useless, so I must ensure that the features are not shuffled so I know which ones are which before doing the thresholding.  But I would still like to shuffle the samples.

### Describe your proposed solution

I think the simplest solution is to allow the `shuffle` parameter to take four different values: `True`, `False` (behaving as at present), `samples` (shuffle the samples only), `features` (shuffle the features only).

An alternative would be introducing a new `shuffle_samples` parameter, taking the values `None` (default: ignore this parameter), `True`, `False` (overriding the `shuffle` parameter).  But I think this alternative is more confusing.

### Describe alternatives you've considered, if relevant

The alternative, which is what I am currently doing, is to manually shuffle the samples myself.

### Additional context

_No response_

COMMENT:
Hi @jeremiedbb!  Thanks, I hadn't spotted that functionality.  So I suggest closing this feature request; it doesn't really make sense to overload this function with this sort of special-casing when it is easy enough to do what is wanted using another sklearn function.

Labels: New Feature, Needs Decision - Include Feature
make_classification: allow shuffling of samples only

### Describe the workflow you want to enable

At present, `make_classification` allows shuffling both the samples and the features.  I would like to create a sample dataset with some categorical features, and I can do that by thresholding some of the features.  I have set some of the features to be informative and some useless, so I must ensure that the features are not shuffled so I know which ones are which before doing the thresholding.  But I would still like to shuffle the samples.

### Describe your proposed solution

I think the simplest solution is to allow the `shuffle` parameter to take four different values: `True`, `False` (behaving as at present), `samples` (shuffle the samples only), `features` (shuffle the features only).

An alternative would be introducing a new `shuffle_samples` parameter, taking the values `None` (default: ignore this parameter), `True`, `False` (overriding the `shuffle` parameter).  But I think this alternative is more confusing.

### Describe alternatives you've considered, if relevant

The alternative, which is what I am currently doing, is to manually shuffle the samples myself.

### Additional context

_No response_

COMMENT:
Thanks for the feedback. Let's close then :)

Labels: New Feature, Needs Decision - Include Feature
How to early stop in GradientBoostingClassifer?

How to track the model performance on an eval set that is provided from outside and early stop the tree building based upon the result?

Currently there is the option of ```validation_fraction``` along with ```n_iter_no_change``` available in the implementation
The issue with that approach:
1. Cannot use k-fold cross-validation.
2. Canont use custom metrics
Currently I solve it using the following, which is kind of hacky

```python
#create a gradient booster
gbc = GradientBoostingClassifier()

#define the metric function that you want to use to early stopping
def accuracy(y_true, y_preds):
    return #return the metric output here
#This class along with the monitor argument will enable early stopping
class early_stopping_gbc():
    def __init__(self, accuracy, eval_set, early_stopping_rounds = 20):
        self.accuracy = accuracy
        self.x_val = eval_set[0]
        self.y_val = eval_set[1]
        self.best_perf = 0.
        self.counter = 0
        self.early_stopping_rounds = early_stopping_rounds
    def __call__(self,i, model, local_vars):
        for counter, preds in enumerate(model.staged_predict_proba(self.x_val)):
            if counter == i:
                break
        acc = self.accuracy(self.y_val,preds[:,1])
        if acc > self.best_perf:
            self.best_perf = acc
            self.counter = 0
        else:
            self.counter += 1
        return self.counter > self.early_stopping_rounds

#Run gradient booster with early stopping on 20 rounds
gbc.fit(X_train,y_train, monitor = early_stopping_gbc(accuracy, [X_val,y_val], early_stopping_rounds = 20))
```

COMMENT:
This seems related to #25460 where we need a way to pass a custom validation set.

We need to design a consistent API across different scikit-learn estimator to handle this need.

Labels: New Feature, API, Needs Decision
Add sample weights to Nearest Neighbors classifiers

### Describe the workflow you want to enable

Both `KNeighborsClassifier` and `RadiusNeighborsClassifier` support providing weights, determined as a function of the distances of the neighboring samples. However, it is not possible to provide `sample_weight`, usually given to `fit`. I.e., I'd like to be able to give a weight for each sample, regardless of its distance.

### Describe your proposed solution

A straightforward implementation would be, during prediction, to multiply the `weights` by the `sample_weight`. For example, in `KNeighborsClassifier.predict_proba` it would look something like:
```python
if self.sample_weight_ is not None:
    weights *= self.sample_weight_[neigh_ind]
```
after having set `self.sample_weight_` during `fit`.

### Describe alternatives you've considered, if relevant

One could argue, that in the case of `KNeighborsClassifier` (as opposed to `RadiusNeighborsClassifier`), in addition to the weighing proposed above, the number of neighbors `n_neighbors` itself should be interpreted as "the number of effective samples after weighing". That means that for each query sample, we'd like to query $k$ nearest neighbors, such that $k$ is the minimal integer satisfying 
$$\sum_{i=0}^{k-1} w_i \ge n$$
where $w_i$ is the weight for the $i$-th nearest neighbor and $n$ is `n_neighbors`.

This is more tricky to implement, as we don't know in advance how many neighbors to query, and the number is different for each sample. A reasonable compromise would be to query "a bit more" (maybe double?) neighbors than `n_neighbors`, and then take only as many neighbors as needed for each sample. 

Another complication is that there can be a situation in which all queried neighbors are not enough to satisfy the condition. This should be handled similarly to the case in which (in the current implementation) `n_neighbors` is larger than the number of training samples, and raise a `ValueError`.

### Additional context

_No response_

COMMENT:
Please can you assign this to me

Labels: New Feature, wontfix