Talip7/scikit-learn-issues-multilabel huggingface dataset:
[MAINT] Modularize Tree code and Splitter utility functions

From #20819 , developers expressed issues with the current tree code.

Part of that is the modularity and as a result, maintainability/upgradability of such code. I propose the following super-short refactors to the `_tree.pyx/pxd` and `_splitter.pyx/pxd` files. This would be the first in a series of PRs to demonstrate that #20819 is fairly straightforward.

### Tree class

The Tree class assumes axis-aligned splits. **However, by modularizing the parts where the node values are set, and the feature values are computed for any given dataset, then any subclass of Tree can easily redefine only these two functions and a new `Splitter` to enable a "new" type of Tree.**

I propose adding the following two functions to the `Tree` class and altering `_add_node()`, `_apply_dense` to accompany these changes:

```cython
    cdef int _set_node_values(self, SplitRecord split_node,
            Node *node) nogil except -1:
        """Set node data.
        """
        node.feature = split_node.feature
        node.threshold = split_node.threshold
        return 1
    
  cdef DTYPE_t _compute_feature(self, const DTYPE_t[:] X_ndarray,
            Node *node) nogil:
        """Compute feature from a given data matrix, X.

        In axis-aligned trees, this is simply the value in the column of X
        for this specific feature.
        """
        # the feature index
        cdef DTYPE_t feature = X_ndarray[node.feature]
        return feature
```



### Splitter

Splitter uses functions only defined in the `.pyx` files. As a result, they are not available via `cimport`. **This poses an issue for #20819 and also for downstream packages that might want to define a new splitter that subclasses `Splitter`.** 

Here I propose adding the following functions into the `_splitter.pxd` file:

```
cdef inline void sort(DTYPE_t* Xf, SIZE_t* samples, SIZE_t n) nogil
cdef inline void swap(DTYPE_t* Xf, SIZE_t* samples, SIZE_t i, SIZE_t j) nogil
# and the other splitter utility functions.
...
`

### Misc Notes

This specifically addresses only issues with dense arrays. A follow-on issue and PR would be necessary for sparse arrays.

COMMENT:
> As part of my initiative to reduce the code in `trees`, (#22630 or #22328) functions are being removed. A concrete example is `_splitter`'s `sort` which may be removed for [c++ sort](https://en.cppreference.com/w/cpp/algorithm/sort). If external libraries want to use those functions, the safest thing to do is to vendor them.

I see that makes sense and would be awesome! In that case, would it be desirable to replace the existing usage of `sort` with the c++ sort? That is the main function that needs to be used in all splitters. Everything else is utility it seems like.

Labels: module:tree, Refactor
[MAINT] Modularize Tree code and Splitter utility functions

From #20819 , developers expressed issues with the current tree code.

Part of that is the modularity and as a result, maintainability/upgradability of such code. I propose the following super-short refactors to the `_tree.pyx/pxd` and `_splitter.pyx/pxd` files. This would be the first in a series of PRs to demonstrate that #20819 is fairly straightforward.

### Tree class

The Tree class assumes axis-aligned splits. **However, by modularizing the parts where the node values are set, and the feature values are computed for any given dataset, then any subclass of Tree can easily redefine only these two functions and a new `Splitter` to enable a "new" type of Tree.**

I propose adding the following two functions to the `Tree` class and altering `_add_node()`, `_apply_dense` to accompany these changes:

```cython
    cdef int _set_node_values(self, SplitRecord split_node,
            Node *node) nogil except -1:
        """Set node data.
        """
        node.feature = split_node.feature
        node.threshold = split_node.threshold
        return 1
    
  cdef DTYPE_t _compute_feature(self, const DTYPE_t[:] X_ndarray,
            Node *node) nogil:
        """Compute feature from a given data matrix, X.

        In axis-aligned trees, this is simply the value in the column of X
        for this specific feature.
        """
        # the feature index
        cdef DTYPE_t feature = X_ndarray[node.feature]
        return feature
```



### Splitter

Splitter uses functions only defined in the `.pyx` files. As a result, they are not available via `cimport`. **This poses an issue for #20819 and also for downstream packages that might want to define a new splitter that subclasses `Splitter`.** 

Here I propose adding the following functions into the `_splitter.pxd` file:

```
cdef inline void sort(DTYPE_t* Xf, SIZE_t* samples, SIZE_t n) nogil
cdef inline void swap(DTYPE_t* Xf, SIZE_t* samples, SIZE_t i, SIZE_t j) nogil
# and the other splitter utility functions.
...
`

### Misc Notes

This specifically addresses only issues with dense arrays. A follow-on issue and PR would be necessary for sparse arrays.

COMMENT:
I'm working on replacing sort all together. It requires minor refactoring of the splitter internals and benchmarks to make sure there are no performance regressions.

Labels: module:tree, Refactor
[MAINT] Modularize Tree code and Splitter utility functions

From #20819 , developers expressed issues with the current tree code.

Part of that is the modularity and as a result, maintainability/upgradability of such code. I propose the following super-short refactors to the `_tree.pyx/pxd` and `_splitter.pyx/pxd` files. This would be the first in a series of PRs to demonstrate that #20819 is fairly straightforward.

### Tree class

The Tree class assumes axis-aligned splits. **However, by modularizing the parts where the node values are set, and the feature values are computed for any given dataset, then any subclass of Tree can easily redefine only these two functions and a new `Splitter` to enable a "new" type of Tree.**

I propose adding the following two functions to the `Tree` class and altering `_add_node()`, `_apply_dense` to accompany these changes:

```cython
    cdef int _set_node_values(self, SplitRecord split_node,
            Node *node) nogil except -1:
        """Set node data.
        """
        node.feature = split_node.feature
        node.threshold = split_node.threshold
        return 1
    
  cdef DTYPE_t _compute_feature(self, const DTYPE_t[:] X_ndarray,
            Node *node) nogil:
        """Compute feature from a given data matrix, X.

        In axis-aligned trees, this is simply the value in the column of X
        for this specific feature.
        """
        # the feature index
        cdef DTYPE_t feature = X_ndarray[node.feature]
        return feature
```



### Splitter

Splitter uses functions only defined in the `.pyx` files. As a result, they are not available via `cimport`. **This poses an issue for #20819 and also for downstream packages that might want to define a new splitter that subclasses `Splitter`.** 

Here I propose adding the following functions into the `_splitter.pxd` file:

```
cdef inline void sort(DTYPE_t* Xf, SIZE_t* samples, SIZE_t n) nogil
cdef inline void swap(DTYPE_t* Xf, SIZE_t* samples, SIZE_t i, SIZE_t j) nogil
# and the other splitter utility functions.
...
`

### Misc Notes

This specifically addresses only issues with dense arrays. A follow-on issue and PR would be necessary for sparse arrays.

COMMENT:
> I'm working on replacing sort all together. It requires minor refactoring of the splitter internals and benchmarks to make sure there are no performance regressions.

Oh awesome! That'll make the diff even smaller in obliquePR. Lmk if you need any help with that.

Labels: module:tree, Refactor
[MAINT] Modularize Tree code and Splitter utility functions

From #20819 , developers expressed issues with the current tree code.

Part of that is the modularity and as a result, maintainability/upgradability of such code. I propose the following super-short refactors to the `_tree.pyx/pxd` and `_splitter.pyx/pxd` files. This would be the first in a series of PRs to demonstrate that #20819 is fairly straightforward.

### Tree class

The Tree class assumes axis-aligned splits. **However, by modularizing the parts where the node values are set, and the feature values are computed for any given dataset, then any subclass of Tree can easily redefine only these two functions and a new `Splitter` to enable a "new" type of Tree.**

I propose adding the following two functions to the `Tree` class and altering `_add_node()`, `_apply_dense` to accompany these changes:

```cython
    cdef int _set_node_values(self, SplitRecord split_node,
            Node *node) nogil except -1:
        """Set node data.
        """
        node.feature = split_node.feature
        node.threshold = split_node.threshold
        return 1
    
  cdef DTYPE_t _compute_feature(self, const DTYPE_t[:] X_ndarray,
            Node *node) nogil:
        """Compute feature from a given data matrix, X.

        In axis-aligned trees, this is simply the value in the column of X
        for this specific feature.
        """
        # the feature index
        cdef DTYPE_t feature = X_ndarray[node.feature]
        return feature
```



### Splitter

Splitter uses functions only defined in the `.pyx` files. As a result, they are not available via `cimport`. **This poses an issue for #20819 and also for downstream packages that might want to define a new splitter that subclasses `Splitter`.** 

Here I propose adding the following functions into the `_splitter.pxd` file:

```
cdef inline void sort(DTYPE_t* Xf, SIZE_t* samples, SIZE_t n) nogil
cdef inline void swap(DTYPE_t* Xf, SIZE_t* samples, SIZE_t i, SIZE_t j) nogil
# and the other splitter utility functions.
...
`

### Misc Notes

This specifically addresses only issues with dense arrays. A follow-on issue and PR would be necessary for sparse arrays.

COMMENT:
FYI, I just have opened https://github.com/scikit-learn/scikit-learn/pull/22760.

Labels: module:tree, Refactor
Unable to compute AgglomerativeClustering with affinity 'precomputed' and linkage 'ward'

### Describe the bug

When trying to compute AgglomerativeClustering with affinity='precomputed', linkage='ward' I get the following error:

`ValueError: precomputed was provided as affinity. Ward can only work with euclidean distances.`
I think this is intended. What is the reason behind it?

### Steps/Code to Reproduce

```
from sklearn.cluster import AgglomerativeClustering
from sklearn.metrics.pairwise import euclidean_distances
import numpy as np

X = np.random.normal(size=(2,4))
dist=euclidean_distances(X)
cluster_dist = AgglomerativeClustering(n_clusters=2, affinity='precomputed', linkage='ward')
results_dist = cluster_dist.fit(dist)
```

### Expected Results

The resulting clusters

### Actual Results

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [122], in <module>
      2 dist=euclidean_distances(X)
      3 cluster_dist = AgglomerativeClustering(n_clusters=n_cluster, affinity='precomputed', linkage='ward')
----> 4 results_dist = cluster_dist.fit(dist)

File ~/Documents/technical/xai_cats_dogs/xai_cats_dogs_venv/lib/python3.9/site-packages/sklearn/cluster/_agglomerative.py:918, in AgglomerativeClustering.fit(self, X, y)
    900 """Fit the hierarchical clustering from features, or distance matrix.
    901 
    902 Parameters
   (...)
    915     Returns the fitted instance.
    916 """
    917 X = self._validate_data(X, ensure_min_samples=2, estimator=self)
--> 918 return self._fit(X)

File ~/Documents/technical/xai_cats_dogs/xai_cats_dogs_venv/lib/python3.9/site-packages/sklearn/cluster/_agglomerative.py:955, in AgglomerativeClustering._fit(self, X)
    950     raise ValueError(
    951         "compute_full_tree must be True if distance_threshold is set."
    952     )
    954 if self.linkage == "ward" and self.affinity != "euclidean":
--> 955     raise ValueError(
    956         "%s was provided as affinity. Ward can only "
    957         "work with euclidean distances." % (self.affinity,)
    958     )
    960 if self.linkage not in _TREE_BUILDERS:
    961     raise ValueError(
    962         "Unknown linkage type %s. Valid options are %s"
    963         % (self.linkage, _TREE_BUILDERS.keys())
    964     )

ValueError: precomputed was provided as affinity. Ward can only work with euclidean distances.
```

### Versions

```shell
System:
    python: 3.9.7 (default, Oct 13 2021, 06:45:31)  [Clang 13.0.0 (clang-1300.0.29.3)]
executable: /Users/cristianpachon/Documents/technical/xai_cats_dogs/xai_cats_dogs_venv/bin/python
   machine: macOS-11.4-x86_64-i386-64bit

Python dependencies:
          pip: 21.3.1
   setuptools: 60.5.0
      sklearn: 1.0.2
        numpy: 1.22.2
        scipy: 1.8.0
       Cython: None
       pandas: 1.4.1
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```
```

COMMENT:
Similar to  #15287.

Labels: Bug, module:cluster, Needs Triage
KBinsDiscretizer calling get_feature_names_out only works for encode = "onehot"

### Describe the bug

When using `KBinsDiscretizer` with encode set to anything but "onehot", calling `get_feature_names_out` on a fitted instance raises an AttributeError as shown below. It looks like as if the `self._encode` attribute is only available when `encode="onehot"`. The example below might look a bit useless, but when integrating  `KBinsDiscretizer` on a `ColumnTransformer`, I usually call `get_feature_names_out` on that instance to see what type of features it produces. This of course does break if any of the transformers used doesn't properly implement the `get_feature_names_out` API.

### Steps/Code to Reproduce

```python
import pandas as pd
from sklearn.preprocessing import KBinsDiscretizer

data = pd.DataFrame({"a":list(range(100))})
kbd = KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='quantile').fit(data)
names = kbd.get_feature_names_out()
```

### Expected Results

The names of the output feature, which is probably the same as the input feature.

### Actual Results

```pytb
Traceback (most recent call last):
  File
line 3251, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-19-fbbefdd5a8b5>", line 1, in <module>
    kbd.get_feature_names_out()
  File "/.venv/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py", line 396, in get_feature_names_out
    return self._encoder.get_feature_names_out(input_features)
AttributeError: 'KBinsDiscretizer' object has no attribute '_encoder'
```

### Versions

```shell
System:
    python: 3.9.7 (default, Nov 10 2021, 10:51:48)  [Clang 11.1.0 ]
   machine: macOS-10.16-x86_64-i386-64bit
Python dependencies:
          pip: 22.0.3
   setuptools: 58.3.0
      sklearn: 1.0.2
        numpy: 1.22.2
        scipy: 1.8.0
       Cython: None
       pandas: 1.4.1
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0
Built with OpenMP: True
```

COMMENT:
I can reproduce. Since we don't use any internal encoder with `ordinal`, the names of the features should be directly the `feature_names_in_`.

Labels: Bug
KBinsDiscretizer calling get_feature_names_out only works for encode = "onehot"

### Describe the bug

When using `KBinsDiscretizer` with encode set to anything but "onehot", calling `get_feature_names_out` on a fitted instance raises an AttributeError as shown below. It looks like as if the `self._encode` attribute is only available when `encode="onehot"`. The example below might look a bit useless, but when integrating  `KBinsDiscretizer` on a `ColumnTransformer`, I usually call `get_feature_names_out` on that instance to see what type of features it produces. This of course does break if any of the transformers used doesn't properly implement the `get_feature_names_out` API.

### Steps/Code to Reproduce

```python
import pandas as pd
from sklearn.preprocessing import KBinsDiscretizer

data = pd.DataFrame({"a":list(range(100))})
kbd = KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='quantile').fit(data)
names = kbd.get_feature_names_out()
```

### Expected Results

The names of the output feature, which is probably the same as the input feature.

### Actual Results

```pytb
Traceback (most recent call last):
  File
line 3251, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-19-fbbefdd5a8b5>", line 1, in <module>
    kbd.get_feature_names_out()
  File "/.venv/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py", line 396, in get_feature_names_out
    return self._encoder.get_feature_names_out(input_features)
AttributeError: 'KBinsDiscretizer' object has no attribute '_encoder'
```

### Versions

```shell
System:
    python: 3.9.7 (default, Nov 10 2021, 10:51:48)  [Clang 11.1.0 ]
   machine: macOS-10.16-x86_64-i386-64bit
Python dependencies:
          pip: 22.0.3
   setuptools: 58.3.0
      sklearn: 1.0.2
        numpy: 1.22.2
        scipy: 1.8.0
       Cython: None
       pandas: 1.4.1
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0
Built with OpenMP: True
```

COMMENT:
I feel like the best course of action in this scenario is to use the make_pipeline function and execute all of the transformations as pipeline steps. My immediate solution was to process the target encoding transformations independently for each feature.

Labels: Bug
KBinsDiscretizer calling get_feature_names_out only works for encode = "onehot"

### Describe the bug

When using `KBinsDiscretizer` with encode set to anything but "onehot", calling `get_feature_names_out` on a fitted instance raises an AttributeError as shown below. It looks like as if the `self._encode` attribute is only available when `encode="onehot"`. The example below might look a bit useless, but when integrating  `KBinsDiscretizer` on a `ColumnTransformer`, I usually call `get_feature_names_out` on that instance to see what type of features it produces. This of course does break if any of the transformers used doesn't properly implement the `get_feature_names_out` API.

### Steps/Code to Reproduce

```python
import pandas as pd
from sklearn.preprocessing import KBinsDiscretizer

data = pd.DataFrame({"a":list(range(100))})
kbd = KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='quantile').fit(data)
names = kbd.get_feature_names_out()
```

### Expected Results

The names of the output feature, which is probably the same as the input feature.

### Actual Results

```pytb
Traceback (most recent call last):
  File
line 3251, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-19-fbbefdd5a8b5>", line 1, in <module>
    kbd.get_feature_names_out()
  File "/.venv/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py", line 396, in get_feature_names_out
    return self._encoder.get_feature_names_out(input_features)
AttributeError: 'KBinsDiscretizer' object has no attribute '_encoder'
```

### Versions

```shell
System:
    python: 3.9.7 (default, Nov 10 2021, 10:51:48)  [Clang 11.1.0 ]
   machine: macOS-10.16-x86_64-i386-64bit
Python dependencies:
          pip: 22.0.3
   setuptools: 58.3.0
      sklearn: 1.0.2
        numpy: 1.22.2
        scipy: 1.8.0
       Cython: None
       pandas: 1.4.1
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0
Built with OpenMP: True
```

COMMENT:
> use the make_pipeline function and execute all of the transformations as pipeline steps

yes, i have tried all the steps above but still get the same error messages

Labels: Bug
How to use Hierarchical Navigable Small Worlds (HNSW) and LSH for Data classification rather than just retrieving Nearest neighbours

I want to use Hierarchical Navigable Small Worlds (HNSW) and LSH for data classification. How can I modify their fit and train functions???

For example if you want to use them like ball tree or kd tree from the scikit learn library for classifying data.

COMMENT:
This is a duplicate of https://github.com/scikit-learn/scikit-learn/discussions/22738 . The issue tracker is for scikit-learn development.

Labels: Needs Triage
Implement Repeated Group CV

https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/model_selection/_split.py#L505

I tried to implement repeated group cv using GroupKFold and _RepeatedSplits. But it did not work unless I included `shuffle=False, random_state=None` to `def __ini__ ()`

I just want to check if I am why `shuffle=False, random_state=None` got excluded from `def __ini__ ()`, and if it is okay for me to do the following implementation: `def __init__(self, n_splits=5, shuffle=False, random_state=None)`

COMMENT:
The public API for `GroupKFold` does not have `shuffle` or `random_state` because there is no random element to the splitter. (`GroupKFold` itself does not support shuffling)

If we go by the design of the [current splitters](https://scikit-learn.org/stable/modules/classes.html#splitter-classes), then a "repeated group k fold" would be it's own class, `RepeatedGroupKFold`. In this case, we would not need to modify `GroupKFold` at all.

Labels: New Feature, module:model_selection, Needs Decision - Include Feature
Implement Repeated Group CV

https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/model_selection/_split.py#L505

I tried to implement repeated group cv using GroupKFold and _RepeatedSplits. But it did not work unless I included `shuffle=False, random_state=None` to `def __ini__ ()`

I just want to check if I am why `shuffle=False, random_state=None` got excluded from `def __ini__ ()`, and if it is okay for me to do the following implementation: `def __init__(self, n_splits=5, shuffle=False, random_state=None)`

COMMENT:
@thomasjpfan We're working with a small dataset and would greatly appreciate the ability to use a `RepeatedGroupKFold` class. Is there anything I can do to help add it to sklearn ASAP?

Labels: New Feature, module:model_selection, Needs Decision - Include Feature