Talip7/scikit-learn-issues-multilabel huggingface dataset:
BUG (maybe) wrong node bound spread in KernelDensity

### Describe the bug

https://github.com/scikit-learn/scikit-learn/blob/a5620f45614ac3f849c430f53146a66319e4908b/sklearn/neighbors/_binary_tree.pxi.tp#L2114-L2116

https://github.com/scikit-learn/scikit-learn/blob/a5620f45614ac3f849c430f53146a66319e4908b/sklearn/neighbors/_binary_tree.pxi.tp#L2121-L2123

These lines do not seem to be correct. The right-hand side seems to be node log max bound but not node log bound spread. Should they instead be

```python
node_log_bound_spreads[i1] = logsubexp(log(N1)+ compute_log_kernel(dist_LB_1, h, kernel),
                                       node_log_min_bounds[i1])
node_log_bound_spreads[i2] = logsubexp(log(N2)+ compute_log_kernel(dist_LB_2, h, kernel),
                                       node_log_min_bounds[i2])
```

I don't really find an unexpected result or something, just got confused when reading the code.

COMMENT:
In reponse to https://github.com/scikit-learn/scikit-learn/pull/27971#discussion_r1687755714 @glemaitre I'm providing some more intuition here:

https://github.com/scikit-learn/scikit-learn/blob/a5620f45614ac3f849c430f53146a66319e4908b/sklearn/neighbors/_binary_tree.pxi.tp#L2111-L2116

In the above snippet `dist_UB_1` and `dist_LB_1` are respectively the maximum and minimum distance between the point to node `i1`. The node lower bound should thus be node weight multiplied by the kernel computed from the maximum distance (because kernel function is decreasing) and if you take the logarithm you can see that the first part is correct. However the second part seems to be computing the node upper bound instead of the node bound spread (upper bound minus lower bound) since it is the same equation as when computing the node lower bound, except replacing maximum distance with minimum. I'm proposing that we should subtract by node lower bound (so `logsubexp` by `node_log_min_bounds`) here.

https://github.com/scikit-learn/scikit-learn/blob/a5620f45614ac3f849c430f53146a66319e4908b/sklearn/neighbors/_binary_tree.pxi.tp#L2251-L2257

The above is one of the other places where node boundaries are updated for your reference.

Labels: Needs Investigation
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=58833&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Sep 11, 2023)
- test_multi_target_sparse_regression[dok_array]

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=58914&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf) on Sep 12, 2023

Labels: Needs Triage
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=58833&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Sep 11, 2023)
- test_multi_target_sparse_regression[dok_array]

COMMENT:
Looks like the bot only cares about the status of the main tests and not the rest e.g. doctests in `rst` doc files, so this can be closed. #27345 will skip the doctests so that the full build will hopefully pass.

Labels: Needs Triage
incorrect intercept in LinearRegression when `copy_X=False`?

### Describe the bug

The intercept is incorrectly computed when using sample weights and `copy_X=False`.

The docs only say that `X` may be overwritten by setting this flag but the intercept also changes.

### Steps/Code to Reproduce

import numpy as np
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(20)
n, p = 50, 3

X = rng.standard_normal((n, p))
Y = rng.standard_normal(n) + 2 + X @ rng.standard_normal(p)
W = rng.uniform(0, 1, size=(n,))

lm_nocp = LinearRegression(copy_X=False)
lm_nocp.fit(X, Y, sample_weight=W)

lm_cp = LinearRegression(copy_X=True)
lm_cp.fit(X, Y, sample_weight=W)

print('checking with weights', np.allclose(lm_nocp.intercept_, lm_cp.intercept_))
print(lm_nocp.intercept_, lm_cp.intercept_)

lm_nocp.fit(X, Y)
lm_cp.fit(X, Y)

print('checking without weights', lm_nocp.intercept_ == lm_cp.intercept_)
print(lm_nocp.intercept_, lm_cp.intercept_)

### Expected Results

checking with weights True
1.6871821183020286 1.6871821183020286
checking without weights True
1.874959984635478 1.874959984635478


### Actual Results

checking with weights False
1.8599658087359068 1.6871821183020286
checking without weights False
1.7039080345298554 1.874959984635478


### Versions

```shell
System:
    python: 3.10.12 (main, Jul  5 2023, 15:02:25) [Clang 14.0.6 ]
executable: /Users/jtaylo/anaconda3/envs/pyglmnet/bin/python
   machine: macOS-13.5-arm64-arm-64bit

Python dependencies:
      sklearn: 1.3.0
          pip: 23.2.1
   setuptools: 68.0.0
        numpy: 1.24.4
        scipy: 1.11.1
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.7.2
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 10
         prefix: libopenblas
       filepath: /Users/jtaylo/anaconda3/envs/pyglmnet/lib/python3.10/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.21
threading_layer: pthreads
   architecture: armv8

       user_api: openmp
   internal_api: openmp
    num_threads: 10
         prefix: libomp
       filepath: /Users/jtaylo/anaconda3/envs/pyglmnet/lib/python3.10/site-packages/sklearn/.dylibs/libomp.dylib
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 10
         prefix: libopenblas
       filepath: /Users/jtaylo/anaconda3/envs/pyglmnet/lib/python3.10/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: armv8
```

COMMENT:
On second thought this difference is probably just the effect of X being overwritten. Will check that first

Docstring not completely clear when X might be overwritten.

Labels: Bug, Needs Triage
incorrect intercept in LinearRegression when `copy_X=False`?

### Describe the bug

The intercept is incorrectly computed when using sample weights and `copy_X=False`.

The docs only say that `X` may be overwritten by setting this flag but the intercept also changes.

### Steps/Code to Reproduce

import numpy as np
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(20)
n, p = 50, 3

X = rng.standard_normal((n, p))
Y = rng.standard_normal(n) + 2 + X @ rng.standard_normal(p)
W = rng.uniform(0, 1, size=(n,))

lm_nocp = LinearRegression(copy_X=False)
lm_nocp.fit(X, Y, sample_weight=W)

lm_cp = LinearRegression(copy_X=True)
lm_cp.fit(X, Y, sample_weight=W)

print('checking with weights', np.allclose(lm_nocp.intercept_, lm_cp.intercept_))
print(lm_nocp.intercept_, lm_cp.intercept_)

lm_nocp.fit(X, Y)
lm_cp.fit(X, Y)

print('checking without weights', lm_nocp.intercept_ == lm_cp.intercept_)
print(lm_nocp.intercept_, lm_cp.intercept_)

### Expected Results

checking with weights True
1.6871821183020286 1.6871821183020286
checking without weights True
1.874959984635478 1.874959984635478


### Actual Results

checking with weights False
1.8599658087359068 1.6871821183020286
checking without weights False
1.7039080345298554 1.874959984635478


### Versions

```shell
System:
    python: 3.10.12 (main, Jul  5 2023, 15:02:25) [Clang 14.0.6 ]
executable: /Users/jtaylo/anaconda3/envs/pyglmnet/bin/python
   machine: macOS-13.5-arm64-arm-64bit

Python dependencies:
      sklearn: 1.3.0
          pip: 23.2.1
   setuptools: 68.0.0
        numpy: 1.24.4
        scipy: 1.11.1
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.7.2
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 10
         prefix: libopenblas
       filepath: /Users/jtaylo/anaconda3/envs/pyglmnet/lib/python3.10/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.21
threading_layer: pthreads
   architecture: armv8

       user_api: openmp
   internal_api: openmp
    num_threads: 10
         prefix: libomp
       filepath: /Users/jtaylo/anaconda3/envs/pyglmnet/lib/python3.10/site-packages/sklearn/.dylibs/libomp.dylib
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 10
         prefix: libopenblas
       filepath: /Users/jtaylo/anaconda3/envs/pyglmnet/lib/python3.10/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: armv8
```

COMMENT:
Yup, that was it. My bad.

Labels: Bug, Needs Triage
RandomForest{Classifier,Regressor} split criterion documentation

### Describe the issue linked to the documentation

There's no where in the documentation that explains what method is used to identify which values to consider as candidate splits. For example, for regression, an exhaustive method would be to sort each feature and then use the halfway point between each feature value. In this case there would be O(N*F) candidate splits where N=# data and F= # features. Another way would be to randomly choose K values between the min and max of each feature.

Does anyone know what method is used? Could someone point me to where in the code this occurs?

Contributors should read this: https://github.com/scikit-learn/scikit-learn/issues/27159#issuecomment-1722203111

COMMENT:
I'm working on this

Labels: Documentation, help wanted
RandomForest{Classifier,Regressor} split criterion documentation

### Describe the issue linked to the documentation

There's no where in the documentation that explains what method is used to identify which values to consider as candidate splits. For example, for regression, an exhaustive method would be to sort each feature and then use the halfway point between each feature value. In this case there would be O(N*F) candidate splits where N=# data and F= # features. Another way would be to randomly choose K values between the min and max of each feature.

Does anyone know what method is used? Could someone point me to where in the code this occurs?

Contributors should read this: https://github.com/scikit-learn/scikit-learn/issues/27159#issuecomment-1722203111

COMMENT:
@howarth just to be clear , you were unable to find the methods for evaluating candidate splits , correct?

Labels: Documentation, help wanted
RandomForest{Classifier,Regressor} split criterion documentation

### Describe the issue linked to the documentation

There's no where in the documentation that explains what method is used to identify which values to consider as candidate splits. For example, for regression, an exhaustive method would be to sort each feature and then use the halfway point between each feature value. In this case there would be O(N*F) candidate splits where N=# data and F= # features. Another way would be to randomly choose K values between the min and max of each feature.

Does anyone know what method is used? Could someone point me to where in the code this occurs?

Contributors should read this: https://github.com/scikit-learn/scikit-learn/issues/27159#issuecomment-1722203111

COMMENT:
/take

Labels: Documentation, help wanted
RandomForest{Classifier,Regressor} split criterion documentation

### Describe the issue linked to the documentation

There's no where in the documentation that explains what method is used to identify which values to consider as candidate splits. For example, for regression, an exhaustive method would be to sort each feature and then use the halfway point between each feature value. In this case there would be O(N*F) candidate splits where N=# data and F= # features. Another way would be to randomly choose K values between the min and max of each feature.

Does anyone know what method is used? Could someone point me to where in the code this occurs?

Contributors should read this: https://github.com/scikit-learn/scikit-learn/issues/27159#issuecomment-1722203111

COMMENT:
> @howarth just to be clear , you were unable to find the methods for evaluating candidate splits , correct?

@Snerdify I was able to find documentation on the split criterion methods like entropy, gini, etc. What I could not find was documentation on how the set of candidate values is populated for each feature (the set of values for which you compute the split criterion, and then take the max or min). In my initial post I gave two example methods that are commonly used, but I don’t know if either of those are use, or if another is used. 

What I’m actually trying to understand is the computational complexity of creating a single node. To know that I need to know how many splits are calculated at each node.

Labels: Documentation, help wanted
RandomForest{Classifier,Regressor} split criterion documentation

### Describe the issue linked to the documentation

There's no where in the documentation that explains what method is used to identify which values to consider as candidate splits. For example, for regression, an exhaustive method would be to sort each feature and then use the halfway point between each feature value. In this case there would be O(N*F) candidate splits where N=# data and F= # features. Another way would be to randomly choose K values between the min and max of each feature.

Does anyone know what method is used? Could someone point me to where in the code this occurs?

Contributors should read this: https://github.com/scikit-learn/scikit-learn/issues/27159#issuecomment-1722203111

COMMENT:
@howarth got it. Working on it

Labels: Documentation, help wanted
RandomForest{Classifier,Regressor} split criterion documentation

### Describe the issue linked to the documentation

There's no where in the documentation that explains what method is used to identify which values to consider as candidate splits. For example, for regression, an exhaustive method would be to sort each feature and then use the halfway point between each feature value. In this case there would be O(N*F) candidate splits where N=# data and F= # features. Another way would be to randomly choose K values between the min and max of each feature.

Does anyone know what method is used? Could someone point me to where in the code this occurs?

Contributors should read this: https://github.com/scikit-learn/scikit-learn/issues/27159#issuecomment-1722203111

COMMENT:
@howarth 
**Current Approach:**
Scikit-learn's decision tree implementation uses the "best split" strategy as the default approach to identify candidate splits. This means that for each feature, all unique values present in the training data are considered as potential splitting points. The criterion (e.g., Gini impurity or entropy) is computed for each unique value, and the value that maximizes the reduction in impurity is selected as the splitting point for that feature.

**Alternative/Fix:**
If you are interested in an alternative approach for selecting candidate splits, you could consider using the "random split" strategy. In this approach, you randomly choose a subset of candidate values from the feature's range and evaluate the criterion for each of those values. This can significantly reduce the computational complexity, especially for features with a large number of unique values.

**Code Location:**
If you want to delve into the code to understand how candidate splits are identified, you can look into the _splitter.pyx file in scikit-learn's source code. This file contains the core logic for finding the best split in decision trees. Specifically, you can explore the `BestSplitter` class, where the process of finding the best split point is implemented. You can find this here:
https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/tree/_splitter.pyx

Labels: Documentation, help wanted
RandomForest{Classifier,Regressor} split criterion documentation

### Describe the issue linked to the documentation

There's no where in the documentation that explains what method is used to identify which values to consider as candidate splits. For example, for regression, an exhaustive method would be to sort each feature and then use the halfway point between each feature value. In this case there would be O(N*F) candidate splits where N=# data and F= # features. Another way would be to randomly choose K values between the min and max of each feature.

Does anyone know what method is used? Could someone point me to where in the code this occurs?

Contributors should read this: https://github.com/scikit-learn/scikit-learn/issues/27159#issuecomment-1722203111

COMMENT:
@AyushChauhan-ui thank you!

Labels: Documentation, help wanted
RandomForest{Classifier,Regressor} split criterion documentation

### Describe the issue linked to the documentation

There's no where in the documentation that explains what method is used to identify which values to consider as candidate splits. For example, for regression, an exhaustive method would be to sort each feature and then use the halfway point between each feature value. In this case there would be O(N*F) candidate splits where N=# data and F= # features. Another way would be to randomly choose K values between the min and max of each feature.

Does anyone know what method is used? Could someone point me to where in the code this occurs?

Contributors should read this: https://github.com/scikit-learn/scikit-learn/issues/27159#issuecomment-1722203111

COMMENT:
@howarth , @AyushChauhan-ui , in the scikit-learn documentation , there is no detailed documentation for gini, variance reduction (which is used for regression trees) , entropy, information gain . Also there is one more splitting method we could add called chi-square splitting method.

Labels: Documentation, help wanted
RandomForest{Classifier,Regressor} split criterion documentation

### Describe the issue linked to the documentation

There's no where in the documentation that explains what method is used to identify which values to consider as candidate splits. For example, for regression, an exhaustive method would be to sort each feature and then use the halfway point between each feature value. In this case there would be O(N*F) candidate splits where N=# data and F= # features. Another way would be to randomly choose K values between the min and max of each feature.

Does anyone know what method is used? Could someone point me to where in the code this occurs?

Contributors should read this: https://github.com/scikit-learn/scikit-learn/issues/27159#issuecomment-1722203111

COMMENT:
Also the complexity that is mentioned is for binary trees and nothing specific about what is the computational complexity for particular decision tree algorithms like CART , ID3. That could be added too.

Labels: Documentation, help wanted
RandomForest{Classifier,Regressor} split criterion documentation

### Describe the issue linked to the documentation

There's no where in the documentation that explains what method is used to identify which values to consider as candidate splits. For example, for regression, an exhaustive method would be to sort each feature and then use the halfway point between each feature value. In this case there would be O(N*F) candidate splits where N=# data and F= # features. Another way would be to randomly choose K values between the min and max of each feature.

Does anyone know what method is used? Could someone point me to where in the code this occurs?

Contributors should read this: https://github.com/scikit-learn/scikit-learn/issues/27159#issuecomment-1722203111

COMMENT:
> @howarth , @AyushChauhan-ui , in the scikit-learn documentation , there is no detailed documentation for gini, variance reduction (which is used for regression trees) , entropy, information gain . 

https://scikit-learn.org/stable/modules/tree.html#mathematical-formulation

Labels: Documentation, help wanted
RandomForest{Classifier,Regressor} split criterion documentation

### Describe the issue linked to the documentation

There's no where in the documentation that explains what method is used to identify which values to consider as candidate splits. For example, for regression, an exhaustive method would be to sort each feature and then use the halfway point between each feature value. In this case there would be O(N*F) candidate splits where N=# data and F= # features. Another way would be to randomly choose K values between the min and max of each feature.

Does anyone know what method is used? Could someone point me to where in the code this occurs?

Contributors should read this: https://github.com/scikit-learn/scikit-learn/issues/27159#issuecomment-1722203111

COMMENT:
> [...] for regression, an exhaustive method would be to sort each feature and then use the halfway point between each feature value. In this case there would be O(N*F) candidate splits where N=# data and F= # features. Another way would be to randomly choose K values between the min and max of each feature.

It's the former. See e.g. https://github.com/scikit-learn/scikit-learn/blob/a05eb6bbe30bc40904274c87afb4ac93bcd60168/sklearn/tree/_splitter.pyx#L472

I don't think it's essential to spell this out in the User Guide, but I also don't think it would hurt. There have been some alternatives used, like left or right endpoints instead of midpoints, see e.g. https://datascience.stackexchange.com/q/32638/55122

Also, uniform selection in the feature range _is_ used, e.g. in IsolationForests.

Labels: Documentation, help wanted