Talip7/scikit-learn-issues-multilabel huggingface dataset:
Balltree's node_data radii seems to be incorrect

### Describe the bug

The node data from constructing a BallTree seems to be incorrect. Each parent node's radius should always be greater than or equal to both its child's radiuses, since the radius is that of the smallest hypersphere containing all the points within that node, and the children of a node is just a subset of that node. Therefore if a child's radius is higher than its parents, it could just adopt the hypersphere of its parent as its current sphere to have a smaller radius.

That being said, I did some testing internally and it is very common for either or both children of a node in a BallTree to have a higher radius than their parent! Can someone point out what I might be doing wrong here, or is the BallTree radius simply an approximation? I don't think it is either, since we see [here](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neighbors/_ball_tree.pyx#L92) that the radius of a node is computed as the largest distance from its centroid (mean of all its vectors) to any vector within the node. 

Edit: I think I understand why child nodes could potentially have a larger radius than their parent, due to the fact that we use a new centroid based on the points within a node each time (instead of comparing to the parent centroid). It's possible that we end up using a worse centroid which nets a larger radius. However, I still think the radius calculation is off - see my follow up comment below.

### Steps/Code to Reproduce
```
from sklearn.neighbors import BallTree    
import numpy as np

np.random.seed(3)
Z = np.random.random((1000, 512)) 
model = BallTree(Z, leaf_size=1, metric='l2')
(
    _,
    idx_array,
    node_data,
    _,
) = model.get_arrays()

root = node_data[0]
lchild = node_data[1]
rchild = node_data[2]

_, _, _, root_radius = root
_, _, _, left_radius = lchild
_, _, _, right_radius = rchild


if left_radius > root_radius:
    print(left_radius, root_radius)
if right_radius > root_radius:
    print(right_radius, root_radius)
```

### Expected Results

Nothing should be printed.

### Actual Results

`6.93039685892859 6.923245275411173`


### Versions

```shell
System:
    python: 3.8.12 (default, Oct 12 2021, 06:23:56)  [Clang 10.0.0 ]
executable: /opt/miniconda3/envs/models/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 21.2.4
   setuptools: 62.1.0
      sklearn: 0.24.2
        numpy: 1.20.3
        scipy: 1.7.1
       Cython: 0.29.24
       pandas: 1.1.5
   matplotlib: 3.4.2
       joblib: 1.0.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
It seems like a bug to me after re-reading https://en.wikipedia.org/wiki/Ball_tree.

Would you be interested in opening a pull request with a candidate fix and your reproducer as non-regression test?

Labels: module:neighbors, Needs Investigation
Balltree's node_data radii seems to be incorrect

### Describe the bug

The node data from constructing a BallTree seems to be incorrect. Each parent node's radius should always be greater than or equal to both its child's radiuses, since the radius is that of the smallest hypersphere containing all the points within that node, and the children of a node is just a subset of that node. Therefore if a child's radius is higher than its parents, it could just adopt the hypersphere of its parent as its current sphere to have a smaller radius.

That being said, I did some testing internally and it is very common for either or both children of a node in a BallTree to have a higher radius than their parent! Can someone point out what I might be doing wrong here, or is the BallTree radius simply an approximation? I don't think it is either, since we see [here](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neighbors/_ball_tree.pyx#L92) that the radius of a node is computed as the largest distance from its centroid (mean of all its vectors) to any vector within the node. 

Edit: I think I understand why child nodes could potentially have a larger radius than their parent, due to the fact that we use a new centroid based on the points within a node each time (instead of comparing to the parent centroid). It's possible that we end up using a worse centroid which nets a larger radius. However, I still think the radius calculation is off - see my follow up comment below.

### Steps/Code to Reproduce
```
from sklearn.neighbors import BallTree    
import numpy as np

np.random.seed(3)
Z = np.random.random((1000, 512)) 
model = BallTree(Z, leaf_size=1, metric='l2')
(
    _,
    idx_array,
    node_data,
    _,
) = model.get_arrays()

root = node_data[0]
lchild = node_data[1]
rchild = node_data[2]

_, _, _, root_radius = root
_, _, _, left_radius = lchild
_, _, _, right_radius = rchild


if left_radius > root_radius:
    print(left_radius, root_radius)
if right_radius > root_radius:
    print(right_radius, root_radius)
```

### Expected Results

Nothing should be printed.

### Actual Results

`6.93039685892859 6.923245275411173`


### Versions

```shell
System:
    python: 3.8.12 (default, Oct 12 2021, 06:23:56)  [Clang 10.0.0 ]
executable: /opt/miniconda3/envs/models/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 21.2.4
   setuptools: 62.1.0
      sklearn: 0.24.2
        numpy: 1.20.3
        scipy: 1.7.1
       Cython: 0.29.24
       pandas: 1.1.5
   matplotlib: 3.4.2
       joblib: 1.0.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
Good spot @rtyasdf 🔍!

As I understand it now, there is no bug in `BallTree` after all.

Is there something we can/should do regarding comments in the code to help people from the future who are trying to understand what is happening?

Labels: module:neighbors, Needs Investigation
⚠️ CI failed on Linux_nogil.pylatest_pip_nogil ⚠️

**CI failed on [Linux_nogil.pylatest_pip_nogil](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=45748&view=logs&j=67fbb25f-e417-50be-be55-3b1e9637fce5)** (Aug 22, 2022)
- test_kernel_gradient[kernel11]

COMMENT:
I can not reproduce this one locally, let's see if it happens consistently in the CI ...

Labels: Build / CI
⚠️ CI failed on Linux_nogil.pylatest_pip_nogil ⚠️

**CI failed on [Linux_nogil.pylatest_pip_nogil](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=45748&view=logs&j=67fbb25f-e417-50be-be55-3b1e9637fce5)** (Aug 22, 2022)
- test_kernel_gradient[kernel11]

COMMENT:
Note that this test also has `@fails_if_pypy` (without any comment unfortunately on the what or the why) so it might trigger some a low-level quirkiness.

For the record, here it fails only with:

```
kernel = 1.41**2 * Matern(length_scale=[0.5, 2], nu=0.5)
```

Labels: Build / CI
⚠️ CI failed on Linux_nogil.pylatest_pip_nogil ⚠️

**CI failed on [Linux_nogil.pylatest_pip_nogil](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=45748&view=logs&j=67fbb25f-e417-50be-be55-3b1e9637fce5)** (Aug 22, 2022)
- test_kernel_gradient[kernel11]

COMMENT:
I will manually retrigger this CI build in Azure to see if it's deterministic or not.

Here is the traceback of the original failure:

```python-traceback
________________________ test_kernel_gradient[kernel11] ________________________
[gw1] linux -- Python 3.9.10 /home/vsts/work/1/s/testvenv/bin/python

kernel = 1.41**2 * Matern(length_scale=[0.5, 2], nu=0.5)

    @fails_if_pypy
    @pytest.mark.parametrize("kernel", kernels)
    def test_kernel_gradient(kernel):
        # Compare analytic and numeric gradient of kernels.
        K, K_gradient = kernel(X, eval_gradient=True)
    
        assert K_gradient.shape[0] == X.shape[0]
        assert K_gradient.shape[1] == X.shape[0]
        assert K_gradient.shape[2] == kernel.theta.shape[0]
    
        def eval_kernel_for_theta(theta):
            kernel_clone = kernel.clone_with_theta(theta)
            K = kernel_clone(X, eval_gradient=False)
            return K
    
        K_gradient_approx = _approx_fprime(kernel.theta, eval_kernel_for_theta, 1e-10)
    
>       assert_almost_equal(K_gradient, K_gradient_approx, 4)
E       AssertionError: 
E       Arrays are not almost equal to 4 decimals
E       
E       Mismatched elements: 8 / 75 (10.7%)
E       Max absolute difference: 71.55003086
E       Max relative difference: 0.67469822
E        x: array([[[2.0000e+000, 1.3853e-309, 9.3709e-310],
        [2.38801479e-01, 4.98647235e-01, 8.87095952e-03],
        [2.00000017e+00, 0.00000000e+00, 0.00000000e+00]]])
eval_kernel_for_theta = <function test_kernel_gradient.<locals>.eval_kernel_for_theta at 0x4c915408d50>
kernel     = 1.41**2 * Matern(length_scale=[0.5, 2], nu=0.5)

```

Labels: Build / CI
⚠️ CI failed on Linux_nogil.pylatest_pip_nogil ⚠️

**CI failed on [Linux_nogil.pylatest_pip_nogil](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=45748&view=logs&j=67fbb25f-e417-50be-be55-3b1e9637fce5)** (Aug 22, 2022)
- test_kernel_gradient[kernel11]

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=45748&view=logs&j=5f71101d-e19c-58af-1684-d0516a86ccf0) on Aug 22, 2022

Labels: Build / CI
⚠️ CI failed on Linux_nogil.pylatest_pip_nogil ⚠️

**CI failed on [Linux_nogil.pylatest_pip_nogil](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=45748&view=logs&j=67fbb25f-e417-50be-be55-3b1e9637fce5)** (Aug 22, 2022)
- test_kernel_gradient[kernel11]

COMMENT:
Ok, so apparently this not deterministic. It could be a race condition of some sort though but given that it's not easy to reproduce, I think we can close for now. If it shows up again we will give it a higher priority to try to investigate the root cause.

Labels: Build / CI
⚠️ CI failed on Linux_nogil.pylatest_pip_nogil ⚠️

**CI failed on [Linux_nogil.pylatest_pip_nogil](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=45748&view=logs&j=67fbb25f-e417-50be-be55-3b1e9637fce5)** (Aug 22, 2022)
- test_kernel_gradient[kernel11]

COMMENT:
I looked a bit more into this one, I can reproduce the failure with pypy. We are using code like:
```py
result = np.divide(numerator, denominator, where=denominator != 0)
```

`result[denominator == 0]` values are undefined when using this. For CPython it seems like the `np.empty` allocated for the return value reuses the temporary array created for `denominator != 0` in the where argument so that `result[denominator == 0]` only contains 0.

A snippet to reproduce a similar behaviour:
```py
import numpy as np

numerator = np.array([1., 1., 1.])
denominator = np.array([0., 1., 1.])
where = denominator != 0

for i in range(5):
    # this temporary variable impacts the values for denominator == 0 ...
    # probably by reusing the memory :-(
    tmp = numerator * (i + denominator)
    print(f"tmp address: {tmp.__array_interface__['data'][0]}")
    del tmp
    divide_result = np.divide(numerator, denominator, where=where)
    print(f"div address: {divide_result.__array_interface__['data'][0]}")
    print(f"{divide_result}")
```

Output with CPython:

```
tmp address: 94334388189392
div address: 94334388189392
[0. 1. 1.]
tmp address: 94334388576960
div address: 94334388576960
[1. 1. 1.]
tmp address: 94334388189392
div address: 94334388189392
[2. 1. 1.]
tmp address: 94334388576960
div address: 94334388576960
[3. 1. 1.]
tmp address: 94334388189392
div address: 94334388189392
[4. 1. 1.]
```

Output with pypy:
```
tmp address: 94000129327360
div address: 94000126685472
[4.64422333e-310 1.00000000e+000 1.00000000e+000]
tmp address: 94000126688656
div address: 94000129178512
[4.64422345e-310 1.00000000e+000 1.00000000e+000]
tmp address: 94000129180688
div address: 94000129179344
[4.64422345e-310 1.00000000e+000 1.00000000e+000]
tmp address: 94000135025200
div address: 94000129181024
[4.64422374e-310 1.00000000e+000 1.00000000e+000]
tmp address: 94000127349824
div address: 94000135025536
[4.64422336e-310 1.00000000e+000 1.00000000e+000]
```

Labels: Build / CI
⚠️ CI failed on Linux_nogil.pylatest_pip_nogil ⚠️

**CI failed on [Linux_nogil.pylatest_pip_nogil](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=45748&view=logs&j=67fbb25f-e417-50be-be55-3b1e9637fce5)** (Aug 22, 2022)
- test_kernel_gradient[kernel11]

COMMENT:
Opened #24245 to fix the problem for `pypy` and probably the intermittent `nogil` one too.

Labels: Build / CI
explained_variance_ratio_ > 1.0

### Describe the bug

`PCA.explained_variance_ratio_` is larger than 1.0

### Steps/Code to Reproduce

```python
import sklearn.decomposition

pca = 64
_pca = sklearn.decomposition.PCA(pca)
vectors = _pca.fit_transform(vectors)
print(f"Explained variance ratio: {_pca.explained_variance_ratio_.sum():.2%}")
```


### Expected Results

I would expect the sum of `explained_variance_ratio_` to be max 1.0 (like explained in the [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html)).

### Actual Results

Explained variance ratio: 111.91%

### Versions

```shell
System:
    python: 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:53)  [GCC 9.4.0]
executable: /opt/conda/envs/morphocluster/bin/python
   machine: Linux-5.4.0-122-generic-x86_64-with-debian-10.6

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: /opt/conda/envs/morphocluster/lib
cblas_libs: mkl_rt, pthread

Python deps:
       pip: 21.3.1
setuptools: 59.8.0
   sklearn: 0.20.4
     numpy: 1.19.5
     scipy: 1.7.3
    Cython: 0.29.32
    pandas: 1.3.5
```

COMMENT:
This seems weird. It would help a lot if you could put together a standalone snippet, in particular with some data that triggers this behaviour, so that someone can try to reproduce. You can have a look at https://stackoverflow.com/help/mcve for more details.

Also I see that you are using sklearn 0.20.4, maybe you can try with a more up-to-date scikit-learn version (latest release is 1.1.2) to see if the issue still persists with the latest version.

Labels: Bug, module:decomposition, Needs Reproducible Code
explained_variance_ratio_ > 1.0

### Describe the bug

`PCA.explained_variance_ratio_` is larger than 1.0

### Steps/Code to Reproduce

```python
import sklearn.decomposition

pca = 64
_pca = sklearn.decomposition.PCA(pca)
vectors = _pca.fit_transform(vectors)
print(f"Explained variance ratio: {_pca.explained_variance_ratio_.sum():.2%}")
```


### Expected Results

I would expect the sum of `explained_variance_ratio_` to be max 1.0 (like explained in the [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html)).

### Actual Results

Explained variance ratio: 111.91%

### Versions

```shell
System:
    python: 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:53)  [GCC 9.4.0]
executable: /opt/conda/envs/morphocluster/bin/python
   machine: Linux-5.4.0-122-generic-x86_64-with-debian-10.6

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: /opt/conda/envs/morphocluster/lib
cblas_libs: mkl_rt, pthread

Python deps:
       pip: 21.3.1
setuptools: 59.8.0
   sklearn: 0.20.4
     numpy: 1.19.5
     scipy: 1.7.3
    Cython: 0.29.32
    pandas: 1.3.5
```

COMMENT:
Closing as no reproducible provided.

Labels: Bug, module:decomposition, Needs Reproducible Code
Upload a Python wheel for sklearn package

As you know a number of people install scikit-learn via `pip install sklearn` (or specify `sklearn` in their dependencies) which works due to the [`sklearn`](https://pypi.org/project/sklearn/#files) package alias. So far we still have no solution to let users know they shouldn't do this https://github.com/scikit-learn/scikit-learn/issues/8215

The problem is that if users try to install a package using the Pyodide distribution in their browser the installation will fail https://github.com/pyodide/pyodide/discussions/2989  because installing  .tar.gz archives from PyPi is not supported (and will not be in the near future due to browser constraints)  One solution is to upload a pure Python python wheel for the `sklearn` package. This will also make the installation a bit faster for all users who happened to use this alias.

The other solution is to try to fix packages that use `sklearn` in their dependencies, but without some centralized mechanism, it's a bit of a hopeless task.

COMMENT:
I don't know what else to say but :sob:.

Maybe some additional motivation to revisit the brownout strategy discussed in #8215.

For good measure I will do a PR on the package that depends on `sklearn` that was found by your pyodide issue.

Labels: Needs Triage
Upload a Python wheel for sklearn package

As you know a number of people install scikit-learn via `pip install sklearn` (or specify `sklearn` in their dependencies) which works due to the [`sklearn`](https://pypi.org/project/sklearn/#files) package alias. So far we still have no solution to let users know they shouldn't do this https://github.com/scikit-learn/scikit-learn/issues/8215

The problem is that if users try to install a package using the Pyodide distribution in their browser the installation will fail https://github.com/pyodide/pyodide/discussions/2989  because installing  .tar.gz archives from PyPi is not supported (and will not be in the near future due to browser constraints)  One solution is to upload a pure Python python wheel for the `sklearn` package. This will also make the installation a bit faster for all users who happened to use this alias.

The other solution is to try to fix packages that use `sklearn` in their dependencies, but without some centralized mechanism, it's a bit of a hopeless task.

COMMENT:
What would the wheel hold? nothing but a dependency on the true scikit-learn package?

I we want to go through the brownout strategy, the wheel might be a problem because there is no way to raise a warning or error at install time, only at import time. But that that would mean that the wheel would hold a fake yet importable `sklearn` top level module but this would collide with the true scikit-learn's `sklearn` top level module so both packages would not be installable at the same time (or behavior would not be predictable).

Labels: Needs Triage
Upload a Python wheel for sklearn package

As you know a number of people install scikit-learn via `pip install sklearn` (or specify `sklearn` in their dependencies) which works due to the [`sklearn`](https://pypi.org/project/sklearn/#files) package alias. So far we still have no solution to let users know they shouldn't do this https://github.com/scikit-learn/scikit-learn/issues/8215

The problem is that if users try to install a package using the Pyodide distribution in their browser the installation will fail https://github.com/pyodide/pyodide/discussions/2989  because installing  .tar.gz archives from PyPi is not supported (and will not be in the near future due to browser constraints)  One solution is to upload a pure Python python wheel for the `sklearn` package. This will also make the installation a bit faster for all users who happened to use this alias.

The other solution is to try to fix packages that use `sklearn` in their dependencies, but without some centralized mechanism, it's a bit of a hopeless task.

COMMENT:
> the wheel might be a problem because there is no way to raise a warning or error at install time, only at import time.

Yes, indeed, I forgot about the brownout idea. And it would indeed not work once a wheel is uploaded, so probably better not do that.

Closing this won't fix it then, thanks for the feedback. At some level, offending packages should fix it. And the worst-case scenario we could hardcode the alias during dependency resolution in Pyodide.

Labels: Needs Triage
Allow X in func or transformer in TransformedTargetRegressor

### Describe the workflow you want to enable

I would like to enable target transformation based on X in data. 

This will come in handy when predicting a quantity that has a natural dependency on a feature. E.g. target value is summed salary of a household and the function is to divide by number of adults in household. This creates a natural scaling of the target but requires features in training data to accomplish. 

In this case the proposed solution would look something like this

```
def func(X, y):
   return y / x[1]

def inverse(X, y):
   return y * X[1]
```

### Describe your proposed solution

Check signature on func and transformer calls to see if it requires X as parameter and if so add it to function calls similar to https://github.com/scikit-learn/scikit-learn/pull/21935

Here is a hacky solution I made myself to get it to work

```
class TargetValueTransformer(TransformedTargetRegressor):
    def __init__(
        self,
        regressor=None,
        *,
        transformer=None,
        func=None,
        inverse_func=None,
        check_inverse=False,
    ):        
        super(TargetValueTransformer, self).__init__(regressor,        
        transformer=transformer,
        func=func,
        inverse_func=inverse_func,
        check_inverse=check_inverse,
)
      
    def _fit_transformer(self, y):
        pass
        
    def fit(self, X, y):
        self.regressor.fit(X, self.func(X, y))
        return self
    
    def predict(self, X):
        return self.inverse_func(X, self.regressor.predict(X))
```
### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Can this be done with custom transformer passed to `transformer` parameter in `TransformedTargetRegressor`?

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
Allow X in func or transformer in TransformedTargetRegressor

### Describe the workflow you want to enable

I would like to enable target transformation based on X in data. 

This will come in handy when predicting a quantity that has a natural dependency on a feature. E.g. target value is summed salary of a household and the function is to divide by number of adults in household. This creates a natural scaling of the target but requires features in training data to accomplish. 

In this case the proposed solution would look something like this

```
def func(X, y):
   return y / x[1]

def inverse(X, y):
   return y * X[1]
```

### Describe your proposed solution

Check signature on func and transformer calls to see if it requires X as parameter and if so add it to function calls similar to https://github.com/scikit-learn/scikit-learn/pull/21935

Here is a hacky solution I made myself to get it to work

```
class TargetValueTransformer(TransformedTargetRegressor):
    def __init__(
        self,
        regressor=None,
        *,
        transformer=None,
        func=None,
        inverse_func=None,
        check_inverse=False,
    ):        
        super(TargetValueTransformer, self).__init__(regressor,        
        transformer=transformer,
        func=func,
        inverse_func=inverse_func,
        check_inverse=check_inverse,
)
      
    def _fit_transformer(self, y):
        pass
        
    def fit(self, X, y):
        self.regressor.fit(X, self.func(X, y))
        return self
    
    def predict(self, X):
        return self.inverse_func(X, self.regressor.predict(X))
```
### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
To complete the answer @GLevV, I think providing a `FunctionTransformer` (https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.FunctionTransformer.html) as a transformer should work for this use case.

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature