Talip7/scikit-learn-issues-multilabel huggingface dataset:
TransformedTargetRegressor with Early Stopping: transforming user-supplied validation sets in fit_params, too

### Describe the workflow you want to enable

Many advanced regressors (CatBoost, XGBoost, LightGBM to name a few) support providing custom early stopping dataset(s) to their **fit** methods. Not all of them have scalar eval_fraction parameters, like GradientBoostingRegressor; even if they would, sometimes there is really a need for custom validation splitting (time-series with daily grouping, for instance).

Currently, TransformedTargetRegressor only transforms the **y** argument, which makes training with early stopping of above-mentioned regressors impossible: they do not converge, as I have just experienced. Or even if they converge, that can theoretically happen, validating on data in different scale can hardly find the same early stopping sweetspots.

### Describe your proposed solution

It would be convenient to support an extra **es_fit_param_name** argument to the init method of TransformedTargetRegressor.

When such param is present, at the fitting time,  fit_params are inspected for it.

```
if it's not in fit_params:
    error is raised
else:
   if it's an iterable of lists/tuples:
       self.transformer_.transform is additionally applied to the second element of each top-level iterable (fit_params will need modifying)
   elif it's an iterable:
       self.transformer_.transform is additionally applied to the second element of iterable (fit_params will need modifying)
   else:
       error is raised
```

Actually, for some most popular libraries relevant es_fit_param_name are known and can be used as defaults, if es_fit_param_name is not provided by the user but params nevertheless exist (can be turned off with an extra **no_es_autosuggest** flag).  But maybe going this deep is too much.

### Describe alternatives you've considered, if relevant

Applying transformer or func/inverse_func to the fit_params manually in every project using custom early stopping splitting.

### Additional context

A main drawback that I am seeing is the necessity to modify the user-provided fit_params, which is not a usual practice (but maybe we can copy fit_params internally first). On the other hand, it should allow thousands of data scientists to keep using both target transformations and custom early stopping in an elegant fashion. Ready to contribute if approved. Any thoughts? Is it worth it?

COMMENT:
I have just ensured my approach works:

![image](https://github.com/scikit-learn/scikit-learn/assets/16359856/a1350023-cc1e-4fea-9402-e2b6e52d2ef8)
![image](https://github.com/scikit-learn/scikit-learn/assets/16359856/eb4b0158-f982-41d7-864e-a0cd39db5f9a)

Labels: New Feature
TransformedTargetRegressor with Early Stopping: transforming user-supplied validation sets in fit_params, too

### Describe the workflow you want to enable

Many advanced regressors (CatBoost, XGBoost, LightGBM to name a few) support providing custom early stopping dataset(s) to their **fit** methods. Not all of them have scalar eval_fraction parameters, like GradientBoostingRegressor; even if they would, sometimes there is really a need for custom validation splitting (time-series with daily grouping, for instance).

Currently, TransformedTargetRegressor only transforms the **y** argument, which makes training with early stopping of above-mentioned regressors impossible: they do not converge, as I have just experienced. Or even if they converge, that can theoretically happen, validating on data in different scale can hardly find the same early stopping sweetspots.

### Describe your proposed solution

It would be convenient to support an extra **es_fit_param_name** argument to the init method of TransformedTargetRegressor.

When such param is present, at the fitting time,  fit_params are inspected for it.

```
if it's not in fit_params:
    error is raised
else:
   if it's an iterable of lists/tuples:
       self.transformer_.transform is additionally applied to the second element of each top-level iterable (fit_params will need modifying)
   elif it's an iterable:
       self.transformer_.transform is additionally applied to the second element of iterable (fit_params will need modifying)
   else:
       error is raised
```

Actually, for some most popular libraries relevant es_fit_param_name are known and can be used as defaults, if es_fit_param_name is not provided by the user but params nevertheless exist (can be turned off with an extra **no_es_autosuggest** flag).  But maybe going this deep is too much.

### Describe alternatives you've considered, if relevant

Applying transformer or func/inverse_func to the fit_params manually in every project using custom early stopping splitting.

### Additional context

A main drawback that I am seeing is the necessity to modify the user-provided fit_params, which is not a usual practice (but maybe we can copy fit_params internally first). On the other hand, it should allow thousands of data scientists to keep using both target transformations and custom early stopping in an elegant fashion. Ready to contribute if approved. Any thoughts? Is it worth it?

COMMENT:
I think most relevant discussions to this one are on https://github.com/scikit-learn/scikit-learn/issues/18748 and https://github.com/scikit-learn/scikit-learn/pull/27124

Labels: New Feature
TransformedTargetRegressor with Early Stopping: transforming user-supplied validation sets in fit_params, too

### Describe the workflow you want to enable

Many advanced regressors (CatBoost, XGBoost, LightGBM to name a few) support providing custom early stopping dataset(s) to their **fit** methods. Not all of them have scalar eval_fraction parameters, like GradientBoostingRegressor; even if they would, sometimes there is really a need for custom validation splitting (time-series with daily grouping, for instance).

Currently, TransformedTargetRegressor only transforms the **y** argument, which makes training with early stopping of above-mentioned regressors impossible: they do not converge, as I have just experienced. Or even if they converge, that can theoretically happen, validating on data in different scale can hardly find the same early stopping sweetspots.

### Describe your proposed solution

It would be convenient to support an extra **es_fit_param_name** argument to the init method of TransformedTargetRegressor.

When such param is present, at the fitting time,  fit_params are inspected for it.

```
if it's not in fit_params:
    error is raised
else:
   if it's an iterable of lists/tuples:
       self.transformer_.transform is additionally applied to the second element of each top-level iterable (fit_params will need modifying)
   elif it's an iterable:
       self.transformer_.transform is additionally applied to the second element of iterable (fit_params will need modifying)
   else:
       error is raised
```

Actually, for some most popular libraries relevant es_fit_param_name are known and can be used as defaults, if es_fit_param_name is not provided by the user but params nevertheless exist (can be turned off with an extra **no_es_autosuggest** flag).  But maybe going this deep is too much.

### Describe alternatives you've considered, if relevant

Applying transformer or func/inverse_func to the fit_params manually in every project using custom early stopping splitting.

### Additional context

A main drawback that I am seeing is the necessity to modify the user-provided fit_params, which is not a usual practice (but maybe we can copy fit_params internally first). On the other hand, it should allow thousands of data scientists to keep using both target transformations and custom early stopping in an elegant fashion. Ready to contribute if approved. Any thoughts? Is it worth it?

COMMENT:
> #27124

Oh nice, when #27124 is implemented, this feature request will also be valid for HistGradientBoostingRegressor.

Labels: New Feature
Enable parallel sklearn.feature_selection.mutual_info_regression

### Describe the workflow you want to enable

I can raise the PR if someone is willing to review and potentially merge.
```
from sklearn.feature_selection import mutual_info_regression
mutual_info = mutual_info_regression(X, y, n_jobs = -1)
```

### Describe your proposed solution

In: https://github.com/scikit-learn/scikit-learn/blob/0ab36990c0a7d02052a3de7b726aa425ee14950f/sklearn/feature_selection/_mutual_info.py#L304
Change
```
    mi = [
        _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)
        for x, discrete_feature in zip(_iterate_columns(X), discrete_mask)
    ]
```
to
```
from joblib import Parallel, delayed
def process_column(x, discrete_feature):
    return _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)

mi = Parallel(n_jobs=n_jobs)(delayed(process_column)(x, discrete_feature) 
                                       for x, discrete_feature in zip(_iterate_columns(X), discrete_mask))
```

### Describe alternatives you've considered, if relevant

None

### Additional context

Enable the user to choose multicore or single core.

COMMENT:
Could you provide a benchmark to show that the parallelization would be beneficial.

Labels: Needs Decision, Needs Benchmarks
Enable parallel sklearn.feature_selection.mutual_info_regression

### Describe the workflow you want to enable

I can raise the PR if someone is willing to review and potentially merge.
```
from sklearn.feature_selection import mutual_info_regression
mutual_info = mutual_info_regression(X, y, n_jobs = -1)
```

### Describe your proposed solution

In: https://github.com/scikit-learn/scikit-learn/blob/0ab36990c0a7d02052a3de7b726aa425ee14950f/sklearn/feature_selection/_mutual_info.py#L304
Change
```
    mi = [
        _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)
        for x, discrete_feature in zip(_iterate_columns(X), discrete_mask)
    ]
```
to
```
from joblib import Parallel, delayed
def process_column(x, discrete_feature):
    return _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)

mi = Parallel(n_jobs=n_jobs)(delayed(process_column)(x, discrete_feature) 
                                       for x, discrete_feature in zip(_iterate_columns(X), discrete_mask))
```

### Describe alternatives you've considered, if relevant

None

### Additional context

Enable the user to choose multicore or single core.

COMMENT:
Indeed, it shows a net benefit. We could then expose an `n_jobs` parameter and use `joblib`.

@FlorinAndrei @netomenoci Would you like to make a pull request?

Labels: Needs Decision, Needs Benchmarks
Enable parallel sklearn.feature_selection.mutual_info_regression

### Describe the workflow you want to enable

I can raise the PR if someone is willing to review and potentially merge.
```
from sklearn.feature_selection import mutual_info_regression
mutual_info = mutual_info_regression(X, y, n_jobs = -1)
```

### Describe your proposed solution

In: https://github.com/scikit-learn/scikit-learn/blob/0ab36990c0a7d02052a3de7b726aa425ee14950f/sklearn/feature_selection/_mutual_info.py#L304
Change
```
    mi = [
        _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)
        for x, discrete_feature in zip(_iterate_columns(X), discrete_mask)
    ]
```
to
```
from joblib import Parallel, delayed
def process_column(x, discrete_feature):
    return _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)

mi = Parallel(n_jobs=n_jobs)(delayed(process_column)(x, discrete_feature) 
                                       for x, discrete_feature in zip(_iterate_columns(X), discrete_mask))
```

### Describe alternatives you've considered, if relevant

None

### Additional context

Enable the user to choose multicore or single core.

COMMENT:
Hey @FlorinAndrei, @glemaitre  here's the pr. https://github.com/scikit-learn/scikit-learn/pull/28085 

I have used

`from ..utils.parallel import Parallel, delayed
`

instead of 
`from joblib import Parallel, delayed`

since it seems that this is the default in sklearn? (The other functions seem to be using it, as it comes with some sklearn checks)

Labels: Needs Decision, Needs Benchmarks
Enable parallel sklearn.feature_selection.mutual_info_regression

### Describe the workflow you want to enable

I can raise the PR if someone is willing to review and potentially merge.
```
from sklearn.feature_selection import mutual_info_regression
mutual_info = mutual_info_regression(X, y, n_jobs = -1)
```

### Describe your proposed solution

In: https://github.com/scikit-learn/scikit-learn/blob/0ab36990c0a7d02052a3de7b726aa425ee14950f/sklearn/feature_selection/_mutual_info.py#L304
Change
```
    mi = [
        _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)
        for x, discrete_feature in zip(_iterate_columns(X), discrete_mask)
    ]
```
to
```
from joblib import Parallel, delayed
def process_column(x, discrete_feature):
    return _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)

mi = Parallel(n_jobs=n_jobs)(delayed(process_column)(x, discrete_feature) 
                                       for x, discrete_feature in zip(_iterate_columns(X), discrete_mask))
```

### Describe alternatives you've considered, if relevant

None

### Additional context

Enable the user to choose multicore or single core.

COMMENT:
merged https://github.com/scikit-learn/scikit-learn/pull/28085

Labels: Needs Decision, Needs Benchmarks
Enable parallel sklearn.feature_selection.mutual_info_regression

### Describe the workflow you want to enable

I can raise the PR if someone is willing to review and potentially merge.
```
from sklearn.feature_selection import mutual_info_regression
mutual_info = mutual_info_regression(X, y, n_jobs = -1)
```

### Describe your proposed solution

In: https://github.com/scikit-learn/scikit-learn/blob/0ab36990c0a7d02052a3de7b726aa425ee14950f/sklearn/feature_selection/_mutual_info.py#L304
Change
```
    mi = [
        _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)
        for x, discrete_feature in zip(_iterate_columns(X), discrete_mask)
    ]
```
to
```
from joblib import Parallel, delayed
def process_column(x, discrete_feature):
    return _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)

mi = Parallel(n_jobs=n_jobs)(delayed(process_column)(x, discrete_feature) 
                                       for x, discrete_feature in zip(_iterate_columns(X), discrete_mask))
```

### Describe alternatives you've considered, if relevant

None

### Additional context

Enable the user to choose multicore or single core.

COMMENT:
@glemaitre I guess it didn't make the cut into 1.4.0. Will it be included in 1.4.1?

Labels: Needs Decision, Needs Benchmarks
Enable parallel sklearn.feature_selection.mutual_info_regression

### Describe the workflow you want to enable

I can raise the PR if someone is willing to review and potentially merge.
```
from sklearn.feature_selection import mutual_info_regression
mutual_info = mutual_info_regression(X, y, n_jobs = -1)
```

### Describe your proposed solution

In: https://github.com/scikit-learn/scikit-learn/blob/0ab36990c0a7d02052a3de7b726aa425ee14950f/sklearn/feature_selection/_mutual_info.py#L304
Change
```
    mi = [
        _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)
        for x, discrete_feature in zip(_iterate_columns(X), discrete_mask)
    ]
```
to
```
from joblib import Parallel, delayed
def process_column(x, discrete_feature):
    return _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)

mi = Parallel(n_jobs=n_jobs)(delayed(process_column)(x, discrete_feature) 
                                       for x, discrete_feature in zip(_iterate_columns(X), discrete_mask))
```

### Describe alternatives you've considered, if relevant

None

### Additional context

Enable the user to choose multicore or single core.

COMMENT:
It will be 1.5 because we don't include any API changes in bug fixes release. So it should be in scikit-learn in May.

Labels: Needs Decision, Needs Benchmarks
AgglomerativeClustering not using cache

### Describe the bug

Hi,
When trying to run `AgglomerativeClustering` on a precomputed distance matrix, cache is not used. 
Tried both `memory='path/to/somewhere'` and `memory=joblib.Memory('path/to/somewhere')`
Cache directory is created, and filled with some 20kb of code, but not the tree/ any large files. 
(using 1.3.0)
Thanks for helping!
-R

I'm aware of #18859 but this is not related.

### Steps/Code to Reproduce

```python
import os
import numpy as np
from joblib import Memory
from sklearn.cluster import AgglomerativeClustering
import sklearn

print(sklearn.__version__)

mat=np.random.rand(10000,10000)
p = '/somepath/'
c = AgglomerativeClustering(
    None,
    metric='precomputed',
    distance_threshold=0.1,
    linkage='average',
    memory=p
)
a = c.fit_predict(mat)
# to show no memory is used
nbytes = sum(
    d.stat().st_size
    for d in os.scandir(p)
    if d.is_file()
)
print(nbytes)
# now with joblib.Memory(path) - this one prints nice things in
# the beginning of fit_predict but doesn't use cache. 
c = AgglomerativeClustering(
    None,
    metric='precomputed',
    distance_threshold=0.1,
    linkage='average',
    memory=Memory(p)
)
a = c.fit_predict(mat)
nbytes = sum(
    d.stat().st_size
    for d in os.scandir(p)
    if d.is_file()
)
print(nbytes)
```


### Expected Results

Any value greater than 130bytes, a value that would indicate real caching is done. 

### Actual Results

0
0

### Versions

```shell
1.3.0
1.3.2
```

COMMENT:
You are not inspecting the right folder. The files will be stored in a specific directory:

```
/somewhat/joblib/sklearn/cluster/_agglomerative/_average_linkage/52ca51783c6ed6db87670be4c3e23f6c
```

where the hash will change depending of the session (if I recall).

Locally, get 2 files for this processing:

```shell
> ls -l
total 480
-rw-r--r--  1 glemaitre  wheel     800 Nov 16 11:31 metadata.json
-rw-r--r--  1 glemaitre  wheel  240315 Nov 16 11:31 output.pkl
```

So caching is working.

Labels: Bug, Needs Triage
Floating Point Precision in RandomForestClassifier is only `1e-7` despite `numpy.float64`

### Describe the bug

When training the classifier, the precision is only about `1e-7`.

This is true even when the `train_X` is of data type `numpy.float64`.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.tree import plot_tree, DecisionTreeClassifier

X = np.array([[0.0], [1e-7]], dtype=np.float64)
y = [0, 1]
model = DecisionTreeClassifier().fit(X, y)
plot_tree(model)
```

### Expected Results

```
[Text(0.5, 0.75, 'x[0] <= 0.0\ngini = 0.5\nsamples = 2\nvalue = [1, 1]'),
Text(0.25, 0.25, 'gini = 0.0\nsamples = 1\nvalue = [1, 0]'),
Text(0.75, 0.25, 'gini = 0.0\nsamples = 1\nvalue = [0, 1]')]
```
![image](https://github.com/scikit-learn/scikit-learn/assets/21100851/2ccd1af7-7177-4157-a864-2936054b3e34)

One would **expect the root node splits** into two children.

This output can be obtained by changing `X` into: `X = np.array([[0.0], [2e-7]], dtype=np.float64)`

### Actual Results

```
[Text(0.5, 0.5, 'gini = 0.5\nsamples = 2\nvalue = [1, 1]')]
```
![image](https://github.com/scikit-learn/scikit-learn/assets/21100851/6ca34d65-96bc-4f33-865b-898d4dbae194)

The decision tree does not split, as if the two rows in `X` are indistinguishable.

### Versions

```shell
System:
    python: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
executable: /usr/bin/python3
   machine: Linux-5.15.120+-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.2
   setuptools: 67.7.2
        numpy: 1.23.5
        scipy: 1.11.3
       Cython: 3.0.5
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/numpy.libs/libopenblas64_p-r0-742d56dc.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 2
         prefix: libgomp
       filepath: /usr/local/lib/python3.10/dist-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Haswell
```

COMMENT:
The internal of the trees are working with `np.float32` so the loss of precision.

It is documented here: https://scikit-learn.org/stable/modules/tree.html#tips-on-practical-use

Labels: Bug, Needs Triage
HuberRegressor failed with ABNORMAL_TERMINATION_IN_LNSRCH on simple dataset

### Describe the bug

sklearn linear_model HuberRegressor fails with a very simple dataset. 

The code below fails with ValueError: HuberRegressor convergence failed: l-BFGS-b solver terminated with ABNORMAL_TERMINATION_IN_LNSRCH



### Steps/Code to Reproduce

```python
import numpy as np
import sklearn
from sklearn.linear_model import HuberRegressor

X = np.array([10, 11, 28]).reshape(-1, 1)
y = np.log(np.array([5000., 5000., 5000.]))
sample_weights = np.array([2., 2., 2.])
h = HuberRegressor().fit(X, y, sample_weights)
```

### Expected Results

no ValueError thrown

### Actual Results

```pytb
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 10
      8 y = np.log(np.array([5000., 5000., 5000.]))
      9 sample_weights = np.array([2., 2., 2.])
---> 10 h = HuberRegressor().fit(X, y, sample_weights)

File ~\Anaconda3\lib\site-packages\sklearn\base.py:1151, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1144     estimator._validate_params()
   1146 with config_context(
   1147     skip_parameter_validation=(
   1148         prefer_skip_nested_validation or global_skip_validation
   1149     )
   1150 ):
-> 1151     return fit_method(estimator, *args, **kwargs)

File ~\Anaconda3\lib\site-packages\sklearn\linear_model\_huber.py:338, in HuberRegressor.fit(self, X, y, sample_weight)
    335 parameters = opt_res.x
    337 if opt_res.status == 2:
--> 338     raise ValueError(
    339         "HuberRegressor convergence failed: l-BFGS-b solver terminated with %s"
    340         % opt_res.message
    341     )
    342 self.n_iter_ = _check_optimize_result("lbfgs", opt_res, self.max_iter)
    343 self.scale_ = parameters[-1]

ValueError: HuberRegressor convergence failed: l-BFGS-b solver terminated with ABNORMAL_TERMINATION_IN_LNSRCH
```

### Versions

```shell
System:
    python: 3.9.18 (main, Sep 11 2023, 13:30:38) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\[username]\Anaconda3\python.exe
   machine: Windows-10-10.0.19044-SP0

Python dependencies:
      sklearn: 1.3.0
          pip: 23.3
   setuptools: 68.0.0
        numpy: 1.24.3
        scipy: 1.11.3
       Cython: None
       pandas: 2.1.1
   matplotlib: 3.8.0
       joblib: 1.2.0
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       filepath: C:\Users\[username]\Anaconda3\Library\bin\mkl_rt.2.dll
         prefix: mkl_rt
       user_api: blas
   internal_api: mkl
        version: 2023.1-Product
    num_threads: 4
threading_layer: intel

       filepath: C:\Users\[username]\Anaconda3\vcomp140.dll
         prefix: vcomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 8
```

COMMENT:
Yes, it should just warn with a ConvergenceWarning.
Maybe the params of lbfgs can be improved similar to logreg/glm.

Labels: help wanted