Talip7/scikit-learn-issues-multilabel huggingface dataset:
Local testing of global_random_seed is not enough

When adding ``global_random_seed`` to a test, it's not enough to check it locally, i.e. on a single machine. Numerical precision issues can come from various factors like OS, CPU, BLAS, ...

When adding ``global_random_seed``, it's important to test **all** random seeds on **all** CI jobs. To do that, you need to push a commit with ``[all random seeds]`` and the list of tests to check in the commit message:
```
some message [all random seeds]
test_something
test_some_other_thing
```

NOTE: It is **mandatory to pass a short list of test function names** after the `[all random seeds]` commit flag. Running the full scikit-learn test suite for all random seeds at once would take too long.

If this is not done, we merge the PR and then the nightly builds fail every once in a while because the tolerance was barely too small for some seed.

COMMENT:
@jeremiedbb I would like to contribute to this, but am new to CI. To improve this issue, my understanding is to add [all random seeds] in tests in .yml files in each CI. Is that correct?

Labels: Numerical Stability
Local testing of global_random_seed is not enough

When adding ``global_random_seed`` to a test, it's not enough to check it locally, i.e. on a single machine. Numerical precision issues can come from various factors like OS, CPU, BLAS, ...

When adding ``global_random_seed``, it's important to test **all** random seeds on **all** CI jobs. To do that, you need to push a commit with ``[all random seeds]`` and the list of tests to check in the commit message:
```
some message [all random seeds]
test_something
test_some_other_thing
```

NOTE: It is **mandatory to pass a short list of test function names** after the `[all random seeds]` commit flag. Running the full scikit-learn test suite for all random seeds at once would take too long.

If this is not done, we merge the PR and then the nightly builds fail every once in a while because the tolerance was barely too small for some seed.

COMMENT:
Hi @kyrajeep, there's no action needed regarding this issue, it's mostly a reminder for reviewers.

Labels: Numerical Stability
Local testing of global_random_seed is not enough

When adding ``global_random_seed`` to a test, it's not enough to check it locally, i.e. on a single machine. Numerical precision issues can come from various factors like OS, CPU, BLAS, ...

When adding ``global_random_seed``, it's important to test **all** random seeds on **all** CI jobs. To do that, you need to push a commit with ``[all random seeds]`` and the list of tests to check in the commit message:
```
some message [all random seeds]
test_something
test_some_other_thing
```

NOTE: It is **mandatory to pass a short list of test function names** after the `[all random seeds]` commit flag. Running the full scikit-learn test suite for all random seeds at once would take too long.

If this is not done, we merge the PR and then the nightly builds fail every once in a while because the tolerance was barely too small for some seed.

COMMENT:
Thank you for letting me know!

Labels: Numerical Stability
Local testing of global_random_seed is not enough

When adding ``global_random_seed`` to a test, it's not enough to check it locally, i.e. on a single machine. Numerical precision issues can come from various factors like OS, CPU, BLAS, ...

When adding ``global_random_seed``, it's important to test **all** random seeds on **all** CI jobs. To do that, you need to push a commit with ``[all random seeds]`` and the list of tests to check in the commit message:
```
some message [all random seeds]
test_something
test_some_other_thing
```

NOTE: It is **mandatory to pass a short list of test function names** after the `[all random seeds]` commit flag. Running the full scikit-learn test suite for all random seeds at once would take too long.

If this is not done, we merge the PR and then the nightly builds fail every once in a while because the tolerance was barely too small for some seed.

COMMENT:
Hey. It seems the CI tries to run all random seeds on all tests when passing more than one test. Or did I do someting wrong? **Edit: yes I did something wrong: each test name should be on a new line.**

This commit:
[trigger CI [all random seeds] test_diabetes_underfit test_importances](https://github.com/scikit-learn/scikit-learn/pull/32190/commits/a3c5d0bc60aaefa7d7b1aa52322a174f1a7cece6)
(from PR https://github.com/scikit-learn/scikit-learn/pull/32190)

Tiggered this in some [Azure pipeline](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=80065&view=logs&j=f71949a9-f9d9-549e-cf45-2e99c7b412d1&t=d5baef2b-8bda-5a4c-e848-157b5caff279&l=215)
```
+ eval 'python -m pytest --showlocals --durations=20 --junitxml=test-data.xml -o junit_family=legacy -n2 -k '\''test_diabetes_underfit test_importances'\'' --pyargs sklearn'
++ python -m pytest --showlocals --durations=20 --junitxml=test-data.xml -o junit_family=legacy -n2 -k 'test_diabetes_underfit test_importances' --pyargs sklearn
============================= test session starts ==============================
platform linux -- Python 3.10.18, pytest-8.4.2, pluggy-1.6.0
rootdir: /home/vsts/work/tmp_folder
configfile: pyproject.toml
plugins: xdist-3.8.0
created: 2/2 workers
2 workers [364334 items]
```
(hopefully, the collection seems to fail, so it doesn't run 364k tests)

Next, I tried with just one test:
[retry with just one test [all random seeds] test_diabetes_underfit](https://github.com/scikit-learn/scikit-learn/pull/32190/commits/adacbf67c0b6f307685a9c90822514458cf81666)
And it worked as expected (see for instance https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=80067&view=logs&jobId=dde5042c-7464-5d47-9507-31bdd2ee0a3a&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a&t=4bd2dad8-62b3-5bf9-08a5-a9880c530c94)

Labels: Numerical Stability
Local testing of global_random_seed is not enough

When adding ``global_random_seed`` to a test, it's not enough to check it locally, i.e. on a single machine. Numerical precision issues can come from various factors like OS, CPU, BLAS, ...

When adding ``global_random_seed``, it's important to test **all** random seeds on **all** CI jobs. To do that, you need to push a commit with ``[all random seeds]`` and the list of tests to check in the commit message:
```
some message [all random seeds]
test_something
test_some_other_thing
```

NOTE: It is **mandatory to pass a short list of test function names** after the `[all random seeds]` commit flag. Running the full scikit-learn test suite for all random seeds at once would take too long.

If this is not done, we merge the PR and then the nightly builds fail every once in a while because the tolerance was barely too small for some seed.

COMMENT:
@cakedev0 each test name should be on a new line in the commit message, see the example in https://github.com/scikit-learn/scikit-learn/issues/28959#issue-2280443257

Labels: Numerical Stability
Local testing of global_random_seed is not enough

When adding ``global_random_seed`` to a test, it's not enough to check it locally, i.e. on a single machine. Numerical precision issues can come from various factors like OS, CPU, BLAS, ...

When adding ``global_random_seed``, it's important to test **all** random seeds on **all** CI jobs. To do that, you need to push a commit with ``[all random seeds]`` and the list of tests to check in the commit message:
```
some message [all random seeds]
test_something
test_some_other_thing
```

NOTE: It is **mandatory to pass a short list of test function names** after the `[all random seeds]` commit flag. Running the full scikit-learn test suite for all random seeds at once would take too long.

If this is not done, we merge the PR and then the nightly builds fail every once in a while because the tolerance was barely too small for some seed.

COMMENT:
Maybe we could change the script to all for one-liners: whatever is after `[all random seeds]` is interpreted as a test name, either separated by whitespace or new line characters.

Labels: Numerical Stability
⚠️ CI failed on Linux_nogil.pylatest_pip_nogil (last failure: May 06, 2024) ⚠️

**CI failed on [Linux_nogil.pylatest_pip_nogil](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=66324&view=logs&j=67fbb25f-e417-50be-be55-3b1e9637fce5)** (May 06, 2024)
- test_pca_solver_equivalence[81-float32-False-True-tall-arpack]

COMMENT:
ping @ogrisel, probably an issue with tolerances in the test you added a month ago.

Labels: Bug
⚠️ CI failed on Linux_nogil.pylatest_pip_nogil (last failure: May 06, 2024) ⚠️

**CI failed on [Linux_nogil.pylatest_pip_nogil](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=66324&view=logs&j=67fbb25f-e417-50be-be55-3b1e9637fce5)** (May 06, 2024)
- test_pca_solver_equivalence[81-float32-False-True-tall-arpack]

COMMENT:
Indeed, I can reproduce locally when linking numpy/scipy to openblas but not when linking to Apple Accelerate.

I will open a PR and run that tests for all seeds on the CI.

Labels: Bug
⚠️ CI failed on Linux_nogil.pylatest_pip_nogil (last failure: May 06, 2024) ⚠️

**CI failed on [Linux_nogil.pylatest_pip_nogil](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=66324&view=logs&j=67fbb25f-e417-50be-be55-3b1e9637fce5)** (May 06, 2024)
- test_pca_solver_equivalence[81-float32-False-True-tall-arpack]

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=66363&view=logs&j=67fbb25f-e417-50be-be55-3b1e9637fce5) on May 07, 2024

Labels: Bug
Add missing values and categorical features when generating datasets

### Describe the workflow you want to enable

I am often using random datasets (typically with make_classification). However I often find myself having to add more realistic features to the dataset:
- missing data, sometime just to test the pipeline (missing at random would be fine), or sometimes to look for more complex phenomenons (missingnes not at random, possibly depending on the target)
- categorical: categoricals variables often need to be handled specifically. I usually introduce categoricals with binning a continuous value, then transforming to strings. 
It would be nice to have both of those in datasets generation. 

### Describe your proposed solution

Introduce parameters to allow for generation of missing data (proportion of missingness, type of missingness - at random, not at random).
Introduce parameters to allow for generation of categorical features (number of features, type of repartition in categories - even - uneven - pareto.  

### Describe alternatives you've considered, if relevant

I usually handle this by hand.

### Additional context

Could be used to illustrate imputing techniques, encoding techniques.

COMMENT:
@lcrmorin This would be great for testing! I would also suggest adding infinities as possible values, bcs they also break stuff quite often. Also, if randomly generated, making sure to always include at least one NaN and inf value

Labels: New Feature, Moderate
Add missing values and categorical features when generating datasets

### Describe the workflow you want to enable

I am often using random datasets (typically with make_classification). However I often find myself having to add more realistic features to the dataset:
- missing data, sometime just to test the pipeline (missing at random would be fine), or sometimes to look for more complex phenomenons (missingnes not at random, possibly depending on the target)
- categorical: categoricals variables often need to be handled specifically. I usually introduce categoricals with binning a continuous value, then transforming to strings. 
It would be nice to have both of those in datasets generation. 

### Describe your proposed solution

Introduce parameters to allow for generation of missing data (proportion of missingness, type of missingness - at random, not at random).
Introduce parameters to allow for generation of categorical features (number of features, type of repartition in categories - even - uneven - pareto.  

### Describe alternatives you've considered, if relevant

I usually handle this by hand.

### Additional context

Could be used to illustrate imputing techniques, encoding techniques.

COMMENT:
@lcrmorin I suggest adding a noise function or something similar which can generate structured randomness so as to make some sense in data and not pseudo-randomness. Perhaps something like Perlin Noise?

Labels: New Feature, Moderate
Add missing values and categorical features when generating datasets

### Describe the workflow you want to enable

I am often using random datasets (typically with make_classification). However I often find myself having to add more realistic features to the dataset:
- missing data, sometime just to test the pipeline (missing at random would be fine), or sometimes to look for more complex phenomenons (missingnes not at random, possibly depending on the target)
- categorical: categoricals variables often need to be handled specifically. I usually introduce categoricals with binning a continuous value, then transforming to strings. 
It would be nice to have both of those in datasets generation. 

### Describe your proposed solution

Introduce parameters to allow for generation of missing data (proportion of missingness, type of missingness - at random, not at random).
Introduce parameters to allow for generation of categorical features (number of features, type of repartition in categories - even - uneven - pareto.  

### Describe alternatives you've considered, if relevant

I usually handle this by hand.

### Additional context

Could be used to illustrate imputing techniques, encoding techniques.

COMMENT:
Regarding the missing values I recall the following issues/PRs: #6284 / #7084. It seems that the consensus was to have something similar to the `ampute` R package.

I almost a similar discussion for categorical features but I could not find. For sure, it would be handy to have those two parameters even though we could limit the complexity (e.g. only have a single missingness pattern)

Labels: New Feature, Moderate
Add missing values and categorical features when generating datasets

### Describe the workflow you want to enable

I am often using random datasets (typically with make_classification). However I often find myself having to add more realistic features to the dataset:
- missing data, sometime just to test the pipeline (missing at random would be fine), or sometimes to look for more complex phenomenons (missingnes not at random, possibly depending on the target)
- categorical: categoricals variables often need to be handled specifically. I usually introduce categoricals with binning a continuous value, then transforming to strings. 
It would be nice to have both of those in datasets generation. 

### Describe your proposed solution

Introduce parameters to allow for generation of missing data (proportion of missingness, type of missingness - at random, not at random).
Introduce parameters to allow for generation of categorical features (number of features, type of repartition in categories - even - uneven - pareto.  

### Describe alternatives you've considered, if relevant

I usually handle this by hand.

### Additional context

Could be used to illustrate imputing techniques, encoding techniques.

COMMENT:
Regarding the categorical features, we have the following related issue: #12433

Labels: New Feature, Moderate
Add missing values and categorical features when generating datasets

### Describe the workflow you want to enable

I am often using random datasets (typically with make_classification). However I often find myself having to add more realistic features to the dataset:
- missing data, sometime just to test the pipeline (missing at random would be fine), or sometimes to look for more complex phenomenons (missingnes not at random, possibly depending on the target)
- categorical: categoricals variables often need to be handled specifically. I usually introduce categoricals with binning a continuous value, then transforming to strings. 
It would be nice to have both of those in datasets generation. 

### Describe your proposed solution

Introduce parameters to allow for generation of missing data (proportion of missingness, type of missingness - at random, not at random).
Introduce parameters to allow for generation of categorical features (number of features, type of repartition in categories - even - uneven - pareto.  

### Describe alternatives you've considered, if relevant

I usually handle this by hand.

### Additional context

Could be used to illustrate imputing techniques, encoding techniques.

COMMENT:
I will take a look on it and try to implement this feature in my side by exploring different possibilities to incorporate NaN and inf values.

Labels: New Feature, Moderate
Add missing values and categorical features when generating datasets

### Describe the workflow you want to enable

I am often using random datasets (typically with make_classification). However I often find myself having to add more realistic features to the dataset:
- missing data, sometime just to test the pipeline (missing at random would be fine), or sometimes to look for more complex phenomenons (missingnes not at random, possibly depending on the target)
- categorical: categoricals variables often need to be handled specifically. I usually introduce categoricals with binning a continuous value, then transforming to strings. 
It would be nice to have both of those in datasets generation. 

### Describe your proposed solution

Introduce parameters to allow for generation of missing data (proportion of missingness, type of missingness - at random, not at random).
Introduce parameters to allow for generation of categorical features (number of features, type of repartition in categories - even - uneven - pareto.  

### Describe alternatives you've considered, if relevant

I usually handle this by hand.

### Additional context

Could be used to illustrate imputing techniques, encoding techniques.

COMMENT:
I will try to work on it!

Labels: New Feature, Moderate
Yeo-Johnson inverse_transform fails silently on extreme skew data

### Describe the bug

The Yeo-Johnson is not a surjective transformation for negative lambdas. Therefore, the inverse transformation returns `np.nan` when inverse transforming values outside the range of the transform. This failure is silent, so it took me quite a while of debugging to understand this behavior.

The problematic lines are

https://github.com/scikit-learn/scikit-learn/blob/8721245511de2f225ff5f9aa5f5fadce663cd4a3/sklearn/preprocessing/_data.py#L3390

and 

https://github.com/scikit-learn/scikit-learn/blob/8721245511de2f225ff5f9aa5f5fadce663cd4a3/sklearn/preprocessing/_data.py#L3386

in which we might compute `np.power(something_negative, not_integral_value)`, which of course returns `np.nan` as per https://numpy.org/doc/stable/reference/generated/numpy.power.html

### Steps/Code to Reproduce

To reproduce for positive values (there is a similar problem for negative values):

```python
import numpy as np
import sklearn.preprocessing
trans = sklearn.preprocessing.PowerTransformer(method='yeo-johnson')
x = np.array([1,1,1e10]).reshape(-1, 1) # extreme skew
trans.fit(x)
lmbda = trans.lambdas_[0] 
print(lmbda)
assert lmbda < 0 # == -0.096 negative value

# any value `psi` for which lambda*psi+1 <= 0 will result in nan due to lacking support, since the forwards transformation 
# is not surjective on negative lambdas. In this specific case, 10*-0.096 < 1
psi = np.array([10]).reshape(-1, 1)
x = trans.inverse_transform(psi).item()
print(x)
assert np.isnan(x)
```

### Expected Results

 The code should either:

1) validate its inputs and raise an exception
2) validate its inputs and raise a warning
3) fail silently, but have it documented behavior

### Actual Results

It just prints

```
-0.0962322261004418
nan
```

### Versions

```shell
System:
    python: 3.11.3 (main, Jan 18 2024, 19:07:12) [Clang 18.0.0 (https://github.com/llvm/llvm-project 75501f53624de92aafce2f1da698
executable: /home/pyodide/this.program
   machine: Emscripten-3.1.46-wasm32-32bit

Python dependencies:
      sklearn: 1.3.1
          pip: None
   setuptools: None
        numpy: 1.26.1
        scipy: 1.11.2
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: False
```

COMMENT:
Thanks for the report and the analysis. I confirm I can reproduce on `main` with the provided reproducer.

I think that calling `inverse_transform` with negative lambda values should at least raise a warning.

Not sure if it would be helpful to raise such a warning at `fit` time though. Maybe some users only care about the `transform` without `inverse_transform` and raising a warning would be annoying for those users.

Labels: Bug
Yeo-Johnson inverse_transform fails silently on extreme skew data

### Describe the bug

The Yeo-Johnson is not a surjective transformation for negative lambdas. Therefore, the inverse transformation returns `np.nan` when inverse transforming values outside the range of the transform. This failure is silent, so it took me quite a while of debugging to understand this behavior.

The problematic lines are

https://github.com/scikit-learn/scikit-learn/blob/8721245511de2f225ff5f9aa5f5fadce663cd4a3/sklearn/preprocessing/_data.py#L3390

and 

https://github.com/scikit-learn/scikit-learn/blob/8721245511de2f225ff5f9aa5f5fadce663cd4a3/sklearn/preprocessing/_data.py#L3386

in which we might compute `np.power(something_negative, not_integral_value)`, which of course returns `np.nan` as per https://numpy.org/doc/stable/reference/generated/numpy.power.html

### Steps/Code to Reproduce

To reproduce for positive values (there is a similar problem for negative values):

```python
import numpy as np
import sklearn.preprocessing
trans = sklearn.preprocessing.PowerTransformer(method='yeo-johnson')
x = np.array([1,1,1e10]).reshape(-1, 1) # extreme skew
trans.fit(x)
lmbda = trans.lambdas_[0] 
print(lmbda)
assert lmbda < 0 # == -0.096 negative value

# any value `psi` for which lambda*psi+1 <= 0 will result in nan due to lacking support, since the forwards transformation 
# is not surjective on negative lambdas. In this specific case, 10*-0.096 < 1
psi = np.array([10]).reshape(-1, 1)
x = trans.inverse_transform(psi).item()
print(x)
assert np.isnan(x)
```

### Expected Results

 The code should either:

1) validate its inputs and raise an exception
2) validate its inputs and raise a warning
3) fail silently, but have it documented behavior

### Actual Results

It just prints

```
-0.0962322261004418
nan
```

### Versions

```shell
System:
    python: 3.11.3 (main, Jan 18 2024, 19:07:12) [Clang 18.0.0 (https://github.com/llvm/llvm-project 75501f53624de92aafce2f1da698
executable: /home/pyodide/this.program
   machine: Emscripten-3.1.46-wasm32-32bit

Python dependencies:
      sklearn: 1.3.1
          pip: None
   setuptools: None
        numpy: 1.26.1
        scipy: 1.11.2
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: False
```

COMMENT:
I think that a warning in `inverse_transform`, behind a check that diagnose whether this error case is hit or not. I suppose it looks something like 

    tmp1 = -(2 - lmbda) * x[~pos] + 1
    tmp2 = x[pos] * lmbda + 1
    lambda < 0 and ( np.any( tmp1< 0) or np.any(tmp2 < 0))

Short circuiting means the `np.any` calls are often not computed at all, so it should be really cheap. `tmp1` and `tmp2` needs to be computed for the `inverse_transform` so that is no extra work.

EDIT: hopefully with better variable names than `tmp1` and `tmp2`...

Labels: Bug