Talip7/scikit-learn-issues-multilabel huggingface dataset:
PCA hangs occasionally when applied to data that fits in memory

### Describe the bug

When trying to `fit` PCA with a large enough array that fits in memory (approximately `300 000 x 750`) with an aim to make about 3 times reduction, unstable results are obtained: in most cases the *fit succeeds*, but occasionaly it *hangs "forever"*.

Random seed is fixed, but unfortunalely it does not help in this case.
The `svd_solver` option is left at the default. Looks like explicitly specifying `svd_solver='arpack'` does not cause this issue.

### Steps/Code to Reproduce

```python
import numpy as np
import random

np.random.seed(42)
random.seed(42)

from sklearn.decomposition import PCA


if __name__ == '__main__':
    embeddings = np.random.RandomState(42).normal(size=(328039, 768))
    embeddings = embeddings.astype(np.float32)

    for i in range(20):
        print('Iteration', i)
        pca = PCA(n_components=260, random_state=42)
        pca = pca.fit(embeddings)

```

### Expected Results

Stable results: consistently successful fits over a finite time.

### Actual Results

Sometimes the process hangs, in most cases in the first 10 iterations. I haven't found any consistency pattern at what time it fails.

### Versions

```shell
System:
   python: 3.9.7 | packaged by conda-forge 
   machine: Windows-10-10.0.19043-SP0
   RAM: 32Gb

Python dependencies:
          pip: 21.3.1
   setuptools: 60.8.1
      sklearn: 1.0.1
        numpy: 1.20.1
        scipy: 1.7.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
It sounds like a bug in the multithreading code of OpenBLAS or MKL. The PCA estimator (with its `svd_solver="randomized"` setting) itself does not use multithreading itself directly. It only does so via OpenBLAS or MKL.

Could some that can reproduce the deadlock use `faulthandler.dump_traceback_later` to dump the state of the threads when the deadlock happens?


```python
import numpy as np
import random
import faulthandler

np.random.seed(42)
random.seed(42)

from sklearn.decomposition import PCA


if __name__ == '__main__':
    embeddings = np.random.RandomState(42).normal(size=(328039, 768))
    embeddings = embeddings.astype(np.float32)

    for i in range(20):
        print('Iteration', i)
        faulthandler.dump_traceback_later(10, exit=True)
        pca = PCA(svd_solver="randomized", n_components=260, random_state=42)
        pca = pca.fit(embeddings)
        faulthandler.cancel_dump_traceback_later()
```

Labels: Bug, module:decomposition, Needs Investigation
PCA hangs occasionally when applied to data that fits in memory

### Describe the bug

When trying to `fit` PCA with a large enough array that fits in memory (approximately `300 000 x 750`) with an aim to make about 3 times reduction, unstable results are obtained: in most cases the *fit succeeds*, but occasionaly it *hangs "forever"*.

Random seed is fixed, but unfortunalely it does not help in this case.
The `svd_solver` option is left at the default. Looks like explicitly specifying `svd_solver='arpack'` does not cause this issue.

### Steps/Code to Reproduce

```python
import numpy as np
import random

np.random.seed(42)
random.seed(42)

from sklearn.decomposition import PCA


if __name__ == '__main__':
    embeddings = np.random.RandomState(42).normal(size=(328039, 768))
    embeddings = embeddings.astype(np.float32)

    for i in range(20):
        print('Iteration', i)
        pca = PCA(n_components=260, random_state=42)
        pca = pca.fit(embeddings)

```

### Expected Results

Stable results: consistently successful fits over a finite time.

### Actual Results

Sometimes the process hangs, in most cases in the first 10 iterations. I haven't found any consistency pattern at what time it fails.

### Versions

```shell
System:
   python: 3.9.7 | packaged by conda-forge 
   machine: Windows-10-10.0.19043-SP0
   RAM: 32Gb

Python dependencies:
          pip: 21.3.1
   setuptools: 60.8.1
      sklearn: 1.0.1
        numpy: 1.20.1
        scipy: 1.7.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
Note that for this kind of data set sizes, the new solver being developed in #27491 should work much faster and use fewer memory.

Labels: Bug, module:decomposition, Needs Investigation
PCA hangs occasionally when applied to data that fits in memory

### Describe the bug

When trying to `fit` PCA with a large enough array that fits in memory (approximately `300 000 x 750`) with an aim to make about 3 times reduction, unstable results are obtained: in most cases the *fit succeeds*, but occasionaly it *hangs "forever"*.

Random seed is fixed, but unfortunalely it does not help in this case.
The `svd_solver` option is left at the default. Looks like explicitly specifying `svd_solver='arpack'` does not cause this issue.

### Steps/Code to Reproduce

```python
import numpy as np
import random

np.random.seed(42)
random.seed(42)

from sklearn.decomposition import PCA


if __name__ == '__main__':
    embeddings = np.random.RandomState(42).normal(size=(328039, 768))
    embeddings = embeddings.astype(np.float32)

    for i in range(20):
        print('Iteration', i)
        pca = PCA(n_components=260, random_state=42)
        pca = pca.fit(embeddings)

```

### Expected Results

Stable results: consistently successful fits over a finite time.

### Actual Results

Sometimes the process hangs, in most cases in the first 10 iterations. I haven't found any consistency pattern at what time it fails.

### Versions

```shell
System:
   python: 3.9.7 | packaged by conda-forge 
   machine: Windows-10-10.0.19043-SP0
   RAM: 32Gb

Python dependencies:
          pip: 21.3.1
   setuptools: 60.8.1
      sklearn: 1.0.1
        numpy: 1.20.1
        scipy: 1.7.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
@ogrisel Is this issue resolved given the advent of https://github.com/scikit-learn/scikit-learn/pull/27491?

Labels: Bug, module:decomposition, Needs Investigation
PCA hangs occasionally when applied to data that fits in memory

### Describe the bug

When trying to `fit` PCA with a large enough array that fits in memory (approximately `300 000 x 750`) with an aim to make about 3 times reduction, unstable results are obtained: in most cases the *fit succeeds*, but occasionaly it *hangs "forever"*.

Random seed is fixed, but unfortunalely it does not help in this case.
The `svd_solver` option is left at the default. Looks like explicitly specifying `svd_solver='arpack'` does not cause this issue.

### Steps/Code to Reproduce

```python
import numpy as np
import random

np.random.seed(42)
random.seed(42)

from sklearn.decomposition import PCA


if __name__ == '__main__':
    embeddings = np.random.RandomState(42).normal(size=(328039, 768))
    embeddings = embeddings.astype(np.float32)

    for i in range(20):
        print('Iteration', i)
        pca = PCA(n_components=260, random_state=42)
        pca = pca.fit(embeddings)

```

### Expected Results

Stable results: consistently successful fits over a finite time.

### Actual Results

Sometimes the process hangs, in most cases in the first 10 iterations. I haven't found any consistency pattern at what time it fails.

### Versions

```shell
System:
   python: 3.9.7 | packaged by conda-forge 
   machine: Windows-10-10.0.19043-SP0
   RAM: 32Gb

Python dependencies:
          pip: 21.3.1
   setuptools: 60.8.1
      sklearn: 1.0.1
        numpy: 1.20.1
        scipy: 1.7.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
The problem is still there if the user code is changed to call `PCA` with `svd_solver="randomized"` explicitly.

It would be interesting to find a reproducer that only involves numpy and scipy and report it upstream.

Labels: Bug, module:decomposition, Needs Investigation
DOC correct use of SAMPLE

### Describe the issue linked to the documentation

The word "sample" in statistics means
> a set of individuals or objects collected or selected

see https://en.wikipedia.org/wiki/Sample_(statistics). Thus, a "single sample" (usually) already consists of several data points or observations (rows), i.e. a sample is a dataset.

In the scikit-lean documentation, however, "samples" is often used to denote the single data points (rows) in a given dataset. A few examples:
- From the glossary: "sample/samples: We usually use this term as a noun to indicate a single feature vector. Elsewhere a sample is called an instance, data point, or observation"
- `n_samples` means "sample size n" and not "there are n samples, i.e. n datasets".
- "However, by partitioning the available data into three sets, we drastically reduce the number of samples which can be used for learning the model ..." from https://scikit-learn.org/stable/modules/cross_validation.html
- "setting the minimum number of samples required at a leaf node" from https://scikit-learn.org/stable/modules/tree.html

### Suggest a potential alternative/fix

My personal point of view is that this is harmful, in particular for education. Therefore, ideally, this would be fixed, at least in the text. I'm well aware for the size of this change and I don't expect that API critical names are changed such as `n_samples` (even if that is unfortunate).

COMMENT:
As a user and an aspiring contributor I can relate to the expressed concern. At times, the usage of "sample"/"samples" becomes especially confusing, as, for example, in [GaussianMixture](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html#sklearn.mixture.GaussianMixture
):
> sample([n_samples]) Generate random samples from the fitted Gaussian distribution.
> score(X[, y]) Compute the per-sample average log-likelihood of the given data X.
> score_samples(X) Compute the log-likelihood of each sample.

Here "sample" is used once as a verb, once in the strict sense (meaning a group), and 3 times in the "vulgar" sense (meaning a point). It is hardly possible to have any confidence based on the above descriptions alone that `score()` returns a single scalar and `score_samples()` returns an array.

Labels: Documentation, Needs Decision
DOC correct use of SAMPLE

### Describe the issue linked to the documentation

The word "sample" in statistics means
> a set of individuals or objects collected or selected

see https://en.wikipedia.org/wiki/Sample_(statistics). Thus, a "single sample" (usually) already consists of several data points or observations (rows), i.e. a sample is a dataset.

In the scikit-lean documentation, however, "samples" is often used to denote the single data points (rows) in a given dataset. A few examples:
- From the glossary: "sample/samples: We usually use this term as a noun to indicate a single feature vector. Elsewhere a sample is called an instance, data point, or observation"
- `n_samples` means "sample size n" and not "there are n samples, i.e. n datasets".
- "However, by partitioning the available data into three sets, we drastically reduce the number of samples which can be used for learning the model ..." from https://scikit-learn.org/stable/modules/cross_validation.html
- "setting the minimum number of samples required at a leaf node" from https://scikit-learn.org/stable/modules/tree.html

### Suggest a potential alternative/fix

My personal point of view is that this is harmful, in particular for education. Therefore, ideally, this would be fixed, at least in the text. I'm well aware for the size of this change and I don't expect that API critical names are changed such as `n_samples` (even if that is unfortunate).

COMMENT:
Being too picky about how terms are used in a specific academic field is not helpful: score, samples are terms that have an acceptance in a statistical literature but are used differently by a wider population. I hear very often, including in academic conference "how many samples do you have" (which is one of the phrases that this issue picks upon). "number of samples" is used in very serious machine learning publications by major authors (for instance https://www.cs.cmu.edu/afs/cs/user/mmv/www/papers/19arxiv-minerl.pdf ), actually a quick scan at ICML papers shows that it's very common to see sentences such as "10000 samples"

We need to make sure that the wordings that we use reflect those that most users would use, and rather than textbooks in mathematical statistics.

While I'm all for avoiding ambiguities, we also need to speak simply.

Labels: Documentation, Needs Decision
DOC correct use of SAMPLE

### Describe the issue linked to the documentation

The word "sample" in statistics means
> a set of individuals or objects collected or selected

see https://en.wikipedia.org/wiki/Sample_(statistics). Thus, a "single sample" (usually) already consists of several data points or observations (rows), i.e. a sample is a dataset.

In the scikit-lean documentation, however, "samples" is often used to denote the single data points (rows) in a given dataset. A few examples:
- From the glossary: "sample/samples: We usually use this term as a noun to indicate a single feature vector. Elsewhere a sample is called an instance, data point, or observation"
- `n_samples` means "sample size n" and not "there are n samples, i.e. n datasets".
- "However, by partitioning the available data into three sets, we drastically reduce the number of samples which can be used for learning the model ..." from https://scikit-learn.org/stable/modules/cross_validation.html
- "setting the minimum number of samples required at a leaf node" from https://scikit-learn.org/stable/modules/tree.html

### Suggest a potential alternative/fix

My personal point of view is that this is harmful, in particular for education. Therefore, ideally, this would be fixed, at least in the text. I'm well aware for the size of this change and I don't expect that API critical names are changed such as `n_samples` (even if that is unfortunate).

COMMENT:
Some ML references with "one sample is a data point", i.e. "a sample of features".
- Elements of Statistical Learning 12th printing Jan 2017:
  "Each basis function is evaluated at each of the N **samples**" (p. 146)
  "Consider a scenario with N = 50 **samples**" (p. 245)
- Shai Shalev-Shwartz and Shai Ben-David (2014) Understanding Machine Learning: From Theory to Algorithms:
  "Explain how the number of available labeled training **samples** will affect your choice." (p. 66)
  "**how many samples are needed for learning**" (p. 100)
- German, Bienenstock, Doursat (1992) Neural Networks and the Bia/Variance Dilemma
  "based on a collection of labeled (pre-classified) training **samples**" (p. 1)
  "given enough training samples" (p. 6)

Note that all those references also use the meaning of "sample" as a set of instances.

**I apologize for the noise.**

As @avm19 points out, there are still places in the doc that could be improved to avoid misunderstanding.


> and rather than textbooks in mathematical statistics.

IMHO, if anything, we should learn/advertise more statistics, not less. (ESL has an "S" in it.)

Labels: Documentation, Needs Decision
DOC correct use of SAMPLE

### Describe the issue linked to the documentation

The word "sample" in statistics means
> a set of individuals or objects collected or selected

see https://en.wikipedia.org/wiki/Sample_(statistics). Thus, a "single sample" (usually) already consists of several data points or observations (rows), i.e. a sample is a dataset.

In the scikit-lean documentation, however, "samples" is often used to denote the single data points (rows) in a given dataset. A few examples:
- From the glossary: "sample/samples: We usually use this term as a noun to indicate a single feature vector. Elsewhere a sample is called an instance, data point, or observation"
- `n_samples` means "sample size n" and not "there are n samples, i.e. n datasets".
- "However, by partitioning the available data into three sets, we drastically reduce the number of samples which can be used for learning the model ..." from https://scikit-learn.org/stable/modules/cross_validation.html
- "setting the minimum number of samples required at a leaf node" from https://scikit-learn.org/stable/modules/tree.html

### Suggest a potential alternative/fix

My personal point of view is that this is harmful, in particular for education. Therefore, ideally, this would be fixed, at least in the text. I'm well aware for the size of this change and I don't expect that API critical names are changed such as `n_samples` (even if that is unfortunate).

COMMENT:
I'd like to add a point related to what @GaelVaroquaux wrote about wordings that "reflect those that most users would use". Scikit-learn Guides and Documentation is a gateway to machine learning for many novices (and for their audience on Medium). Thus, a vicious cycle may form. I am not saying this is the case with "N samples", but more as a general remark.

Labels: Documentation, Needs Decision
Some ideas for a developer-level API

Scikit-Learn developers have long been debating about improving the developer utilities (see #15801 and #20657). The current discussion revolves around the adoption of a **specific syntax** to differentiate the develop API from the common user API. Here, I'd like to propose an alternative solution: **package layering**.

The idea is to break scikit-learn into 2 separate packages, `sklearn` and `sklearn-core`, with the former depending on the latter and the latter providing common utilities for the former. The package `sklearn` defines the user-level API for common users, and the package `sklearn-core` defines the develop-level API for developers (including the third-party ones).

![drawing](https://user-images.githubusercontent.com/6860682/153353262-046b8840-f382-4a46-8356-3d9f7ab0f65f.svg)

I see this solution as a neater one than using a specific syntax. The decomposition of the package explicitly and unambiguously defines the develop API rather than implicitly. It also encourages the core developers to think more carefully when developing scikit-learn. In addition, it has an extra benefit for third-party packages: they can now depend on the light-weight `sklearn-core` than the heavy-duty `sklearn`.

COMMENT:
Naming wise, `sklearn-dev` would make more sense than `sklearn-core` to me. Indeed, with this subset of tools, you are not going to have anything working.

Having 2 packages could be fine but it should not come with 2 repositories. Otherwise, the maintenance will be hell. I see that `dask` is having a single repository and it is just a matter of organizing the setup with the `packages` parameter.

> they can now depend on the light-weight sklearn-core than the heavy-duty sklearn

What I am not really sure about here is where to put the cursor. Basically, you could limit `sklearn-dev` to only what defines a minimal API that one should support. The package could maybe be considered as a core package if we decide to include base classes (e.g. linear model, ensemble, etc.) as well as numerical optimization algorithms or loss functions that could be shared. But the core package becomes the scikit-learn package.

Labels: Needs Decision
Some ideas for a developer-level API

Scikit-Learn developers have long been debating about improving the developer utilities (see #15801 and #20657). The current discussion revolves around the adoption of a **specific syntax** to differentiate the develop API from the common user API. Here, I'd like to propose an alternative solution: **package layering**.

The idea is to break scikit-learn into 2 separate packages, `sklearn` and `sklearn-core`, with the former depending on the latter and the latter providing common utilities for the former. The package `sklearn` defines the user-level API for common users, and the package `sklearn-core` defines the develop-level API for developers (including the third-party ones).

![drawing](https://user-images.githubusercontent.com/6860682/153353262-046b8840-f382-4a46-8356-3d9f7ab0f65f.svg)

I see this solution as a neater one than using a specific syntax. The decomposition of the package explicitly and unambiguously defines the develop API rather than implicitly. It also encourages the core developers to think more carefully when developing scikit-learn. In addition, it has an extra benefit for third-party packages: they can now depend on the light-weight `sklearn-core` than the heavy-duty `sklearn`.

COMMENT:
Let's close the issue. Maybe in the future the discussion will comeback but this is not the case for the moment.

Labels: Needs Decision
GPR `sample_y` enforce `n_targets=1` before calling `fit`

In `GaussianProcessRegressor`, sampling in the prior before calling `fit` via `sample_y` will assume that `y` is made of a single target. However, this is not necessarily the case. Therefore, the shape of the output of `sample_y` before and after `fit` is different.

In order to solve this inconsistency, we need to introduce a new parameter `n_targets=None`. Before calling `fit` this parameter should be explicitly set by the user. After `fit`, we can use the information of the target seen during `fit` without explicitly setting the parameter.

COMMENT:
I see that we have the same issue with `predict` indeed.

Labels: Bug, API, module:gaussian_process
GPR `sample_y` enforce `n_targets=1` before calling `fit`

In `GaussianProcessRegressor`, sampling in the prior before calling `fit` via `sample_y` will assume that `y` is made of a single target. However, this is not necessarily the case. Therefore, the shape of the output of `sample_y` before and after `fit` is different.

In order to solve this inconsistency, we need to introduce a new parameter `n_targets=None`. Before calling `fit` this parameter should be explicitly set by the user. After `fit`, we can use the information of the target seen during `fit` without explicitly setting the parameter.

COMMENT:
Hi @glemaitre, I've opened PR #23099 for a potential fix to this issue, could you help take a look if you got time, thank you !

Labels: Bug, API, module:gaussian_process