Talip7/scikit-learn-issues-multilabel huggingface dataset:
GaussianProcessRegressor (predict)

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/22925

<div type='discussions-op-text'>

<sup>Originally posted by **jecampagne** March 23, 2022</sup>
Hello,
I am questioning the code of `predict` of the  `GaussianProcessRegressor`. The code is based on Alg. 2.1 of C. E. Rasmussen & C. K. I. Williams (2003). I have the 2006 version and I do not know if there have been a modification between the two versions. Well, the algorithm is based on the following formula:

![image](https://user-images.githubusercontent.com/20539759/159709115-293c9cc3-aaf9-497f-9c0a-a7c9c10da73f.png)

![image](https://user-images.githubusercontent.com/20539759/159711732-5c3b305a-ecb0-4874-b79b-11680eb670a5.png)


Notice that the `K(X,X)` (ie, X=X_train) is the only one that contains the "noise" parameter while `K(X*, X)` and `K(X*,X*)` (ie, X*=X_test) do not get this additional part to the diagonal. And this is ok.

Looking now at the code it is ok for the default kernel (ie. RBF with fixed scale and length):
```python
self.kernel_ = C(1.0, constant_value_bounds="fixed") * RBF(
                1.0, length_scale_bounds="fixed"
            )
```
But, if the user pass a kernel composed with the `WhiteKernel`as
```python
kernel = C(1.0) * RBF() + WhiteKernel(0.5)
```
then it seems that `K(X*, X)` and `K(X*,X*)` will use the `WhiteKernel` part which is not what the Alg. 2.1 of Rasmussen & Williams does.




</div>

COMMENT:
[This](https://infallible-thompson-49de36.netlify.app) is a nice resource (linked from [here](https://distill.pub/2019/visual-exploration-gaussian-processes)) that discusses aspects of noise modeling. Using `WhiteKernel` is `sklearn`'s way to learn a noise model by optimizing `WhiteKernel`'s `noise_level` param.

A way to force a fixed `noise_level` is

```py
WhiteKernel(noise_level=0.01, noise_level_bounds="fixed")
```

Labels: module:gaussian_process
GaussianProcessRegressor (predict)

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/22925

<div type='discussions-op-text'>

<sup>Originally posted by **jecampagne** March 23, 2022</sup>
Hello,
I am questioning the code of `predict` of the  `GaussianProcessRegressor`. The code is based on Alg. 2.1 of C. E. Rasmussen & C. K. I. Williams (2003). I have the 2006 version and I do not know if there have been a modification between the two versions. Well, the algorithm is based on the following formula:

![image](https://user-images.githubusercontent.com/20539759/159709115-293c9cc3-aaf9-497f-9c0a-a7c9c10da73f.png)

![image](https://user-images.githubusercontent.com/20539759/159711732-5c3b305a-ecb0-4874-b79b-11680eb670a5.png)


Notice that the `K(X,X)` (ie, X=X_train) is the only one that contains the "noise" parameter while `K(X*, X)` and `K(X*,X*)` (ie, X*=X_test) do not get this additional part to the diagonal. And this is ok.

Looking now at the code it is ok for the default kernel (ie. RBF with fixed scale and length):
```python
self.kernel_ = C(1.0, constant_value_bounds="fixed") * RBF(
                1.0, length_scale_bounds="fixed"
            )
```
But, if the user pass a kernel composed with the `WhiteKernel`as
```python
kernel = C(1.0) * RBF() + WhiteKernel(0.5)
```
then it seems that `K(X*, X)` and `K(X*,X*)` will use the `WhiteKernel` part which is not what the Alg. 2.1 of Rasmussen & Williams does.




</div>

COMMENT:
But then if one uses `WhiteKernel(noise_level=0.01, noise_level_bounds="fixed")` is treated for `predict` in a same way by  `sklearn` if 
1) `WhiteKernel(noise_level=0.01)` (ie not fixed) but at the end of the hyperparameter optimisation the `noise_level` is very close to 0.01
2) `alpha` parameter of `GaussianProcessRegressor` which here would be set to 0.01.

Labels: module:gaussian_process
GaussianProcessRegressor (predict)

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/22925

<div type='discussions-op-text'>

<sup>Originally posted by **jecampagne** March 23, 2022</sup>
Hello,
I am questioning the code of `predict` of the  `GaussianProcessRegressor`. The code is based on Alg. 2.1 of C. E. Rasmussen & C. K. I. Williams (2003). I have the 2006 version and I do not know if there have been a modification between the two versions. Well, the algorithm is based on the following formula:

![image](https://user-images.githubusercontent.com/20539759/159709115-293c9cc3-aaf9-497f-9c0a-a7c9c10da73f.png)

![image](https://user-images.githubusercontent.com/20539759/159711732-5c3b305a-ecb0-4874-b79b-11680eb670a5.png)


Notice that the `K(X,X)` (ie, X=X_train) is the only one that contains the "noise" parameter while `K(X*, X)` and `K(X*,X*)` (ie, X*=X_test) do not get this additional part to the diagonal. And this is ok.

Looking now at the code it is ok for the default kernel (ie. RBF with fixed scale and length):
```python
self.kernel_ = C(1.0, constant_value_bounds="fixed") * RBF(
                1.0, length_scale_bounds="fixed"
            )
```
But, if the user pass a kernel composed with the `WhiteKernel`as
```python
kernel = C(1.0) * RBF() + WhiteKernel(0.5)
```
then it seems that `K(X*, X)` and `K(X*,X*)` will use the `WhiteKernel` part which is not what the Alg. 2.1 of Rasmussen & Williams does.




</div>

COMMENT:
1. Yes, if the optimizer lands in the same optimum of the log marginal likelihood, i.e. you get the same kernel params (`length_scale` in `RBF()`) as you would when optimizing both `length_scale` and `noise_level`.

2. Yes (`y_mean`), no (`y_std` / `y_cov`): You can hack a fixed `noise_level` by using

   ```py
   GaussianProcessRegressor(
       kernel=RBF(...),
       alpha=noise_level,
       ...
       )
   ```

   and you will get the same fit weights `GaussianProcessRegressor.alpha_` and thus the same predicted `y_mean` as with

   ```py
   GaussianProcessRegressor(
       kernel = RBF(...)
        + WhiteKernel(noise_level=noise_level, noise_level_bounds="fixed")
       alpha=0,
       ...
       )
   ```

   But `y_cov` and `y_std` will be different (like `predict_noiseless` as discussed above) b/c you don't use `WhiteKernel`. This is what the `sklearn` part of the above linked ~gist~ repo shows.

Labels: module:gaussian_process
GaussianProcessRegressor (predict)

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/22925

<div type='discussions-op-text'>

<sup>Originally posted by **jecampagne** March 23, 2022</sup>
Hello,
I am questioning the code of `predict` of the  `GaussianProcessRegressor`. The code is based on Alg. 2.1 of C. E. Rasmussen & C. K. I. Williams (2003). I have the 2006 version and I do not know if there have been a modification between the two versions. Well, the algorithm is based on the following formula:

![image](https://user-images.githubusercontent.com/20539759/159709115-293c9cc3-aaf9-497f-9c0a-a7c9c10da73f.png)

![image](https://user-images.githubusercontent.com/20539759/159711732-5c3b305a-ecb0-4874-b79b-11680eb670a5.png)


Notice that the `K(X,X)` (ie, X=X_train) is the only one that contains the "noise" parameter while `K(X*, X)` and `K(X*,X*)` (ie, X*=X_test) do not get this additional part to the diagonal. And this is ok.

Looking now at the code it is ok for the default kernel (ie. RBF with fixed scale and length):
```python
self.kernel_ = C(1.0, constant_value_bounds="fixed") * RBF(
                1.0, length_scale_bounds="fixed"
            )
```
But, if the user pass a kernel composed with the `WhiteKernel`as
```python
kernel = C(1.0) * RBF() + WhiteKernel(0.5)
```
then it seems that `K(X*, X)` and `K(X*,X*)` will use the `WhiteKernel` part which is not what the Alg. 2.1 of Rasmussen & Williams does.




</div>

COMMENT:
I think the above is now covered by the [current docs](https://scikit-learn.org/stable/modules/gaussian_process.html#gaussian-process-regression-gpr) and [this example](https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_noisy.html#sphx-glr-auto-examples-gaussian-process-plot-gpr-noisy-py) reasonably well, but brief enough to keep the user guide compact:

* `alpha` : fixed known scalar ("homoscedastic") noise or 1d array ("heteroscedastic")
* `WhiteKernel` learn scalar noise

What could be added / stated more clearly in the user guide is that

* define fixed known scalar noise with `WhiteKernel(noise_level=noise_level, noise_level_bounds="fixed")` instead of `alpha` is also possible, and I would in fact recommend this approach
* only when using `WhiteKernel` will the noise $\sigma_n^2$ be added to the kernel matrix diag when calling `predict()`

The above discussion and the last two points are covered in detail [over here](https://elcorto.github.io/gp_playground/content/gp_pred_comp/intro.html), but I'm hesitant to duplicate this in the sklearn docs. If people find this useful, I can send a PR to link to this material.

Further, I believe that learning a 1d array (heteroscedastic noise) is not possible with sklearn as far as I could see. If that's the case, then maybe the user guide could highlight that.

Labels: module:gaussian_process
AttributeError when use ‘CountVectorizer‘ with ‘preprocessor ’

I define a preprocessing function like:

def clean_parens(text):
    text = re.sub("=","e",text)
    return text

Then, word vector extraction is defined as：

SelfDicCvB = CountVectorizer(input='filename',vocabulary=voc['word'].tolist(),
                             decode_error='ignore',
                             token_pattern="[x=_\\\?\+\-\&\#\$\(\[\{\*\/\'\"\}\.\;\,\|\^\~\<\>\`]+", 
                             lowercase=True,
                             preprocessor = clean_parens)

And save to pkl file.
with open("SelfDicCvB.model",'wb') as file:
    pkl.dump(SelfDicCvB,file)

When I read the model from the pkl file, I get the following error：

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
AttributeError: Can't get attribute 'clean_parens' on <module '__main__' (built-in)>

The problem is how to save to .pkl file when using  'preprocessor '?

COMMENT:
This is a common Python issue when pickling a function in `__main__`. A common alternative is to use [dill](https://github.com/uqfoundation/dill) or [cloudpickle](https://github.com/cloudpipe/cloudpickle) instead of pickle.

Labels: Needs Triage
sklearn.preprocessing.LabelEncoder: new feature to manage unknown value in `transform()` method

### Describe the workflow you want to enable

Hi, I'm working with `from sklearn.preprocessing import LabelEncoder` and I would find interesting (and useful) to have an option to not get an error in case of when using the `transform()` method there is an unknown label as in this toy example.

Currently:

```python
import pandas as pd
from sklearn.preprocessing import LabelEncoder


le = LabelEncoder()
train = pd.DataFrame({"x1": ["a", "b", "c"], "y": [1, 2, 3]})
test = pd.DataFrame({"x1": ["a", "c", "d"], "y": [1, 3, 4]})
print(le.fit_transform(train["x1"]))
print(le.transform(test["x1"]))
```

output is:
```
[0 1 2]
KeyError
.
.
.
ValueError: y contains previously unseen labels: 'd'
```

Wanted behaviour/feature:
```python
import pandas as pd
from sklearn.preprocessing import LabelEncoder


le = LabelEncoder()
train = pd.DataFrame({"x1": ["a", "b", "c"], "y": [1, 2, 3]})
test = pd.DataFrame({"x1": ["a", "c", "d"], "y": [1, 3, 4]})
print(le.fit_transform(train["x1"]))
print(le.transform(test["x1"], allow_unkown=True))
```

output is something like:
```
[0 1 2]
[0 2 None]
```

### Describe your proposed solution

We could do this by adding a new parameter to the `transform()` method of `LabelEncoder()` class or in the `__init__()`method of the `LabelEncoder()` class. The parameter could also looks like `replace_unknown=False` if we don't want this new feature (=default value) or if `replace_unknown!=False` we will replace the unknown value (not provided during the `fit()`) by the specified value (for example: `replace_unknown=None` will replace the unknown values by `None` or `replace_unknown=0` will replace the unknown values by `0`).
I could open a PR to propose this change

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
We have a very similar feature for `OrdinalEncoder`, that encodes unknown categorical features:

```python
import pandas as pd
from sklearn.preprocessing import OrdinalEncoder

# We use unknown_value=-1 so that the encoded output is numerical
le = OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1)
train = pd.DataFrame({"x1": ["a", "b", "c"], "y": [1, 2, 3]})
test = pd.DataFrame({"x1": ["a", "c", "d"], "y": [1, 3, 4]})
le.fit(train[["x1"]])

le.transform(test[["x1"]])
# array([[ 0.],
#        [ 2.],
#        [-1.]])
```


To better understand the issue, can you describe your use case for allowing unknown classes in `LabelEncoder`?

Labels: New Feature, module:preprocessing
sklearn.preprocessing.LabelEncoder: new feature to manage unknown value in `transform()` method

### Describe the workflow you want to enable

Hi, I'm working with `from sklearn.preprocessing import LabelEncoder` and I would find interesting (and useful) to have an option to not get an error in case of when using the `transform()` method there is an unknown label as in this toy example.

Currently:

```python
import pandas as pd
from sklearn.preprocessing import LabelEncoder


le = LabelEncoder()
train = pd.DataFrame({"x1": ["a", "b", "c"], "y": [1, 2, 3]})
test = pd.DataFrame({"x1": ["a", "c", "d"], "y": [1, 3, 4]})
print(le.fit_transform(train["x1"]))
print(le.transform(test["x1"]))
```

output is:
```
[0 1 2]
KeyError
.
.
.
ValueError: y contains previously unseen labels: 'd'
```

Wanted behaviour/feature:
```python
import pandas as pd
from sklearn.preprocessing import LabelEncoder


le = LabelEncoder()
train = pd.DataFrame({"x1": ["a", "b", "c"], "y": [1, 2, 3]})
test = pd.DataFrame({"x1": ["a", "c", "d"], "y": [1, 3, 4]})
print(le.fit_transform(train["x1"]))
print(le.transform(test["x1"], allow_unkown=True))
```

output is something like:
```
[0 1 2]
[0 2 None]
```

### Describe your proposed solution

We could do this by adding a new parameter to the `transform()` method of `LabelEncoder()` class or in the `__init__()`method of the `LabelEncoder()` class. The parameter could also looks like `replace_unknown=False` if we don't want this new feature (=default value) or if `replace_unknown!=False` we will replace the unknown value (not provided during the `fit()`) by the specified value (for example: `replace_unknown=None` will replace the unknown values by `None` or `replace_unknown=0` will replace the unknown values by `0`).
I could open a PR to propose this change

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> We have a very similar feature for `OrdinalEncoder`, that encodes unknown categorical features:
> 
> ```python
> import pandas as pd
> from sklearn.preprocessing import OrdinalEncoder
> 
> # We use unknown_value=-1 so that the encoded output is numerical
> le = OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1)
> train = pd.DataFrame({"x1": ["a", "b", "c"], "y": [1, 2, 3]})
> test = pd.DataFrame({"x1": ["a", "c", "d"], "y": [1, 3, 4]})
> le.fit(train[["x1"]])
> 
> le.transform(test[["x1"]])
> # array([[ 0.],
> #        [ 2.],
> #        [-1.]])
> ```
> 
> To better understand the issue, can you describe your use case for allowing unknown classes in `LabelEncoder`?

Thanks for the very quick reply @thomasjpfan. I think I was wrongly using `LabelEncoder` because I'm encoding features data X and not label variable y as it's well specified [here](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html#sklearn-preprocessing-labelencoder) `This transformer should be used to encode target values, i.e. y, and not the input X.`

So I'm closing the issue.

Thank you again @thomasjpfan !

Labels: New Feature, module:preprocessing
CI: CircleCI artifact link has changed

see https://184075-843222-gh.circle-artifacts.com/0/doc/_changed.html
or https://184079-843222-gh.circle-artifacts.com/0/doc/_changed.html

COMMENT:
It looks like the URL changed. Here is an example of the URL to one of the PRS:

https://output.circle-artifacts.com/output/job/2a3449a2-96ea-4e27-997e-79462d4ecef7/artifacts/0/doc/_changed.html

Labels: Documentation, Build / CI
CI: CircleCI artifact link has changed

see https://184075-843222-gh.circle-artifacts.com/0/doc/_changed.html
or https://184079-843222-gh.circle-artifacts.com/0/doc/_changed.html

COMMENT:
Cross linking the issue opened upstream : https://github.com/larsoner/circleci-artifacts-redirector/issues/8

Labels: Documentation, Build / CI
CI: CircleCI artifact link has changed

see https://184075-843222-gh.circle-artifacts.com/0/doc/_changed.html
or https://184079-843222-gh.circle-artifacts.com/0/doc/_changed.html

COMMENT:
The link has been fixed upstream but only for the action for now (https://github.com/larsoner/circleci-artifacts-redirector-action/pull/20). It's a good opportunity to switch to using circleci-artifact-redirector as an action. PR here https://github.com/scikit-learn/scikit-learn/pull/22991

Labels: Documentation, Build / CI
roc_auc_score, support `average=None` for multiclass labels

### Describe the workflow you want to enable

Currently as of version 1.0.2 If you try to calculate the roc_auc_score for a multiclass `y_true`  with `average=None` you get an error:

````
import sklearn.metrics as metrics
import numpy as np

y_true = [0, 1, 2, 2]
y_score = np.array([[0.2, 0.3, 0.5], [0.3, 0.6, 0.1], [0.1, 0.1, 0.8] ,[0.3, 0.6, 0.1]])

metrics.roc_auc_score(y_true, y_score, multi_class="ovr", average=None)
````

```
ValueError: average must be one of ('macro', 'weighted') for multiclass problems
```



### Describe your proposed solution

The Area under the ROC Curve can still be calculated for every group in both the ``ovr`` and ``ovo`` setting. Furthermore, it can be calculated using the current ``roc_auc_score`` implementation.

Example:

````
import sklearn.metrics as metrics
import numpy as np

y_true = [0, 1, 2, 2]
y_score = np.array([[0.2, 0.3, 0.5], [0.3, 0.6, 0.1], [0.1, 0.1, 0.8] ,[0.3, 0.6, 0.1]])

metrics.roc_auc_score(y_true, y_score, multi_class="ovr")
````
> 0.5972222222222222


Now we can calculate the roc-auc for every group in the ``ovr``

````
roc_labels = []
for y_i in np.unique(y_true):
    roc_labels.append(
        metrics.roc_auc_score(
            np.vectorize(
                {y_i: 1, **{y_k: 0 for y_k in np.unique(y_true) if y_k != y_i}}.get
            )(y_true),
            y_score[:, y_i],
        )
    )
````

And we get the same result
``
roc_labels
``
> [0.33333333333333337, 0.8333333333333334, 0.625]

``
np.mean(roc_labels)
``
> 0.5972222222222222



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thank you for opening the issue! This feature was recently added in https://github.com/scikit-learn/scikit-learn/commit/a4c78b4b939540d8e4d56e68b453054f0c393145 and will be in the 1.1 release. On the dev build:

```python
import sklearn.metrics as metrics
import numpy as np

y_true = [0, 1, 2, 2]
y_score = np.array([[0.2, 0.3, 0.5], [0.3, 0.6, 0.1], [0.1, 0.1, 0.8] ,[0.3, 0.6, 0.1]])

metrics.roc_auc_score(y_true, y_score, multi_class="ovr", average=None)
# array([0.33333333, 0.83333333, 0.625     ])
```

You can try this feature out by installing the nightly build:

```bash
pip install --pre --extra-index https://pypi.anaconda.org/scipy-wheels-nightly/simple scikit-learn
```

Labels: New Feature, module:metrics
R squared is missing in Scikit learn

### Describe the bug

For the calculation of R squared, you need to determine the Correlation coefficient, and then you need to square the result. 

Is not that R2 score with negative values... 

### Steps/Code to Reproduce

from sklearn.metrics import r2_score
y_true = [1, 2, 3]
y_pred = [3, 2, 1]
r2_score(y_true, y_pred)


### Expected Results

The results is -3!!!!

R2 should be 1.

you have a perfect correlation coeficient of -1.

The correct R2 score should be available! Or, all Scikit reputation is in stake. Not even the basics are correct... I can not trust a more complex fuction, when a basic test fails!

### Actual Results

-3.0

### Versions

```shell
import sklearn; sklearn.show_versions()
```

COMMENT:
scikit-learn uses the most [general definition of r2](https://scikit-learn.org/stable/modules/model_evaluation.html#r2-score-the-coefficient-of-determination):

![Screen Shot 2022-03-23 at 11 31 50 AM](https://user-images.githubusercontent.com/5402633/159736647-3203c755-dca6-4d2f-bdaf-6353432dd828.jpg)

which can also be seen as:

![Screen Shot 2022-03-23 at 11 22 30 AM](https://user-images.githubusercontent.com/5402633/159736732-ef2e7ce4-ad33-4c89-bcb3-c0a75f88817e.jpg)


where `SS_res` is the residual sum of squares and `SS_tot` is the total sum of squares. In your case, we were to compute the r2 manually to get the same result as `r2_score`:

```python
import numpy as np

y_true = np.asarray([1.0, 2.0, 3.0])
y_pred = np.asarray([3.0, 2.0, 1.0])

ss_residual = np.sum((y_true - y_pred)**2)
ss_total = np.sum((y_true - np.mean(y_true))**2)

print(1 - ss_residual / ss_total)
# -3.0
```

Labels: Bug, Needs Triage
R squared is missing in Scikit learn

### Describe the bug

For the calculation of R squared, you need to determine the Correlation coefficient, and then you need to square the result. 

Is not that R2 score with negative values... 

### Steps/Code to Reproduce

from sklearn.metrics import r2_score
y_true = [1, 2, 3]
y_pred = [3, 2, 1]
r2_score(y_true, y_pred)


### Expected Results

The results is -3!!!!

R2 should be 1.

you have a perfect correlation coeficient of -1.

The correct R2 score should be available! Or, all Scikit reputation is in stake. Not even the basics are correct... I can not trust a more complex fuction, when a basic test fails!

### Actual Results

-3.0

### Versions

```shell
import sklearn; sklearn.show_versions()
```

COMMENT:
Well, this says alot about scikit-learn.
What is the correlation of 1 , 2 , 3 with 3 , 2 , 1?
scikit-learn answer -3
LOL

Labels: Bug, Needs Triage
R squared is missing in Scikit learn

### Describe the bug

For the calculation of R squared, you need to determine the Correlation coefficient, and then you need to square the result. 

Is not that R2 score with negative values... 

### Steps/Code to Reproduce

from sklearn.metrics import r2_score
y_true = [1, 2, 3]
y_pred = [3, 2, 1]
r2_score(y_true, y_pred)


### Expected Results

The results is -3!!!!

R2 should be 1.

you have a perfect correlation coeficient of -1.

The correct R2 score should be available! Or, all Scikit reputation is in stake. Not even the basics are correct... I can not trust a more complex fuction, when a basic test fails!

### Actual Results

-3.0

### Versions

```shell
import sklearn; sklearn.show_versions()
```

COMMENT:
It is fairly common to use the [general definition for the coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination).

For example, [R's caret package](https://rdrr.io/cran/caret/) will also output -3 when configured to use the traditional definition:

```R
library(caret)
pred <- c(1, 2, 3)
true <- c(3, 2, 1)

R2(pred, true, formula="traditional")
# [1] -3
```

Labels: Bug, Needs Triage