Talip7/scikit-learn-issues-multilabel huggingface dataset:
BUG log_loss renormalizes the predictions

### Describe the bug

`log_loss(y_true, y_pred)` renormalizes `y_pred` internally such that it sums to 1. This way, a really bad model, the predictions of which do not sum to 1, gets a better loss then it actually has.

### Steps/Code to Reproduce

```python
from scipy.special import xlogy
from sklearn.metrics import log_loss

y_true = [[0, 1]]
y_pred = [[0.2, 0.3]]

log_loss(y_true, y_pred)
```

### Expected Results

```python
-xlogy(y_true, y_pred).sum(axis=1)
```
Result: `1.2039728`

### Actual Results

Result: `0.5108256237659907`

### Versions

```shell
System:
    python: 3.9.14
   machine: macOS

Python dependencies:
      sklearn: 1.1.2
```

COMMENT:
can u share me the refernce of the code where the bug was there

Labels: Bug, help wanted, module:metrics
BUG log_loss renormalizes the predictions

### Describe the bug

`log_loss(y_true, y_pred)` renormalizes `y_pred` internally such that it sums to 1. This way, a really bad model, the predictions of which do not sum to 1, gets a better loss then it actually has.

### Steps/Code to Reproduce

```python
from scipy.special import xlogy
from sklearn.metrics import log_loss

y_true = [[0, 1]]
y_pred = [[0.2, 0.3]]

log_loss(y_true, y_pred)
```

### Expected Results

```python
-xlogy(y_true, y_pred).sum(axis=1)
```
Result: `1.2039728`

### Actual Results

Result: `0.5108256237659907`

### Versions

```shell
System:
    python: 3.9.14
   machine: macOS

Python dependencies:
      sklearn: 1.1.2
```

COMMENT:
@TomDLT I'd be interested in you opinion. We hit this in https://github.com/scikit-learn/scikit-learn/pull/24365#discussion_r976815764.

Labels: Bug, help wanted, module:metrics
BUG log_loss renormalizes the predictions

### Describe the bug

`log_loss(y_true, y_pred)` renormalizes `y_pred` internally such that it sums to 1. This way, a really bad model, the predictions of which do not sum to 1, gets a better loss then it actually has.

### Steps/Code to Reproduce

```python
from scipy.special import xlogy
from sklearn.metrics import log_loss

y_true = [[0, 1]]
y_pred = [[0.2, 0.3]]

log_loss(y_true, y_pred)
```

### Expected Results

```python
-xlogy(y_true, y_pred).sum(axis=1)
```
Result: `1.2039728`

### Actual Results

Result: `0.5108256237659907`

### Versions

```shell
System:
    python: 3.9.14
   machine: macOS

Python dependencies:
      sklearn: 1.1.2
```

COMMENT:
I feel like computing a log loss with probabilities not summing to one does not make sense, so I am ok with the renormalization.

> a really bad model, the predictions of which do not sum to 1

To me this is not a bad model (a model that has poor prediction accuracy), this is an incorrect model (a model that does not return results in the expected format), as would be a model that predicts negative probabilities. We could almost raise an error if the probabilities do not sum to one (or close), but I feel like it is simpler to just renormalize and compute the loss.

Labels: Bug, help wanted, module:metrics
BUG log_loss renormalizes the predictions

### Describe the bug

`log_loss(y_true, y_pred)` renormalizes `y_pred` internally such that it sums to 1. This way, a really bad model, the predictions of which do not sum to 1, gets a better loss then it actually has.

### Steps/Code to Reproduce

```python
from scipy.special import xlogy
from sklearn.metrics import log_loss

y_true = [[0, 1]]
y_pred = [[0.2, 0.3]]

log_loss(y_true, y_pred)
```

### Expected Results

```python
-xlogy(y_true, y_pred).sum(axis=1)
```
Result: `1.2039728`

### Actual Results

Result: `0.5108256237659907`

### Versions

```shell
System:
    python: 3.9.14
   machine: macOS

Python dependencies:
      sklearn: 1.1.2
```

COMMENT:
> I feel like computing a log loss with probabilities not summing to one does not make sense, so I am ok with the renormalization.

To be honest, I strongly think renormalizing in the metric is methodologically wrong because this way the metric is not measuring the model prediction anymore. The model should take care of it's own predictions (i.e. make them sum to 1), not the metric!

A metric is like a measuring device, say a scale (for measuring weight). We are measuring objects for a flight transport, so lighter is better. Renormalization is like measuring the weight of objects with their packaging removed. But the flight will have to carry the whole objects, with packaging included.

Labels: Bug, help wanted, module:metrics
BUG log_loss renormalizes the predictions

### Describe the bug

`log_loss(y_true, y_pred)` renormalizes `y_pred` internally such that it sums to 1. This way, a really bad model, the predictions of which do not sum to 1, gets a better loss then it actually has.

### Steps/Code to Reproduce

```python
from scipy.special import xlogy
from sklearn.metrics import log_loss

y_true = [[0, 1]]
y_pred = [[0.2, 0.3]]

log_loss(y_true, y_pred)
```

### Expected Results

```python
-xlogy(y_true, y_pred).sum(axis=1)
```
Result: `1.2039728`

### Actual Results

Result: `0.5108256237659907`

### Versions

```shell
System:
    python: 3.9.14
   machine: macOS

Python dependencies:
      sklearn: 1.1.2
```

COMMENT:
> We could almost raise an error if the probabilities do not sum to one (or close), but I feel like it is simpler to just renormalize and compute the loss.

I would be fine with raising a warning first and an error in the future.

Labels: Bug, help wanted, module:metrics
BUG log_loss renormalizes the predictions

### Describe the bug

`log_loss(y_true, y_pred)` renormalizes `y_pred` internally such that it sums to 1. This way, a really bad model, the predictions of which do not sum to 1, gets a better loss then it actually has.

### Steps/Code to Reproduce

```python
from scipy.special import xlogy
from sklearn.metrics import log_loss

y_true = [[0, 1]]
y_pred = [[0.2, 0.3]]

log_loss(y_true, y_pred)
```

### Expected Results

```python
-xlogy(y_true, y_pred).sum(axis=1)
```
Result: `1.2039728`

### Actual Results

Result: `0.5108256237659907`

### Versions

```shell
System:
    python: 3.9.14
   machine: macOS

Python dependencies:
      sklearn: 1.1.2
```

COMMENT:
I agree with @lorentzenchr that the current behavior is very surprising and therefore it's a bug to me.

Labels: Bug, help wanted, module:metrics
BUG log_loss renormalizes the predictions

### Describe the bug

`log_loss(y_true, y_pred)` renormalizes `y_pred` internally such that it sums to 1. This way, a really bad model, the predictions of which do not sum to 1, gets a better loss then it actually has.

### Steps/Code to Reproduce

```python
from scipy.special import xlogy
from sklearn.metrics import log_loss

y_true = [[0, 1]]
y_pred = [[0.2, 0.3]]

log_loss(y_true, y_pred)
```

### Expected Results

```python
-xlogy(y_true, y_pred).sum(axis=1)
```
Result: `1.2039728`

### Actual Results

Result: `0.5108256237659907`

### Versions

```shell
System:
    python: 3.9.14
   machine: macOS

Python dependencies:
      sklearn: 1.1.2
```

COMMENT:
What should be the behavior for larger `eps` values? For example, the following has `eps=0.1`:

```python
from sklearn.metrics import log_loss
import numpy as np

y_true = [0, 1, 2]
y_pred = [[0, 0, 1], [0, 1, 0], [0, 0, 1]]

eps = 0.1
log_loss(y_true, y_pred, eps=eps)
# 0.9330788879075577

# `log_loss` will use a clipped `y_pred` for computing the log loss:
np.clip(y_pred, eps, 1 - eps)
# array([[0.1, 0.1, 0.9],
#        [0.1, 0.9, 0.1],
#        [0.1, 0.1, 0.9]])
```

We could just validate the input, i.e. `np.isclose(y_pred.sum(axis=1), 1.0)`, but `log_loss` will use the clipped version for computation. For reference, here is the clipping:

https://github.com/scikit-learn/scikit-learn/blob/d52e946fa4fca4282b0065ddcb0dd5d268c956e7/sklearn/metrics/_classification.py#L2629-L2630

For me, I think that `eps` shouldn't be a parameter and should always be `np.finfo(y_pred.dtype).eps`. For reference, PyTorch's `binary_cross_entropy` [clips the log to -100 by default](https://github.com/pytorch/pytorch/blob/e47af44eb81b9cd0c3583de91b0a2d4f56a5cf8d/aten/src/ATen/native/Loss.cpp#L292-L297)

Labels: Bug, help wanted, module:metrics
BUG log_loss renormalizes the predictions

### Describe the bug

`log_loss(y_true, y_pred)` renormalizes `y_pred` internally such that it sums to 1. This way, a really bad model, the predictions of which do not sum to 1, gets a better loss then it actually has.

### Steps/Code to Reproduce

```python
from scipy.special import xlogy
from sklearn.metrics import log_loss

y_true = [[0, 1]]
y_pred = [[0.2, 0.3]]

log_loss(y_true, y_pred)
```

### Expected Results

```python
-xlogy(y_true, y_pred).sum(axis=1)
```
Result: `1.2039728`

### Actual Results

Result: `0.5108256237659907`

### Versions

```shell
System:
    python: 3.9.14
   machine: macOS

Python dependencies:
      sklearn: 1.1.2
```

COMMENT:
I agree with @thomasjpfan that `eps` should not be a parameter. In my opinion, the perfect default value is 0 (what is the use case of having it in the first place?). I'm also fine with `np.finfo(y_pred.dtype).eps`.

Labels: Bug, help wanted, module:metrics
BUG log_loss renormalizes the predictions

### Describe the bug

`log_loss(y_true, y_pred)` renormalizes `y_pred` internally such that it sums to 1. This way, a really bad model, the predictions of which do not sum to 1, gets a better loss then it actually has.

### Steps/Code to Reproduce

```python
from scipy.special import xlogy
from sklearn.metrics import log_loss

y_true = [[0, 1]]
y_pred = [[0.2, 0.3]]

log_loss(y_true, y_pred)
```

### Expected Results

```python
-xlogy(y_true, y_pred).sum(axis=1)
```
Result: `1.2039728`

### Actual Results

Result: `0.5108256237659907`

### Versions

```shell
System:
    python: 3.9.14
   machine: macOS

Python dependencies:
      sklearn: 1.1.2
```

COMMENT:
@lorentzenchr Would it be okay for me to work on this issue?

Labels: Bug, help wanted, module:metrics
BUG log_loss renormalizes the predictions

### Describe the bug

`log_loss(y_true, y_pred)` renormalizes `y_pred` internally such that it sums to 1. This way, a really bad model, the predictions of which do not sum to 1, gets a better loss then it actually has.

### Steps/Code to Reproduce

```python
from scipy.special import xlogy
from sklearn.metrics import log_loss

y_true = [[0, 1]]
y_pred = [[0.2, 0.3]]

log_loss(y_true, y_pred)
```

### Expected Results

```python
-xlogy(y_true, y_pred).sum(axis=1)
```
Result: `1.2039728`

### Actual Results

Result: `0.5108256237659907`

### Versions

```shell
System:
    python: 3.9.14
   machine: macOS

Python dependencies:
      sklearn: 1.1.2
```

COMMENT:
@OmarManzoor If the solution is clear to you, then yes.

Labels: Bug, help wanted, module:metrics
BUG log_loss renormalizes the predictions

### Describe the bug

`log_loss(y_true, y_pred)` renormalizes `y_pred` internally such that it sums to 1. This way, a really bad model, the predictions of which do not sum to 1, gets a better loss then it actually has.

### Steps/Code to Reproduce

```python
from scipy.special import xlogy
from sklearn.metrics import log_loss

y_true = [[0, 1]]
y_pred = [[0.2, 0.3]]

log_loss(y_true, y_pred)
```

### Expected Results

```python
-xlogy(y_true, y_pred).sum(axis=1)
```
Result: `1.2039728`

### Actual Results

Result: `0.5108256237659907`

### Versions

```shell
System:
    python: 3.9.14
   machine: macOS

Python dependencies:
      sklearn: 1.1.2
```

COMMENT:
> @OmarManzoor If the solution is clear to you, then yes.

From what I can understand we need to raise a warning if `y_pred` does not sum to 1 and we need to remove eps as a parameter to this metric and instead use a value of `np.finfo(y_pred.dtype).eps`.

Labels: Bug, help wanted, module:metrics
BUG log_loss renormalizes the predictions

### Describe the bug

`log_loss(y_true, y_pred)` renormalizes `y_pred` internally such that it sums to 1. This way, a really bad model, the predictions of which do not sum to 1, gets a better loss then it actually has.

### Steps/Code to Reproduce

```python
from scipy.special import xlogy
from sklearn.metrics import log_loss

y_true = [[0, 1]]
y_pred = [[0.2, 0.3]]

log_loss(y_true, y_pred)
```

### Expected Results

```python
-xlogy(y_true, y_pred).sum(axis=1)
```
Result: `1.2039728`

### Actual Results

Result: `0.5108256237659907`

### Versions

```shell
System:
    python: 3.9.14
   machine: macOS

Python dependencies:
      sklearn: 1.1.2
```

COMMENT:
@OmarManzoor Yes. You are very welcome to go ahead, open a PR and link it with this issue. Keep in mind that `eps` needs a a deprecation cycle, i.e. throw a warning when set for 2 releases. Be prepared that a few concerns might be raised during review.

Labels: Bug, help wanted, module:metrics
BUG log_loss renormalizes the predictions

### Describe the bug

`log_loss(y_true, y_pred)` renormalizes `y_pred` internally such that it sums to 1. This way, a really bad model, the predictions of which do not sum to 1, gets a better loss then it actually has.

### Steps/Code to Reproduce

```python
from scipy.special import xlogy
from sklearn.metrics import log_loss

y_true = [[0, 1]]
y_pred = [[0.2, 0.3]]

log_loss(y_true, y_pred)
```

### Expected Results

```python
-xlogy(y_true, y_pred).sum(axis=1)
```
Result: `1.2039728`

### Actual Results

Result: `0.5108256237659907`

### Versions

```shell
System:
    python: 3.9.14
   machine: macOS

Python dependencies:
      sklearn: 1.1.2
```

COMMENT:
> @OmarManzoor Yes. You are very welcome to go ahead, open a PR and link it with this issue. Keep in mind that `eps` needs a a deprecation cycle, i.e. throw a warning when set for 2 releases. Be prepared that a few concerns might be raised during review.

Sure thank you.

Labels: Bug, help wanted, module:metrics
Sparse random projection description is incorrect in docs

### Describe the bug

See: https://scikit-learn.org/stable/modules/generated/sklearn.random_projection.SparseRandomProjection.html#sklearn-random-projection-sparserandomprojection

The docs say that if s = 1 / density, then the weights for drawing the values of the sparse matrix should be 1/(2s), 1 - 1/s, and 1/(2s). However, the reference gives weights 1/(2\*density), etc. The sklearn docs should give weights of 1/2\*s, 1-s, and 1/2\*s. (It looks like someone just read 1/2s as 1/(2s) instead of 1/2\*s.) Hopefully this is implemented correctly in the actual code!

### Steps/Code to Reproduce

N/a

### Expected Results

N/a

### Actual Results

N/a

### Versions

```shell
Online docs of sklearn 1.1.2
```

COMMENT:
Sorry, I am wrong -- just got confused by all the fractions!

Labels: Bug, Needs Triage
⚠️ CI failed on Linux_Nightly_ICC.pylatest_conda_forge_mkl ⚠️

**CI failed on [Linux_Nightly_ICC.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=47016&view=logs&j=8628a494-79d0-53fa-274c-1b00464f7121)** (Sep 24, 2022)
Unable to find junit file. Please see link for details.

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=47027&view=logs&j=8628a494-79d0-53fa-274c-1b00464f7121) on Sep 25, 2022

Labels: Needs Triage
⚠️ CI failed on Linux_Nightly_ICC.pylatest_conda_forge_mkl ⚠️

**CI failed on [Linux_Nightly_ICC.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=47016&view=logs&j=8628a494-79d0-53fa-274c-1b00464f7121)** (Sep 24, 2022)
Unable to find junit file. Please see link for details.

COMMENT:
The failing CI tests was because of a random network issue. I think this is safe to close.

Labels: Needs Triage
RFC Should pairwise_distances preserve float32 ?

Currently the dtype of the distance matrix returned by `pairwise_distances` is not very consistent, depending on the metric and on the value of n_jobs.

For float64 input, everything is consistent: the returned matrix is always in float64.
For mixed float64 X and float32 Y, the return matrix is also always in float64 and this is what should be expected imo.

The troubles come when both X and Y are float32.
- for sklearn metrics:
  - `euclidean` and `cosine`: result is always float32
  - `manhattan`: result is float64 if n_jobs=1 and float32 otherwise
- for scipy metrics: result is float64 if n_jobs=1 and float32 otherwise
  Note that scipy cdist/pdist always returns float64.

Hence the question: should `pairwise_distances` preserve float32 ?

My opinion is that it should since `pairwise_distances` can be used as an intermediate step during fit and since there's ongoing work towards preserving float32 in estimators (see https://github.com/scikit-learn/scikit-learn/issues/11000 for transfromers for instance).

An argument against that could be reducing the numerical instabilities. A potential solution could be to use float64 accumulators for the intermediate computations only, still returning a float32 dist matrix. Note that with https://github.com/scikit-learn/scikit-learn/pull/23958 we might not need to use the scipy metrics anymore, in favor of the ones defined in `dist_metrics`, and using float64 accumulators would be easier to implement generally.

Answering this question will help to not go in the wrong direction in https://github.com/scikit-learn/scikit-learn/pull/23958

COMMENT:
If possible, I think if `X` and `Y` are both float32 then the return should be float32.

>  A potential solution could be to use float64 accumulators for the intermediate computations only, still returning a float32 dist matrix.

For me, using a float64 accumulator is a requirement if we want to preserve float32.

Labels: RFC, module:metrics
RFC Should pairwise_distances preserve float32 ?

Currently the dtype of the distance matrix returned by `pairwise_distances` is not very consistent, depending on the metric and on the value of n_jobs.

For float64 input, everything is consistent: the returned matrix is always in float64.
For mixed float64 X and float32 Y, the return matrix is also always in float64 and this is what should be expected imo.

The troubles come when both X and Y are float32.
- for sklearn metrics:
  - `euclidean` and `cosine`: result is always float32
  - `manhattan`: result is float64 if n_jobs=1 and float32 otherwise
- for scipy metrics: result is float64 if n_jobs=1 and float32 otherwise
  Note that scipy cdist/pdist always returns float64.

Hence the question: should `pairwise_distances` preserve float32 ?

My opinion is that it should since `pairwise_distances` can be used as an intermediate step during fit and since there's ongoing work towards preserving float32 in estimators (see https://github.com/scikit-learn/scikit-learn/issues/11000 for transfromers for instance).

An argument against that could be reducing the numerical instabilities. A potential solution could be to use float64 accumulators for the intermediate computations only, still returning a float32 dist matrix. Note that with https://github.com/scikit-learn/scikit-learn/pull/23958 we might not need to use the scipy metrics anymore, in favor of the ones defined in `dist_metrics`, and using float64 accumulators would be easier to implement generally.

Answering this question will help to not go in the wrong direction in https://github.com/scikit-learn/scikit-learn/pull/23958

COMMENT:
I agree with @thomasjpfan.

Labels: RFC, module:metrics
RFC Should pairwise_distances preserve float32 ?

Currently the dtype of the distance matrix returned by `pairwise_distances` is not very consistent, depending on the metric and on the value of n_jobs.

For float64 input, everything is consistent: the returned matrix is always in float64.
For mixed float64 X and float32 Y, the return matrix is also always in float64 and this is what should be expected imo.

The troubles come when both X and Y are float32.
- for sklearn metrics:
  - `euclidean` and `cosine`: result is always float32
  - `manhattan`: result is float64 if n_jobs=1 and float32 otherwise
- for scipy metrics: result is float64 if n_jobs=1 and float32 otherwise
  Note that scipy cdist/pdist always returns float64.

Hence the question: should `pairwise_distances` preserve float32 ?

My opinion is that it should since `pairwise_distances` can be used as an intermediate step during fit and since there's ongoing work towards preserving float32 in estimators (see https://github.com/scikit-learn/scikit-learn/issues/11000 for transfromers for instance).

An argument against that could be reducing the numerical instabilities. A potential solution could be to use float64 accumulators for the intermediate computations only, still returning a float32 dist matrix. Note that with https://github.com/scikit-learn/scikit-learn/pull/23958 we might not need to use the scipy metrics anymore, in favor of the ones defined in `dist_metrics`, and using float64 accumulators would be easier to implement generally.

Answering this question will help to not go in the wrong direction in https://github.com/scikit-learn/scikit-learn/pull/23958

COMMENT:
> If possible, I think if X and Y are both float32 then the return should be float32.

+1 as well. I have the feeling that internal `float64` accumulator is unlikely to hold more meaningfully precise information than its float32-downcast version once the computation has of the distances has finished. But maybe I am wrong, if so I would be happy to see a counter example and I would update my vote accordingly.

Labels: RFC, module:metrics
RFC Should pairwise_distances preserve float32 ?

Currently the dtype of the distance matrix returned by `pairwise_distances` is not very consistent, depending on the metric and on the value of n_jobs.

For float64 input, everything is consistent: the returned matrix is always in float64.
For mixed float64 X and float32 Y, the return matrix is also always in float64 and this is what should be expected imo.

The troubles come when both X and Y are float32.
- for sklearn metrics:
  - `euclidean` and `cosine`: result is always float32
  - `manhattan`: result is float64 if n_jobs=1 and float32 otherwise
- for scipy metrics: result is float64 if n_jobs=1 and float32 otherwise
  Note that scipy cdist/pdist always returns float64.

Hence the question: should `pairwise_distances` preserve float32 ?

My opinion is that it should since `pairwise_distances` can be used as an intermediate step during fit and since there's ongoing work towards preserving float32 in estimators (see https://github.com/scikit-learn/scikit-learn/issues/11000 for transfromers for instance).

An argument against that could be reducing the numerical instabilities. A potential solution could be to use float64 accumulators for the intermediate computations only, still returning a float32 dist matrix. Note that with https://github.com/scikit-learn/scikit-learn/pull/23958 we might not need to use the scipy metrics anymore, in favor of the ones defined in `dist_metrics`, and using float64 accumulators would be easier to implement generally.

Answering this question will help to not go in the wrong direction in https://github.com/scikit-learn/scikit-learn/pull/23958

COMMENT:
So I guess everyone agrees on dtypes' preservation.

#23958 has been updated accordingly and would treat most cases once merged.

Labels: RFC, module:metrics