Talip7/scikit-learn-issues-multilabel huggingface dataset:
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
No idea, could be. But those were both added 7 years ago according to `git blame` so maybe it's a new Sphinx version thing. Also FWIW it looks like they're both in there in separate entries:
```
larsoner:~/python/expyfun$ python -m intersphinx_registry.lookup sklearn | grep "    Y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    Y                                                                                : glossary.html#term-Y
larsoner:~/python/expyfun$ python -m intersphinx_registry.lookup sklearn | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
maybe sphinx changed their check to be case-insensitive or something?

Labels: Needs Triage
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
Would be worth opening an issue on the intersphinx side maybe?

Labels: Needs Triage
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
Actually it looks like this has been fixed in 1.5.1 released yesterday (3 July)?

I don't get any warnings when running this locally:
```
python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
```
Sphinx version:
```
❯ sphinx-build --version
sphinx-build 7.2.6
```

Labels: Needs Triage
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
Agreed, seems to have been magically fixed! 🪄

Labels: Needs Triage
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
This issue seems to be back:

```
$ pip list | grep  Sphinx
Sphinx                        7.4.2
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```

Labels: Needs Triage
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
@maxnoe would you be able to create an issue on the sphinx side maybe? This seems an issue there.

Labels: Needs Triage
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
Done

Labels: Needs Triage
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
As I commented in https://github.com/sphinx-doc/sphinx/issues/12585#issuecomment-2228415308 the warning has started to happen with sphinx 7.4.0 (this is why I was not getting the warning in https://github.com/scikit-learn/scikit-learn/issues/29337#issuecomment-2209049572), let's continue the discussion in https://github.com/sphinx-doc/sphinx/issues/12585.

Labels: Needs Triage
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
> Thanks for reporting this. Could this be because [`main`/doc/glossary.rst](https://github.com/scikit-learn/scikit-learn/blob/main/doc/glossary.rst?rgh-link-date=2024-06-24T08%3A24%3A51Z) defines an entry for both `y` and `Y`?

This is right -- in this case it isn't an issue as both point to the same target (glossary definition), but e.g. if the definitions were in different pages or separated in some other way, `intersphinx` wouldn't know which to use (because resolution is done in a case-insensitive fashion).

A

Labels: Needs Triage
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
Note that at least in scikit-learn, the case matters sometimes. `y` and `Y` don't have to mean the same thing, and it's possible that we'd have a case sensitive target.

Labels: Needs Triage
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
I've a potential change open at https://github.com/sphinx-doc/sphinx/pull/12586 that would filter these warnings _provided that the item being pointed to is the same_ -- so the current warning about `y` and `Y` would disappear.. but if they are modified to point to distinct targets, the warning would reappear (because the Intersphinx v2 inventory format does not support case-sensitive matches).

I realize that could present some friction in cases where there _is_ a desire to use names that vary only by character-casing; but I think we should surface the ambiguity.

Labels: Needs Triage
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
Thanks a lot @jayaddison for the fix and release, I confirm that sphinx 7.4.4 fixes the warning about scikit-learn, i.e the following does not show any warning:

```
python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
```

I think this is good enough to close this issue.

Out of curiosity, I looked a bit at the [sphinx glossary doc](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary) but I could not figure out if there is a way to have an alias for a glossary entry, i.e. have `Y` as the glossary entry but be able to refer to it by:
```
term:`ycapital`
```

Labels: Needs Triage
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
> Thanks a lot @jayaddison for the fix and release, I confirm that sphinx 7.4.4 fixes the warning about scikit-learn, i.e the following does not show any warning:

Thank you @lesteve for the feedback, and I'm glad to hear that the noise from this is reduced :+1: 
 
> Out of curiosity, I looked a bit at the [sphinx glossary doc](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary) but I could not figure out if there is a way to have an alias for a glossary entry, i.e. have `Y` as the glossary entry but be able to refer to it by:
> 
> ```
> term:`ycapital`
> ```

This might require some code archaeology, and I can't promise to get around to it, but I'll try to figure out some more of what is possible here.  If you have time, a minimal example (a single `.rst` file or two perhaps) to send across to the Sphinx bugtracker could be super helpful - I'll see what I can figure out either way though.

Labels: Needs Triage
BUG: Stable docs intersphinx has a duplicate std:term:y

When building with intersphinx we get a warning which you can replicate with `intersphinx` directly
```
$ python -m sphinx.ext.intersphinx https://scikit-learn.org/stable/objects.inv | grep "    y "
WARNING:sphinx.sphinx.util.inventory:inventory <> contains multiple definitions for std:term:y
    y                                                                                : glossary.html#term-y
```
Only one term shows up in the `grep` presumably because intersphinx chooses one of the two, but the `WARNING` line actually shows up in doc builds.

COMMENT:
Thanks @jayaddison, my question was mostly out of curiosity and at this time I feel this is not worth opening a sphinx issue about it because this is a YAGNI. If the use case really happened in scikit-learn and somehow `y` and `Y` are moved to separate glossary entries, we will reconsider.

As a summary of the previous comments:
- `y` and `Y` are the same entry in the scikit-learn glossary so we can live with the limitation that `objects.inv` is case insensitive
- sphinx 7.4.4 remove the puzzling warning for scikit-learn `objects.inv` consumers

Labels: Needs Triage
Discussion: citation file

### Describe the issue linked to the documentation

Any thoughts to adding a citation.cff file to this repo?

Here is an example from Seaborn:
https://github.com/mwaskom/seaborn/blob/master/CITATION.cff

This example was shared in NASA Open Science training. 

### GitHub
https://citation-file-format.github.io/

### Suggest a potential alternative/fix

COMMENT:
Don't mind. But it'd be really nice to write a new paper for scikit-learn.

Labels: Easy, Documentation, help wanted
Discussion: citation file

### Describe the issue linked to the documentation

Any thoughts to adding a citation.cff file to this repo?

Here is an example from Seaborn:
https://github.com/mwaskom/seaborn/blob/master/CITATION.cff

This example was shared in NASA Open Science training. 

### GitHub
https://citation-file-format.github.io/

### Suggest a potential alternative/fix

COMMENT:
cc @scikit-learn/core-devs

Labels: Easy, Documentation, help wanted
Discussion: citation file

### Describe the issue linked to the documentation

Any thoughts to adding a citation.cff file to this repo?

Here is an example from Seaborn:
https://github.com/mwaskom/seaborn/blob/master/CITATION.cff

This example was shared in NASA Open Science training. 

### GitHub
https://citation-file-format.github.io/

### Suggest a potential alternative/fix

COMMENT:
+1 for having one with the historical paper.

I am not against participating to writing another one. :slightly_smiling_face:

Labels: Easy, Documentation, help wanted
Discussion: citation file

### Describe the issue linked to the documentation

Any thoughts to adding a citation.cff file to this repo?

Here is an example from Seaborn:
https://github.com/mwaskom/seaborn/blob/master/CITATION.cff

This example was shared in NASA Open Science training. 

### GitHub
https://citation-file-format.github.io/

### Suggest a potential alternative/fix

COMMENT:
No strong opinion on the matter.

Labels: Easy, Documentation, help wanted
Discussion: citation file

### Describe the issue linked to the documentation

Any thoughts to adding a citation.cff file to this repo?

Here is an example from Seaborn:
https://github.com/mwaskom/seaborn/blob/master/CITATION.cff

This example was shared in NASA Open Science training. 

### GitHub
https://citation-file-format.github.io/

### Suggest a potential alternative/fix

COMMENT:
First things first, someone with enough motivation and time should open a PR to add `CITATION.cff` with the scikit-learn paper. All the info is [here](https://scikit-learn.org/stable/about.html#citing-scikit-learn) this should not be that hard to figure out how to translate that in the `CITATION.cff` format

Labels: Easy, Documentation, help wanted
Discussion: citation file

### Describe the issue linked to the documentation

Any thoughts to adding a citation.cff file to this repo?

Here is an example from Seaborn:
https://github.com/mwaskom/seaborn/blob/master/CITATION.cff

This example was shared in NASA Open Science training. 

### GitHub
https://citation-file-format.github.io/

### Suggest a potential alternative/fix

COMMENT:
Hello, I had time to work on this and created a pull request :)

The PR would give an option to cite this repository within GitHub, this is the generated BibTeX example:

@article{Pedregosa_Scikit-learn_Machine_Learning_2011,
author = {Pedregosa, Fabian and Varoquaux, Gaël and Gramfort, Alexandre and Michel, Vincent and Thirion, Bertrand and Grisel, Olivier and Blondel, Mathieu and Prettenhofer, Peter and Weiss, Ron and Dubourg, Vincent and Vanderplas, Jake and Passos, Alexandre and Cournapeau, David and Brucher, Matthieu and Perrot, Matthieu and Duchesnay, Édouard},
journal = {Journal of Machine Learning Research},
pages = {2825--2830},
title = {{Scikit-learn: Machine Learning in Python}},
url = {https://jmlr.csail.mit.edu/papers/v12/pedregosa11a.html},
volume = {12},
year = {2011}
}

Labels: Easy, Documentation, help wanted
⚠️ CI failed on Wheel builder (last failure: Jun 21, 2024) ⚠️

**CI failed on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/9607956232)** (Jun 21, 2024)

COMMENT:
"Carton plein" as we say in French :wink:
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/b446e90c-f312-4bd6-8f81-58daf8e0ef97)

Same thing as https://github.com/scikit-learn/scikit-learn/issues/29325#issuecomment-2181967547 where I put more details.

Labels: Needs Triage
improve performance of cdist/pdist pairwise_distance for boolean matrices

### Describe the workflow you want to enable

If we have a boolean matrix, we can pack bits, use XOR and then `np.bitwise_count` (in numpy 2.0) to compute distances much faster than current `cdist` or `pdist` or `pairwise_distance`. It would be awesome to accelerate these functions.

### Describe your proposed solution

import numpy as np
import time
from sklearn.metrics import pairwise_distances
from scipy.spatial.distance import pdist, cdist 

qq1 = (np.random.rand(230099, 4097) > 0.5)
mm1 = (np.random.rand(810, 4097) > 0.5)

t1 = time.time()
d2a = cdist(qq1, mm1, metric='sqeuclidean')
print(f"{time.time() - t1}")
# returns 186.7363

t1 = time.time()
d2aa = cdist(qq1, mm1, metric='cityblock')
print(f"{time.time() - t1}")
# returns 335s

def fast_binary_dist(X, Y):
    out = np.zeros([X.shape[0], Y.shape[0]])
    for j in range(Y.shape[0]):
        row = Y.take(j, axis=0)
        out[:, j] = np.sum(np.bitwise_count(X ^ row), axis=1, dtype=int)
    return out

t1 = time.time()
qq1p = np.packbits(qq1, axis=1)
mm1p = np.packbits(mm1, axis=1)
d2c = fast_binary_dist(qq1p, mm1p)
print(f"{time.time() - t1}")
# returns 43.1 seconds

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi @vasilyev-mit, since we rely on scipy's `cdist`and `pdist` for all metrics but `euclidean`, `manhattan` and `cosine`, I guess that it would be more appropriate to open dedicated issue on the scipy repo.

Since the euclidean metric is already well optimized in scikit-learn, I don't think the specialized version for booleans would be beneficial. The following looks a lot faster on my machine (and better takes advantage of multi-cores).
```py
d = pairwise_distances(qq1, mm1, metric="euclidean")
d ** 2
```

Labels: Performance
improve performance of cdist/pdist pairwise_distance for boolean matrices

### Describe the workflow you want to enable

If we have a boolean matrix, we can pack bits, use XOR and then `np.bitwise_count` (in numpy 2.0) to compute distances much faster than current `cdist` or `pdist` or `pairwise_distance`. It would be awesome to accelerate these functions.

### Describe your proposed solution

import numpy as np
import time
from sklearn.metrics import pairwise_distances
from scipy.spatial.distance import pdist, cdist 

qq1 = (np.random.rand(230099, 4097) > 0.5)
mm1 = (np.random.rand(810, 4097) > 0.5)

t1 = time.time()
d2a = cdist(qq1, mm1, metric='sqeuclidean')
print(f"{time.time() - t1}")
# returns 186.7363

t1 = time.time()
d2aa = cdist(qq1, mm1, metric='cityblock')
print(f"{time.time() - t1}")
# returns 335s

def fast_binary_dist(X, Y):
    out = np.zeros([X.shape[0], Y.shape[0]])
    for j in range(Y.shape[0]):
        row = Y.take(j, axis=0)
        out[:, j] = np.sum(np.bitwise_count(X ^ row), axis=1, dtype=int)
    return out

t1 = time.time()
qq1p = np.packbits(qq1, axis=1)
mm1p = np.packbits(mm1, axis=1)
d2c = fast_binary_dist(qq1p, mm1p)
print(f"{time.time() - t1}")
# returns 43.1 seconds

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@jeremiedbb - thank you very much for clarification!! I had no idea about the performance of the "euclidean" metric being faster, in fact, than "sqeuclidean"! Is there an explanation to this? 

On my machine, your code runs in amazing 6 seconds! 
However, if I use `metric="sqeuclidean`, it runs in 186 seconds. What is happening?

Labels: Performance