Talip7/scikit-learn-issues-multilabel huggingface dataset:
RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar?

While working on the representation imposed by NEP51, I found out that we recently made the `accuracy_score` to return a Python scalar while, up-to-now, other metric are returning NumPy scalar.

This change was made due to the array API work:

https://github.com/scikit-learn/scikit-learn/blob/b0da1b7706054f0b78f0a0582a9362a188e1fa38/sklearn/utils/_array_api.py#L448-L454

I assume that we are getting to an intersection where we should make the output of our metrics consistent but also foresee potential requirements: as the comment indicate, calling `float()` will be a sync point but it might not be the best strategy for lazy computation.

This RFC is a placeholder to discuss what strategy we should be implementing.

COMMENT:
@ogrisel it seems some of your comment is misformatted or missing words? Could you take a look?

Labels: help wanted, RFC
RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar?

While working on the representation imposed by NEP51, I found out that we recently made the `accuracy_score` to return a Python scalar while, up-to-now, other metric are returning NumPy scalar.

This change was made due to the array API work:

https://github.com/scikit-learn/scikit-learn/blob/b0da1b7706054f0b78f0a0582a9362a188e1fa38/sklearn/utils/_array_api.py#L448-L454

I assume that we are getting to an intersection where we should make the output of our metrics consistent but also foresee potential requirements: as the comment indicate, calling `float()` will be a sync point but it might not be the best strategy for lazy computation.

This RFC is a placeholder to discuss what strategy we should be implementing.

COMMENT:
@betatim I edited my comment.

Labels: help wanted, RFC
RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar?

While working on the representation imposed by NEP51, I found out that we recently made the `accuracy_score` to return a Python scalar while, up-to-now, other metric are returning NumPy scalar.

This change was made due to the array API work:

https://github.com/scikit-learn/scikit-learn/blob/b0da1b7706054f0b78f0a0582a9362a188e1fa38/sklearn/utils/_array_api.py#L448-L454

I assume that we are getting to an intersection where we should make the output of our metrics consistent but also foresee potential requirements: as the comment indicate, calling `float()` will be a sync point but it might not be the best strategy for lazy computation.

This RFC is a placeholder to discuss what strategy we should be implementing.

COMMENT:
Thanks a lot!

I think my least favourite option is (c). It feels like we are delegating the complexity to our users, who probably have even less clue about "the right thing to do" than we have.

I like (b) and think that it isn't a big deal that something like `Estimator.score` can't be lazy. Trying to make it lazy seems to add a lot of complexity (you need a `LazyScalar` and explain to users) and I can't think of a use-case where making the computation lazy would allow you to make the computation more efficient. Compared to for example reading data from a file, where you can make the reading more efficient by knowing that certain columns are never used.

I also wonder if we need to specify that the type of the return value of `Estimator.score` is a Python scalar and not rely on duck typing (so it can be a Python scalar or a Numpy scalar).

Labels: help wanted, RFC
RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar?

While working on the representation imposed by NEP51, I found out that we recently made the `accuracy_score` to return a Python scalar while, up-to-now, other metric are returning NumPy scalar.

This change was made due to the array API work:

https://github.com/scikit-learn/scikit-learn/blob/b0da1b7706054f0b78f0a0582a9362a188e1fa38/sklearn/utils/_array_api.py#L448-L454

I assume that we are getting to an intersection where we should make the output of our metrics consistent but also foresee potential requirements: as the comment indicate, calling `float()` will be a sync point but it might not be the best strategy for lazy computation.

This RFC is a placeholder to discuss what strategy we should be implementing.

COMMENT:
I also like (b) for the same reasons as @betatim mentions

Labels: help wanted, RFC
RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar?

While working on the representation imposed by NEP51, I found out that we recently made the `accuracy_score` to return a Python scalar while, up-to-now, other metric are returning NumPy scalar.

This change was made due to the array API work:

https://github.com/scikit-learn/scikit-learn/blob/b0da1b7706054f0b78f0a0582a9362a188e1fa38/sklearn/utils/_array_api.py#L448-L454

I assume that we are getting to an intersection where we should make the output of our metrics consistent but also foresee potential requirements: as the comment indicate, calling `float()` will be a sync point but it might not be the best strategy for lazy computation.

This RFC is a placeholder to discuss what strategy we should be implementing.

COMMENT:
There seems to be some amount of consensus and no new comments for a while. Should we try to wrap this up?

@glemaitre what do you think of the options Oliver listed? Do you also like option (b)?

Labels: help wanted, RFC
RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar?

While working on the representation imposed by NEP51, I found out that we recently made the `accuracy_score` to return a Python scalar while, up-to-now, other metric are returning NumPy scalar.

This change was made due to the array API work:

https://github.com/scikit-learn/scikit-learn/blob/b0da1b7706054f0b78f0a0582a9362a188e1fa38/sklearn/utils/_array_api.py#L448-L454

I assume that we are getting to an intersection where we should make the output of our metrics consistent but also foresee potential requirements: as the comment indicate, calling `float()` will be a sync point but it might not be the best strategy for lazy computation.

This RFC is a placeholder to discuss what strategy we should be implementing.

COMMENT:
Thanks @betatim to keep track of this issue. Option (b) looks the right trade-off right now.
So we can therefore settle on converting output to Python scalar.

Labels: help wanted, RFC
RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar?

While working on the representation imposed by NEP51, I found out that we recently made the `accuracy_score` to return a Python scalar while, up-to-now, other metric are returning NumPy scalar.

This change was made due to the array API work:

https://github.com/scikit-learn/scikit-learn/blob/b0da1b7706054f0b78f0a0582a9362a188e1fa38/sklearn/utils/_array_api.py#L448-L454

I assume that we are getting to an intersection where we should make the output of our metrics consistent but also foresee potential requirements: as the comment indicate, calling `float()` will be a sync point but it might not be the best strategy for lazy computation.

This RFC is a placeholder to discuss what strategy we should be implementing.

COMMENT:
/take

Labels: help wanted, RFC
RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar?

While working on the representation imposed by NEP51, I found out that we recently made the `accuracy_score` to return a Python scalar while, up-to-now, other metric are returning NumPy scalar.

This change was made due to the array API work:

https://github.com/scikit-learn/scikit-learn/blob/b0da1b7706054f0b78f0a0582a9362a188e1fa38/sklearn/utils/_array_api.py#L448-L454

I assume that we are getting to an intersection where we should make the output of our metrics consistent but also foresee potential requirements: as the comment indicate, calling `float()` will be a sync point but it might not be the best strategy for lazy computation.

This RFC is a placeholder to discuss what strategy we should be implementing.

COMMENT:
@ckosten this is not a good issue to begin with. You can look for good first issues and help wanted tags to find one.

Labels: help wanted, RFC
RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar?

While working on the representation imposed by NEP51, I found out that we recently made the `accuracy_score` to return a Python scalar while, up-to-now, other metric are returning NumPy scalar.

This change was made due to the array API work:

https://github.com/scikit-learn/scikit-learn/blob/b0da1b7706054f0b78f0a0582a9362a188e1fa38/sklearn/utils/_array_api.py#L448-L454

I assume that we are getting to an intersection where we should make the output of our metrics consistent but also foresee potential requirements: as the comment indicate, calling `float()` will be a sync point but it might not be the best strategy for lazy computation.

This RFC is a placeholder to discuss what strategy we should be implementing.

COMMENT:
It was assigned as a beginner problem in a scikit learn workshop recently...

> @ckosten this is not a good issue to begin with. You can look for good first issues and help wanted tags to find one.

Labels: help wanted, RFC
RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar?

While working on the representation imposed by NEP51, I found out that we recently made the `accuracy_score` to return a Python scalar while, up-to-now, other metric are returning NumPy scalar.

This change was made due to the array API work:

https://github.com/scikit-learn/scikit-learn/blob/b0da1b7706054f0b78f0a0582a9362a188e1fa38/sklearn/utils/_array_api.py#L448-L454

I assume that we are getting to an intersection where we should make the output of our metrics consistent but also foresee potential requirements: as the comment indicate, calling `float()` will be a sync point but it might not be the best strategy for lazy computation.

This RFC is a placeholder to discuss what strategy we should be implementing.

COMMENT:
Shall we close this issue and open matching meta-issue to track the remaining work to do?

It might be redundant with the array API meta issue at #26024 which is already well under way and since most scalar-returning metric function will likely such a treatment to have their tests pass on PyTorch with CUDA.

Maybe we can leave it open for now and close it once all the metric function referenced in #26024 have been addressed.

Labels: help wanted, RFC
RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar?

While working on the representation imposed by NEP51, I found out that we recently made the `accuracy_score` to return a Python scalar while, up-to-now, other metric are returning NumPy scalar.

This change was made due to the array API work:

https://github.com/scikit-learn/scikit-learn/blob/b0da1b7706054f0b78f0a0582a9362a188e1fa38/sklearn/utils/_array_api.py#L448-L454

I assume that we are getting to an intersection where we should make the output of our metrics consistent but also foresee potential requirements: as the comment indicate, calling `float()` will be a sync point but it might not be the best strategy for lazy computation.

This RFC is a placeholder to discuss what strategy we should be implementing.

COMMENT:
>  will likely such a treatment to have their tests pass on PyTorch with CUDA.

To be clear, you mean for them to return a scalar? Then yeah I'm happy to have this closed.

Labels: help wanted, RFC
Feature Request: Out-of-Vocabulary (OOV) Token Handling in CountVectorizer

### Describe the workflow you want to enable

Users of CountVectorizer frequently encounter situations where, during the transformation of test data or new unseen data, there are words that the vectorizer hasn't seen during the training (fit) phase. In such situations, the default behavior is to ignore these words. However, for many NLP applications (like sentiment analysis, named entity recognition, etc.), having a representation for these OOV words is crucial. Hence, an inbuilt mechanism to handle OOV words by representing them with a special token would be beneficial.

### Describe your proposed solution

1. Additional Parameter: Introduce a new optional parameter in CountVectorizer called oov_token which defaults to None.

When oov_token is None, the vectorizer behaves as it currently does.
When oov_token is provided (e.g., <OOV>), any word encountered during the transform phase that is not in the learned vocabulary gets represented by this token.
2. Vocabulary Inclusion: The oov_token, if provided, should be part of the vocabulary, and its position/index in the vocabulary should be consistent.

3. OOV Token Counting: The count of the OOV token should reflect the total count of all OOV words in a document.

### Describe alternatives you've considered, if relevant

Custom Vectorizer: Users currently have to extend the CountVectorizer class to handle OOV, which adds extra overhead. An inbuilt feature would be more efficient and user-friendly.


### Additional context

_No response_

COMMENT:
This doesn't sound unreasonable to me. Very similar to `handle_unknown` from `OneHotEncoder`: https://scikit-learn.org/dev/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder

Labels: New Feature
Feature Request: Out-of-Vocabulary (OOV) Token Handling in CountVectorizer

### Describe the workflow you want to enable

Users of CountVectorizer frequently encounter situations where, during the transformation of test data or new unseen data, there are words that the vectorizer hasn't seen during the training (fit) phase. In such situations, the default behavior is to ignore these words. However, for many NLP applications (like sentiment analysis, named entity recognition, etc.), having a representation for these OOV words is crucial. Hence, an inbuilt mechanism to handle OOV words by representing them with a special token would be beneficial.

### Describe your proposed solution

1. Additional Parameter: Introduce a new optional parameter in CountVectorizer called oov_token which defaults to None.

When oov_token is None, the vectorizer behaves as it currently does.
When oov_token is provided (e.g., <OOV>), any word encountered during the transform phase that is not in the learned vocabulary gets represented by this token.
2. Vocabulary Inclusion: The oov_token, if provided, should be part of the vocabulary, and its position/index in the vocabulary should be consistent.

3. OOV Token Counting: The count of the OOV token should reflect the total count of all OOV words in a document.

### Describe alternatives you've considered, if relevant

Custom Vectorizer: Users currently have to extend the CountVectorizer class to handle OOV, which adds extra overhead. An inbuilt feature would be more efficient and user-friendly.


### Additional context

_No response_

COMMENT:
Would love to take this up if it's a feature that's useful!

Labels: New Feature
Feature Request: Out-of-Vocabulary (OOV) Token Handling in CountVectorizer

### Describe the workflow you want to enable

Users of CountVectorizer frequently encounter situations where, during the transformation of test data or new unseen data, there are words that the vectorizer hasn't seen during the training (fit) phase. In such situations, the default behavior is to ignore these words. However, for many NLP applications (like sentiment analysis, named entity recognition, etc.), having a representation for these OOV words is crucial. Hence, an inbuilt mechanism to handle OOV words by representing them with a special token would be beneficial.

### Describe your proposed solution

1. Additional Parameter: Introduce a new optional parameter in CountVectorizer called oov_token which defaults to None.

When oov_token is None, the vectorizer behaves as it currently does.
When oov_token is provided (e.g., <OOV>), any word encountered during the transform phase that is not in the learned vocabulary gets represented by this token.
2. Vocabulary Inclusion: The oov_token, if provided, should be part of the vocabulary, and its position/index in the vocabulary should be consistent.

3. OOV Token Counting: The count of the OOV token should reflect the total count of all OOV words in a document.

### Describe alternatives you've considered, if relevant

Custom Vectorizer: Users currently have to extend the CountVectorizer class to handle OOV, which adds extra overhead. An inbuilt feature would be more efficient and user-friendly.


### Additional context

_No response_

COMMENT:
I think you could go ahead and have a PR. Please take a look at the details of `handle_unknown` from OHE, probably some code can be refactored to be reused.

Labels: New Feature
Feature Request: Out-of-Vocabulary (OOV) Token Handling in CountVectorizer

### Describe the workflow you want to enable

Users of CountVectorizer frequently encounter situations where, during the transformation of test data or new unseen data, there are words that the vectorizer hasn't seen during the training (fit) phase. In such situations, the default behavior is to ignore these words. However, for many NLP applications (like sentiment analysis, named entity recognition, etc.), having a representation for these OOV words is crucial. Hence, an inbuilt mechanism to handle OOV words by representing them with a special token would be beneficial.

### Describe your proposed solution

1. Additional Parameter: Introduce a new optional parameter in CountVectorizer called oov_token which defaults to None.

When oov_token is None, the vectorizer behaves as it currently does.
When oov_token is provided (e.g., <OOV>), any word encountered during the transform phase that is not in the learned vocabulary gets represented by this token.
2. Vocabulary Inclusion: The oov_token, if provided, should be part of the vocabulary, and its position/index in the vocabulary should be consistent.

3. OOV Token Counting: The count of the OOV token should reflect the total count of all OOV words in a document.

### Describe alternatives you've considered, if relevant

Custom Vectorizer: Users currently have to extend the CountVectorizer class to handle OOV, which adds extra overhead. An inbuilt feature would be more efficient and user-friendly.


### Additional context

_No response_

COMMENT:
+1 for having a new constructor parameter `handle_unkown="ignore"` by default and with an option to use a dedicated column. We need to find a good parameter name and column naming strategy but we can discuss that in a PR.

Labels: New Feature
Feature Request: Out-of-Vocabulary (OOV) Token Handling in CountVectorizer

### Describe the workflow you want to enable

Users of CountVectorizer frequently encounter situations where, during the transformation of test data or new unseen data, there are words that the vectorizer hasn't seen during the training (fit) phase. In such situations, the default behavior is to ignore these words. However, for many NLP applications (like sentiment analysis, named entity recognition, etc.), having a representation for these OOV words is crucial. Hence, an inbuilt mechanism to handle OOV words by representing them with a special token would be beneficial.

### Describe your proposed solution

1. Additional Parameter: Introduce a new optional parameter in CountVectorizer called oov_token which defaults to None.

When oov_token is None, the vectorizer behaves as it currently does.
When oov_token is provided (e.g., <OOV>), any word encountered during the transform phase that is not in the learned vocabulary gets represented by this token.
2. Vocabulary Inclusion: The oov_token, if provided, should be part of the vocabulary, and its position/index in the vocabulary should be consistent.

3. OOV Token Counting: The count of the OOV token should reflect the total count of all OOV words in a document.

### Describe alternatives you've considered, if relevant

Custom Vectorizer: Users currently have to extend the CountVectorizer class to handle OOV, which adds extra overhead. An inbuilt feature would be more efficient and user-friendly.


### Additional context

_No response_

COMMENT:
I am not sure if anyone is working on this. I was looking at the code because this seemed like a useful feature. But I think the tricky part is to not characterize a stop_word as an "out of vocabulary" token.

It looks like we cannot rely on having the stop word set (I see this comment in the documentation):
"
Notes
    -----
    The ``stop_words_`` attribute can get large and increase the model size
    when pickling. This attribute is provided only for introspection and can
    be safely removed using delattr or set to None before pickling.
"

I am assuming stop words will not be a part of the vocabulary built during "fit". If we don't have access to the "stop_words_" set at the time of transform, we will incorrectly count them as "out of vocabulary".

Labels: New Feature
Convergence Warning message!!!

The optimization algorithm is not converging to a solution within the specified number of iterations.
Any Solution?

COMMENT:
Can you please include the full warning message you get and ideally a minimal reproducer?

https://scikit-learn.org/dev/developers/minimal_reproducer.html

In general, the solutions to this kind of problem include some of the following:

- increasing the number of iterations (setting `max_iter=1000` or more),
- preprocessing the input feature to make sure that they all lie on comparable scales of variation (see: https://scikit-learn.org/stable/modules/preprocessing.html),
- adding more regularization to the optimization problem (e.g. decreasing `C` for a `LogisticRegression` model or increasing `alpha` for linear regression models).
- trying a different solver.

Labels: Needs Triage
1 threads instead of multi-threading

### Describe the bug

Hello, I have a question regarding the following code: why is it running with only one thread instead of utilizing multiple threads? Thank you for your assistance.

X shape :(90, 16384) Y shape:  (90, 261880)
R=RidgeCV(alphas=[0.1, 1, 100], gcv_mode='svd').fit(X, Y)

the output of `pprint(threadpool_info())` is 

```python
[{'filepath': '/cvmfs/[../gentoo/2020/usr/lib64/gcc/x86_64-pc-linux-gnu/11.3.0/libgomp.so.1.0.0](http://../gentoo/2020/usr/lib64/gcc/x86_64-pc-linux-gnu/11.3.0/libgomp.so.1.0.0)',
  'internal_api': 'openmp',
  'num_threads': 1,
  'prefix': 'libgomp',
  'user_api': 'openmp',
  'version': None},
 {'architecture': 'zen2',
  'filepath': '/cvmfs/[../easybuild/software/2020/avx2/Core/bliscore/0.8.1/lib/libblis.so.3.0.1](http://../easybuild/software/2020/avx2/Core/bliscore/0.8.1/lib/libblis.so.3.0.1)',
  'internal_api': 'blis',
  'num_threads': 1,
  'prefix': 'libblis',
  'threading_layer': 'openmp',
  'user_api': 'blas',
  'version': '0.8.1'}]
```
### Steps/Code to Reproduce

X shape :(90, 16384) Y shape:  (90, 261880)
R=RidgeCV(alphas=[0.1, 1, 100], gcv_mode='svd').fit(X, Y)

then pprint(threadpool_info()) .


### Expected Results

execution through multi-threading

### Actual Results

utilizing only one thread

### Versions

```shell
sklearn version 1.3.0
python 3.10
```

COMMENT:
Hi @Sana3883, it seems that:
 - you are using shared library of OpenMP and OpenBLAS which are different from the usual ones (e.g. the ones from their associated packages on conda-forge or the ones vendored). Are you building them yourself on your machine? In that case, you might need to activate multi-threading support explicitly via their respective build systems.
 - the process are constrained to only use one thread, making the output of `threadpool_info` showing that only one thread is used.
 - are you using Gentoo which specific USE flags?

Labels: Bug