Talip7/scikit-learn-issues-multilabel huggingface dataset:
setuptools >= 61 package discovery issue with python setup.py clean

Originally reported in https://github.com/scikit-learn/scikit-learn/discussions/23413

This seems to be a change in setuptools 61 about package discovery ... probably a related issue is https://github.com/pypa/setuptools/issues/3197

```
mamba create -n test -c conda-forge python=3.9 numpy scipy cython -y
conda activate test
python setup.py clean
```

```
error: Multiple top-level packages discovered in a flat-layout: ['prof', 'sklearn', 'maint_tools', 'build_tools', 'asv_benchmarks', 'cache_directory'].

To avoid accidental inclusion of unwanted files or directories,
setuptools will not proceed with this build.

If you are trying to create a single distribution with multiple packages
on purpose, you should not rely on automatic discovery.
Instead, consider the following options:

1. set up custom discovery (`find` directive with `include` or `exclude`)
2. use a `src-layout`
3. explicitly set `py_modules` or `packages` with a list of names

To find more information, look for "package discovery" on setuptools docs.
```

Weirdly enough the other `setup.py` commands seems to work OK e.g. `python setup.py develop` or `python setup.py install` ...

COMMENT:
Maybe we should follow numpy and not support `python setup.py clean`: https://github.com/numpy/numpy/blob/ad1096884e82a8176e7b5125d168ceeca77c3bf7/setup.py#L367-L373. They have done this 7 years ago apparently.

We can change `make clean` to do `git clean -Xdf` (remove only ignored files so amongst other things compiled files and leave unversioned files alone) as suggested by the numpy error message.

Labels: Bug, Build / CI
HalvingRandomSearchCV - Custom Factor

### Describe the workflow you want to enable

There is often the problem with successive Halving that there are too many candidates and not enough resources. At the moment you can handle this by using Aggressive Elimination or trying to adjust the halving factor.

I would like to enable a workflow where two factors are used, one for the number of candidates and one for the number of samples per round. Example: Increase data with factor of 2 each round, but dividde the candidates by 3.

### Describe your proposed solution

The solution would be rather easy:
 
1. Add a second factor_cand for candidates, which be default is not used
2. If the user sets factor_cand manually in gridsearch or by means of initialization it scales the number of candidates independently of the already implemented scaling factor.
3. Optionally: Add some flags and checks to see if this works out with the number of resources, though I see no issues here.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Can you provide a reference or a paper that studies using a custom factor in successive halving? There is also [scikit-learn's inclusion criterion](https://scikit-learn.org/dev/faq.html#what-are-the-inclusion-criteria-for-new-algorithms) to take into consideration.

Labels: New Feature, module:model_selection, Needs Decision - Include Feature
Bug: Not Multiplying by 100 in Mean Absolute Percentage Error

Hi,

I am using Scikit-learn version 1.1.0`.

As I was looking into the implementation of `mean_absolute_percentate_error`, I found that the implementation is missing multiplication by 100 to convert it into a percentage. 

See the code below,
https://github.com/scikit-learn/scikit-learn/blob/16625450b58f555dc3955d223f0c3b64a5686984/sklearn/metrics/_regression.py#L365-L379

I see the corresponding [scikit documentation](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-percentage-error) for this metric and there too, multiplication with 100 is missing/avoided.

On [wiki](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error#:~:text=The%20mean%20absolute%20percentage%20error,t%20is%20the%20forecast%20value.), they have mentioned multiplication by 100. 

Please check if multiplication by 100 is required?

COMMENT:
It was done on purpose and documented in the API documentation:

```
Note here that the output is not a percentage in the range [0, 100] and a value of 100 does not mean 100% but 1e2
```

Labels: Needs Triage
Bug: Not Multiplying by 100 in Mean Absolute Percentage Error

Hi,

I am using Scikit-learn version 1.1.0`.

As I was looking into the implementation of `mean_absolute_percentate_error`, I found that the implementation is missing multiplication by 100 to convert it into a percentage. 

See the code below,
https://github.com/scikit-learn/scikit-learn/blob/16625450b58f555dc3955d223f0c3b64a5686984/sklearn/metrics/_regression.py#L365-L379

I see the corresponding [scikit documentation](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-percentage-error) for this metric and there too, multiplication with 100 is missing/avoided.

On [wiki](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error#:~:text=The%20mean%20absolute%20percentage%20error,t%20is%20the%20forecast%20value.), they have mentioned multiplication by 100. 

Please check if multiplication by 100 is required?

COMMENT:
It came as a surprise to me. "Percentage" in function name can cause confusion. Could you let me know the reasoning behind having such behaviour of the function?

Labels: Needs Triage
Bug: Not Multiplying by 100 in Mean Absolute Percentage Error

Hi,

I am using Scikit-learn version 1.1.0`.

As I was looking into the implementation of `mean_absolute_percentate_error`, I found that the implementation is missing multiplication by 100 to convert it into a percentage. 

See the code below,
https://github.com/scikit-learn/scikit-learn/blob/16625450b58f555dc3955d223f0c3b64a5686984/sklearn/metrics/_regression.py#L365-L379

I see the corresponding [scikit documentation](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-percentage-error) for this metric and there too, multiplication with 100 is missing/avoided.

On [wiki](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error#:~:text=The%20mean%20absolute%20percentage%20error,t%20is%20the%20forecast%20value.), they have mentioned multiplication by 100. 

Please check if multiplication by 100 is required?

COMMENT:
You can refer to the original PR where it was debated: https://github.com/scikit-learn/scikit-learn/pull/15007#issuecomment-601094206

I think the reason is that we stick to the Wikipedia definition.

Labels: Needs Triage
Bug: Not Multiplying by 100 in Mean Absolute Percentage Error

Hi,

I am using Scikit-learn version 1.1.0`.

As I was looking into the implementation of `mean_absolute_percentate_error`, I found that the implementation is missing multiplication by 100 to convert it into a percentage. 

See the code below,
https://github.com/scikit-learn/scikit-learn/blob/16625450b58f555dc3955d223f0c3b64a5686984/sklearn/metrics/_regression.py#L365-L379

I see the corresponding [scikit documentation](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-percentage-error) for this metric and there too, multiplication with 100 is missing/avoided.

On [wiki](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error#:~:text=The%20mean%20absolute%20percentage%20error,t%20is%20the%20forecast%20value.), they have mentioned multiplication by 100. 

Please check if multiplication by 100 is required?

COMMENT:
In the wiki definition, they have used multiplication by 100. Please [check](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error#:~:text=For%20forecasts%20which%20are%20too,limit%20to%20the%20percentage%20error.).

Labels: Needs Triage
Bug: Not Multiplying by 100 in Mean Absolute Percentage Error

Hi,

I am using Scikit-learn version 1.1.0`.

As I was looking into the implementation of `mean_absolute_percentate_error`, I found that the implementation is missing multiplication by 100 to convert it into a percentage. 

See the code below,
https://github.com/scikit-learn/scikit-learn/blob/16625450b58f555dc3955d223f0c3b64a5686984/sklearn/metrics/_regression.py#L365-L379

I see the corresponding [scikit documentation](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-percentage-error) for this metric and there too, multiplication with 100 is missing/avoided.

On [wiki](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error#:~:text=The%20mean%20absolute%20percentage%20error,t%20is%20the%20forecast%20value.), they have mentioned multiplication by 100. 

Please check if multiplication by 100 is required?

COMMENT:
It is multiplied by 100% and not 100:

![image](https://user-images.githubusercontent.com/7454015/169286908-9e3ff18d-daaf-469b-91d7-7ce204012906.png)

Labels: Needs Triage
Bug: Not Multiplying by 100 in Mean Absolute Percentage Error

Hi,

I am using Scikit-learn version 1.1.0`.

As I was looking into the implementation of `mean_absolute_percentate_error`, I found that the implementation is missing multiplication by 100 to convert it into a percentage. 

See the code below,
https://github.com/scikit-learn/scikit-learn/blob/16625450b58f555dc3955d223f0c3b64a5686984/sklearn/metrics/_regression.py#L365-L379

I see the corresponding [scikit documentation](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-percentage-error) for this metric and there too, multiplication with 100 is missing/avoided.

On [wiki](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error#:~:text=The%20mean%20absolute%20percentage%20error,t%20is%20the%20forecast%20value.), they have mentioned multiplication by 100. 

Please check if multiplication by 100 is required?

COMMENT:
So sklearn implemented MARE, but just wrong. At least they are not shy and brag about doing it wrong in the docs.

Labels: Needs Triage
LassoLars: improve precision at lower regularization values

[LassoLars](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLars.html?highlight=lassolars#sklearn.linear_model.LassoLars) can be quite imprecise at low regularization values (e.g. alpha=alpha_max/1000).
This can be easily solved by modifying the definition of the variable `equality_tolerance` (on file linear_model/_least_angle.py) so that it becomes proportional to the value of alpha, for instance as follows:

`equality_tolerance = min(tiny32, np.finfo(np.float32).eps * alpha_min)`

COMMENT:
May you provide a code snippet with sample data that show cases the issue with low regularization?

Labels: module:linear_model, Needs Investigation
LassoLars: improve precision at lower regularization values

[LassoLars](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLars.html?highlight=lassolars#sklearn.linear_model.LassoLars) can be quite imprecise at low regularization values (e.g. alpha=alpha_max/1000).
This can be easily solved by modifying the definition of the variable `equality_tolerance` (on file linear_model/_least_angle.py) so that it becomes proportional to the value of alpha, for instance as follows:

`equality_tolerance = min(tiny32, np.finfo(np.float32).eps * alpha_min)`

COMMENT:
Here is a code example and the corresponding output.
(Disclaimer: it takes around 10 minutes to run, since a coordinate descent solver is run with very high precision as a comparison)
```py
import numpy as np
from sklearn import linear_model
from numpy.linalg import norm
from sklearn.datasets import fetch_openml
from matplotlib import pyplot as plt

# Load MEG data
dataset = fetch_openml(data_id=43884)
all_data = dataset.data.to_numpy()
X = all_data[:, :7498]  # shape (305, 7498)
y = all_data[:, 7498 + 27]  #  Left Auditory condition
n_samples = X.shape[0]

# Maximum regularization parameter
alpha_max = abs(X.T @ (y - y.mean())).max() / n_samples
# alpha = 1 * alpha_max

# Running for several regularization parameters
alphas = np.array([1e-2, 5e-3, 2e-3, 1e-3]) * alpha_max
subopt = []
for alpha in alphas:

    #  Fitting models: LARS and Coordinate Descent with high precision (as a reference)
    lars = linear_model.LassoLars(alpha=alpha, fit_intercept=True, normalize=False)
    lars.fit(X, y)

    cd = linear_model.Lasso(alpha=alpha, fit_intercept=True, max_iter=1e6, tol=1e-12)
    cd.fit(X, y)

    # Computing objective function
    obj_lars = (
        norm(y - X @ lars.coef_ - lars.intercept_) ** 2 / (2 * n_samples)
        + alpha * abs(lars.coef_).sum()
    )

    obj_cd = (
        norm(y - X @ cd.coef_ - cd.intercept_) ** 2 / (2 * n_samples)
        + alpha * abs(cd.coef_).sum()
    )

    #  Suboptimality (objective function value)
    subopt.append(obj_lars - min(obj_cd, obj_lars))

# Plotting suboptimality of Lasso-LARS solver
plt.loglog(alphas / alpha_max, subopt, label="LARS")
plt.title("Suboptimality of Lasso-Lars solver")
plt.ylabel(r"$F(x) - F(x^*)$")
plt.xlabel(r"Relative regularization ($\alpha/\alpha_{max}$)")
plt.legend()
```
![output](https://user-images.githubusercontent.com/774058/178243789-3fe66eed-a7a5-4482-8aa5-32eaee3a3342.png)

We can see that Lars suboptimality increases significantly as the regularization parameter gets smaller.

Labels: module:linear_model, Needs Investigation
Grouping of infrequent categories in 𝗢𝗻𝗲𝗛𝗼𝘁𝗘𝗻𝗰𝗼𝗱𝗲𝗿

Dear all,

This feature is very similar to the one presented in this paper:
https://ieeexplore.ieee.org/document/8851888

which states the following paragraph:
The goal of the PCP transform is to substantially reduce the input memory and processing requirements while keeping the most relevant levels. Fig. 2 exemplifies the effect of this transform, by presenting the 1,000 highest frequency values (decreasing order) for the user city attribute for the TEST traffic data (which contains a total of 10,690 levels). For this attribute and when P = 10%, PCP selects only the most frequent 688 levels (dashed vertical line in Fig. 2), merging the other 10,002 infrequent levels into the "Others" label. 

<img width="883" alt="Captura de ecrã 2022-05-17, às 16 13 10" src="https://user-images.githubusercontent.com/39952471/168860640-dbe54b09-2771-4c9f-a0d9-17d4058bf29b.png">

Just asking, did the authors took notice of this paper? It can also improve this feature (instead of using a fixed number of categories it could be using a percentage categorical values to keep). 


Thanks for the attention!

COMMENT:
In the recently released 1.1, `OneHotEncoder` can group infrequent categories through `min_frequency` or `max_categories`, you can read more about it [here](https://scikit-learn.org/stable/modules/preprocessing.html#one-hot-encoder-infrequent-categories)

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
Grouping of infrequent categories in 𝗢𝗻𝗲𝗛𝗼𝘁𝗘𝗻𝗰𝗼𝗱𝗲𝗿

Dear all,

This feature is very similar to the one presented in this paper:
https://ieeexplore.ieee.org/document/8851888

which states the following paragraph:
The goal of the PCP transform is to substantially reduce the input memory and processing requirements while keeping the most relevant levels. Fig. 2 exemplifies the effect of this transform, by presenting the 1,000 highest frequency values (decreasing order) for the user city attribute for the TEST traffic data (which contains a total of 10,690 levels). For this attribute and when P = 10%, PCP selects only the most frequent 688 levels (dashed vertical line in Fig. 2), merging the other 10,002 infrequent levels into the "Others" label. 

<img width="883" alt="Captura de ecrã 2022-05-17, às 16 13 10" src="https://user-images.githubusercontent.com/39952471/168860640-dbe54b09-2771-4c9f-a0d9-17d4058bf29b.png">

Just asking, did the authors took notice of this paper? It can also improve this feature (instead of using a fixed number of categories it could be using a percentage categorical values to keep). 


Thanks for the attention!

COMMENT:
> In the recently released 1.1, `OneHotEncoder` can group infrequent categories through `min_frequency` or `max_categories`, you can read more about it [here](https://scikit-learn.org/stable/modules/preprocessing.html#one-hot-encoder-infrequent-categories)

I know, im just asking if the authors of this new update took notice of the paper I linked....
It is from 2019...
It also could provide with a categorical transformation like the PCP (which is similar to the new OneHotEncoder infrequent categories, but instead of a fixed number, you provide a percentage cut)

You can check the package that does this in here as well:

https://pypi.org/project/cane/

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
Grouping of infrequent categories in 𝗢𝗻𝗲𝗛𝗼𝘁𝗘𝗻𝗰𝗼𝗱𝗲𝗿

Dear all,

This feature is very similar to the one presented in this paper:
https://ieeexplore.ieee.org/document/8851888

which states the following paragraph:
The goal of the PCP transform is to substantially reduce the input memory and processing requirements while keeping the most relevant levels. Fig. 2 exemplifies the effect of this transform, by presenting the 1,000 highest frequency values (decreasing order) for the user city attribute for the TEST traffic data (which contains a total of 10,690 levels). For this attribute and when P = 10%, PCP selects only the most frequent 688 levels (dashed vertical line in Fig. 2), merging the other 10,002 infrequent levels into the "Others" label. 

<img width="883" alt="Captura de ecrã 2022-05-17, às 16 13 10" src="https://user-images.githubusercontent.com/39952471/168860640-dbe54b09-2771-4c9f-a0d9-17d4058bf29b.png">

Just asking, did the authors took notice of this paper? It can also improve this feature (instead of using a fixed number of categories it could be using a percentage categorical values to keep). 


Thanks for the attention!

COMMENT:
@lesteve Did you check the work? 
It's from 2019, and it does the same as your new feature (even has a possible good solution based on percentages instead of a specific number)

That was also explored in this paper:

https://ieeexplore.ieee.org/abstract/document/8710472
Which is named CP (Categorical Pruned)

Another suggestion is the method of IDF for categorical data (also in the link I have send and in the packaged I have posted)

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
Grouping of infrequent categories in 𝗢𝗻𝗲𝗛𝗼𝘁𝗘𝗻𝗰𝗼𝗱𝗲𝗿

Dear all,

This feature is very similar to the one presented in this paper:
https://ieeexplore.ieee.org/document/8851888

which states the following paragraph:
The goal of the PCP transform is to substantially reduce the input memory and processing requirements while keeping the most relevant levels. Fig. 2 exemplifies the effect of this transform, by presenting the 1,000 highest frequency values (decreasing order) for the user city attribute for the TEST traffic data (which contains a total of 10,690 levels). For this attribute and when P = 10%, PCP selects only the most frequent 688 levels (dashed vertical line in Fig. 2), merging the other 10,002 infrequent levels into the "Others" label. 

<img width="883" alt="Captura de ecrã 2022-05-17, às 16 13 10" src="https://user-images.githubusercontent.com/39952471/168860640-dbe54b09-2771-4c9f-a0d9-17d4058bf29b.png">

Just asking, did the authors took notice of this paper? It can also improve this feature (instead of using a fixed number of categories it could be using a percentage categorical values to keep). 


Thanks for the attention!

COMMENT:
> @lesteve Did you check the work?

No sorry ...

It's not clear to me what your would like to happen out of this issue, it would help a lot if you could clarify:
1. a new parameter in `OneHotEncoder` based on a cumulative frequency, rather than an individual per-value frequency (`min_frequency` is doing the latter), i.e. put group all the category values, whose cumulative frequency amounts to 10% of the number of samples, in the category value "Other"
2. some kind of attribution. You would like your work to be referenced in the scikit-learn code for the `min_frequency` feature that already exists
3. nothing in particular, it was more of a "for your information" discussion, i.e. something like this: "this feature is cool, we found it useful in practice, you can read more about more about our work here"
4. something else: sorry I am not a mind reader yet :wink:

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
Grouping of infrequent categories in 𝗢𝗻𝗲𝗛𝗼𝘁𝗘𝗻𝗰𝗼𝗱𝗲𝗿

Dear all,

This feature is very similar to the one presented in this paper:
https://ieeexplore.ieee.org/document/8851888

which states the following paragraph:
The goal of the PCP transform is to substantially reduce the input memory and processing requirements while keeping the most relevant levels. Fig. 2 exemplifies the effect of this transform, by presenting the 1,000 highest frequency values (decreasing order) for the user city attribute for the TEST traffic data (which contains a total of 10,690 levels). For this attribute and when P = 10%, PCP selects only the most frequent 688 levels (dashed vertical line in Fig. 2), merging the other 10,002 infrequent levels into the "Others" label. 

<img width="883" alt="Captura de ecrã 2022-05-17, às 16 13 10" src="https://user-images.githubusercontent.com/39952471/168860640-dbe54b09-2771-4c9f-a0d9-17d4058bf29b.png">

Just asking, did the authors took notice of this paper? It can also improve this feature (instead of using a fixed number of categories it could be using a percentage categorical values to keep). 


Thanks for the attention!

COMMENT:
> > @lesteve Did you check the work?
> 
> 
> 
> No sorry ...
> 
> 
> 
> It's not clear to me what your would like to happen out of this issue, it would help a lot if you could clarify:
> 
> 1. a new parameter in `OneHotEncoder` based on a cumulative frequency, rather than an individual per-value frequency (`min_frequency` is doing the latter), i.e. put group all the category values, whose cumulative frequency amounts to 10% of the number of samples, in the category value "Other"
> 
> 2. some kind of attribution. You would like your work to be referenced in the scikit-learn code for the `min_frequency` feature that already exists
> 
> 3. nothing in particular, it was more of a "for your information" discussion, i.e. something like this: "this feature is cool, we found it useful in practice, you can read more about more about our work here"
> 
> 4. something else: sorry I am not a mind reader yet :wink: 

@lesteve 
Well I would love to see the first 2 points happen to be honest 😃 particularly it could be an improvement for the package and the one hot encoding processing using the PCP transformation as proposed in the paper I have linked 😃

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
Grouping of infrequent categories in 𝗢𝗻𝗲𝗛𝗼𝘁𝗘𝗻𝗰𝗼𝗱𝗲𝗿

Dear all,

This feature is very similar to the one presented in this paper:
https://ieeexplore.ieee.org/document/8851888

which states the following paragraph:
The goal of the PCP transform is to substantially reduce the input memory and processing requirements while keeping the most relevant levels. Fig. 2 exemplifies the effect of this transform, by presenting the 1,000 highest frequency values (decreasing order) for the user city attribute for the TEST traffic data (which contains a total of 10,690 levels). For this attribute and when P = 10%, PCP selects only the most frequent 688 levels (dashed vertical line in Fig. 2), merging the other 10,002 infrequent levels into the "Others" label. 

<img width="883" alt="Captura de ecrã 2022-05-17, às 16 13 10" src="https://user-images.githubusercontent.com/39952471/168860640-dbe54b09-2771-4c9f-a0d9-17d4058bf29b.png">

Just asking, did the authors took notice of this paper? It can also improve this feature (instead of using a fixed number of categories it could be using a percentage categorical values to keep). 


Thanks for the attention!

COMMENT:
Is there any open-access pre-print for this article?

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature