Talip7/scikit-learn-issues-multilabel huggingface dataset:
Allow batch based metrics calculation of sklearn.metrics

### Describe the workflow you want to enable

I have a lot of data and need to calculate metrics such as accuracy_score, jaccard_score, f1_score, recall, precision etc.

### Describe your proposed solution

 When I try to calculate these it can literally take days, so i created a small solution which can batch and avg in the end, or for the weighted metrics it can do a weighted avg of each, this accelerated the calculation to just a couple of minutes, because I have a 32 core CPU. I'm willing to contribute with the proper guidance as I'm unfamiliar with the codebase, but I think many people can benefit from this. I'm unsure if there is already a work around of this present in the codebase, but if there is one do let me know, thanks a lot.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> When I try to calculate it in this way, all the predictions and targets are calculated pretty quickly. When I flatten and give it to all the metrics(jaccard, f1, recall, precision, etc). It takes a long time, most probably because this is a lot of data in my case, roughly 17000 images, 512x512, flattened.

Can you paste the alternative code which is much slower?

Labels: Performance, help wanted, module:metrics, Needs Investigation
Allow batch based metrics calculation of sklearn.metrics

### Describe the workflow you want to enable

I have a lot of data and need to calculate metrics such as accuracy_score, jaccard_score, f1_score, recall, precision etc.

### Describe your proposed solution

 When I try to calculate these it can literally take days, so i created a small solution which can batch and avg in the end, or for the weighted metrics it can do a weighted avg of each, this accelerated the calculation to just a couple of minutes, because I have a 32 core CPU. I'm willing to contribute with the proper guidance as I'm unfamiliar with the codebase, but I think many people can benefit from this. I'm unsure if there is already a work around of this present in the codebase, but if there is one do let me know, thanks a lot.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think you misunderstood me. This is the slow code.
What I was trying to say is that the predictions and targets(all_preds, all_targets is computed before being passed to the metric functions) are computed pretty quickly; later, when those predictions and targets are passed to the scikit functions, it takes a long time, because I think it's working in a single-threaded manner.

Labels: Performance, help wanted, module:metrics, Needs Investigation
Allow batch based metrics calculation of sklearn.metrics

### Describe the workflow you want to enable

I have a lot of data and need to calculate metrics such as accuracy_score, jaccard_score, f1_score, recall, precision etc.

### Describe your proposed solution

 When I try to calculate these it can literally take days, so i created a small solution which can batch and avg in the end, or for the weighted metrics it can do a weighted avg of each, this accelerated the calculation to just a couple of minutes, because I have a 32 core CPU. I'm willing to contribute with the proper guidance as I'm unfamiliar with the codebase, but I think many people can benefit from this. I'm unsure if there is already a work around of this present in the codebase, but if there is one do let me know, thanks a lot.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Ok, for us to continue, please provide a minimal reproducer which runs slowly, which can nicely be profiled for us to see where we can improve.

Labels: Performance, help wanted, module:metrics, Needs Investigation
Allow batch based metrics calculation of sklearn.metrics

### Describe the workflow you want to enable

I have a lot of data and need to calculate metrics such as accuracy_score, jaccard_score, f1_score, recall, precision etc.

### Describe your proposed solution

 When I try to calculate these it can literally take days, so i created a small solution which can batch and avg in the end, or for the weighted metrics it can do a weighted avg of each, this accelerated the calculation to just a couple of minutes, because I have a 32 core CPU. I'm willing to contribute with the proper guidance as I'm unfamiliar with the codebase, but I think many people can benefit from this. I'm unsure if there is already a work around of this present in the codebase, but if there is one do let me know, thanks a lot.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
This should give you an idea about the issue
```
import numpy as np
from sklearn.metrics import accuracy_score, jaccard_score, f1_score, recall_score, precision_score
import time

num_samples = 10000
height, width = 512, 512

all_preds = np.random.rand(num_samples, 1, height, width).astype(np.float32)

all_targets = np.random.randint(0, 2, size=(num_samples, 1, height, width)).astype(np.uint8)

start = time.time()

preds_binary = (all_preds > 0.5).astype(int)

accuracy = accuracy_score(all_targets.flatten(), preds_binary.flatten())
jaccard = jaccard_score(all_targets.flatten(), preds_binary.flatten(), average='binary')
f1 = f1_score(all_targets.flatten(), preds_binary.flatten(), average='binary')
recall = recall_score(all_targets.flatten(), preds_binary.flatten(), average='binary')
precision = precision_score(all_targets.flatten(), preds_binary.flatten(), average='binary')

end = time.time()

print(f"Accuracy: {accuracy:.4f}, Jaccard: {jaccard:.4f}, F1 Score: {f1:.4f}")
print(f"Recall: {recall:.4f}, Precision: {precision:.4f}")
print(f"\nTime taken: {end - start:.2f} seconds")

```

Labels: Performance, help wanted, module:metrics, Needs Investigation
Allow batch based metrics calculation of sklearn.metrics

### Describe the workflow you want to enable

I have a lot of data and need to calculate metrics such as accuracy_score, jaccard_score, f1_score, recall, precision etc.

### Describe your proposed solution

 When I try to calculate these it can literally take days, so i created a small solution which can batch and avg in the end, or for the weighted metrics it can do a weighted avg of each, this accelerated the calculation to just a couple of minutes, because I have a 32 core CPU. I'm willing to contribute with the proper guidance as I'm unfamiliar with the codebase, but I think many people can benefit from this. I'm unsure if there is already a work around of this present in the codebase, but if there is one do let me know, thanks a lot.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi, I'm interested in investigating this performance issue. I'll start by profiling the provided code snippet to identify the exact bottleneck within the metric functions. Is anyone else currently working on this?

Labels: Performance, help wanted, module:metrics, Needs Investigation
Allow batch based metrics calculation of sklearn.metrics

### Describe the workflow you want to enable

I have a lot of data and need to calculate metrics such as accuracy_score, jaccard_score, f1_score, recall, precision etc.

### Describe your proposed solution

 When I try to calculate these it can literally take days, so i created a small solution which can batch and avg in the end, or for the weighted metrics it can do a weighted avg of each, this accelerated the calculation to just a couple of minutes, because I have a 32 core CPU. I'm willing to contribute with the proper guidance as I'm unfamiliar with the codebase, but I think many people can benefit from this. I'm unsure if there is already a work around of this present in the codebase, but if there is one do let me know, thanks a lot.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thanks @pushkar-hue , go for it.

Labels: Performance, help wanted, module:metrics, Needs Investigation
Allow batch based metrics calculation of sklearn.metrics

### Describe the workflow you want to enable

I have a lot of data and need to calculate metrics such as accuracy_score, jaccard_score, f1_score, recall, precision etc.

### Describe your proposed solution

 When I try to calculate these it can literally take days, so i created a small solution which can batch and avg in the end, or for the weighted metrics it can do a weighted avg of each, this accelerated the calculation to just a couple of minutes, because I have a 32 core CPU. I'm willing to contribute with the proper guidance as I'm unfamiliar with the codebase, but I think many people can benefit from this. I'm unsure if there is already a work around of this present in the codebase, but if there is one do let me know, thanks a lot.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
What's your numpy version? Can you check against nightly build? `unique1d` doesn't call `sort` on many dtypes and that has been a solved issue now.

Labels: Performance, help wanted, module:metrics, Needs Investigation
Allow batch based metrics calculation of sklearn.metrics

### Describe the workflow you want to enable

I have a lot of data and need to calculate metrics such as accuracy_score, jaccard_score, f1_score, recall, precision etc.

### Describe your proposed solution

 When I try to calculate these it can literally take days, so i created a small solution which can batch and avg in the end, or for the weighted metrics it can do a weighted avg of each, this accelerated the calculation to just a couple of minutes, because I have a 32 core CPU. I'm willing to contribute with the proper guidance as I'm unfamiliar with the codebase, but I think many people can benefit from this. I'm unsure if there is already a work around of this present in the codebase, but if there is one do let me know, thanks a lot.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I'm confused as you're using different `num_samples`. Please also leave the actual script for this; since I'm a bit surprised at the times and the small size of the data.

Labels: Performance, help wanted, module:metrics, Needs Investigation
Allow batch based metrics calculation of sklearn.metrics

### Describe the workflow you want to enable

I have a lot of data and need to calculate metrics such as accuracy_score, jaccard_score, f1_score, recall, precision etc.

### Describe your proposed solution

 When I try to calculate these it can literally take days, so i created a small solution which can batch and avg in the end, or for the weighted metrics it can do a weighted avg of each, this accelerated the calculation to just a couple of minutes, because I have a 32 core CPU. I'm willing to contribute with the proper guidance as I'm unfamiliar with the codebase, but I think many people can benefit from this. I'm unsure if there is already a work around of this present in the codebase, but if there is one do let me know, thanks a lot.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I apologize  you're absolutely right, the data size is quite small. I was running into out-of-memory (OOM) issues on my machine when trying with more than 500 samples, so I had to reduce the size to avoid crashing.

Here’s the full script I used for profiling:

```
import numpy as np
from sklearn.metrics import accuracy_score, jaccard_score, f1_score, recall_score, precision_score
import cProfile
import pstats
import io
import time

def calculate_all_metrics():
    num_samples = 250  
    height, width = 512, 512

    all_preds = np.random.rand(num_samples, 1, height, width).astype(np.float32)
    all_targets = np.random.randint(0, 2, size=(num_samples, 1, height, width)).astype(np.uint8)

    preds_binary = (all_preds > 0.5).astype(int)

    accuracy = accuracy_score(all_targets.flatten(), preds_binary.flatten())
    jaccard = jaccard_score(all_targets.flatten(), preds_binary.flatten(), average='binary')
    f1 = f1_score(all_targets.flatten(), preds_binary.flatten(), average='binary')
    recall = recall_score(all_targets.flatten(), preds_binary.flatten(), average='binary')
    precision = precision_score(all_targets.flatten(), preds_binary.flatten(), average='binary')

    print(f"Accuracy: {accuracy:.4f}, Jaccard: {jaccard:.4f}, F1 Score: {f1:.4f}")
    print(f"Recall: {recall:.4f}, Precision: {precision:.4f}")

# Timer
start = time.time()

# Run profiler
pr = cProfile.Profile()
pr.enable()
calculate_all_metrics()
pr.disable()
end = time.time()

print(f"\nTotal Time taken: {end - start:.2f} seconds")

# Capture profiler output
s = io.StringIO()
ps = pstats.Stats(pr, stream=s).sort_stats("tottime")
ps.print_stats(15)

# Print profiler output
print("\n--- PROFILER RESULTS ---")
print(s.getvalue())

```

Labels: Performance, help wanted, module:metrics, Needs Investigation
Allow batch based metrics calculation of sklearn.metrics

### Describe the workflow you want to enable

I have a lot of data and need to calculate metrics such as accuracy_score, jaccard_score, f1_score, recall, precision etc.

### Describe your proposed solution

 When I try to calculate these it can literally take days, so i created a small solution which can batch and avg in the end, or for the weighted metrics it can do a weighted avg of each, this accelerated the calculation to just a couple of minutes, because I have a 32 core CPU. I'm willing to contribute with the proper guidance as I'm unfamiliar with the codebase, but I think many people can benefit from this. I'm unsure if there is already a work around of this present in the codebase, but if there is one do let me know, thanks a lot.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi @tahamukhtar20 and scikit-learn maintainers, I’m interested in contributing a batch-based metrics calculation feature as described in this issue. I propose implementing a utility to process metrics like accuracy_score, f1_score, etc., in batches with averaging or weighted averaging, leveraging parallel processing (e.g., via joblib). Could maintainers confirm if this aligns with project goals? I’d also appreciate pointers on integrating with sklearn.metrics and any specific conventions to follow.

Labels: Performance, help wanted, module:metrics, Needs Investigation
Allow batch based metrics calculation of sklearn.metrics

### Describe the workflow you want to enable

I have a lot of data and need to calculate metrics such as accuracy_score, jaccard_score, f1_score, recall, precision etc.

### Describe your proposed solution

 When I try to calculate these it can literally take days, so i created a small solution which can batch and avg in the end, or for the weighted metrics it can do a weighted avg of each, this accelerated the calculation to just a couple of minutes, because I have a 32 core CPU. I'm willing to contribute with the proper guidance as I'm unfamiliar with the codebase, but I think many people can benefit from this. I'm unsure if there is already a work around of this present in the codebase, but if there is one do let me know, thanks a lot.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi, I go ahead and add function batch_f1_score to address this issue. 
Here is my pull request : [https://github.com/scikit-learn/scikit-learn/pull/31897](url).
Please take a look and let me know if something that needs to be fixed before it can be merged into main. 
Thank you.

Labels: Performance, help wanted, module:metrics, Needs Investigation
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
Adding the loss seems quite reasonable. Adding a custom loss might be a bit more complicated, and hard to do with good performance. If you think you can add the loss with a PR, we'd welcome that.

I wonder if @lorentzenchr has an opinion on this one.

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
I'd be interested in contributing Huber loss support to HistGradientBoostingRegressor. Let me know if this is something the maintainers would support.

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
I would prefer to not implement Huber for HGBT. Mainly because it is not that clear, what a model minimizing Huber loss actually estimates/predicts, something between mean and median. Maybe, @AndrewYRoyal's use case can be solved differently? It would help if you could outline our use case briefly.

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
Thanks for all your attention to the matter. My particular use case has lots of _very_ noisy training observations (label measurement error is +10x the scale of actual heterogeneity in labels); I can handle this partially using sample weights, but I've found HuberLoss to be effective at training a model that does well on both L1 and L2 evaluation metrics. 

I'm in a position now where I'm seeing better L1 and L2 performance while using `GradientBoostingRegressor` with HuberLoss (though at the cost of much longer training time). The performance drop when using `HistGradientBoostingRegressor` could be due to a number of reasons, but I suspect that the loss function could be the culprit-- FWIW I've also found that the linear HuberRegression generates better squared/absolute loss metrics with my data when compared to Lasso & Ridge.

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
Can you detect the outliers? Have you tried `HistGradientBoostingRegressor(loss="quantile", quantile=0.5)` to estimate the median instead of the mean?

For this kind of discussion, training time is only of second interest for me. (Has to do with non-smooth loss.)

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
Yeah it looks like using quantile regression does improve model performance. I still think it could be nice to see whether using Huber loss improves performance further, but I also understand your point about the ambiguity of what it's optimizing for.

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
What do you think here @antoinebaker ?

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
Could you provide some information about your dataset(s)? How many samples (rows)? How many features (columns)? When you say L1 and L2 evaluation metrics are often better with a model minimizing Huber loss, is it on a separate hold-out/test set or on the training set? If you do cross validation, are the L1/L2 CV scores within the uncertainty/std_deviation of the other model (e.g. GradientBoostingRegressor with Huber and one with absolute error).

Labels: New Feature, help wanted, Hard