Talip7/scikit-learn-issues-multilabel huggingface dataset:
Regression error characteristic curve

### Describe the workflow you want to enable

Add more fine-grained diagnostic, similar to ROC or Precision-Recall curves, to regression problems. It appears that this library has a lot of excellent tools for classification, and I believe it would benefit from some additional tools for regression.

### Describe your proposed solution

Compute Regression Error Characteristic (REC) [1] curve - for each error threshold the percentage of samples whose error is below that threshold. This is essentially the CDF of the regression errors. Its function is similar to that of ROC curves - allows comparing performance profiles of regressors beyond just one summary statistic, such as RMSE or MAE.

I already implement a pull-request:
https://github.com/scikit-learn/scikit-learn/pull/31380

Screenshot from the merge request:

![Image](https://github.com/user-attachments/assets/1974e8e7-03da-47c7-adb5-5c75eb24d61e)

If you believe this feature is useful, please help me with reviewing and merging it.

### Describe alternatives you've considered, if relevant

Regression Receiver Operating Characteristic (RROC) curves, proposed [2], which plot over-prediction vs under-prediction, are a different form of diagnostic curves for regression. They may also be useful, but I think we should begin from somewhere, and I belive it's better to begin from REC, both because the paper has more citations, and because it turned out to be very useful for me at work, and I believe it can be similarly useful to other scientists.

### Additional context

**References**
---

[1]: Bi, J. and Bennett, K.P., 2003. Regression error characteristic curves. In Proceedings of the 20th international conference on machine learning (ICML-03) (pp. 43-50).
[2]: Hernández-Orallo, J., 2013. ROC curves for regression. Pattern Recognition, 46(12), pp.3395-3411.

COMMENT:
Counter question: What do you learn from a fitted model by looking at the CDF of some error (loss function or score) of its predictions?

Labels: New Feature, Needs Triage
Regression error characteristic curve

### Describe the workflow you want to enable

Add more fine-grained diagnostic, similar to ROC or Precision-Recall curves, to regression problems. It appears that this library has a lot of excellent tools for classification, and I believe it would benefit from some additional tools for regression.

### Describe your proposed solution

Compute Regression Error Characteristic (REC) [1] curve - for each error threshold the percentage of samples whose error is below that threshold. This is essentially the CDF of the regression errors. Its function is similar to that of ROC curves - allows comparing performance profiles of regressors beyond just one summary statistic, such as RMSE or MAE.

I already implement a pull-request:
https://github.com/scikit-learn/scikit-learn/pull/31380

Screenshot from the merge request:

![Image](https://github.com/user-attachments/assets/1974e8e7-03da-47c7-adb5-5c75eb24d61e)

If you believe this feature is useful, please help me with reviewing and merging it.

### Describe alternatives you've considered, if relevant

Regression Receiver Operating Characteristic (RROC) curves, proposed [2], which plot over-prediction vs under-prediction, are a different form of diagnostic curves for regression. They may also be useful, but I think we should begin from somewhere, and I belive it's better to begin from REC, both because the paper has more citations, and because it turned out to be very useful for me at work, and I believe it can be similarly useful to other scientists.

### Additional context

**References**
---

[1]: Bi, J. and Bennett, K.P., 2003. Regression error characteristic curves. In Proceedings of the 20th international conference on machine learning (ICML-03) (pp. 43-50).
[2]: Hernández-Orallo, J., 2013. ROC curves for regression. Pattern Recognition, 46(12), pp.3395-3411.

COMMENT:
Well, the x axis of the curve serves as an error threshold, so directly what you observe is what portion of the data has errors less than $\epsilon$ for various values of $\epsilon$. So this may already be directly relevant in terms of business - "we have errors < 0.5 for 95% of the data", and you can see it visually for any error threshold beyond 0.5.

Secondly, it lets you do is compare models. In some sense, model A is better than model B if for any threshold $\epsilon$ it has more data whose errors are under $\epsilon$.

Finally, If your curve quickly grows, stops at, say,  0.7, and grows very slowly towards 1, you understand that you have a long tail of samples with a large error (a kind of a upwards knee shape). Alternatively, if it grows a bit slowly but saturates at 1, you see that you do not have a long tail. Plotting several curves of various models lets you compare this behavior, because "quickly" and "slowly" are more appreciated by humans as relative to some baseline, rather than some absolute qualitative concept. And there is always a baseline on the plot - a constant predictor.

So for me these aspects were very useful. But as I said, I need to understand CAP curves more deeply to understand if similar observations can be made using CAP curves.

Labels: New Feature, Needs Triage
Regression error characteristic curve

### Describe the workflow you want to enable

Add more fine-grained diagnostic, similar to ROC or Precision-Recall curves, to regression problems. It appears that this library has a lot of excellent tools for classification, and I believe it would benefit from some additional tools for regression.

### Describe your proposed solution

Compute Regression Error Characteristic (REC) [1] curve - for each error threshold the percentage of samples whose error is below that threshold. This is essentially the CDF of the regression errors. Its function is similar to that of ROC curves - allows comparing performance profiles of regressors beyond just one summary statistic, such as RMSE or MAE.

I already implement a pull-request:
https://github.com/scikit-learn/scikit-learn/pull/31380

Screenshot from the merge request:

![Image](https://github.com/user-attachments/assets/1974e8e7-03da-47c7-adb5-5c75eb24d61e)

If you believe this feature is useful, please help me with reviewing and merging it.

### Describe alternatives you've considered, if relevant

Regression Receiver Operating Characteristic (RROC) curves, proposed [2], which plot over-prediction vs under-prediction, are a different form of diagnostic curves for regression. They may also be useful, but I think we should begin from somewhere, and I belive it's better to begin from REC, both because the paper has more citations, and because it turned out to be very useful for me at work, and I believe it can be similarly useful to other scientists.

### Additional context

**References**
---

[1]: Bi, J. and Bennett, K.P., 2003. Regression error characteristic curves. In Proceedings of the 20th international conference on machine learning (ICML-03) (pp. 43-50).
[2]: Hernández-Orallo, J., 2013. ROC curves for regression. Pattern Recognition, 46(12), pp.3395-3411.

COMMENT:
@lorentzenchr I read the literature in the issue about CAP curves, and I don't see how it applies to regression problems. How do you measure the proportion of "positive" events out of your top-k scores? What does "positive" mean? If you mean thresholding the prediction, then for every threshold you get a different curve.

And how do you get business insights such as this one's I described above?

Am I missing something?

Labels: New Feature, Needs Triage
Regression error characteristic curve

### Describe the workflow you want to enable

Add more fine-grained diagnostic, similar to ROC or Precision-Recall curves, to regression problems. It appears that this library has a lot of excellent tools for classification, and I believe it would benefit from some additional tools for regression.

### Describe your proposed solution

Compute Regression Error Characteristic (REC) [1] curve - for each error threshold the percentage of samples whose error is below that threshold. This is essentially the CDF of the regression errors. Its function is similar to that of ROC curves - allows comparing performance profiles of regressors beyond just one summary statistic, such as RMSE or MAE.

I already implement a pull-request:
https://github.com/scikit-learn/scikit-learn/pull/31380

Screenshot from the merge request:

![Image](https://github.com/user-attachments/assets/1974e8e7-03da-47c7-adb5-5c75eb24d61e)

If you believe this feature is useful, please help me with reviewing and merging it.

### Describe alternatives you've considered, if relevant

Regression Receiver Operating Characteristic (RROC) curves, proposed [2], which plot over-prediction vs under-prediction, are a different form of diagnostic curves for regression. They may also be useful, but I think we should begin from somewhere, and I belive it's better to begin from REC, both because the paper has more citations, and because it turned out to be very useful for me at work, and I believe it can be similarly useful to other scientists.

### Additional context

**References**
---

[1]: Bi, J. and Bennett, K.P., 2003. Regression error characteristic curves. In Proceedings of the 20th international conference on machine learning (ICML-03) (pp. 43-50).
[2]: Hernández-Orallo, J., 2013. ROC curves for regression. Pattern Recognition, 46(12), pp.3395-3411.

COMMENT:
(With a high bias as coauthor,) I suggest reading [Gini Index and Friends](https://ssrn.com/abstract=4248143). CAP and Gini require that your response/target `y` is positive.

What you suggest can be seen as a variation of residual analysis (gradient of the squared error) which answers different questions. (Again with a high bias), I suggest my blog post [Model Diagnostics: Statistics vs Machine Learning](https://lorentzen.ch/index.php/2025/05/01/model-diagnostics-statistics-vs-machine-learning/) for differences in diagnostic tools for different use cases.

If the error/loss per row is of interest for the use case, what you suggest makes absolutely sense. What I am saying is that I think the trade-off between maintenance burden (of maintainers) and benefit for scikit-learn (users) is not good enough for inclusion. BTW, I appreciate the discussion with you and the fact that you started with an issue instead of an implementation (=PR).

@scikit-learn/core-devs @scikit-learn/communication-team @scikit-learn/documentation-team Your opinion would be valuable.

Labels: New Feature, Needs Triage
Regression error characteristic curve

### Describe the workflow you want to enable

Add more fine-grained diagnostic, similar to ROC or Precision-Recall curves, to regression problems. It appears that this library has a lot of excellent tools for classification, and I believe it would benefit from some additional tools for regression.

### Describe your proposed solution

Compute Regression Error Characteristic (REC) [1] curve - for each error threshold the percentage of samples whose error is below that threshold. This is essentially the CDF of the regression errors. Its function is similar to that of ROC curves - allows comparing performance profiles of regressors beyond just one summary statistic, such as RMSE or MAE.

I already implement a pull-request:
https://github.com/scikit-learn/scikit-learn/pull/31380

Screenshot from the merge request:

![Image](https://github.com/user-attachments/assets/1974e8e7-03da-47c7-adb5-5c75eb24d61e)

If you believe this feature is useful, please help me with reviewing and merging it.

### Describe alternatives you've considered, if relevant

Regression Receiver Operating Characteristic (RROC) curves, proposed [2], which plot over-prediction vs under-prediction, are a different form of diagnostic curves for regression. They may also be useful, but I think we should begin from somewhere, and I belive it's better to begin from REC, both because the paper has more citations, and because it turned out to be very useful for me at work, and I believe it can be similarly useful to other scientists.

### Additional context

**References**
---

[1]: Bi, J. and Bennett, K.P., 2003. Regression error characteristic curves. In Proceedings of the 20th international conference on machine learning (ICML-03) (pp. 43-50).
[2]: Hernández-Orallo, J., 2013. ROC curves for regression. Pattern Recognition, 46(12), pp.3395-3411.

COMMENT:
We certainly have room to improve tools we provide to inspect models, and this falls into that bucket. We also have somebody starting in October, who would focus on displays and similar tools. So I'd say from that perspective, it'd be nice to have a list of ideas to implement / include / review.

However, I don't have the right background to judge inclusion of this particular plot; but I think it would be very nice to have you @alexshtf involved in improving those aspects of the library.

Labels: New Feature, Needs Triage
Regression error characteristic curve

### Describe the workflow you want to enable

Add more fine-grained diagnostic, similar to ROC or Precision-Recall curves, to regression problems. It appears that this library has a lot of excellent tools for classification, and I believe it would benefit from some additional tools for regression.

### Describe your proposed solution

Compute Regression Error Characteristic (REC) [1] curve - for each error threshold the percentage of samples whose error is below that threshold. This is essentially the CDF of the regression errors. Its function is similar to that of ROC curves - allows comparing performance profiles of regressors beyond just one summary statistic, such as RMSE or MAE.

I already implement a pull-request:
https://github.com/scikit-learn/scikit-learn/pull/31380

Screenshot from the merge request:

![Image](https://github.com/user-attachments/assets/1974e8e7-03da-47c7-adb5-5c75eb24d61e)

If you believe this feature is useful, please help me with reviewing and merging it.

### Describe alternatives you've considered, if relevant

Regression Receiver Operating Characteristic (RROC) curves, proposed [2], which plot over-prediction vs under-prediction, are a different form of diagnostic curves for regression. They may also be useful, but I think we should begin from somewhere, and I belive it's better to begin from REC, both because the paper has more citations, and because it turned out to be very useful for me at work, and I believe it can be similarly useful to other scientists.

### Additional context

**References**
---

[1]: Bi, J. and Bennett, K.P., 2003. Regression error characteristic curves. In Proceedings of the 20th international conference on machine learning (ICML-03) (pp. 43-50).
[2]: Hernández-Orallo, J., 2013. ROC curves for regression. Pattern Recognition, 46(12), pp.3395-3411.

COMMENT:
@adrinjalali thanks for your reply. There are other regression diagnosis tools, such as the ones pointed out by Chris, and also RROC curves that I talked about in the issue description. You can also see in the description that I already implemented a pull-request for REC curves:
https://github.com/scikit-learn/scikit-learn/pull/31380

So you can clone my fork and play with it to get a feeling of what it does. And please, if there are any discussions about regression diagnostic curves, feel free to include me.

Labels: New Feature, Needs Triage
The libomp.dylib shipped with the macOS x86_64 package does not have an SDK version set

### Describe the bug

I want to build an macOS app that uses scikit-learn as a dependency. Using the arm64 package of scikit-learn for this works flawlessly. However, if I want to do the same using the macOS x86_64 packages Apple's notarizing step always breaks the app. This is likely due to the shipped libomp.dylib in the x86_64 package (installed using pip) does not have an SDK version set:
```
otool -l ./venv/lib/python3.12/site-packages/sklearn/.dylibs/libomp.dylib | grep -A 3 LC_VERSION_MIN_MACOSX
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.9
      sdk n/a
```
The arm64 version has this set:
```
otool -l ./venv/lib/python3.12/site-packages/sklearn/.dylibs/libomp.dylib | grep -A 4 LC_BUILD_VERSION
      cmd LC_BUILD_VERSION
  cmdsize 32
 platform 1
    minos 11.0
      sdk 11.0
```
It would be great, if you could set this (to at least 10.9; would probably need a rebuild of the dylib from source). I already tried some workarounds, but so far none have been successful. Is there any chance you would consider that :)?

### Steps/Code to Reproduce

```
% otool -l ./venv/lib/python3.12/site-packages/sklearn/.dylibs/libomp.dylib | grep -A 3 LC_VERSION_MIN_MACOSX
      cmd LC_VERSION_MIN_MACOSX
```

### Expected Results

```
  cmdsize 16
  version 10.9
      sdk 10.9
```

### Actual Results

```
  cmdsize 16
  version 10.9
      sdk n/a
```

### Versions

```shell
scikit-learn==1.6.1 (from pip freeze)
```

COMMENT:
I’d like to work on this issue and attempt to fix the missing SDK version in the x86_64 libomp.dylib. I’ll investigate how to ensure the library is built with the correct SDK version so that notarization passes.

Please let me know if anyone is already working on this, or if there are any specific guidelines I should follow while contributing.

Thanks!

Labels: Bug, Needs Triage
The libomp.dylib shipped with the macOS x86_64 package does not have an SDK version set

### Describe the bug

I want to build an macOS app that uses scikit-learn as a dependency. Using the arm64 package of scikit-learn for this works flawlessly. However, if I want to do the same using the macOS x86_64 packages Apple's notarizing step always breaks the app. This is likely due to the shipped libomp.dylib in the x86_64 package (installed using pip) does not have an SDK version set:
```
otool -l ./venv/lib/python3.12/site-packages/sklearn/.dylibs/libomp.dylib | grep -A 3 LC_VERSION_MIN_MACOSX
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.9
      sdk n/a
```
The arm64 version has this set:
```
otool -l ./venv/lib/python3.12/site-packages/sklearn/.dylibs/libomp.dylib | grep -A 4 LC_BUILD_VERSION
      cmd LC_BUILD_VERSION
  cmdsize 32
 platform 1
    minos 11.0
      sdk 11.0
```
It would be great, if you could set this (to at least 10.9; would probably need a rebuild of the dylib from source). I already tried some workarounds, but so far none have been successful. Is there any chance you would consider that :)?

### Steps/Code to Reproduce

```
% otool -l ./venv/lib/python3.12/site-packages/sklearn/.dylibs/libomp.dylib | grep -A 3 LC_VERSION_MIN_MACOSX
      cmd LC_VERSION_MIN_MACOSX
```

### Expected Results

```
  cmdsize 16
  version 10.9
      sdk 10.9
```

### Actual Results

```
  cmdsize 16
  version 10.9
      sdk n/a
```

### Versions

```shell
scikit-learn==1.6.1 (from pip freeze)
```

COMMENT:
I had some discussions with the devs from PyInstaller about this. Sounded like they have some experience what's necessary and what's the culprits of not having it in there. Maybe get in touch with them :).

Labels: Bug, Needs Triage
The libomp.dylib shipped with the macOS x86_64 package does not have an SDK version set

### Describe the bug

I want to build an macOS app that uses scikit-learn as a dependency. Using the arm64 package of scikit-learn for this works flawlessly. However, if I want to do the same using the macOS x86_64 packages Apple's notarizing step always breaks the app. This is likely due to the shipped libomp.dylib in the x86_64 package (installed using pip) does not have an SDK version set:
```
otool -l ./venv/lib/python3.12/site-packages/sklearn/.dylibs/libomp.dylib | grep -A 3 LC_VERSION_MIN_MACOSX
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.9
      sdk n/a
```
The arm64 version has this set:
```
otool -l ./venv/lib/python3.12/site-packages/sklearn/.dylibs/libomp.dylib | grep -A 4 LC_BUILD_VERSION
      cmd LC_BUILD_VERSION
  cmdsize 32
 platform 1
    minos 11.0
      sdk 11.0
```
It would be great, if you could set this (to at least 10.9; would probably need a rebuild of the dylib from source). I already tried some workarounds, but so far none have been successful. Is there any chance you would consider that :)?

### Steps/Code to Reproduce

```
% otool -l ./venv/lib/python3.12/site-packages/sklearn/.dylibs/libomp.dylib | grep -A 3 LC_VERSION_MIN_MACOSX
      cmd LC_VERSION_MIN_MACOSX
```

### Expected Results

```
  cmdsize 16
  version 10.9
      sdk 10.9
```

### Actual Results

```
  cmdsize 16
  version 10.9
      sdk n/a
```

### Versions

```shell
scikit-learn==1.6.1 (from pip freeze)
```

COMMENT:
I'm not sure if that file is built from scratch as part of the scikit-learn build. I suspect not though. This means asking "more upstream" is probably the right thing to do.

Please re-open if my assumption is wrong and it is indeed built from source as part of scikit-learn's build process

Labels: Bug, Needs Triage
Cannot recover DBSCAN from memory-overuse

### Describe the bug

I also just ran into this issue that the program gets killed when running DBSCAN, similar to:
https://github.com/scikit-learn/scikit-learn/issues/22531

The documentation update already helps and I think it's ok for the algorithm to fail. But currently there is no way for me to recover, and a more informative error message would be useful. Since now DBSCAN just reports `killed` and it requires a bit of search to see what fails:
```
>>> DBSCAN(eps=1, min_samples=2).fit(np.random.rand(10_000_000, 3))
Killed
```

e.g., something like how `numpy` does it:
```
>>> n = int(1e6)
>>> np.random.rand(n, n)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "numpy/random/mtrand.pyx", line 1219, in numpy.random.mtrand.RandomState.rand
  File "numpy/random/mtrand.pyx", line 437, in numpy.random.mtrand.RandomState.random_sample
  File "_common.pyx", line 307, in numpy.random._common.double_fill
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 7.28 TiB for an array with shape (1000000, 1000000) and data type float64
```

Additionally, I noted that the memory accumulated with consecutive calling of DBSCAN. Which can lead to a killed program even though there is enough memory when running a single fit.
I was able to resolve this by explicitly calling `import gc; gc.collect()` after each run. Maybe this could be invoked at the end of each DBSCAN fit?

### Steps/Code to Reproduce

```python
try:
    DBSCAN(eps=1, min_samples=2).fit(np.random.rand(10_000_000, 3))
except:
    print("Caught exception")
```


### Expected Results

```python
Caught exception
```

### Actual Results

```python
Killed
```

### Versions

```shell
>>> import sklearn; sklearn.show_versions()

System:
    python: 3.10.12 (main, Feb  4 2025, 14:57:36) [GCC 11.4.0]
executable: /usr/bin/python3
   machine: Linux-6.14.6-arch1-1-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.6.1
          pip: None
   setuptools: 80.7.1
        numpy: 1.26.4
        scipy: 1.15.3
       Cython: None
       pandas: 2.2.3
   matplotlib: 3.10.3
       joblib: 1.5.0
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Prescott

       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libscipy_openblas
       filepath: /usr/local/lib/python3.10/dist-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 20
         prefix: libgomp
       filepath: /usr/local/lib/python3.10/dist-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
I have made my conclusion https://github.com/scikit-learn/scikit-learn/pull/31526#discussion_r2142638083

Labels: Bug, help wanted, Hard, Needs Reproducible Code, Needs Investigation