Talip7/scikit-learn-issues-multilabel huggingface dataset:
ReliefF and RReliefF feature selectors

### Describe the workflow you want to enable

Consider to add ReliefF and RReliefF feature selectors to sklearn.feature_selection.

### Describe your proposed solution

Add ReliefF and RReliefF feature selectors.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Could you please motivate this feature request better by:

- linking to the relevant paper. I cannot find the PDF for RRelief paper in particular. The link from https://en.wikipedia.org/wiki/Relief_(feature_selection) and other sources is broken.
- please explain why it's useful to you;
- do you know any particular application domain / benchmark dataset where this method yields results that are much better than alternatives already implemented in scikit-learn. If that the case, please be specific about how you define "better" (e.g. in terms of a downstream supervised ML metric, speed, memory usage, or ideally a combination of both);
- please link to any existing working implementation that you like in Python or any other programming language that could server as reference.

Labels: New Feature
ReliefF and RReliefF feature selectors

### Describe the workflow you want to enable

Consider to add ReliefF and RReliefF feature selectors to sklearn.feature_selection.

### Describe your proposed solution

Add ReliefF and RReliefF feature selectors.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@ogrisel Please check the implementation [here](https://www.mathworks.com/help/stats/relieff.html)

Labels: New Feature
ReliefF and RReliefF feature selectors

### Describe the workflow you want to enable

Consider to add ReliefF and RReliefF feature selectors to sklearn.feature_selection.

### Describe your proposed solution

Add ReliefF and RReliefF feature selectors.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I've never heard of the algorithms, but the 3 articles cited in that page together have about 5000 citations. So one could argue they're pretty established. Seems like they've been included in Matlab since 2010.

The algorithms themselves seems quite simple as well. So I wouldn't be opposed to seeing a PR to see how it'd look.

Labels: New Feature
ReliefF and RReliefF feature selectors

### Describe the workflow you want to enable

Consider to add ReliefF and RReliefF feature selectors to sklearn.feature_selection.

### Describe your proposed solution

Add ReliefF and RReliefF feature selectors.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@ncooder are you aware of https://github.com/EpistasisLab/scikit-rebate (`pip install skrebate`) and have you tried it on your use case? See https://github.com/scikit-learn/scikit-learn/pull/29704#issuecomment-2306943089 for a bit more details.

Labels: New Feature
ReliefF and RReliefF feature selectors

### Describe the workflow you want to enable

Consider to add ReliefF and RReliefF feature selectors to sklearn.feature_selection.

### Describe your proposed solution

Add ReliefF and RReliefF feature selectors.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Seems like based on the discussion in https://github.com/scikit-learn/scikit-learn/pull/29704 we've decided against this.

Labels: New Feature
mRMR feature selector

### Describe the workflow you want to enable

Consider adding minimum redundancy maximum relevance (mRMR) feature selector to sklearn.feature_selection.

### Describe your proposed solution

mRMR as another feature selector. 

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
duplicate of https://github.com/scikit-learn/scikit-learn/issues/8889

Labels: New Feature, Needs Triage
DOC Update advanced installation instructions from macOS

I think we should kind of wait for the dust to settle on https://github.com/scikit-learn/scikit-learn/issues/29546 but I am pretty sure that with the improvements in OpenMP detection in Meson 1.5 https://github.com/mesonbuild/meson/pull/13350, you don't need to set any environment variables and that our [macOS installation doc](https://scikit-learn.org/dev/developers/advanced_installation.html#macos-compilers-from-homebrew) can be simplified.

We may be setting environment variable in our CI as well, this is worth a look if we can remove them.

cc @EmilyXinyi if you feel like working on it at one point.

I guess it's good to note that this is a positive side-effect of moving away from setuptools to Meson: some things just work better out of the box. I am certainly slightly biased but I am personally convinced that the cost of switching was worth it. Future will tell if I was wrong but I am reasonably confident about this :wink:.

COMMENT:
Thank you @lesteve I think you are right and I can update the macOS installation doc. In terms of the environment variable, I am not 100% sure in the CI but on my local at least, I think we need to update `test_openmp_parallelism_enabled` and any code associated with it.

Labels: Documentation
DOC Update advanced installation instructions from macOS

I think we should kind of wait for the dust to settle on https://github.com/scikit-learn/scikit-learn/issues/29546 but I am pretty sure that with the improvements in OpenMP detection in Meson 1.5 https://github.com/mesonbuild/meson/pull/13350, you don't need to set any environment variables and that our [macOS installation doc](https://scikit-learn.org/dev/developers/advanced_installation.html#macos-compilers-from-homebrew) can be simplified.

We may be setting environment variable in our CI as well, this is worth a look if we can remove them.

cc @EmilyXinyi if you feel like working on it at one point.

I guess it's good to note that this is a positive side-effect of moving away from setuptools to Meson: some things just work better out of the box. I am certainly slightly biased but I am personally convinced that the cost of switching was worth it. Future will tell if I was wrong but I am reasonably confident about this :wink:.

COMMENT:
Also for people on macOS M1 machines like @ogrisel and @glemaitre I am wondering whether this may be a way to avoid having to install `compilers` from conda-forge and avoid the ccache limitation with known but quirky work-arounds https://github.com/conda-forge/compilers-feedstock/issues/31.

Of course, there may be other caveats when switching between conda-forge `compilers` (current way of working for people on macOS machine I would say) to system clang with libomp installed with brew.

Labels: Documentation
DOC Update advanced installation instructions from macOS

I think we should kind of wait for the dust to settle on https://github.com/scikit-learn/scikit-learn/issues/29546 but I am pretty sure that with the improvements in OpenMP detection in Meson 1.5 https://github.com/mesonbuild/meson/pull/13350, you don't need to set any environment variables and that our [macOS installation doc](https://scikit-learn.org/dev/developers/advanced_installation.html#macos-compilers-from-homebrew) can be simplified.

We may be setting environment variable in our CI as well, this is worth a look if we can remove them.

cc @EmilyXinyi if you feel like working on it at one point.

I guess it's good to note that this is a positive side-effect of moving away from setuptools to Meson: some things just work better out of the box. I am certainly slightly biased but I am personally convinced that the cost of switching was worth it. Future will tell if I was wrong but I am reasonably confident about this :wink:.

COMMENT:
> In terms of the environment variable, I am not 100% sure in the CI but on my local at least, I think we need to update `test_openmp_parallelism_enabled` and any code associated with it.

So if you build locally scikit-learn without OpenMP support and you run all the tests, you need to set the environment variable `SKLEARN_SKIP_OPENMP_TEST` to skip this particular test. This is what is done in the CI indeed, see https://github.com/scikit-learn/scikit-learn/blob/45cf8ec555a026c4263e8bef12850755a83df10e/azure-pipelines.yml#L265-L270

Labels: Documentation
Please extend the loss-functions in SGDRegressor to allow incremental learning of Poisson / Gamma / Tweedie regressors

### Describe the workflow you want to enable

I would like SGDRegressor to be able to accept additional values in its 'loss' arguments to be able to incrementally train additional regressors, that are currently available in SkLearn in the form of non-incremental regressors (PoissonRegressor / GammaRegressor/TweedieRegressor).

### Describe your proposed solution

Add additional loss function implementations (providing gradients) to support the additional regressor types.

### Describe alternatives you've considered, if relevant

Learning with L1/L2 loss in log-space. But it doesn't work well when the label of a given sample is zero. Poisson regression handles it gracefully.

### Additional context

_No response_

COMMENT:
That should be doable indeed now that `SGDRegressor` has been refactored to reuse the loss functions from the shared `sklearn.linear_model._loss` module as part of #28049.

Any opinion @lorentzenchr?

Labels: New Feature, Moderate, module:linear_model
Please extend the loss-functions in SGDRegressor to allow incremental learning of Poisson / Gamma / Tweedie regressors

### Describe the workflow you want to enable

I would like SGDRegressor to be able to accept additional values in its 'loss' arguments to be able to incrementally train additional regressors, that are currently available in SkLearn in the form of non-incremental regressors (PoissonRegressor / GammaRegressor/TweedieRegressor).

### Describe your proposed solution

Add additional loss function implementations (providing gradients) to support the additional regressor types.

### Describe alternatives you've considered, if relevant

Learning with L1/L2 loss in log-space. But it doesn't work well when the label of a given sample is zero. Poisson regression handles it gracefully.

### Additional context

_No response_

COMMENT:
Very easy. Basically:
- Allow a new loss function
- Call inverse link function in `predict`
- Think about tests

Labels: New Feature, Moderate, module:linear_model
Please extend the loss-functions in SGDRegressor to allow incremental learning of Poisson / Gamma / Tweedie regressors

### Describe the workflow you want to enable

I would like SGDRegressor to be able to accept additional values in its 'loss' arguments to be able to incrementally train additional regressors, that are currently available in SkLearn in the form of non-incremental regressors (PoissonRegressor / GammaRegressor/TweedieRegressor).

### Describe your proposed solution

Add additional loss function implementations (providing gradients) to support the additional regressor types.

### Describe alternatives you've considered, if relevant

Learning with L1/L2 loss in log-space. But it doesn't work well when the label of a given sample is zero. Poisson regression handles it gracefully.

### Additional context

_No response_

COMMENT:
Hi, could I take up this issue?

Labels: New Feature, Moderate, module:linear_model
Please extend the loss-functions in SGDRegressor to allow incremental learning of Poisson / Gamma / Tweedie regressors

### Describe the workflow you want to enable

I would like SGDRegressor to be able to accept additional values in its 'loss' arguments to be able to incrementally train additional regressors, that are currently available in SkLearn in the form of non-incremental regressors (PoissonRegressor / GammaRegressor/TweedieRegressor).

### Describe your proposed solution

Add additional loss function implementations (providing gradients) to support the additional regressor types.

### Describe alternatives you've considered, if relevant

Learning with L1/L2 loss in log-space. But it doesn't work well when the label of a given sample is zero. Poisson regression handles it gracefully.

### Additional context

_No response_

COMMENT:
/take

Labels: New Feature, Moderate, module:linear_model
sklearn/cluster/_optics File Size and Complexity

### Problem

The [sklearn/cluster/_optics.py](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/cluster/_optics.py) file in our project has grown to over 1200 lines of code. This size can make it difficult to navigate, maintain, and understand. The current structure includes both the main Optics class and numerous helper functions, all within the same file. Having a large file like this can introduce several issues:
- **Maintenance Complexity:** It becomes challenging to locate specific pieces of code and understand their relationships.
- **Limited Modularity:** The file seems to be handling multiple responsibilities, which could be separated to enhance code clarity and modularity.
- **Difficulties in Testing:** Testing individual components can be more complex when they are tightly coupled in a single file.

### Proposed Solution 
To address these issues, I propose the following:
- **Separation of Concerns:** Move the Optics class into its own file (e.g., cluster/optics_class.py).
- **Modularize Helper Functions:** Extract the helper functions into a separate module (e.g., cluster/optics_helpers.py). This module can then be imported into cluster/optics_class.py as needed.

COMMENT:
Thanks for the issue and the PRs. However, 1200 lines is really not that large for our project's conventions. We have files a lot larger than this. And since those functions are only used inside the file itself, I don't see why we'd move them. So closing unless others think otherwise.

Labels: Needs Triage
BaggingRegressor with **fit_params with CatBoostRegressor fit(..., eval_set= ())

### Describe the issue linked to the documentation

How can we use the new **fit_params of the BaggingRegressor  to add the eval_set of Catboost or LightGBM when calling the .fit() function ? The metadata routing documentation is incomplete about this !

### Suggest a potential alternative/fix

_No response_

COMMENT:
I don't think the metadata routing feature can be used to leverage extensions to the scikit-learn API in third party libraries.

However, I agree that we need a functional way to specify and route for early stopping validation sets in pipelines and other meta-estimators.

There are ongoing discussions and candidate implementation here:

- #18748
- #28440
- #28901

Labels: Documentation
BaggingRegressor with **fit_params with CatBoostRegressor fit(..., eval_set= ())

### Describe the issue linked to the documentation

How can we use the new **fit_params of the BaggingRegressor  to add the eval_set of Catboost or LightGBM when calling the .fit() function ? The metadata routing documentation is incomplete about this !

### Suggest a potential alternative/fix

_No response_

COMMENT:
Let's close as duplicate.

Labels: Documentation
Replace confusing buildtype=debugoptimized by buildtype=release in meson.build

So we use `buildtype=debugoptimized` mostly because I took it from the other projects I looked at (numpy, scipy, scikit-image) without understanding too much what this meant.

https://github.com/scikit-learn/scikit-learn/blob/70a84ea2db68d3cab1df30ed374445be2ac67dd4/meson.build#L8

All the scikit-learn developers are going through `pip` to build scikit-learn and I have recently realised that when you go through `pip` meson-python sets the buildtype=release see https://mesonbuild.com/meson-python/explanations/default-options.html. This makes buildtype=debugoptimized a bit confusing since it is actually almost never used, so should be use `buildtype=release` instead? For it to be used you would need to do `meson setup` + `ninja` manually which happens super rarely in a scikit-learn context.

Contrast this with most other projects that historically (my crude understanding at least) use out-of-tree build i.e. for example `spin build` + set `PYTHONPATH` when doing `spin test` (although some of the developers use editable installs for example because better integration into IDE/editors I think).

For these other projects, it may make sense to have developer build in debugoptimized mode and let `meson-python` build in release for wheels. I guess that's a possible reason for this, but I am not sure ....

cc @rgommers in case you have some advice about this.

cc @adam2392 because you asked the question in our Discord https://discord.com/channels/731163543038197871/1046822941586898974/1263909669181722655

COMMENT:
Thanks for summarizing and looking into this!

> All the scikit-learn developers are going through `pip` to build scikit-learn and I have recently realised that when you go through `pip` meson-python sets the buildtype=release see https://mesonbuild.com/meson-python/explanations/default-options.html. This makes buildtype=debugoptimized a bit confusing since it is actually almost never used, so should be use `buildtype=release` instead? For it to be used you would need to do `meson setup` + `ninja` manually which happens super rarely in a scikit-learn context.
> Contrast this with most other projects that historically (my crude understanding at least) use out-of-tree build i.e. for example `spin build` + set `PYTHONPATH` when doing `spin test` (although some of the developers use editable installs for example because better integration into IDE/editors I think).

Interesting. I did not know this. I maintain another package that is downstream of scikit-learn and copied the "boiler plate" and unknowingly added the `buildtype=debugoptimized` option as well. But I go through `spin`, so does this imply that the resulting compilation is suboptimal? Vs if I went thru pip, it would be fine (i.e. what scikit-learn `make dev-meson` does)? 

Seems like a point of confusion to float up somewhere upstream if that's the case.

Labels: Build / CI
Replace confusing buildtype=debugoptimized by buildtype=release in meson.build

So we use `buildtype=debugoptimized` mostly because I took it from the other projects I looked at (numpy, scipy, scikit-image) without understanding too much what this meant.

https://github.com/scikit-learn/scikit-learn/blob/70a84ea2db68d3cab1df30ed374445be2ac67dd4/meson.build#L8

All the scikit-learn developers are going through `pip` to build scikit-learn and I have recently realised that when you go through `pip` meson-python sets the buildtype=release see https://mesonbuild.com/meson-python/explanations/default-options.html. This makes buildtype=debugoptimized a bit confusing since it is actually almost never used, so should be use `buildtype=release` instead? For it to be used you would need to do `meson setup` + `ninja` manually which happens super rarely in a scikit-learn context.

Contrast this with most other projects that historically (my crude understanding at least) use out-of-tree build i.e. for example `spin build` + set `PYTHONPATH` when doing `spin test` (although some of the developers use editable installs for example because better integration into IDE/editors I think).

For these other projects, it may make sense to have developer build in debugoptimized mode and let `meson-python` build in release for wheels. I guess that's a possible reason for this, but I am not sure ....

cc @rgommers in case you have some advice about this.

cc @adam2392 because you asked the question in our Discord https://discord.com/channels/731163543038197871/1046822941586898974/1263909669181722655

COMMENT:
> For these other projects, it may make sense to have developer build in debugoptimized mode and let `meson-python` build in release for wheels. I guess that's a possible reason for this, but I am not sure ....

Yes exactly.

For more context:
- `meson-python` is optimized for releases, and the explicit setting was deemed necessary to ensure that a `pip install some-pkg` from source would get an optimized build
- the numpy/scipy default of `debugoptimized` I chose because:
  - I wanted debug symbols by default for a dev build, since we have a lot of compiled code
  - I wanted reasonable performance by default for benchmarking, so no plain `debug` (which means `-O0`)
  - It matches what distutils used to do (it hardcodes flags, the default is `-O2 -g` for GCC)

Editable installs inherently give you less choice here as a result. If those are used a lot, then I'd probably choose to not specify any default in `meson.build` so that there's no divergence between editable installs and `spin` or plain `meson setup` usage.

Labels: Build / CI
Replace confusing buildtype=debugoptimized by buildtype=release in meson.build

So we use `buildtype=debugoptimized` mostly because I took it from the other projects I looked at (numpy, scipy, scikit-image) without understanding too much what this meant.

https://github.com/scikit-learn/scikit-learn/blob/70a84ea2db68d3cab1df30ed374445be2ac67dd4/meson.build#L8

All the scikit-learn developers are going through `pip` to build scikit-learn and I have recently realised that when you go through `pip` meson-python sets the buildtype=release see https://mesonbuild.com/meson-python/explanations/default-options.html. This makes buildtype=debugoptimized a bit confusing since it is actually almost never used, so should be use `buildtype=release` instead? For it to be used you would need to do `meson setup` + `ninja` manually which happens super rarely in a scikit-learn context.

Contrast this with most other projects that historically (my crude understanding at least) use out-of-tree build i.e. for example `spin build` + set `PYTHONPATH` when doing `spin test` (although some of the developers use editable installs for example because better integration into IDE/editors I think).

For these other projects, it may make sense to have developer build in debugoptimized mode and let `meson-python` build in release for wheels. I guess that's a possible reason for this, but I am not sure ....

cc @rgommers in case you have some advice about this.

cc @adam2392 because you asked the question in our Discord https://discord.com/channels/731163543038197871/1046822941586898974/1263909669181722655

COMMENT:
Has anyone measured the practical impact of building scikit-learn and scipy with `debugoptimized` vs `release` on:

- the .whl file size and the total package folder size under `site-packages`?
- the ability to get meaningful `backtrace` outputs in `gdb` in case of post-mortem investigations of segfaults (or interactive debugging sessions with manual break points)?

Is there a way to pass extra command-line arguments to `pip install --editable` so that the user can decide between the two without having to edit a version controlled file in the source tree?

Labels: Build / CI