Talip7/scikit-learn-issues-multilabel huggingface dataset:
Docs instructions for installing  LLVM OpenMP with Homebrew may need updating

### Describe the issue linked to the documentation

Environment variables CFLAGS, CXXFLAGS, CXXFLAGS mentioned here:
https://scikit-learn.org/dev/developers/advanced_installation.html#compiler-macos:~:text=Set%20the%20following%20environment%20variables%3A
may be for Intel-based Macs only.

So when trying to do this:
```
make clean
pip install --editable . \
    --verbose --no-build-isolation \
    --config-settings editable-verbose=true
```
I got  `../../meson.build:1:0: ERROR: Compiler /usr/bin/clang cannot compile programs.`

The reason being that `Homebrew` installed `libomp` here: `/opt/homebrew/opt/libomp` and not here`/usr/local/opt/libomp/`.


### Suggest a potential alternative/fix

Modify the env variables that I mentioned above to the right path to `libomp` for M2 macs.

Please note:

- I'm not sure if the variables should be updated or have the two mac versions (Intel vs M1/M2).
- I didn't test that all works for an Intel mac. 
- Modifying the variables to the correct path, I was able to make the new environment.

COMMENT:
```
(sklearn-env-test) *[main][~/projects/scikit-learn]$ python -c 'import sklearn; sklearn.show_versions()'
meson-python: building scikit-learn: /Users/dealeon/projects/scikit-learn/sklearn-env-test/bin/ninja
[3/3] Linking target sklearn/linear_model/_cd_fast.cpython-313-darwin.so

System:
    python: 3.13.6 (v3.13.6:4e665351082, Aug  6 2025, 11:22:35) [Clang 16.0.0 (clang-1600.0.26.6)]
executable: /Users/dealeon/projects/scikit-learn/sklearn-env-test/bin/python
   machine: macOS-15.6-arm64-arm-64bit-Mach-O

Python dependencies:
      sklearn: 1.8.dev0
          pip: 25.2
   setuptools: None
        numpy: 2.3.2
        scipy: 1.16.1
       Cython: 3.1.2
       pandas: None
   matplotlib: None
       joblib: 1.5.1
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /opt/homebrew/Cellar/libomp/20.1.8/lib/libomp.dylib
        version: None
```

The thing with `pytest` is unrelated but some step may be missing in the instructions. Please see:

```
(sklearn-env-test) *[main][~/projects/scikit-learn]$ which pytest
/Library/Frameworks/Python.framework/Versions/3.12/bin/pytest
```
and 
```
(sklearn-env-test) *[main][~/projects/scikit-learn]$ python -m pytest
/Users/dealeon/projects/scikit-learn/sklearn-env-test/bin/python: No module named pytest
```
I have seen this issue before, but didn't spend the time to investigate. I just went back to an old environment that worked. 😅

If I do the same on the other environment (setting the environment variables myself): 
```
(sklearn-envAug2025) *[main][~/projects/scikit-learn]$ which pytest
/Users/dealeon/projects/scikit-learn/sklearn-envAug2025/bin/pytest
```

I can still clean up the docs removing the instructions that are not needed. 😇

Labels: Documentation
Docs instructions for installing  LLVM OpenMP with Homebrew may need updating

### Describe the issue linked to the documentation

Environment variables CFLAGS, CXXFLAGS, CXXFLAGS mentioned here:
https://scikit-learn.org/dev/developers/advanced_installation.html#compiler-macos:~:text=Set%20the%20following%20environment%20variables%3A
may be for Intel-based Macs only.

So when trying to do this:
```
make clean
pip install --editable . \
    --verbose --no-build-isolation \
    --config-settings editable-verbose=true
```
I got  `../../meson.build:1:0: ERROR: Compiler /usr/bin/clang cannot compile programs.`

The reason being that `Homebrew` installed `libomp` here: `/opt/homebrew/opt/libomp` and not here`/usr/local/opt/libomp/`.


### Suggest a potential alternative/fix

Modify the env variables that I mentioned above to the right path to `libomp` for M2 macs.

Please note:

- I'm not sure if the variables should be updated or have the two mac versions (Intel vs M1/M2).
- I didn't test that all works for an Intel mac. 
- Modifying the variables to the correct path, I was able to make the new environment.

COMMENT:
OK thanks so it looks like the build does find OpenMP (`Built with OpenMP: True`) and meson finds correctly the OpenMP installed with brew.

This confirms we can remove the setting of the environment variable from the doc [^1] and remove the mention 
> For Apple Silicon M1 hardware, only the conda-forge method below is known to work at the time of writing

[^1]: for completeness I did try a while ago on a macOS Intel VM I had access to and it worked without setting the env variable.

> I can still clean up the docs removing the instructions that are not needed. 😇

Glad to hear that 🙏! Don't hesitate to ping me when you have a PR ready!

This seems like what I guessed about your pytest issue. You need to install `pytest` in your venv for example with `python -m pip install pytest`. What happens is that `pytest` is found from another place than your `venv` (`/Library/Frameworks/Python.framework/Versions/3.12/bin/pytest` maybe your system Python on macOS, not 100% sure) so that `pytest` actually does use this other Python environment that does not have scikit-learn installed. This is a common trap and `python -m pytest` is a bit less misleading.

Labels: Documentation
Docs instructions for installing  LLVM OpenMP with Homebrew may need updating

### Describe the issue linked to the documentation

Environment variables CFLAGS, CXXFLAGS, CXXFLAGS mentioned here:
https://scikit-learn.org/dev/developers/advanced_installation.html#compiler-macos:~:text=Set%20the%20following%20environment%20variables%3A
may be for Intel-based Macs only.

So when trying to do this:
```
make clean
pip install --editable . \
    --verbose --no-build-isolation \
    --config-settings editable-verbose=true
```
I got  `../../meson.build:1:0: ERROR: Compiler /usr/bin/clang cannot compile programs.`

The reason being that `Homebrew` installed `libomp` here: `/opt/homebrew/opt/libomp` and not here`/usr/local/opt/libomp/`.


### Suggest a potential alternative/fix

Modify the env variables that I mentioned above to the right path to `libomp` for M2 macs.

Please note:

- I'm not sure if the variables should be updated or have the two mac versions (Intel vs M1/M2).
- I didn't test that all works for an Intel mac. 
- Modifying the variables to the correct path, I was able to make the new environment.

COMMENT:
Thanks for the `pytest` advice. 
Just for info: 

The version of Python on macOS that I have is 3.13, not 3.12 like the one we see here:
> pytest is found from another place than your venv (/Library/Frameworks/Python.framework/Versions/3.12/bin/pytest
maybe your system Python on macOS, not 100% sure

And when setting the env variables (the step we don't need any more), the correct `pytest` is found - the one in the activated environment. 
It's just a "side note". 🙂

Labels: Documentation
Docs instructions for installing  LLVM OpenMP with Homebrew may need updating

### Describe the issue linked to the documentation

Environment variables CFLAGS, CXXFLAGS, CXXFLAGS mentioned here:
https://scikit-learn.org/dev/developers/advanced_installation.html#compiler-macos:~:text=Set%20the%20following%20environment%20variables%3A
may be for Intel-based Macs only.

So when trying to do this:
```
make clean
pip install --editable . \
    --verbose --no-build-isolation \
    --config-settings editable-verbose=true
```
I got  `../../meson.build:1:0: ERROR: Compiler /usr/bin/clang cannot compile programs.`

The reason being that `Homebrew` installed `libomp` here: `/opt/homebrew/opt/libomp` and not here`/usr/local/opt/libomp/`.


### Suggest a potential alternative/fix

Modify the env variables that I mentioned above to the right path to `libomp` for M2 macs.

Please note:

- I'm not sure if the variables should be updated or have the two mac versions (Intel vs M1/M2).
- I didn't test that all works for an Intel mac. 
- Modifying the variables to the correct path, I was able to make the new environment.

COMMENT:
> And when setting the env variables (the step we don't need any more), the correct pytest is found

Hmmm that doesn't make too much sense to me, but 🤷. These environment variables are about how to build scikit-learn. Your issue seems to me about pytest not installed in your venv and/or where the `pytest` command is found (e.g. look at your `$PATH`) ...

Labels: Documentation
Docs instructions for installing  LLVM OpenMP with Homebrew may need updating

### Describe the issue linked to the documentation

Environment variables CFLAGS, CXXFLAGS, CXXFLAGS mentioned here:
https://scikit-learn.org/dev/developers/advanced_installation.html#compiler-macos:~:text=Set%20the%20following%20environment%20variables%3A
may be for Intel-based Macs only.

So when trying to do this:
```
make clean
pip install --editable . \
    --verbose --no-build-isolation \
    --config-settings editable-verbose=true
```
I got  `../../meson.build:1:0: ERROR: Compiler /usr/bin/clang cannot compile programs.`

The reason being that `Homebrew` installed `libomp` here: `/opt/homebrew/opt/libomp` and not here`/usr/local/opt/libomp/`.


### Suggest a potential alternative/fix

Modify the env variables that I mentioned above to the right path to `libomp` for M2 macs.

Please note:

- I'm not sure if the variables should be updated or have the two mac versions (Intel vs M1/M2).
- I didn't test that all works for an Intel mac. 
- Modifying the variables to the correct path, I was able to make the new environment.

COMMENT:
Maybe I did something different when I was following the steps.. Like skipping to install it for example?! 

This issue should have been closed with my PR. I'll close it now. 
Thanks.

Labels: Documentation
Docs instructions for installing  LLVM OpenMP with Homebrew may need updating

### Describe the issue linked to the documentation

Environment variables CFLAGS, CXXFLAGS, CXXFLAGS mentioned here:
https://scikit-learn.org/dev/developers/advanced_installation.html#compiler-macos:~:text=Set%20the%20following%20environment%20variables%3A
may be for Intel-based Macs only.

So when trying to do this:
```
make clean
pip install --editable . \
    --verbose --no-build-isolation \
    --config-settings editable-verbose=true
```
I got  `../../meson.build:1:0: ERROR: Compiler /usr/bin/clang cannot compile programs.`

The reason being that `Homebrew` installed `libomp` here: `/opt/homebrew/opt/libomp` and not here`/usr/local/opt/libomp/`.


### Suggest a potential alternative/fix

Modify the env variables that I mentioned above to the right path to `libomp` for M2 macs.

Please note:

- I'm not sure if the variables should be updated or have the two mac versions (Intel vs M1/M2).
- I didn't test that all works for an Intel mac. 
- Modifying the variables to the correct path, I was able to make the new environment.

COMMENT:
> Maybe I did something different when I was following the steps.. Like skipping to install it for example?!

That would be one of my guess indeed

Labels: Documentation
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thanks for raising this. I also noticed that topic quality from the VB-based LDA in scikit-learn is sometimes weaker compared to Gibbs Sampling implementations like MALLET. as i understand, the main request here is to explore integrating a Gibbs Sampling-based LDA implementation into sklearn possibly side by side with VB. Should we prefer different class like LatentDirichletAllocationGS or something like solver = 'gs  option in the current LDA implementation.                                                                                                         I’d be interested in experimenting with a simple Python prototype of Gibbs Sampling LDA to compare outputs. Would such a prototype be useful to move this discussion forward?

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi @adityasingh345 , I was thinking that adding something like a `solver="gs"` would be nice and consistent with other scikit-learn classes. I think creating a prototype would be immensely helpful. I have seen some Python implementations of the collapsed gibbs sampler for LDA on the internet with permissive licenses. Maybe we could take inspiration from there? I can link some of them a bit later if that can help you.

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thanks for the suggestion! 🙌 I agree that adding a solver="gs" option would be consistent with other scikit-learn estimators. My idea is to extend the existing LatentDirichletAllocation class by introducing Gibbs Sampling as an additional inference backend, while keeping the current batch and online variational solvers intact.

Conceptually, the interface would remain the same (fit/transform/partial_fit), but internally the E-step would switch to collapsed Gibbs updates when solver="gs". The main differences would be:

maintaining word-topic and doc-topic count matrices instead of variational parameters,

running Gibbs sweeps for max_iter or until convergence,

exposing reproducibility through a random_state parameter,

and returning topic-word distributions normalized from counts for compatibility with existing outputs.

I can start by prototyping a simple Gibbs sampler version in pure NumPy to validate the API and later align it with scikit-learn’s estimator structure.

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
https://github.com/adityasingh345/LDA_gibbs_sampling/tree/main
Hi all,

I’ve implemented a simple LDA Gibbs Sampling version in Google Colab to experiment with the concerns raised about topic quality in scikit-learn’s current LatentDirichletAllocation

The notebook includes:

A basic Gibbs sampler implementation (NumPy-based, no extra dependencies).

Step-by-step explanation of the update equations.

Small demo documents + generated topics.

I’m sharing this as a reference implementation and to invite feedback. Do you think it could be useful for:

Benchmarking scikit-learn’s LDA against Gibbs sampling?

Guiding improvements or discussions around adding a Gibbs-based variant in the library?

Happy to refine this further or run larger comparisons if that would be helpful.

Thanks!

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
You are very fast @adityasingh345 ! Thanks for sending the implementation.

I have highlighted some inconsistencies with the API in [this issue](https://github.com/adityasingh345/LDA_gibbs_sampling/issues/1).

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Actually I am working on this from one week

Labels: New Feature, Needs Decision - Include Feature