Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thanks for reporting this issue. Could you please post a notebook that runs a comparative study on an example dataset where scikit-learn's VB-based estimators and Gibbs sampling-based estimators (from tomotopy or Mallet or both) yield significant performance measurements (e.g. in terms of likelihood of held-out data)?

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think I might have seen a blog post doing just this, but it would be great to replicate it. I will try to find it.

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
https://colab.research.google.com/github/adityasingh345/LDA_gibbs_sampling/blob/main/LDA_gibbs_sampling.ipynb

Could you please review this implementation and share your feedback or suggestions for improvement?

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thanks @adityasingh345. Could you please extend your study as suggested in https://github.com/scikit-learn/scikit-learn/issues/31925#issuecomment-3219361680 ?

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
https://colab.research.google.com/drive/1XsXqEIIErHibdFp_hmUAohXTlt96ZIyE?usp=sharing
@ogrisel 
As requested , i am sharing a google colab notebook comparing scikit learn's VB LDA and tomotopy's Gibbs Sampling LDA on 20 newsgroups. as its clearly shown gibbs sampling has lower value of perplexity.

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thanks for your work @adityasingh345 ! Can I ask you to give me access to the notebook, it does not allow me to see it as of now, and I suppose that is the case with others as well.

As far as I know perplexity can be a misleading score to look at sometimes. Can we perhaps also check with some topic quality metrics, like word embedding coherence and diversity? I have implementations for these in the [topic-benchmark](https://github.com/x-tabdeveloping/topic-benchmark/tree/main/topic_benchmark/metrics) repo.

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
https://colab.research.google.com/drive/1XsXqEIIErHibdFp_hmUAohXTlt96ZIyE?usp=sharing

hey @x-tabdeveloping  here is the link as you have said use for coherence and diversity i have tried that 
The Scikit-learn LDA model achieved higher topic diversity (0.77), suggesting its topics are more distinct and use less overlapping vocabulary.

The Tomotopy LDA model achieved slightly higher coherence (0.464 vs 0.442), suggesting its topics are semantically more meaningful.

This highlights the common trade-off in topic modeling: coherence vs. diversity. Depending on the application, one may prefer more interpretable topics (higher coherence) or more distinct topics (higher diversity).

please have a look @ogrisel

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@adityasingh345 I looked at the notebook and I have a couple of suggestions that could make your analysis better:
 
You are not using the same tokenization approach for all models, which might be responsible for much of the variation we see in performance. For instance, you are not removing overly frequent words and stop words for tomotopy. I suggest you create the vectorizer, and the use its `build_analyzer` method to create a tokenizer function, sort of like this:
```python
vectorizer = CountVectorizer(max_df=0.95, min_df=2, stop_words="english")
X = vectorizer.fit_transform(data.data)

tokenizer = vectorizer.build_analyzer()
tokens = [tokenizer(doc) for doc in data.data]
```
I suspect that much of the disparity in diversity and perplexity can be attributed to this.

In addition, you should probably run the analysis over multiple numbers of topics (let's say from 10 to 50) instead of doing a single run.
I think printing some topics would also make a lot of sense. I have worked quite a bit with topic model evaluation and these topic quality metrics are not exactly perfect. Some qualitative assessment is likely necessary.

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
You’re absolutely right that I didn’t use a consistent tokenization approach across models, and I can see how that might have caused the performance differences. I’ll update the code to build tokens from the same CountVectorizer analyzer so the comparison is fair.

I’ll also try running the experiments over a wider range of topics (10–50) and print out some representative topics for qualitative evaluation, since metrics like perplexity and diversity can sometimes be misleading.

thanks for pointing out . @x-tabdeveloping

Labels: New Feature, Needs Decision - Include Feature
Add a better implementation of Latent Dirichlet Allocation

### Describe the workflow you want to enable

While this remains to be rigorously tested, the scikit-learn implementation of Latent Dirichlet Allocation is, in the [unanimous experience of topic modelling scholars](https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html), outperformed by Gibbs-Sampling implementations, such as the ones in MALLET and tomotopy when it comes to topic quality. I have personally been criticised for using the scikit-learn implementation of LDA in my publications as a baseline, since other scholars do not think this implementation does justice to how well LDA can actually work in practice.
This is quite sad, since scikit-learn otherwise has a very authoritative position when it comes to machine learning, and many research and industry workflows build on your well-thought out and convenient API.

It would be of immense value for both industry and academia if Latent Dirichlet Allocation had multiple implementations, and preferably another one were the default.

### Describe your proposed solution

Include the implementation of LDA from the following publication:
[Distributed Algorithms for Topic Models](https://jmlr.org/papers/volume10/newman09a/newman09a.pdf)

This implementation has been around for a while, is used both in tomotopy and MALLET, is published in a reputable journal and has been cited more than 600 times according to Google Scholar.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I failed to find the blogpost I was thinking about, but the tomotopy documentation has a figure to compare the log-likelihood per word in VI and GS. As you can see the log-likelihood significantly goes down with higher number of topics with VI (note that the y axis is inverted), but stays roughly the same with GS.

<img width="1223" height="803" alt="Image" src="https://github.com/user-attachments/assets/ae4b8c80-a9d1-45e2-bea3-7e37f1e6bfd9" />

Labels: New Feature, Needs Decision - Include Feature
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev (last failure: Aug 10, 2025) ⚠️

**CI failed on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=78962&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Aug 10, 2025)
- test_dtype_preprocess_data[73-True-True]

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=79026&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf) on Aug 13, 2025

Labels: Needs Triage