Talip7/scikit-learn-issues-multilabel huggingface dataset:
Dirichlet Multinomial Mixture Model

### Describe the workflow you want to enable

Is there an intention to implement the Dirichlet Multinomial Mixture Model with EM algorithm?

Dirichlet Multinomial Mixture Model is a popular clustering model in NLP, Information Retrieval and Bioinformatics. 

### Describe your proposed solution

I have implemented a usable Dirichlet Multinomial Mixture Model with the Base Mixture class in scikit-learn. If it is needed, I can refine the doc and details and pull it to scikit-learn.

https://github.com/Jck-R/pyDIMM

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hey there @Jck-R, thanks for opening an issue for this. Regarding inclusion of the model, I believe there's enough credibility to the underlying model to warrant consideration, however there's a few things we would like to see to make inclusion easier or more likely:

1. Do you have meaningful use cases in which this model is state of the art, or at least competitive? I'm familiar with some of its use in bioinformatics, however especially with NLP it seems to have fallen a bit out of favor (from what I've seen).
2. Could you provide testing associated with your implementation? Rigorous tests on the underlying mechanisms are especially useful in reviewing and understanding the algorithm.
3. Could you provide example uses of this algorithm on datasets, comparing against existing scikit-learn estimators which demonstrate that this proposed model is, in some sense, superior or preferable?
4. Could you provide benchmarks regarding memory footprint and computation time to achieve competitive results? Of course the implementation will be optimized over time (e.g. during review for inclusion), but it is an important sanity-check to ensure that it is not _wildly_ expensive.

Please let me know if you have any questions or concerns. Once again, thank you for proposing this and including your sample implementation :)

Labels: New Feature, module:cluster, Needs Decision - Include Feature
Dirichlet Multinomial Mixture Model

### Describe the workflow you want to enable

Is there an intention to implement the Dirichlet Multinomial Mixture Model with EM algorithm?

Dirichlet Multinomial Mixture Model is a popular clustering model in NLP, Information Retrieval and Bioinformatics. 

### Describe your proposed solution

I have implemented a usable Dirichlet Multinomial Mixture Model with the Base Mixture class in scikit-learn. If it is needed, I can refine the doc and details and pull it to scikit-learn.

https://github.com/Jck-R/pyDIMM

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think this would be very useful. The ideal implementation would also include a non-parametric version based on Dirichlet process, same as [BayesianGaussianMixture](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.BayesianGaussianMixture.html), and would have the same interface.

Labels: New Feature, module:cluster, Needs Decision - Include Feature
Transformer for nominal categories, with the goal of improving category support in decision trees

I'd like to find out how keen people would be for adding a transformer like this.

### Describe the workflow you want to enable

Improved support for nominal categories in tree based models. Nominal categories are ones with no order to them, for example colour or country (c.f. ordinal categories that are ordered).

### Describe your proposed solution

In #12866 @amueller linked to [Splitting on categorical predictors in random forests](https://peerj.com/articles/6339). This proposes to replace categorical values (red, green, blue, ..) by cleverly computed numerical values (blue -> 0.643, green ->0.123, ...) that allow you to achieve similar/equal performance by using `<` as splitting decision in a tree node compared to performing an exhaustive search of all possible categorical splits.

I would implement this as a transformer that you use together with a random forest or other tree based model in a pipeline.

### Describe alternatives you've considered, if relevant

There have been several PRs attempting to add native category support to decision trees. #12866 is the latest one. My impression is that it would be cool to have this in trees but that it is not easy to do, several people have tried but no PR has landed yet. The "Breimann trick" that is used is also limited in the number of categorical values it supports, where this new idea seems to support unlimited categorical values and multi class classification.

### Additional context

An earlier paper that is cited in "Splitting categorical predictors...": https://link.springer.com/article/10.1023/A:1009869804967

An open question for me is how to measure how good this kind of transformation is compared to the exhaustive split (which is "the best possible" but too expensive to compute to be used in practice). The method proposed by the paper seems simple (famous last words), and they make claims about the performance but it would be nice to know how close this gets us. Trust, but verify ;)

COMMENT:
Some notes on how to compute the various quantities. Mostly so I don't lose them. Based on the example given in "Splitting categorical predictors ..".

```python
# contingency matrix, given in the example, would have to construct it
labels = ["PS", "NP", "PR"]  # unused, just for completeness
categories = ["G","B","P","DO","LO","Y","R"]

C = np.array([[20, 3, 1], [19,4,2], [25,4,7], [64,4,20], [16,1,5], [100,2,3], [200,3,4]])
P = (C / C.sum(axis=1).reshape(-1,1))
pbar = C.sum(axis=0) / C.sum()

sigma = np.zeros((3,3))
for a in range(P.shape[0]):
  sigma +=  (C.sum(axis=1)[a] * np.outer((P[a, :] - pbar), (P[a,:] - pbar).T))

w, v = np.linalg.eig(sigma)
v0 = v[:, np.argmax(w)]

for a, category in enumerate(categories):
    print(category, a, v0 @ P[a, :])
```

Each categorical value of a feature should be replaced by `v0 @P[a, :]`, where `a` is the index that corresponds to the value of the category.

Labels: New Feature
Transformer for nominal categories, with the goal of improving category support in decision trees

I'd like to find out how keen people would be for adding a transformer like this.

### Describe the workflow you want to enable

Improved support for nominal categories in tree based models. Nominal categories are ones with no order to them, for example colour or country (c.f. ordinal categories that are ordered).

### Describe your proposed solution

In #12866 @amueller linked to [Splitting on categorical predictors in random forests](https://peerj.com/articles/6339). This proposes to replace categorical values (red, green, blue, ..) by cleverly computed numerical values (blue -> 0.643, green ->0.123, ...) that allow you to achieve similar/equal performance by using `<` as splitting decision in a tree node compared to performing an exhaustive search of all possible categorical splits.

I would implement this as a transformer that you use together with a random forest or other tree based model in a pipeline.

### Describe alternatives you've considered, if relevant

There have been several PRs attempting to add native category support to decision trees. #12866 is the latest one. My impression is that it would be cool to have this in trees but that it is not easy to do, several people have tried but no PR has landed yet. The "Breimann trick" that is used is also limited in the number of categorical values it supports, where this new idea seems to support unlimited categorical values and multi class classification.

### Additional context

An earlier paper that is cited in "Splitting categorical predictors...": https://link.springer.com/article/10.1023/A:1009869804967

An open question for me is how to measure how good this kind of transformation is compared to the exhaustive split (which is "the best possible" but too expensive to compute to be used in practice). The method proposed by the paper seems simple (famous last words), and they make claims about the performance but it would be nice to know how close this gets us. Trust, but verify ;)

COMMENT:
> An open question for me is how to measure how good this kind of transformation is compared to the exhaustive split 

I assume that we should at least benchmark as in:

https://scikit-learn.org/dev/auto_examples/ensemble/plot_gradient_boosting_categorical.html#sphx-glr-auto-examples-ensemble-plot-gradient-boosting-categorical-py

I am not against having this encoder until we can show that it works in practice and that it kind of scales. However, I think that we should tend to have a similar user experience for tree-based models (DT and RF, even exact GBDT) as the HGBDT, meaning going towards the implementation proposed in https://github.com/scikit-learn/scikit-learn/pull/24907, where the magic happens within the trees.

Labels: New Feature
Transformer for nominal categories, with the goal of improving category support in decision trees

I'd like to find out how keen people would be for adding a transformer like this.

### Describe the workflow you want to enable

Improved support for nominal categories in tree based models. Nominal categories are ones with no order to them, for example colour or country (c.f. ordinal categories that are ordered).

### Describe your proposed solution

In #12866 @amueller linked to [Splitting on categorical predictors in random forests](https://peerj.com/articles/6339). This proposes to replace categorical values (red, green, blue, ..) by cleverly computed numerical values (blue -> 0.643, green ->0.123, ...) that allow you to achieve similar/equal performance by using `<` as splitting decision in a tree node compared to performing an exhaustive search of all possible categorical splits.

I would implement this as a transformer that you use together with a random forest or other tree based model in a pipeline.

### Describe alternatives you've considered, if relevant

There have been several PRs attempting to add native category support to decision trees. #12866 is the latest one. My impression is that it would be cool to have this in trees but that it is not easy to do, several people have tried but no PR has landed yet. The "Breimann trick" that is used is also limited in the number of categorical values it supports, where this new idea seems to support unlimited categorical values and multi class classification.

### Additional context

An earlier paper that is cited in "Splitting categorical predictors...": https://link.springer.com/article/10.1023/A:1009869804967

An open question for me is how to measure how good this kind of transformation is compared to the exhaustive split (which is "the best possible" but too expensive to compute to be used in practice). The method proposed by the paper seems simple (famous last words), and they make claims about the performance but it would be nice to know how close this gets us. Trust, but verify ;)

COMMENT:
From a quick read of #18894 which seems to be the "source" of #24907 the proposal there is to use a category encoder inside hist gradient boosting. This means users won't have to create a pipeline anymore to do the encoding themselves, the estimator takes care of it (kinda like a "built in" pipeline). Having an additional transformer like the one proposed here that can deal with nominal (no order) categories seems like a good thing.

Labels: New Feature
Transformer for nominal categories, with the goal of improving category support in decision trees

I'd like to find out how keen people would be for adding a transformer like this.

### Describe the workflow you want to enable

Improved support for nominal categories in tree based models. Nominal categories are ones with no order to them, for example colour or country (c.f. ordinal categories that are ordered).

### Describe your proposed solution

In #12866 @amueller linked to [Splitting on categorical predictors in random forests](https://peerj.com/articles/6339). This proposes to replace categorical values (red, green, blue, ..) by cleverly computed numerical values (blue -> 0.643, green ->0.123, ...) that allow you to achieve similar/equal performance by using `<` as splitting decision in a tree node compared to performing an exhaustive search of all possible categorical splits.

I would implement this as a transformer that you use together with a random forest or other tree based model in a pipeline.

### Describe alternatives you've considered, if relevant

There have been several PRs attempting to add native category support to decision trees. #12866 is the latest one. My impression is that it would be cool to have this in trees but that it is not easy to do, several people have tried but no PR has landed yet. The "Breimann trick" that is used is also limited in the number of categorical values it supports, where this new idea seems to support unlimited categorical values and multi class classification.

### Additional context

An earlier paper that is cited in "Splitting categorical predictors...": https://link.springer.com/article/10.1023/A:1009869804967

An open question for me is how to measure how good this kind of transformation is compared to the exhaustive split (which is "the best possible" but too expensive to compute to be used in practice). The method proposed by the paper seems simple (famous last words), and they make claims about the performance but it would be nice to know how close this gets us. Trust, but verify ;)

COMMENT:
Hey @betatim - is there a possibility I could help contribute on this? Would love to help and learn. Thank you.

Labels: New Feature
Transformer for nominal categories, with the goal of improving category support in decision trees

I'd like to find out how keen people would be for adding a transformer like this.

### Describe the workflow you want to enable

Improved support for nominal categories in tree based models. Nominal categories are ones with no order to them, for example colour or country (c.f. ordinal categories that are ordered).

### Describe your proposed solution

In #12866 @amueller linked to [Splitting on categorical predictors in random forests](https://peerj.com/articles/6339). This proposes to replace categorical values (red, green, blue, ..) by cleverly computed numerical values (blue -> 0.643, green ->0.123, ...) that allow you to achieve similar/equal performance by using `<` as splitting decision in a tree node compared to performing an exhaustive search of all possible categorical splits.

I would implement this as a transformer that you use together with a random forest or other tree based model in a pipeline.

### Describe alternatives you've considered, if relevant

There have been several PRs attempting to add native category support to decision trees. #12866 is the latest one. My impression is that it would be cool to have this in trees but that it is not easy to do, several people have tried but no PR has landed yet. The "Breimann trick" that is used is also limited in the number of categorical values it supports, where this new idea seems to support unlimited categorical values and multi class classification.

### Additional context

An earlier paper that is cited in "Splitting categorical predictors...": https://link.springer.com/article/10.1023/A:1009869804967

An open question for me is how to measure how good this kind of transformation is compared to the exhaustive split (which is "the best possible" but too expensive to compute to be used in practice). The method proposed by the paper seems simple (famous last words), and they make claims about the performance but it would be nice to know how close this gets us. Trust, but verify ;)

COMMENT:
My current plan is to write a bit of code and open a PR tomorrow. I'm not sure yet how we could best collaborate on the PR (maybe Prs to the PR?). Getting another pair of eyes on it is good though because reviewer time is often the bottleneck. So maybe that is a way to collaborate.

Labels: New Feature
Transformer for nominal categories, with the goal of improving category support in decision trees

I'd like to find out how keen people would be for adding a transformer like this.

### Describe the workflow you want to enable

Improved support for nominal categories in tree based models. Nominal categories are ones with no order to them, for example colour or country (c.f. ordinal categories that are ordered).

### Describe your proposed solution

In #12866 @amueller linked to [Splitting on categorical predictors in random forests](https://peerj.com/articles/6339). This proposes to replace categorical values (red, green, blue, ..) by cleverly computed numerical values (blue -> 0.643, green ->0.123, ...) that allow you to achieve similar/equal performance by using `<` as splitting decision in a tree node compared to performing an exhaustive search of all possible categorical splits.

I would implement this as a transformer that you use together with a random forest or other tree based model in a pipeline.

### Describe alternatives you've considered, if relevant

There have been several PRs attempting to add native category support to decision trees. #12866 is the latest one. My impression is that it would be cool to have this in trees but that it is not easy to do, several people have tried but no PR has landed yet. The "Breimann trick" that is used is also limited in the number of categorical values it supports, where this new idea seems to support unlimited categorical values and multi class classification.

### Additional context

An earlier paper that is cited in "Splitting categorical predictors...": https://link.springer.com/article/10.1023/A:1009869804967

An open question for me is how to measure how good this kind of transformation is compared to the exhaustive split (which is "the best possible" but too expensive to compute to be used in practice). The method proposed by the paper seems simple (famous last words), and they make claims about the performance but it would be nice to know how close this gets us. Trust, but verify ;)

COMMENT:
Totally up for it @betatim, PR to a PR or with anything that could help you save time. Looking forward to it.

Labels: New Feature
Transformer for nominal categories, with the goal of improving category support in decision trees

I'd like to find out how keen people would be for adding a transformer like this.

### Describe the workflow you want to enable

Improved support for nominal categories in tree based models. Nominal categories are ones with no order to them, for example colour or country (c.f. ordinal categories that are ordered).

### Describe your proposed solution

In #12866 @amueller linked to [Splitting on categorical predictors in random forests](https://peerj.com/articles/6339). This proposes to replace categorical values (red, green, blue, ..) by cleverly computed numerical values (blue -> 0.643, green ->0.123, ...) that allow you to achieve similar/equal performance by using `<` as splitting decision in a tree node compared to performing an exhaustive search of all possible categorical splits.

I would implement this as a transformer that you use together with a random forest or other tree based model in a pipeline.

### Describe alternatives you've considered, if relevant

There have been several PRs attempting to add native category support to decision trees. #12866 is the latest one. My impression is that it would be cool to have this in trees but that it is not easy to do, several people have tried but no PR has landed yet. The "Breimann trick" that is used is also limited in the number of categorical values it supports, where this new idea seems to support unlimited categorical values and multi class classification.

### Additional context

An earlier paper that is cited in "Splitting categorical predictors...": https://link.springer.com/article/10.1023/A:1009869804967

An open question for me is how to measure how good this kind of transformation is compared to the exhaustive split (which is "the best possible" but too expensive to compute to be used in practice). The method proposed by the paper seems simple (famous last words), and they make claims about the performance but it would be nice to know how close this gets us. Trust, but verify ;)

COMMENT:
For linking purposes, https://github.com/scikit-learn/scikit-learn/pull/17323 introduces "impact encoder" or "target encoder" that converts categories into numerical values. I plan on revisiting it soon and likely bring it closer to cuml's `TargetEncoder`. 

As for this issue, I suspect there are not enough citations to adopt the proposed feature.

Labels: New Feature
Transformer for nominal categories, with the goal of improving category support in decision trees

I'd like to find out how keen people would be for adding a transformer like this.

### Describe the workflow you want to enable

Improved support for nominal categories in tree based models. Nominal categories are ones with no order to them, for example colour or country (c.f. ordinal categories that are ordered).

### Describe your proposed solution

In #12866 @amueller linked to [Splitting on categorical predictors in random forests](https://peerj.com/articles/6339). This proposes to replace categorical values (red, green, blue, ..) by cleverly computed numerical values (blue -> 0.643, green ->0.123, ...) that allow you to achieve similar/equal performance by using `<` as splitting decision in a tree node compared to performing an exhaustive search of all possible categorical splits.

I would implement this as a transformer that you use together with a random forest or other tree based model in a pipeline.

### Describe alternatives you've considered, if relevant

There have been several PRs attempting to add native category support to decision trees. #12866 is the latest one. My impression is that it would be cool to have this in trees but that it is not easy to do, several people have tried but no PR has landed yet. The "Breimann trick" that is used is also limited in the number of categorical values it supports, where this new idea seems to support unlimited categorical values and multi class classification.

### Additional context

An earlier paper that is cited in "Splitting categorical predictors...": https://link.springer.com/article/10.1023/A:1009869804967

An open question for me is how to measure how good this kind of transformation is compared to the exhaustive split (which is "the best possible" but too expensive to compute to be used in practice). The method proposed by the paper seems simple (famous last words), and they make claims about the performance but it would be nice to know how close this gets us. Trust, but verify ;)

COMMENT:
Do you know how many citations the target encoding in cuml (or really any of the many variations of target encoding) have? The cuml docs link to https://maxhalford.github.io/blog/target-encoding/ which, after a quick browse, doesn't link to a paper.

Labels: New Feature