Add more robust Kmeans initializer for mixture.GaussianMixture

### Describe the workflow you want to enable

The [GaussianMixture](https://github.com/scikit-learn/scikit-learn/blob/24f3006fb2d054b8afb26382209ae33629a8dfe0/sklearn/mixture/_gaussian_mixture.py#L456 ) class has an option of initializing using `K-means`, another method, or passing in your own initializer values, with the default being `K-Means`. 

The expectation maximization (EM) algorithm used by the `.fit` method of this class is very sensitive to initial conditions, as also mentioned in the docs. However, the `K-means` initializer uses hardcoded parameters which are passed via a super() call to the parent [BaseMixture](https://github.com/scikit-learn/scikit-learn/blob/24f3006fb2d054b8afb26382209ae33629a8dfe0/sklearn/mixture/_base.py#L50) class.

The workflow I want to enable is one where a more robust K-means initializer can be readily used (~1 line code) to initialize the GMM model. 

### Describe your proposed solution

There are two simple possibilities, which can be implemented alongside each other:

1. Allow passing in a `initializer_kwargs = {}` or `kmeans_kwargs` dictionary to the initializer, which is empty/None by default. These kwargs should be unpacked and overwrite the  keyword arguments passed to the initializer algorithm. Allowing optionally using a more robust but costly set of parameters.    
2. Provide a helper function in `mixtures` or a method to `cluster.Kmeans` to produce the desired inputs expected by `mixture.GaussianMixture`. This logic is currently buried in a mix of the [base class](https://github.com/scikit-learn/scikit-learn/blob/24f3006fb2d054b8afb26382209ae33629a8dfe0/sklearn/mixture/_base.py#L128) and `GaussianMixture` class and uses private ('_'functions) which are not exposed in `sklearn.mixture`, so need to be absolute imported. 

The benefit to option 1 is that the docstring of `GaussianMixture` can be modified to easily explain this and provide an example. It also would take the least amount of code. For example `sklearn.mixture.GaussianMixture(kmeans_kwargs={'n_init':1}). It is also immune to breaking changes in either cluster.Kmeans or mixture.GaussianMixture. 

Option 2 has the nice property that it can be used internally inside of mixture.GaussianMixture, instead of the adhoc current code. This can be especially useful if this option is propagated to other GMM mixture classes such as the Bayesian one. It also allows one mess around until an optimal initializer is found by experimenting with the KMeans class, and finally taking the desired result to use to init the GMM class.

### Describe alternatives you've considered, if relevant

It is already easy, especially if one understand the codebase or the math, to run your own Kmeans (or other method) as an initializer. Take the output, calculate the required parameters, which is actually the responsibilities, but  `means` (provided), `weights` (not-provided by Kmeans) and technically `precisions` (inverse of covariance matrix, although this can be calculated inside the class). This however takes several lines of code and benefits from using private/non-exposed functions inside mixture module. Therefore, it's not an ideal alternative.

Another alternative is to use composition and allow passing in an instance of a clustering algorithm class (like Kmeans), with the parameters already set. While this is useful, in practice it would mean validating many other clustering algorithms in sklearn to make sure they are usable, or requiring users to write their own decorators (or class extensions) to provide the values needed by GMM. I think it's too technically onerous for the small gain, even though I would love more composition in sklearn between classes and modules. This would also mean adding a potentially confusing extra way to pass an init_params argument that is a callable instead of a string, or modifying that parameter to confusingly accept strings as well as callables (instances of a clustering or other initializer algorithm). 

Final alternative is to allow passing in user-derived `responsibilities` instead of the above 3 separate parameters, which are actually what the super initialize class uses and all that is needed; As well as, provide a property on Kmeans which calculates the responsibilities. This has the added benefit that users can easily get responsibilities from Kmeans and that users only have to pass a single parameter instead of 3. But it would mean a bit more involved (still relatively easy) rewrite of GaussianMixtures class. However, it would also then likely deviate from other mixture classes and introduce an change to Kmeans.

**TL;DR**: Add: (1) optional keyword arguments to GMM class that is passed to the initializer. (2) public helper function to convert user-derived K-means model to easily initialize a GMM class with 1 line of code. Alternatives (such as using composition) are valid, even preferable, but likely too much of a code review and maintenance burden for the minor improvement. The two proposed solutions add minimal review and maintenance burdens. 

### Additional context

In version 1.1 of sklearn `kmeans++` will be added as an alternative initializer, however, while better, it also suffers from the lack of being able to change its parameters. Additionally, it's not a very clear name so Kmeans will likely still be often used with a non-robust set of parameters. 

**I would like to submit a PR on this** if that's acceptable to the maintainers. But since it would be my first PR to sklearn, I would appreciate any feedback on my proposal so I can get a better understanding of how contributing works for this package.

COMMENT:
I started working on implementing both versions but after trying it, I agree that the extra kwargs version, while the most simple and changes least amount of code (to be frank, current code really doesn’t fit the contributing guidelines but I don’t want to create a review burden so I am not touching it), is less user friendly than passing in a callable.

Also I tested the more robust GMM initialization for an astronomy paper I’m involved in and we got better results (higher completion factors for known labels) using it, so I’m convinced that it’s useful.

When I get a chance I can open up a WIP PR following the callable as arg syntax from Kmeans. Actually, it would be great to just pass the fitted Kmeans class and avoid having to use my proposed function. That can be _private one used under the hood instead. So I am leaning towards your view as well.

If anyone, not limited to maintainers, has input I’m highly inclined to listen and adapt the WIP. And, thanks for taking the time to read and comment on this issue.


Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Jérémie du Boisberranger ***@***.***>
Sent: Wednesday, April 27, 2022 11:13:48 AM
To: scikit-learn/scikit-learn ***@***.***>
Cc: Emir ***@***.***>; Mention ***@***.***>
Subject: Re: [scikit-learn/scikit-learn] Add more robust Kmeans initializer for mixture.GaussianMixture (Issue #23195)


Hi @emirkmo<https://github.com/emirkmo>,

I think it's reasonable to extend the initialization possibilities of GMM. I would not add an initializer_kwargs parameter because having parameters that only do something based on another parameter are not very user friendly imo.

What would you think about adding the possibility to pass a callable to init_params ? This is for instance possible in KMeans.
Another possibility to directly pass initial responsibilities would also be interesting.

public helper function to convert user-derived K-means model to easily initialize a GMM

That would be helpful for both callable and custom init.

I would like to submit a PR on this if that's acceptable to the maintainers.

Feel free to do that, but be aware that we are a bit short on review time so don't worry if you're receiving reviews immediately.
We can also wait a little more to see if other maintainers are interested and be sure we're on the same page regarding the proposed solution.

—
Reply to this email directly, view it on GitHub<https://github.com/scikit-learn/scikit-learn/issues/23195#issuecomment-1110762851>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACVQXWQRDSO7B7RTUMRX42LVHEAMZANCNFSM5UDKFAZA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>

Labels: New Feature