Repository: graspologic-org/graspologic

Issue:
merge Autogmm and GaussianCluster
Seems like there is enough redundant code, they are both doing something similar, and we can still get both behaviors with different options


Discussion:
to clarify, AutoGMM does everything that GaussianCluster does and more. The difference is, GaussianCluster has an option to run kmeans initialization multiple times, which I have found empirically can lead to better results than any of the AutoGMM initializations. If AutoGMM had a way of running multiple kmeans inits and taking the best, we could deprecate GaussianCluster
I'll get this done this semester
https://arxiv.org/abs/1909.02688
@bdpedigo I will take this one, sounds fun! : )

DoD:

- Implementation:
   - Add option in AutoGMM class to run k-means multiple times:
     - Must be simple and intuitive to call for the user
     - Must be able to specify min_components and max_components to use
            - If max_components is None then it must utilize min_components as maximum number(analogously to gclust)
     - Must be integrated with the k-means method already present in the AutoGMM class
- Testing:
    - Must have identical output as gclust. Meaning that the difference between number of clusters assigned by gclust and the merged functionality to autoGMM must be identically 0 given same input data and same settings

This is an initial very much qualitative assesment of what the final implementation should do. I am sure it will progressively become more and more quantitative. 

Please let me know if you have any suggestions/ideas. 
    
https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLarsIC.html
see `criterion_` for example
`criterion_: array-like of shape (n_alphas,)
The value of the information criteria (‘aic’, ‘bic’) across all alphas. The alpha which has the smallest information criterion is chosen. This value is larger by a factor of n_samples compared to Eqns. 2.15 and 2.16 in (Zou et al, 2007).`
@PerifanosPrometheus: @tliu68 is working on some changes to the internals of AutoGMM (see #371). I think this will just affect the initialization. You two might want to coordinate a bit so that the merge is easy at the end!
@tliu68 this PR from @PerifanosPrometheus will more or less add 1 or more runs of kmeans as an initialization in AutoGMM, and maybe will rename the class itself.
> @tliu68 this PR from @PerifanosPrometheus will more or less add 1 or more runs of kmeans as an initialization in AutoGMM, and maybe will rename the class itself.

Cool! @PerifanosPrometheus I'll let you know what specific changes I'm going to make soon. Thanks!
> > @tliu68 this PR from @PerifanosPrometheus will more or less add 1 or more runs of kmeans as an initialization in AutoGMM, and maybe will rename the class itself.
> 
> Cool! @PerifanosPrometheus I'll let you know what specific changes I'm going to make soon. Thanks!

@tliu68 Currently, I am also on the planning stage so I haven't finalized any changes yet. Please let me know if there is anything that I should particularly be careful dealing with
> @tliu68 Currently, I am also on the planning stage so I haven't finalized any changes yet. Please let me know if there is anything that I should particularly be careful dealing with

@PerifanosPrometheus I've opened a draft PR #589. Please see my description on all major changes I made. My plan for next step is to try to clean up/reorganize the for loops and see if I can incorporate parallel processing. But I believe the major updates we wanted to make have been addressed. Please let me know if you have any question and all suggestion/advice welcome!
@tliu68 @PerifanosPrometheus please coordinate
@tliu68 I would like to continue working on this issue, however I noticed that PR #589 has been merged. I would like to touch back with you to have an update on changes merged. 

Mainly, I believe that fixing this issue will involve modifying the param_grid dict in the previous implementation, has that been changed? 

Lastly, I heard that you were planning on merging the function with sklearn(not entirely sure this info is correct) in that case what would be the timeline to make my changes(if even desirable).
> @tliu68 I would like to continue working on this issue, however I noticed that PR #589 has been merged. I would like to touch back with you to have an update on changes merged.
> 
> Mainly, I believe that fixing this issue will involve modifying the param_grid dict in the previous implementation, has that been changed?
> 
> Lastly, I heard that you were planning on merging the function with sklearn(not entirely sure this info is correct) in that case what would be the timeline to make my changes(if even desirable).

Yes, I am working on merging this into sklearn (described in issue #601). The previous PR #589 basically changed the initialization process so that instead of running AgglomerativeClustering and GaussianMixture for all combinations in the old `param_grid` in parallel, now 

- for each set of agglomerative parameters, 
- AgglomerativeClustering is run for the smallest n_components (which is why I added another output `param_grid_ag` from `_process_paramgrid` --  to separate out the agglomerative parameters)
- and a hierarchy of clusters is generated based on that assignment, 
- then after calculating all agglomerative clustering assignments, GaussianMixture is run for all combinations in `param_grid`

Does that make sense? Also, Ben and I will probably make more changes (to be discussed) but it would be great if we can start a PR into sklearn soon. Do you mind sharing your plan?

Resolving patch:
diff --git a/graspologic/cluster/autogmm.py b/graspologic/cluster/autogmm.py
--- a/graspologic/cluster/autogmm.py
+++ b/graspologic/cluster/autogmm.py
@@ -15,6 +15,7 @@
     _estimate_gaussian_parameters,
 )
 from sklearn.model_selection import ParameterGrid
+from sklearn.utils import check_scalar
 
 from .base import BaseCluster
 
@@ -116,6 +117,12 @@ class AutoGMMCluster(BaseCluster):
         If provided, min_components and ``max_components`` must match the number of
         unique labels given here.
 
+    kmeans_n_init : int, optional (default = 1)
+        If ``kmeans_n_init`` is larger than 1 and ``label_init`` is None, additional
+        ``kmeans_n_init``-1 runs of :class:`sklearn.mixture.GaussianMixture`
+        initialized with k-means will be performed
+        for all covariance parameters in ``covariance_type``.
+
     max_iter : int, optional (default = 100).
         The maximum number of EM iterations to perform.
 
@@ -215,6 +222,7 @@ def __init__(
         covariance_type="all",
         random_state=None,
         label_init=None,
+        kmeans_n_init=1,
         max_iter=100,
         verbose=0,
         selection_criteria="bic",
@@ -362,6 +370,8 @@ def __init__(
         if max_agglom_size is not None and max_agglom_size < 2:
             raise ValueError("Must use at least 2 points for `max_agglom_size`")
 
+        check_scalar(kmeans_n_init, name="kmeans_n_init", target_type=int, min_val=1)
+
         self.min_components = min_components
         self.max_components = max_components
         self.affinity = affinity
@@ -369,13 +379,14 @@ def __init__(
         self.covariance_type = new_covariance_type
         self.random_state = random_state
         self.label_init = labels_init
+        self.kmeans_n_init = kmeans_n_init
         self.max_iter = max_iter
         self.verbose = verbose
         self.selection_criteria = selection_criteria
         self.max_agglom_size = max_agglom_size
         self.n_jobs = n_jobs
 
-    def _fit_cluster(self, X, X_subset, y, params, agg_clustering):
+    def _fit_cluster(self, X, X_subset, y, params, agg_clustering, seed):
         label_init = self.label_init
         if label_init is not None:
             onehot = _labels_to_onehot(label_init)
@@ -400,6 +411,7 @@ def _fit_cluster(self, X, X_subset, y, params, agg_clustering):
             gm_params["init_params"] = "kmeans"
         gm_params["reg_covar"] = 0
         gm_params["max_iter"] = self.max_iter
+        gm_params["random_state"] = seed
 
         criter = np.inf  # if none of the iterations converge, bic/aic is set to inf
         # below is the regularization scheme
@@ -514,10 +526,18 @@ def fit(self, X, y=None):
             linkage=self.linkage,
             covariance_type=self.covariance_type,
             n_components=range(lower_ncomponents, upper_ncomponents + 1),
-            random_state=[self.random_state],
         )
         param_grid = list(ParameterGrid(param_grid))
-        param_grid_ag, param_grid = _process_paramgrid(param_grid)
+
+        param_grid_ag, param_grid = _process_paramgrid(
+            param_grid, self.kmeans_n_init, self.label_init
+        )
+
+        if isinstance(self.random_state, int):
+            np.random.seed(self.random_state)
+            seeds = np.random.randint(np.iinfo(np.int32).max, size=len(param_grid))
+        else:
+            seeds = [self.random_state] * len(param_grid)
 
         n = X.shape[0]
         if self.max_agglom_size is None or n <= self.max_agglom_size:
@@ -539,17 +559,17 @@ def fit(self, X, y=None):
                     )
                     ag_labels.append(hierarchical_labels)
 
-        def _fit_for_data(p):
+        def _fit_for_data(p, seed):
             n_clusters = p[1]["n_components"]
             if (p[0]["affinity"] != "none") and (self.label_init is None):
                 index = param_grid_ag.index(p[0])
                 agg_clustering = ag_labels[index][:, n_clusters - self.min_components]
             else:
                 agg_clustering = []
-            return self._fit_cluster(X, X_subset, y, p, agg_clustering)
+            return self._fit_cluster(X, X_subset, y, p, agg_clustering, seed)
 
         results = Parallel(n_jobs=self.n_jobs, verbose=self.verbose)(
-            delayed(_fit_for_data)(p) for p in param_grid
+            delayed(_fit_for_data)(p, seed) for p, seed in zip(param_grid, seeds)
         )
         results = pd.DataFrame(results)
 
@@ -646,7 +666,7 @@ def _labels_to_onehot(labels):
     return onehot
 
 
-def _process_paramgrid(paramgrid):
+def _process_paramgrid(paramgrid, kmeans_n_init, label_init):
     """
     Removes combinations of affinity and linkage that are not possible.
 
@@ -664,7 +684,7 @@ def _process_paramgrid(paramgrid):
     ag_paramgrid_processed : list of dicts
         options for AgglomerativeClustering
     """
-    gm_keys = ["covariance_type", "n_components", "random_state"]
+    gm_keys = ["covariance_type", "n_components"]
     ag_keys = ["affinity", "linkage"]
     ag_params_processed = []
     paramgrid_processed = []
@@ -686,6 +706,16 @@ def _process_paramgrid(paramgrid):
             ag_params = {key: params[key] for key in ag_keys}
             if ag_params not in ag_params_processed:
                 ag_params_processed.append(ag_params)
+            if (
+                ag_params["affinity"] == "none"
+                and kmeans_n_init > 1
+                and label_init is None
+            ):
+                more_kmeans_init = gm_params.copy()
+                more_kmeans_init.update({"n_init": 1})
+                paramgrid_processed += [
+                    [{"affinity": "none", "linkage": "none"}, more_kmeans_init]
+                ] * (kmeans_n_init - 1)
 
             paramgrid_processed.append([ag_params, gm_params])
     return ag_params_processed, paramgrid_processed


Test patch:
diff --git a/tests/cluster/test_autogmm.py b/tests/cluster/test_autogmm.py
--- a/tests/cluster/test_autogmm.py
+++ b/tests/cluster/test_autogmm.py
@@ -225,6 +225,31 @@ def test_two_class(self):
         # Asser that we get perfect clustering
         assert_allclose(AutoGMM.ari_, 1)
 
+    def test_two_class_multiple_kmeans_inits(self):
+        """
+        Easily separable two gaussian problem.
+        """
+        np.random.seed(1)
+
+        n = 100
+        d = 3
+
+        X1 = np.random.normal(2, 0.5, size=(n, d))
+        X2 = np.random.normal(-2, 0.5, size=(n, d))
+        X = np.vstack((X1, X2))
+        y = np.repeat([0, 1], n)
+
+        AutoGMM = AutoGMMCluster(max_components=5, kmeans_n_init=2)
+        AutoGMM.fit(X, y)
+
+        n_components = AutoGMM.n_components_
+
+        # Assert that the two cluster model is the best
+        assert_equal(n_components, 2)
+
+        # Asser that we get perfect clustering
+        assert_allclose(AutoGMM.ari_, 1)
+
     def test_two_class_parallel(self):
         """
         Easily separable two gaussian problem.