Repository: biolab/orange3

Issue:
k-means restores different clusters when loading a workflow
##### Expected behavior
Saving a workflow and opening it should return exactly the same clusters. 

Perhaps k-means should also have a replicable option similar to the data sampler widget. But I think that saving and opening a workflow should return the same clusters even if replicable is not selected. 

##### Actual behavior
K-means results change if you save and open schemas. Because also the order of classes changes, everything downstream is problematic.


Discussion:
I agree with a replicable checkbox.

In order to always return the same clusters when a workflow is reopened, the clusters would need to be stored in settings. The same behaviour could also be expected from sample data widget, jittering in scatter plot, cross validation, random forest, ... I am not sure we want to open that door. As a side note, when workflow is reopened, we have no guarantee that the dataset is the same as before, so we would have to handle that as well.
I would open that pandoras box. 

I know there are many such widgets, but would we really need anything else that to just save some kind of random seed?
The problem is how long to remember it for. If the widget would pick a random number when it is added to the workflow remember it and use it as a seed each time it does the clustering, it could work.

If we want a different seed each time apply is pressed, this could be problematic. If this is what you had in mind, let me know and I can explain the problem in detail.
I tried to think about the problematic solution, with random seed changing on each apply, except when applied the first time after widget init.

So, what am I missing? :)
When I concatenate multiple datasets in car registrations workflow, some of them get loaded automatically and some not, because they are too large. So when I open the workflow k-means would get multiple apply signals, one for all datasets that loaded automatically and then one for each dataset once I click reload.
I agree there is a problem. Thanks!

As I understand it now Data Table does not send anything if is is waiting for user confirmation. Would it perhaps work in that case, if Concatenate would wait for a signal on all inputs before constructing the output? 

If we could make above work and if each widget returned only one output per a set of inputs, would we still have a problem?
Probably.

I am thinking more along the line of having a single k-means widget always produce the same clusters for the same dataset (selecting a random seed in `__init__`). If you want to see multiple clusterings, we can add an explicit "new random" button, or you can use a separate k-means widget.
+1

Same data set + same algorithm = same results, unless explicitly randomized.
@astaric, I like your approach of selecting a random seed in __init__. It is certainly easier to implement than fixing all (at least some of) other widgets.
For a start, I would propose that every time this widget runs a clustering, it sets the random seed. This is also done when trying different k. Only advanced users would try to see how the clustering is changing with different seeds, and they would achieve the same effect through shuffling of the data (via Data Sampler, I hope). We have many more users that expect replicable results and for which fixing a random seed would work just fine.
@BlazZupan, PR #3088 only tries solves a problem with users reopening their saved workflows.

If we want that every user fresh-build workflow gets exactly the same result, we always need the same random seed. Then, the only way to get different results would be to shuffle the data. Did you have this in mind? This would be even simpler and perhaps less confusing.

Resolving patch:
diff --git a/Orange/widgets/unsupervised/owkmeans.py b/Orange/widgets/unsupervised/owkmeans.py
--- a/Orange/widgets/unsupervised/owkmeans.py
+++ b/Orange/widgets/unsupervised/owkmeans.py
@@ -19,6 +19,9 @@
 from Orange.widgets.widget import Input, Output
 
 
+RANDOM_STATE = 0
+
+
 class ClusterTableModel(QAbstractTableModel):
     def __init__(self, parent=None):
         super().__init__(parent)
@@ -252,14 +255,14 @@ def enough_data_instances(self, k):
         return len(self.data) >= k
 
     @staticmethod
-    def _compute_clustering(data, k, init, n_init, max_iter, silhouette):
+    def _compute_clustering(data, k, init, n_init, max_iter, silhouette, random_state):
         # type: (Table, int, str, int, int, bool) -> KMeansModel
         if k > len(data):
             raise NotEnoughData()
 
         return KMeans(
             n_clusters=k, init=init, n_init=n_init, max_iter=max_iter,
-            compute_silhouette_score=silhouette,
+            compute_silhouette_score=silhouette, random_state=random_state,
         )(data)
 
     @Slot(int, int)
@@ -321,6 +324,7 @@ def __launch_tasks(self, ks):
             n_init=self.n_init,
             max_iter=self.max_iterations,
             silhouette=True,
+            random_state=RANDOM_STATE,
         ) for k in ks]
         watcher = FutureSetWatcher(futures)
         watcher.resultReadyAt.connect(self.__clustering_complete)


Test patch:
diff --git a/Orange/widgets/unsupervised/tests/test_owkmeans.py b/Orange/widgets/unsupervised/tests/test_owkmeans.py
--- a/Orange/widgets/unsupervised/tests/test_owkmeans.py
+++ b/Orange/widgets/unsupervised/tests/test_owkmeans.py
@@ -426,6 +426,25 @@ def test_correct_smart_init(self):
             self.commit_and_wait()
             self.assertEqual(compute.call_args[1]['init'], "random")
 
+    def test_always_same_cluster(self):
+        """The same random state should always return the same clusters"""
+        self.send_signal(self.widget.Inputs.data, self.iris[::10], wait=5000)
+
+        def cluster():
+            self.widget.invalidate()  # reset caches
+            self.commit_and_wait()
+            return self.get_output(self.widget.Outputs.annotated_data).metas[:, 0]
+
+        def assert_all_same(l):
+            for a1, a2 in zip(l, l[1:]):
+                np.testing.assert_equal(a1, a2)
+
+        self.widget.smart_init = 0
+        assert_all_same([cluster() for _ in range(5)])
+
+        self.widget.smart_init = 1
+        assert_all_same([cluster() for _ in range(5)])
+
 
 if __name__ == "__main__":
     unittest.main()