Repository: biolab/orange3

Issue:
Replicable sampling in Test & Score
Add an option to choose replicable sampling in Test & Score widget. It should do the same as the Replicable checkbox in Data Sampler


Discussion:
I can have a look at this issue.
@MaxGreil, how is it going? Did you encounter any problems adding the checkbox?
Hi @astaric , I am sorry but I couldn't start to implement yet. I had some troubles installing the software on my Ubuntu `(ImportError: ../lib/python3.5/site-packages/PyQt5/QtCore.so: undefined symbol: PySlice_AdjustIndices)`.
The solution to this was to add `pyqt5==5.10.1` in the `requirements-doc.txt`.
Similar to #3104, no? @markotoplak Would you be interested in adding this? 
Hmmm, @astaric, the Test & Score already seems to pass a fixed random seed forward. Do we need a non-replicable option?
Is there another way to get different AUC results on a dataset?
Shuffle your data? But perhaps in a different widget? 

Currently, data sampler can do a len-1 sample, which is almost shuffling.

Resolving patch:
diff --git a/Orange/widgets/data/owdatasampler.py b/Orange/widgets/data/owdatasampler.py
--- a/Orange/widgets/data/owdatasampler.py
+++ b/Orange/widgets/data/owdatasampler.py
@@ -60,7 +60,7 @@ class Warning(OWWidget.Warning):
 
     class Error(OWWidget.Error):
         too_many_folds = Msg("Number of folds exceeds data size")
-        sample_larger_than_data = Msg("Sample must be smaller than data")
+        sample_larger_than_data = Msg("Sample can't be larger than data")
         not_enough_to_stratify = Msg("Data is too small to stratify")
         no_data = Msg("Dataset is empty")
 
@@ -88,7 +88,7 @@ def set_sampling_type_i():
         self.sampleSizePercentageSlider = gui.hSlider(
             gui.indentedBox(sampling), self,
             "sampleSizePercentage",
-            minValue=0, maxValue=99, ticks=10, labelFormat="%d %%",
+            minValue=0, maxValue=100, ticks=10, labelFormat="%d %%",
             callback=set_sampling_type(self.FixedProportion),
             addSpace=12)
 
@@ -265,7 +265,7 @@ def updateindices(self):
         else:
             assert self.sampling_type == self.Bootstrap
 
-        if not repl and size is not None and (data_length <= size):
+        if not repl and size is not None and (size > data_length):
             self.Error.sample_larger_than_data()
         if not repl and data_length <= num_classes and self.stratify:
             self.Error.not_enough_to_stratify()
@@ -386,7 +386,12 @@ def __call__(self, table):
             o[sample] = 0
             others = np.nonzero(o)[0]
             return others, sample
-        if self.stratified and table.domain.has_discrete_class:
+        if self.n == len(table):
+            rgen = np.random.RandomState(self.random_state)
+            sample = np.arange(self.n)
+            rgen.shuffle(sample)
+            return np.array([], dtype=int), sample
+        elif self.stratified and table.domain.has_discrete_class:
             test_size = max(len(table.domain.class_var.values), self.n)
             splitter = skl.StratifiedShuffleSplit(
                 n_splits=1, test_size=test_size,


Test patch:
diff --git a/Orange/widgets/data/tests/test_owdatasampler.py b/Orange/widgets/data/tests/test_owdatasampler.py
--- a/Orange/widgets/data/tests/test_owdatasampler.py
+++ b/Orange/widgets/data/tests/test_owdatasampler.py
@@ -100,6 +100,22 @@ def test_bigger_output_warning(self):
         self.set_fixed_sample_size(3, with_replacement=True)
         self.assertTrue(self.widget.Warning.bigger_sample.is_shown())
 
+    def test_shuffling(self):
+        self.send_signal('Data', self.iris)
+
+        self.set_fixed_sample_size(150)
+        self.assertFalse(self.widget.Warning.bigger_sample.is_shown())
+        sample = self.get_output("Data Sample")
+        self.assertTrue((self.iris.ids != sample.ids).any())
+        self.assertEqual(set(self.iris.ids), set(sample.ids))
+
+        self.select_sampling_type(self.widget.FixedProportion)
+        self.widget.sampleSizePercentage = 100
+        self.widget.commit()
+        sample = self.get_output("Data Sample")
+        self.assertTrue((self.iris.ids != sample.ids).any())
+        self.assertEqual(set(self.iris.ids), set(sample.ids))
+
     def set_fixed_sample_size(self, sample_size, with_replacement=False):
         """Set fixed sample size and return the number of gui spin.