Repository: scikit-image/scikit-image

Issue:
ransac selects duplicate data points in random sample
## Description
I don't know if this behavior is intentional, but to me it seems a bit odd.
When the ransac-method selects the random sample to fit the model, it can select the same data point multiple times.

## Way to reproduce
```python
import numpy as np
from skimage.measure import ransac

np.random.seed(seed=0)

data = np.arange(10)

class Model(object):

    """Dummy model"""

    def estimate(self, data):
        if np.unique(data).size != data.size:
            print("Duplicate points: ", data)
        return True

    def residuals(self, data):
        return 1.0

ransac(data, Model, min_samples=3, residual_threshold=0.0, max_trials=10)
```
Which results in
```python
('Duplicate points: ', array([8, 8, 1]))
('Duplicate points: ', array([6, 7, 7]))
('Duplicate points: ', array([9, 8, 9]))
```

## Version information
```python
# Paste the output of the following python commands
2.7.13 (default, Sep 26 2018, 18:42:22) 
[GCC 6.3.0 20170516]
Linux-4.9.0-8-amd64-x86_64-with-debian-9.4
scikit-image version: 0.14.2
numpy version: 1.16.0
```


Discussion:
Thanks for pointing this out, @johanjeppsson! I would say that this is unintentional. @stefanv?

Here is the relevant bit of code:

https://github.com/scikit-image/scikit-image/blob/7949b582285bc46a043bf038e48f571351420189/skimage/measure/fit.py#L805-L813

You *could* get around this by writing:

```python
def no_repeats(lst):
    return len(set(lst)) == len(lst)

ransac(..., is_data_valid=no_repeats)
```

But this seems clunky for what, in my view, should be the default behaviour. I expect that part of the reason is that NumPy didn't have `np.random.choice` when this was written.

If we get a consensus that we should move to that, I'll switch to `choice`. Having said that, it might require a tricky deprecation cycle. @stefanv might have some insight about the NumPy random numbers reproducibility policy and whether we want to be as strict as that or not.
sorry, it's a bit more tricky than the above, because `is_data_valid` gets the samples themselves, not the indices...
Thanks for the quick response!

I saw the is_valid option as well, and, like you said, it is possible to use that with e.g. np.unique to check if you have duplicates in the sample data.

```python
def check_dup(data):
    return np.unique(data).size == data.size
```
which works given one dimensional data. For more dimensions, maybe the axis-argument to unique can be used.

But I agree, if feels like this is a little bit of a weird extra hoop to jump through.
This is definitely a bug. The random subset should only contain unique data points. This should be done using the following logic (can be implemented more efficiently as not the whole array has to be shuffled but just the first elements):
```
idxs = np.arange(num_data_points)
for i in range(num_iters): np.suffle(idxs); subset_idxs = idxs[:num_subset_points]
```
why can't we just use `random.choice replace=False`????

This seams to pass the tests:

```
        # choose random sample set
        random_idxs = random_state.choice(num_samples, min_samples,
                                          replace=False)
```
@hmaarrfk That's obviously also correct, but didn't know about that helper function. It probably executes the same logic inside anyway.
alright, cool! seems like there is a consensus on the approach, now it just needs a test.... won't be too easy due to the inherent randomness of the algorithm
> it probably executes the same logic inside anyway.

Nope. =) For any value significantly smaller than your population size, the best way to sample without replacement is to sample with replacement and then discard the repeat samples. (See this [StackExchange question](https://stats.stackexchange.com/questions/20590/how-do-i-sample-without-replacement-using-a-sampling-with-replacement-function) asked ages ago by yours truly!) This is in fact what NumPy does:

https://github.com/numpy/numpy/blob/808bdcf20ffeaa7241259086a1f7c9ef13ce29f3/numpy/random/mtrand/mtrand.pyx#L1166-L1197

except in the case of num_samples = pop_size.

The NumPy version is probably not the fastest because it works hard to not "use up" more random samples than it needs. However in RANSAC I don't think it matters because typically num_samples is much smaller than pop_size and repeats are probably quite rare, but something to think about.
Note: if we treat this as a bug and don't worry about exact reproducibility, we can just replace the sampling. This would be my preferred approach for practical reasons, but happy for someone to pop in and object.
> > it probably executes the same logic inside anyway.

> Nope. =)

Oh LOL, upon re-reading of the code, it *is* indeed what it does, except for non-uniform probabilities. :man_facepalming: :sweat_smile: Next time I'll wait until *after* finishing my coffee to write a response. =P
I think we should at least have a test that asks for 3 samples out of 4, 10 times, and checks that non are repeating.

Resolving patch:
diff --git a/skimage/measure/fit.py b/skimage/measure/fit.py
--- a/skimage/measure/fit.py
+++ b/skimage/measure/fit.py
@@ -804,7 +804,8 @@ def ransac(data, model_class, min_samples, residual_threshold,
 
         # choose random sample set
         samples = []
-        random_idxs = random_state.randint(0, num_samples, min_samples)
+        random_idxs = random_state.choice(num_samples, min_samples,
+                                          replace=False)
         for d in data:
             samples.append(d[random_idxs])
 


Test patch:
diff --git a/skimage/measure/tests/test_fit.py b/skimage/measure/tests/test_fit.py
--- a/skimage/measure/tests/test_fit.py
+++ b/skimage/measure/tests/test_fit.py
@@ -348,3 +348,23 @@ def test_ransac_invalid_input():
     with testing.raises(ValueError):
         ransac(np.zeros((10, 2)), None, min_samples=2,
                residual_threshold=0, stop_probability=1.01)
+
+
+def test_ransac_sample_duplicates():
+    class DummyModel(object):
+
+        """Dummy model to check for duplicates."""
+
+        def estimate(self, data):
+            # Assert that all data points are unique.
+            assert_equal(np.unique(data).size, data.size)
+            return True
+
+        def residuals(self, data):
+            return 1.0
+
+    # Create dataset with four unique points. Force 10 iterations
+    # and check that there are no duplicated data points.
+    data = np.arange(4)
+    ransac(data, DummyModel, min_samples=3, residual_threshold=0.0,
+           max_trials=10)