Repository: scikit-image/scikit-image

Issue:
measure.find_contours Key Error Stacktrace/fail
measure.find_contours failing with a KeyError message.

I can provide the image file (png) if needed. Note the image is inverted prior to this call.

image = Image.open(imgpath)
image_inv = PIL.ImageOps.invert(image)
npai = image_inv
npaif = np.array(npai, dtype='float')
npaif_contours = measure.find_contours(npaif,level=128.0,fully_connected='low',positive_orientation='low')

Traceback (most recent call last):
  File "lib\skimagelearn.py", line 44, in <module>
    npaif_contours = measure.find_contours(npaif,level=128.0,fully_connected='low',positive_orientation='low')
  File "E:\p\pythonlib64\lib\skimage\measure\find_contours.py", line 120, in find_contours
    contours = _assemble_contours(_take_2(point_list))
  File "E:\p\pythonlib64\lib\skimage\measure\find_contours.py", line 170, in _assemble_contours
    del ends[tail[-1]]
KeyError: (0.0, 396)



Discussion:
I can't reproduce with various grayscale uint8 PNG files lying around. Could you provide a link to the troublesome image?

Yes, attached, test code, and image. You will eed to update the imgpath,
and probably the py64libpath. Thx

When i set both 'low' parameter values to 'high' the method works without
error.

[image: Inline image 1]n

On Fri, Nov 15, 2013 at 3:32 PM, Josh Warner notifications@github.comwrote:

> I can't reproduce with various grayscale uint8 PNG files lying around.
> Could you provide a link to the troublesome image?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/scikit-image/scikit-image/issues/817#issuecomment-28601099
> .
> 
> #http://scikit-image.org/docs/0.9.x/api/skimage.filter.html

import pdb,os,sys
py64libpath = os.path.dirname(os.path.dirname(sys.executable))
sys.path.insert(0,py64libpath)
import numpy as np
sys.path.append(os.path.join(py64libpath,'matplotlib'))
sys.path.append(os.path.join(py64libpath,'pyparsingpkg'))
import pyparsing
import matplotlib.pyplot as mplt
import matplotlib.image  as mimg
import h5py
sys.path.append(os.path.join(py64libpath,'opencv'))
import cv2
from pylibtiff import TIFF
import skimage
import pylab
import scipy
import scipy.ndimage
from PIL import Image
import PIL.ImageOps
from skimage import measure
from skimage import util

# MAIN

dst='_skil'
fn = '02988.png'
imgpath = r'E:\p\hhmi\tools\cmsl_cmsl\TASK_001_CLV\bock11_hdf5_00_051200_052224_081920_082944\png\%s'%fn

image = Image.open(imgpath)
image_inv = PIL.ImageOps.invert(image)
npai = image_inv

npaif = np.array(npai, dtype='float')
npaif_contours = measure.find_contours(npaif,level=128.0,fully_connected='low',positive_orientation='low')
n1 = npaif_contours

mplt.xlim(0,1024)
mplt.ylim(0,1024)
for c1 in n1:
    mplt.plot(c1[:,0],c1[:,1])
mplt.show()

It appears GitHub stripped out any image(s) attached or embedded in your email. 

Probably the easiest way to attach an image to an issue is to use the "view it on GitHub" link at the bottom to come here, drag the image into the text box, preview and post it as a comment. That way it will definitely get to us.

![02988](https://f.cloud.github.com/assets/1430861/1554328/a96c19e6-4e43-11e3-8d36-157262532a76.png)

image posted.

i did not mean to close.

Is this issue still relevant? I cannot reproduce...

I can reproduce the error.

The problem is that npaif has values in [0, 255] but is a floating point image.

I guess this should not be a pre-requisite for finding contours?

https://www.dropbox.com/s/rxux0bjsi738noo/npaif.npy

```
import numpy as np
npaif = np.load('npaif.npy')
from skimage import measure
measure.find_contours(npaif,level=128.0,fully_connected='low',positive_orientation='low')
```

Less simple than it appears, because if you try dividing both the image and level by 255.0 (which is then a correct floating point image representation on range [0, 1]) the exact same KeyError is thrown.

``` python
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-32-79a02e4a4b3d> in <module>()
----> 1 npaif_contours = skimage.measure.find_contours(npaif / 255., level=128. / 255., fully_connected='low', positive_orientation='low')

/Users/[username]/src/scikit-image/skimage/measure/_find_contours.pyc in find_contours(array, level, fully_connected, positive_orientation)
    118     point_list = _find_contours_cy.iterate_and_store(array, level,
    119                                                      fully_connected == 'high')
--> 120     contours = _assemble_contours(_take_2(point_list))
    121     if positive_orientation == 'high':
    122         contours = [c[::-1] for c in contours]

/Users/[username]/src/scikit-image/skimage/measure/_find_contours.pyc in _assemble_contours(points_iterator)
    168                     # remove all traces of tail:
    169                     del starts[to_point]
--> 170                     del ends[tail[-1]]
    171                     del contours[tail_num]
    172                     # remove the old end of head and add the new end.

KeyError: (0.0, 396)

```

Are you sure?  It works fine for me.

Just triple checked, even made a new array. Still getting the exact same error and traceback when running

```
npaif_contours = skimage.measure.find_contours(
    npaif / 255., level=128. / 255., fully_connected='low', positive_orientation='low')
```

Notably, the error does not occur if you use `level=0.5`. But for `level=128. / 255` (which is slightly greater than the round value 0.5) you get the same error/traceback.

Finally, `level=127. / 255` which is slightly less than 0.5 passes without error.

This is an edge case, but it's repeatable across image representations for that exact set of scaled inputs. IMO worth tracking down and squashing.

Yes, agreed.  Mine worked because I did not modify the level correctly.

@JDWarner Did you by any chance figure out what was happening here?

Honestly I never investigated further, but will dive into this over the next few days. Should be interesting!

Stefan: No, I provided a sample image which should reproduce the issue, I
hope that you observe the same failure. I chose to 'live' with it, however
I am interested in any updates. Josh appears to have a good handle on the
area causing the error.

On Mon, Feb 16, 2015 at 10:23 PM, Josh Warner notifications@github.com
wrote:

> Honestly I never investigated further, but will dive into this over the
> next few days. Should be interesting!
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/scikit-image/scikit-image/issues/817#issuecomment-74611833
> .

I cannot reproduce this under Python 3.  @JDWarner Are you able to?

I can reproduce under Python 3.

Right, does not happen when I load the image but it does when I load the numpy file.  Datafile can now be found here:

https://www.dropbox.com/s/22am3de09ogdaa7/npaif.npz?dl=0

I think there is a corner case not accounted for in the logic.

@scikit-image/core I've gone round in circles on this one for about a week. The line that's failing is 

``` python
del ends[tail[-1]]
```

which is supposed to snip out the `tail` contour from the `ends` dict. This presupposes that the `tail` counter is actually already in the `ends` dict - but it isn't. The trouble in tracking this back is that the exception happens later, when the deletion is attempted, rather than at the point where this entry was supposed to be added to `ends` in the first place.

However.

The purpose of that line is to remove the tail from `ends`. Except... it isn't there to be deleted. From a pragmatic perspective what happens if we just ignore this? The goal was to remove that entry - it's not there - mission complete? All of the other commands work, _including what really matters_ which is that `head` is extended by `tail` and then that result gets appended to `ends`. So, the contour is preserved and all is well if this specific line is wrapped in a `try: ... except`.

I've run it this way with a custom warning printed, and the result is as expected. Tests pass. The contour in question is correctly present. While this might not be the most intellectually satisfying solution, it certainly seems like a user would prefer getting perfectly good results from working code. 

If this sounds like a reasonable short-term fix to bridge into 0.11, I'll make a PR with the minor fix. Otherwise we may need another set of eyes on this one.

Thank you for following up and digging into the corner case.

On Sat, Feb 28, 2015 at 2:14 PM, Josh Warner notifications@github.com
wrote:

> @scikit-image/core https://github.com/orgs/scikit-image/teams/core I've
> gone round in circles on this one for about a week. The line that's failing
> is
> 
> del ends[tail[-1]]
> 
> which is supposed to snip out the tail contour from the ends dict. This
> presupposes that the tail counter is actually already in the ends dict -
> but it isn't. The trouble in tracking this back is that the exception
> happens later, when the deletion is attempted, rather than at the point
> where this entry was supposed to be added to ends in the first place.
> 
> However.
> 
> The purpose of that line is to remove the tail from ends. Except... it
> isn't there to be deleted. From a pragmatic perspective what happens if we
> just ignore this? The goal was to remove that entry - it's not there -
> mission complete? All of the other commands work, _including what really
> matters_ which is that head is extended by tail and then that result gets
> appended to ends. So, the contour is preserved and all is well if this
> specific line is wrapped in a try: ... except.
> 
> I've run it this way with a custom warning printed, and the result is as
> expected. Tests pass. The contour in question is correctly present. While
> this might not be the most intellectually satisfying solution, it certainly
> seems like a user would prefer getting perfectly good results from working
> code.
> 
> If this sounds like a reasonable short-term fix to bridge into 0.11, I'll
> make a PR with the minor fix. Otherwise we may need another set of eyes on
> this one.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/scikit-image/scikit-image/issues/817#issuecomment-76540861
> .

@JDWarner FWIW, I came up with the same workaround, except that I didn't have nearly such a nice description as to why it would be a good one!

Hmm, though this workaround seems to work, I suspect that it MUST be masking some deeper bug. The loop invariant here is:
- There is exactly one entry in `starts` and one entry in `ends` for every open contour in `contours`. (Open contours are those for which `contour[0] != contour[-1]`.)
- For each open contour, `contour[0]` is in `starts` and `contour[-1]` is in `ends`.

Plugging in a quick test that this invariant holds at each iteration would very rapidly pinpoint when things change that there is a contour that can't be indexed as `ends[contour[-1]]`. This would then be "the point where this entry was supposed to be added to `ends` in the first place" that Josh was looking for. My suspicion is either there's some floating-point-comparison bizarreness going on (though I'm not sure how that would work), or there's some degenerate case that I hadn't previously thought of. (E.g. where a contour makes some kind of lariat structure and gets closed too early? But I'm not sure how that would work either.)

Sorry I don't have time right now to sort this out in more detail myself, but I at least wanted to note my thoughts quickly. Probably this should be left as an open issue of some sort, even with the workaround in place. 

Also, note also that as symmetric as the code is, there's probably an equivalent issue with `del starts[head[0]]` on [_find_contours.py:182](https://github.com/scikit-image/scikit-image/blob/master/skimage/measure/_find_contours.py#L182).


Resolving patch:
diff --git a/skimage/measure/_find_contours.py b/skimage/measure/_find_contours.py
--- a/skimage/measure/_find_contours.py
+++ b/skimage/measure/_find_contours.py
@@ -1,5 +1,5 @@
 import numpy as np
-from . import _find_contours_cy
+from ._find_contours_cy import _get_contour_segments
 
 from collections import deque
 
@@ -114,47 +114,38 @@ def find_contours(array, level,
     [array([[0. , 0.5],
            [0.5, 0. ]])]
     """
-    array = np.asarray(array, dtype=np.double)
+    if fully_connected not in _param_options:
+        raise ValueError('Parameters "fully_connected" must be either '
+                         '"high" or "low".')
+    if positive_orientation not in _param_options:
+        raise ValueError('Parameters "positive_orientation" must be either '
+                         '"high" or "low".')
+    if array.shape[0] < 2 or array.shape[1] < 2:
+        raise ValueError("Input array must be at least 2x2.")
     if array.ndim != 2:
         raise ValueError('Only 2D arrays are supported.')
-    level = float(level)
     if mask is not None:
-        if not np.can_cast(mask.dtype, bool, casting='safe'):
-            raise TypeError('Parameter "mask" must be a binary array.')
-        mask = mask.view(np.uint8)
         if mask.shape != array.shape:
             raise ValueError('Parameters "array" and "mask"'
                              ' must have same shape.')
-    if (fully_connected not in _param_options or
-       positive_orientation not in _param_options):
-        raise ValueError('Parameters "fully_connected" and'
-        ' "positive_orientation" must be either "high" or "low".')
-    point_list = _find_contours_cy.iterate_and_store(array, level,
-                                                     fully_connected == 'high',
-                                                     mask=mask)
-    contours = _assemble_contours(_take_2(point_list))
+        if not np.can_cast(mask.dtype, bool, casting='safe'):
+            raise TypeError('Parameter "mask" must be a binary array.')
+        mask = mask.astype(np.uint8, copy=False)
+
+    segments = _get_contour_segments(array.astype(np.double), float(level),
+                                     fully_connected == 'high', mask=mask)
+    contours = _assemble_contours(segments)
     if positive_orientation == 'high':
         contours = [c[::-1] for c in contours]
     return contours
 
 
-def _take_2(seq):
-    iterator = iter(seq)
-    while True:
-        try:
-            n1 = next(iterator)
-            n2 = next(iterator)
-            yield (n1, n2)
-        except StopIteration:
-            return
-
-
-def _assemble_contours(points_iterator):
+def _assemble_contours(segments):
     current_index = 0
     contours = {}
     starts = {}
     ends = {}
-    for from_point, to_point in points_iterator:
+    for from_point, to_point in segments:
         # Ignore degenerate segments.
         # This happens when (and only when) one vertex of the square is
         # exactly the contour level, and the rest are above or below.
@@ -163,20 +154,15 @@ def _assemble_contours(points_iterator):
         if from_point == to_point:
             continue
 
-        tail_data = starts.get(to_point)
-        head_data = ends.get(from_point)
+        tail, tail_num = starts.pop(to_point, (None, None))
+        head, head_num = ends.pop(from_point, (None, None))
 
-        if tail_data is not None and head_data is not None:
-            tail, tail_num = tail_data
-            head, head_num = head_data
+        if tail is not None and head is not None:
             # We need to connect these two contours.
             if tail is head:
                 # We need to closed a contour.
-                # Add the end point, and remove the contour from the
-                # 'starts' and 'ends' dicts.
+                # Add the end point
                 head.append(to_point)
-                del starts[to_point]
-                del ends[from_point]
             else:  # tail is not head
                 # We need to join two distinct contours.
                 # We want to keep the first contour segment created, so that
@@ -185,47 +171,36 @@ def _assemble_contours(points_iterator):
                     # tail was created second. Append tail to head.
                     head.extend(tail)
                     # remove all traces of tail:
-                    del starts[to_point]
-                    try:
-                        del ends[tail[-1]]
-                    except KeyError:
-                        pass
-                    del contours[tail_num]
-                    # remove the old end of head and add the new end.
-                    del ends[from_point]
+                    ends.pop(tail[-1])
+                    contours.pop(tail_num, None)
+                    # Update contour starts end ends
+                    starts[head[0]] = (head, head_num)
                     ends[head[-1]] = (head, head_num)
                 else:  # tail_num <= head_num
                     # head was created second. Prepend head to tail.
                     tail.extendleft(reversed(head))
                     # remove all traces of head:
-                    del starts[head[0]]
-                    del ends[from_point]
-                    del contours[head_num]
-                    # remove the old start of tail and add the new start.
-                    del starts[to_point]
+                    starts.pop(head[0])
+                    contours.pop(head_num, None)
+                    # Update contour starts end ends
                     starts[tail[0]] = (tail, tail_num)
-        elif tail_data is None and head_data is None:
+                    ends[tail[-1]] = (tail, tail_num)
+        elif tail is None and head is None:
             # we need to add a new contour
-            current_index += 1
-            new_num = current_index
             new_contour = deque((from_point, to_point))
-            contours[new_num] = new_contour
-            starts[from_point] = (new_contour, new_num)
-            ends[to_point] = (new_contour, new_num)
-        elif tail_data is not None and head_data is None:
-            tail, tail_num = tail_data
+            contours[current_index] = new_contour
+            starts[from_point] = (new_contour, current_index)
+            ends[to_point] = (new_contour, current_index)
+            current_index += 1
+        elif head is None:  # tail is not None
             # We've found a single contour to which the new segment should be
             # prepended.
             tail.appendleft(from_point)
-            del starts[to_point]
             starts[from_point] = (tail, tail_num)
-        elif tail_data is None and head_data is not None:
-            head, head_num = head_data
+        else:  # tail is None and head is not None:
             # We've found a single contour to which the new segment should be
             # appended
             head.append(to_point)
-            del ends[from_point]
             ends[to_point] = (head, head_num)
-    # end iteration over from_ and to_ points
 
-    return [np.array(contour) for (num, contour) in sorted(contours.items())]
+    return [np.array(contour) for _, contour in sorted(contours.items())]


Test patch: