Repository: scikit-image/scikit-image

Issue:
threshold_li changed from version 0.14.0 to version 0.14.1
## Description
The code for threshold_li in filters/thresholding.py changed from version 0.14.0 to 0.14.1. The new method does not work at all for my images.

Example data (test.npy must be unzipped): [test.zip](https://github.com/scikit-image/scikit-image/files/2691279/test.zip)

```python
import matplotlib.pylab as plt
import numpy as np
from skimage.filters import threshold_li

image = np.load("test.npy")
thresh = threshold_li(image)

ax1 = plt.subplot(121)
ax1.imshow(image)

ax2 = plt.subplot(122)
ax2 = plt.imshow(image > thresh)

plt.savefig("result.png")
```

Old code (0.14.0):
https://github.com/scikit-image/scikit-image/blob/v0.14.0/skimage/filters/thresholding.py#L519-L533
![result_14 0](https://user-images.githubusercontent.com/2853022/50166366-374ced00-02e7-11e9-8150-adf119dc6774.png)

New code (0.14.1):
https://github.com/scikit-image/scikit-image/blob/v0.14.1/skimage/filters/thresholding.py#L594-L604
![result_14 1](https://user-images.githubusercontent.com/2853022/50166481-754a1100-02e7-11e9-8e8f-8fffb079424d.png)

Not sure whether this change is intended. The while loop in the new code seems to be different. For one, the parameter ``threshold`` is not used there.


Discussion:
@jni This is something you changed in this commit... Do you remember? https://github.com/scikit-image/scikit-image/commit/f328085c6d2771fdffc13822b9c135cd089b1c00#diff-3d45990596f953f3e60ccfb305cf8924
@sciunto @paulmueller yep, I changed this in https://github.com/scikit-image/scikit-image/pull/3402 and I remember it well, because I spent a long time reviewing the code and the original paper. Long story short: unless I am severely mistaken, our implementation of Li thresholding was incorrect before and it was just a coincidence that it gave good results for your use case.

How to test my hypothesis:

- `from skimage.filters.thresholding import _cross_entropy`
- try *all possible threshold values* and measure the cross entropy with `_cross_entropy(image, threshold)`. This is the measure that `threshold_li` tries to minimise.
- Check whether the returned value by `threshold_li` matches the above-found threshold

More practically, maybe another thresholding method like `threshold_otsu` would work better on your image?

I intend to try all these things out later today myself but thought I'd get you started if you have time. Sorry about breaking your pipeline!
Ach, I take it back! What's changed is something about our tolerance and how it interacts with this image.

Here is the cross-entropy vs threshold plot:

![cross-entropy-test](https://user-images.githubusercontent.com/492549/50205449-58334200-03bc-11e9-8f95-e57184309467.png)

And here is what `threshold_li` finds:

```python
In [38]: threshold_li(image)
Out[38]: 118.14975555282838
```

So it looks like the optimization iteration got stuck in that local minimum/low gradient region. Here's what happens when we decrease the tolerance:

```python
In [39]: threshold_li(image, tolerance=0.0001)
Out[39]: 143.45882223438937
```

What I don't understand is why the default tolerance no longer works, because that condition hasn't changed, as far as I can tell. We didn't actually backport the tolerance keyword argument to 0.14.x so you can't try it out on that release. @paulmueller are you on Python 3.x or 2.7? If 3.x then this problem would be solved (for you at least) if we release 0.15.0. The next few days are a good candidate for this since hopefully we'll have some cool stuff merged after the ongoing Paris sprint.

Anyway, I'll keep investigating why there would be a difference to see whether we need to change the default tolerance value or something. Sorry again! And sorry for my earlier unwarranted confidence! Is there a humble pie emoji?
Ok so here is the old code:

https://github.com/scikit-image/scikit-image/blob/5f91f5456e42370beb2fd05b75f5e4ac0fdc6364/skimage/filters/thresholding.py#L521-L539

vs the new code

https://github.com/scikit-image/scikit-image/blob/5b6791eb9def2fd8298f5e035a3151205f35229d/skimage/filters/thresholding.py#L588-L601

Notice how in the old code, the threshold gets updated *twice* in the iteration: first, the threshold used inside the while loop is `threshold + tolerance`, rather than `threshold`. This pushes the threshold along in the positive direction. Then in the update, `tolerance` gets added to `t_next` *again*. So, in effect, in the old code, the iteration was massively biased towards moving in the increasing direction. This makes sense in many images because most of the image is dark/background, so the mean will be lower than the desired threshold, and the iteration will indeed "want" to move rightward. But it's *not* Li thresholding, it's something more akin to "biased accelerated Li thresholding".

Through trial and error (with binary search) I found that I could get away with 1/60th of the current default tolerance, which is `1/2 * 1/256 * im_range`. So probably we should use something along those lines as the new default, maybe `1/128 * 1/256 * im_range`?
@jni Thanks for your replies. I am on Python 3.6, so I will just stick to 0.14.0 for now.

[EDIT]: I would like to point out that these changes were not mentioned in the release notes https://github.com/scikit-image/scikit-image/blob/master/doc/release/release_0.14.rst.
@paulmueller 

> these changes were not mentioned in the release notes

a very good point! Sorry about that, I don't know how or why I missed that PR when compiling the notes. I'll aim to fix that also.

@scikit-image/core any comments on changing the default tolerance?
My opinion: Keep the new version that sticks to the original implementation. Change the default tolerance (It's not problematic since we changed the implementation). Backport in 0.14.2. Add all this in the notes :)


On Wed, 19 Dec 2018 22:37:17 -0800, François Boulogne wrote:
> My opinion: Keep the new version that sticks to the original
> implementation. Change the default tolerance (It's not problematic
> since we changed the implementation). Backport in 0.14.2. Add all this
> in the notes :)

Changing public API in a point release is problematic.  Even if we
consider this a bug-fix of sorts, I recommend a warning to the user.

@stefanv the public API was not changed before and will not change now. The tolerance is an internal parameter.
But we are changing behavior.  I.e., the same API call gives different results between 0.14.0 and 0.14.1.
Our policy has always been to do this silently when it was a bug fix. Running scikit-image would be a warning salad if we warned every time we changed some internal thing. It was definitely a mistake not to mention anything in the release notes, but other than that I think fixing an algorithm should be silent. And anyway, the goal of changing the tolerance now is for the behavior in 0.14.2 to match 0.14.0 (minus a 1/255 offset which again was buggy before).
If 0.14.2 will match 0.14.0, that sounds right.
@paulmueller could we have your permission to use your image in our documentation and testing? For this we would need it to be licensed as CC0 (public domain) or CC-BY (attribution) (@stefanv can you confirm that we could include CC-BY?).
@jni: CC0 is fine, use it!
Awesome, thanks! What should we call it? `data.beads()`? =) If you have any additional information, e.g. the pixel spacing, that would be nice-to-have (obviously not essential).
It should be called ``data.cells()`` or ``data.cells_phase()``/``data.cells_qpi()`` (more accurate, "qpi" stands for quantitative phase imaging).
Though, I just realized that the data are preprocessed with another filter. I will dig out a better picture (also with more cells) that reproduces the bug and upload it here with a more detailed description.
Here is the new dataset: [cells_qpi_zipped.zip](https://github.com/scikit-image/scikit-image/files/2712598/cells_qpi_zipped.zip)

Meta data (include as much as seems necessary):
```
This is a quantitative phase image retrieved from a digital hologram using the Python library qpformat. The data show two cells (left and right) with high phase values and an imaging artifact of the optical system (center) with phase values close to the background phase.

sample: HL60/S4 cells
medium: phosphate buffered saline (PBS, n=1.335)
imaging wavelength: 633nm
pixel size: 0.107µm
phase units: radians
imaging technique: digital holography
date of acquisition: 2015/10/29

context:
These data were part of a comparison between several refractive index retrieval
techniques for spherical objects as part of the publication (figure 5d):

  Paul Müller, Mirjam Schürmann, Salvatore Girardo, Gheorghe Cojoc,
  and Jochen Guck 
  "Accurate evaluation of size and refractive index for spherical
  objects in quantitative phase imaging,"
  Optics Express 26(8): 10729-10743, 2018.
  doi:10.1364/OE.26.010729

license: CC0
These data are dedicated to the public domain. You can copy, modify,
or distribute them without asking permission.
```

In order to test ``threshold_li``, the data have to be preprocessed with ``threshold_local``.
This breaks it down pretty much:

```python
import matplotlib.pylab as plt
import numpy as np
from skimage.filters import threshold_li, threshold_local

# load original phase data
data = np.load("cells_qpi.npz")
phase = np.array(data["arr_0"], dtype=float)

# apply local threshold
locthr = threshold_local(phase, block_size=331)
phaset = phase - locthr

# reproduce li threshold
thresh = threshold_li(phaset)
assert thresh > 1  # I got 1.0479158553337644


plt.subplot(131, title="original")
plt.imshow(phase)

plt.subplot(132, title="preprocessed")
plt.imshow(phaset)


plt.subplot(133, title="threshold li")
plt.imshow(phaset > thresh)


plt.savefig("thresh.png")
plt.show()
```
![thresh_ex](https://user-images.githubusercontent.com/2853022/50483116-8c28eb80-09ea-11e9-9d86-997282bfac65.png)



Thanks Paul! This is great!
@scikit-image/core how do we feel about adding this image (1.8MB) to our data? It's large, but imho this will be in for one release before we implement remote data. Or do we want to wait until remote data is implemented? If the latter, then I think #3622 should be merged before tests and a gallery example are implemented, because it is a regression bug fix, so I'd like it to be in the next release.
are we planning to just host data on github? 

Files are currently at 20MB+. I feel like adding 2MB is not insignificant. Any way to compress the data? or to drop the resolution/quality. at least temporarily? PNG compression maybe?
I would also prefer a more conservative approach to this especially if a long term solution would make this obsolete.
@hmaarrfk I agree with you, which is why I stopped myself before adding the data. The data is already compressed. The problem is that the problem happens when you have a very different magnitude between the range of the data and the min diff between two values. So there's no simple way to test this.

But, I'll point out, in #3622 the current tests pass, the coverage is the same, and the issue is fixed, so I'm strongly in favour of merging that but keeping this issue open to add the regression test once the new data machinery is in place.

Resolving patch:
diff --git a/skimage/filters/thresholding.py b/skimage/filters/thresholding.py
--- a/skimage/filters/thresholding.py
+++ b/skimage/filters/thresholding.py
@@ -1,3 +1,4 @@
+from __future__ import division, print_function
 import itertools
 import math
 import numpy as np
@@ -529,11 +530,6 @@ def threshold_li(image):
     image : ndarray
         Input image.
 
-    tolerance : float, optional
-        Finish the computation when the change in the threshold in an iteration
-        is less than this value. By default, this is half of the range of the
-        input image, divided by 256.
-
     Returns
     -------
     threshold : float
@@ -582,8 +578,7 @@ def threshold_li(image):
     # Li's algorithm requires positive image (because of log(mean))
     image_min = np.min(image)
     image -= image_min
-    image_range = np.max(image)
-    tolerance = 0.5 * image_range / 256
+    tolerance = np.min(np.diff(np.unique(image))) / 2
 
     # Initial estimate
     t_curr = np.mean(image)


Test patch: