Repository: scikit-image/scikit-image

Issue:
RGB to HED conversion after version 0.17
## Description
After upgrading our [project](https://github.com/histolab/histolab) environment to version 0.18.x we have noticed that the RGB to HED conversion behaves unexpectedly according to the biological meaning of the HED color space. We have done some research and carefully read the [PR](https://github.com/scikit-image/scikit-image/pull/4725) changing the stain deconvolution method, however the HED conversion in version 0.17.x seemed to be more appropriate; in particular the H channel highlights the nuclei while the DAB channel highlights the brown stain:

**version 0.17.x**
![image](https://user-images.githubusercontent.com/31658006/103347675-f3754c80-4a97-11eb-9ba5-f8c267773fc5.png)

We are not able to find the same biological meaning in version 0.18.x:

**version 0.18.x**
![image](https://user-images.githubusercontent.com/31658006/103347671-ed7f6b80-4a97-11eb-86c3-b10a49687862.png)

Regards,
Nicole, Alessia, Ernesto (histolab team)

RGB to HED conversion after version 0.17
## Description
After upgrading our [project](https://github.com/histolab/histolab) environment to version 0.18.x we have noticed that the RGB to HED conversion behaves unexpectedly according to the biological meaning of the HED color space. We have done some research and carefully read the [PR](https://github.com/scikit-image/scikit-image/pull/4725) changing the stain deconvolution method, however the HED conversion in version 0.17.x seemed to be more appropriate; in particular the H channel highlights the nuclei while the DAB channel highlights the brown stain:

**version 0.17.x**
![image](https://user-images.githubusercontent.com/31658006/103347675-f3754c80-4a97-11eb-9ba5-f8c267773fc5.png)

We are not able to find the same biological meaning in version 0.18.x:

**version 0.18.x**
![image](https://user-images.githubusercontent.com/31658006/103347671-ed7f6b80-4a97-11eb-86c3-b10a49687862.png)

Regards,
Nicole, Alessia, Ernesto (histolab team)



Discussion:
Hi @nicolebussola thanks for reporting this. It's true that the doc example https://scikit-image.org/docs/dev/auto_examples/color_exposure/plot_ihc_color_separation.html does not look great. @alexdesiqueira since you worked on this PR could you please comment?
Hi @nicolebussola, thank you for opening this issue. Yeah @emmanuelle, the doc example doesn't "feel right" anymore.
I will need some time to check that, Nicole. As you saw, there was a lot of research — and people! — involved in #4725, and according to the papers, the solution we have now seem appropriate. However, as seen, something doesn't feel right...
My course of action here would be:

* Check out @crisluengo's implementation in [DIPLib](https://github.com/DIPlib/diplib/blob/28cdcc53f0735ab474d3e6e8862f83ea1f54f8bb/src/microscopy/unmix_stains.cpp). I'll run a couple of tests using their library; let's see their results using the same images. Maybe there's something missing in our latest implementation.
* Check if there's something we need to improve that example. I'd bet that a better colormap is needed to present the results. Not sure, though.

That said, how does that sound, Nicole? Is there something I'm missing, or would you like to add something else?
Thank you again!
In `skimage.color.separate_stains()` we're missing a clip operation. Adding `ihc_hed[ihc_hed<0] = 0` before display corrects things. I missed this in our rewrite of these functions, my apologies.

Current code for `separate_stains`:
```python
rgb = _prepare_colorarray(rgb, force_copy=True)
np.maximum(rgb, 1E-6, out=rgb)  # avoiding log artifacts
log_adjust = np.log(1E-6)  # used to compensate the sum above
stains = (np.log(rgb) / log_adjust) @ conv_matrix
```
In the last line, `np.log(rgb) / log_adjust` produces a correct image in the range [0,1]. But the rotation (`@ conv_matrix`) can lead to negative values. These negative values spoil the fun (actually spoil the display, which has a min-max stretch built in). Adding
```python
np.minimum(stains, 0, out= stains)
```
should fix the function.

If we create the input artificially by combining positive values and using the colors as assumed in `conv_matrix`, then the function will be guaranteed to produce only positive values, but the real world is not ideal: the stain colors are not the same as those given in `conv_matrix`, and the image has noise also.

Yes, this clipping makes the round-trip not idempotent. But if you start with a synthetic image, the clip will do nothing and hence the tests should continue to work. On the other hand,
```python
ihc_hed = rgb2hed(ihc_rgb)
reconstructed = hed2rgb(ihc_hed)
```
shows in `reconstructed` which colors for each stain are chosen in `conv_matrix` (actually `hed_from_rgb`). These colors don't exactly match the data: the DAB is more red, less yellow, and the hematoxylin is somewhat more purple. This also shows that the unmixing is inexact: the only way to get the exact colors is to measure them specifically for the microscope and chemistry sets used. Every lab produces different colors for the same stains...

Also, the example at https://scikit-image.org/docs/dev/auto_examples/color_exposure/plot_ihc_color_separation.html has a bug:
```python
cmap_eosin = LinearSegmentedColormap.from_list('mycmap', ['darkviolet', 'white'])
```
The two colors are reversed, this should be:
```python
cmap_eosin = LinearSegmentedColormap.from_list('mycmap', ['white', 'darkviolet'])
```
Nobody noticed this before because the image has no Eosin, and so this channel has only residue (= negative values). Reversing the colormap actually looks good in this case, but the image should be all zeros (white), which we do see after clipping negative values.

![Figure_2](https://user-images.githubusercontent.com/7302924/103558158-79670e00-4e71-11eb-822d-0f861f9ba910.png)

Finally, the DAB channel (brown) looks rather dim. This is because there is one pixel that happens to be really dense (maybe noise?) and the image is scaled to that. But this is just a display issue. If you want to avoid this, you can do like in the ImageJ plugin, where they don't use 1e-6 but 1/255 in the initial clip. This causes the darkest stain to be less dark. This is a question about what you prefer: better display or better data.
OK, one last comment from me and then I'll shut up. :)

I'd change the demo as follows for proper display of the unmixed stains. What we're doing here is creating an image in "hed color space" that has only one stain, then using `hed2rgb` to produce a correct representation of that stain. There is now no worry about scaling of the images. The result of `hed2rgb` should be comparable in intensities to the original input.

```python
import matplotlib.pyplot as plt
import numpy as np

from skimage import data
from skimage.color import rgb2hed, hed2rgb


ihc_rgb = data.immunohistochemistry()
ihc_hed = rgb2hed(ihc_rgb)
ihc_hed[ihc_hed<0] = 0  # TODO: remove this after fixing `rgb2hed()`

null = np.zeros_like(ihc_hed[:,:,0])
ihc_h = hed2rgb(np.stack((ihc_hed[:,:,0], null, null), axis=-1))
ihc_e = hed2rgb(np.stack((null, ihc_hed[:,:,1], null), axis=-1))
ihc_d = hed2rgb(np.stack((null, null, ihc_hed[:,:,2]), axis=-1))

fig, axes = plt.subplots(2, 2, figsize=(7, 6), sharex=True, sharey=True)
ax = axes.ravel()

ax[0].imshow(ihc_rgb)
ax[0].set_title("Original image")

ax[1].imshow(ihc_h)
ax[1].set_title("Hematoxylin")

ax[2].imshow(ihc_e)
ax[2].set_title("Eosin")

ax[3].imshow(ihc_d)
ax[3].set_title("DAB")

for a in ax.ravel():
    a.axis('off')

fig.tight_layout()
plt.show()
```

It is still fine to use `ihc_hed[:,:,0]` etc. as in the second part of the example.

![Figure_3](https://user-images.githubusercontent.com/7302924/103562923-24c79100-4e79-11eb-915b-512b72a62d57.png)

Thanks for sorting this out @crisluengo! This is looking much better and the fix is pretty straightforward. Let us know if you want to make a PR, otherwise we can take care of it. 
Hi @nicolebussola thanks for reporting this. It's true that the doc example https://scikit-image.org/docs/dev/auto_examples/color_exposure/plot_ihc_color_separation.html does not look great. @alexdesiqueira since you worked on this PR could you please comment?
Hi @nicolebussola, thank you for opening this issue. Yeah @emmanuelle, the doc example doesn't "feel right" anymore.
I will need some time to check that, Nicole. As you saw, there was a lot of research — and people! — involved in #4725, and according to the papers, the solution we have now seem appropriate. However, as seen, something doesn't feel right...
My course of action here would be:

* Check out @crisluengo's implementation in [DIPLib](https://github.com/DIPlib/diplib/blob/28cdcc53f0735ab474d3e6e8862f83ea1f54f8bb/src/microscopy/unmix_stains.cpp). I'll run a couple of tests using their library; let's see their results using the same images. Maybe there's something missing in our latest implementation.
* Check if there's something we need to improve that example. I'd bet that a better colormap is needed to present the results. Not sure, though.

That said, how does that sound, Nicole? Is there something I'm missing, or would you like to add something else?
Thank you again!
In `skimage.color.separate_stains()` we're missing a clip operation. Adding `ihc_hed[ihc_hed<0] = 0` before display corrects things. I missed this in our rewrite of these functions, my apologies.

Current code for `separate_stains`:
```python
rgb = _prepare_colorarray(rgb, force_copy=True)
np.maximum(rgb, 1E-6, out=rgb)  # avoiding log artifacts
log_adjust = np.log(1E-6)  # used to compensate the sum above
stains = (np.log(rgb) / log_adjust) @ conv_matrix
```
In the last line, `np.log(rgb) / log_adjust` produces a correct image in the range [0,1]. But the rotation (`@ conv_matrix`) can lead to negative values. These negative values spoil the fun (actually spoil the display, which has a min-max stretch built in). Adding
```python
np.minimum(stains, 0, out= stains)
```
should fix the function.

If we create the input artificially by combining positive values and using the colors as assumed in `conv_matrix`, then the function will be guaranteed to produce only positive values, but the real world is not ideal: the stain colors are not the same as those given in `conv_matrix`, and the image has noise also.

Yes, this clipping makes the round-trip not idempotent. But if you start with a synthetic image, the clip will do nothing and hence the tests should continue to work. On the other hand,
```python
ihc_hed = rgb2hed(ihc_rgb)
reconstructed = hed2rgb(ihc_hed)
```
shows in `reconstructed` which colors for each stain are chosen in `conv_matrix` (actually `hed_from_rgb`). These colors don't exactly match the data: the DAB is more red, less yellow, and the hematoxylin is somewhat more purple. This also shows that the unmixing is inexact: the only way to get the exact colors is to measure them specifically for the microscope and chemistry sets used. Every lab produces different colors for the same stains...

Also, the example at https://scikit-image.org/docs/dev/auto_examples/color_exposure/plot_ihc_color_separation.html has a bug:
```python
cmap_eosin = LinearSegmentedColormap.from_list('mycmap', ['darkviolet', 'white'])
```
The two colors are reversed, this should be:
```python
cmap_eosin = LinearSegmentedColormap.from_list('mycmap', ['white', 'darkviolet'])
```
Nobody noticed this before because the image has no Eosin, and so this channel has only residue (= negative values). Reversing the colormap actually looks good in this case, but the image should be all zeros (white), which we do see after clipping negative values.

![Figure_2](https://user-images.githubusercontent.com/7302924/103558158-79670e00-4e71-11eb-822d-0f861f9ba910.png)

Finally, the DAB channel (brown) looks rather dim. This is because there is one pixel that happens to be really dense (maybe noise?) and the image is scaled to that. But this is just a display issue. If you want to avoid this, you can do like in the ImageJ plugin, where they don't use 1e-6 but 1/255 in the initial clip. This causes the darkest stain to be less dark. This is a question about what you prefer: better display or better data.
OK, one last comment from me and then I'll shut up. :)

I'd change the demo as follows for proper display of the unmixed stains. What we're doing here is creating an image in "hed color space" that has only one stain, then using `hed2rgb` to produce a correct representation of that stain. There is now no worry about scaling of the images. The result of `hed2rgb` should be comparable in intensities to the original input.

```python
import matplotlib.pyplot as plt
import numpy as np

from skimage import data
from skimage.color import rgb2hed, hed2rgb


ihc_rgb = data.immunohistochemistry()
ihc_hed = rgb2hed(ihc_rgb)
ihc_hed[ihc_hed<0] = 0  # TODO: remove this after fixing `rgb2hed()`

null = np.zeros_like(ihc_hed[:,:,0])
ihc_h = hed2rgb(np.stack((ihc_hed[:,:,0], null, null), axis=-1))
ihc_e = hed2rgb(np.stack((null, ihc_hed[:,:,1], null), axis=-1))
ihc_d = hed2rgb(np.stack((null, null, ihc_hed[:,:,2]), axis=-1))

fig, axes = plt.subplots(2, 2, figsize=(7, 6), sharex=True, sharey=True)
ax = axes.ravel()

ax[0].imshow(ihc_rgb)
ax[0].set_title("Original image")

ax[1].imshow(ihc_h)
ax[1].set_title("Hematoxylin")

ax[2].imshow(ihc_e)
ax[2].set_title("Eosin")

ax[3].imshow(ihc_d)
ax[3].set_title("DAB")

for a in ax.ravel():
    a.axis('off')

fig.tight_layout()
plt.show()
```

It is still fine to use `ihc_hed[:,:,0]` etc. as in the second part of the example.

![Figure_3](https://user-images.githubusercontent.com/7302924/103562923-24c79100-4e79-11eb-915b-512b72a62d57.png)

Thanks for sorting this out @crisluengo! This is looking much better and the fix is pretty straightforward. Let us know if you want to make a PR, otherwise we can take care of it. 

Resolving patch:
diff --git a/doc/examples/color_exposure/plot_ihc_color_separation.py b/doc/examples/color_exposure/plot_ihc_color_separation.py
--- a/doc/examples/color_exposure/plot_ihc_color_separation.py
+++ b/doc/examples/color_exposure/plot_ihc_color_separation.py
@@ -19,35 +19,38 @@
        American Society of Cytology, vol. 23, no. 4, pp. 291-9, Aug. 2001.
 
 """
+import numpy as np
 import matplotlib.pyplot as plt
 
 from skimage import data
-from skimage.color import rgb2hed
-from matplotlib.colors import LinearSegmentedColormap
-
-# Create an artificial color close to the original one
-cmap_hema = LinearSegmentedColormap.from_list('mycmap', ['white', 'navy'])
-cmap_dab = LinearSegmentedColormap.from_list('mycmap', ['white',
-                                             'saddlebrown'])
-cmap_eosin = LinearSegmentedColormap.from_list('mycmap', ['darkviolet',
-                                               'white'])
+from skimage.color import rgb2hed, hed2rgb
 
+# Example IHC image
 ihc_rgb = data.immunohistochemistry()
+
+# Separate the stains from the IHC image
 ihc_hed = rgb2hed(ihc_rgb)
 
+# Create an RGB image for each of the stains
+null = np.zeros_like(ihc_hed[:, :, 0])
+ihc_h = hed2rgb(np.stack((ihc_hed[:, :, 0], null, null), axis=-1))
+ihc_e = hed2rgb(np.stack((null, ihc_hed[:, :, 1], null), axis=-1))
+ihc_d = hed2rgb(np.stack((null, null, ihc_hed[:, :, 2]), axis=-1))
+
+# Display
 fig, axes = plt.subplots(2, 2, figsize=(7, 6), sharex=True, sharey=True)
 ax = axes.ravel()
 
 ax[0].imshow(ihc_rgb)
 ax[0].set_title("Original image")
 
-ax[1].imshow(ihc_hed[:, :, 0], cmap=cmap_hema)
+ax[1].imshow(ihc_h)
 ax[1].set_title("Hematoxylin")
 
-ax[2].imshow(ihc_hed[:, :, 1], cmap=cmap_eosin)
-ax[2].set_title("Eosin")
+ax[2].imshow(ihc_e)
+ax[2].set_title("Eosin")  # Note that there is no Eosin stain in this image
 
-ax[3].imshow(ihc_hed[:, :, 2], cmap=cmap_dab)
+ax[3].imshow(ihc_d)
 ax[3].set_title("DAB")
 
 for a in ax.ravel():
@@ -59,13 +62,16 @@
 ######################################################################
 # Now we can easily manipulate the hematoxylin and DAB "channels":
 
-import numpy as np
 from skimage.exposure import rescale_intensity
 
 # Rescale hematoxylin and DAB signals and give them a fluorescence look
-h = rescale_intensity(ihc_hed[:, :, 0], out_range=(0, 1))
-d = rescale_intensity(ihc_hed[:, :, 2], out_range=(0, 1))
-zdh = np.dstack((np.zeros_like(h), d, h))
+h = rescale_intensity(ihc_hed[:, :, 0], out_range=(0, 1),
+                      in_range=(0, np.percentile(ihc_hed[:, :, 0], 99)))
+d = rescale_intensity(ihc_hed[:, :, 2], out_range=(0, 1),
+                      in_range=(0, np.percentile(ihc_hed[:, :, 2], 99)))
+
+# Put the two channels into an RGB image as green and blue channels
+zdh = np.dstack((null, d, h))
 
 fig = plt.figure()
 axis = plt.subplot(1, 1, 1, sharex=ax[0], sharey=ax[0])
diff --git a/skimage/color/colorconv.py b/skimage/color/colorconv.py
--- a/skimage/color/colorconv.py
+++ b/skimage/color/colorconv.py
@@ -1451,6 +1451,8 @@ def separate_stains(rgb, conv_matrix):
 
     stains = (np.log(rgb) / log_adjust) @ conv_matrix
 
+    np.maximum(stains, 0, out=stains)
+
     return stains
 
 


Test patch:
diff --git a/skimage/color/tests/test_colorconv.py b/skimage/color/tests/test_colorconv.py
--- a/skimage/color/tests/test_colorconv.py
+++ b/skimage/color/tests/test_colorconv.py
@@ -47,6 +47,7 @@ class TestColorconv(TestCase):
     img_rgba = np.array([[[0, 0.5, 1, 0],
                           [0, 0.5, 1, 1],
                           [0, 0.5, 1, 0.5]]]).astype(float)
+    img_stains = img_as_float(img_rgb) * 0.3
 
     colbars = np.array([[1, 1, 0, 0, 1, 1, 0, 0],
                         [1, 1, 1, 1, 0, 0, 0, 0],
@@ -183,32 +184,33 @@ def test_xyz_rgb_roundtrip(self):
         img_rgb = img_as_float(self.img_rgb)
         assert_array_almost_equal(xyz2rgb(rgb2xyz(img_rgb)), img_rgb)
 
-    # RGB<->HED roundtrip with ubyte image
+    # HED<->RGB roundtrip with ubyte image
     def test_hed_rgb_roundtrip(self):
-        img_rgb = img_as_ubyte(self.img_rgb)
-        new = img_as_ubyte(hed2rgb(rgb2hed(img_rgb)))
-        assert_equal(new, img_rgb)
+        img_in = img_as_ubyte(self.img_stains)
+        img_out = rgb2hed(hed2rgb(img_in))
+        assert_equal(img_as_ubyte(img_out), img_in)
 
-    # RGB<->HED roundtrip with float image
+    # HED<->RGB roundtrip with float image
     def test_hed_rgb_float_roundtrip(self):
-        img_rgb = img_as_float(self.img_rgb)
-        assert_array_almost_equal(hed2rgb(rgb2hed(img_rgb)), img_rgb)
-
-    # RGB<->HDX roundtrip with ubyte image
-    def test_hdx_rgb_roundtrip(self):
-        from skimage.color.colorconv import hdx_from_rgb, rgb_from_hdx
-        img_rgb = self.img_rgb
-        conv = combine_stains(separate_stains(img_rgb, hdx_from_rgb),
-                              rgb_from_hdx)
-        assert_equal(img_as_ubyte(conv), img_rgb)
-
-    # RGB<->HDX roundtrip with float image
-    def test_hdx_rgb_roundtrip_float(self):
-        from skimage.color.colorconv import hdx_from_rgb, rgb_from_hdx
-        img_rgb = img_as_float(self.img_rgb)
-        conv = combine_stains(separate_stains(img_rgb, hdx_from_rgb),
-                              rgb_from_hdx)
-        assert_array_almost_equal(conv, img_rgb)
+        img_in = self.img_stains
+        img_out = rgb2hed(hed2rgb(img_in))
+        assert_array_almost_equal(img_out, img_in)
+
+    # BRO<->RGB roundtrip with ubyte image
+    def test_bro_rgb_roundtrip(self):
+        from skimage.color.colorconv import bro_from_rgb, rgb_from_bro
+        img_in = img_as_ubyte(self.img_stains)
+        img_out = combine_stains(img_in, rgb_from_bro)
+        img_out = separate_stains(img_out, bro_from_rgb)
+        assert_equal(img_as_ubyte(img_out), img_in)
+
+    # BRO<->RGB roundtrip with float image
+    def test_bro_rgb_roundtrip_float(self):
+        from skimage.color.colorconv import bro_from_rgb, rgb_from_bro
+        img_in = self.img_stains
+        img_out = combine_stains(img_in, rgb_from_bro)
+        img_out = separate_stains(img_out, bro_from_rgb)
+        assert_array_almost_equal(img_out, img_in)
 
     # RGB to RGB CIE
     def test_rgb2rgbcie_conversion(self):