Repository: scikit-image/scikit-image

Issue:
warp/rotate: inconsistency with clipping when cval is outside of the input range
## Description

For warp/rotate, when specifying `clip=True` (which is the default), the behavior when `cval` is outside of the range of the input image depends on the specified interpolation order.  For orders 0, 1, and 3 (which use native code), any pixels in the output image that are `cval` are preserved as `cval`.  For orders 2, 4, and 5 (which fall back to SciPy code), the pixels in the output image that are `cval` invariably get clipped to be within the input range.  This happens because the check in `_clip_warp_output()`:
https://github.com/scikit-image/scikit-image/blob/e562d9f8914c804151b5247bd2e778d53510904f/skimage/transform/_warps.py#L727
is too strict now that the SciPy code is called using the mode `grid-constant` instead of `constant` as of scikit-image 0.19.  That is, values outside of the original image are no longer exactly equal to `cval` because of the way `grid-constant` works.

<!--
(Note: for guidance on how to use `scikit-image`, please post instead on https://forum.image.sc/tag/scikit-image)
-->

## Way to reproduce
```python
import matplotlib.pyplot as plt
import numpy as np
from skimage.transform import rotate

image = np.ones((20, 20))
image[5:-5, 5:-5] = 2

fig, axs = plt.subplots(2, 6, figsize=(13, 4))

for i in range(6):
    axs[0, i].set_title(f'order={i}')
    axs[0, i].imshow(rotate(image, 30, order=i, cval=0, resize=True, clip=False), vmin=0, vmax=3)
    axs[1, i].imshow(rotate(image, 30, order=i, cval=0, resize=True, clip=True), vmin=0, vmax=3)
    
axs[0, 0].set_ylabel('clip=False')
axs[1, 0].set_ylabel('clip=True')

plt.show()
```
![download (49)](https://user-images.githubusercontent.com/991759/160432313-b1e9696d-4f1c-451f-84ff-f96f2110df02.png)

## Version information
```python
# Paste the output of the following python commands
from __future__ import print_function
import sys; print(sys.version)
import platform; print(platform.platform())
import skimage; print(f'scikit-image version: {skimage.__version__}')
import numpy; print(f'numpy version: {numpy.__version__}')
```

```python
3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 15:50:08) [MSC v.1916 64 bit (AMD64)]
Windows-10-10.0.19041-SP0
scikit-image version: 0.19.2
numpy version: 1.20.2
```
And scipy is 1.8.0
warp : inconsistency between cval=np.nan and cval=0 (or any other values)
Hello,

Boundary positions can differ when using cval=np.nan in comparison to real values.
It could be good to have the same behaviour regardless the value you pass to cval.
Thanks,
Patrick

numpy : 1.19.5
scikit-image : 0.18.3

Hereafter : 
- 1rst block with 'pixel' transformation : **no difference** in the boundary position between cval=np.nan and cval=0
- 2nd block with 'subpixel' transformation : **difference** in the boundary position

```
>>> import numpy as np
>>> from skimage.transform import AffineTransform, warp

>>> img = np.ones((5, 5))

>>> tmat = np.array([[1, 0, -2], [0, 1, 0], [0, 0, 1]])
>>> tform = AffineTransform(matrix=tmat)
>>> warp(img, tform, cval=np.nan)
array([[nan, nan,  1.,  1.,  1.],
       [nan, nan,  1.,  1.,  1.],
       [nan, nan,  1.,  1.,  1.],
       [nan, nan,  1.,  1.,  1.],
       [nan, nan,  1.,  1.,  1.]])
>>> warp(img, tform, cval=0)
array([[0., 0., 1., 1., 1.],
       [0., 0., 1., 1., 1.],
       [0., 0., 1., 1., 1.],
       [0., 0., 1., 1., 1.],
       [0., 0., 1., 1., 1.]])

>>> tmat = np.array([[1, 0, -2.3], [0, 1, 0], [0, 0, 1]])
>>> tform = AffineTransform(matrix=tmat)
>>> warp(img, tform, cval=np.nan)
array([[nan, nan, nan,  1.,  1.],
       [nan, nan, nan,  1.,  1.],
       [nan, nan, nan,  1.,  1.],
       [nan, nan, nan,  1.,  1.],
       [nan, nan, nan,  1.,  1.]])
>>> warp(img, tform, cval=0)
array([[0., 0., 1., 1., 1.],
       [0., 0., 1., 1., 1.],
       [0., 0., 1., 1., 1.],
       [0., 0., 1., 1., 1.],
       [0., 0., 1., 1., 1.]])
```


Discussion:
Thanks for reporting this @ayshih. Indeed, it looks like we should adjust the clipping range to take `cval` into account when `mode='constant'`. Are you interested in making a PR to fix this?

I can confirm this on current main e562d9f8914c804151b5247bd2e778d53510904f. @ayshih thank you for the descriptive report! We'd be happy to help you with a PR. 
Thank you @patquem for the report, I can reproduce the bug. `scipy.ndimage` gives the expected result:
```python
>>> from skimage.transform import warp_coords
>>> from scipy import ndimage as ndi
>>> coords = warp_coords(tform, (5, 5))
>>> ndi.map_coordinates(img, coords)
array([[0., 0., 0., 1., 1.],
       [0., 0., 0., 1., 1.],
       [0., 0., 0., 1., 1.],
       [0., 0., 0., 1., 1.],
       [0., 0., 0., 1., 1.]])
```
The result with cval=0 is actually the following before a final call to `_clip_warp_output`:

```Python
array([[0. , 0. , 0.7, 1. , 1. ],
       [0. , 0. , 0.7, 1. , 1. ],
       [0. , 0. , 0.7, 1. , 1. ],
       [0. , 0. , 0.7, 1. , 1. ],
       [0. , 0. , 0.7, 1. , 1. ]])
```
This matches what one would get in SciPy 1.6+ with `ndi.map_coordinates(img, coords, mode='grid-constant')`. Note that SciPy's `grid-constant` mode is different than `constant` mode and interpolates between the edge value of 1.0 and the background of 0.0 to give 0.7. scikit-image's Cython-based 2D interpolation routines in 'constant' mode behave like scipy.ndimage's `'grid-constant'`. By contrast, scipy's mode='constant' sets any value outside the image range immediately to cval rather than interpolating between the image edge and the background.

As for why the values of 0.7 become 1.0 in your example, that seems to be a strange behavior (bug?) of  `_clip_warp_output` when the `cval` is outside the range of `image.min()` or `image.max()`. For your uniform image of ones, the clip is actually applied with min=1, max=1 which gives an output that is all ones, but then any values that were originally equal to `cval` are restored to `cval` ([see code here](https://github.com/scikit-image/scikit-image/blob/d44ceda6241cb23a22dc8abf09a05090ed14da7f/skimage/transform/_warps.py#L721-L733)). Since 0.7 was not originally == 0.0, it does not get set to 0.

If you call the `warp` function with `clip=False` you should get the values with 0.7 as above.

This value of 0.7 behavior is consistent with a value of np.nan in the NaN case since interpolation between a value of 1.0 and np.nan results in np.nan.
Many thanks @grlee77  for your detailed explanations.
Indeed, when working with order=0, 'boundaries' are the same between cval=np.nan and cval=0.
That confirms the interpolation effect you mentionned above.
```
>>> tmat = np.array([[1, 0, -2.3], [0, 1, 0], [0, 0, 1]])
>>> tform = AffineTransform(matrix=tmat)
>>> warp(img, tform, cval=np.nan, order=0)
array([[nan, nan,  1.,  1.,  1.],
       [nan, nan,  1.,  1.,  1.],
       [nan, nan,  1.,  1.,  1.],
       [nan, nan,  1.,  1.,  1.],
       [nan, nan,  1.,  1.,  1.]])
>>> warp(img, tform, cval=0, order=0)
array([[0., 0., 1., 1., 1.],
       [0., 0., 1., 1., 1.],
       [0., 0., 1., 1., 1.],
       [0., 0., 1., 1., 1.],
       [0., 0., 1., 1., 1.]])
```

Thank you @grlee77 for your investigations :wink:.

> As for why the values of 0.7 become 1.0 in your example, that seems to be a strange behavior (bug?) of `_clip_warp_output` when the `cval` is outside the range of `image.min()` or `image.max()`. For your uniform image of ones, the clip is actually applied with min=1, max=1 which gives an output that is all ones, but then any values that were originally equal to `cval` are restored to `cval` ([see code here](https://github.com/scikit-image/scikit-image/blob/d44ceda6241cb23a22dc8abf09a05090ed14da7f/skimage/transform/_warps.py#L721-L733)). Since 0.7 was not originally == 0.0, it does not get set to 0.

I don't see this as a bug, but for sure a strange behavior to be at least documented. An other option would be to allow giving a couple of values for the `clip` parameter corresponding to the clip values...

Resolving patch:
diff --git a/skimage/transform/_warps.py b/skimage/transform/_warps.py
--- a/skimage/transform/_warps.py
+++ b/skimage/transform/_warps.py
@@ -717,20 +717,32 @@ def _clip_warp_output(input_image, output_image, mode, cval, clip):
 
     """
     if clip:
-        min_val = input_image.min()
-        max_val = input_image.max()
+        min_val = np.min(input_image)
+        if np.isnan(min_val):
+            # NaNs detected, use NaN-safe min/max
+            min_func = np.nanmin
+            max_func = np.nanmax
+            min_val = min_func(input_image)
+        else:
+            min_func = np.min
+            max_func = np.max
+        max_val = max_func(input_image)
 
-        preserve_cval = (mode == 'constant' and not
-                         (min_val <= cval <= max_val))
+        # Check if cval has been used such that it expands the effective input
+        # range
+        preserve_cval = (mode == 'constant'
+                         and not min_val <= cval <= max_val
+                         and min_func(output_image) <= cval <= max_func(output_image))
 
+        # expand min/max range to account for cval
         if preserve_cval:
-            cval_mask = output_image == cval
+            # cast cval to the same dtype as the input image
+            cval = input_image.dtype.type(cval)
+            min_val = min(min_val, cval)
+            max_val = max(max_val, cval)
 
         np.clip(output_image, min_val, max_val, out=output_image)
 
-        if preserve_cval:
-            output_image[cval_mask] = cval
-
 
 def warp(image, inverse_map, map_args={}, output_shape=None, order=None,
          mode='constant', cval=0., clip=True, preserve_range=False):


Test patch:
diff --git a/skimage/transform/tests/test_warps.py b/skimage/transform/tests/test_warps.py
--- a/skimage/transform/tests/test_warps.py
+++ b/skimage/transform/tests/test_warps.py
@@ -110,6 +110,79 @@ def test_warp_clip():
     assert_array_almost_equal(outx.max(), 1)
 
 
+@pytest.mark.parametrize('order', [0, 1, 3])
+def test_warp_clip_image_containing_nans(order):
+    # Test that clipping works as intended on an image with NaNs
+    # Orders 2, 4, and 5 do not produce good output when the input image has
+    # NaNs, so those orders are not tested
+
+    x = np.ones((15, 15), dtype=np.float64)
+    x[7, 7] = np.nan
+
+    outx = rotate(x, 45, order=order, cval=2, resize=True, clip=True)
+
+    assert_array_almost_equal(np.nanmin(outx), 1)
+    assert_array_almost_equal(np.nanmax(outx), 2)
+
+
+@pytest.mark.parametrize('order', [0, 1, 3])
+def test_warp_clip_cval_is_nan(order):
+    # Test that clipping works as intended when cval is NaN
+    # Orders 2, 4, and 5 do not produce good output when cval is NaN, so those
+    # orders are not tested
+
+    x = np.ones((15, 15), dtype=np.float64)
+    x[5:-5, 5:-5] = 2
+
+    outx = rotate(x, 45, order=order, cval=np.nan, resize=True, clip=True)
+
+    assert_array_almost_equal(np.nanmin(outx), 1)
+    assert_array_almost_equal(np.nanmax(outx), 2)
+
+
+@pytest.mark.parametrize('order', range(6))
+def test_warp_clip_cval_outside_input_range(order):
+    # Test that clipping behavior considers cval part of the input range
+
+    x = np.ones((15, 15), dtype=np.float64)
+
+    # Specify a cval that is outside the input range to check clipping
+    with expected_warnings(['Bi-quadratic.*bug'] if order == 2 else None):
+        outx = rotate(x, 45, order=order, cval=2, resize=True, clip=True)
+
+    # The corners should be cval for all interpolation orders
+    assert_array_almost_equal([outx[0, 0], outx[0, -1],
+                               outx[-1, 0], outx[-1, -1]], 2)
+
+    # For all interpolation orders other than nearest-neighbor, the clipped
+    # output should have some pixels with values between the input (1) and
+    # cval (2) (i.e., clipping should not set them to 1)
+    if order > 0:
+        assert np.sum(np.less(1, outx) * np.less(outx, 2)) > 0
+
+
+@pytest.mark.parametrize('order', range(6))
+def test_warp_clip_cval_not_used(order):
+    # Test that clipping does not consider cval part of the input range if it
+    # is not used in the output image
+
+    x = np.ones((15, 15), dtype=np.float64)
+    x[5:-5, 5:-5] = 2
+
+    # Transform the image by stretching it out by one pixel on each side so
+    # that cval will not actually be used
+    transform = AffineTransform(scale=15/(15+2), translation=(1, 1))
+    with expected_warnings(['Bi-quadratic.*bug'] if order == 2 else None):
+        outx = warp(x, transform, mode='constant', order=order, cval=0,
+                    clip=True)
+
+    # At higher orders of interpolation, the transformed image has overshoots
+    # beyond the input range that should be clipped to the range 1 to 2.  Even
+    # though cval=0, the minimum value of the clipped output image should be
+    # 1 and not affected by the unused cval.
+    assert_array_almost_equal(outx.min(), 1)
+
+
 def test_homography():
     x = np.zeros((5, 5), dtype=np.float64)
     x[1, 1] = 1