Repository: kornia/kornia

Issue:
Background color selection in warp_perspective() function
## 🚀 Feature

Possibility to select the color (RGB) for the background pixels in the `warp_perspective()` function (pixels whose color des not come from the original warped image).

## Motivation

OpenCV's [`warpPerspective()`](https://docs.opencv.org/master/da/d54/group__imgproc__transform.html#gaf73673a7e8e18ec6963e3774e6a94b87) function has a `borderValue` parameter allowing to select the color of the background pixels when the `borderMode` is set to `BORDER_CONSTANT`. This is very useful when the desired background color is not black (default). Currently Kornia's [`warp_perspective()`](https://kornia.readthedocs.io/en/latest/geometry.transform.html) only supports black (`zeros` option for `border_mode`).

## Pitch

I'd like to have a parameter similar to OpenCV's `borderValue` to select the color of the background when the `border_mode` of the `warp_perspective` function in Kornia is set to `zeros` (and in that case, probably this name should change from `zeros` to something like `constant`, being black the default color).



Discussion:
hi @arubior thanksl for reporting this. In fact, we should do support what you suggest through `border_mode` since it's internally mapped to `padding_mode` for `F.grid_sample`. Could you verify that ?

See also that's there's a small change in this coming PR: 
https://github.com/kornia/kornia/pull/896/files#diff-393441a973b582a54d0dc25e96233c9d5bf2e8d07dc21597cd2de3c96f386484R106
Hi @edgarriba thanks for pointing that out, now I see that it's internally mapped to `padding_mode` of `F.grid_sample`, but according to [their documentation](https://pytorch.org/docs/stable/nn.functional.html), they only support `'zeros' | 'border' | 'reflection'`, am I right?
true - there's a missing feature from pytorch side here in `grid_sample`. I think it's worth opening a feature request there. In meantime, I believe we could hack this around with masks and blending.
Cool, I was thinking on handling it on my side with masks, but if you include it as a feature in kornia it'd be great.
It's not a priority right now - but I'll mark it as future future.
Being able to specify a padding mode with the same semantics as "zeros" but instead allowing a custom RGB triple would be incredibly useful for me. If someone wants to point me to where this would live in the code base I'd be happy to take a crack at it.
@dribnet I believe this should happen right after this call
https://github.com/kornia/kornia/blob/master/kornia/geometry/transform/imgwarp.py#L117

Ideally this feature should be coming from pytorch `grid_sample` /cc @fmassa

In the meantime as @arubior suggested we could implement somehow with masks. What API do you find more suitable for this feature - If ```mode == "constant"``` we require to provide a float or tuple[float,float,float] or tensor (in case we want to specify a different background color per image in the batch) ? /cc @ducha-aiki 
😢
<img width="656" alt="Screen Shot 2021-10-23 at 7 34 23 PM" src="https://user-images.githubusercontent.com/945979/138545577-97bc37c4-eab3-41f7-a267-315509ebcc89.png">


> cry <img alt="Screen Shot 2021-10-23 at 7 34 23 PM" width="656" src="https://user-images.githubusercontent.com/945979/138545577-97bc37c4-eab3-41f7-a267-315509ebcc89.png">

Our resources/bandwidth is quite limited right now - We leave this open to the community to take it over.
@dribnet
We (Edgar, Luis, me) work at other jobs, e.g. I am PhD student at CTU in Prague and implement those functions first, which are required for our other (read: main) jobs. This way we can work on kornia at our working time. Regarding the rest - it depends on the our (very limited) free resources. 
We also pay for the website and CI from our own pockets - the donations are not big enough to cover that. 

If you would like for some function to be implemented, the best way is to do a PR with it :)

Hi @ducha-aiki - and sorry - I was just being cheeky and letting you know I still had my eye on this one! I think kornia is amazing thanks to the active dev team behind it and I will try to get you a PR and with any luck we'll see this in the next release instead. 
the @pixray project would like to discuss a donation to help cover website and CI costs for korina. please [DM me (@dribnet) on discord pixray channel](https://discord.gg/x2g9TWrNKe) to arrange details.

Resolving patch:
diff --git a/kornia/geometry/transform/imgwarp.py b/kornia/geometry/transform/imgwarp.py
--- a/kornia/geometry/transform/imgwarp.py
+++ b/kornia/geometry/transform/imgwarp.py
@@ -37,6 +37,7 @@ def warp_perspective(
     mode: str = 'bilinear',
     padding_mode: str = 'zeros',
     align_corners: bool = True,
+    fill_value: torch.Tensor = torch.zeros(3),  # needed for jit
 ) -> torch.Tensor:
     r"""Apply a perspective transformation to an image.
 
@@ -56,8 +57,9 @@ def warp_perspective(
         M: transformation matrix with shape :math:`(B, 3, 3)`.
         dsize: size of the output image (height, width).
         mode: interpolation mode to calculate output values ``'bilinear'`` | ``'nearest'``.
-        padding_mode: padding mode for outside grid values ``'zeros'`` | ``'border'`` | ``'reflection'``.
-        align_corners(bool, optional): interpolation flag.
+        padding_mode: padding mode for outside grid values ``'zeros'`` | ``'border'`` | ``'reflection'`` | ``'fill'``.
+        align_corners: interpolation flag.
+        fill_value: tensor of shape :math:`(3)` that fills the padding area. Only supported for RGB.
 
     Returns:
         the warped input image :math:`(B, C, H, W)`.
@@ -88,6 +90,11 @@ def warp_perspective(
     if not (len(M.shape) == 3 and M.shape[-2:] == (3, 3)):
         raise ValueError(f"Input M must be a Bx3x3 tensor. Got {M.shape}")
 
+    # fill padding is only supported for 3 channels because we can't set fill_value default
+    # to None as this gives jit issues.
+    if padding_mode == "fill" and fill_value.shape != torch.Size([3]):
+        raise ValueError(f"Padding_tensor only supported for 3 channels. Got {fill_value.shape}")
+
     B, _, H, W = src.size()
     h_out, w_out = dsize
 
@@ -102,6 +109,8 @@ def warp_perspective(
     )
     grid = transform_points(src_norm_trans_dst_norm[:, None, None], grid)
 
+    if padding_mode == "fill":
+        return _fill_and_warp(src, grid, align_corners=align_corners, mode=mode, fill_value=fill_value)
     return F.grid_sample(src, grid, align_corners=align_corners, mode=mode, padding_mode=padding_mode)
 
 
@@ -112,6 +121,7 @@ def warp_affine(
     mode: str = 'bilinear',
     padding_mode: str = 'zeros',
     align_corners: bool = True,
+    fill_value: torch.Tensor = torch.zeros(3),  # needed for jit
 ) -> torch.Tensor:
     r"""Apply an affine transformation to a tensor.
 
@@ -129,8 +139,9 @@ def warp_affine(
         M: affine transformation of shape :math:`(B, 2, 3)`.
         dsize: size of the output image (height, width).
         mode: interpolation mode to calculate output values ``'bilinear'`` | ``'nearest'``.
-        padding_mode (str): padding mode for outside grid values ``'zeros'`` | ``'border'`` | ``'reflection'``.
+        padding_mode: padding mode for outside grid values ``'zeros'`` | ``'border'`` | ``'reflection'`` | ``'fill'``.
         align_corners : mode for grid_generation.
+        fill_value: tensor of shape :math:`(3)` that fills the padding area. Only supported for RGB.
 
     Returns:
         the warped tensor with shape :math:`(B, C, H, W)`.
@@ -162,6 +173,11 @@ def warp_affine(
     if not (len(M.shape) == 3 or M.shape[-2:] == (2, 3)):
         raise ValueError(f"Input M must be a Bx2x3 tensor. Got {M.shape}")
 
+    # fill padding is only supported for 3 channels because we can't set fill_value default
+    # to None as this gives jit issues.
+    if padding_mode == "fill" and fill_value.shape != torch.Size([3]):
+        raise ValueError(f"Padding_tensor only supported for 3 channels. Got {fill_value.shape}")
+
     B, C, H, W = src.size()
 
     # we generate a 3x3 transformation matrix from 2x3 affine
@@ -173,9 +189,37 @@ def warp_affine(
 
     grid = F.affine_grid(src_norm_trans_dst_norm[:, :2, :], [B, C, dsize[0], dsize[1]], align_corners=align_corners)
 
+    if padding_mode == "fill":
+        return _fill_and_warp(src, grid, align_corners=align_corners, mode=mode, fill_value=fill_value)
     return F.grid_sample(src, grid, align_corners=align_corners, mode=mode, padding_mode=padding_mode)
 
 
+def _fill_and_warp(
+    src: torch.Tensor,
+    grid: torch.Tensor,
+    mode: str,
+    align_corners: bool,
+    fill_value: torch.Tensor,
+) -> torch.Tensor:
+    r"""Warp a mask of ones, then multiple with fill_value and add to default warp.
+
+    Args:
+        src: input tensor of shape :math:`(B, 3, H, W)`.
+        grid: grid tensor from `transform_points`.
+        mode: interpolation mode to calculate output values ``'bilinear'`` | ``'nearest'``.
+        align_corners: interpolation flag.
+        fill_value: tensor of shape :math:`(3)` that fills the padding area. Only supported for RGB.
+
+    Returns:
+        the warped and filled tensor with shape :math:`(B, 3, H, W)`.
+    """
+    ones_mask = torch.ones_like(src)
+    fill_value = fill_value.to(ones_mask)[None, :, None, None]  # cast and add dimensions for broadcasting
+    inv_ones_mask = 1 - F.grid_sample(ones_mask, grid, align_corners=align_corners, mode=mode, padding_mode="zeros")
+    inv_color_mask = inv_ones_mask * fill_value
+    return F.grid_sample(src, grid, align_corners=align_corners, mode=mode, padding_mode="zeros") + inv_color_mask
+
+
 def get_perspective_transform(src, dst):
     r"""Calculate a perspective transform from four pairs of the corresponding
     points.


Test patch:
diff --git a/test/geometry/transform/test_imgwarp.py b/test/geometry/transform/test_imgwarp.py
--- a/test/geometry/transform/test_imgwarp.py
+++ b/test/geometry/transform/test_imgwarp.py
@@ -171,6 +171,23 @@ def test_gradcheck(self, device, dtype):
         img_b = utils.tensor_to_gradcheck_var(img_b)  # to var
         assert gradcheck(kornia.geometry.warp_affine, (img_b, aff_ab, (height, width)), raise_exception=True)
 
+    def test_fill_padding_translation(self, device, dtype):
+        offset = 1.0
+        h, w = 3, 4
+        aff_ab = torch.eye(2, 3, device=device, dtype=dtype)[None]
+        aff_ab[..., -1] += offset
+
+        img_b = torch.arange(float(3 * h * w), device=device, dtype=dtype).view(1, 3, h, w)
+
+        # normally fill_value will also be converted to the right device and type in warp_affine
+        fill_value = torch.tensor([0.5, 0.2, 0.1], device=device, dtype=dtype)
+
+        img_a = kornia.geometry.warp_affine(img_b, aff_ab, (h, w), padding_mode="fill", fill_value=fill_value)
+        top_row_mean = img_a[..., :1, :].mean(dim=[0, 2, 3])
+        first_col_mean = img_a[..., :1].mean(dim=[0, 2, 3])
+        assert_close(top_row_mean, fill_value)
+        assert_close(first_col_mean, fill_value)
+
 
 class TestWarpPerspective:
     def test_smoke(self, device, dtype):
@@ -359,6 +376,23 @@ def test_gradcheck(self, device, dtype):
         H_ab = utils.tensor_to_gradcheck_var(H_ab, requires_grad=False)  # to var
         assert gradcheck(kornia.geometry.warp_perspective, (img_b, H_ab, (height, width)), raise_exception=True)
 
+    def test_fill_padding_translation(self, device, dtype):
+        offset = 1.0
+        h, w = 3, 4
+
+        img_b = torch.arange(float(3 * h * w), device=device, dtype=dtype).view(1, 3, h, w)
+        homo_ab = kornia.eye_like(3, img_b)
+        homo_ab[..., :2, -1] += offset
+
+        # normally fill_value will also be converted to the right device and type in warp_perspective
+        fill_value = torch.tensor([0.5, 0.2, 0.1], device=device, dtype=dtype)
+
+        img_a = kornia.geometry.warp_perspective(img_b, homo_ab, (h, w), padding_mode="fill", fill_value=fill_value)
+        top_row_mean = img_a[..., :1, :].mean(dim=[0, 2, 3])
+        first_col_mean = img_a[..., :1].mean(dim=[0, 2, 3])
+        assert_close(top_row_mean, fill_value)
+        assert_close(first_col_mean, fill_value)
+
 
 class TestRemap:
     def test_smoke(self, device, dtype):