Repository: kornia/kornia

Issue:
Add Guided filter
## 🚀 Feature
<!-- A clear and concise description of the feature proposal -->

Add Guided filter. It is an edge-preserving image smoothing filter, similar to bilateral filter.

https://kaiminghe.github.io/publications/eccv10guidedfilter.pdf

https://en.wikipedia.org/wiki/Guided_filter

OpenCV:

- https://docs.opencv.org/4.x/da/d17/group__ximgproc__filters.html#ga86813d59f8452a9600e979c6287805f5
- https://github.com/opencv/opencv_contrib/blob/4.x/modules/ximgproc/src/guided_filter.cpp

Matlab: https://www.mathworks.com/help/images/ref/imguidedfilter.html

Some PyTorch implementations

- https://github.com/wuhuikai/DeepGuidedFilter/blob/master/GuidedFilteringLayer/GuidedFilter_PyTorch/guided_filter_pytorch/guided_filter.py (grayscale image only)
- https://github.com/perrying/guided-filter-pytorch/blob/master/guided_filter.py (grayscale and RGB)

Guided filter for 1-channel (grayscale) image is simple and straight-forward. However, for multi-channel (RGB), we need to calculate the covariance matrix for the channels (treat each channel as a random variable), and calculate matrix inverse. perrying's implementation hard-codes this calculation.

## Motivation

<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->

## Pitch

<!-- A clear and concise description of what you want to happen. -->

Perhaps we can add guided filter for single channel first? Then we can decide if we want to only support RGB images (then it's possible to hard-code the calculations) or >3-channel images.

## Alternatives

<!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->

______________________________________________________________________

#### Consider also to contribute to Kornia universe projects :)

<sub>

- [**Tutorials**](https://github.com/kornia/tutorials): our repository containing the tutorials.

</sub>



Discussion:
+1 for this
@gau-nernst are you planning to work soon on that ? otherwise i can take that one
I can work on this this week. Have you decided how we should handle multi-channels? Single channel (grayscale) image is straightforward. Multi-channel (e.g. RGB) is complicated since we need to perform matrix inversion (see Equation 14 - https://kaiminghe.github.io/publications/eccv10guidedfilter.pdf)
ok, let's add first for single channel. Playing with vectorisation we cannot make it agnostic to compute the inverse directly with pytorch ?
PyTorch has matrix inverse operation with batch support: https://pytorch.org/docs/stable/generated/torch.linalg.inv.html

Perhaps I can play around with it and let you know

Resolving patch:
diff --git a/docs/generate_example_images.py b/docs/generate_example_images.py
--- a/docs/generate_example_images.py
+++ b/docs/generate_example_images.py
@@ -379,6 +379,7 @@ def main():
         "box_blur": (((5, 5),), 1),
         "median_blur": (((5, 5),), 1),
         "gaussian_blur2d": (((5, 5), (1.5, 1.5)), 1),
+        "guided_blur": (((5, 5), 0.01), 1),
         "motion_blur": ((5, 90.0, 1.0), 1),
         "max_blur_pool2d": ((5,), 1),
         "blur_pool2d": ((5,), 1),
@@ -394,6 +395,8 @@ def main():
         if fn_name == "joint_bilateral_blur":
             guide = K.geometry.resize(img2.repeat(num_samples, 1, 1, 1), img_in.shape[-2:])
             args_in = (img_in, guide, *args)
+        elif fn_name == "guided_blur":
+            args_in = (img_in, img_in, *args)
         else:
             args_in = (img_in, *args)
         # import function and apply
diff --git a/kornia/filters/__init__.py b/kornia/filters/__init__.py
--- a/kornia/filters/__init__.py
+++ b/kornia/filters/__init__.py
@@ -14,6 +14,7 @@
 from .dexined import DexiNed
 from .filter import filter2d, filter2d_separable, filter3d
 from .gaussian import GaussianBlur2d, gaussian_blur2d, gaussian_blur2d_t
+from .guided import GuidedBlur, guided_blur
 from .kernels import (
     gaussian,
     get_binary_kernel2d,
@@ -63,6 +64,7 @@
     "get_sobel_kernel2d",
     "get_diff_kernel2d",
     "gaussian_blur2d",
+    "guided_blur",
     "laplacian",
     "laplacian_1d",
     "unsharp_mask",
@@ -102,4 +104,5 @@
     "joint_bilateral_blur",
     "BilateralBlur",
     "JointBilateralBlur",
+    "GuidedBlur",
 ]
diff --git a/kornia/filters/guided.py b/kornia/filters/guided.py
new file mode 100644
--- /dev/null
+++ b/kornia/filters/guided.py
@@ -0,0 +1,213 @@
+from __future__ import annotations
+
+import torch
+from torch.nn.functional import interpolate
+
+from kornia.core import Module, Tensor
+from kornia.core.check import KORNIA_CHECK, KORNIA_CHECK_IS_TENSOR, KORNIA_CHECK_SHAPE
+
+from .blur import box_blur
+from .kernels import _unpack_2d_ks
+
+
+def _preprocess_fast_guided_blur(
+    guidance: Tensor, input: Tensor, kernel_size: tuple[int, int] | int, subsample: int = 1
+) -> tuple[Tensor, Tensor, tuple[int, int]]:
+    ky, kx = _unpack_2d_ks(kernel_size)
+    if subsample > 1:
+        s = 1 / subsample
+        guidance_sub = interpolate(guidance, scale_factor=s, mode="nearest")
+        input_sub = guidance_sub if input is guidance else interpolate(input, scale_factor=s, mode="nearest")
+        ky, kx = ((k - 1) // subsample + 1 for k in (ky, kx))
+    else:
+        guidance_sub = guidance
+        input_sub = input
+    return guidance_sub, input_sub, (ky, kx)
+
+
+def _guided_blur_grayscale_guidance(
+    guidance: Tensor,
+    input: Tensor,
+    kernel_size: tuple[int, int] | int,
+    eps: float | Tensor,
+    border_type: str = 'reflect',
+    subsample: int = 1,
+) -> Tensor:
+    guidance_sub, input_sub, kernel_size = _preprocess_fast_guided_blur(guidance, input, kernel_size, subsample)
+
+    mean_I = box_blur(guidance_sub, kernel_size, border_type)
+    corr_I = box_blur(guidance_sub.square(), kernel_size, border_type)
+    var_I = corr_I - mean_I.square()
+
+    if input is guidance:
+        mean_p = mean_I
+        cov_Ip = var_I
+
+    else:
+        mean_p = box_blur(input_sub, kernel_size, border_type)
+        corr_Ip = box_blur(guidance_sub * input_sub, kernel_size, border_type)
+        cov_Ip = corr_Ip - mean_I * mean_p
+
+    if isinstance(eps, Tensor):
+        eps = eps.view(-1, 1, 1, 1)  # N -> NCHW
+
+    a = cov_Ip / (var_I + eps)
+    b = mean_p - a * mean_I
+
+    mean_a = box_blur(a, kernel_size, border_type)
+    mean_b = box_blur(b, kernel_size, border_type)
+
+    if subsample > 1:
+        mean_a = interpolate(mean_a, scale_factor=subsample, mode="bilinear")
+        mean_b = interpolate(mean_b, scale_factor=subsample, mode="bilinear")
+
+    return mean_a * guidance + mean_b
+
+
+def _guided_blur_multichannel_guidance(
+    guidance: Tensor,
+    input: Tensor,
+    kernel_size: tuple[int, int] | int,
+    eps: float | Tensor,
+    border_type: str = 'reflect',
+    subsample: int = 1,
+) -> Tensor:
+    guidance_sub, input_sub, kernel_size = _preprocess_fast_guided_blur(guidance, input, kernel_size, subsample)
+    B, C, H, W = guidance_sub.shape
+
+    mean_I = box_blur(guidance_sub, kernel_size, border_type).permute(0, 2, 3, 1)
+    II = (guidance_sub.unsqueeze(1) * guidance_sub.unsqueeze(2)).flatten(1, 2)
+    corr_I = box_blur(II, kernel_size, border_type).permute(0, 2, 3, 1)
+    var_I = corr_I.reshape(B, H, W, C, C) - mean_I.unsqueeze(-2) * mean_I.unsqueeze(-1)
+
+    if guidance is input:
+        mean_p = mean_I
+        cov_Ip = var_I
+
+    else:
+        mean_p = box_blur(input_sub, kernel_size, border_type).permute(0, 2, 3, 1)
+        Ip = (input_sub.unsqueeze(1) * guidance_sub.unsqueeze(2)).flatten(1, 2)
+        corr_Ip = box_blur(Ip, kernel_size, border_type).permute(0, 2, 3, 1)
+        cov_Ip = corr_Ip.reshape(B, H, W, C, -1) - mean_p.unsqueeze(-2) * mean_I.unsqueeze(-1)
+
+    if isinstance(eps, Tensor):
+        _eps = torch.eye(C, device=guidance.device, dtype=guidance.dtype).view(1, 1, 1, C, C) * eps.view(-1, 1, 1, 1, 1)
+    else:
+        _eps = guidance.new_full((C,), eps).diag().view(1, 1, 1, C, C)
+    a = torch.linalg.solve(var_I + _eps, cov_Ip)  # B, H, W, C_guidance, C_input
+    b = mean_p - (mean_I.unsqueeze(-2) @ a).squeeze(-2)  # B, H, W, C_input
+
+    mean_a = box_blur(a.flatten(-2).permute(0, 3, 1, 2), kernel_size, border_type)
+    mean_b = box_blur(b.permute(0, 3, 1, 2), kernel_size, border_type)
+
+    if subsample > 1:
+        mean_a = interpolate(mean_a, scale_factor=subsample, mode="bilinear")
+        mean_b = interpolate(mean_b, scale_factor=subsample, mode="bilinear")
+    mean_a = mean_a.view(B, C, -1, H * subsample, W * subsample)
+
+    # einsum might not be contiguous, thus mean_b is the first argument
+    return mean_b + torch.einsum("BCHW,BCcHW->BcHW", guidance, mean_a)
+
+
+def guided_blur(
+    guidance: Tensor,
+    input: Tensor,
+    kernel_size: tuple[int, int] | int,
+    eps: float | Tensor,
+    border_type: str = 'reflect',
+    subsample: int = 1,
+) -> Tensor:
+    r"""Blur a tensor using a Guided filter.
+
+    .. image:: _static/img/guided_blur.png
+
+    The operator is an edge-preserving image smoothing filter. See :cite:`he2010guided`
+    and :cite:`he2015fast` for details. Guidance and input can have different number of channels.
+
+    Arguments:
+        guidance: the guidance tensor with shape :math:`(B,C,H,W)`.
+        input: the input tensor with shape :math:`(B,C,H,W)`.
+        kernel_size: the size of the kernel.
+        eps: regularization parameter. Smaller values preserve more edges.
+        border_type: the padding mode to be applied before convolving.
+          The expected modes are: ``'constant'``, ``'reflect'``,
+          ``'replicate'`` or ``'circular'``. Default: ``'reflect'``.
+        subsample: subsampling factor for Fast Guided filtering. Default: 1 (no subsampling)
+
+    Returns:
+        the blurred tensor with same shape as `input` :math:`(B, C, H, W)`.
+
+    Examples:
+        >>> guidance = torch.rand(2, 3, 5, 5)
+        >>> input = torch.rand(2, 4, 5, 5)
+        >>> output = guided_blur(guidance, input, 3, 0.1)
+        >>> output.shape
+        torch.Size([2, 4, 5, 5])
+    """
+
+    KORNIA_CHECK_IS_TENSOR(guidance)
+    KORNIA_CHECK_SHAPE(guidance, ['B', 'C', 'H', 'W'])
+    if input is not guidance:
+        KORNIA_CHECK_IS_TENSOR(input)
+        KORNIA_CHECK_SHAPE(input, ['B', 'C', 'H', 'W'])
+        KORNIA_CHECK(
+            (guidance.shape[0] == input.shape[0]) and (guidance.shape[-2:] == input.shape[-2:]),
+            "guidance and input should have the same batch size and spatial dimensions",
+        )
+
+    if guidance.shape[1] == 1:
+        return _guided_blur_grayscale_guidance(guidance, input, kernel_size, eps, border_type, subsample)
+    else:
+        return _guided_blur_multichannel_guidance(guidance, input, kernel_size, eps, border_type, subsample)
+
+
+class GuidedBlur(Module):
+    r"""Blur a tensor using a Guided filter.
+
+    The operator is an edge-preserving image smoothing filter. See :cite:`he2010guided`
+    and :cite:`he2015fast` for details. Guidance and input can have different number of channels.
+
+    Arguments:
+        kernel_size: the size of the kernel.
+        eps: regularization parameter. Smaller values preserve more edges.
+        border_type: the padding mode to be applied before convolving.
+          The expected modes are: ``'constant'``, ``'reflect'``,
+          ``'replicate'`` or ``'circular'``. Default: ``'reflect'``.
+        subsample: subsampling factor for Fast Guided filtering. Default: 1 (no subsampling)
+
+    Returns:
+        the blurred input tensor.
+
+    Shape:
+        - Input: :math:`(B, C, H, W)`, :math:`(B, C, H, W)`
+        - Output: :math:`(B, C, H, W)`
+
+    Examples:
+        >>> guidance = torch.rand(2, 3, 5, 5)
+        >>> input = torch.rand(2, 4, 5, 5)
+        >>> blur = GuidedBlur(3, 0.1)
+        >>> output = blur(guidance, input)
+        >>> output.shape
+        torch.Size([2, 4, 5, 5])
+    """
+
+    def __init__(
+        self, kernel_size: tuple[int, int] | int, eps: float, border_type: str = 'reflect', subsample: int = 1
+    ):
+        super().__init__()
+        self.kernel_size = kernel_size
+        self.eps = eps
+        self.border_type = border_type
+        self.subsample = subsample
+
+    def __repr__(self) -> str:
+        return (
+            f"{self.__class__.__name__}"
+            f"(kernel_size={self.kernel_size}, "
+            f"eps={self.eps}, "
+            f"border_type={self.border_type}, "
+            f"subsample={self.subsample})"
+        )
+
+    def forward(self, guidance: Tensor, input: Tensor) -> Tensor:
+        return guided_blur(guidance, input, self.kernel_size, self.eps, self.border_type, self.subsample)


Test patch:
diff --git a/test/filters/test_guided.py b/test/filters/test_guided.py
new file mode 100644
--- /dev/null
+++ b/test/filters/test_guided.py
@@ -0,0 +1,188 @@
+import pytest
+import torch
+
+from kornia.filters import GuidedBlur, guided_blur
+from kornia.testing import BaseTester, tensor_to_gradcheck_var
+
+
+class TestGuidedBlur(BaseTester):
+    @pytest.mark.parametrize("batch_size", [1, 2])
+    @pytest.mark.parametrize("guide_dim", [1, 3])
+    @pytest.mark.parametrize("input_dim", [1, 3])
+    @pytest.mark.parametrize("kernel_size", [5, (3, 5)])
+    @pytest.mark.parametrize("eps", [0.1, 0.01])
+    def test_smoke(self, batch_size, guide_dim, input_dim, kernel_size, eps, device, dtype):
+        H, W = 8, 16
+        guide = torch.randn(batch_size, guide_dim, H, W, device=device, dtype=dtype)
+        inp = torch.randn(batch_size, input_dim, H, W, device=device, dtype=dtype)
+
+        # tensor eps -> with batch dim
+        eps = torch.rand(batch_size, device=device, dtype=dtype)
+        actual_A = guided_blur(guide, inp, kernel_size, eps)
+        assert isinstance(actual_A, torch.Tensor)
+        assert actual_A.shape == (batch_size, input_dim, H, W)
+
+        # float and tuple sigmas -> same sigmas across batch
+        eps_ = eps[0].item()
+        actual_B = guided_blur(guide, inp, kernel_size, eps_)
+        assert isinstance(actual_B, torch.Tensor)
+        assert actual_B.shape == (batch_size, input_dim, H, W)
+
+        self.assert_close(actual_A[0], actual_B[0])
+
+        # fast guided filter
+        actual_C = guided_blur(guide, inp, kernel_size, eps_, subsample=4)
+        assert isinstance(actual_C, torch.Tensor)
+        assert actual_C.shape == (batch_size, input_dim, H, W)
+
+        # self-guidance
+        actual_D = guided_blur(inp, inp, kernel_size, eps_)
+        assert isinstance(actual_D, torch.Tensor)
+        assert actual_D.shape == (batch_size, input_dim, H, W)
+
+    @pytest.mark.parametrize("shape", [(1, 1, 8, 15), (2, 3, 11, 7)])
+    @pytest.mark.parametrize("kernel_size", [5, (3, 5)])
+    def test_cardinality(self, shape, kernel_size, device, dtype):
+        guide = torch.zeros(shape, device=device, dtype=dtype)
+        inp = torch.zeros(shape, device=device, dtype=dtype)
+        actual = guided_blur(guide, inp, kernel_size, 0.1)
+        assert actual.shape == shape
+
+    def test_exception(self):
+        with pytest.raises(Exception) as errinfo:
+            guided_blur(torch.rand(1, 1, 5, 5), 3, 3, 0.1)
+        assert 'Not a Tensor type. Go' in str(errinfo)
+
+        with pytest.raises(Exception) as errinfo:
+            guided_blur(torch.rand(1, 1, 5, 5), torch.rand(2, 1, 5, 5), 3, 0.1)
+        assert 'same batch size and spatial dimensions' in str(errinfo)
+
+    def test_noncontiguous(self, device, dtype):
+        batch_size = 3
+        guide = torch.rand(3, 5, 5, device=device, dtype=dtype).expand(batch_size, -1, -1, -1)
+        inp = torch.rand(3, 5, 5, device=device, dtype=dtype).expand(batch_size, -1, -1, -1)
+
+        actual = guided_blur(guide, inp, 3, 0.1)
+        assert actual.is_contiguous()
+
+    def test_gradcheck(self, device):
+        guide = torch.rand(1, 2, 5, 4, device=device)
+        img = torch.rand(1, 2, 5, 4, device=device)
+        guide = tensor_to_gradcheck_var(guide)  # to var
+        img = tensor_to_gradcheck_var(img)
+        self.gradcheck(guided_blur, (guide, img, 3, 0.1))
+
+        eps = torch.rand(1, device=device)
+        eps = tensor_to_gradcheck_var(eps)
+        self.gradcheck(guided_blur, (guide, img, 3, eps))
+
+    @pytest.mark.parametrize("shape", [(1, 1, 8, 16), (2, 3, 12, 8)])
+    @pytest.mark.parametrize("kernel_size", [5, (3, 5)])
+    @pytest.mark.parametrize("eps", [0.1, 0.01])
+    @pytest.mark.parametrize("subsample", [1, 2])
+    def test_module(self, shape, kernel_size, eps, subsample, device, dtype):
+        guide = torch.rand(shape, device=device, dtype=dtype)
+        img = torch.rand(shape, device=device, dtype=dtype)
+
+        op = guided_blur
+        op_module = GuidedBlur(kernel_size, eps, subsample=subsample)
+        self.assert_close(op_module(guide, img), op(guide, img, kernel_size, eps, subsample=subsample))
+
+    @pytest.mark.parametrize('kernel_size', [5, (5, 7)])
+    @pytest.mark.parametrize("subsample", [1, 2])
+    def test_dynamo(self, kernel_size, subsample, device, dtype, torch_optimizer):
+        guide = torch.ones(2, 3, 8, 8, device=device, dtype=dtype)
+        inpt = torch.ones(2, 3, 8, 8, device=device, dtype=dtype)
+        op = GuidedBlur(kernel_size, 0.1, subsample=subsample)
+        op_optimized = torch_optimizer(op)
+
+        self.assert_close(op(guide, inpt), op_optimized(guide, inpt))
+
+        op = GuidedBlur(kernel_size, torch.tensor(0.1, device=device, dtype=dtype), subsample=subsample)
+        op_optimized = torch_optimizer(op)
+
+        self.assert_close(op(guide, inpt), op_optimized(guide, inpt))
+
+    def test_opencv_grayscale(self, device, dtype):
+        guide = [[100, 130, 58, 36], [215, 142, 173, 166], [114, 150, 190, 60], [23, 83, 84, 216]]
+        guide = torch.tensor(guide, device=device, dtype=dtype).view(1, 1, 4, 4) / 255
+
+        img = [[95, 130, 108, 228], [98, 142, 187, 166], [114, 166, 190, 141], [150, 83, 174, 216]]
+        img = torch.tensor(img, device=device, dtype=dtype).view(1, 1, 4, 4) / 255
+
+        kernel_size = 3
+        eps = 0.01
+
+        # Expected output generated with OpenCV:
+        # import cv2
+        # expected = cv2.ximgproc.guidedFilter(
+        #   guide.squeeze().numpy(),
+        #   img.squeeze().numpy(),
+        #   (kernel_size - 1) // 2,
+        #   eps,
+        # )
+        expected = [
+            [0.4487294, 0.5163902, 0.5981981, 0.70094436],
+            [0.4850059, 0.53724647, 0.62616897, 0.6686147],
+            [0.5010369, 0.5631456, 0.6808387, 0.5960593],
+            [0.5304646, 0.53203756, 0.57674146, 0.80308396],
+        ]
+        expected = torch.tensor(expected, device=device, dtype=dtype).view(1, 1, 4, 4)
+
+        # OpenCV uses hard-coded BORDER_REFLECT mode, which also reflects the outermost pixels
+        # https://github.com/opencv/opencv_contrib/blob/853144ef93c4ffa55661619b861539090943c5b6/modules/ximgproc/src/guided_filter.cpp#L162
+        # PyTorch's `reflect` border type corresponds to OpenCV's BORDER_REFLECT_101
+        # To match the border's behavior, we use kernel_size = 3 and border_type="replicate" for testing
+        out = guided_blur(guide, img, kernel_size, eps, border_type="replicate")
+        self.assert_close(out, expected)
+
+    def test_opencv_rgb(self, device, dtype):
+        guide = [
+            [[170, 89, 182, 255], [199, 209, 216, 205], [196, 213, 218, 191], [207, 126, 224, 249]],
+            [[61, 104, 274, 225], [65, 112, 14, 148], [78, 247, 176, 120], [124, 69, 155, 211]],
+            [[73, 111, 94, 175], [77, 117, 123, 130], [83, 139, 163, 120], [132, 84, 137, 155]],
+        ]
+        guide = torch.tensor(guide, device=device, dtype=dtype).view(1, 3, 4, 4) / 255
+
+        img = [
+            [[170, 189, 182, 255], [169, 239, 206, 215], [196, 213, 28, 191], [207, 16, 234, 240]],
+            [[61, 144, 74, 225], [20, 112, 176, 148], [34, 147, 116, 120], [124, 61, 155, 211]],
+            [[73, 111, 90, 175], [177, 117, 163, 130], [89, 139, 163, 120], [132, 84, 137, 135]],
+        ]
+        img = torch.tensor(img, device=device, dtype=dtype).view(1, 3, 4, 4) / 255
+
+        kernel_size = 3
+        eps = 0.01
+
+        # Expected output generated with OpenCV:
+        # import cv2
+        # expected = cv2.ximgproc.guidedFilter(
+        #   guide.squeeze().permute(1, 2, 0).numpy(),
+        #   img.squeeze().permute(1, 2, 0).numpy(),
+        #   (kernel_size - 1) // 2,
+        #   eps,
+        # ).transpose(2, 0, 1)
+        expected = [
+            [
+                [0.7039907, 0.7277061, 0.7474556, 0.904094],
+                [0.7095674, 0.76176095, 0.77444744, 0.7774203],
+                [0.67807436, 0.7721572, 0.70001286, 0.7042719],
+                [0.73099065, 0.28477466, 0.7464762, 0.8454268],
+            ],
+            [
+                [0.25627214, 0.4922768, 0.3593133, 0.76788116],
+                [0.21797341, 0.42890117, 0.56577384, 0.58102953],
+                [0.25184435, 0.5643642, 0.59704626, 0.5153022],
+                [0.42154774, 0.24721909, 0.56817913, 0.7258603],
+            ],
+            [
+                [0.431774, 0.40672457, 0.39094293, 0.63833976],
+                [0.47457936, 0.51558167, 0.58189815, 0.5340911],
+                [0.45442006, 0.5345709, 0.5615816, 0.5071402],
+                [0.49547666, 0.37159446, 0.5301453, 0.55153173],
+            ],
+        ]
+        expected = torch.tensor(expected, device=device, dtype=dtype).view(1, 3, 4, 4)
+
+        out = guided_blur(guide, img, kernel_size, eps, border_type="replicate")
+        self.assert_close(out, expected)