Is there anyway to do gaussian filtering for an image(2D,3D) in pytorch?

isalirezag: <p>I am wondering if pytorch has gaussian filtering (convolution).<br>
For example, if I want to do low pass Gaussian filter on an image, is it possible?<br>
In other words, Im trying to do this:<br>
scipy.ndimage.gaussian_filter(a,b)<br>
<a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.gaussian_filter.html" class="onebox" target="_blank" rel="nofollow noopener">https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.gaussian_filter.html</a></p>

ptrblck: <p>You can create a <code>Conv2d</code> layer and specify the weights to be gaussian.<br>
Then just apply the conv layer on your image.</p>

tetratrio: <p>For anyone who has a problem implementing this here is a solution entirely written in pytorch:</p>
<pre><code class="lang-auto"># Set these to whatever you want for your gaussian filter
kernel_size = 15
sigma = 3

# Create a x, y coordinate grid of shape (kernel_size, kernel_size, 2)
x_cord = torch.arange(kernel_size)
x_grid = x_cord.repeat(kernel_size).view(kernel_size, kernel_size)
y_grid = x_grid.t()
xy_grid = torch.stack([x_grid, y_grid], dim=-1)

mean = (kernel_size - 1)/2.
variance = sigma**2.

# Calculate the 2-dimensional gaussian kernel which is
# the product of two gaussian distributions for two different
# variables (in this case called x and y)
gaussian_kernel = (1./(2.*math.pi*variance)) *\
                  torch.exp(
                      -torch.sum((xy_grid - mean)**2., dim=-1) /\
                      (2*variance)
                  )
# Make sure sum of values in gaussian kernel equals 1.
gaussian_kernel = gaussian_kernel / torch.sum(gaussian_kernel)

# Reshape to 2d depthwise convolutional weight
gaussian_kernel = gaussian_kernel.view(1, 1, kernel_size, kernel_size)
gaussian_kernel = gaussian_kernel.repeat(channels, 1, 1, 1)

gaussian_filter = nn.Conv2d(in_channels=channels, out_channels=channels,
                            kernel_size=kernel_size, groups=channels, bias=False)

gaussian_filter.weight.data = gaussian_kernel
gaussian_filter.weight.requires_grad = False
</code></pre>
<p>You probably want to keep the size of your input the same after it has been filtered. In that case just add some padding to the input (I use reflection padding).</p>

isalirezag: <p><a class="mention" href="/u/tetratrio">@tetratrio</a> I assume you have a class when you define it, right? otherwise the self.* does not make sense</p>

tetratrio: <p>Oh yea, I forgot to remove that. Thought it would make the code more readable without the class</p>

isalirezag: <p>Yeah!<br>
Good stuff!<br>
Thanks</p>

James_Burton: <p>This doesn’t work with the latest version of PyTorch (1.0), as the LongTensor objects round values when multiplying by floats and end up with a 0 sum, which blows up the torch.exp (causing Python kernel to restart in Jupyter).</p>
<p>The fix however is easy … ensure xy_grid is a FloatTensor, by changing adding .float() to that one line like this:</p>
<pre><code class="lang-auto">#...
xy_grid = torch.stack([x_grid, y_grid], dim=-1).float()
#...
</code></pre>
<p>Here is the full function version for PyTorch 1.0:-</p>
<pre><code class="lang-auto"># From https://discuss.pytorch.org/t/is-there-anyway-to-do-gaussian-filtering-for-an-image-2d-3d-in-pytorch/12351/3
def get_gaussian_kernel(kernel_size=3, sigma=2, channels=3):
    # Create a x, y coordinate grid of shape (kernel_size, kernel_size, 2)
    x_coord = torch.arange(kernel_size)
    x_grid = x_coord.repeat(kernel_size).view(kernel_size, kernel_size)
    y_grid = x_grid.t()
    xy_grid = torch.stack([x_grid, y_grid], dim=-1).float()

    mean = (kernel_size - 1)/2.
    variance = sigma**2.

    # Calculate the 2-dimensional gaussian kernel which is
    # the product of two gaussian distributions for two different
    # variables (in this case called x and y)
    gaussian_kernel = (1./(2.*math.pi*variance)) *\
                      torch.exp(
                          -torch.sum((xy_grid - mean)**2., dim=-1) /\
                          (2*variance)
                      )

    # Make sure sum of values in gaussian kernel equals 1.
    gaussian_kernel = gaussian_kernel / torch.sum(gaussian_kernel)

    # Reshape to 2d depthwise convolutional weight
    gaussian_kernel = gaussian_kernel.view(1, 1, kernel_size, kernel_size)
    gaussian_kernel = gaussian_kernel.repeat(channels, 1, 1, 1)

    gaussian_filter = nn.Conv2d(in_channels=channels, out_channels=channels,
                                kernel_size=kernel_size, groups=channels, bias=False)

    gaussian_filter.weight.data = gaussian_kernel
    gaussian_filter.weight.requires_grad = False
    
    return gaussian_filter
</code></pre>

tetratrio: <p>Cheers!</p>
<p>Since my last post there has also been the addition of <code>torch.meshgrid</code>. This is what I currently use (it does not contain parameters and works for 1d, 2d and 3d data):</p>
<pre><code class="lang-auto">import math
import numbers
import torch
from torch import nn
from torch.nn import functional as F

class GaussianSmoothing(nn.Module):
    """
    Apply gaussian smoothing on a
    1d, 2d or 3d tensor. Filtering is performed seperately for each channel
    in the input using a depthwise convolution.
    Arguments:
        channels (int, sequence): Number of channels of the input tensors. Output will
            have this number of channels as well.
        kernel_size (int, sequence): Size of the gaussian kernel.
        sigma (float, sequence): Standard deviation of the gaussian kernel.
        dim (int, optional): The number of dimensions of the data.
            Default value is 2 (spatial).
    """
    def __init__(self, channels, kernel_size, sigma, dim=2):
        super(GaussianSmoothing, self).__init__()
        if isinstance(kernel_size, numbers.Number):
            kernel_size = [kernel_size] * dim
        if isinstance(sigma, numbers.Number):
            sigma = [sigma] * dim

        # The gaussian kernel is the product of the
        # gaussian function of each dimension.
        kernel = 1
        meshgrids = torch.meshgrid(
            [
                torch.arange(size, dtype=torch.float32)
                for size in kernel_size
            ]
        )
        for size, std, mgrid in zip(kernel_size, sigma, meshgrids):
            mean = (size - 1) / 2
            kernel *= 1 / (std * math.sqrt(2 * math.pi)) * \
                      torch.exp(-((mgrid - mean) / (2 * std)) ** 2)

        # Make sure sum of values in gaussian kernel equals 1.
        kernel = kernel / torch.sum(kernel)

        # Reshape to depthwise convolutional weight
        kernel = kernel.view(1, 1, *kernel.size())
        kernel = kernel.repeat(channels, *[1] * (kernel.dim() - 1))

        self.register_buffer('weight', kernel)
        self.groups = channels

        if dim == 1:
            self.conv = F.conv1d
        elif dim == 2:
            self.conv = F.conv2d
        elif dim == 3:
            self.conv = F.conv3d
        else:
            raise RuntimeError(
                'Only 1, 2 and 3 dimensions are supported. Received {}.'.format(dim)
            )

    def forward(self, input):
        """
        Apply gaussian filter to input.
        Arguments:
            input (torch.Tensor): Input to apply gaussian filter on.
        Returns:
            filtered (torch.Tensor): Filtered output.
        """
        return self.conv(input, weight=self.weight, groups=self.groups)

</code></pre>
<p>Again, I suggest adding reflection padding.</p>
<pre><code class="lang-auto">smoothing = GaussianSmoothing(3, 5, 1)
input = torch.rand(1, 3, 100, 100)
input = F.pad(input, (2, 2, 2, 2), mode='reflect')
output = smoothing(input)
</code></pre>

ntipakos: <p>Thank you for the helpful implementation, but I think line:</p>
<pre><code class="lang-auto">kernel *= 1 / (std * math.sqrt(2 * math.pi)) * \
          torch.exp(-((mgrid - mean) / (2 * std)) ** 2)
</code></pre>
<p>should change to</p>
<pre><code class="lang-auto">kernel *= 1 / (std * math.sqrt(2 * math.pi)) * \
          torch.exp((-((mgrid - mean) / std) ** 2) / 2)
</code></pre>
<p>since:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/af30d25a340bee164b6fa8b07927c7cbbe49c7b5.png" alt="pytorch_gaussian" data-base62-sha1="oZO9KItNHUEYLvFiCjPzEzYDyPH" width="442" height="221"></p>

tetratrio: <p>Hey, nice spot there, you’re totally right. I can’t edit my previous post, hope people dont just copy the code without looking at your answer first. Ill post an altered copy here as well</p>
<pre><code class="lang-auto">import math
import numbers
import torch
from torch import nn
from torch.nn import functional as F

class GaussianSmoothing(nn.Module):
    """
    Apply gaussian smoothing on a
    1d, 2d or 3d tensor. Filtering is performed seperately for each channel
    in the input using a depthwise convolution.
    Arguments:
        channels (int, sequence): Number of channels of the input tensors. Output will
            have this number of channels as well.
        kernel_size (int, sequence): Size of the gaussian kernel.
        sigma (float, sequence): Standard deviation of the gaussian kernel.
        dim (int, optional): The number of dimensions of the data.
            Default value is 2 (spatial).
    """
    def __init__(self, channels, kernel_size, sigma, dim=2):
        super(GaussianSmoothing, self).__init__()
        if isinstance(kernel_size, numbers.Number):
            kernel_size = [kernel_size] * dim
        if isinstance(sigma, numbers.Number):
            sigma = [sigma] * dim

        # The gaussian kernel is the product of the
        # gaussian function of each dimension.
        kernel = 1
        meshgrids = torch.meshgrid(
            [
                torch.arange(size, dtype=torch.float32)
                for size in kernel_size
            ]
        )
        for size, std, mgrid in zip(kernel_size, sigma, meshgrids):
            mean = (size - 1) / 2
            kernel *= 1 / (std * math.sqrt(2 * math.pi)) * \
                      torch.exp(-((mgrid - mean) / std) ** 2 / 2)

        # Make sure sum of values in gaussian kernel equals 1.
        kernel = kernel / torch.sum(kernel)

        # Reshape to depthwise convolutional weight
        kernel = kernel.view(1, 1, *kernel.size())
        kernel = kernel.repeat(channels, *[1] * (kernel.dim() - 1))

        self.register_buffer('weight', kernel)
        self.groups = channels

        if dim == 1:
            self.conv = F.conv1d
        elif dim == 2:
            self.conv = F.conv2d
        elif dim == 3:
            self.conv = F.conv3d
        else:
            raise RuntimeError(
                'Only 1, 2 and 3 dimensions are supported. Received {}.'.format(dim)
            )

    def forward(self, input):
        """
        Apply gaussian filter to input.
        Arguments:
            input (torch.Tensor): Input to apply gaussian filter on.
        Returns:
            filtered (torch.Tensor): Filtered output.
        """
        return self.conv(input, weight=self.weight, groups=self.groups)


smoothing = GaussianSmoothing(3, 5, 1)
input = torch.rand(1, 3, 100, 100)
input = F.pad(input, (2, 2, 2, 2), mode='reflect')
output = smoothing(input)
</code></pre>

ntp: <p>Another solution would be to directly use PIL’s <a href="https://pillow.readthedocs.io/en/5.1.x/reference/ImageFilter.html#PIL.ImageFilter.GaussianBlur" rel="nofollow noopener"><code>ImageFilter.GaussianBlur</code></a> transform.</p>
<p>You can then create the transform:</p>
<pre><code class="lang-auto">import numbers
import numpy as np
from PIL import ImageFilter

class GaussianSmoothing(object):
    def __init__(self, radius):
        if isinstance(radius, numbers.Number):
            self.min_radius = radius
            self.max_radius = radius
        elif isinstance(radius, list):
            if len(radius) != 2:
                raise Exception(
                    "`radius` should be a number or a list of two numbers")
            if radius[1] &lt; radius[0]:
                raise Exception(
                    "radius[0] should be &lt;= radius[1]")
            self.min_radius = radius[0]
            self.max_radius = radius[1]
        else:
            raise Exception(
                "`radius` should be a number or a list of two numbers")

    def __call__(self, image):
        radius = np.random.uniform(self.min_radius, self.max_radius)
        return image.filter(ImageFilter.GaussianBlur(radius))
</code></pre>
<p>Then you can simply add it to your composed transform before <code>ToTensor()</code>:</p>
<pre><code class="lang-auto">from PIL import Image
import torchvision.transforms as T

# just add it before ToTensor(), which is ommited here 
transform = T.Compose([
    GaussianSmoothing([0, 5])
]) 

TEST_IMG = "path/to/image"
img = Image.open(TEST_IMG)
img_t = transform(img)

img_t.show()
</code></pre>

Frida: <p>thanks a lot for the code, if you are working with that class through the training - do you need to backprop through it? I mean, I don’t want to change the coefficients just to correctly update other layers before. thanks in advance</p>

tetratrio: <p>No problem, glad you found it useful!</p>
<p>The actual weights are registered as a buffer</p>
<pre><code class="lang-auto">self.register_buffer('weight', kernel)
</code></pre>
<p>Since they are not parameters (<code>torch.nn.Parameter</code>) they are not trained and will remain static when you use the layer.</p>
<pre><code class="lang-python">class MyModel(torch.nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.layer = torch.nn.Conv2d(5, 5, kernel_size=3, padding=1)
        self.smoothing = GaussianSmoothing(5, kernel_size=3, sigma=1)

    def forward(self, x):
        x = self.layer(x)
        x = torch.nn.functional.pad(x, (1, 1, 1, 1), mode='reflect')
        x = self.smoothing(x)
        return x

my_model = MyModel()
opt = torch.nn.optim.SGD(my_model.parameters(), lr=1e-3)
output = my_model(torch.randn(1, 5, 10, 10))
loss = output.mean()
opt.zero_grad()
loss.backward()
opt.step()
</code></pre>
<p>The smoothing layer will remain static during training</p>

Frida: <p>ok. thanks a lot!<br>
just a general thought: If the smoothing operator is the one of the layers in the network - in order to perform backprop - do you need to calculate the derivatives of that operator</p>

edgarriba: <p>you can also do that using <a href="https://kornia.readthedocs.io/en/latest/filters.html#kornia.filters.gaussian_blur2d" rel="nofollow noopener"><code>kornia.gaussian_blur2d</code></a></p>

ssalome: <p>Hi<br>
Can you explain the working of the code in brief?<br>
Is there any way to adjust the level of denoising in the code?</p>

ntp: <p>Hi Salome,</p>
<p>the level of blurring is given by the radius. The higher it is, the blurrier the result is. For a radius of 0, there is no blurring applied. Most of the code deals with the input, which can be either a number (the radius) or a list of two elements, in which case the radius is chosen randomly from this interval.</p>

ssalome: <p>Okay.  <img src="https://discuss.pytorch.org/images/emoji/apple/+1/4.png?v=9" title=":+1:t4:" class="emoji" alt=":+1:t4:"> Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

Jeffrey_Danowitz: <p><a class="mention" href="/u/tetratrio">@tetratrio</a><br>
Thanks for the post - I used this and it really helped me.<br>
Just 1 comment:</p>
<pre><code class="lang-auto">for size, std, mgrid in zip(kernel_size, sigma, meshgrids):
            mean = (size - 1) / 2
            kernel *= 1 / (std * math.sqrt(2 * math.pi)) * \
                      torch.exp(-((mgrid - mean) / std) ** 2 / 2)

        # Make sure sum of values in gaussian kernel equals 1.
        kernel = kernel / torch.sum(kernel)
</code></pre>
<p>Since you do: kernel = kernel / torch.sum(kernel) then there is no reason to divide by:<br>
std * math.sqrt(2 * math.pi)<br>
The moment you normalize the sum to be 1 divisions by a constant (depending or not on the std) will not effect the final result.<br>
Great work - your code taught me alot about how to use conv (1, 2, 3) in pytorch. It was really cryptic before that.</p>

tetratrio: <p>You are correct! It is an unnecessary extra calculation step but it also may reduce confusion about calculating the kernel as it now is a pure normal density function.</p>
<p>But I have to admit I did not think about that when writing the code so its a nice catch!</p>
<p>Also glad you found it useful. I like the idea of implementing convolutional layers and models to be generic in regards to the number of dimensions in the data!</p>