Where is the noise layer in pytorch?

cold_wind: <p>If I want to add some zero-centered Gaussian noise,it only active in training process. Dose pytorch has this function? Keras has it(<a href="https://keras.io/layers/noise/" rel="nofollow noopener">noise layer in Keras</a>)</p>

smth: <p>torch.randn generates noise</p>

Ben: <p>Here is my code. Hope it helps.<br>
The known issue is it slower my training process about 25%.</p>
<pre><code class="lang-auto">class GaussianNoise(nn.Module):
    def __init__(self, stddev):
        super().__init__()
        self.stddev = stddev

    def forward(self, din):
        if self.training:
            return din + torch.autograd.Variable(torch.randn(din.size()).cuda() * self.stddev)
        return din
</code></pre>

YannDubs1: <p>Here is what I normally use: the major difference being that I do not pass the noise to GPU at every call which should speed things up <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">class GaussianNoise(nn.Module):
    """Gaussian noise regularizer.

    Args:
        sigma (float, optional): relative standard deviation used to generate the
            noise. Relative means that it will be multiplied by the magnitude of
            the value your are adding the noise to. This means that sigma can be
            the same regardless of the scale of the vector.
        is_relative_detach (bool, optional): whether to detach the variable before
            computing the scale of the noise. If `False` then the scale of the noise
            won't be seen as a constant but something to optimize: this will bias the
            network to generate vectors with smaller values.
    """

    def __init__(self, sigma=0.1, is_relative_detach=True):
        super().__init__()
        self.sigma = sigma
        self.is_relative_detach = is_relative_detach
        self.noise = torch.tensor(0).to(device)

    def forward(self, x):
        if self.training and self.sigma != 0:
            scale = self.sigma * x.detach() if self.is_relative_detach else self.sigma * x
            sampled_noise = self.noise.repeat(*x.size()).normal_() * scale
            x = x + sampled_noise
        return x 
</code></pre>

Kenzo: <p>Hey <a class="mention" href="/u/yanndubs1">@YannDubs1</a>,</p>
<p>Thanks for sharing the code, I am curious to know how can I use your code as a part of my implementation.<br>
My model has 2 part: Encoder and Decoder and I want to add small noise (N(mean = 0, std = 0.1)) to the output of the Encoder but I don’t know how to do that.</p>

YannDubs1: <p>Something along those lines:</p>
<pre><code class="lang-auto">class Encoder(nn.Module)

    def __init__(self, ....):
        ....
        self.noise = GaussianNoise()

    def forward(self, x):
        ....
        output = self.noise(output)
        return output
</code></pre>

Kenzo: <p>Thanks for your help but still I am confused about how to add small noise to my network.</p>
<p>The code below is my Generator model which I would like to add small noise to the output of encoder OR input of decoder part of my model. In the code ‘down’ is Encoder part and ‘up’ is Decoder part</p>
<pre><code class="lang-auto">class UnetGenerator(nn.Module):

    def __init__(self, input_nc, output_nc, num_downs, ngf=64, norm_layer=nn.BatchNorm2d, use_dropout=False):

        super(UnetGenerator, self).__init__()
        unet_block = UnetSkipConnectionBlock(ngf * 8, ngf * 8, input_nc=None, submodule=None, norm_layer=norm_layer, innermost=True)

        for i in range(num_downs - 5):
            unet_block = UnetSkipConnectionBlock(ngf * 8, ngf * 8, input_nc=None, submodule=unet_block, norm_layer=norm_layer, use_dropout=use_dropout)

        unet_block = UnetSkipConnectionBlock(ngf * 4, ngf * 8, input_nc=None, submodule=unet_block, norm_layer=norm_layer)
        unet_block = UnetSkipConnectionBlock(ngf * 2, ngf * 4, input_nc=None, submodule=unet_block, norm_layer=norm_layer)
        unet_block = UnetSkipConnectionBlock(ngf, ngf * 2, input_nc=None, submodule=unet_block, norm_layer=norm_layer)
        self.model = UnetSkipConnectionBlock(output_nc, ngf, input_nc=input_nc, submodule=unet_block, outermost=True, norm_layer=norm_layer)

    def forward(self, input):

        return self.model(input)


class UnetSkipConnectionBlock(nn.Module):

    def __init__(self, outer_nc, inner_nc, input_nc=None,
                 submodule=None, outermost=False, innermost=False, norm_layer=nn.BatchNorm2d, use_dropout=False):

        super(UnetSkipConnectionBlock, self).__init__()

        self.outermost = outermost

        if type(norm_layer) == functools.partial:
            use_bias = norm_layer.func == nn.InstanceNorm2d
        else:
            use_bias = norm_layer == nn.InstanceNorm2d

        if input_nc is None:
            input_nc = outer_nc

        downconv = nn.Conv2d(input_nc, inner_nc, kernel_size=4,
                             stride=2, padding=1, bias=use_bias)
        downrelu = nn.LeakyReLU(0.2, True)
        downnorm = norm_layer(inner_nc)
        uprelu = nn.ReLU(True)
        upnorm = norm_layer(outer_nc)


        if outermost:
            upconv = nn.ConvTranspose2d(inner_nc * 2, outer_nc,
                                        kernel_size=4, stride=2,
                                        padding=1)
            down = [downconv]
            up = [uprelu, upconv, nn.Tanh()]
            model = down + [submodule] + up

        elif innermost:
            upconv = nn.ConvTranspose2d(inner_nc, outer_nc,
                                        kernel_size=4, stride=2,
                                        padding=1, bias=use_bias)

            down = [downrelu, downconv]
            up = [uprelu, upconv, upnorm]
            model = down + up
        else:
            upconv = nn.ConvTranspose2d(inner_nc * 2, outer_nc,
                                        kernel_size=4, stride=2,
                                        padding=1, bias=use_bias)


            down = [downrelu, downconv, downnorm]
            up = [uprelu, upconv, upnorm]

            if use_dropout:
                model = down + [submodule] + up + [nn.Dropout(0.5)]
            else:
                model = down + [submodule] + up

        self.model = nn.Sequential(*model)

    def forward(self, x):
        if self.outermost:
            out = self.model(x)
            return out
        else:   # skip connections
            out = torch.cat([x, self.model(x)], 1)
           return out
</code></pre>
<p>Thanks in advance!</p>

chantk: <p>Thanks for the code but somehow, the code will give me an error “normal_cuda not implemented for Long”. I had to add .float() to the following line</p>
<p>sampled_noise = self.noise.repeat(*x.size()).normal_() * scale</p>
<p>so that it become</p>
<p>sampled_noise = self.noise.repeat(*x.size()).float().normal_() * scale</p>
<p>to resolve the error. But I’m not sure will it cause any difference/error.</p>

111329: <p>you can try</p>
<pre><code class="lang-python">
noise = torch.randn_like(x)
x = ((noise + x).detach() - x).detach() + x

</code></pre>

TriKri: <p><a class="mention" href="/u/111329">@111329</a> What is the purpose of doing <code>x = ((noise + x).detach() - x).detach() + x</code>? Doesn’t this have the same result as doing <code>x = noise + x</code>, while being computationally more expensive?</p>

111329: <p>noise is not differential, so we should use the STE trick to make gradients pass through as no noise.</p>

TriKri: <p><a class="mention" href="/u/111329">@111329</a> What is the STE trick? Do you mean the reparameterization trick? If so, I think the code <code>x = noise + x</code> already uses that trick.</p>
<p>The reparameterization trick is basically just to make sure that you don’t let the random number generation depend on your learnable parameters in any way (directly or indirectly), which it doesn’t do here. That is, you need to let the parameters you use to generate the random numbers be constants, and for example not generate random numbers that are sampled from a distribution with mean <code>x</code>, because then you use <code>x</code> as a parameter to the random number generation and <code>x</code> in turn depends on your learnable parameters.</p>

111329: <p>maybe you can read this paper, I did not read it, STE is a basic trick in quantization aware training.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/1903.05662">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.4/images/icons/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/1903.05662" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://arxiv.org/abs/1903.05662" target="_blank" rel="noopener nofollow ugc">Understanding Straight-Through Estimator in Training Activation Quantized...</a></h3>

  <p>Training activation quantized neural networks involves minimizing a piecewise
constant function whose gradient vanishes almost everywhere, which is
undesirable for the standard back-propagation or chain rule. An empirical way
around this issue is to...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


TriKri: <p><a class="mention" href="/u/111329">@111329</a> Okay, maybe you can say that you can use the STE (I have never heard about the STE in the context of noise addition, though, but you usually talk about using the reparameterization trick).</p>
<p>But I still don’t think the code <code>x = ((noise + x).detach() - x).detach() + x</code> should behave any differently than <code>x = noise + x</code>; could you explain what the difference would be?</p>

111329: <p>You can try use <code>torch.rand</code> directly if <code>torch.rand</code> has normal gradients as you wish, else you should use some trick like STE to solve the backward problem.</p>

TriKri: <p>My point is that there is no problem with the code <code>x = noise + x</code>. It will work just as fine as the code that you provided (and it is much simpler so it should be preferred). There is no problem that you’re solving by <code>detach</code>ing, subtracting <code>x</code>, <code>detach</code>ing and adding <code>x</code> again.</p>
<p>Also, the straight-through estimator is used when you use a discontinuous activation function, so it doesn’t apply to this case because there is no activation function that is being used. I still think that the trick you are thinking of is the reparameterization trick. (Also, the STE is usually referred to as an “estimator”—hence the “E” in “STE”—and not as a “trick”.)</p>