GaussianDropout implementation

username: Mark_Esteins
cooked: <p>Is this a proper way of implement Gaussian Dropout or I’m missing something?</p>
<pre><code class="lang-auto">class GaussianDropout(nn.Module):
    def __init__(self, alpha=1.0):
        super(GaussianDropout, self).__init__()
        self.alpha = alpha
        
    def forward(self, x):
        if self.training:
            epsilon = torch.randn_like(x) * self.alpha + 1
            return x * epsilon
        else:
            return x
</code></pre>
<p>Would appreciate any advice or clarification on this one.</p>
<p>Thanks in advance.</p>

username: tom
cooked: <p>The PyTorch bits seem OK.</p>
<p>But one thing to consider is whether <code>alpha</code> is that descriptive a name for the standard deviation and whether it is a good parameter convention.<br>
PyTorch’s standard dropout with Bernoulli takes the rate <code>p</code>. The multiplicator will have mean 1 and standard deviation <code>(p * (1-p))**0.5 /  (1-p) = (p/(1-p))**0.5</code> (on the left side, the numerator <code>(p*(1-p))**0.5</code>  is the standard deviation of the Bernoulli and the denominator 1-p comes from the scaling.<br>
So if you want to match more what (Bernoulli) Dropout does in terms of mean and std, you could take an argument <code>p</code> and use the standard deviation <code>(p/(1-p))**0.5</code> instead of self.alpha.<br>
(I think e.g. Keras’ Gaussian dropout does that, too.)</p>
<p>Best regards</p>
<p>Thomas</p>

username: Mark_Esteins
cooked: <p>Thanks <a class="mention" href="/u/tom">@tom</a> if I understood it correctly and looking at the Keras code, your suggestion would look like this right?</p>
<pre><code class="lang-auto">class GaussianDropout(nn.Module):
    def __init__(self, p=0.5):
        super(GaussianDropout, self).__init__()
        self.p = p
        
    def forward(self, x):
        if self.training:
            assert 0 &lt; p &lt; 1
            mean = torch.ones_like(x)
            stddev = (self.p / (1.0 - self.p))**0.5
            epsilon = torch.normal(mean = mean, std = stddev) 
            return x * epsilon
        else:
            return x
</code></pre>

username: tom
cooked: <p>Almost!<br>
I’d do the assert (or better RuntimeError, assert should be for “internal”  dev errors) in the init and probably would just use torch.randn_like(…) * stddev. Other than that it looks good to me at first sight.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Mark_Esteins
cooked: <p>Thanks! Will leave the final implementation here in case it’s useful for someone else.</p>
<pre><code class="lang-auto">class GaussianDropout(nn.Module):
    def __init__(self, p=0.5):
        super(GaussianDropout, self).__init__()
        if p &lt;= 0 or p &gt;= 1:
            raise Exception("p value should accomplish 0 &lt; p &lt; 1")
        self.p = p
        
    def forward(self, x):
        if self.training:
            stddev = (self.p / (1.0 - self.p))**0.5
            epsilon = torch.randn_like(x) * stddev
            return x * epsilon
        else:
            return x
</code></pre>

username: tom
cooked: <aside class="quote no-group" data-username="Mark_Esteins" data-post="5" data-topic="151756">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/85f322/48.png" class="avatar"> Mark_Esteins:</div>
<blockquote>
<p>Will leave the final implementation here in case it’s useful for someone else.</p>
</blockquote>
</aside>
<p>Thank you, it’ll be valuable for the next person!</p>
<p>P.P.S.: For v2 of the code, I’d probably allow p=0. It can be handy to use a hyperparameter to disable dropout for experimentation (even if not having the module would be more efficient, but changing the structure this can lead to hickups e.g. with nn.Sequential when you want to compare parameters etc.).</p>