How should I add a Gaussian noise to the weights of network?

username: S_M
cooked: <p>I’m new in PyTorch. I am trying to write a function that adds some arbitrary Gaussian noise to the wights during the training process. my code is like this</p>
<p>for m in model.modules():<br>
if hasattr(m, ‘weight’):<br>
m.weight.add_(np.random.normal(my_mean, my_std, m.shape)*noise_strength)<br>
and my question is the shape of “m” how can I create noise with its shape?<br>
I’m confused how should I do that, any help would be appreciated.</p>

username: vainaijr
cooked: <p>do you mean something like this?</p>
<pre><code class="lang-auto">with torch.no_grad():
  x.weight.add_(torch.randn(x.weight.size()))
</code></pre>

username: S_M
cooked: <p>Actually my model is some triple network like below:</p>
<p>class EmbeddingNet(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(EmbeddingNet, self).<strong>init</strong>()<br>
self.convnet = nn.Sequential(nn.Conv2d(1, 32, 5), nn.PReLU(),<br>
nn.MaxPool2d(2, stride=2),<br>
nn.Conv2d(32, 64, 5), nn.PReLU(),<br>
nn.MaxPool2d(2, stride=2))</p>
<pre><code>    self.fc = nn.Sequential(nn.Linear(64 * 4 * 4, 256),
                            nn.PReLU(),
                            nn.Linear(256, 256),
                            nn.PReLU(),
                            nn.Linear(256, 2)
                            )

def forward(self, x):
    output = self.convnet(x)
    output = output.view(output.size()[0], -1)
    output = self.fc(output)
    return output

def get_embedding(self, x):
    return self.forward(x)
</code></pre>
<p>class TripletNet(nn.Module):<br>
def <strong>init</strong>(self, embedding_net):<br>
super(TripletNet, self).<strong>init</strong>()<br>
self.embedding_net = embedding_net<br>
embedding_net = EmbeddingNet()<br>
model = TripletNet(embedding_net)</p>
<p>So, I am using define some arbitrary mean and std then use them into make Gaussian noise how can i add this noise during the training process of this network in each epoch. because in each epoch i define a new mean and std.</p>

username: vainaijr
cooked: <p>do you mean sampling from normal distribution with new mean and std, then we could use,</p>
<pre><code class="lang-auto">normal_dist = torch.distributions.Normal(loc=torch.tensor([0.]), scale=torch.tensor([1.0]))
</code></pre>
<p>replace torch.tensor([0.]) with mean value, and torch.tensor([1.0]) with std value<br>
one example would be</p>
<pre><code class="lang-auto">x = nn.Linear(3, 3)
t = normal_dist.sample((x.weight.view(-1).size())).reshape(x.weight.size())
with torch.no_grad():
  x.weight.add_(t)
</code></pre>

username: TinfoilHat0
cooked: <p>First get the parameters of your model as a vector</p>
<pre><code class="lang-auto">from torch.nn.utils import vector_to_parameters, parameters_to_vector

param_vector = parameters_to_vector(model.parameters())
</code></pre>
<p>Then sample a gaussian noise of the same size as this vector and add it.</p>
<pre><code class="lang-auto">n_params = len(param_vector)
noise = Normal(0, 1).sample_n(n_params)
param_vector.add_(noise)
</code></pre>
<p>Finally, load the parameters back to your model.<br>
<code>vector_to_parameters(param_vector, model.parameters())</code></p>

username: S_M
cooked: <p>hi Ravin Jain, thanks for your good comment. I have a question about your code:<br>
x.weigh.data.add_(t) or x.weight.add_(t) what is the difference between of them and which is correct?</p>