Fix random weights initialized by models in default

Marya1995: <p>First of all, I know how to fix the randomness of the used weights if I set them manually for the model layers by using (torch.manual_seed(a number) )</p>
<p>my question,<br>
when I create a model, it initializes the weights and biases by default using random values. How to fix the randomness here if I am using the default initialization for the weights and biases?<br>
I need it to be able to reproduce the results.<br>
thanks</p>

ptrblck: <p>Seeding the code should work as seen here:</p>
<pre><code class="lang-python">def checksum(model):
    s = torch.sum(torch.stack([p.double().abs().sum() for p in model.parameters()]))
    return s

for _ in range(10):
    torch.manual_seed(0)
    model = models.resnet50()
    s = checksum(model)
    print(s)
# tensor(764318.5419, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764318.5419, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764318.5419, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764318.5419, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764318.5419, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764318.5419, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764318.5419, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764318.5419, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764318.5419, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764318.5419, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
    
for _ in range(10):
    torch.manual_seed(1)
    model = models.resnet50()
    s = checksum(model)
    print(s)
# tensor(764287.9182, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764287.9182, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764287.9182, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764287.9182, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764287.9182, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764287.9182, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764287.9182, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764287.9182, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764287.9182, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(764287.9182, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
</code></pre>

Marya1995: <p>Thanks for replying,<br>
so basically it is the same, if I want to set the weights manually or keep the default ones …<br>
One more question please,<br>
what is the used approach for weight initialization, is it Xavier ?</p>

ptrblck: <aside class="quote no-group" data-username="Marya1995" data-post="3" data-topic="181229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marya1995/48/59375_2.png" class="avatar"> Marya1995:</div>
<blockquote>
<p>so basically it is the same, if I want to set the weights manually or keep the default ones …</p>
</blockquote>
</aside>
<p>I don’t understand this statement.<br>
My code shows that seeding the code will result in the same parameters created by the default random init methods for each layer. Manually setting the weights will of course use your manually defined values.</p>
<aside class="quote no-group" data-username="Marya1995" data-post="3" data-topic="181229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marya1995/48/59375_2.png" class="avatar"> Marya1995:</div>
<blockquote>
<p>what is the used approach for weight initialization, is it Xavier ?</p>
</blockquote>
</aside>
<p>It depends on the layer and is defined in the <code>reset_parameters</code> method.<br>
E.g. <code>nn.Linear</code> uses <a href="https://github.com/pytorch/pytorch/blob/1204463bd044e347890148189005291a9bede348/torch/nn/modules/linear.py#L103-L111">this code</a>:</p>
<pre data-code-wrap="python"><code class="lang-python">    def reset_parameters(self) -&gt; None:
        # Setting a=sqrt(5) in kaiming_uniform is the same as initializing with
        # uniform(-1/sqrt(in_features), 1/sqrt(in_features)). For details, see
        # https://github.com/pytorch/pytorch/issues/57109
        init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in) if fan_in &gt; 0 else 0
            init.uniform_(self.bias, -bound, bound)
</code></pre>

Marya1995: <p>Sorry for inconvenience for the first part,  I meant that (torch.manual_seed(a number) ) is used either with the default random initialization or with the manually set approach for random initialization e.g., change the initially defined weights to Xavier manually. I understood from your answer is YES we use the torch.manual_seed().</p>
<p>The last question please,<br>
is there a way to make the model  initializing all layers by default with Xavier?</p>

ptrblck: <p>Yes, you can use <code>torch.nn.init.xavier_normal_/.xavier_uniform_</code> to initialize parameters by passing them directly to this method or by using it in a function passed to <code>model.apply</code>.</p>