Question on Initializing weights

Estabi: <p>Hello,<br>
I am new to deep learning and using pytorch.</p>
<p>I have gone through some codes and I always see examples of weight initialization.</p>
<p>My questions are -</p>
<p>When do I initialize weights and what is the intuition behind weights initialization for neural networks.</p>
<p>Does random seeding have anything to do with weights iniitialization?</p>
<p>Say if I have nn Module before a pretrained model do I initialize weights for the nn Module? see below example<br>
Thank you for the help.</p>
<pre><code class="lang-auto">class inNet(nn.Module):
    def __init__(self):
        super(inNet, self).__init__()
        nC = 3
        inC = nC * 2
        self.bn1 = nn.BatchNorm2d(nC)
        self.conv1 = nn.Conv2d(nC, inC, kernel_size=1, bias=False)
        self.bn2 = nn.BatchNorm2d(inC)
        self.conv2 = nn.Conv2d(inC, nC, kernel_size=3, padding=1, bias=False)
        self.avg2 = nn.MaxPool2d(2)

    def forward(self, x):
        out = self.conv1(F.relu(self.bn1(x)))
        out = self.conv2(F.relu(self.bn2(out)))
        out = self.avg2(out)
        return out
</code></pre>
<pre><code class="lang-auto">class FinalNet(nn.Module):
    def __init__(self):
        super(FinalNet, self).__init__()
        model = resnet18(pretrained=True)
        self.down_network = model()
        self.up_network = inNet()
        self.final_layer = inNet()
        self.fc = nn.Linear(....)


    def forward(self, x):
       ......
       ......

</code></pre>

Estabi: <p>Can anyone please help answer this?</p>
<p>Thank you.</p>

ptrblck: <aside class="quote no-group" data-username="Estabi" data-post="1" data-topic="163744">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/edb3f5/48.png" class="avatar"> Estabi:</div>
<blockquote>
<p>When do I initialize weights and what is the intuition behind weights initialization for neural networks.</p>
</blockquote>
</aside>
<p>A proper weight initialization allows models to converge faster or to converge at all. A custom weight initialization is used if the default built-in initialization isn’t working for your use case.</p>
<aside class="quote no-group" data-username="Estabi" data-post="1" data-topic="163744">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/edb3f5/48.png" class="avatar"> Estabi:</div>
<blockquote>
<p>Does random seeding have anything to do with weights iniitialization?</p>
</blockquote>
</aside>
<p>Yes, since random values are usually drawn from a specific distribution, which use the pseudorandom number generation which you can seed.</p>
<aside class="quote no-group" data-username="Estabi" data-post="1" data-topic="163744">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/edb3f5/48.png" class="avatar"> Estabi:</div>
<blockquote>
<p>Say if I have nn Module before a pretrained model do I initialize weights for the nn Module?</p>
</blockquote>
</aside>
<p>You can use a custom weight init for the newly created layers or just stick to the default random init defined by PyTorch.</p>

Estabi: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> . These answer it all.</p>