How to initialize weights inside a sequential container

username: Nath_NWA
cooked: <p>Hey,</p>
<p>So I’ve just finished re-configuring a network. I replaced <code>nn.Upsample</code> with the <code>upConv</code> sequential container shown in the code below. I’ve verified that everything is lined up by running <code>summary(UNetPP, (3, 128, 128))</code> which runs with no issue.</p>
<pre><code class="lang-auto">def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)
        
        
class blockUNetPP(nn.Module):
    def __init__(self, in_channels, middle_channels, out_channels):
        super().__init__()
        self.relu = nn.LeakyReLU(0.2, inplace=True)
        self.conv1 = nn.Conv2d(in_channels, middle_channels, 3, padding=1)
        self.bn1 = nn.BatchNorm2d(middle_channels)
        self.conv2 = nn.Conv2d(middle_channels, out_channels, 3, padding=1)
        self.bn2 = nn.BatchNorm2d(out_channels)
    

    def forward(self, x):
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)
        out = self.relu(out)

        return out
    
class upConv(nn.Module):
    def __init__(self, in_ch, out_ch):
        super().__init__()
        self.upc = nn.Sequential(
                                 nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True),
                                 nn.Conv2d(in_ch, out_ch*2, 3, stride=1, padding=1),
                                 nn.BatchNorm2d(out_ch*2),
                                 nn.ReLU(inplace=True)
                                 )
    def forward(self, x):
        out = self.upc(x)
        return out
</code></pre>
<p>My issue is that when I try to start training the model I get the following issue:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "runTrain.py", line 90, in &lt;module&gt;
    netG.apply(weights_init)
  File "C:\Users\Anaconda3\envs\CFD\lib\site-packages\torch\nn\modules\module.py", line 289, in apply
    module.apply(fn)
  File "C:\Users\Anaconda3\envs\CFD\lib\site-packages\torch\nn\modules\module.py", line 290, in apply
    fn(self)
  File "D:\Thesis Models\Deep_learning_models\UNet\train\NetC.py", line 8, in weights_init
    m.weight.data.normal_(0.0, 0.02)
  File "C:\Users\Anaconda3\envs\CFD\lib\site-packages\torch\nn\modules\module.py", line 594, in __getattr__
    type(self).__name__, name))
AttributeError: 'upConv' object has no attribute 'weight'
</code></pre>
<p>I’ve looked up  <a href="https://discuss.pytorch.org/t/how-to-initialize-weights-in-nn-sequential-container/8534">solutions</a> which suggest looping over container modules, but I’m already doing this with <code>weights_init(m)</code>. Could someone explain whats wrong with my current setup?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>There are some issues regarding your weight init.</p>
<aside class="quote no-group" data-username="Nath_NWA" data-post="1" data-topic="84512">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nath_nwa/48/12563_2.png" class="avatar"> Nath_NWA:</div>
<blockquote>
<pre><code class="lang-auto">def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)
 
</code></pre>
</blockquote>
</aside>
<ol>
<li>Never use <code>.data</code> for changing the weights or biases, it may cause problems.</li>
<li>Instead of trying to find the proper module using searching its name in string format, try to use exact object instance using <code>isinstance</code> method:</li>
</ol>
<pre><code class="lang-auto">def weights_init(m):
    if isinstance(m, nn.Conv2d):
        nn.init.normal_(m.weight, 0.0, 0.02)
    elif isinstance(m, nn.BatchNorm2d):
        nn.init.normal_(m.weight, 0.0, 0.02)
        nn.init.constant_(m.bias, 0)
</code></pre>
<p>Bests</p>

username: Nath_NWA
cooked: <p>Cheers Doosti, up and running now!</p>

username: Nikronic
cooked: <p>Great, I think the best way to learn conventions is to read the source code of models such as ResNet. Actually, I copy pasted this method and just changed it.</p>