Why some model's parameters' grad is zero?

LingFengZhang24: <p>Hi, when I training my model, I find  some model parameters’ grad(such as conv1.weight and conv2.weight…) is zero during the training process. Here are some code snippet:</p>
<pre><code class="lang-auto">class DC_Net(nn.Module):
    def __init__(self, L):
        self.L = L
        super(DC_Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 16, 2, 2)
        self.conv2 = nn.Conv1d(16, 32, (2, 1), 2)
        self.conv3 = nn.Conv1d(32, 64, (2, 1), 2)
        self.maxp_1 = nn.MaxPool1d(2)
        self.maxp_2 = nn.MaxPool1d(2)
        self.maxp_3 = nn.MaxPool1d(2)

        self.conv1_ = nn.Conv2d(1, 16, 2, 2)
        self.conv2_ = nn.Conv1d(16, 32, (2, 1), 2)
        self.conv3_ = nn.Conv1d(32, 64, (2, 1), 2)
        self.maxp_1_ = nn.MaxPool1d(2)
        self.maxp_2_ = nn.MaxPool1d(2)
        self.maxp_3_ = nn.MaxPool1d(2)
        self.fc1 = nn.Linear(self.L, self.L-1)
        self.fc2 = nn.Linear(self.L-1, 1)
        self.fc1_ = nn.Linear(self.L, self.L-1)
        self.fc2_ = nn.Linear(self.L-1, 1)
        self.fc3 = nn.Linear(2, 1)

    def forward(self, x):
        out1 = self.maxp_1(F.relu(self.conv1(x)).squeeze())

        out1 = self.maxp_2(F.relu(self.conv2(out1.unsqueeze(-1))).squeeze())

        out1 = self.maxp_3(F.relu(self.conv3(out1.unsqueeze(-1))).squeeze())

        out2 = self.maxp_1_(F.relu(self.conv1_(x)).squeeze())
        out2 = self.maxp_2_(F.relu(self.conv2_(out2.unsqueeze(-1))).squeeze())
        out2 = self.maxp_3_(F.relu(self.conv3_(out2.unsqueeze(-1))).squeeze())

        out1 = out1.view(-1, self.L)

        out2 = out2.view(-1, self.L)

        out1 = F.relu(self.fc1(out1))
        out2 = F.relu(self.fc1_(out2))
        out1 = F.relu(self.fc2(out1))
        out2 = F.relu(self.fc2_(out2))

        out = torch.cat((out1, out2), 1)
        w = self.fc3(out)
        return w
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68b89851f6bde70ce0209c772e2771c1e1261891.png" data-download-href="https://discuss.pytorch.org/uploads/default/68b89851f6bde70ce0209c772e2771c1e1261891" title="%E6%8D%95%E8%8E%B7"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68b89851f6bde70ce0209c772e2771c1e1261891_2_690x259.png" alt="%E6%8D%95%E8%8E%B7" data-base62-sha1="eWp9ibnfaLPjz4eKZhGUZ7tS2oV" width="690" height="259" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68b89851f6bde70ce0209c772e2771c1e1261891_2_690x259.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68b89851f6bde70ce0209c772e2771c1e1261891_2_1035x388.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/8/68b89851f6bde70ce0209c772e2771c1e1261891.png 2x" data-dominant-color="F5F1F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E6%8D%95%E8%8E%B7</span><span class="informations">1116×419 61.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
could someone tell me how to fix this problem?</p>

vainaijr: <p>did you add these scalars to tensorboard after optimizer.zero_grad()?<br>
if you add them after loss.backward() and before optimizer.zero_grad(), then it will show up.</p>

LingFengZhang24: <p>Hi, vainaijr, I add them after loss.backward() and before optimizer.zero_grad(), layer connect with <strong>out2</strong>  have the grad, but without with <strong>out1</strong> , I feel something wrong with the  part of <strong>out1</strong>, but I don’t know how to fix it.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f11947ec3a1fa006ae61124816fd8a4af1a529e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/f11947ec3a1fa006ae61124816fd8a4af1a529e0" title="%E6%8D%95%E8%8E%B7"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f11947ec3a1fa006ae61124816fd8a4af1a529e0_2_690x260.png" alt="%E6%8D%95%E8%8E%B7" data-base62-sha1="yoRexEfg14f4hyJitzux3H8kC2c" width="690" height="260" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f11947ec3a1fa006ae61124816fd8a4af1a529e0_2_690x260.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f11947ec3a1fa006ae61124816fd8a4af1a529e0_2_1035x390.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f11947ec3a1fa006ae61124816fd8a4af1a529e0_2_1380x520.png 2x" data-dominant-color="F9F1EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E6%8D%95%E8%8E%B7</span><span class="informations">1454×548 90.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>