Loss is 1 but gradients are zero

Ravi_Raja: <p>I am facing this issue of gradient being 0 even though the loss is not zero. loss stays at 1 while gradients are 0. I’m using the MSE loss function. Can anyone please help me here in debugging this?</p>
<p>Training code snippet:</p>
<pre><code class="lang-auto"># Train network
    max_epochs = max_epochs+1
    epoch = 1
    last_acc = 0
    while epoch &lt; max_epochs:
        gcln.train()
        optimizer.zero_grad()
        train_epoch_loss = 0
        accuracy = 0
        datalen = 0
        train_size = 0
        output = []
        target = []
        for batch_idx, (inps, tgts) in enumerate(train_loader):
            tgts = tgts.reshape((tgts.size(0), -1)).to(device)
            tgts = tgts.round()
            inps = inps.to(torch.float)
            outs = gcln(inps).to(device)
            gcln_ = copy.deepcopy(gcln)
            gcln_.cnf_layer_1.layer_or_weights = torch.nn.Parameter(
                gcln_.cnf_layer_1.layer_or_weights.round())
            gcln_.cnf_layer_1.layer_and_weights = torch.nn.Parameter(
                gcln_.cnf_layer_1.layer_and_weights.round())
            gcln_.cnf_layer_2.layer_or_weights = torch.nn.Parameter(
                gcln_.cnf_layer_2.layer_or_weights.round())
            gcln_.cnf_layer_2.layer_and_weights = torch.nn.Parameter(
                gcln_.cnf_layer_2.layer_and_weights.round())
            out_ = gcln_(inps)
            print("out_", out_.shape)
            if architecture == 1:
                t_loss = criterion(
                    outs, tgts[:, current_output].unsqueeze(-1))
                train_epoch_loss += t_loss.item()
            elif architecture == 2:
                t_loss = criterion(outs, tgts)
                train_epoch_loss += t_loss.item()
            elif architecture == 3:
                l = []
                for i in range(num_of_outputs):
                    l.append(criterion(outs[:, i], tgts[:, i]))
                t_loss = sum(l)
                train_epoch_loss += t_loss.item()/num_of_outputs
            train_size += outs.shape[0]

            optimizer.zero_grad()
            t_loss.backward()
            optimizer.step()
            if architecture == 1:
                output.append([abs(e)
                              for e in out_.round().flatten().tolist()])
                target.append(tgts[:, current_output].tolist())
                accuracy += (out_.round().squeeze() ==
                             tgts[:, current_output]).sum()
            elif architecture &gt; 1:
                output.append([abs(e)
                              for e in out_.round().flatten().tolist()])
                target.append(tgts.flatten().tolist())
                accuracy += (out_.round() == tgts).sum()
        train_loss.append(train_epoch_loss)
</code></pre>
<p>Thanks</p>

thecho7: <p>To clarify your problem, please give us a code snippet.</p>

Ravi_Raja: <p>added the code snippet for training loop</p>

thecho7: <p>I think that code is well-written even though having some weird parts…<br>
Loss value can be stuck with non-zero gradient but another case is not possible if your model is nicely defined.</p>
<p>Check every layer in your model to find any NaN or something.</p>

InnovArul: <aside class="quote no-group" data-username="Ravi_Raja" data-post="1" data-topic="149296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ravi_raja/48/32723_2.png" class="avatar"> Ravi_Raja:</div>
<blockquote>
<pre><code class="lang-auto">            gcln_.cnf_layer_1.layer_or_weights = torch.nn.Parameter(
                gcln_.cnf_layer_1.layer_or_weights.round())
            gcln_.cnf_layer_1.layer_and_weights = torch.nn.Parameter(
                gcln_.cnf_layer_1.layer_and_weights.round())
            gcln_.cnf_layer_2.layer_or_weights = torch.nn.Parameter(
                gcln_.cnf_layer_2.layer_or_weights.round())
            gcln_.cnf_layer_2.layer_and_weights = torch.nn.Parameter(
                gcln_.cnf_layer_2.layer_and_weights.round())
</code></pre>
</blockquote>
</aside>
<p>Are you sure that these <code>round()</code> operations do not lead to all elements of weights being <code>0</code>’s?</p>

thecho7: <p>I don’t think <code>glcn_</code> and <code>glcn</code> are related.<br>
The <code>t_loss</code> comes from the <code>glcn</code>.</p>

InnovArul: <aside class="quote no-group" data-username="thecho7" data-post="6" data-topic="149296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/eada6e/48.png" class="avatar"> thecho7:</div>
<blockquote>
<p>I don’t think <code>glcn_</code> and <code>glcn</code> are related.</p>
</blockquote>
</aside>
<p>I see. You are right. Didn’t notice that:)<br>
Also, there are many unknown things in the question.<br>
What are the range of values in <code>tgts</code>, what about the network design and the final layer activation etc.</p>

Ravi_Raja: <p><a class="mention" href="/u/thecho7">@thecho7</a> there are weird parts as the problem in itself is unique. I need the weights to be interpretable. A particular weight value ranges between 0 and 1 but at the end I want them to be either 0 or 1. After training I have to read formula using the learned weights of the network.</p>
<p>Here’s the code for model:</p>
<pre><code class="lang-auto">class CNF_Netowrk(torch.nn.Module):
    def __init__(self, input_size, output_size, hidden_size, device) -&gt; None:
        super().__init__()
        self.device = device
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size

        self.layer_or_weights = torch.nn.Parameter(
            torch.Tensor(
                self.input_size, self.hidden_size
            ).uniform_(0., 1.).to(dtype=torch.double).to(self.device)
        )

        self.layer_and_weights = torch.nn.Parameter(
            torch.Tensor(
                self.hidden_size, self.output_size
            ).uniform_(0., 1.).to(dtype=torch.double).to(self.device)
        )

    def apply_gates(self, x, y):
        return torch.mul(x, y)

    def forward(self, inputs):
        with torch.no_grad():
            self.layer_or_weights.data.clamp_(0.0, 1.0)
            self.layer_and_weights.data.clamp_(0.0, 1.0)

        # gated_inputs.shape: batch_size x hidden_size
        gated_inputs = self.apply_gates(self.layer_or_weights, inputs)
        o = 1 - gated_inputs

        # or_res.shape: batch_size x K
        or_res = 1 - util.tnorm_n_inputs(o)
        or_res = or_res.unsqueeze(-1)

        # gated_or_res.shape: batch_size x K
        gated_or_res = self.apply_gates(self.layer_and_weights, or_res)
        gated_or_res = torch.add(
            gated_or_res, 1.0 - self.layer_and_weights, alpha=1)

        # out.shape: batch_size x output_size
        outs = util.tnorm_n_inputs(gated_or_res).unsqueeze(-1)

        return outs
</code></pre>
<p>Also, I checked the outputs of each layer, there’s no NaNs in them.</p>
<p><a class="mention" href="/u/innovarul">@InnovArul</a> gcln_ is deepcopied to round the weights to 0’s and 1’s so that I can use them to get binary outputs to be compared with tgts.</p>
<p>tgts is binary vector.</p>

mMagmer: <p>hi,<br>
maybe you’re cutting computation graph somewhere in forward pass.<br>
i think you can check it by filling parameter.grad with value other than zero.<br>
then backward loss to see if it changes at all.</p>

InnovArul: <aside class="quote no-group" data-username="Ravi_Raja" data-post="8" data-topic="149296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ravi_raja/48/32723_2.png" class="avatar"> Ravi_Raja:</div>
<blockquote>
<p>Also, I checked the outputs of each layer, there’s no NaNs in them.</p>
</blockquote>
</aside>
<p>Just a hypothesis. Can you plot the distribution of <code>self.layer_and_weights</code> for every training iteration?<br>
Is there a chance that after some iterations of training, this <code>self.layer_and_weights</code> is going <code>0</code>?<br>
Can you verify that?</p>

Ravi_Raja: <p>No the weights are not going to 0 but it remains constant after some time. Which is because gradient being 0.</p>

Ravi_Raja: <p><a class="mention" href="/u/mmagmer">@mMagmer</a> yes you are right I guess. gradients doesn’t change at all after doing what you said to do. But why is this not happening in case of other examples. And how to find out what is cutting the computation graph?</p>

mMagmer: <p>now that i think about it, the test is not the right way to check for computation graph.<br>
i wanna say it’s in <code>util.tnorm_n_inputs</code> part, but i’m not sure.</p>

Ravi_Raja: <p>Here’s the implementation for tnorm_n_inputs. I’m using the product case. Do you think that is problematic?</p>
<pre><code class="lang-auto">def tnorm_n_inputs(self, inp):
        '''
        Fuzzy alternative for Logical AND
        '''

        if self.name == "godel":
            out, _ = torch.min(inp, dim=-2)
            return out
        elif self.name == "product":
            return torch.prod(inp, -2)
        else:
            print("Wrong Name!")
</code></pre>

mMagmer: <aside class="quote no-group" data-username="Ravi_Raja" data-post="14" data-topic="149296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ravi_raja/48/32723_2.png" class="avatar"> Ravi_Raja:</div>
<blockquote>
<pre><code class="lang-auto">elif self.name == "product":
            return torch.prod(inp, -2)
</code></pre>
</blockquote>
</aside>
<p>no problem in this case. but <code>torch.min </code> will cut the graph,</p>

Ravi_Raja: <p>Yeah. I’m not using that anywhere.</p>

mMagmer: <aside class="quote" data-post="2" data-topic="40020">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-are-hooks-used-for/40020/2">What are hooks used for?</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    You could pass a function as the hook to register_hook, which will be called every time the gradient is calculated. 
This might be useful for debugging purposes, e.g. just printing the gradient or its statistics, or you could of course manipulate the gradient in a custom way, e.g. normalizing it somehow etc.
  </blockquote>
</aside>


mMagmer: <aside class="quote no-group" data-username="mMagmer" data-post="9" data-topic="149296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmagmer/48/43283_2.png" class="avatar"> mMagmer:</div>
<blockquote>
<p>maybe you’re cutting computation graph somewhere in forward pass</p>
</blockquote>
</aside>
<p>i think i am wrong about this,<br>
sorry</p>

Ravi_Raja: <p>No worries. Thanks for helping.</p>

Ravi_Raja: <p><a class="mention" href="/u/innovarul">@InnovArul</a> can you tell what happens if layer_and_weights goes to zero after some iterations?</p>