Using repeat() on differentiable layer parameters

username: gcortes
cooked: <p>I have a custom nonlinearity whose initialization is given by</p>
<pre><code class="lang-auto">def __init__(self):
    super().__init__()
    bias = torch.Tensor(100)
    self.bias = nn.Parameter(bias)
    nn.init.normal_(self.bias)
</code></pre>
<p>and whose forward pass is given by</p>
<pre><code class="lang-auto">def forward(self, x):
    batch_size = x.shape[0]
    frob_norms = torch.linalg.norm(x, axis=2)
    bias = self.bias.repeat(batch_size).view(batch_size, -1)
    relative = frob_norms - bias
    ...
    return out
</code></pre>
<p>The shape of <code>x</code> is given by <code>(batch_size, 100, 2)</code>, i.e. each sample is an array of 100 vectors of dimension 2. For each sample, I’d like to subtract the learnable <code>self.bias</code> parameter from the vector norms. The <strong>same</strong> biases should be used for each sample. However, since I’d like this operation to occur across a batch, I figured I could repeat my <code>self.bias</code> parameter as shown above and then compute the difference. I printed the contents of <code>self.bias</code> during training, only to find that it consists purely of <code>NaN</code> types, thus preventing the model from learning. How can I correct this issue? I’m not sure that I want to <code>detach()</code>, as it’s my understanding that this requires passing to the CPU which diminishes efficiency.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="gcortes" data-post="1" data-topic="160513">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/d2c977/48.png" class="avatar"> gcortes:</div>
<blockquote>
<p>I printed the contents of <code>self.bias</code> during training, only to find that it consists purely of <code>NaN</code> types, thus preventing the model from learning.</p>
</blockquote>
</aside>
<p>This should not happen and could point towards invalid gradients etc. so you should check why the <code>bias</code> is updated with invalid values or overflows etc.</p>
<aside class="quote no-group" data-username="gcortes" data-post="1" data-topic="160513">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/d2c977/48.png" class="avatar"> gcortes:</div>
<blockquote>
<p>I’m not sure that I want to <code>detach()</code>, as it’s my understanding that this requires passing to the CPU which diminishes efficiency.</p>
</blockquote>
</aside>
<p><code>detach()</code> does not decrease the efficiency and does not move the data to the CPU. However, I also don’t know how it would help in this case as detaching a tensor from the computation graph would stop the gradient propagation at this point.</p>

username: gcortes
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>. Thanks for your response. How would you suggest I check what’s causing the invalid gradients?</p>

username: ptrblck
cooked: <p>I would start with making sure the <code>bias</code> value contains valid values at the beginning and then observe when the first NaN or Inf is seen it in. Afterwards, I would probably check its <code>.grad</code> attribute to see if the gradient is invalid or what’s causing the invalid update.</p>

username: gcortes
cooked: <p>I found the issue! It’s a divide by zero error. I have a line further down that reads <code>out = torch.divide(relative, frob_norms)</code>. But I’m not sure how to handle this. Since <code>relative</code> depends on <code>bias</code> and <code>out</code> depends on <code>relative</code>, surely <code>out</code> is added to the computation graph. I tried clamping <code>frob_norms</code> to a minimum value of <code>1e-20</code>, but this still ends up introducing <code>nan</code>, probably because I’m now getting huge outputs. Ultimately, I would just like for <code>torch.divide()</code> to insert a <code>0</code> whenever a divide-by-zero were to occur, but clearly I cannot naively replace <code>nan</code> with <code>0</code> after using <code>torch.divide()</code>.</p>

username: ptrblck
cooked: <p>The common approach is to add a small <code>eps</code> value to the denominator such as <code>1e-6</code> to avoid dividing by zero.</p>