How to multiply a tensor with itself?

Bachstelze: <p>I am incorporating some custom functions into <a href="https://github.com/karpathy/nanoGPT" rel="noopener nofollow ugc">nanoGPT</a>. Performing a linear combination just works fine:</p>
<pre><code class="lang-auto">def static_combination(batch):
  sequence_length, dimension = batch.size()
  for sequence in range(1, sequence_length):
    scalar = 1.0/(sequence+1)
    batch[sequence] = torch.add(batch[sequence]*scalar, batch[sequence-1], alpha=(1-scalar))
  return batch
</code></pre>
<p>Changing the operation to a multiplication yields a RunTimeError:</p>
<pre><code class="lang-auto"># multiply the token with the previous 
def token_multiplication(batch):
  sequence_length, token_dimension = batch.size()
  for token_number in range(1, sequence_length):
    batch[token_number] = torch.mul(batch[token_number], batch[token_number-1])
  return batch
</code></pre>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2, 128]], which is output 0 of AsStridedBackward0, is at version 63; expected version 62 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>With <code>torch.autograd.set_detect_anomaly(True)</code>:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2, 128]], which is output 0 of AsStridedBackward0, is at version 63; expected version 62 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later.
</code></pre>
<p>As far as I know <code>retain_graph=True</code> isn’t used.<br>
The environment setting is:</p>
<ul>
<li><code>transformers</code> version: 4.40.2</li>
<li>Platform: Linux-6.1.85±x86_64-with-glibc2.35</li>
<li>Python version: 3.10.12</li>
<li>Huggingface_hub version: 0.23.0</li>
<li>Safetensors version: 0.4.3</li>
<li>Accelerate version: not installed</li>
<li>Accelerate config: not found</li>
<li>PyTorch version (GPU?): 2.2.1+cu121 (False)</li>
<li>Tensorflow version (GPU?): 2.15.0 (False)</li>
<li>Flax version (CPU?/GPU?/TPU?): 0.8.3 (cpu)</li>
<li>Jax version: 0.4.26</li>
<li>JaxLib version: 0.4.26</li>
<li>Using GPU in script?: No</li>
<li>Using distributed or parallel set-up in script?:No</li>
</ul>

Bachstelze: <p>Detaching and cloning the previous tensor seems to solve the issue:</p>
<pre><code class="lang-auto">def token_multiplication(batch):
  sequence_length, token_dimension = batch.size()
  for token_number in range(1, sequence_length):
    # copy the result for the backpropagation
    batch[token_number] = torch.mul(batch[token_number], batch[token_number-1].detach().clone())
  return batch
</code></pre>
<p>Is this the correct way for the backpropagation?</p>

KFrank: <p>Hi Wags!</p>
<aside class="quote no-group" data-username="Bachstelze" data-post="2" data-topic="203170" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bachstelze/48/70185_2.png" class="avatar"> Bachstelze:</div>
<blockquote>
<p>Detaching and cloning the previous tensor seems to solve the issue:</p>
</blockquote>
</aside>
<p>You almost certainly don’t want to do this.</p>
<p>Try <code>clone()</code>ing <code>batch[token_number]</code> (without a <code>.detach()</code>), rather than<br>
<code>.detach()</code>ing <code>batch[token_number - 1]</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    # copy the result for the backpropagation
    batch[token_number] = torch.mul(batch[token_number], batch[token_number-1].detach().clone())
</code></pre>
</blockquote>
</aside>
<p>My take on what is going on:</p>
<p>An inplace-modification error happens when two things occur:  first, a tensor from<br>
the forward pass is needed to compute gradients during the backward pass; and<br>
second, that tensor is modified inplace.  Note that assigning into a tensor using<br>
indexing or slicing counts as an inplace modification.</p>
<p>With your original:</p>
<pre data-code-wrap="python"><code class="lang-python">batch[token_number] = torch.mul(batch[token_number], batch[token_number-1])
</code></pre>
<p>in order to compute the gradient with respect to  <code>batch[token_number - 1]</code>,<br>
<code>.backward()</code> needs to know what <code>batch[token_number - 1]</code> was multiplied by,<br>
namely <code>batch[token_number]</code>, which is to say, the whole <code>batch</code> tensor.  But you<br>
modify <code>batch</code> inplace, hence the error.</p>
<p>When you <code>.detach()</code> <code>batch[token_number - 1]</code>, <code>.backward()</code> no longer tries to<br>
compute this part of the gradient, so <code>batch</code> (in particular, <code>batch[token_number]</code>)<br>
is no longer needed and it doesn’t matter that it was modified inplace, so the error<br>
goes away.</p>
<aside class="quote no-group">
<blockquote>
<p>Is this the correct way for the backpropagation?</p>
</blockquote>
</aside>
<p>No, because you are no longer computing the piece of the gradient that comes<br>
from <code>batch[token_number - 1]</code>.  You probably do want this part of the gradient,<br>
so your “fix” is probably wrong.</p>
<p>By <code>.clone()</code>ing <code>batch[token_number]</code> (before overwriting it by modifying<br>
<code>batch</code> inplace), you let the computation graph hold on to (the unmodified)<br>
<code>batch[token_number]</code>, which is then used by <code>.backward()</code> to compute the<br>
gradient.</p>
<p>Best.</p>
<p>K. Frank</p>