Sofmaxing over concatenated subsets of minibatch output logits yields 'variables modified by inplace operation' error

MFajcik1: <p>Hello,<br>
in my current work in open-domain QA, I am trying to implement model that is processing mini-batch of documents and applies softmax over logits from multiple documents - subsets of mini-batch - computed via same model. For example:</p>
<pre><code class="lang-auto">Minibatch form:
=== logits from doc 0 ===
=== logits from doc 1 === 
=== logits from doc 2 ===
=== logits from doc 3 === 
=== logits from doc 4 ===
=== logits from doc 5 === 
=== logits from doc 6 === 
=== logits from doc 7 ===

(0,3),(3,8) are ranges to be normalized over
loss_term1 = CE(softmax(=== logits from doc 0 === logits from doc 1 ===  logits from doc 2))
loss_term2 = CE(softmax(=== logits from doc 3 === ... ===  logits from doc 7))
loss = loss_term1+loss_term2
</code></pre>
<p>Unfortunately, I am running into ‘<em>one of the variables needed for gradient computation has been modified by an inplace operation</em>’ error.</p>
<p>An example of iteration of the following code follows:</p>
<ol>
<li>Model processes batch of e.g. 20 documents, obtaining logits for each token in the document <code>logprobs_S, logprobs_E = reader_model(batch)</code>. <code>logprobs_S</code> has size <code>batch</code> x <code>longest_document_in_minibatch</code>.</li>
<li>Model iterates over each range of documents to be processed from set of all documents - e.g. <code>paragraph_ranges=[(0,5),(5,20),(20,22)]</code>. A subset of <code>paragraph_ranges</code> list is always processed via 1 mini-batch. Assume that since our model has minibatch size 20, it has processed documents <code>currently_processed_range = (0,20)</code>.</li>
<li>Since <code>currently_processed_range = (0,20)</code>, the document subsets given by ranges <code>paragraph_ranges[0]</code> and <code>paragraph_ranges[1]</code> have been processed in this minibatch. Now I would like to compute 1 loss term for each document subset.</li>
<li>Logits from <code>logprobs_S</code>,<code>logprobs_E</code> for each processed range are indexed as <code>log_probs_S_per_ex = logprobs_S[to_be_processed_range[0] - os:to_be_processed_range[1] - os]</code>. For instance for <code>to_be_processed_range=(0,5)</code>,<br>
we will pick <code>logprobs_S[0:5]</code>.</li>
<li>As we would like to maximize the probability of logits, we need to compute cross entropy with ground truth answer span indices, which denote the offset from start of the file. Since we want to normalize over more files, we need to adjust these indices by offset given by number of logits in previous documents (done in part where <code>gt_start += offset</code>).</li>
<li>At last, after picking the logits for documents from the same <code>to_be_processed_range</code> and adjusting ground_truths by offset, I unroll logits from these documents into 1 vector and compute loss for these logits.</li>
<li>After processing all <code>to_be_processed_range</code> processed in minibatch <code>(0,5),(5,20)</code> in this example, we compute loss and occasionaly add gradients to the parameters (we accumulate gradients).</li>
</ol>
<pre><code class="lang-python">logprobs_S, logprobs_E = reader_model(batch)

print(f"Currently processed range {currently_processed_range}")
loss = 0
for example_ix, to_be_processed_range in enumerate(paragraph_ranges):

    # check whether paragraphs of to_be_processed_range have been processed
    # if they have been processed, to_be_processed_range is inside currently_processed_range
    if to_be_processed_range[0] &gt;= currently_processed_range[0] and \
            to_be_processed_range[1] &lt;= currently_processed_range[1]:

        # offset that must be substracted from to_be_processed_range
        # to align with currenty processed range index
        # e.g. if we previously processed range (0,20), 
        # and now we are processing (20,40), 
        # the 20th output is actually 0th output in logprobs_S/logprobs_E

        os = currently_processed_range[0]

        print(f"Processing range {to_be_processed_range[0]} - {to_be_processed_range[1]}.")
        print(f"Picking logit indices {to_be_processed_range[0] - os} - {to_be_processed_range[1] - os}.")
        # pick logits and answers corresponding to paragraphs aligned with 1 question
        log_probs_S_per_ex = logprobs_S[to_be_processed_range[0] - os:to_be_processed_range[1] - os]
        log_probs_E_per_ex = logprobs_E[to_be_processed_range[0] - os:to_be_processed_range[1] - os]
        answers_per_e = answers[to_be_processed_range[0] - os:to_be_processed_range[1] - os]

        # pick gt answer token indices for this example
        gt_start, gt_end = answers_per_e[flat_gt_index[example_ix]][0]

        # compute offset -  how many tokens (there is 1 logit for each token) are in documents before GT documents
        offset = log_probs_S_per_ex[:flat_gt_index[example_ix]].numel()

        # add offset to closed-domain ground truths
        gt_start += offset
        gt_end += offset

        # flatten paragraphs
        log_probs_S_per_ex = log_probs_S_per_ex.view(-1)
        log_probs_E_per_ex = log_probs_E_per_ex.view(-1)

        # compute loss over flatten paragraphs per example
        loss_s = reader_loss(log_probs_S_per_ex.unsqueeze(0), gt_start)
        loss_e = reader_loss(log_probs_E_per_ex.unsqueeze(0), gt_end)
        loss += loss_s + loss_e

        loss_elements += 1
        total_updates_so_far += 1

if loss != 0:
    total_loss += loss.item()
    loss.backward()
if loss_elements &gt;= self.config["reader_batch_size"]:
    pbar_iterator.set_description(f"Training loss: {total_loss / total_updates_so_far}")
    loss_elements = 0
    torch.nn.utils.clip_grad_norm_(filter(lambda p: p.requires_grad, reader_model.parameters()), 5.)
    reader_optimizer.step()
    reader_optimizer.zero_grad()
</code></pre>
<p>Unfortunately, I am getting a following error immediately after calling <code>loss.backward()</code>for the first time. This happens only in case if I want to process more document subsets within one batch - e.g. <code>(0,5)</code>, <code>(5,20)</code>. So far I have failed to exactly figure out what to do to prevent it and I made it work by iterating only over 1 set of documents per mini-batch, which is a very slow solution. Is there any obvious problem I am missing?</p>
<pre><code class="lang-auto">/pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:57: UserWarning: Traceback of forward call that caused the error:
  File "/home/researchpc/projects/QA/scripts/openqa/run_openret.py", line 77, in &lt;module&gt;
    framework.fit(query_enc_model, reader_model, reader_config=val_reader_config)
  File "/home/researchpc/projects/QA/scripts/openqa/openretrieval_faster.py", line 189, in fit
    train_iter, fields, reader_loss=CrossEntropyLoss())
  File "/home/researchpc/projects/QA/scripts/openqa/openretrieval_faster.py", line 420, in train_epoch_normalized_GTonly
    loss_e = reader_loss(log_probs_E_per_ex.unsqueeze(0), gt_end)
  File "/opt/anaconda3/envs/mlenv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/anaconda3/envs/mlenv/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/opt/anaconda3/envs/mlenv/lib/python3.6/site-packages/torch/nn/functional.py", line 1995, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/opt/anaconda3/envs/mlenv/lib/python3.6/site-packages/torch/nn/functional.py", line 1824, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

2019-09-30 14:31:23,235 ERROR root: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [1]] is at version 8; expected version 7 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Traceback (most recent call last):
  File "/home/researchpc/projects/QA/scripts/openqa/run_openret.py", line 80, in &lt;module&gt;
    raise be
  File "/home/researchpc/projects/QA/scripts/openqa/run_openret.py", line 77, in &lt;module&gt;
    framework.fit(query_enc_model, reader_model, reader_config=val_reader_config)
  File "/home/researchpc/projects/QA/scripts/openqa/openretrieval_faster.py", line 189, in fit
    train_iter, fields, reader_loss=CrossEntropyLoss())
  File "/home/researchpc/projects/QA/scripts/openqa/openretrieval_faster.py", line 427, in train_epoch_normalized_GTonly
    loss.backward()
  File "/opt/anaconda3/envs/mlenv/lib/python3.6/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/anaconda3/envs/mlenv/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [1]] is at version 8; expected version 7 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!```</code></pre>

albanD: <p>Hi,</p>
<p>One way to pinpoint such errors is to enable <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly" rel="nofollow noopener">anomaly detection</a> to know which operation in the forward caused the error in the backward.</p>
<p>You don’t seem to be using a lot of inplace operations. Maybe replace the <code>loss +=</code> by loss = loss +`?</p>

MFajcik1: <p>Thank for reply <a class="mention" href="/u/alband">@albanD</a>,<br>
I have enabled the anomaly detection, otherwise there would be no stacktrace n.1.<br>
After writing this long post after hours of debugging, I have replaced torch Crossentropy loss with my own implementation and now it seems to work <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> .</p>
<pre><code class="lang-auto">loss_s = -F.log_softmax(log_probs_S_per_ex, 0)[gt_start[0]]
loss_e = -F.log_softmax(log_probs_E_per_ex, 0)[gt_end[0]]
loss += loss_s + loss_e
</code></pre>
<p>Following your advice, I have also tried changing code to <code>loss = loss + loss_s + loss_e</code>, but results are the same when using torch’s CrossEntropyLoss. Is there an inplace operation somewhere in CrossEntropyLoss loss?</p>

albanD: <p>Ho,</p>
<p>Sorry about that. I missed the first stacktrace!</p>
<p>That is interesting. Can you manage to reproduce that with a small code sample (20/30 lines)? I would be interesting in finding why this happens!</p>

MFajcik1: <p>I have been trying until now, but everything simple I have created seems to work… I will make my code code public, in time, and add the reference here. (If I won’t forget) in future.</p>

albanD: <p>That must be some very specific interaction in your code then <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
Have you tried <code>.clone()</code> on each input and output of the CrossEntropyLoss? If one of them fixes the issue, then it is this one that was modified inplace <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:">  like <code>loss_s = reader_loss(log_probs_S_per_ex.unsqueeze(0).clone(), gt_start.clone()).clone()</code></p>

MFajcik1: <p>Interestingly, cloning gt really works!</p>
<pre><code class="lang-auto">loss_s = reader_loss(log_probs_S_per_ex.unsqueeze(0), gt_start.clone())
loss_e = reader_loss(log_probs_E_per_ex.unsqueeze(0), gt_end.clone())
</code></pre>
<p>and after closer investigation, I have found the guilty! It is the <code>+=</code> operator applied to offset<br>
Changing</p>
<pre><code class="lang-auto">offset = log_probs_S_per_ex[:flat_gt_index[example_ix]].numel()
# add offset to closed-domain ground truths
gt_start += offset
gt_end += offset
</code></pre>
<p>to</p>
<pre><code class="lang-auto">offset = log_probs_S_per_ex[:flat_gt_index[example_ix]].numel()
# add offset to closed-domain ground truths
gt_start = gt_start+ offset
gt_end = gt_end+ offset
</code></pre>
<p>solves this issue (without any explicit cloning)!</p>

albanD: <p>Nice catch !</p>
<p>Cloning has the side effect of making inplace operations out of place. So it can help tracking this kind of issues!</p>