An error on loss.backward() -> RuntimeError: could not create a primitive

username: Mohammad_78
cooked: <p>Hi,<br>
I am training a joint segmentation-classification model and the loss is the weighted average of the losses.</p>
<pre><code class="lang-python">                    loss = w_seg * loss_seg + w_cls * loss_cls
                    model.zero_grad()
                    print(loss)
                    loss.backward()
</code></pre>
<p>loss is created and printed but I have an error on loss.backward():</p>
<p>tensor(0.6947, grad_fn=&lt; AddBackward0 &gt;)<br>
Traceback (most recent call last):<br>
File “main.py”, line 140, in <br>
main(config)<br>
File “main.py”, line 80, in main<br>
solver.train()<br>
File “/project/6027897/meyta/Codes/Joint/4-11/Compile_nets.py”, line 346, in train<br>
loss.backward()<br>
File “/home/meyta/.local/lib/python3.8/site-packages/torch/tensor.py”, line 245, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/home/meyta/.local/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py”, line 145, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: could not create a primitive</p>
<p>Does anyone has an idea about the error?</p>

username: ptrblck
cooked: <p>This error was recently reported in <a href="https://discuss.pytorch.org/t/runtimeerror-could-not-create-a-primitive/117519">this thread</a>. As I couldn’t reproduce it, I’ve asked the user to create an issue on GitHub, but cannot find any issue with this error message, so could you create an issue instead with an executable code snippet (if possible), please?</p>

username: Mohammad_78
cooked: <p>Thank you for your reply. The code works properly in some versions, and in some others the error changes to:</p>
<pre><code class="lang-auto">  File "main.py", line 140, in &lt;module&gt;
    main(config)
  File "main.py", line 80, in main
    solver.train()
  File "/project/6027897/meyta/Joint/4-11/Compile_nets.py", line 344, in train
    loss.backward()
  File "/home/meyta/ENV/lib/python3.6/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/meyta/ENV/lib/python3.6/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: label is too far
</code></pre>
<p>I tried with a few systems and versions, but I couldn’t find out what was the exact source of the error. So, reproducing errors is not straightforward for me. Now, the code is working on my system but I still face the error on the server (computecanada).</p>

username: ptrblck
cooked: <p>That’s interesting, as I cannot find any references for this error message. Were you able to find any similar issue?</p>

username: willdone1337
cooked: <p>File “run.py”, line 83, in <br>
learn.fit_one_cycle(epochs=30,max_lr=1e-3)<br>
File “C:\Users\v.huseynov\PycharmProjects\dim\learn.py”, line 85, in fit_one_cycle<br>
self._fit(epochs=epochs, cyclic=True,)<br>
File “C:\Users\v.huseynov\PycharmProjects\dim\learn.py”, line 137, in <em>fit<br>
loss.backward()<br>
File “C:\ProgramData\Anaconda3\envs\dim\lib\site-packages\torch_tensor.py”, line 307, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File "C:\ProgramData\Anaconda3\envs\dim\lib\site-packages\torch\autograd_<em>init</em></em>.py", line 156, in backward<br>
allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag<br>
RuntimeError: could not create a primitive</p>
<p>HI , i got this error running model on cpu device. When i track running in colab track that in starting second epoch ram is dramatically increase. maybe problem in ram???</p>

username: ptrblck
cooked: <p>I don’t know, as I wasn’t able to reproduce or isolate another error with the same error message.<br>
Based on <a href="https://discuss.pytorch.org/t/runtimeerror-could-not-create-a-primitive/117519/4">this post</a> I could find some references to potentially missing CPU instructions, but don’t know if that was indeed the root cause.</p>

username: willdone1337
cooked: <p>I can fix this error removing tensorboard wirter logging from code part. seems tensorboard use this part of ram</p>