Runtime Error in backpropagation of the graph second time

username: I3aer
cooked: <p>Hi,</p>
<p>I am implementing a GNN, which uses Chebyshev’s polynomials. I have a for loop that uses the recursive relation between Chebyshev’s polynomials:</p>
<pre><code class="lang-auto"># for k = 0, T_0(L) = I
Tx = T0_x = graphs.clone()
    
if self.cheby_K &gt; 1: # T_1(L) = L
    T1_x = laplacians @ graphs
    Tx = torch.cat((T0_x, T1_x),dim=-1)
    # T_k(x) = 2*L*T_{k-1}(x) - T_{k-2}(x)
for k in range(2, self.cheby_K):
   Tk_x = 2 * laplacians @ self.T1_x - T0_x
   Tx = torch.cat((Tx,Tk_x),dim=-1)
   T1_x, T0_x = Tk_x, T1_x
</code></pre>
<p>When I test my network with the following snippet:</p>
<pre><code class="lang-auto">with torch.autograd.set_detect_anomaly(True):
        with torch.enable_grad():
            
            graphs = torch.unsqueeze(nodes,0).contiguous().to(device)
            
            laplacians = torch.unsqueeze(laplacian,0).to(device)
                
            for i in range(0,100):
                
                optimizer.zero_grad()
                
                logits, out_nodes, out_laplacians = gnn(graphs,laplacians)
                
                loss = criterion(logits,labels)
                
                print('epoch:{1:d}, loss: {0:.4f}'.format(loss,i))
                
                probs = torch.softmax(logits,dim=-1)
                
                print('predictions:\n',probs.type(torch.float16).data.numpy())
                
                loss.backward()
                
                optimizer.step()
</code></pre>
<p>I receive the following error message</p>
<blockquote>
<p>Blockquote “RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the first time.”</p>
</blockquote>
<p>According to the error, I must specify the retain_graph=True when calling .backward in my loop above, but the training code again raises another RunTimeError:</p>
<blockquote>
<p>Blockquote RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [8, 32]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<p>I dont use any hidden state. Also, if I set self.cheby_K=1, the training runs without an error.<br>
Could you help me to figure out what I am doing wrong?</p>
<p>Thanks</p>
<p>Edit:Typo</p>

username: pascal_notsawo
cooked: <p>Is the line concerned by the error not pointed out in the whole error message? If so, please let me see it.</p>

username: I3aer
cooked: <p>The following is the whole error message when I dont use the retain_graph=True in .backward():</p>
<blockquote>
<p>Blockquote C:\Users\I3ase\miniconda3\lib\site-packages\torch\autograd_<em>init</em>_.py:145: UserWarning: Error detected in ReluBackward0. Traceback of forward call that caused the error:<br>
File “C:\Users\I3ase\miniconda3\lib\runpy.py”, line 197, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “C:\Users\I3ase\miniconda3\lib\runpy.py”, line 87, in <em>run_code<br>
exec(code, run_globals)<br>
File "C:\Users\I3ase\miniconda3\lib\site-packages\spyder_kernels\console_<em>main</em></em>.py", line 23, in <br>
start.main()<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\spyder_kernels\console\start.py”, line 296, in main<br>
kernel.start()<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\ipykernel\kernelapp.py”, line 612, in start<br>
self.io_loop.start()<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\tornado\platform\asyncio.py”, line 199, in start<br>
self.asyncio_loop.run_forever()<br>
File “C:\Users\I3ase\miniconda3\lib\asyncio\base_events.py”, line 596, in run_forever<br>
self._run_once()<br>
File “C:\Users\I3ase\miniconda3\lib\asyncio\base_events.py”, line 1890, in _run_once<br>
handle._run()<br>
File “C:\Users\I3ase\miniconda3\lib\asyncio\events.py”, line 80, in _run<br>
self._context.run(self._callback, *self._args)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\tornado\ioloop.py”, line 688, in <br>
lambda f: self._run_callback(functools.partial(callback, future))<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\tornado\ioloop.py”, line 741, in _run_callback<br>
ret = callback()<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\tornado\gen.py”, line 814, in inner<br>
self.ctx_run(self.run)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\tornado\gen.py”, line 775, in run<br>
yielded = self.gen.send(value)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\ipykernel\kernelbase.py”, line 365, in process_one<br>
yield gen.maybe_future(dispatch(*args))<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\tornado\gen.py”, line 234, in wrapper<br>
yielded = ctx_run(next, result)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\ipykernel\kernelbase.py”, line 268, in dispatch_shell<br>
yield gen.maybe_future(handler(stream, idents, msg))<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\tornado\gen.py”, line 234, in wrapper<br>
yielded = ctx_run(next, result)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\ipykernel\kernelbase.py”, line 543, in execute_request<br>
self.do_execute(<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\tornado\gen.py”, line 234, in wrapper<br>
yielded = ctx_run(next, result)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\ipykernel\ipkernel.py”, line 306, in do_execute<br>
res = shell.run_cell(code, store_history=store_history, silent=silent)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\ipykernel\zmqshell.py”, line 536, in run_cell<br>
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\IPython\core\interactiveshell.py”, line 2894, in run_cell<br>
result = self._run_cell(<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\IPython\core\interactiveshell.py”, line 2940, in _run_cell<br>
return runner(coro)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\IPython\core\async_helpers.py”, line 68, in <em>pseudo_sync_runner<br>
coro.send(None)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\IPython\core\interactiveshell.py”, line 3165, in run_cell_async<br>
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\IPython\core\interactiveshell.py”, line 3357, in run_ast_nodes<br>
if (await self.run_code(code, result,  async</em>=asy)):<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\IPython\core\interactiveshell.py”, line 3437, in run_code<br>
exec(code_obj, self.user_global_ns, self.user_ns)<br>
File “”, line 1, in <br>
runfile(‘C:/Users/I3ase/Documents/projects/agcn/unit_test_sgc_ll.py’, wdir=‘C:/Users/I3ase/Documents/projects/agcn’)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 565, in runfile<br>
exec_code(file_code, filename, ns_globals, ns_locals,<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 453, in exec_code<br>
exec(compiled, ns_globals, ns_locals)<br>
File “C:\Users\I3ase\Documents\projects\agcn\unit_test_sgc_ll.py”, line 96, in <br>
logits, out_nodes, out_laplacians = gnn(graphs,laplacians)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\torch\nn\modules\module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\I3ase\Documents\projects\agcn\unit_test_sgc_ll.py”, line 24, in forward<br>
graphs2, laplacians2 = self.layer2(graphs1, laplacians1)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\torch\nn\modules\module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\I3ase\Documents\projects\agcn\sgc_ll.py”, line 183, in forward<br>
out_graphs = F.relu(res_graph + sc_graph)<br>
File “C:\Users\I3ase\miniconda3\lib\site-packages\torch\nn\functional.py”, line 1206, in relu<br>
result = torch.relu(input)<br>
(Triggered internally at  …\torch\csrc\autograd\python_anomaly_mode.cpp:104.)<br>
Variable._execution_engine.run_backward(<br>
Traceback (most recent call last):</p>
</blockquote>
<p>File “C:\Users\I3ase\Documents\projects\agcn\unit_test_sgc_ll.py”, line 106, in <br>
loss.backward()</p>
<p>File “C:\Users\I3ase\miniconda3\lib\site-packages\torch\tensor.py”, line 245, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)</p>
<p>File “C:\Users\I3ase\miniconda3\lib\site-packages\torch\autograd_<em>init</em>_.py”, line 145, in backward<br>
Variable._execution_engine.run_backward(</p>
<p>RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the first time.</p>
<p>In addition, DGL uses a similar loop <a href="https://docs.dgl.ai/_modules/dgl/nn/pytorch/conv/chebconv.html#ChebConv.forward" rel="noopener nofollow ugc">ChebNet</a></p>

username: I3aer
cooked: <p>Hi,</p>
<p>I have solved the problem. I have changed the name of the updated laplacian in the network. I named both that tensor and the input tensor with the same name ‘laplacians’. In addition, I’ve changed the addition-assignment to addition, i.e.,<br>
laplacians += res_laplacians  → u_laplacians = laplacians + res_laplacians</p>