Gradient problem using two joined networks

EugenioMuttio: <p>Hello!</p>
<p>My intention is to use two networks, in which the output of the first one is modified before used as input to the second one. Here the picture shows that I am using two inputs X_n+1 and P_n, the output of the first net is thought of as an incremental value of P. Then, the sum of P_n and incremental P is P_n+1 which is used for the second network.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d35ed98fd30b6cbabae39cbce25c74050db62db.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0d35ed98fd30b6cbabae39cbce25c74050db62db" title="doublenet"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d35ed98fd30b6cbabae39cbce25c74050db62db_2_690x388.jpeg" alt="doublenet" data-base62-sha1="1SRKlF3sGyLRfe3RLTTKue4Ypm3" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d35ed98fd30b6cbabae39cbce25c74050db62db_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d35ed98fd30b6cbabae39cbce25c74050db62db_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d35ed98fd30b6cbabae39cbce25c74050db62db.jpeg 2x" data-dominant-color="FDFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">doublenet</span><span class="informations">1058×595 19.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried to do this using a single forward pass, the problem is that during the backward process there is an error:</p>
<pre><code class="lang-auto">File "/home/eugeniomuttio/anaconda3/envs/py37torch/lib/python3.7/site-packages/torch/tensor.py", line 245, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/eugeniomuttio/anaconda3/envs/py37torch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 147, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>Is there a consistent way to use two networks like this?</p>
<p>My forward pass is:</p>
<pre><code class="lang-auto">    def forward(self, x):
        batch_size, features = x.size()

        # First Network
        relu_1_out = torch.relu(self.relu_1(x))
        internal_inc = self.linear_1(relu_1_out)

        # Adding previous internal variable
        internal_var = torch.add(x[:, 1], internal_inc)
        x2 = torch.cat((x[:, 0], internal_var[:, 0]), dim=0)

        # Second Network
        out = self.linear_2(x2)

        return out, internal_var
</code></pre>
<p>Here I am using X that effectively is X = [X_n+1, P_n], internal_var is incremental P and then I add this with P_n to obtain X2 = [X_n+1, P_n+1]. I return internal_var because outside this I update X for the next forward pass.</p>