Backpropagation is not updating all the models in the nn.ModuleList

ala_baccar: <p>Hi,<br>
I created a detector with nn.ModuleList which contains many DNN networks each has 2 hidden layers and one output layers, and the forward function is as follows: each dnn in the nn.Modulelist takes the output of its previous dnn and gives new prediction which will be given to the next dnn in the nn.ModuleList, but when I checked training with params.grad, only the grads of the last dnn of the modulelist are being calculated but all of the rest are None, I don’t understand why backpropagation is not going through all the DNNs of the modulelist.<br>
So basically on the params of the last DNN are being optimized.</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue?</p>

ala_baccar: <p>The code is pretty long and have many details here are the main parts:</p>
<pre><code class="lang-auto">     ############### 

    def calculate_posteriors(self,sub_model,previous_prob,rx,H,i):
        """
        Propagates the probabilities through the learnt networks.
        Returns the probability matrix(users over columns) of symbols at iteration i
        Prev_prob here is the Sym vector(sym_vec)
        """
        next_probs_vec = (torch.zeros((H.shape[0],self.output,self.n_users))).to(self.device)

        for user in range(self.n_users):
           .................
            
            if i==self.n_iter-1:
                output = sub_model[user * self.n_iter + i](input.float())
            else:
                output = self.softmax1(sub_model[user * self.n_iter + i](input.float()))
            next_probs_vec[:,:,user]=output
            next_probs_vec=next_probs_vec.to(self.device)

        return next_probs_vec



    def forward(self,rx,H):

        probs_vec= (torch.ones((H.shape[0],self.output,self.n_users))/self.output).to(self.device)
        sym_vec=(torch.ones((H.shape[0],1,self.n_users),dtype=torch.complex64))

        for i in range(self.n_iter):
            '''
            Encoding propagated probabilities and calculating interfering term which will give it to each dnn of each user
            '''
            ####### Post-Processing
                  ................
            ######################### Updating the predictions ###################################
       #####The self.detector here is the module list
            sym_vec=sym_vec.to(self.device)
            probs_vec=self.calculate_posteriors(self.detector,sym_vec,rx,H,i)
        return probs_vec
</code></pre>
<p>Then I just take the probs_vec from the forward which is the output of the last DNNs and calculate loss with cross-entropy which only updates these last DNNs but not the previous.<br>
I hope this snippet of code gives a little idea about the context.</p>
<p><strong>Just forgot to mention that here the sym_vec that I give it to the next DNNs is some kind of transformations I do it to the probs_vec(previous output), so before giving the prediction of the prev DNN to the next I encode it then give it to the next dnn for its calculations.</strong></p>

ala_baccar: <p>Even in this computational graph, only the DNNs of the last iteration figure out. (5,11,17,23) from the whole 24 networksg<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f63f6dcdafa33f1805f0a32b5153fccb312323a.png" data-download-href="https://discuss.pytorch.org/uploads/default/5f63f6dcdafa33f1805f0a32b5153fccb312323a" title="graph"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f63f6dcdafa33f1805f0a32b5153fccb312323a_2_690x248.png" alt="graph" data-base62-sha1="dBRwZUbUbBxCYDpV6TOf3Uhe854" width="690" height="248" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f63f6dcdafa33f1805f0a32b5153fccb312323a_2_690x248.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f63f6dcdafa33f1805f0a32b5153fccb312323a_2_1035x372.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f63f6dcdafa33f1805f0a32b5153fccb312323a_2_1380x496.png 2x" data-dominant-color="EDF0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">graph</span><span class="informations">1913×689 183 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>It’s unclear what exactly causes this behavior based on the provided code snippet. You could add debug <code>print</code> statements checking if input tensors from previous modules have a valid <code>.grad_fn</code> and if it disappears at one point to narrow down where the computation graph is cut.</p>

ala_baccar: <p>I printed the grad_fn of all the inputs, that are being forwarded between the sub-models and it gives only none. Although the input.requires_grad is set to True.</p>
<p>I also checked the grad_fn of the outputs and they are not none, but I have to mention here that the input to the next dnn is not only the output of the previous dnn, I concatenate this output with another tensor then I give it to the next dnn.</p>

ptrblck: <p>Concatenating a tensor from a computation graph won’t break the graph as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(1, requires_grad=True)
out = torch.cat((x, torch.randn(1)))
print(out.grad_fn)
# &lt;CatBackward0 object at 0x7fcd2b3fbcd0&gt;
out.mean().backward()
print(x.grad)
# tensor([0.5000])
</code></pre>
<p>so you might want to check why the <code>.grad_fn</code> of the inputs to the modules is set to <code>None</code>.</p>

ala_baccar: <p>I think the problem is that I am not exactly using that prediction as input, I am encoding it in a new tensor then I concatenate that encoded tensor with other variable and give it as input, I don’t know how to include this new whole input (which includes the encoded version of the DNN output) in the computational graph. I thought just setting requires_grad=True will solve the issue.</p>

ptrblck: <p>No, creating a new tensor will detach it. Setting the <code>.requires_grad</code> attribute to <code>True</code> afterwards won’t reattach the tensor, so you would have to use <code>torch.stack</code> or <code>torch.cat</code> to concatenate tensors as seen in my example.</p>
<p>EDIT: also numpy arrays are not tracked by Autograd so the <code>torch.from_numpy</code> tensor also does not have a gradient history.</p>

ala_baccar: <p>So this means that there is no way I can do an encoding to the previous output (Which is attached to the computational graph) and give it to the next DNN in a way that stays attached to the graph?</p>

ptrblck: <p>My previous code snippet shows that <code>torch.cat</code> works fine. A minimal and executable code snippet showing the exact operations detaching the graph would be helpful.</p>

ala_baccar: <p>It should be here in the last line where I used the sym_vec tensor (encoded version of the output tensor) as input, which cannot be added to the graph in the first place after its creation</p>
<pre><code class="lang-auto">
            for iue in range(probs_vec.shape[2]):
                for batch in range(H.shape[0]):
                    idx = torch.argmax(probs_vec[batch, :, iue]).item()
                    bb= OneHotDeco(self.Onehot[idx])
                    sym_vec[batch,:, iue] = torch.from_numpy(qam.encode(bb.astype(int)))

            sym_vec=sym_vec.to(self.device)
            sym_vec.requires_grad=True
            probs_vec=self.calculate_posteriors(self.detector,sym_vec,rx,H,i)
</code></pre>

ptrblck: <p>It’s still unclear where <code>sym_vec</code> comes from. If it’s an output of a previous differentiable operation, the code will still work even though it’s treating the <code>torch.from_numpy</code> tensor as a constant:</p>
<pre data-code-wrap="python"><code class="lang-python">w = torch.randn(2, requires_grad=True)
sym_vec = w * 2
print(sym_vec.grad_fn)
# &lt;MulBackward0 object at 0x7fcd2b468100&gt;

# replace with static tensor, which is not attached to a computation graph
sym_vec[1] = torch.from_numpy(np.random.randn(1))

print(sym_vec.grad_fn)
# &lt;CopySlices object at 0x7fcd2b468550&gt;

sym_vec.mean().backward()
print(w.grad)
# tensor([1., 0.])
</code></pre>
<p>If you replace all values of <code>sym_vec</code>, it will still work to call <code>backward()</code> on the tensor, but all gradients will be zero, since no gradient history is attached to the numpy arrays.</p>

ala_baccar: <p>Actually sym_vec is a tensor that I initialized(torch.zeros) in the beginning of the forward function, so it contains the encoding values of probs_vec, then I give this sym_vec to the next dnn, it is initialized outside the loop of n_iter.</p>
<p>The thing here is that the encoding that I am doing is not a simple product or a sum I do it to a differentiable variable(prob_vec).</p>
<p>I think the only solution is to directly overwrite the values of prob_vec without initializing this new sym_vec, but still they don’t have same shape xD.</p>