How to calculate the gradient when using fixed-weight filters in multiple branches?

Miffy: <p>Recently, I have been implementing a paper at <a href="https://www.sciencedirect.com/science/article/pii/S0893608019301315" rel="noopener nofollow ugc">https://www.sciencedirect.com/science/article/pii/S0893608019301315</a>.<br>
The paper describes a branched CNN architecture in which the authors use fixed-weight filters after the general convolution layer to obtain additional feature maps.<br>
They mention in the paper that during backpropagation, the gradients of these feature maps (including the original feature map) are in an accumulative state.</p>
<p>I have created a simple architecture diagram as follows. It’s not a very complex structure, but I am puzzled about how backpropagation works in this context.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d3e1a14d160d082fd5bb373b489918e82bd6f8d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5d3e1a14d160d082fd5bb373b489918e82bd6f8d" title="nnn"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d3e1a14d160d082fd5bb373b489918e82bd6f8d_2_690x105.jpeg" alt="nnn" data-base62-sha1="diRsmj7ScTT0tiI69q8IDkjf8yV" width="690" height="105" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d3e1a14d160d082fd5bb373b489918e82bd6f8d_2_690x105.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d3e1a14d160d082fd5bb373b489918e82bd6f8d.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d3e1a14d160d082fd5bb373b489918e82bd6f8d.jpeg 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nnn</span><span class="informations">716×109 10.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
ps.“Cat” means concatenate!</p>
<p>I know that Pytorch can automatically compute gradients and perform backpropagation by using .backward(), but do I need to add any specific instructions, or will Pytorch automatically handle the accumulative part for the original feature maps and additional feature maps for me?</p>
<p>Thank you very much for your responses!</p>

ptrblck: <aside class="quote no-group" data-username="Miffy" data-post="1" data-topic="189377">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/miffy/48/54884_2.png" class="avatar"> Miffy:</div>
<blockquote>
<p>They mention in the paper that during backpropagation, the gradients of these feature maps (including the original feature map) are in an accumulative state.</p>
</blockquote>
</aside>
<p>I don’t fully understand this statement since feature maps (assuming these refer to forward activations) are created for each new input in the forward pass and will thus not accumulate gradients. Trainable parameters will accumulate gradients in their <code>.grad</code> attribute, but you’ve also mentioned that the <code>weight</code> is fixed, so I assume it’s not trainable.</p>

Miffy: <p>Dear ptrblck,</p>
<p>I think that during backpropagation, the original feature maps are generated by the convolution layer, so the architecture I mentioned above can only update the convolution layer’s filter weights and the three fc layers during backpropagation.</p>
<p>I think there was something wrong with my previous statement, and I’d like to clarify by referring to the original text of the paper.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/5398e3f2f5ef93aec0c863ef6d4f60704afe6d8d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5398e3f2f5ef93aec0c863ef6d4f60704afe6d8d" title="err"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/5398e3f2f5ef93aec0c863ef6d4f60704afe6d8d_2_623x500.jpeg" alt="err" data-base62-sha1="bVxcatFvteefzSYwCq41WO5MeQJ" width="623" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/5398e3f2f5ef93aec0c863ef6d4f60704afe6d8d_2_623x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/5398e3f2f5ef93aec0c863ef6d4f60704afe6d8d_2_934x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/3/5398e3f2f5ef93aec0c863ef6d4f60704afe6d8d.jpeg 2x" data-dominant-color="EDEEED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">err</span><span class="informations">977×783 136 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I think the original text shows that the error which is propagated to the original feature maps, includes the error from the additional feature maps, and the error is equal to the gradient, so the authors use the title <code>Accumulative back-propagation</code>.<br>
That is my understanding, but I’m not sure if it is correct.</p>
<p>And, if my understanding is correct, will Pytorch automatically do the “accumulative back-propagation” for me or I should do it by myself?</p>