How two decoders share same encoder features? Do I need to use detach()?

username: John1231983
cooked: <p>I have a network that includes one encoder and two decoders. The two decoders share the same encoder. The <a href="https://arxiv.org/abs/1804.00521" rel="noopener nofollow ugc">paper</a> said</p>
<blockquote>
<p>SO and CO branches - We take the downsampling branch of a U-Net as it is, however we split the upsampling branch into two halves, one to obtain the Region of Interest and the other for Complementary aka non region of interest. Losses here are negative dice for ROI and positive dice for Non-ROI region.</p>
</blockquote>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/48fc1f14f9d308ad5fc1a6d56e79fa00b0e59554.png" data-download-href="https://discuss.pytorch.org/uploads/default/48fc1f14f9d308ad5fc1a6d56e79fa00b0e59554" title="Screenshot%20from%202019-01-02%2010-05-25"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/48fc1f14f9d308ad5fc1a6d56e79fa00b0e59554_2_504x499.png" alt="Screenshot%20from%202019-01-02%2010-05-25" data-base62-sha1="apExOWm6ZN2Y1jk66aI0ucvwKfa" width="504" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/48fc1f14f9d308ad5fc1a6d56e79fa00b0e59554_2_504x499.png, https://discuss.pytorch.org/uploads/default/original/2X/4/48fc1f14f9d308ad5fc1a6d56e79fa00b0e59554.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/4/48fc1f14f9d308ad5fc1a6d56e79fa00b0e59554.png 2x" data-dominant-color="E7E8E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-01-02%2010-05-25</span><span class="informations">673×667 99.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Assume that the encoder provides the feature likes</p>
<p><code>encoder_feature =[ f1, f2, f3, f4, f5]</code></p>
<p>So, both <code>decoder_SO</code> and <code>decoder_CO</code> use encoder features to update the weights. Which one is correct?</p>
<pre><code class="lang-auto">decoder_SO = [ up_sample(f1), up_sample(f2), up_sample(f3), up_sample(f4), up_sample(f5) ]
decoder_CO = [ up_sample(f1), up_sample(f2), up_sample(f3), up_sample(f4), up_sample(f5) ]
</code></pre>
<p>Or using <code>detach()</code></p>
<pre><code class="lang-auto">decoder_SO = [ up_sample(f1), up_sample(f2), up_sample(f3), up_sample(f4), up_sample(f5) ]
decoder_CO = [ up_sample(f1.detach()), up_sample(f2.detach()), up_sample(f3.detach()), up_sample(f4.detach()), up_sample(f5.detach()) ]
</code></pre>

username: justusschock
cooked: <p>If you want to train the encoder together with both decoders the first variant is the one to go with.</p>
<p>Version 2 trains the encoder together with decoder_SO and the other decoder will be trained separately.</p>

username: John1231983
cooked: <aside class="quote no-group" data-username="justusschock" data-post="2" data-topic="33601">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/justusschock/48/4637_2.png" class="avatar"> justusschock:</div>
<blockquote>
<p>2 trains the encoder together with decoder_SO and the other decoder will be trained separately.</p>
</blockquote>
</aside>
<p>If I want to train encoder using both information from decoder_SO and decoder_CO. So I need to use option 1, Is it right? I am not clear about your answer in second case</p>

username: justusschock
cooked: <p>Yes, you have to use Version 1.</p>
<p>The second version would detach the decoders input from the encoders computational graph. This means, that no gradients could be propagated back through the encoder and thus the encoder would only be trained with one decoder.</p>

username: meshghi
cooked: <p>Hi, I’m trying to implement a similar architecture. I’m confused about what the loss function would look like? Any help/suggestion/advice would be greatly appreciated</p>