Variable parts of hidden layers in a network

cherepanovic: <p>Hello community,</p>
<p>my question is about a variable output or some parts of a net which can vary. For example a flag would direct which output (or some part of a net) is to choose. That means that I have different hidden layers and with means of a flag it will be decided which is to take. Is there some examples available, could you provide someone.</p>
<p>here a small illustration:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/fae1f0688b4246a04d6eb0b699d330df694ae321.png" data-download-href="https://discuss.pytorch.org/uploads/default/fae1f0688b4246a04d6eb0b699d330df694ae321" title="bla"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fae1f0688b4246a04d6eb0b699d330df694ae321_2_539x500.png" alt="bla" data-base62-sha1="zNprgWoOcee4hU8AIKT7spf1Lpf" width="539" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fae1f0688b4246a04d6eb0b699d330df694ae321_2_539x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/fae1f0688b4246a04d6eb0b699d330df694ae321_2_808x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/fae1f0688b4246a04d6eb0b699d330df694ae321.png 2x" data-dominant-color="F9FBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">bla</span><span class="informations">828×768 59.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>best regards!</p>

ptrblck: <p>You can just pass a flag into your model’s <code>forward</code> to chose a certain path:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.path1 = nn.Linear(5, 10)
        self.path2 = nn.Linear(10, 10)
        
    def forward(self, x, path):
        if path == 'path1':
            x = self.path1(x)
        elif path == 'path2':
            x = self.path2(x)
        else:
            print('unknown path')
        return x

model = MyModel()
x1 = torch.randn(1, 5)
output1 = model(x1, 'path1')
x2 = torch.randn(1, 10)
output2 = model(x2, 'path2')
</code></pre>

cherepanovic: <p>Thank you for your answer!</p>
<p>One question more, how does learning work (the backpropagation?)</p>

iArunava: <h3>In very brief (for supervised settting)</h3>
<ol>
<li>You do a forward pass, after all the calculations, the final linear layer outputs a vector [1, nc], where <code>nc</code> is the number of classes.</li>
<li>This gets compared to the true classes, and we get a value of <code>loss</code>
</li>
<li>This loss gets backpropagated.</li>
<li>Every parameter in the model gets updated with respect to this loss.</li>
<li>You have a better model! Hurray!</li>
</ol>
<h3>Let’s revise, the major the magic word backpropagation</h3>
<p>This is an algorithm where each model parameter is compared to the <code>loss</code>.<br>
And we calculate a <code>gradient</code> which is basically a number which has <strong>direction</strong> and <strong>magnitude</strong><br>
Now, this direction is indicated by <code>torch.sign</code> of this value, (i.e. whether positive or negative)<br>
and moving the model parameter in this direction will increase the value of loss by a factor of <code>magnitude</code>.<br>
So, we move in the opposite direction (i.e. negate the <code>gradient</code>) and thus decrease the <code>loss</code> <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
<p>That was not very brief, but I think it’s clear.</p>
<p>Also read this: <a href="https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b" rel="nofollow noopener">https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b</a></p>

cherepanovic: <p>I didn’t ask what the backpropagation is. Your post does not fit to my question.</p>

iArunava: <blockquote>
<p>how does learning work (the backpropagation?)</p>
</blockquote>
<p>My bad! Seeing the question mark after backprop. I derived that’s your question.</p>

ptrblck: <p>Since the computation graph will be created dynamically during the forward pass, only the parameters will be updated (and get a valid gradient) which were used to calculate the loss.<br>
I.e. if you only use <code>path1</code> during training, only <code>self.path1</code> will be upgraded, while <code>self.path2</code> keeps its initial values.</p>