Two branches architecture, and back-propagate just one branch loss

username: algeriapy
cooked: <p>Hi Dears,</p>
<p>I created the following architecture which has one main branch, but in the decision layer it has two braches where resnet FC layer is shared:</p>
<pre><code class="lang-auto">class Net1(nn.Module):

    def __init__(self, in_features, out_features):
        super(Net1, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.weight = Parameter(torch.Tensor(out_features, in_features))

    def forward(self, input, label):
        out = torch.mm(input,  self.weight.t())
        return out

classifier =  net1(1024, 624)  

class MyRes(nn.Module):
    def __init__(self, in_features = 1024, Num_classes = 624):
        super(MyRes, self).__init__()
        self.in_features = in_features
        self.res_model =  torchvision.models.resnet50(pretrained=True) 
        self.res_model.fc = nn.Sequential(
            nn.BatchNorm1d(2048),
            nn.Dropout(p=0.4),
            nn.Linear(2048, in_features),
            nn.BatchNorm1d(in_features),  
        )
        self.classifier = classifier
        self.res_model.classifier = nn.Linear(in_features, Num_classes)
        
    def forward(self, x, labels):        
        x = self.res_model(x)
        l = self.classifier(x, labels)            
        x = self.res_model.classifier(x)
           
        return x, l
</code></pre>
<p>In the training, I calculated the loss for just the first branch then back-propagated  it</p>
<pre><code class="lang-auto">model =  MyRes()
optimizer = optim.Adam(model.parameters(), lr = lr)

model.train()
pred, preds2 = model(images)

loss = criterion(preds, labels)        
optimizer.zero_grad()
loss.backward()
optimizer.step() 

</code></pre>
<p>I am wondering what happens at the main branch, especially at the connection point of the two branches !!.<br>
Is there will be two gradient (from self.classifier and self.res_model.classifier ), In back propagation they summed !!!<br>
What happens to the weights  in (self.weight) from Net1, Are they will be learned with the network training or stay fixed as they initialized !!</p>
<p>Best Regards</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="algeriapy" data-post="1" data-topic="139072">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/34f0e0/48.png" class="avatar"> algeriapy:</div>
<blockquote>
<p>but in the decision layer it has two braches where resnet FC layer is shared:</p>
</blockquote>
</aside>
<p>Your current architecture creates two new classifiers and assigns one to <code>self.classifier</code> and the other one to a new attribute called <code>self.res_model.classifier</code>. Note that you are not replacing the last linear layer of the resnet in the latter case, as it’s assigned to <code>self.res_model.fc</code> (and already replaced with the <code>nn.Sequential</code> module). This isn’t necessarily wrong, but you should be aware that the forward pass of <code>self.res_model</code> will not call <code>self.res_model.classifier</code>.</p>
<aside class="quote no-group" data-username="algeriapy" data-post="1" data-topic="139072">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/34f0e0/48.png" class="avatar"> algeriapy:</div>
<blockquote>
<p><code>self.weight = Parameter(torch.Tensor(out_features, in_features))</code></p>
</blockquote>
</aside>
<p>This will use uninitialized memory, which is most likely not what you want.<br>
Don’t use the <code>torch.Tensor</code> constructor, but the factory methods instead (<code>torch.randn</code>, <code>torch.ones</code>, etc.).</p>
<aside class="quote no-group" data-username="algeriapy" data-post="1" data-topic="139072">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/34f0e0/48.png" class="avatar"> algeriapy:</div>
<blockquote>
<p>Is there will be two gradient (from self.classifier and self.res_model.classifier ), In back propagation they summed !!!</p>
</blockquote>
</aside>
<p>No, since you are calculating the <code>loss</code> from one output only (<code>preds</code> in the posted code snippet, which is undefined, so I don’t know which output you are using). If you are calculating the loss from both outputs, both computation paths will be used to calculate the gradients of the parameters in the main branch.</p>
<aside class="quote no-group" data-username="algeriapy" data-post="1" data-topic="139072">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/34f0e0/48.png" class="avatar"> algeriapy:</div>
<blockquote>
<p>What happens to the weights in (self.weight) from Net1, Are they will be learned with the network training or stay fixed as they initialized !!</p>
</blockquote>
</aside>
<p>If you are using the model’s output to calculate the loss (again unclear based on your posted code), then it will be updated unless you freeze the parameters by setting their <code>.requires_grad</code> attribute to <code>False</code>.</p>

username: algeriapy
cooked: <p>Dear ptrblck,</p>
<p>Thanks for your reply,</p>
<p>Actually you are right the shared point is the output of (the replacement of <code>self.res_model.fc</code>)  <code>nn.Sequential</code> model (which is 1024 )</p>
<p>The idea here, I am using CosFace  <code>self.classifier</code> along side with another FC layer for classification,</p>
<p>In more details,<br>
In the training phase,</p>
<ol>
<li>Foreword pass: the two branches are used   <code>self.classifier</code> (Cosface) and <code>self.res_model.classifier</code> (the classification output), but the loss is calculated from just the classification branch,</li>
<li>back-propagation, the loss calculated from the classification output is used to back propagate, (no use of the cosface)</li>
</ol>
<p>In the testing phase:<br>
only the classification branch is active (the cosface was not used at all)</p>
<p>My main question, since I am not using the CosFace branch to calculate the loss and back propagate it, how it influence my training (recall the results are very different from when I calculate the loss from the CosFace and back propagate it, also when I dont use the CosFace at all)</p>
<p>Best Regards</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="algeriapy" data-post="3" data-topic="139072">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/34f0e0/48.png" class="avatar"> algeriapy:</div>
<blockquote>
<p>My main question, since I am not using the CosFace branch to calculate the loss and back propagate it, how it influence my training</p>
</blockquote>
</aside>
<p>It shouldn’t influence the training at all, if you are just calling its <code>forward</code> without using any output.</p>

username: algeriapy
cooked: <p>Thanks a lot,</p>
<p>I found adding FC layers to resnet improved my results a lot<br>
So the cosface was not used  at all</p>
<p>Best Regards</p>