Add multiple FC layers in parallel

kl_divergence: <p>I am using pretrained ResNet 50, I am performing finetuning, My code is as follows:</p>
<pre><code class="lang-auto">class ResNet50(nn.Module):
    def __init__(self,num_classes,loss={'xent'},**kwargs):
        super(ResNet50,self).__init__()
        self.loss = loss
        resnet50 = torchvision.models.resnet50(pretrained=True)
        num_ftrs = resnet50.fc.in_features
        self.base= nn.Sequential(*list(resnet50.children())[:-2])
        self.fc1 = nn.Linear(2048,num_classes)
        self.fc2 = nn.Linear(2048, num_classes)


    def forward(self,x):
        x = self.base(x)
        x = F.avg_pool2d(x,x.size()[2:])
        f = x.view(x.size(0),-1)
        if not self.training:
            return f
        y = self.classifier(f)

</code></pre>
<p>I want to add multiple fc layers in parallel after ResNet</p>
<ol>
<li>How can I add multiple FC layers in parallel ?,</li>
</ol>

justusschock: <p>To answer your first question I removed the unneccessary code (loss etc) to get a minimum example. You could simply calculate the fc outputs one after the other (as multiprocessing on GPUs can be hard to implement) and later on return a dictionary (or a list or whatever you want).</p>
<pre><code class="lang-auto">class ResNet50(nn.Module):
    def __init__(self,num_classes,num_fcs=3, loss={'xent'},**kwargs):
        super(ResNet50,self).__init__()
        self.loss = loss
        resnet50 = torchvision.models.resnet50(pretrained=True)
        self.base= nn.Sequential(*list(resnet50.children())[:-2])
        self.num_fcs = num_fcs
        for i in range(num_fcs):
            setattr(self, "fc%d" % i, nn.Linear(2048, num_classes))


    def forward(self,x):
        x = self.base(x)
        x = F.avg_pool2d(x,x.size()[2:])
        f = x.view(x.size(0),-1)

        clf_outputs = {}
        for i in range(self.num_fcs):
            clf_outputs["fc%d" % i] = getattr(self, "fc%d" % i)(f)

        return clf_outputs

</code></pre>
<p>For your second question you could simply try it like this:</p>
<pre><code class="lang-auto"># create class instance
model = ResNet50(5)

# freeze layer (can be done with every fc layer)
model.fc0.train(False)
</code></pre>

kl_divergence: <p>What does 5 signify here in ResNet50(5) ?</p>

justusschock: <p>If you look at the class definition there has to be one positional argument specifying the number of classes (5 was just an arbitrary value to create an instance).</p>

kl_divergence: <p>Thanks a lot! How do i manage y in this case (from my_code)  ?</p>

justusschock: <p>I don’t know, what y is supposed to be. In your code is no definition of <code>self.classifier</code>.</p>
<p>If you defined it somewhere else you could simply return it together with the dictionary</p>

kl_divergence: <p>self.classifier = nn.Linear(2048,num_classes) . Missed it</p>

justusschock: <pre><code class="lang-auto">def forward(self,x):
        x = self.base(x)
        x = F.avg_pool2d(x,x.size()[2:])
        f = x.view(x.size(0),-1)

        clf_outputs = {}
        for i in range(self.num_fcs):
            clf_outputs["fc%d" % i] = getattr(self, "fc%d" % i)(f)

        clf_outputs["y"] = self.classifier(f)

        return clf_outputs
</code></pre>

kl_divergence: <p>The thing is I’ll freeze one fc layer, and then train the model with second fc layer. But in second phase, I’ll train the model with both fc layers (unfreeze) on different dataset, so how will i compute loss pertaining to two fc layers unlike the code which I provided.</p>

justusschock: <p>you could simply calculate separate losses for each of the output and add them up (or calculate the mean) and then call backward on the total loss</p>

kl_divergence: <p>I have to remove self.classifier because now i have multiple fc layers</p>

kl_divergence: <p>When I’ll join train (fc0 and fc1 turned on), how will i manage loss then ?</p>

justusschock: <p>calculate a mean/sum of both separate losses and call backward() on it. Autograd should handle the rest for you since only the parameters which are involved in the forward pass will be updated for each part of the loss.</p>

kl_divergence: <p>You mean this ?</p>
<pre><code class="lang-auto">def forward(self,x):
        x = self.base(x)
        x = F.avg_pool2d(x,x.size()[2:])
        f = x.view(x.size(0),-1)
        clf_outputs = {}
        for i in range(self.num_fcs):
            clf_outputs["fc%d" %i] = getattr(self, "fc%d" %i)(f)

        clf_outputs["y1"] = self.fc0(f)
        clf_outputs=["y2"] = self.fc1(f)
        l = (y1+y2)/2

        if self.loss == {'xent'}:
            return l
        elif self.loss == {'xent','htri'}:
            return l,f
        elif self.loss == {'cent'}:
            return l,f
        else:
            raise KeyError("Unsupported loss:{}".format(self.loss))
        l.backward()
        return clf_outputs
</code></pre>

justusschock: <p>First of all:</p>
<aside class="quote no-group" data-username="kl_divergence" data-post="14" data-topic="19008">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kl_divergence/48/5069_2.png" class="avatar"> kl_divergence:</div>
<blockquote>
<pre><code class="lang-auto">clf_outputs["y1"] = self.fc0(f) 
clf_outputs=["y2"] = self.fc1(f)
</code></pre>
</blockquote>
</aside>
<p>you don’t need this part as</p>
<aside class="quote no-group" data-username="kl_divergence" data-post="14" data-topic="19008">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kl_divergence/48/5069_2.png" class="avatar"> kl_divergence:</div>
<blockquote>
<pre><code class="lang-auto">for i in range(self.num_fcs): 
    clf_outputs["fc%d" %i] = getattr(self, "fc%d" %i)(f)
</code></pre>
</blockquote>
</aside>
<p>also stores the same results in <code>clf_outputs["fc0"]</code> and <code>clf_outputs["fc1"]</code> instead of <code>clf_outputs["y1"]</code> and <code>clf_outputs["y2"]</code></p>
<p>Second: thats not what I meant. you simply sould use your forward function like</p>
<pre><code class="lang-auto">def forward(self,x):
        x = self.base(x)
        x = F.avg_pool2d(x,x.size()[2:])
        f = x.view(x.size(0),-1)
        clf_outputs = {}
        for i in range(self.num_fcs):
            clf_outputs["fc%d" %i] = getattr(self, "fc%d" %i)(f)

        if self.loss == {'xent'}:
            return clf_outputs
        elif self.loss == {'xent','htri'}:
            return clf_outputs,f
        elif self.loss == {'cent'}:
            return clf_outputs,f
        else:
            raise KeyError("Unsupported loss:{}".format(self.loss))

</code></pre>
<p>after you created a model instance with <code>model = ResNet50(10)</code> you cann do some predictions with <code>clf_outputs, f = model(data_tensor)</code> and later on calculate a loss for each of the values in <code>clf_outputs</code>:</p>
<p>In the following snippet I assume you have a list of targets (one per fc layer)!</p>
<pre><code class="lang-auto">model = ResNet(10, num_fcs=2)
optim = SGD(model.parameters())
clf_outputs, f = model(data_tensor)

# for simplicity I'm using MSE-Loss but you could simply use any other loss function as well
loss_fn = MSELoss()
loss_value = 0
for k, v in clf_outputs.items():
    _curr_loss = loss_fn(v, target_list[int(k.replace("fc", ""))])
    loss_val = loss_val + _curr_loss

optim.zero_grad()
loss_value.backward()
optim.step()
</code></pre>

kl_divergence: <p>When I freeze FC layer 2</p>
<pre><code class="lang-auto">resnet50.fc1.train(False)
</code></pre>
<p>, it provides me with this:</p>
<pre><code class="lang-auto">AttributeError: 'ResNet' object has no attribute 'fc1'
</code></pre>

justusschock: <p>how did you create your model?</p>

kl_divergence: <p>I am using parse arguments</p>
<pre><code class="lang-auto">parser.add_argument('-a', '--arch', type=str, default='resnet50', choices=models.get_names())
</code></pre>
<p>Then this is the relevant part of train function</p>
<pre><code class="lang-auto">def train(epoch, model, criterion_xent, criterion_htri, optimizer, trainloader, use_gpu):
    losses = AverageMeter()
    batch_time = AverageMeter()
    data_time = AverageMeter()

    model.train()

    for batch_idx, (imgs, pids, _) in enumerate(trainloader):
        if use_gpu:
            imgs, pids = imgs.cuda(), pids.cuda()
        
        outputs, features = model(imgs)
        if args.htri_only:
            if isinstance(features, tuple):
                loss = DeepSupervision(criterion_htri, features, pids)
            else:
                loss = criterion_htri(features, pids)
        else:
            if isinstance(outputs, tuple):
                xent_loss = DeepSupervision(criterion_xent, outputs, pids)
            else:
                xent_loss = criterion_xent(outputs, pids)
                  
            
         loss = xent_loss + htri_loss
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>Then i am using this</p>
<pre><code class="lang-auto">model = models.init_model(name=args.arch, num_classes=dataset.num_train_pids, loss={'xent', 'htri'})
</code></pre>

justusschock: <p>and what does your <code>init_model</code> function look like?</p>

kl_divergence: <pre><code class="lang-auto">def init_model(name, *args, **kwargs):
    if name not in __factory.keys():
        raise KeyError("Unknown model: {}".format(name))
    return __factory[name](*args, **kwargs)
</code></pre>