Using Parallel Layers Makes My NN not Converge

randomnd_random: <p>I am tuning VGG16 network by adding multiple classifiers as the final classification layers. Based on a recommendation on this site, I created this class:</p>
<pre><code class="lang-auto">class Parallel(torch.nn.Module):
    def __init__(self, modules: List[torch.nn.Module]):
        super().__init__()
        self.modules = modules

    def forward(self, inputs):
        return [module(inputs) for module in self.modules]
</code></pre>
<p>And then I replaced the last layer in VGG16 using this command:</p>
<pre><code class="lang-auto">net = torchvision.models.vgg16_bn(pretrained=True)
device = 'cuda' if torch.cuda.is_available() else 'cpu'
for param in net.parameters():
    param.requires_grad = True
k = 6
net.classifier._modules[str(k)] = Parallel(
    [nn.Linear(4096, classes_num).to(device) for i in range(classifiers_num)]).to(device)
</code></pre>
<p>I noticed when <code>classifiers_num</code> is equal to 1, the training error never decreases. However, when I replace the last layer with <code>nn.Linear</code> without using my <code>Parallel</code> class, it does decrease:</p>
<pre><code class="lang-auto">net.classifier._modules[str(k)] = nn.Linear(4096, classes_num)

</code></pre>
<p>I am not sure what the different is. What am I missing?</p>

KFrank: <p>Hi Randomnd!</p>
<aside class="quote no-group" data-username="randomnd_random" data-post="1" data-topic="151725" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/randomnd_random/48/49631_2.png" class="avatar"> randomnd_random:</div>
<blockquote>
<pre><code class="lang-auto">    def forward(self, inputs):
        return [module(inputs) for module in self.modules]
</code></pre>
</blockquote>
</aside>
<p>This returns a <code>list</code> of whatever the <code>module</code>s return.</p>
<aside class="quote no-group">
<blockquote>
<p>And then I replaced the last layer in VGG16 using this command:</p>
<pre><code class="lang-auto">net.classifier._modules[str(k)] = Parallel(
    [nn.Linear(4096, classes_numeber).to(device) for i in range(classifiers_num)]).to(device)
</code></pre>
</blockquote>
</aside>
<p>So in this <code>Parallel</code> case <code>_modules[str(k)]</code> returns a <code>list</code> of the<br>
outputs of the <code>Linear</code>s, so presumably a <code>list</code> of tensors of shape<br>
<code>[nBatch, classes_numeber]</code>.</p>
<p>Note, even if <code>classifiers_num = 1</code>, you still return a <code>list</code> that contains<br>
a single tensor, not just the tensor, itself.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">net.classifier._modules[str(k)] = nn.Linear(4096, regions_num)
</code></pre>
</blockquote>
</aside>
<p>However, in this case without <code>Parallel</code>, <code>_modules[str(k)]</code> is simply a<br>
<code>Linear</code> and so returns a tensor of shape <code>[nBatch, classes_numeber]</code><br>
and does not return a <code>list</code>.</p>
<p>If your code that computes a loss from the output of <code>net</code> does <em>not</em> treat<br>
these two cases – <code>list</code> vs. tensor – differently, then you are likely not<br>
computing the loss correctly in (at least) one of the cases.</p>
<p>Best.</p>
<p>K. Frank</p>

randomnd_random: <p>Thanks for the reply. My code does treat the two cases but I avoided including them.</p>
<pre><code class="lang-auto">#loop over the `classifiers_num `
for k, outputs in enumerate(batch_outputs):
         loss = loss + criterion(outputs, labels_batched[:, k])
</code></pre>
<p>In case of only using a <code>Linear</code> without a list, then I would do this : <code> batch_outputs = [net(data_batched)]</code> and execute the piece of code above.</p>

KFrank: <p>Hi Randomnd!</p>
<aside class="quote no-group" data-username="randomnd_random" data-post="3" data-topic="151725" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/randomnd_random/48/49631_2.png" class="avatar"> randomnd_random:</div>
<blockquote>
<pre><code class="lang-auto">for k, outputs in enumerate(batch_outputs):
         loss = loss + criterion(outputs, labels_batched[:, k])
</code></pre>
<p>In case of only using a <code>Linear</code> without a list, then I would do this : <code> batch_outputs = [net(data_batched)]</code></p>
</blockquote>
</aside>
<p>Indexing <code>labels_batched</code> with <code>k</code> looks fishy to me. How are <code>data_batched</code><br>
and <code>labels_batched</code> supposed to line up with one another?  How is<br>
<code>labels_batched</code> supposed to know in advance how many elements you<br>
have in your <code>Parallel</code> and, therefore, what the length of <code>batch_outputs</code><br>
will be?</p>
<p>Irrespective of that detail, for further debugging I would start by setting up<br>
<code>net</code> with a single <code>Linear</code> as its last layer, and then again with a <code>Parallel</code><br>
containing a single <code>Linear</code> as its last layer, making sure that the two<br>
<code>Linear</code>s are initialized <em>identically</em>.  (Or maybe you could just reuse the<br>
same <code>Linear</code> for both cases.)  Then run the same input data (one batch)<br>
through the two versions of <code>net</code> and check that you get the same <code>loss</code>.<br>
If that works, try performing a single backward pass, and check that you<br>
get the same <code>grad</code> for the parameters of <code>net</code> and, in particular, the same<br>
grad for you last-layer <code>Linear</code>.</p>
<p>Best.</p>
<p>K. Frank</p>