BCE Loss Stalling for Multilabel, Multiclass Image Classification

Felix_Labelle: <p>I’m trying to predict a multilabel multiclass output from a series of image features (i.e. panoramic images passed through Resnet-152 giving a tensor of BATCH_SIZE X NUM_IMAGES X 36 (NUM_VIEWS) x 2048 (FEATURE_SIZE)). The output of the model is BATCH_SIZE X NUM_CLASSES (~3000). Note that there is a heavy imbalance in the classes. One series of images may have multiple objects in it, so I used BCE loss (with logits).</p>
<p>CV is not my specialty. Below is the model I’ve been using (which may be part of the problem). There is an assumption that one particular image contributes to the output vector, which is why Conv1D was used (I wasn’t sure how to process only one set of panoramic images at a time) and the outputs of each convolution (all of CLASS_SIZE) are max pooled together.</p>
<pre><code class="lang-auto">class LandMarkPredictionModule(nn.Module):
    def __init__(self, num_classes, num_views=36, kernel_size = 4):
        super(LandMarkPredictionModule, self).__init__()
        self.num_views = num_views
        self.kernel_size = kernel_size
        self.num_classes = num_classes
        self.conv = torch.nn.Conv1d(in_channels = self.num_views, out_channels= 1, kernel_size = self.kernel_size) # figure out
        self.fc = nn.Linear(in_features = 2045, out_features = self.num_classes) # How to calculate this value automatically
        self.act_func = nn.Sigmoid()

    def forward(self, images):
        batch_size, length, num_views, feat_size = images.shape
        reshaped_images = images.reshape(batch_size * length, num_views, feat_size)
        convolved_images = self.conv(reshaped_images).squeeze(1)
        output = self.act_func(self.fc(convolved_images))
        output = output.reshape(batch_size,length, self.num_classes)
        return output.max(dim=1)[0]
</code></pre>
<p>The loss stalls out at ~0.69, even in a case where I try to over fit to just 1 data point. Below is some example code and the data used.</p>
<pre><code class="lang-auto">import torch.optim as optim

pred_module = LandMarkPredictionModule(len(landmark_classes))
criterion = nn.BCEWithLogitsLoss()
optimizer = optim.Adam(pred_module.parameters(), lr=1e-3)
# USE BCE LOSS
# Manage class imbalances
NUM_EPOCHS = 100
tst_images = torch.from_numpy(np.stack((image_class_pairs[0][0],image_class_pairs[3][0]),axis=0))
tst_labels = torch.from_numpy(np.stack((image_class_pairs[0][1],image_class_pairs[3][1]),axis=0))

for epoch in range(NUM_EPOCHS):
    optimizer.zero_grad()

    logits = pred_module(tst_images)
    loss = criterion(logits,tst_labels)
    loss.backward()
    print(loss.item())
    optimizer.step()
</code></pre>
<p>The sizes of the inputs and outputs above are:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d671d8482e600a57f8cc595315669f738d9be96f.png" data-download-href="https://discuss.pytorch.org/uploads/default/d671d8482e600a57f8cc595315669f738d9be96f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d671d8482e600a57f8cc595315669f738d9be96f.png" alt="image" data-base62-sha1="uB45Aw9m43PQJJ3RemqY9EWFTkz" width="690" height="89" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1147×149 3.83 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m in the process of experimenting with the model, but beyond that what else could cause the loss to stay stuck (Data, Bug in the Code)?</p>

KFrank: <p>Hi Felix!</p>
<aside class="quote no-group quote-modified" data-username="Felix_Labelle" data-post="1" data-topic="98327" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/felix_labelle/48/11625_2.png" class="avatar"> Felix_Labelle:</div>
<blockquote>
<p>I used BCE loss (with logits).<br>
…</p>
<pre><code class="lang-auto">        self.fc = nn.Linear(in_features = 2045, out_features = self.num_classes) # How to calculate this value automatically
        self.act_func = nn.Sigmoid()
...
        output = self.act_func(self.fc(convolved_images))
        output = output.reshape(batch_size,length, self.num_classes)
...
criterion = nn.BCEWithLogitsLoss()
</code></pre>
</blockquote>
</aside>
<p>Because your are using <code>BCEWithLogitsLoss</code> (seemingly reasonable<br>
for your use case), you should get rid of the <code>Sigmoid</code> after your final<br>
<code>Linear</code> layer, <code>fc</code>.  (<code>Sigmoid</code> converts logits to probabilities, likely not<br>
what you want.)</p>
<p>Other than that, I don’t have much intuition about your use case, or<br>
what “NUM_IMAGES” (apparently <code>6</code>) might mean conceptually.</p>
<p>Having said that, the shenanigans where you <code>reshape</code> your data<br>
into a “pseudo-batch” of size <code>batch_size * length</code>, pass it through<br>
your network, and then <code>reshape</code> it back to a batch of <code>batch_size</code><br>
strike me as a little odd.</p>
<p>Best.</p>
<p>K. Frank</p>

Felix_Labelle: <blockquote>
<p>Because your are using <code>BCEWithLogitsLoss</code> (seemingly reasonable<br>
for your use case), you should get rid of the <code>Sigmoid</code> after your final<br>
<code>Linear</code> layer, <code>fc</code> . ( <code>Sigmoid</code> converts logits to probabilities, likely not<br>
what you want.)</p>
</blockquote>
<p>Thank you K. Frank!! I face palmed hard after reading this, I hadn’t considered the effects of that activation layer…</p>
<p>Changing that activation function to LeakyReLU helped, the loss now goes down to 0.05 after 100 epochs over 2 samples.</p>
<blockquote>
<p>Other than that, I don’t have much intuition about your use case, or<br>
what “NUM_IMAGES” (apparently <code>6</code> ) might mean conceptually.</p>
</blockquote>
<p>The network’s input is a tensor of image features representing panoramic views along a path (36 views separated by 30 degrees, with 12 views looking up, 12 looking forward, 12 looking down). NUM_IMAGES is the length of the path (could have been named better), in the example case there are 6 points. The goal of the network is to find which objects are present at a given time step.</p>
<p>The problem is that I don’t have a ground truth for the objects present at each point in the path, but I do know which objects appear in the path. The idea is that a model trained to predict objects present in the path, with an independence assumption between time steps aggregated using max pooling, could be used to classify single points afterwards.</p>
<blockquote>
<p>Having said that, the shenanigans where you <code>reshape</code> your data<br>
into a “pseudo-batch” of size <code>batch_size * length</code> , pass it through<br>
your network, and then <code>reshape</code> it back to a batch of <code>batch_size</code><br>
strike me as a little odd.</p>
</blockquote>
<p>To process each point in the path separately, I decided to use a 1d convolution, but that required reshaping the output. Rather than flatten it and lose some of the benefits of the current structure, I tried what you described above.</p>
<p>Originally the plan was to use a 2d convolution, but it seems to require iterating through the data to avoid using multiple points of the path at once.</p>
<p>Since the images do capture a panoramic view, I wanted to capture “interactions” between neighboring images. A fully connected layer could do the trick, but it would be big ~(36 x 2048 x HIDDEN_SIZE), hence the use of convolutions.</p>

KFrank: <p>Hi Felix!</p>
<aside class="quote no-group" data-username="Felix_Labelle" data-post="3" data-topic="98327" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/felix_labelle/48/11625_2.png" class="avatar"> Felix_Labelle:</div>
<blockquote>
<p>Changing that activation function to LeakyReLU helped, the loss now goes down to 0.05 after 100 epochs over 2 samples.</p>
</blockquote>
</aside>
<p>In the most common usage, you would have <em>no</em> activation between<br>
your last <code>Linear</code> layer and your <code>BCEWithLogitsLoss</code> loss criterion,<br>
neither a <code>Sigmoid</code> nor a <code>LeakyReLU</code>.</p>
<p>Also, on further inspection, I see that you do <em>not</em> have a nonlinear<br>
activation between your <code>Conv1D</code> and your <code>Linear</code> layer.  Again,<br>
in the most common usage, you would.  (<code>ReLU,</code>, <code>Sigmoid</code>, and<br>
<code>LeakyReLU</code> could all be reasonable choices.)</p>
<aside class="quote no-group">
<blockquote>
<p>The network’s input is a tensor of image features representing panoramic views along a path (36 views separated by 30 degrees, with 12 views looking up, 12 looking forward, 12 looking down). NUM_IMAGES is the length of the path.</p>
</blockquote>
</aside>
<p>Just thinking out loud here …</p>
<p>It seems to me that there is something inherently “sequential” about<br>
the 12 views in each of the three horizontal “slices”  that make up your<br>
panoramic “images” – hence a convolution could make sense – but<br>
that the 36 views together don’t make up a clean “sequence.”  Perhaps<br>
the “up,” “forward,” and “down” views should be “channels,” and not<br>
convolved over.  Or perhaps a vertical up-forward-down “slice” counts<br>
as a sequence, so maybe a <code>Conv2d</code> would make sense.</p>
<p>In a similar vein, if the 6 points in your paths are “close enough”<br>
together, perhaps they should be treated as a sequence, and<br>
convolved over.</p>
<p>So perhaps you should have a three-channel (up-forward-down)<br>
<code>Conv2d</code> that convolves over the six points in your paths and the<br>
twelve views in your horizontal panorama slices.  Or perhaps even<br>
a full <code>Conv3d</code> over all three.</p>
<p>Best.</p>
<p>K. Frank</p>

Felix_Labelle: <blockquote>
<p>In the most common usage, you would have <em>no</em> activation between<br>
your last <code>Linear</code> layer and your <code>BCEWithLogitsLoss</code> loss criterion,<br>
neither a <code>Sigmoid</code> nor a <code>LeakyReLU</code> .</p>
</blockquote>
<p>Makes sense, considering the shape of the sigmoid adding a ReLU after the linear layer would make my lowest possible value close to 0.5.<br>
For now, I’ve also tried with 3 FC layers and it outperforms the previous code (5-fold cross validation).</p>
<pre><code class="lang-auto">        self.input_layer = nn.Sequential(nn.Dropout(self.dropout_rate),
                                         nn.Linear(2048,hidden_sizes[0]),
                                         nn.LeakyReLU(),
                                         nn.Flatten(start_dim=-2),
                                         nn.Linear(hidden_sizes[0] * 36, hidden_sizes[1]),
                                         nn.LeakyReLU(),
                                         nn.Linear(hidden_sizes[1], num_classes)
                                        )
</code></pre>
<blockquote>
<p>In a similar vein, if the 6 points in your paths are “close enough”<br>
together, perhaps they should be treated as a sequence, and<br>
convolved over.</p>
</blockquote>
<p>The points are physically rather far and may contain dissimilar information. Think navigating in street view on google maps, but with obstructions between the navigable points. Moreover, wouldn’t that output a smaller length sequence at the output? The goal is to get a probability distribution at each time step at eval time (it’s for a down stream task).</p>
<blockquote>
<p>So perhaps you should have a three-channel (up-forward-down)<br>
<code>Conv2d</code> that convolves over the six points in your paths and the<br>
twelve views in your horizontal panorama slices. Or perhaps even<br>
a full <code>Conv3d</code> over all three.</p>
</blockquote>
<p>Good point! I don’t think that the points are arranged like that. From what I remember the first 12 views are bottom, next 12 are middle, last 12 are top, I’ll check.  With a stride of 12 it should work. It’s getting late, but I’ll try that tomorrow and see how it performs.</p>