One layer network fails at a binary classification task

username: tanitz
cooked: <p>Hello,<br>
I have been trying to train a one layer network on features extracted with another network pre-trained.<br>
The same features are discriminative when I train linear non-DL classifiers such as a Linear SVM or a logistic regression. However for some reason, when I try to train a neural network on these same features, the loss never gets close enough to zero and at some point starts to stagnate. I have tried everything I could think of: I normalized the features before passing them to the network, I increased/decreased the learning rate…decreased the batch size…<br>
Initially this neural network was intended to be trained on a dataset (that is then passed to the same pre-trained feature extractor) way bigger (2 million samples) than the one I used for the non-DL classifiers (13 000 samples),  but I’m trying it now on even the same dataset used for the non-DL classifiers and it’s still not working. I am really puzzled and would appreciate some help.<br>
Here are relevant parts of the code:</p>
<p>the one layer neural network</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, embedding_size=512):
        super(Net, self).__init__()

        self.embedding_size= embedding_size 
        self.fc=nn.Linear(self.embedding_size,1)
    
    def forward(self,x):
        x = self.fc(x)
        return x
</code></pre>
<p>optimizer and loss</p>
<pre><code class="lang-auto">advBackbone=Net(embedding_size=cfg.embedding_size).to(device)
opt_adv_backbone = torch.optim.Adam(
        params=[{'params': advBackbone.parameters()}],
        lr=0.01,
        )
advcriterion = BCEWithLogitsLoss()
</code></pre>
<p>training loop</p>
<pre><code class="lang-auto">for epoch in range(start_epoch, cfg.num_epoch):
        for i, (img, _, g) in enumerate(train_loader):
            for p in backbone.parameters(): # fixing the pre-trained feature extractor
                p.requires_grad = False

            global_step += 1
            opt_adv_backbone.zero_grad()
            img = img.to(device)
            g = g.to(device)
            features = F.normalize(backbone(img))
            output = advBackbone(features.detach())
            loss_a=advcriterion(output, g.unsqueeze(1).float())
            loss_a.backward()
            opt_adv_backbone.step()
            loss.update(loss_a, 1)
</code></pre>
<p>by the way, when I try to let it overfit on one batch, the loss does get close to zero (~0.005). When I try one the whole data, it decreases and then from a certain epoch stagnates around a certain value (0.2) for the small data and (0.3) for the large dataset. What I don’t understand, is how come it fails when a logistic regression or a linear SVM does converge. The fact that these do means that the features, at least extracted for the small data are indeed linearly separable and the neural network defined should be able to converge as well…</p>
<p>Thanks in advance for anyone willing to help</p>

username: KFrank
cooked: <p>Hi Tanitz!</p>
<aside class="quote no-group quote-modified" data-username="tanitz" data-post="1" data-topic="151406" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tanitz/48/49400_2.png" class="avatar"> tanitz:</div>
<blockquote>
<p>I have been trying to train a one layer network on features extracted with another network pre-trained.<br>
The same features are discriminative when I train linear non-DL classifiers such as a Linear SVM or a logistic regression. However for some reason, when I try to train a neural network on these same features, the loss never gets close enough to zero and at some point starts to stagnate.<br>
…<br>
I am really puzzled and would appreciate some help.<br>
…<br>
the one layer neural network</p>
<pre><code class="lang-auto">        self.fc=nn.Linear(self.embedding_size,1)
</code></pre>
</blockquote>
</aside>
<p>I agree with your assessment that you should be able to successfully<br>
train a single <code>Linear</code> on this task.  The point is that training the single<br>
<code>Linear</code> is fully equivalent to training a linear support-vector machine<br>
(SVM), albeit with a non-standard (and presumably somewhat less<br>
efficient) optimization algorithm.</p>
<p>Some comments:</p>
<p>If your training samples are indeed linearly separable then the prediction<br>
accuracy of your trained linear SVM should be 100%.  Do you achieve<br>
perfect accuracy with your SVM?</p>
<p>I do think you should be able to train your <code>Linear</code> network down to a loss<br>
close to zero and a loss of <code>0.2</code> or <code>0.3</code> does seem too high.  But these loss<br>
values can be a little hard to interpret.</p>
<p>What prediction <em>accuracy</em> do you get from your trained <code>Linear</code>?  By this I<br>
mean threshold the predictions – as directly output by the <code>Linear</code> – against<br>
zero.  Negative means your <code>Linear</code> is predicting class-0 (or the “negative”<br>
class) and positive means class-1 (or the “positive” class).  Then calculate<br>
your accuracy as the percentage of these predictions that are correct.</p>
<p>If things were working, you could well have 100% accuracy without having<br>
the loss be zero (but I would nonetheless expect the loss to be much smaller<br>
than, say, 0.2).</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">opt_adv_backbone = torch.optim.Adam(
        params=[{'params': advBackbone.parameters()}],
        lr=0.01,
        )
advcriterion = BCEWithLogitsLoss()
</code></pre>
</blockquote>
</aside>
<p>The <code>Adam</code> optimizer has the reputation of typically training significantly<br>
faster, but sometimes being more susceptible to getting stuck.</p>
<p>Just to make things simpler and easier to understand, you might try<br>
optimizing with <code>SGD</code>.  I would suggest not using weight decay, or at<br>
least using only a very mild weight decay.  Using momentum would<br>
probably make sense, but I would suggest at least carrying out a<br>
baseline training run without momentum.</p>
<p>If things still aren’t working, I would recommend distilling your problem<br>
down to a pure numerical linearly-separable classification problem.</p>
<p>Preprocess your data by running it through <code>backbone()</code> (whatever that<br>
is) and store the resulting features as a “new” dataset together with the<br>
labels (which, I imagine, do not need to be preprocessed).  Then train<br>
a SVM and a <code>Linear</code> on this distilled dataset.</p>
<p>The point would be to get rid of any extraneous code – <code>train_loader</code>,<br>
<code>backbone</code>, etc. – that might have some bugs hiding in it.  Maybe that<br>
fixes something that’s been easy to overlook.</p>
<p>Lastly, if your SVM and <code>Linear</code> still don’t agree, extract the discriminative<br>
hyperplane from your (working) SVM.  This will be the vector in “feature<br>
space” (of length equal to the number of features which I presume is<br>
<code>embedding_size</code>) that is orthogonal to the discriminative hyperplane<br>
plus the single scalar that is the offset of the hyperplane from the origin.<br>
These map to the <code>Linear</code>’s <code>weight</code> matrix (of shape <code>[1, embedding_size]</code>)<br>
and the <code>Linear</code>’s <code>bias</code> vector (of shape <code>[1]</code>, so a scalar).  (There may be<br>
a minus sign in either the <code>weight</code> or the <code>bias</code> mapping, depending on<br>
how things are defined inside of your SVM.)</p>
<p>Then use the extracted SVM parameters to <em>initialize</em> your <code>Linear</code><br>
network and verify that, when so initialized, your <code>Linear</code> network<br>
makes the same (thresholded) predictions as your SVM.  Assuming<br>
this works, try training your <code>Linear</code>, just to make sure that your training<br>
procedure isn’t broken and somehow messing up your <code>Linear</code>.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: tanitz
cooked: <p>Hi KFrank!</p>
<p>Thank you for your suggestions. I have said before that the features are linearly separable but actually they aren’t “perfectly” linearly separable. The linear SVM does find a hyperplane that classifies most of the points correctly but not all of them (over 90% accuracy, and with an area under the ROC curve over 90% as well…). I do expect the linear network to converge to a similar solution… I will calculate the accuracy at the stagnating lowest loss point and let you know…</p>