Feature extraction Using Pretrained Models

kareem: <p>So, I want to use the pretrained models to feature extract features from images, so I used “resnet50 , incepton_v3, Xception, inception_resnet” models, removed the classifier or FC depends on the model architecture, as some models have model.fc and other have model.classifier and other have model.classi , then I concatenated the features and trained the concatenated features in a deep neural network, but I got validation accuracy 1% only, although when I did the same network in Tensorflow I got accuracy of 95%, so can someone clarify what I did wrong in my pytorch notebook?</p>
<p>model_resnet = models.resnet50(pretrained=True).cuda()<br>
model_resnet.fc=nn.Sequential()</p>
<p>model_incep = models.inception_v3 (pretrained=True).cuda()<br>
model_incep.fc=nn.Sequential()</p>
<p>model_xcep = timm.create_model(‘xception’, pretrained=True).cuda()<br>
model_xcep.fc=nn.Sequential()</p>
<p>model_incep_res = timm.create_model(‘ens_adv_inception_resnet_v2’, pretrained=True).cuda()<br>
model_incep_res.classif=nn.Sequential()</p>
<p>def features_extract(loader, model,features):<br>
num_correct = 0<br>
num_samples = 0<br>
model.eval()<br>
i=0<br>
batchsize=64<br>
score=torch.empty((10222,features))<br>
with torch.no_grad():<br>
for x, y in iter(loader):<br>
x = x.to(device=device)<br>
y = y.to(device=device)<br>
scores = model(x)<br>
score[i*64:(i+1)*64]=scores<br>
i+=1<br>
model.train()<br>
return score</p>
<p>score_res=features_extract(trainloader,model_resnet,2048)<br>
score_incep=features_extract(trainloader,model_incep,2048)<br>
score_incep_res=features_extract(trainloader,model_incep_res,1536)<br>
score_xcep=features_extract(trainloader,model_xcep,2048)</p>
<p>features_extracted=torch.cat([score_res,score_incep,score_incep_res,score_xcep],axis=1)</p>
<p>from collections import OrderedDict<br>
my_model=nn.Sequential(OrderedDict([(‘fc1’, nn.Linear(7680, 5000)),<br>
(‘drop’, nn.Dropout(p=0.5)),<br>
(‘fc2’, nn.Linear(5000, 120)),<br>
(‘output’, nn.Softmax(dim=0))])).cuda()</p>
<p>criterion = nn.CrossEntropyLoss()<br>
optimizer = optim.Adam(my_model.parameters(), 0.001 )</p>
<p>from tqdm import tqdm<br>
epochs = 5<br>
check_every = 1<br>
iters = 0<br>
train_loss=0<br>
modle.to(device)</p>
<p>from tqdm import tqdm<br>
epochs = 5<br>
check_every = 1<br>
iters = 0<br>
train_loss=0<br>
modle.to(device)</p>
<p>for epoch in range(epochs):<br>
for i in range(features_extracted.shape[0]):<br>
# Get data to cuda if possible<br>
data = features_extracted[i].to(device=device)<br>
targets = y_train[i].to(device=device)</p>
<pre><code>    # forward
    scores = modle(torch.reshape(data,(-1,7680)))
    scores=max(scores)
    loss = criterion(scores, targets)
    # backward
    optimizer.zero_grad()
    loss.backward()
    # gradient descent or adam step
    optimizer.step()
</code></pre>
<p>def check_accuracy(features_extracted, model,y_train):<br>
num_correct = 0<br>
num_samples = 0<br>
model.eval()</p>
<pre><code>with torch.no_grad():
    for i in range(features_extracted.shape[0]):
        # Get data to cuda if possible
        data = features_extracted[i].to(device=device)
        targets = y_train[i].to(device=device)
        global scores
        scores = model(data)
        predictions = torch.argmax(scores)
        if(predictions==y_train[i]):
            num_correct+=1

model.train()
return num_correct/features_extracted.shape[0]
</code></pre>
<p>print(f"Accuracy on Validation set: {check_accuracy(features_val, modle,y_val):.2f}")</p>

ptrblck: <p>Your code is quite hard to read since you haven’t formatted it properly.<br>
From the first look it seems these parts might be wrong:</p>
<pre><code class="lang-python">(‘output’, nn.Softmax(dim=0))])).cuda()
criterion = nn.CrossEntropyLoss()
</code></pre>
<p><code>nn.CrossEntropyLoss</code> expects raw logits, so remove the <code>Softmax</code> layer.</p>
<pre><code class="lang-python">    scores=max(scores)
    loss = criterion(scores, targets)
</code></pre>
<p>I don’t know why <code>max</code> is used here instead of passing the logits to the criterion directly.</p>

kareem: <p>when I did remove the Softmax, the last fc layer got large values not probabilities, for better code formate please go to this link :<br>
<a href="https://discuss.pytorch.org/t/feature-extraction-using-pretrained-models/161023">Feature extraction Using Pretrained Models - vision - PyTorch Forums</a><br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Yes, that’s expected since logits are unbound and can have any value in <code>[-Inf, Inf]</code> (of course an <code>Inf</code> value would not be useful). You can still use a <code>softmax</code> to print the probabilities but should not pass them to <code>nn.CrossEntropyLoss</code> as this loss function will apply <code>F.log_softmax</code> and <code>nn.NLLLoss</code> internally.</p>

kareem: <p>Okay, I did delete the softmax, and still the validation accuracy is 2.9% maximum ?<br>
I think the problem is with the features extracted !!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c20cab657567b3ce4ce5a1e6edb2196e0e484f1b.png" data-download-href="https://discuss.pytorch.org/uploads/default/c20cab657567b3ce4ce5a1e6edb2196e0e484f1b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c20cab657567b3ce4ce5a1e6edb2196e0e484f1b.png" alt="image" data-base62-sha1="rGDLXEdpXdfwyhIrx3Un1RaW08X" width="690" height="413" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">757×454 14 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>