Problem Testing Image classification predictions with torchvision.datasets.ImageFolder vs torchvision.transforms.Compose on Resnet50 and my custom architecture

LouisBao: <p>When I training my custom model and fine-tuning the pretrained resnet50 with using torchvision.datasets.ImageFolder, I have underfitting/overfitting during training and get good 98% accuracy on validation. If I use torchvision.datasets.ImageFolder to load the test folder, i get very good prediction and confusion matrix but If I reuse this transform to convert image when I scan the test folder , I always get poor prediction. I don’t know what is this problem in my cas.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/5/959c2805942fa7c21d06099cd868244527d02d10.png" data-download-href="https://discuss.pytorch.org/uploads/default/959c2805942fa7c21d06099cd868244527d02d10" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/959c2805942fa7c21d06099cd868244527d02d10_2_682x500.png" alt="image" data-base62-sha1="llvHNNbRK80MXOt1cwZ8vK5QP6w" width="682" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/959c2805942fa7c21d06099cd868244527d02d10_2_682x500.png, https://discuss.pytorch.org/uploads/default/original/3X/9/5/959c2805942fa7c21d06099cd868244527d02d10.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/5/959c2805942fa7c21d06099cd868244527d02d10.png 2x" data-dominant-color="E2E3EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">901×660 61.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is confusion matrix with torchvision.datasets.ImageFolder<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/02c1d9b3497aa94ee01ed0cb2626326051e40a4e.png" data-download-href="https://discuss.pytorch.org/uploads/default/02c1d9b3497aa94ee01ed0cb2626326051e40a4e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02c1d9b3497aa94ee01ed0cb2626326051e40a4e_2_493x500.png" alt="image" data-base62-sha1="oohcFGA9RBSwauceDTwcEnNAfQ" width="493" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02c1d9b3497aa94ee01ed0cb2626326051e40a4e_2_493x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02c1d9b3497aa94ee01ed0cb2626326051e40a4e_2_739x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/2/02c1d9b3497aa94ee01ed0cb2626326051e40a4e.png 2x" data-dominant-color="EFF2F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">742×752 103 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
model = models.resnet50(pretrained = False)<br>
model.fc = nn.Sequential(nn.Linear(2048, 512),<br>
nn.Linear(512, 20))<br>
model=model.to(device)<br>
checkpoint = torch.load(os.path.join(w_dir,‘checkpoints’,test_name,‘last.pth’), map_location=device)<br>
model.load_state_dict(checkpoint[‘model_state_dict’])<br>
test_transform = transforms.Compose([<br>
transforms.Resize((img_size, img_size)),<br>
transforms.ToTensor(),<br>
transforms.Normalize(<br>
mean=[0, 0, 0],<br>
std=[1, 1, 1])])<br>
dataset_name=‘datasets3*’<br>
test_path=os.path.join(w_dir,‘dataset_cropped_20cl’,<br>
dataset_name,‘test’)<br>
test_dataset = datasets.ImageFolder(<br>
root=test_path,<br>
transform=test_transform)<br>
test_loader = DataLoader(<br>
test_dataset, batch_size=BATCH_SIZE, shuffle=False,<br>
num_workers=4, pin_memory=True)<br>
model.eval()<br>
print(‘Make prediction’)<br>
pred_list=list()<br>
true_list=list()<br>
true_name_list=list()<br>
pred_name_list=list()<br>
save_pred_path=os.path.join(w_dir,‘CNN_prediction’,‘checkpoints’)<br>
if os.path.exists(save_pred_path):<br>
shutil.rmtree(save_pred_path)<br>
os.makedirs(save_pred_path)<br>
with torch.no_grad():<br>
for i, data in tqdm(enumerate(test_loader), total=len(test_loader)):<br>
image, labels = data<br>
image = image.to(device)<br>
labels = labels.to(device)<br>
outputs = model(image)<br>
<span class="hashtag-raw">#output_label</span> = torch.topk(outputs, 1)<br>
<span class="hashtag-raw">#pred_class</span> = labels[int(output_label.indices.item())]<br>
<span class="hashtag-raw">#pred_list</span>.append(int(output_label.indices))<br>
<span class="hashtag-raw">#true_list</span>.append(int(labels))<br>
_, preds = torch.max(outputs.data, 1)<br>
pred_list.append(int(preds))<br>
true_list.append(int(labels))<br>
true_name_list.append(id_to_labels[int(labels)])<br>
pred_name_list.append(id_to_labels[int(preds)])<br>
if int(preds)==int(labels):<br>
print(‘prediction is good’)<br>
print(f’ true class is {int(labels)}, predicted class is {int(preds)}‘)<br>
print(f"GT: {id_to_labels[int(labels)]}, pred: {id_to_labels[int(preds)]}")<br>
else:<br>
print(‘prediction is not good’)<br>
print(f’ true class is {int(labels)}, predicted class is {int(preds)}')<br>
print(f"GT: {id_to_labels[int(labels)]}, pred: {id_to_labels[int(preds)]}")<br>
This is ccconfusion matrix with using my transform<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d39ec7b51261636f36fd165bdee1affecbb1d608.png" data-download-href="https://discuss.pytorch.org/uploads/default/d39ec7b51261636f36fd165bdee1affecbb1d608" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d39ec7b51261636f36fd165bdee1affecbb1d608_2_491x500.png" alt="image" data-base62-sha1="uc4VJ36ket7u1LkXtAj4zEi7a52" width="491" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d39ec7b51261636f36fd165bdee1affecbb1d608_2_491x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d39ec7b51261636f36fd165bdee1affecbb1d608_2_736x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d39ec7b51261636f36fd165bdee1affecbb1d608.png 2x" data-dominant-color="EFF1F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">736×749 103 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
test_transform = transforms.Compose([<br>
transforms.ToPILImage(),<br>
transforms.Resize((img_size, img_size)),<br>
transforms.ToTensor(),<br>
transforms.Normalize(<br>
mean=[0, 0, 0],<br>
std=[1, 1, 1])])<br>
dataset_name=‘datasets3*’<br>
dataset_path=os.path.join(w_dir,‘dataset_cropped_20cl’,dataset_name,‘test’)<br>
pred_list=list()<br>
true_list=list()<br>
true_name_list=list()<br>
pred_name_list=list()<br>
model.eval()<br>
with torch.no_grad():<br>
<span class="hashtag-raw">#scan</span> test folder<br>
for folder_name in os.listdir(dataset_path):<br>
gt_class=folder_name<br>
for roots,dirs,files in os.walk(os.path.join(dataset_path,folder_name)):<br>
for file in files:<br>
img_path=os.path.join(roots,file)<br>
image = cv2.imread(img_path)<br>
orig_image = image.copy()<br>
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)<br>
image=test_transform(image)<br>
image = torch.unsqueeze(image, 0)<br>
with torch.no_grad():<br>
outputs = model(image.to(device))<br>
_, pred_class = torch.max(outputs.data, 1)<br>
pred_list.append(int(pred_class))<br>
true_list.append(int(gt_class))<br>
true_name_list.append(id_to_labels[int(gt_class)])<br>
pred_name_list.append(id_to_labels[int(pred_class)])</p>

LouisBao: <p>Thanks for your helps</p>

ptrblck: <p>Your code unfortunately not properly formatted so I’m unsure what kind of transformation you are using in which run, but note that:</p>
<pre><code class="lang-python">transforms.ToTensor(),
transforms.Normalize(
    mean=[0, 0, 0],
    std=[1, 1, 1])])
</code></pre>
<p>will not normalize the tensors since zeros will be subtracted from them and afterwards you are dividing by ones. In case you used another <code>Normalize</code> transformation during training you should also reuse it during testing.</p>

LouisBao: <ul>
<li>Yes, I also used the same transformation for test as transformation during training.</li>
</ul>
<pre><code class="lang-auto">train_transform = transforms.Compose([
                                transforms.Resize((img_size, img_size)),
                                transforms.RandomHorizontalFlip(p=0.5),
                                transforms.RandomVerticalFlip(p=0.5),
                                transforms.RandomAffine(180,scale=(0.25,2)),
                                transforms.RandomPerspective(p=0.5),
                                transforms.ColorJitter(brightness=0.5,hue=0.2),
                                transforms.ToTensor(),
                                transforms.Normalize(
                                                    mean=[0.5, 0.5, 0.5],
                                                    std=[0.5, 0.5, 0.5]),
                                ])

valid_transform = transforms.Compose([
                                    transforms.Resize((img_size, img_size)),
                                    transforms.ToTensor(),
                                    transforms.Normalize(
                                                        mean=[0.5, 0.5, 0.5],
                                                        std=[0.5, 0.5, 0.5]),
                                    ])
train_path=os.path.join(w_dir,'dataset_cropped_20cl',
                        dataset_name,'train')
val_path=os.path.join(w_dir,'dataset_cropped_20cl',
                      dataset_name,'val')


train_dataset = datasets.ImageFolder(root=train_path,
                                    transform=train_transform)
valid_dataset = datasets.ImageFolder(root=val_path,
                                     transform=valid_transform)
train_loader = DataLoader(train_dataset,batch_size=BATCH_SIZE,
                          shuffle=True,num_workers=4, pin_memory=True)
valid_loader = DataLoader(valid_dataset, batch_size=BATCH_SIZE, 
                          shuffle=False,num_workers=4, pin_memory=True)
</code></pre>
<ul>
<li>Reload the trained model</li>
</ul>
<pre><code class="lang-auto"> #recall model
if model_name=='custom':
    model=CNN_network(img_size=img_size,
                      in_channels=3,
                      conv_depths=[8,16,32,64,128,256],
                      lin_depths=[1024,512],
                      dropout_conv=False,
                      dropout_li=False,
                      nb_classes=nbClasses,
                      classifer_activation=None,
                      activation='relu').to(device)
elif model_name=='resnet50':
    model = models.resnet50(pretrained = False)
    model.fc = nn.Sequential(nn.Linear(2048, 512),
                             nn.Linear(512, 20))
    model=model.to(device)
  
#reload model
checkpoint = torch.load(os.path.join(w_dir,'checkpoints',
                                     test_name,'last.pth'), 
                                     map_location=device)
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()
</code></pre>
<ul>
<li>Using ImageFolder to test model, i got the good confusion matrix</li>
</ul>
<pre><code class="lang-auto">test_transform = transforms.Compose([
                                    transforms.Resize((img_size, img_size)),
                                    transforms.ToTensor(),
                                    transforms.Normalize(
                                        mean=[0.5, 0.5, 0.5],
                                        std=[0.5, 0.5, 0.5])])
 dataset_name='datasets3*'
 test_path=os.path.join(w_dir,'dataset_cropped_20cl',
                        dataset_name,'test')
 test_dataset = datasets.ImageFolder(root=test_path,
                                    transform=test_transform)
 test_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE,
                         shuffle=False,num_workers=4, pin_memory=True)
 model.eval()
 with torch.no_grad():
      for i, data in tqdm(enumerate(test_loader), total=len(test_loader)):
          image, labels = data
          image = image.to(device)
          labels = labels.to(device)
          
          #get prediction
          outputs = model(image)
          _, preds = torch.max(outputs.data, 1)

</code></pre>
<ul>
<li>Without using ImageFolder, I scan my the test folder. In this cas, I used cv2 to load images and convert them to transformation. I got the poor confusion matrix</li>
</ul>
<pre><code class="lang-auto"> test_transform = transforms.Compose([
                                    transforms.ToPILImage(),
                                    transforms.Resize((img_size, img_size)),
                                    transforms.ToTensor(),
                                    transforms.Normalize(
                                        mean=[0.5, 0.5, 0.5],
                                        std=[0.5, 0.5, 0.5]) ])
 dataset_name='datasets3*'
 dataset_path=os.path.join(w_dir,'dataset_cropped_20cl',
                                  dataset_name,'test')
  model.eval()
  with torch.no_grad():
            for folder_name in os.listdir(dataset_path):
                gt_class=folder_name
                for roots,dirs,files in os.walk(
                                   os.path.join(dataset_path,folder_name)):
                    for file in files:
                        #load and convert image to true format
                        img_path=os.path.join(roots,file)
                        image = cv2.imread(img_path)
                        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
                        image=test_transform(image)
                        image = torch.unsqueeze(image, 0).float()
                        
                        #get predictions
                        outputs = model(image.to(device))
                        _, pred_class = torch.max(outputs.data, 1)
</code></pre>
<ul>
<li>I also try PIL.Image to load images but i got the same result.</li>
</ul>
<pre><code class="lang-auto">test_transform = transforms.Compose([
                                    transforms.Resize((img_size, img_size)),
                                    transforms.ToTensor(),
                                    transforms.Normalize(
                                        mean=[0.5, 0.5, 0.5],
                                        std=[0.5, 0.5, 0.5]) ])
model.eval()
with torch.no_grad():
    for folder_name in os.listdir(dataset_path):
        gt_class=folder_name
        for roots,dirs,files in os.walk(
                           os.path.join(dataset_path,folder_name)):
            for file in files:
                 img_path=os.path.join(roots,file)
                 image=Image.open(img_path)
                 image=test_transform(image)
                 image = torch.unsqueeze(image, 0).float()
                 #get predictions
                 outputs = model(image.to(device))
                 _, pred_class = torch.max(outputs.data, 1)

</code></pre>

ptrblck: <p>How are you defining the targets in your manual approach and did you compare the class indices to what <code>ImageFolder</code> returns?</p>

LouisBao: <ul>
<li>I was wrong when I was structuring the dataset folder. I have 20 classes and I marked the class folder as a number, I thought that the ImageFolder returned the correct target number but when I checked the dataset.class_to_idx, I got wrong label_to_number conversion.</li>
</ul>
<pre><code class="lang-auto">train_dataset.class_to_idx={'0': 0, '1': 1, '10': 2, '11': 3, '12': 4, '13': 5, '14': 6, '15': 7, '16': 8, '17': 9, '18': 10, '19': 11, '2': 12, '3': 13, '4': 14, '5': 15, '6': 16, '7': 17, '8': 18, '9': 19}
</code></pre>
<p>I think I don’t need to label the class folder name as a number!  My problem is resolved by modify the<br>
prediction number_to_class indices when I make prediction and confusion matrix.</p>
<p>But I have other question:</p>
<ul>
<li>
<p>Why we don’t need to convert the targets to [0,0,1,0,…,0] instead of torch.tensor([3]) when using CrossEntropyLoss?</p>
</li>
<li>
<p>When I used softmax in the last linear layer of my custom CNN network or I added in the last layer of Resnet50, during the training, the model didn’t achieve as good accuracy as without using torch.nn.Sotfmax for both the cas.</p>
</li>
</ul>

ptrblck: <p>To get the desired logit from the model output you could either multiply the logits with a one-hot encoded target tensor or you could directly index the logits creating the same results.<br>
Storing the target with a single integer instead of a large one-hot encoded tensor containing mostly zeros sounds reasonable.<br>
However, note that <code>nn.CrossEntropyLoss</code> accepts also “soft” targets (and thus also one-hot encoded targets) in newer versions if you pass the target as a floating point type.</p>
<p>That’s expected as <code>nn.CrossEntropyLoss</code> expects raw logits as the model outputs since internally <code>F.log_softmax</code> will be applied.</p>