Imbalanced class + Augmentation problem

username: ivolis
cooked: <p>Hi everyone!</p>
<p>Im trying a new approach on nodule recognition in chest x-rays with JSRT dataset where I concatenate a filtered image but i’m having problemas as my dataset is SMALL (247 images) and IMBALANCED (154 images with nodules and without).</p>
<p>I have something like this</p>
<pre><code class="lang-auto">original_transformation = transforms.Compose([
        transforms.Grayscale(num_output_channels=1),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.5], std=[0.5])
    ])

# Loading original images
original_dataset = torchvision.datasets.ImageFolder(
        root=dataset_folder_path,
        transform=original_transformation
    )
</code></pre>
<p>I have a folder where the original image has the filtered one directly after (i.e:  1.jpg and 1A.jpg), so in order to concatenate I use something like this</p>
<pre><code class="lang-auto">dataset = []

for i in range(0,len(original_dataset)):
  if(i%2==0):
    conc_img = torch.cat((original_dataset[i][0], original_dataset[i+1][0]), 0)
    conc_tuple = (conc_img , original_dataset[i][1])
    dataset.append(conc_tuple)
</code></pre>
<p>This is a quick and dirty solution and it is kinda bad because the dataset losts all of the object methods and attributes, so in the rest of the code you will see some loops in order to get the labels, for example.</p>
<p>For the splitting (where I think the problem begins) I have two options <strong>which bring the same output</strong> (obviusly I comment one to run the other one). Both give me the same proportion of with/without nodule images for training and testing.</p>
<pre><code class="lang-auto">train_size = int(0.7 * len(dataset))
test_size = len(dataset) - train_size

# Random split *
train_dataset, test_dataset = torch.utils.data.random_split(dataset, [train_size, test_size])

# Stritifed split *
from sklearn.model_selection import train_test_split

labels_dataset = []
for i in range(0,len(dataset)):
  labels_dataset.append(dataset[i][1])

train_indices, test_indices = train_test_split(list(range(len(dataset))), test_size=test_size, stratify=labels_dataset)
train_dataset = torch.utils.data.Subset(dataset, train_indices)
test_dataset = torch.utils.data.Subset(dataset, test_indices)
</code></pre>
<p>For data augmentation I do something likes this (i have some doubts about it)</p>
<pre><code class="lang-auto"># First Aug
print("Augmentating...")
aug_dataset1 = []
first_transform = torchvision.transforms.RandomVerticalFlip(p=1)

for i in range(0,len(train_dataset)):
  transf_img = first_transform(train_dataset[i][0])
  transf_tuple = (transf_img , train_dataset[i][1])
  aug_dataset1.append(transf_tuple)


# Second Aug
print("Augmentating...")
aug_dataset2 = []
second_transform = torchvision.transforms.RandomHorizontalFlip(p=0.5)

for i in range(0,len(train_dataset)):
  transf_img = second_transform(train_dataset[i][0])
  transf_tuple = (transf_img , train_dataset[i][1])
  aug_dataset2.append(transf_tuple)

train_dataset = torch.utils.data.ConcatDataset([train_dataset,aug_dataset1])
train_dataset = torch.utils.data.ConcatDataset([train_dataset,aug_dataset2])

</code></pre>
<p>Finally, I <strong>wanted to balance classes</strong> so I used a code by <a class="mention" href="/u/ptrblck">@ptrblck</a> for the dataloaders:</p>
<pre><code class="lang-auto"># Dataloaders

# https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2?u=ptrblck
def get_sampler(target):
  class_sample_count = np.unique(target, return_counts=True)[1]
  weight = 1. / class_sample_count
  samples_weight = weight[target]

  samples_weight = torch.from_numpy(samples_weight)
  samples_weigth = samples_weight.double()
  sampler = torch.utils.data.WeightedRandomSampler(samples_weight, len(samples_weight))
  
  return sampler


train_targets = []
for i in range(0,len(train_dataset)):
  train_targets.append(train_dataset[i][1])


test_targets = []
for i in range(0,len(test_dataset)):
  test_targets.append(train_dataset[i][1])


train_sampler = get_sampler(train_targets)
test_sampler = get_sampler(test_targets)

b_size = 25

train_loader = torch.utils.data.DataLoader(
    train_dataset,
    batch_size=b_size,
    num_workers=0,
    sampler = train_sampler
    #shuffle=True
)

test_loader = torch.utils.data.DataLoader(
    test_dataset,
    batch_size=b_size,
    num_workers=0,
    sampler = test_sampler
    #shuffle=True # true?
)
</code></pre>
<p>The network looks like this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, num_classes = 3):
      super(Net, self).__init__()

      self.conv1 = nn.Conv2d(in_channels=2, out_channels=25, kernel_size=5, stride=1, padding=1)
      self.conv2 = nn.Conv2d(in_channels=25, out_channels=55, kernel_size=5, stride=1, padding=1)
      self.pool = nn.MaxPool2d(kernel_size=2)
      self.dropout1 = nn.Dropout2d(0.4)
      self.dropout2 = nn.Dropout2d(0.6)

      self.fc1 = nn.Linear(in_features=50 * 50 * 55, out_features= 1882)
      self.fc2 = nn.Linear(in_features=1882, out_features=num_classes)

    def forward(self, x):
      x = F.relu(self.pool(self.conv1(x)))
      x = F.relu(self.pool(self.conv2(x)))  
      x = F.dropout(self.dropout1(x), training=self.training)
      # Flattening
      x = x.view(-1, 50 * 50 * 55)
      x = self.fc1(x)
      x = F.relu(x)
      x = self.dropout2(x)
      x = self.fc2(x)

      return x

device = "cpu"
if (torch.cuda.is_available()):
    device = "cuda"

model = Net(num_classes=len(classes)).to(device)

print(model)
</code></pre>
<p><strong>And the real problem is that not only the accuracy is low (about 60%), but also when I make a Confusion Matrix seems to show that the classes are not balanced at all</strong>:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d36ca2177525b62c8921bf6b72cfb7fc17e42c2.png" alt="image" data-base62-sha1="4areqJJk58XpZsylGqryZan6bqa" width="318" height="325"></p>
<pre><code class="lang-auto">#Confusion matrix
truelabels = []
predictions = []
model.eval()
print("Getting predictions from test set…")
for data, target in test_loader:
  data = data.to(device)
  for label in target.data.numpy():
    truelabels.append(label)
  for prediction in model(data):
    predictions.append(torch.argmax(prediction).item())


cm = confusion_matrix(truelabels, predictions)
tick_marks = np.arange(len(classes))

df_cm = pd.DataFrame(cm, index = classes, columns = classes)
plt.figure(figsize = (7,7))
sns.heatmap(df_cm, annot=True, cmap=plt.cm.Blues, fmt='g')
plt.xlabel("Predicted Shape", fontsize = 20)
plt.ylabel("True Shape", fontsize = 20)
plt.show()
</code></pre>
<p><strong>What I’m doing wrong?</strong> Im really new not only in Pytorch but also on Python and I really want to learn how to this the right way. Really sorry for the bad english and the long post, but I already tried so many approaches that I don’t know what else I can do. Any feedback will be life saving!</p>
<p>Thank you very much in advance!</p>

username: ptrblck
cooked: <p>Could you check, if this loop is indeed getting the right class predictions?</p>
<pre><code class="lang-python">  for prediction in model(data):
    predictions.append(torch.argmax(prediction).item())
</code></pre>
<p>Usually you would keep the output batch and use <code>torch.argmax(output, dim=1)</code>.<br>
I think your code should work, but might be worth double checked.</p>
<p>RE: balanced sampling:<br>
could you iterate the <code>DataLoader</code> and check the class frequency in each batch?</p>

username: ivolis
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your fast answer!</p>
<blockquote>
<p>Usually you would keep the output batch and use <code>torch.argmax(output, dim=1)</code> .<br>
I think your code should work, but might be worth double checked.</p>
</blockquote>
<p>I’m sorry you mean instead of using that for-loop just pass the confusion matrix method <code> torch.argmax(output, dim=1)</code> (being output the output batch from <code>model(data)</code>?</p>
<p><strong>balanced samplig:</strong><br>
Is this what you are saying about class frequency in each batch?</p>
<pre><code class="lang-auto">print("Training batchs proportion...")
for i, (data, target) in enumerate(train_loader):
    print ("batch index {}, 0/1: {}/{}".format(
        i,
        len(np.where(target.numpy() == 0)[0]),
        len(np.where(target.numpy() == 1)[0])))
    
print("Testing batchs proportion...")
for i, (data, target) in enumerate(test_loader):
    print ("batch index {}, 0/1: {}/{}".format(
        i,
        len(np.where(target.numpy() == 0)[0]),
        len(np.where(target.numpy() == 1)[0])))
</code></pre>
<p>I get the following output (it changes every time I run it):</p>
<blockquote>
<p>Training batchs proportion…<br>
batch index 0, 0/1: 13/12<br>
batch index 1, 0/1: 15/10<br>
batch index 2, 0/1: 14/11<br>
batch index 3, 0/1: 9/16<br>
batch index 4, 0/1: 11/14<br>
batch index 5, 0/1: 12/13<br>
batch index 6, 0/1: 9/16<br>
batch index 7, 0/1: 7/18<br>
batch index 8, 0/1: 15/10<br>
batch index 9, 0/1: 9/16<br>
batch index 10, 0/1: 12/13<br>
batch index 11, 0/1: 7/18<br>
batch index 12, 0/1: 14/11<br>
batch index 13, 0/1: 11/14<br>
batch index 14, 0/1: 13/12<br>
batch index 15, 0/1: 14/11<br>
batch index 16, 0/1: 9/16<br>
batch index 17, 0/1: 8/17<br>
batch index 18, 0/1: 12/13<br>
batch index 19, 0/1: 14/11<br>
batch index 20, 0/1: 7/9<br>
Testing batchs proportion…<br>
batch index 0, 0/1: 6/19<br>
batch index 1, 0/1: 12/13<br>
batch index 2, 0/1: 5/20</p>
</blockquote>
<p>and if I run it again</p>
<blockquote>
<p>Training batchs proportion…<br>
batch index 0, 0/1: 13/12<br>
batch index 1, 0/1: 9/16<br>
batch index 2, 0/1: 6/19<br>
batch index 3, 0/1: 15/10<br>
batch index 4, 0/1: 12/13<br>
batch index 5, 0/1: 14/11<br>
batch index 6, 0/1: 9/16<br>
batch index 7, 0/1: 11/14<br>
batch index 8, 0/1: 14/11<br>
batch index 9, 0/1: 17/8<br>
batch index 10, 0/1: 12/13<br>
batch index 11, 0/1: 10/15<br>
batch index 12, 0/1: 8/17<br>
batch index 13, 0/1: 16/9<br>
batch index 14, 0/1: 14/11<br>
batch index 15, 0/1: 14/11<br>
batch index 16, 0/1: 12/13<br>
batch index 17, 0/1: 12/13<br>
batch index 18, 0/1: 10/15<br>
batch index 19, 0/1: 15/10<br>
batch index 20, 0/1: 7/9<br>
Testing batchs proportion…<br>
batch index 0, 0/1: 11/14<br>
batch index 1, 0/1: 12/13<br>
batch index 2, 0/1: 13/12</p>
</blockquote>
<p>EDIT: I noticed that any time I run the code for the confusion matrix it changes, is that normal aswell?</p>

username: ptrblck
cooked: <p>Based on the output it seems balancing might be working. To make sure, you could increase the batch size as it would smooth the random sampling a bit and might create more balanced batches.</p>
<p>Yes, it’s expected to get duplicates in the sampling as the <code>WeightedRandomSampler</code> is randomly sampling the data corresponding to the sample weights with replacement.<br>
Usually you wouldn’t apply this sampler on the validation or test datasets as they should correspond to the expected “real world” data.</p>

username: ivolis
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> based on what you said, I have only two last questions <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20">:</p>
<p>Looking at the confusion matrix on the original post, it seems like there are 13+39=52 “yes” images and 13+10=23 images when my network is testing in order to make the confusion matrix (which is not really balanced)</p>
<p>However sometimes I get something more “balanced” like this:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ceea2affafbcfabfc8266327ca3913eaa248b6c0.png" alt="image" data-base62-sha1="tws3OfZoWlkpbfl89JsiDbPDgpq" width="434" height="438"></p>
<p>Where I have 36 “no” and 39 “yes”.</p>
<p><strong>1)</strong> This numbers are the sum of the batchs, right? For example:</p>
<ul>
<li>52 yes vs 23 no images → batches are something like:</li>
</ul>
<blockquote>
<p>test bach INDEX: <span class="hashtag-raw">#yes</span> / <span class="hashtag-raw">#no</span><br>
batch 0: 16/7<br>
batch 1: 17/6<br>
batch 2: 19/10</p>
</blockquote>
<ul>
<li>39 yes vs 36 no images → batches are something like:</li>
</ul>
<blockquote>
<p>test bach INDEX: <span class="hashtag-raw">#yes</span> / <span class="hashtag-raw">#no</span><br>
batch 0: 13/11<br>
batch 1: 14/13<br>
batch 2: 12/12</p>
</blockquote>
<p><strong>2)</strong> Related to last question, seeing that numbers it seems hat my class <strong>IS</strong> sometimes umbalanced when testing? Even using a <code>WeightedRandomSampler</code>? Is there a way I can get a consistency balance when testing? <strong>Why is that happening?</strong> (maybe related to what you said about the batch size?)</p>
<p>Thank you in advance for your patience!</p>

username: ptrblck
cooked: <ol>
<li>
<p>Yes, the number of yes/no (or positive/negative) samples corresponds to the used target labels used to create the confusion matrix (and how you’ve samples these targets in the first place).</p>
</li>
<li>
<p>As previously said, I wouldn’t expect to use a weighted sampler during validation or testing, but your use case might very. As the sampling process is random, noise is expected. To verify that your sampling is indeed working correctly, you could increase the batch size and see if this would smooth the class imbalance.</p>
</li>
</ol>