Balanced Sampling between classes with torchvision DataLoader

Jordi_de_la_Torre: <p>Hi all,</p>
<p>I’m trying to find a way to make a balanced sampling using ImageFolder and DataLoader with a imbalanced dataset. I suppose that I should build a new sampler.</p>
<p>I’m not sure if I’m missing something. Is there an already implemented way of do it?</p>
<p>Thanks</p>
<p>Code:</p>
<p>train_loader = torch.utils.data.DataLoader(<br>
datasets.ImageFolder(traindir, transforms.Compose([<br>
transforms.Scale(600),<br>
transforms.RandomSizedCrop(512),<br>
transforms.ToTensor(),<br>
normalize<br>
])),<br>
batch_size=args.batch_size, shuffle=True,<br>
num_workers=args.workers, pin_memory=True)</p>

smth: <p>you should look at <a href="https://discuss.pytorch.org/t/how-to-prevent-overfitting/1902/6?u=smth">How to Prevent Overfitting</a></p>

Jordi_de_la_Torre: <p>For a dataset coming from a torchvision ImageFolder dataset the code that I’ve finally used, based on the suggestion of <a class="mention" href="/u/smth">@smth</a>  is the next:</p>
<p>First, define the function:</p>
<pre><code class="lang-python">def make_weights_for_balanced_classes(images, nclasses):                        
    count = [0] * nclasses                                                      
    for item in images:                                                         
        count[item[1]] += 1                                                     
    weight_per_class = [0.] * nclasses                                      
    N = float(sum(count))                                                   
    for i in range(nclasses):                                                   
        weight_per_class[i] = N/float(count[i])                                 
    weight = [0] * len(images)                                              
    for idx, val in enumerate(images):                                          
        weight[idx] = weight_per_class[val[1]]                                  
    return weight                                                               
</code></pre>
<p>And after this, use it in the next way:</p>
<pre><code class="lang-python">dataset_train = datasets.ImageFolder(traindir)                                                                         
                                                                                
# For unbalanced dataset we create a weighted sampler                       
weights = make_weights_for_balanced_classes(dataset_train.imgs, len(dataset_train.classes))                                                                
weights = torch.DoubleTensor(weights)                                       
sampler = torch.utils.data.sampler.WeightedRandomSampler(weights, len(weights))                     
                                                                                
train_loader = torch.utils.data.DataLoader(dataset_train, batch_size=args.batch_size, shuffle = True,                              
                                                             sampler = sampler, num_workers=args.workers, pin_memory=True)     
</code></pre>

FuriouslyCurious: <p>Super helpful - thank you!</p>

mderakhshani: <p>Could you please tell me how did you iterate over train_loader? I mean the for LOOP.</p>

ptrblck: <p>You can just loop over it:</p>
<pre><code>for data, target in train_loader:
    data = ...</code></pre>

mderakhshani: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, Thanks for your response! My code is like following:</p>
<pre><code>data = torch.FloatTensor(numClass, numDataPoints, data_dim)
target = torch.zeros(numClass, numDataPoints)
weight = torch.histc(target, bins=10, min = 0, max = 9)
weight = 1/(weight + 1e-3)
sampler = torch.utils.data.sampler.WeightedRandomSampler(weight.type('torch.DoubleTensor'), bs)
data = data.view(-1,data_dim)
target = target.view(-1,1)
dataset_x = data.numpy()
dataset_y = target.numpy().astype(int)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(dataset_x,
                                                    dataset_y,
                                                    test_size=0.33,
                                                    random_state=42,
                                                    stratify = dataset_y)
trainDataset = torch.utils.data.TensorDataset(torch.FloatTensor(X_train), torch.LongTensor(y_train.astype(int)))
validDataset = torch.utils.data.TensorDataset(torch.FloatTensor(X_test), torch.LongTensor(y_test.astype(int)))

trainLoader = torch.utils.data.DataLoader(dataset = trainDataset, batch_size=bs, num_workers=1, sampler = sampler)
testLoader = torch.utils.data.DataLoader(dataset = validDataset, batch_size=bs, shuffle=False, num_workers=1) 
</code></pre>
<p>As you have mentioned above, I have written a code:</p>
<pre><code>for data, labels in train_loader:
         # print something
</code></pre>
<p>But something went wrong here. It gave me an error like this:</p>
<p>‘StopIteration’</p>
<p>Could you please tell me what is wrong with my code? Actually, I would like to get samples from my data loader 1000 times and then at each step train the network and do backward!</p>

ptrblck: <p>In your code the weights seen to be invalid, since you initialize them with an all zero target.<br>
The <code>histc</code> method will just return a histogram with all samples being 0.<br>
Maybe this is your error?</p>
<p>Here is a sample snippet:</p>
<pre><code>batch_size = 20
class_sample_count = [10, 1, 20, 3, 4] # dataset has 10 class-1 samples, 1 class-2 samples, etc.
weights = 1 / torch.Tensor(class_sample_count)
weights = weights.double()
sampler = torch.utils.data.sampler.WeightedRandomSampler(weights, batch_size)
trainloader = torch.utils.data.DataLoader(trainDataset, batch_size = batch_size, sampler = sampler)</code></pre>

mderakhshani: <p>Sorry, <a class="mention" href="/u/ptrblck">@ptrblck</a>,  please assume that the target is initialized with some random integer numbers between 0 and 9. In practice, they are not all zero. So, with this clarification, what is wrong with the code?</p>

ptrblck: <p>Ok, I found some other issues. I changed your code a bit. This should work:</p>
<pre><code>numDataPoints = 1000
data_dim = 5
bs = 100

data = torch.FloatTensor(numDataPoints, data_dim)
target = np.hstack((np.zeros(int(numDataPoints * 0.9), dtype=np.int32),
                    np.ones(int(numDataPoints * 0.1), dtype=np.int32)))

dataset_x = data.numpy()
dataset_y = target
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(dataset_x,
                                                    dataset_y,
                                                    test_size=0.33,
                                                    random_state=42,
                                                    stratify = dataset_y)
print 'target train: {}/{}'.format(len(np.where(y_train==0)[0]),
                                   len(np.where(y_train==1)[0]))

class_sample_count = np.array([len(np.where(y_train==t)[0]) for t in np.unique(y_train)])
weight = 1. / class_sample_count
samples_weight = np.array([weight[t] for t in y_train])

samples_weight = torch.from_numpy(samples_weight)
sampler = WeightedRandomSampler(samples_weight.type('torch.DoubleTensor'), len(samples_weight))

trainDataset = torch.utils.data.TensorDataset(torch.FloatTensor(X_train), torch.LongTensor(y_train.astype(int)))
validDataset = torch.utils.data.TensorDataset(torch.FloatTensor(X_test), torch.LongTensor(y_test.astype(int)))

trainLoader = torch.utils.data.DataLoader(dataset = trainDataset, batch_size=bs, num_workers=1, sampler = sampler)
testLoader = torch.utils.data.DataLoader(dataset = validDataset, batch_size=bs, shuffle=False, num_workers=1) 

for i, (data, target) in enumerate(trainLoader):
    print "batch index {}, 0/1: {}/{}".format(
        i, len(np.where(target.numpy()==0)[0]), len(np.where(target.numpy()==1)[0]))
</code></pre>
<p>It seems that weights should have the same length as your number of samples.<br>
I created a dummy data set with a target imbalance of 9 to 1.<br>
The for loop should loop through all your train samples with each batch containing approx. the same amount of<br>
zeros and ones.</p>
<p>I hope it works for you!</p>

mderakhshani: <p>Thanks for your great response! But one more question about your code. Could please explain how can I determine how many times does <strong>For</strong> execute? <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>No problem <img src="//discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"><br>
Sure, you can call <code>len(trainLoader)</code>, which will return 7 for the example. In this particular case, it will return 6 full batches of 100 samples and a final one with only 70 samples.</p>

tshrjn: <p>Update: With version pytorch 0.2, <code>sampler</code> is mutually exclusive with <code>shuffle</code>.</p>

AruniRC: <p>By default the WeightedRandomSampler will use <code>replacement=True</code>. In which case, the samples that would be in a batch would not necessarily be unique.</p>
<p>Also, shouldn’t the function call to be like <code>sampler = WeightedRandomSampler(samples_weight.type('torch.DoubleTensor'), batch_size)</code> ?</p>

ptrblck: <p>You are right! Since <code>replacement=True</code> is the default option, there will probably be repetitions.</p>
<p>If you use the <code>batch_size</code> for the <code>num_samples</code> argument, your <code>DataLoader</code> will just return one batch of samples in the loop.</p>

Jumabek_Alikhanov: <p>I was able to use this weighted random data sampling using <a class="mention" href="/u/ptrblck">@ptrblck</a> 's code.</p>
<p>here is a snippet:</p>
<pre><code class="lang-auto">   print('Loading data...')
   train_dataset = datasets.ImageFolder(root=traindir,
                                    transform=transform_train)
   test_dataset = datasets.ImageFolder(root=valdir,
                                   transform=transform_test)
   print('Loading is Done!')

   num_classes = len(train_dataset.classes)

   #TODO: read class counts from the file
   class_sample_counts = [10647, 5659, 31445, 40283,  800,  407, 1111, 22396,  610, 1288, 5708, 1538, 1848, 26015, 17639, 3859,  473, 2509,  579, 2636,  822, 1616, 1226,  949, 1725, 1306, 1758, 1704, 10637, 1091, 1036, 1292,  474,  569, 1682,  553,  506, 7571, 3598, 2280, 24291, 5725, 1319,  824, 5456, 1781, 4074, 2538, 5032,  503, 1623, 7251,  599, 9037, 12221, 2128, 2290,  459, 1549, 1739, 2297,  838,  469,  674, 1030,  994,  704,  672, 1690, 2442,  766,  578, 2032,  534,  552, 13934, 1138, 1372]

   # compute weight for all the samples in the dataset
   # samples_weights contain the probability for each example in dataset to be sampled  
   class_weights = 1./torch.Tensor(class_sample_counts)
   train_targets = [sample[1] for sample in train_dataset.imgs]
   train_samples_weight = [class_weights[class_id] for class_id in train_targets]
   test_targets = [sample[1] for sample in test_dataset.imgs]
   test_samples_weight = [class_weights[class_id] for class_id in test_targets]

   # now lets initialize samplers 
   train_sampler = torch.utils.data.sampler.WeightedRandomSampler(train_samples_weight, len(train_dataset))
   test_sampler = torch.utils.data.sampler.WeightedRandomSampler(test_samples_weight, len(test_dataset))

   train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=args.batch_size,sampler=train_sampler, **kwargs)
   val_loader = torch.utils.data.DataLoader(test_dataset, batch_size=args.batch_size, sampler=test_sampler, **kwargs)

</code></pre>

Jumabek_Alikhanov: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I have a question:</p>
<p>with plain sampler I needed 10GB gpu memory for the batch size of 64<br>
However, with WeightedRandomSampler same batch size is using only 2.3GB gpu memory.</p>
<p>does it has to do with repetitions? In fact, my data is heavily imbalanced.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/d/d4580332be2165046fc3ef3ab607d1d1bf6b9fad.png" alt="180611_165227_classification_1_Item_jumabek" data-base62-sha1="uitMXlnq6r1nyDqPS4BIJHMJ48R" width="600" height="400"><br>
Or should I be concerned?</p>
<p>Thanks!</p>

ptrblck: <p>The sampling shouldn’t influence the GPU memory.<br>
I assume your data is on the CPU, you sample from it, and push it inside the training loop to the GPU.<br>
Therefore, your training procedure does not know anything about the sampling procedure.<br>
Your issue sounds quite strange.</p>
<p>Did you make sure the batch size stays the same?</p>

Jumabek_Alikhanov: <p>I confirmed that every iteration is using the same batch size compared to normal sampler case.<br>
and yes, I am moving input to gpu.</p>
<p>Would you please help me identify the problem. I am pasting training code here:</p>
<pre><code class="lang-auto">    for i, (input, target) in enumerate(train_loader):
        print('#images loaded: {} '.format(target.size()))
        
        if args.cuda and torch.cuda.is_available():
            target = target.cuda(async=True)
            input = input.cuda(async=True)

        # compute output
        output = model(input)
        loss = criterion(output, target)

        # compute gradient and do SGD step
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

</code></pre>

Jumabek_Alikhanov: <p>Issue has to do with frozen model weights.<br>
For the same batch size:</p>
<ol>
<li>10GB gpu memory required for training all resnet101 layers</li>
<li>2.3GB gpu memory required for training only layer4  and fc layers of resnet101 . That’s because it does not need to keep track of gradients for layer1,layer2,layer3 of resnet101.</li>
</ol>