Using WeightedRandomSampler for an imbalanced classes

username: Amir
cooked: <p>Hello,<br>
I have an imbalanced dataset in 6 classes, and I’m using the “WeightedRandomSampler”, but when I load the dataset, the train doesn’t work. My code is here:</p>
<blockquote></blockquote>
<pre><code>train_transforms = transforms.Compose([
transforms.Resize((sz, sz)),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

valid_transforms = transforms.Compose([
transforms.Resize((sz, sz)),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

train_ds = datasets.ImageFolder(train path, train_transforms)
valid_ds = datasets.ImageFolder(valid path, valid_transforms)

sample_count = [224, 477, 5027, 4497, 483, 247]
weight = 1 / torch.Tensor(sample_count)
sampler = WeightedRandomSampler(weight, batch_size) 

train_dl = torch.utils.data.DataLoader(train_ds, batch_size=batch_size, sampler=sampler)
valid_dl = torch.utils.data.DataLoader(valid_ds, batch_size=batch_size, shuffle=True)

train_ds_sz = len(train_ds)
valid_ds_sz = len(valid_ds)

print('Train size: {}\nValid size: {} ({:.2f})'.format(train_ds_sz, valid_ds_sz, valid_ds_sz/(train_ds_sz + 
valid_ds_sz)))

class_names = train_ds.classes
</code></pre>
<p>Any help would be appreciated.</p>

username: Amir
cooked: <p>I found that something is wrong in target because it’s zero but I don’t know why?!</p>
<blockquote>
<p>inputs, targets = next(iter(train_dl))<br>
print(targets)</p>
</blockquote>
<p>tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])</p>

username: charan_Vjy
cooked: <aside class="quote no-group" data-username="Amir" data-post="1" data-topic="74571">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amir/48/22073_2.png" class="avatar"> Amir:</div>
<blockquote>
<p>the train doesn’t work</p>
</blockquote>
</aside>
<p>What do you mean? Is there a syntax error? If yes, post the trace.<br>
Is it a problem of accuracy?</p>

username: Amir
cooked: <p><a class="mention" href="/u/charan_vjy">@charan_Vjy</a><br>
No, when I run it, nothing happens. My model train is here:</p>
<blockquote></blockquote>
<pre><code>num_epochs = 10
losses = []
for epoch in range(num_epochs):
for i, (inputs, targets) in enumerate(train_dl):
    inputs = to_var(inputs)
    targets = to_var(targets)
    
    #forward pass
    optimizer.zero_grad()
    outputs = model(inputs)
    
    #loss
    loss = criterion(outputs, targets)
    losses += [loss.data]
    
    #backward pass
    loss.backward()
    
    #update parameters
    optimizer.step()
    
    #report
    if (i+1) % 50 == 0:
        print('Epoch [%2d/%2d], Step [%3d, %3d], Loss: %.4f'
             % (epoch + 1, num_epochs, i + 1, len(train_ds) // batch_size, loss.data))   
</code></pre>
<p>As I told above, I found that something is wrong in the target.</p>

username: charan_Vjy
cooked: <p>Print out something every step rather than every first 50 steps. Print out the losses. We need to first figure out what’s happening. As for the target, why is having targets as ‘0’ a problem? Was there supposed to be someother value?</p>

username: Amir
cooked: <p>I found an example to create a sample <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">here</a> and modified it to create a sampler for my data as below:</p>
<blockquote></blockquote>
<pre><code>cls0 = np.zeros(224, dtype=np.int32)
cls1 = np.ones(477, dtype=np.int32)
cls2 = np.full(5027, 2, dtype=np.int32)
cls3 = np.full(4497, 3, dtype=np.int32)
cls4 = np.full(483, 4, dtype=np.int32)
cls5 = np.full(247, 5, dtype=np.int32)

target = np.hstack((cls0, cls1))
target = np.hstack((target, cls2))
target = np.hstack((target, cls3))
target = np.hstack((target, cls4))
target = np.hstack((target, cls5))

class_sample_count = np.unique(target, 
return_counts=True)[1]

weight = 1. / class_sample_count
samples_weight = weight[target]

samples_weight = torch.from_numpy(samples_weight)
sampler = WeightedRandomSampler(samples_weight, 
len(samples_weight))
</code></pre>
<p>I’m not sure that is correct, but with this sampler, the targets get value.</p>
<blockquote>
<p>inputs, targets = next(iter(train_dl))     # Get a batch of training data<br>
print(targets)</p>
</blockquote>
<p>tensor([1, 5, 3, 4, 3, 0, 5, 2, 0, 0, 4, 1, 5, 0, 5, 5, 5, 5, 2, 5, 1, 1, 0, 3])</p>
<p>and the train runs, but the number of loaded data is the same as the total number of data.</p>
<p>total number of data = 10955<br>
batch_size = 24<br>
step = 10955/24 = 456</p>
<blockquote>
<p>Epoch [ 1/ 2], Step [ 50, 456], Loss: 1.5504<br>
Epoch [ 1/ 2], Step [100, 456], Loss: 1.6046<br>
Epoch [ 1/ 2], Step [150, 456], Loss: 1.6864<br>
Epoch [ 1/ 2], Step [200, 456], Loss: 1.6291<br>
Epoch [ 1/ 2], Step [250, 456], Loss: 1.4469<br>
Epoch [ 1/ 2], Step [300, 456], Loss: 1.7395<br>
Epoch [ 1/ 2], Step [350, 456], Loss: 1.6110<br>
Epoch [ 1/ 2], Step [400, 456], Loss: 1.4821<br>
Epoch [ 1/ 2], Step [450, 456], Loss: 1.7239<br>
Epoch [ 2/ 2], Step [ 50, 456], Loss: 1.3867<br>
Epoch [ 2/ 2], Step [100, 456], Loss: 1.6165<br>
Epoch [ 2/ 2], Step [150, 456], Loss: 1.6229<br>
Epoch [ 2/ 2], Step [200, 456], Loss: 1.4635<br>
Epoch [ 2/ 2], Step [250, 456], Loss: 1.5007<br>
Epoch [ 2/ 2], Step [300, 456], Loss: 1.6607<br>
Epoch [ 2/ 2], Step [350, 456], Loss: 1.6613<br>
Epoch [ 2/ 2], Step [400, 456], Loss: 1.5939<br>
Epoch [ 2/ 2], Step [450, 456], Loss: 1.4794</p>
</blockquote>

username: charan_Vjy
cooked: <p>Note that the input to the <code>WeightedRandomSampler</code> in pytorch’s example is <code>weight[target]</code> and not <code>weight</code>. The length of  <code>weight_target</code>  is target whereas the length of <code>weight</code> is equal to the number of classes. This is probably the reason for the difference. Try using <code>WeightedRandomSampler(..,...,..,replacement=False)</code> to prevent it from happening.</p>
<p>As far as the loss for each steps go, it looks good. See if you could aggregate together all the losses and check if the loss for every subsequent epoch is decreasing.<br>
However, having a batch with the same class is definitely an issue.</p>

username: Amir
cooked: <p><a class="mention" href="/u/charan_vjy">@charan_Vjy</a><br>
Thanks for your help. I didn’t understand what exactly I need to do.<br>
Should the number of data in the “WeightedRandomSampler” be the total number of data or batch_size or the length of the smallest class?<br>
And also, Are my target values wrong in <a href="https://discuss.pytorch.org/t/using-weightedrandomsampler-for-an-imbalanced-classes/74571/6">this</a> way?<br>
I made a change like below and got the error when I want to make the targets. If you could show me by code, that would be great.</p>
<blockquote>
<p>sampler = WeightedRandomSampler([224,477,5027,4497,483,247], len(samples_weight), replacement=False)</p>
</blockquote>
<p>RuntimeError: cannot sample n_sample &gt; prob_dist.size(-1) samples without replacement</p>

username: charan_Vjy
cooked: <p>The weights should correspond to each sample in the train set. If their are 10,000 samples in the train set, the weights should correspond to each of the 10,000 samples. You would want to do something like this:</p>
<pre><code class="lang-auto">inputs,targets = train_ds ##Modify this to get all targets 
weight = 1. / class_sample_count
samples_weight = weight [ targets ]  ### You should get the labels in 
</code></pre>

username: Amir
cooked: <p>When I try to get targets from the train_ds, it receives zero. I think I got all the targets correctly in a previous way, and the only thing that I haven’t understood is the target of a batch of data, which is still imbalanced. For example, I changed the batch_size to 6, which is the number of my classes and passed it as the number of data into WeightedRandomSampler and after loading a batch of data I expected to have a target with one sample of each class but I got different:</p>
<pre><code class="lang-auto">weight = 1. / class_sample_count
samples_weight = weight[target]

sampler = WeightedRandomSampler(samples_weight, batch_size, replacement=True)

inputs, targets = next(iter(train_dl))     # Get a batch of training data
print(targets)
</code></pre>
<p>tensor([5, 3, 1, 4, 5, 5])</p>

username: charan_Vjy
cooked: <p>Below are examples from Pytorch’s forums which address your question.</p>
<blockquote>
<p>list(WeightedRandomSampler([0.1, 0.9, 0.4, 0.7, 3.0, 0.6], 5, replacement=True))<br>
Output: [0, 0, 0, 1, 0]<br>
list(WeightedRandomSampler([0.9, 0.4, 0.05, 0.2, 0.3, 0.1], 5, replacement=False))<br>
Output: [0, 1, 4, 3, 2]</p>
</blockquote>
<p>For a batch size &lt; no_of classes, using Replacement = False would generate independent samples. If batch size &gt; no_of classes, it would throw this error, <code>RuntimeError: cannot sample n_sample &gt; prob_dist.size(-1) samples without replacement</code>.</p>

username: Amir
cooked: <p>I’m so confused. Here is what I did and its result:</p>
<pre><code class="lang-auto">batch_size = 6
weight = 1. / class_sample_count
samples_weight = weight[target]
sampler = WeightedRandomSampler(samples_weight, batch_size, replacement=False)
print('samlper= ', list(sampler))
</code></pre>
<p>samlper=  [8857, 190, 210, 8028, 10662, 1685]</p>
<pre><code class="lang-auto">train_dl = torch.utils.data.DataLoader(train_ds, batch_size=batch_size, sampler=sampler, num_workers = 16)
valid_dl = torch.utils.data.DataLoader(valid_ds, batch_size=batch_size, shuffle=True, num_workers = 16)

inputs, targets = next(iter(train_dl))     # Get a batch of training data
print('targets= ', targets)
</code></pre>
<p>targets=  tensor([4, 5, 5, 0, 3, 3])</p>
<pre><code class="lang-auto">Loss after 15 Epochs:

Epoch [ 1/15], Step [  1, 1825], Loss: 1.5785
Epoch [ 2/15], Step [  1, 1825], Loss: 1.9562
Epoch [ 3/15], Step [  1, 1825], Loss: 1.8681
Epoch [ 4/15], Step [  1, 1825], Loss: 2.0667
Epoch [ 5/15], Step [  1, 1825], Loss: 1.9168
Epoch [ 6/15], Step [  1, 1825], Loss: 1.8286
Epoch [ 7/15], Step [  1, 1825], Loss: 1.9063
Epoch [ 8/15], Step [  1, 1825], Loss: 1.8187
Epoch [ 9/15], Step [  1, 1825], Loss: 1.6252
Epoch [10/15], Step [  1, 1825], Loss: 2.3157
Epoch [11/15], Step [  1, 1825], Loss: 1.7716
Epoch [12/15], Step [  1, 1825], Loss: 2.1706
Epoch [13/15], Step [  1, 1825], Loss: 1.7085
Epoch [14/15], Step [  1, 1825], Loss: 1.7478
Epoch [15/15], Step [  1, 1825], Loss: 2.0860
</code></pre>
<pre><code class="lang-auto">evaluate_model(model, valid_dl)
</code></pre>
<p>accuracy: 19.58</p>
<pre><code class="lang-auto">evaluate_model(model, train_dl)
</code></pre>
<p>accuracy: 0.00</p>
<p>It seems that something goes wrong!</p>

username: charan_Vjy
cooked: <p>This is interesting. The values in the batches are not unique in spite of  using replacement = False.<br>
As far as the loss is concerned, This could be down to a couple of problems. Try the following out</p>
<ul>
<li>Try out different learning rates (smaller than the one you are currently using).</li>
<li>Remove all regularization and momentum until the loss starts decreasing.</li>
<li>Check the inputs right before it goes into the model (detach and plot it). Check correspondance with labels.</li>
<li>You may also be updating the gradients way too many times as a consequence of a small batch size. As the targets are still not unique, you may as well keep a larger batch.</li>
</ul>