WeightedRandomSampler with multi-dimensional batch

Saswat: <p>I’m working on a classification problem (100 classes) and my dataset has a huge class imbalance. To tackle this, I’m considering using torch’s <code>WeightedRandomSampler</code> to oversample the minority class. I took help from <a href="https://discuss.pytorch.org/t/is-there-a-limit-on-how-disbalanced-a-train-set-can-be/26334/6">this post</a> which seemed pretty straightforward. Only concern with this is the nature of my dataset.</p>
<p>In my case, each sample (1 point in a batch) contains 8 points. Each of these 8 points have one true class out of 100 classes. So my output shape is like this: <code>(bs x 8)</code>. Hence, the final weight variable has <code>total_dataset_length*8</code> length.</p>
<p>Here’s my implementation:</p>
<pre><code class="lang-auto">y_org = np.load('target.npy')                 # 5000 x 8
samples_per_class = np.unique(y_org.ravel(), return_counts=True)[1]

class_weights = class_weight.compute_class_weight(class_weight='balanced', \
                                                  classes=np.unique(y_org.ravel()), \
                                                  y=y_org.ravel())
weights = class_weights[y_org.ravel()]
sampler = WeightedRandomSampler(weights, len(y_org.ravel()), replacement=True)
</code></pre>
<p>To count the number of occurrences of class index, I have to unroll (ravel) the ground truth array along the first dimension. Since the final <code>weight</code> variable has <code>total_dataset_length*8</code> , it causes indexing errors during loading</p>
<p><code>IndexError: list index out of range</code></p>
<p>How can I use WeightedRandomSampler in such cases?</p>

ptrblck: <p>Could you explain why the samples are created as <code>[batch_size, 8, ...]</code> tensors?<br>
Assuming each “point” is independent, you should be able to flatten the weights and sample each point in the <code>__getitem__</code> separately. Then in the training loop you might want to create a view in your desired shape.<br>
However, if each call into <code>__getitem__</code> creates a tensor of <code>[1, 8]</code>, then you would have to think about an approach to create a proper weight value for all 8 points.</p>

Saswat: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> The problem that I’m working on, concerns assigning a class to a bunch (of 8) vectors in 8 different bins. These 8 vectors are independent of each other and are procured directly from <code>__getitem__</code> function itself. Here, each “point” comprises 8 vectors, and the dimensions of 1 point are <code>1x8</code></p>

ptrblck: <p>Does it mean the actual target is created in the <code>__getitem__</code> and not known before?<br>
If so, I doubt you could easily use a weighted sampler, since you would need to know the targets to be able to add a weight to each index.</p>

Saswat: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Yes, the target is created inside the <code>__getitem__</code> function. It has previously been stored in a .npy file.</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
    def __init__(self, A_path, B_path, targetA_path, targetB_path):

        self.matA = np.load(A_path)             # (65098, 8, 4)
        self.matB = np.load(B_path)             # (65098, 500, 4)
        self.targetA = np.load(targetA_path)    # (65098, 8, 500)
        self.targetB = np.load(targetB_path)    # (65098, 8)
                
    def __len__(self):
        return self.matA.shape[0]
        
    def __getitem__(self, index):
        return torch.from_numpy(self.matA[index]), \
                torch.from_numpy(self.matB[index]), \
                torch.from_numpy(self.targetA[index]), \
                torch.from_numpy(self.targetB[index])
</code></pre>
<p>This is how the dataset class is defined. Here, 65098 denotes the number of datapoints, 500 the number of classes, 4 is the dimensionality of each input vector point.</p>
<p>I’m trying to calculate the weight matrix across <code>targetA</code> and then sample accordingly</p>

ptrblck: <p>Sorry, I misunderstood your use case as it seems the targets are not generated in the <code>__getitem__</code>, but just loaded.<br>
Also it seems that the <code>self.matX</code> are just indexed and contain multiple values.<br>
You are generally supposed to add a weight to each sample which is indexed via the passed <code>index</code> to <code>__getitem__</code>. I’m thus unsure why you would need to flatten one of the input tensors since you are directly indexing them via <code>[index]</code>.</p>

Saswat: <p>The class indices in targetA are in this form:</p>
<pre><code class="lang-auto">[5,5,5,5,4,5,5,5]
[99,0,0,0,1,2,2,2]
[99,99,99,99,99,99,99,99]
[1,1,1,1,2,2,76,76]
and so on ...
</code></pre>
<p>So flattening it gets me the number of occurrences for each class and then weighing becomes possible</p>

ptrblck: <p>Since each sample contains multiple classes, I don’t think you can easily add a weight to it, since again each <code>index</code> will be drawn using the weight. You might need to apply a more advanced technique as described <a href="https://discuss.pytorch.org/t/unbalanced-data-in-multiple-object-detection/88104/2">here</a> or just return samples containing a single target.</p>

Saswat: <p>Sure. I’ll look into it. Thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>