Per-Class and per-samples WeightSampler explanation needed

Alex_Ge: <p>Hello,</p>
<p>Let me start by saying I’ve searched, and searched and then searched some more. Not only on PyTorch but on Github and other sources.<br>
I believe what confused me is the fact I blindly copied samples from this forum.<br>
I have read topics and posts such as:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/balanced-sampling-between-classes-with-torchvision-dataloader/2703/17">Class Sample Counts</a></li>
<li><a href="https://discuss.pytorch.org/t/balanced-sampling-between-classes-with-torchvision-dataloader/2703/7">Weight Calculation</a></li>
<li><a href="https://discuss.pytorch.org/t/per-class-and-per-sample-weighting/25530/4">Per Class Per Sample Weights</a></li>
</ul>
<p>There seem to be three issues I would really appreciate clarity on:</p>
<ol>
<li>Does the <code>WeightSampler</code> take as argument the <strong>length</strong> of the training set, or the batch size?<br>
The behaviour changes dramatically if I (wrongly I presume) pass the batch size</li>
<li>How exactly should I calculate the weights for the <code>WeightSampler</code>? I’ve seen snippets from other users and from <a class="mention" href="/u/ptrblck">@ptrblck</a> and they all vary to the point where I think this has made me calculate them wrongly (see the snippet I’m posting below which is the actual code I use).</li>
<li>If I use a <code>WeightedSampler</code> does the <code>CE Loss</code> criterion and the way it is back-propagated change? I seen <a href="https://discuss.pytorch.org/t/per-class-and-per-sample-weighting/25530/4">a snippet from here</a> (again from <a class="mention" href="/u/ptrblck">@ptrblck</a>) which suggests that in the event of using a WeightedSampler with CE Loss, then the backprop should be different? Furthermore in the same snippet, constructing the <code>nn.CrossEntropyLoss</code> object seems to take as parameters the actual weights?</li>
</ol>
<p>The way I calculate the actual weights. Method <code>class_samples()</code> returns a list where each class/label has its rows in the dataset counted. Order is preserved and I have manually verified this.<br>
Most examples I have seen stop at line <span class="hashtag">#3</span> and don’t carry on with the <code>cat</code> operation.<br>
<strong>EDIT</strong> I tried removing the bottom three lines (<code>torch.cat</code> and such) and the network Loss drops to zero but Top-1 gets stuck at 50% so I am assuming this is the correct approach creating weight samples.</p>
<pre><code class="lang-auto">sample_index    = dataset.class_samples()
num_samples    = len(dataset)
classes_weight = 1. / torch.tensor(sample_index, dtype=torch.float)
target                 = torch.cat((torch.zeros(int(num_samples * 0.99), dtype=torch.long),
                                torch.ones(int(num_samples * 0.01), dtype=torch.long)))
samples_weight = torch.tensor([classes_weight[t] for t in target]) 
</code></pre>
<p>The remaining code is more or less templated:</p>
<ul>
<li>I use a ResNet34 for single class classification of PDF forms (uploaded by users) which are very well structured most of the time</li>
<li>I have a highly unbalanced dataset, some classes have 4000 samples, some only have 40, hence the use of the <code>WeightedSampler</code>
</li>
<li>I get very good Top-1 accuracy which is reproducible (99% to 100%) however  when doing cross-evaluation the SoftMax’ed output finds and approximates similarity between <code>unseen</code> input and <code>known</code> input when it really shouldn’t. There seems to be a high variability (more than 30%) of the Top-1 when testing with <code>unseen</code> input. I would assume this is too much generalisation/approximation whilst the network reports over-fitting, which makes no sense to me.</li>
</ul>
<p>Any help would be greatly appreciated, and hopefully would put the matter to rest.</p>

ptrblck: <ol>
<li>Usually you would set the length in <code>WeightedRandomSampler</code> to the length of your dataset. However, this is not a requirement and you could also use <code>replacement=True</code> and double the length, if that fits your use case.</li>
<li>Your code looks alright. Basically each sample should get its own weight. The usual approach would be to use the corresponding class index to assign the weight to the sample. Could you please post the posts with varying implementations, as we should fix them?</li>
<li>The <code>weight</code> argument in <code>nn.CrossEntropyLoss</code> is using a class weight on the loss and is orthogonal to weighted sampling. In the linked post, the user wanted a sample based loss weighting in its training, which is also a special case and unrelated to random sampling. (Although all approaches might be used to counter effects of an imbalanced dataset)</li>
</ol>

Alex_Ge: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for the reply (I owe you more than once at this point!)</p>
<ol>
<li>When you say <code>replacement=True</code> doubles the length, do you mean it will allow duplication (up-sampling) of the minority classes?</li>
<li>That is what I have understood, although I’m not clear what <code>torch.cat</code> does and why you assign 99% of them to zeros and 1% to ones. I’ll post the code below, including an explanation of the dataloader.</li>
<li>Similarly that was what I understood, although I’m guessing how it works I’d like to be certain. Is the class weight being used to account for imbalance by correctly attributing propagation error in a better ratio? If so, shouldn’t the sampler account for that imbalance and make the use of weights unnecessary?<br>
I have seen snippets of yours were you set the <code>CrossEntropyLoss</code> weights, the <code>reduction='none'</code> but then I am not very clear how you use the <code>criterion</code> and <code>loss</code> to correctly update the algorithm.</li>
</ol>
<p>The <code>dataloader</code> I have is using IP patent forms (uploaded by users) which are very structured documents, with small deviation (especially after resizing). To give you an idea, I’ve calculated the mean and std for the entire data-set to be (after resizing to 224x224):</p>
<pre><code class="lang-auto">RGB mean: 0.9497, 0.9497, 0.9497
RGB std: 0.0363, 0.0363, 0.0363
</code></pre>
<p>Which is kinda telling the story (since those values are normalised) most images are white (because of the background being a white page) and some sparse pixels are usually black, or less often coloured (logos, etc.). I don’t know if it would help pre-processing by removing the background or other techniques which may help more.</p>
<p>The loader places data tuples in a list of <code>(input.cuda().float(), output.cuda().long())</code> where input is the tensorified RGB image in 224x224 which is only resized and normalised. The output is a fixed class C length tensor, where each output activates one value (only one can be active so no multiclass).<br>
The actual labels are saved in an <code>self.index = OrderedDict</code> of tuples to actual output index, e.g., <code>(country, form) =&gt; output_index</code> for simplicity’s shake, so at any point I can retrieve the actual label by looking at the output and vice versa, all trivial stuff so far.</p>
<p>I calculate the class samples by using the following:</p>
<pre><code class="lang-auto">    def class_samples(self):
        label_samples = [None] * self.num_outputs()
        for i in range(len(self.index)):
            key        = self.get_label(i)
            count      = self.labels.count(key)
            label_samples[i] = count
        return label_samples

    def num_outputs(self):
        output_size = len(self.index)
        assert output_size &gt; 0
        return output_size

    def get_index(self, item: tuple) -&gt; int:
        return self.index.get(item)

    def get_label(self, idx) -&gt; tuple:
        for key, value in self.index.items():
            if value == idx:
                return key
        return None
</code></pre>
<p>You’ve already seen how I calculate class weights and then sample weights:</p>
<pre><code class="lang-auto">    def weights(self):
        sample_index   = self.class_samples()
        num_samples    = len(self)
        classes_weight = 1. / torch.tensor(sample_index, dtype=torch.float)
        target         = torch.cat((torch.zeros(int(num_samples * 0.99), dtype=torch.long),
                                    torch.ones(int(num_samples * 0.01), dtype=torch.long)))
        #
        # I am unsure what this does, it appears to be populating the samples weight based
        # on the classes weights ? However, I would assume that
        # the correct way of doing this would be to assign each sample, the correct corresponding
        # weight, based on which class it belongs to.
        #
        samples_weight = torch.tensor([classes_weight[t] for t in target])
        return (samples_weight, classes_weight)
</code></pre>
<p>Interestingly, when I unit test it, I see that one sample is not indexed/missing:</p>
<pre><code class="lang-auto">samples weights: torch.Size([4375]) 
classes weights: torch.Size([13])
total samples: 4376
</code></pre>
<p>The <code>weights</code> method is invoked before training:</p>
<pre><code class="lang-auto">(samples_w, classes_w) = dataset.weights()
classes_w = classes_w.cuda().float()
</code></pre>
<p>In the training script I do allow replacement and I do pass the class weights to CE Loss:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss(weight=classes_w, reduction='none').cuda()
sampler = torch.utils.data.sampler.WeightedRandomSampler(samples_w,
                                                         len(train_ds),
                                                         replacement=True)
</code></pre>
<p>Finally, when I do the back-prop:</p>
<pre><code class="lang-auto">for epoch in range(args.epochs):
    # batch/step iteration
    for i, (inputs, labels) in enumerate(train_loader):
        optimizer.zero_grad()
        logits        = net(inputs)
        ideal         = labels.argmax(1)
        loss          = criterion(logits, ideal)
        loss.mean().backward()
        optimizer.step()
    #
    scheduler.step()
</code></pre>
<p>Any help is greatly appreciated!</p>