Apply WeightedRandomSampler to VALIDATION/TEST splits makes sense?

username: pinazawa123
cooked: <p>Hello,</p>
<p>I’m currently working on a NLP multi-class classification problem, where I have an unbalanced dataset.</p>
<p>After some research, I found out that using <code>WeightedRandomSampler</code>, I could avoid the problem of always having the same biggest class being trained and predicted over and over again, with only sometimes other classes showing up.</p>
<p>( this wrong training leads to Val Loss going up, Train Loss and a high Val Accuracy going Down)</p>
<p><em><strong>My question is the following: Is it correct/fair to apply the <code>WeightedRandomSampler</code> in the TEST and validation datasets? Or it shouldn`t make any difference at all?</strong></em></p>
<p>Here is how I do my dataset</p>
<pre><code class="lang-auto">from torch.utils.data import WeightedRandomSampler

# create dataset from numpy
train_dataset = TensorDataset(tensor_x_train,tensor_y_train) 
valid_dataset = TensorDataset(tensor_x_valid,tensor_y_valid) 
test_dataset = TensorDataset(tensor_x_test,tensor_y_test) 

# Calculates weights of each SAMPLE
weights = 1. / torch.tensor(class_sample_count, dtype=torch.float)
weights = weights.double()

# Apply weights to each sample
sample_weight_train = weights[tensor_y_train]
sample_weight_test =weights[tensor_y_test]
sample_weight_val =weights[tensor_y_valid]

# create samplers
sampler_train = WeightedRandomSampler(
                              weights=sample_weight_train,
                              num_samples=len(sample_weight_train),
                              replacement=True)
sampler_test = WeightedRandomSampler(
                              weights=sample_weight_test,
                              num_samples=len(sample_weight_test))

sampler_val = WeightedRandomSampler(
                              weights=sample_weight_val,
                              num_samples=len(sample_weight_val))

# create Dataloader
train_dataloader = DataLoader(train_dataset,
                              batch_size=BATCH_SIZE,
                              sampler = sampler_train)
valid_dataloader = DataLoader(valid_dataset,
                              batch_size=BATCH_SIZE,
                             sampler = sampler_val) 
test_dataloader = DataLoader(test_dataset,
                             batch_size=BATCH_SIZE,
                            sampler = sampler_test) 
</code></pre>
<p>Now this code is how I measure the accuracy of my model in TEST set.</p>
<pre><code class="lang-auto">with torch.no_grad():
      for x_test, y_test in test_dataloader:
    
        y_pred = model(x_test)

        loss = criterion(y_pred, y_test)
        acc = binary_accuracy(y_pred, y_test) # Will return a number from 0.0 to 1.1

        epoch_loss += loss.item()
        epoch_acc += acc.item()

print("LOSS: ", epoch_loss/len(val_iter))
print("ACC: ",epoch_acc/len(val_iter))
</code></pre>

username: pinazawa123
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Could you please give me your opinion about this?<br>
Thanks</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="pinazawa123" data-post="1" data-topic="93733">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinazawa123/48/28008_2.png" class="avatar"> pinazawa123:</div>
<blockquote>
<p>My question is the following: Is it correct/fair to apply the <code>WeightedRandomSampler</code> in the TEST and validation datasets? Or it shouldn`t make any difference at all?</p>
</blockquote>
</aside>
<p>I don’t think applying a weighted sampling to the validation and test data would be a valid step.<br>
In my opinion the validation dataset is a proxy of the test dataset and should give you a signal how well your model would perform an new unseen data and can be used for e.g. early stopping.<br>
Once you’ve finished training the model and think the performance is good enough you would run a final iteration on the test dataset to get the model performance on completely new real world data.<br>
If your future task (e.g. when deploying the model) is also dealing with imbalanced data, which should be the case otherwise the test data wouldn’t be imbalanced, I would recommend to leave these datasets as they are.<br>
That being said, you might need to use another metric than the accuracy to measure the performance of your model due to the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">Accuracy Paradox</a>.</p>

username: pinazawa123
cooked: <p>Thanks! I will take a look at some other metrics too.</p>