WightedRandomSampler does not sample uniformly

username: damianofranzo
cooked: <p>Hi there!</p>
<p>I am facing some problem with WeightedRandomSampler since I want to sample <em>uniformly</em> data among several labels.</p>
<p>A really short description of the problem:<br>
The problem concerns plankton image classification. Data is tremendously unbalanced as the majority of images are just “detritus”(which is not even plankton).<br>
In numbers, the most common label is “detritus” with ~100k images in the dataset whether the least has just 12 samples. I want to have a high f1 “Macro” score, so I want to train my models through uniform distribution of labels.</p>
<p>Here I build the data loaders:</p>
<pre><code class="lang-python"># weights for weighted random sampler

X_ = plankton_df['objid']
y_ = plankton_df['level2']

X_train, X_test, y_train, y_test = train_test_split(X_, y_, test_size=0.15, stratify=y_)

weights_df = plankton_df[plankton_df['objid'].isin(X_train.values)] # assign weights only to samples in the training set
weights_df['level2'] = weights_df['level2'].map(label_mapping)
weights_df['count'] = weights_df.groupby('level2')['level2'].transform('count')
weights_df['count'] = 1. / (weights_df['count'])
weights = weights_df['count'].values
weights = torch.DoubleTensor(weights)
detritus_wrs = torch.utils.data.sampler.WeightedRandomSampler(weights, len(weights), replacement=True)

# Extract images in memory

plank_train_dataset = PlanktonDataset(X_train.values, y_train.values,
                                        img_files = img_files,
                                        label_mapping = label_mapping,
                                        transform = training_transformations,
                                        )
plank_train_dataloader = DataLoader(plank_train_dataset, batch_size=64, sampler=detritus_wrs, num_workers=4)

</code></pre>
<p>with the line of code “weights_df[‘count’] = 1. / (weights_df[‘count’])” I just want to make sure that for each sample it is assigned the inverse of the cardinality of his label.</p>
<p>But if I run this cell</p>
<pre><code class="lang-python">minibatch = next(iter(plank_train_dataloader))
print(minibatch['label'])
</code></pre>
<p>I get as output (labels are not uniform at all):</p>
<pre><code class="lang-auto">tensor([ 1,  0,  2,  4,  0,  0,  0,  2,  0, 15,  1,  3,  0,  0, 20, 11,  2,  0,
         0,  0,  0,  0,  1,  0,  0,  0,  2,  3,  1,  1,  2,  0, 12,  0,  0,  0,
         0, 15, 26,  0, 21,  0,  0, 27,  9,  2,  0,  3,  0,  0,  0,  2,  0,  0,
         0, 16,  0,  3,  0,  0,  1,  4,  0,  0])
</code></pre>
<p>Do you have any idea? It is more likely that there are some mistakes in my code rather than bugs on the library <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>
<p>I tried to be as more concise as possible, but If you need other pieces of code or other information of the problem just write here.</p>
<p>Thanks in advance for your consideration</p>

username: ptrblck
cooked: <p>I’m not sure what kind of padas magic you are using, but could you print some values of your <code>weights</code> tensor? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I assume you are assigning the inverse class count to each sample, is that correct?</p>

username: damianofranzo
cooked: <p>Thanks for the answer!</p>
<p>Your intuition is right, that’s exactly what I do.</p>
<pre><code class="lang-auto">print(weights_df[:10])
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">       objid  level2     count
0   32756761       0  0.000007
2   32758055      13  0.000463
3   32758988       5  0.000178
5   32760828       0  0.000007
6   32760820       0  0.000007
10  32760656       5  0.000178
11  32760819       1  0.000037
12  32760823       0  0.000007
13  32636402      17  0.001184
15  32761023       5  0.000178
</code></pre>
<p>N.B.Labels(<em>level2</em> is the label) are mapped into integer numbers. Label numbers are sorted according to their frequency in decreasing order. Label 0 is <em>detritus</em>, which is the label with ~100k samples. and label 5 is <em>silks</em> with 5629 samples.</p>
<p>If I print the values instead (which are the ones that I use into the WeightedRandomSampler) that’s what I get.</p>
<pre><code class="lang-auto">print(weights_df['count'].values[:10])
</code></pre>
<pre><code class="lang-auto">[7.22e-06 4.63e-04 1.78e-04 7.22e-06 7.22e-06 1.78e-04 3.71e-05 7.22e-06
 1.18e-03 1.78e-04]
</code></pre>
<p>Here I print the tensor:</p>
<pre><code class="lang-auto">print(weights[:10])
</code></pre>
<pre><code class="lang-auto">tensor([7.2234e-06, 4.6296e-04, 1.7764e-04, 7.2234e-06, 7.2234e-06, 1.7764e-04,
        3.7124e-05, 7.2234e-06, 1.1838e-03, 1.7764e-04], dtype=torch.float64)
</code></pre>

username: ptrblck
cooked: <p>Your code looks alright.<br>
Could you post the class counts or compare your code with <a href="https://github.com/ptrblck/pytorch_misc/blob/master/weighted_sampling.py" rel="nofollow noopener">this small example</a> and check for differences?</p>

username: damianofranzo
cooked: <p>I have run your code and it actually works on my cluster. It allows to perfectly balance between the two classes.<br>
Output:</p>
<pre><code class="lang-auto">target train 0/1: 990/10
batch index 0, 0/1: 48/52
batch index 1, 0/1: 49/51
batch index 2, 0/1: 48/52
batch index 3, 0/1: 54/46
batch index 4, 0/1: 52/48
batch index 5, 0/1: 51/49
batch index 6, 0/1: 47/53
batch index 7, 0/1: 53/47
batch index 8, 0/1: 48/52
batch index 9, 0/1: 55/45
</code></pre>
<p>I don’t see any particular difference between the two pieces of code. The only curious difference was the number of workers. But even setting it to one I don’t get a correct distribution.</p>

username: ptrblck
cooked: <p>That’s strange. Could you post the class counts, so that I could just create dummy data using your numbers?</p>

username: damianofranzo
cooked: <p>Thanks to your example, finally I got the error. As usual, it is a quite simple error.</p>
<p><code>X_train</code> and <code>weights_df</code> do not share the same order of samples, so weights were perfectly set just for <code>weights_df</code> but not for <code>X_train</code>.</p>
<p><code>X_train</code> is obtained from the function <code>train_test_split</code> with the option <code>stratify=y_</code>, which makes sure that the distributions of labels in training and validation set are set as close as possibles. Therefore the order of the dataset is not maintained since the previous option</p>
<p>In practice, I just substituted the line</p>
<pre><code class="lang-auto">weights_df = plankton_df[plankton_df['objid'].isin(X_train.values)] 
</code></pre>
<p>with</p>
<pre><code class="lang-auto">weights_df = pd.DataFrame({'objid':X_train.values, 'level2': y_train.values})
</code></pre>
<p>Thanks very much, your questions lead me to the solution <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>Awesome you’ve figured it out! Sounds like a nasty bug <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>