Index error when using WeightedRandomSampler

username: Matej_Novosad
cooked: <p>Hi there, I noticed an imbalance between my classes and wanted to battle this. I decided to use WeightedRandomSampler instead of random shuffle in my data loader.<br>
Here is an example of the code I changed. Maybe the <code>Subset</code> class does not have <code>__len__</code> properly implemented by default?</p>
<p>Before:</p>
<pre><code class="lang-auto">trainloader = torch.utils.data.DataLoader(train_dataset, batch_size=8, shuffle=True,num_workers=0)
#train_dataset is a subset of the whole dataset
</code></pre>
<p>After:</p>
<pre><code class="lang-auto">with open("class_weights.txt", "r") as f:
    weights = [float(i) for i in f.readlines()]

sampler = WeightedRandomSampler(weights=weights, num_samples=5856, replacement=True)
trainloader = torch.utils.data.DataLoader(train_dataset, batch_size=8, sampler=sampler,num_workers=0)
#train_dataset is a subset of the whole dataset
</code></pre>
<p>After this change I get the following error.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "C:/Users/Matej/PycharmProjects/pneumonia_detection/normal_neural_network.py", line 153, in &lt;module&gt;
    train_net(net, trainloader, valloader, device, 30)
  File "C:/Users/Matej/PycharmProjects/pneumonia_detection/normal_neural_network.py", line 93, in train_net
    for i, data in enumerate(trainloader, 0):
  File "C:\Users\Matej\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\Users\Matej\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\Matej\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\Matej\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\Matej\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\utils\data\dataset.py", line 257, in __getitem__
    return self.dataset[self.indices[idx]]
IndexError: list index out of range
</code></pre>
<p>Thanks in advance for your answers and time.</p>

username: ptrblck
cooked: <p>Could you check the length of <code>weights</code> as well as <code>train_dataset</code>?<br>
Based on the error I guess you might have created more weights than samples contained in the dataset (Subset).</p>

username: Irfan_Ali
cooked: <p>Hi,<br>
Was this ever resolved. I am facing the same issue with WeightedRandomSampler. I have checked the length of weights (<code>len</code> is 9475) and train_dataset (<code>len</code> is 7580). My DataLoader works fine without sampler but with sampler I get<br>
<code>IndexError: list index out of range</code></p>
<p><code>sampler = WeightedRandomSampler(sample_weights, len(train_dataset), replacement=replacement, generator=None)</code></p>
<p>It doesn’t make sense that <code>weights</code> length is smaller than <code>num_samples</code> to draw or maybe I am missing something here.</p>

username: Irfan_Ali
cooked: <p>Actually I solved it. I was passing whole to dataset to <code>WeightedRandomSampler</code> instead of train_dataset</p>