"index 1 is out of bounds for dimension 0 with size 1" error in pytorch custom dataset generator

Ashwath: <p>i’m trying to iterate through a dataloader created from a pytorch dataset. when I run the loop it gives me <strong>"index 1 is out of bounds for dimension 0 with size 1 error "</strong>. below is my dataset class<br>
class traindataset(Dataset):</p>
<pre><code class="lang-auto">class traindataset(Dataset):
    def __init__(self,data,train_end_idx,augmentation=None):
        '''
        data: data is a pandas dataframe generated from csv file where it has columns-&gt; [name,labels,col 1,col2,...,col784]. shape of data-&gt;(10000, 786)
        
        
        '''
        self.data=data
        self.augmentation=augmentation
        self.train_end=train_end_idx
        self.target=self.data.iloc[:self.train_end,1].values
        
    def __len__(self):
        return len(self.target);
    def __getitem__(self,idx):
        target=self.target
        image=self.data.iloc[:self.train_end,2:].values
        if self.augmentation is not None:
            image = self.augmentation(image)
        return torch.tensor(target[idx]),image[idx]

</code></pre>
<p>below is my augmenataion and dataloader generator</p>
<pre><code class="lang-auto">torchvision_transform = transforms.Compose([
    np.uint8,
    transforms.ToPILImage(),
    transforms.Resize((28,28)),
    transforms.RandomRotation([45,135]),
    transforms.ToTensor()
    ])
</code></pre>
<p>below is the loop I’m running, where I’m getting the mentioned error</p>
<pre><code class="lang-auto">for label,image in trainloader:
    print(label,train)
</code></pre>
<p>below is the complete error I have received.</p>
<pre><code class="lang-auto">IndexError                                Traceback (most recent call last)
/tmp/ipykernel_41/1540740000.py in &lt;module&gt;
----&gt; 1 for label,image in trainloader:
      2     print(label,train)

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    519             if self._sampler_iter is None:
    520                 self._reset()
--&gt; 521             data = self._next_data()
    522             self._num_yielded += 1
    523             if self._dataset_kind == _DatasetKind.Iterable and \

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    559     def _next_data(self):
    560         index = self._next_index()  # may raise StopIteration
--&gt; 561         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    562         if self._pin_memory:
    563             data = _utils.pin_memory.pin_memory(data)

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in &lt;listcomp&gt;(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/tmp/ipykernel_41/1814544681.py in __getitem__(self, idx)
     18         if self.augmentation is not None:
     19             image = self.augmentation(image)
---&gt; 20         return torch.tensor(target[idx]),image[idx]

IndexError: index 1 is out of bounds for dimension 0 with size 1
</code></pre>
<p>note: code works fine without augmentation.</p>

ptrblck: <p>Check the shape of <code>target</code> as it seems to have a size of 1 in <code>dim0</code>, while you are trying to index it with <code>idx = 1</code>.</p>

Ashwath: <p>since  i’m taking the .values of a dataframe its using the numpy array. and in <code>def __getitem__(self,idx</code>) if i return only <code>return torch.tensor(target[idx])</code> it works fine. and also i have printed the shape of target it’s <code>tensor([2000])</code></p>

ptrblck: <p>In that case I might have misinterpreted the error message and <code>image</code> is the culprit, which seems to have the aforementioned shape.</p>

Ashwath: <p>yes the problem is in the image after augmentation.<br>
image shape before augmentation-&gt;<code> (2000, 784)</code>-&gt;2000 rows of data passing from csv.<br>
image shape after augmentation-&gt; <code>([1, 28, 28])</code>–&gt;(28,28) is because in augmentation i’m using resize(28,28)<br>
image shape after augmentation is what, <code>get_item()</code> function is returning. i don’t know why augmentation is returning <code>[1,28,28]</code>  since my batch size is 2, i think it should be of shape[2,28,28]. i’m very new to pytorch can you please help me fix this?</p>

ptrblck: <p>The <code>Dataset.__getitem__</code> method will get a single index by default and will thus load and process a single sample.<br>
The <code>DataLoader</code> will pass <code>batch_size</code> indices one-by-one to the <code>__getitem__</code> method and create the batch afterwards using its <code>collate_fn</code>.<br>
So inside <code>__getitem__</code> treat <code>idx</code> as a single scalar which should load a single sample and target.</p>

Ashwath: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for patiently and quickly replying, I have made the below changes in my code but it still has the same issue.</p>
<pre><code class="lang-auto">class traindataset(Dataset):
    def __init__(self,data,train_end_idx,augmentation = None):
        '''
        data: pandas dataframe
        
        '''
        self.data=data
        self.augmentation=augmentation
        self.train_end=train_end_idx
        self.target=self.data.iloc[:self.train_end,1].values
        self.image=self.data.iloc[:self.train_end,2:].values #contains full data
        
    def __len__(self):
        return len(self.target);
    def __getitem__(self,idx):
        self.target=self.target
        image=self.image[idx].reshape(1,784) #only takes the selected index
        if self.augmentation is not None:
            image = self.augmentation(image)
        
        return torch.tensor(self.target[idx]),torch.tensor(image)
</code></pre>
<p>changes I have done are:</p>
<ol>
<li>in init I put my whole dataset rather than in getitem()-&gt;image=self.image[idx].reshape(1,784)</li>
<li>in get item I’m passing only one image by specifying its index<br>
-&gt;image=self.image[idx].reshape(1,784)</li>
</ol>

my3bikaht: <p><strong>get_item</strong> is dealing with just one image[idx]</p>
<p>It is better to move augmentation to <strong>init</strong> and transform all images altogether, not 1by1 in <em>get_item</em>_, also check that your augmentation doesn’t change shape to (1, 28, 28) where 1 is the number of channels.</p>

ptrblck: <p>Check the shape of <code>self.image</code> in the <code>__getitem__</code> and make sure you can properly index it in <code>dim0</code>.</p>
<p><a class="mention" href="/u/my3bikaht">@my3bikaht</a> NIT: if your data augmentation applies random transformations, applying it in the <code>__init__</code> would transform each sample only once and no random augmentation would be used.</p>