TypeError: 'Dataset' object does not support indexing

Niki: <p>Hi,</p>
<p>I want to concatenate testing samples and training samples (CIFAR-10), and then using this dataset in the test. I used <code>__getitem</code>  :</p>
<pre><code class="lang-auto">class MyTestDataset():
        def __init__(self, transform_test=None, transform_train=None):
            Train = datasets.CIFAR10(root='~/data', train=True,download=True,transform=transform_train)
            Test = datasets.CIFAR10(root='~/data', train=False,download=False,transform=transform_test)
            self.cifar_len = 10
        
            rand_idx = torch.randperm(len(Train.data))[:self.cifar_len]
            self.Train_data = Train.data[rand_idx]
            
            rand_idxt = torch.randperm(len(Test.data))[:self.cifar_len]
            self.Test_data = Test.data[rand_idxt]

        def __getitem(self, index):    
            x1, y1 = self.Train_data[index]
            
            x2, y2 = self.Test_data[index]
            
            x = torch.stack((x1, x2))
            y = torch.stack((torch.tensor(y1), torch.tensor(y2)))
            
            return x, y

        def __len__(self):
            return self.cifar_len + self.cifar_len
</code></pre>
<pre><code class="lang-auto"> dataset = MyTestDataset(transform_test=transform_test, transform_train=transform_train) 
    
loader = torch.utils.data.DataLoader(dataset,batch_size=64,shuffle=False, num_workers=8)
</code></pre>
<p>and in the test, I used:</p>
<pre><code class="lang-auto">for inputs, targets in loader:
</code></pre>
<p>but this gives error:</p>
<pre><code class="lang-auto">  File "...", line 339, in test
    for inputs, targets in loader:
  File "...", line 637, in __next__
    return self._process_next_batch(batch)
  File "...", line 658, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
TypeError: Traceback (most recent call last):
  File "...", line 138, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "...", line 138, in &lt;listcomp&gt;
    samples = collate_fn([dataset[i] for i in batch_indices])
TypeError: 'MyTestDataset' object does not support indexing
</code></pre>

ptrblck: <p>The <code>__getitem__</code> definition is missing the two underscores on the right hand side.</p>

Niki: <p>Thank you for your reply, <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>I fixed that, now it’s giving this error:</p>
<pre><code class="lang-auto">line 190, in __getitem__
    x1, y1 = self.Train_data[index]
ValueError: too many values to unpack (expected 2)
</code></pre>

ptrblck: <p><code>self.Train_data</code> only contains the data, not the target:</p>
<pre><code class="lang-python">self.Train_data = Train.data[rand_idx]
</code></pre>

Niki: <p>if I want both images and labels, what should I do? I used <code>rand_idx</code> to select random samples from train and test samples.</p>

ptrblck: <p>You could draw the data and target separately and then index them in <code>__getitem__</code>:</p>
<pre><code class="lang-python">rand_idx = torch.randperm(len(Train.data))[:10]
data = Train.data[rand_idx]            
target = torch.tensor(Train.targets)[rand_idx]
</code></pre>
<p>However, the usual approach would be to shuffle the indices, which will be passed to <code>__getitem__(self, index)</code>, so that you don’t have to shuffle it manually in your <code>__init__</code>.</p>

Niki: <p>So you mean if I change <code>shuffle=True</code> I can rewrite my code as</p>
<pre><code class="lang-auto">class MyTestDataset():
        def __init__(self, transform_test=None, transform_train=None):
            self.Train = datasets.CIFAR10(root='~/data', train=True,download=False,transform=transform_train)
            self.Test = datasets.CIFAR10(root='~/data', train=False,download=False,transform=transform_test)

        def __getitem__(self, index):    
            x1, y1 = self.Train[index]
            x2, y2 = self.Test[index]
            
            x = torch.stack((x1,x2))
            y = torch.stack((torch.tensor(y1), torch.tensor(y2)))
            
            return x, y, index

        def __len__(self):
            return len(self.Train)+len(self.Test)     
        

dataset = MyTestDataset(transform_test=transform_test, transform_train=transform_train) 
    
loader = torch.utils.data.DataLoader(dataset,batch_size=64,shuffle=True, num_workers=8)
</code></pre>
<p>but how can I limit index for the test data in <code>__getitem__</code>? now it gives this error:</p>
<pre><code class="lang-auto">in __getitem__
    img, target = self.data[index], self.targets[index]
IndexError: index 31041 is out of bounds for axis 0 with size 10000
</code></pre>

DAlolicorn: <p>Have you tried <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.ConcatDataset" rel="nofollow noopener">torch.utils.data.ConcatDataset</a> to concat the two datasets?<br>
I haven’t used it before, but I think it’s for this need?</p>
<p>By the way, the you are returning the concat of one training data and one testing data in your <code>__getitem__</code> method, thus you get an error since the training dataset is larger than the testing dataset.<br>
And if you code it this way, the length of the new dataset will not be the sum of two datasets.</p>

Niki: <p>Thank you for your guidance, <a class="mention" href="/u/dalolicorn">@DAlolicorn</a> .</p>
<p>I changed the <code>__len__</code> that the problem is solved.<br>
I also changed <code>x = torch.stack((x1,x2))</code> to <code>x = torch.cat((x1,x2),2)</code> since <code>torch.stack</code> was generating the images with dimension <code>torch.Size([64, 2, 3, 32, 32])</code>.</p>
<p>Regarding <code>ConcatDataset</code> , I am not sure by having the current <code>__getitem__</code> how I should feed datasets to ConcatDataset?</p>
<p>I also was trying:</p>
<p><code> for batch_idx, (inputs, targets, idx) in loader:</code></p>
<p>but it shows an error, this for loop is wrong?</p>

DAlolicorn: <p>Let me confirm that your objective is to get a dataset that contains both training and testing set of CIFAR-10?</p>
<pre><code class="lang-auto">Train = datasets.CIFAR10(root='~/data', train=True,download=True,transform=transform_train)
Test = datasets.CIFAR10(root='~/data', train=False,download=False,transform=transform_test)
new_set = torch.utils.data.ConcatDataset([Train, Test])
loader = torch.utils.data.DataLoader(new_set,batch_size=64,shuffle=True, num_workers=8)
for inputs, targets in loader:
    #do something
</code></pre>
<p>The <code>(inputs, tagets)</code> pair will be from either the training set or the testing set.<br>
It’s like having a huge dataset containing both training and testing data</p>

Niki: <p>Thank you for your reply, <a class="mention" href="/u/dalolicorn">@DAlolicorn</a>.</p>
<p>I want to have a concatenation of test and train with the size of the test dataset for CIFAR-10. <code>torch.utils.data.ConcatDataset</code> will provide the dataset with the size of both datasets? I mean 50000+10000?</p>
<p>I also looking for the index of images. Is there any way, to get the index in  <code>__getitem__</code>  for each train and test dataset, not both of them one index?</p>

DAlolicorn: <p>Yes, the output of <code>torch.utils.data.ConcatDataset</code> should contain 60000 samples.</p>
<p>As for the index, you mean:</p>
<pre><code class="lang-auto">for batch_idx, (inputs, targets) in enumerate(loader):
</code></pre>

Niki: <p>I am looking for the index of each image, I mean the index here:</p>
<pre><code class="lang-auto">def __getitem__(self, index):    
            x1, y1 = self.Train[index]
            x2, y2 = self.Test[index]
            
            x = torch.stack((x1,x2))
            y = torch.stack((torch.tensor(y1), torch.tensor(y2)))
            
            return x, y, index
</code></pre>
<p>but this index is for concatenated images, if I am correct? I want to track the index of each image apart from their concatenation.</p>

DAlolicorn: <p>So you want your dataset to output two images (one from the training dataset and one from the testing dataset) with the same index, and output the index as well?</p>
<p>I thought that you want to have a huge dataset containing both training and testing data, and you only output one image either from training or testing one time.</p>

Niki: <p>Yes, I want the dataset to output two images one from testing dataset and one from training dataset. I think that I was misunderstand the index part, it seems in <code>__getitem__</code> the index for both images will be the same, correct?</p>

DAlolicorn: <p>Is this what you need?</p>
<pre><code class="lang-auto">class MyTestDataset():
        def __init__(self, transform_test=None, transform_train=None):
            self.Train = datasets.CIFAR10(root='~/data', train=True,download=False,transform=transform_train)
            self.Test = datasets.CIFAR10(root='~/data', train=False,download=False,transform=transform_test)

        def __getitem__(self, index):    
            x1, y1 = self.Train[index]
            x2, y2 = self.Test[index]
            return x1, x2, y1, y2, index

        def __len__(self):
            return len(self.Test)     
        

dataset = MyTestDataset(transform_test=transform_test, transform_train=transform_train) 
    
loader = torch.utils.data.DataLoader(dataset,batch_size=64,shuffle=True, num_workers=8)

for inputs_train, inputs_test, targets_train, targets_test, idx in loader:
</code></pre>
<p>Then the len should be the size of the testing set (10000), since you want the index to be the same.<br>
This is what confuses me since you are not using the last 40000 data in the training dataset…</p>

Niki: <p>Thank you, <a class="mention" href="/u/dalolicorn">@DAlolicorn</a>.</p>
<p>I got the answer. I was looking for the index, that you mentioned by idx.<br>
I think as you said that <code>len(self.Test)</code> determined the size of dataset, which is 10000 in this case.</p>
<p>I also would like to know if I want to have testing data and the shuffled version of it (instead of training data), this is correct?</p>
<pre><code class="lang-auto">class MyTestDataset():
        def __init__(self, transform_test=None):
            self.Test = datasets.CIFAR10(root='~/data', train=False,download=False,transform=transform_test)
            self.cifar_len = 10
         
            rand_idx = torch.randperm(len(self.Test))[:self.cifar_len]
            self.Test_shuffeled = self.Test[rand_idx]
           

        def __getitem__(self, index):    
            x1, y1 = self.Test_shuffeled[index]
            x2, y2 = self.Test[index]
            x = torch.cat((x1,x2),2)
            y = torch.stack((torch.tensor(y1), torch.tensor(y2)))
            
            return x, y, index
        
        def __len__(self):
            return len(self.Test)
</code></pre>
<pre><code class="lang-auto"> loader = torch.utils.data.DataLoader(dataset,batch_size=64,shuffle=False, num_workers=8)
</code></pre>

DAlolicorn: <p>I think if there are no error message out, it should be correct.<br>
You can check by printing a pair of output, if they are not the same, it should be shuffled correctly.</p>
<p>Yet I would recommend just use <code>x1, y1 = self.Test[self.rand_idx[index]]</code>.<br>
In this case, you don’t need a new memory space for <code>self.Test_shuffeled</code>.</p>

Niki: <p>when I used</p>
<pre><code class="lang-auto">            rand_idx = torch.randperm(len(self.Test))[:self.cifar_len]
            self.Test_shuffeled = self.Test[rand_idx]
          
        def __getitem__(self, index):    
            x1, y1 = self.Test[index]
            x2, y2 = self.Test_shuffeled[index]
</code></pre>
<p>it gives this error:</p>
<pre><code class="lang-auto"> File "...", line 181, in __init__
    self.Test_shuffeled = self.Test[rand_idx]
  File "....", line 117, in __getitem__
    img, target = self.data[index], self.targets[index]
TypeError: only integer tensors of a single element can be converted to an index
</code></pre>
<p>and when I tried <code>self.Test[self.rand_idx[index]]</code> it gives this error:</p>
<pre><code class="lang-auto">IndexError: index 10 is out of bounds for dimension 0 with size 10
</code></pre>

DAlolicorn: <p>You can just use numpy:</p>
<pre><code class="lang-auto">self.rand_idx = np.arange(len(self.Test))
np.random.shuffle(self.rand_idx)
</code></pre>
<p>The second error is because the <code>self.cifar_len = 10</code>, your index should be less then 10.<br>
I don’t know why you limited your shuffled version to only contain the first 10 data, but if you need to do so, the len of dataset should then be 10, since now it’s 10000 it tries to request something like self.rand_idx[50] which doesn’t exists and gives you the error.</p>