(how to iterate subset after random_split) TypeError: 'DataLoader' object is not subscriptable

username: Janine
cooked: <p>Hi all,</p>
<p>This might be a trivial error, but I could not find a way to get over it, my sincere appreciation if someone can help me here.<br>
I have run into <code>TypeError: 'DataLoader' object is not subscriptable</code> when trying to iterate through my training dataset after <code>random_split</code> the full set. This is how my full set looks like and how I randomly split it:</p>
<pre><code class="lang-auto">clean_loader.dataset
</code></pre>
<pre><code class="lang-auto">Dataset ImageFolderWithPaths
    Number of datapoints: 6929
    Root location: ./data/clean_images
    StandardTransform
Transform: Compose(
               Resize(size=128, interpolation=PIL.Image.BILINEAR)
               CenterCrop(size=(128, 128))
               ToTensor()
               Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
           )
</code></pre>
<pre><code class="lang-auto">for data in clean_loader:
    print(data)
    break
</code></pre>
<pre><code class="lang-auto">[tensor([[[[-0.3198, -0.3027, -0.5253,  ..., -0.2684,  0.5536,  1.3584],
          [-0.3369, -0.3883, -0.6965,  ..., -0.4739, -0.0629,  1.1187],
          [-0.6623, -0.6965, -0.9534,  ..., -0.4739, -0.6623,  0.1254],
          ...,
          [-0.2684,  0.9132,  1.2899,  ..., -0.5767, -0.2684,  0.2282],
          [ 0.6734,  0.7248,  1.1187,  ..., -0.3541,  0.2624,  0.6734],
          [ 0.8618,  0.6392,  0.9988,  ..., -0.3369,  0.4166,  0.6734]],

         [[-1.1954, -1.1253, -1.3354,  ..., -1.0728, -0.1800,  0.6954],
          [-1.1954, -1.2129, -1.4580,  ..., -1.2479, -0.7402,  0.4853],
          [-1.4930, -1.5105, -1.7206,  ..., -1.2304, -1.3179, -0.4601],
          ...,
          [-1.0028,  0.2927,  0.9055,  ..., -1.3004, -0.9853, -0.4776],
          [ 0.0126,  0.1527,  0.7479,  ..., -1.1429, -0.5126, -0.0924],
          [ 0.2402,  0.0826,  0.6429,  ..., -1.1253, -0.3725, -0.1099]],

         [[ 0.2348,  0.1825, -0.2010,  ...,  0.3393,  1.2805,  2.1346],
          [ 0.1651,  0.0605, -0.3404,  ...,  0.1825,  0.6182,  1.8383],
          [-0.2532, -0.3055, -0.6018,  ...,  0.2348, -0.0267,  0.7402],
          ...,
          [ 0.3742,  1.5420,  1.8905,  ...,  0.2173,  0.5659,  1.0714],
          [ 1.5420,  1.5594,  1.8731,  ...,  0.5136,  1.1585,  1.6117],
          [ 1.8383,  1.5420,  1.8383,  ...,  0.5834,  1.3851,  1.6814]]]]), tensor([0]), ('1',)]
</code></pre>
<p>I used random_split to split the full set:</p>
<pre><code class="lang-auto">clean_train_set, clean_test_set = torch.utils.data.random_split(clean_loader, [round(len(clean_loader)*0.8),(len(clean_loader) - round(len(clean_loader)*0.8))])
</code></pre>
<p>However, after the split, I cannot iterate through the subset:</p>
<pre><code class="lang-auto">for data in clean_train_set:
    print(data)
    break
</code></pre>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in 
----&gt; 1 for data in clean_train_set:
      2     print(data)
      3     break

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataset.py in __getitem__(self, idx)
    255 
    256     def __getitem__(self, idx):
--&gt; 257         return self.dataset[self.indices[idx]]
    258 
    259     def __len__(self):

TypeError: 'DataLoader' object is not subscriptable
</code></pre>
<p>I have also tried the <code>enumerate</code> way to do the iteration:</p>
<pre><code class="lang-auto">for step, data in enumerate(clean_train_set):
    print(data)
    break
</code></pre>
<p>But get the same error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in 
----&gt; 1 for step, data in enumerate(clean_train_set):
      2     print(data)
      3     break

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataset.py in __getitem__(self, idx)
    255 
    256     def __getitem__(self, idx):
--&gt; 257         return self.dataset[self.indices[idx]]
    258 
    259     def __len__(self):

TypeError: 'DataLoader' object is not subscriptable
</code></pre>
<p>I guess this might be an elephant-in-the-room issue, but I have googled thoroughly and cannot find a cure. Urgently need some help to get over this error, thanks a bunch!</p>

username: Janine
cooked: <p>I have also tried to pass the Subset to a DataLoader, but still no luck:</p>
<pre><code class="lang-auto">clean_train_loader = torch.utils.data.DataLoader(
    clean_train_set,
    batch_size=1,
    shuffle=True,
    num_workers=0,
)
</code></pre>
<pre><code class="lang-auto">for step, data in enumerate(clean_train_loader):
    print(data)
    break
</code></pre>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in 
----&gt; 1 for step, data in enumerate(clean_train_loader):
      2     print(data)
      3     break

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in __next__(self)
    343 
    344     def __next__(self):
--&gt; 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
    383     def _next_data(self):
    384         index = self._next_index()  # may raise StopIteration
--&gt; 385         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    386         if self._pin_memory:
    387             data = _utils.pin_memory.pin_memory(data)

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py in (.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataset.py in __getitem__(self, idx)
    255 
    256     def __getitem__(self, idx):
--&gt; 257         return self.dataset[self.indices[idx]]
    258 
    259     def __len__(self):

TypeError: 'DataLoader' object is not subscriptable
</code></pre>
<p>I can somehow iterate over the dataset using <code>clean_train_loader.dataset.dataset</code>, but it seems like it is actually the original full set which has been enumerated over, not the subset:</p>
<pre><code class="lang-auto">print(len(clean_loader))
print(len(clean_train_loader))
i = 0
for step, data in enumerate(clean_train_loader.dataset.dataset):
    i += 1
print(i)
</code></pre>
<pre><code class="lang-auto">6929
5543
6929
</code></pre>
<p>I found some discussion referring that by doing <code>data_loader.dataset.dataset</code> it traces back to the original full set, but if such being the case, how can I iterate through the subset per se? I am really confused here. Desperately need some help here. <img src="https://discuss.pytorch.org/images/emoji/apple/disappointed_relieved.png?v=9" title=":disappointed_relieved:" class="emoji" alt=":disappointed_relieved:"></p>

username: Janine
cooked: <p>BTW, I have also tried out the <code>SubsetRandomSampler</code>, but the results are somehow the same:</p>
<pre><code class="lang-auto">from torch.utils.data.sampler import SubsetRandomSampler
test_split = .2
shuffle_dataset = True
random_seed= 42

# Creating data indices for training and validation splits:
dataset_size = len(clean_loader)
indices = list(range(dataset_size))
split = int(np.floor(test_split * dataset_size))
if shuffle_dataset :
    np.random.seed(random_seed)
    np.random.shuffle(indices)
train_indices, test_indices = indices[split:], indices[:split]

# Creating PT data samplers and loaders:
train_sampler = SubsetRandomSampler(train_indices)
test_sampler = SubsetRandomSampler(test_indices)

train_loader = torch.utils.data.DataLoader(clean_loader, batch_size=1, 
                                           sampler=train_sampler)
test_loader = torch.utils.data.DataLoader(clean_loader, batch_size=1,
                                                sampler=test_sampler)
i = 0
for data in train_loader.dataset:
    i += 1
print(i)

i = 0
for step, data in enumerate(train_loader.dataset):
    i += 1
print(i)

i = 0
for (images, labels, index) in train_loader.dataset:
    i += 1
print(i)

for data in train_loader:
    print(data)
</code></pre>
<pre><code class="lang-auto">6929
6929
6929
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in 
----&gt; 1 for data in train_loader:
      2     print(data)

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in __next__(self)
    343 
    344     def __next__(self):
--&gt; 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
    383     def _next_data(self):
    384         index = self._next_index()  # may raise StopIteration
--&gt; 385         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    386         if self._pin_memory:
    387             data = _utils.pin_memory.pin_memory(data)

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py in (.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

TypeError: 'DataLoader' object is not subscriptable
</code></pre>
<p>Basically, I have tried every style of iteration approach I could find, but still with no luck</p>

username: Janine
cooked: <p>Solved the problem. It turns out need to pass  <code>clean_loader.dataset</code>  instead of  <code>clean_loader</code>  to  <code>DataLoader</code>. Just would like to document it here in case someone else also get caught with the same mistake.</p>