Issues with torch.utils.data.random_split

mariosfourn: <p>I’m getting an error when I use a DataLoader based on split dataset through <code>torch.utils.data.random_split()</code>.<br>
I have tried creating  a dataloader with the un-split dataset and everything works fine. So I assume it’s something do with the splitting.</p>
<p>I have a custom dataset defined as:</p>
<pre><code class="lang-auto"> class AntsDataset(Dataset):

    def __init__(self, root_dir, csv_file, transform=None):
        """
        Args:
            csv_file (string): Path to the csv_file with rotations
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.rotations = pd.read_csv(csv_file,header=None)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.rotations)

    def __getitem__(self, idx):
        #import ipdb; ipdb.set_trace()
        img_name = os.path.join(self.root_dir,
                                self.rotations.iloc[idx, 0])
        image = plt.imread(img_name,format='RGB')
        rotation = self.rotations.iloc[idx, 1].astype('float')

        if self.transform is not None:
            image=self.transform(image)

        return (image, rotation)
</code></pre>
<p>I then create a dataset, split and form a data_loader:</p>
<pre><code class="lang-auto">ants_dataset=AntsDataset(ants1_root_dir, ants1_rot_file,
        transform=transforms.Compose([transforms.ToPILImage(),
        transforms.Resize((120,120)),
        transforms.RandomCrop(size=100, pad_if_needed=True),
        transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.1, hue=0.07),
        transforms.ToTensor()]))

dataloader=torch.utils.data.DataLoader(ants_dataset,
        batch_size=10, shuffle=True)

train_length=int(0.7* len(ants_dataset))

test_length=len(ants_dataset)-train_length

train_dataset,test_dataset=torch.utils.data.random_split(ants_dataset,(train_length,test_length))

dataloader_train=torch.utils.data.DataLoader(train_dataset,
        batch_size=10, shuffle=True)

for batch_idx, (data,rotations) in enumerate(dataloader_train):
    print(rotations)
</code></pre>
<p>When I try to loop over the dataloader I get the following error:</p>
<pre><code class="lang-auto">--------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-82-f629e71651de&gt; in &lt;module&gt;()
      1 dataloader_train=torch.utils.data.DataLoader(train_dataset,
      2         batch_size=10, shuffle=True)
----&gt; 3 for batch_idx, (data,rotations) in enumerate(dataloader_train):
      4     print(rotations)

~/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    312         if self.num_workers == 0:  # same-process loading
    313             indices = next(self.sample_iter)  # may raise StopIteration
--&gt; 314             batch = self.collate_fn([self.dataset[i] for i in indices])
    315             if self.pin_memory:
    316                 batch = pin_memory_batch(batch)

~/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py in &lt;listcomp&gt;(.0)
    312         if self.num_workers == 0:  # same-process loading
    313             indices = next(self.sample_iter)  # may raise StopIteration
--&gt; 314             batch = self.collate_fn([self.dataset[i] for i in indices])
    315             if self.pin_memory:
    316                 batch = pin_memory_batch(batch)

~/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataset.py in __getitem__(self, idx)
    101 
    102     def __getitem__(self, idx):
--&gt; 103         return self.dataset[self.indices[idx]]
    104 
    105     def __len__(self):

&lt;ipython-input-44-01f6586a3276&gt; in __getitem__(self, idx)
     20         #import ipdb; ipdb.set_trace()
     21         img_name = os.path.join(self.root_dir,
---&gt; 22                                 self.rotations.iloc[idx, 0])
     23         image = plt.imread(img_name,format='RGB')
     24         rotation = self.rotations.iloc[idx, 1].astype('float')

~/anaconda3/envs/pytorch/lib/python3.6/site-packages/pandas/core/indexing.py in __getitem__(self, key)
   1470             except (KeyError, IndexError):
   1471                 pass
-&gt; 1472             return self._getitem_tuple(key)
   1473         else:
   1474             # we by definition only have the 0th axis

~/anaconda3/envs/pytorch/lib/python3.6/site-packages/pandas/core/indexing.py in _getitem_tuple(self, tup)
   2011     def _getitem_tuple(self, tup):
   2012 
-&gt; 2013         self._has_valid_tuple(tup)
   2014         try:
   2015             return self._getitem_lowerdim(tup)

~/anaconda3/envs/pytorch/lib/python3.6/site-packages/pandas/core/indexing.py in _has_valid_tuple(self, key)
    220                 raise IndexingError('Too many indexers')
    221             try:
--&gt; 222                 self._validate_key(k, i)
    223             except ValueError:
    224                 raise ValueError("Location based indexing can only have "

~/anaconda3/envs/pytorch/lib/python3.6/site-packages/pandas/core/indexing.py in _validate_key(self, key, axis)
   1965             l = len(self.obj._get_axis(axis))
   1966 
-&gt; 1967             if len(arr) and (arr.max() &gt;= l or arr.min() &lt; -l):
   1968                 raise IndexError("positional indexers are out-of-bounds")
   1969         else:

TypeError: len() of unsized object 
</code></pre>

ptrblck: <p>The error points to <code>pandas</code> and the <code>self.rotations.iloc[idx, 0]</code> call in particular.<br>
Could you print out <code>idx</code> and use this last <code>idx</code> which causes this error to call the function manually, i.e.:</p>
<pre><code class="lang-python">ants_dataset.rotations.iloc[idx, 0]
</code></pre>

mariosfourn: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="22298">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>auses this error to call the function manually, i.e.:</p>
</blockquote>
</aside>
<p>I found what the issue is: <code>torch.utils.data.random_split()</code> returns <code>idx</code> as <code>torch.Tensor</code> rather than a float. I assume this is a bug?</p>

ptrblck: <p>I’m not sure if that’s the issue here, as it seems your <code>Dataset</code> was successfully split.<br>
Could you try to get a single sample after the splitting using:</p>
<pre><code class="lang-python">train_dataset[0]
test_dataset[0]
</code></pre>
<p>Also, were you successful in getting the <code>idx</code> before the crash occurs?</p>

mariosfourn: <p>I get exactly the same error, when I try that. I put an <code>ipdb.set_trace</code>  in the custom dataset class to inspect the incoming <code>idx</code>. See image attached <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/2f6a50d339d7f551573b7e667173dbbf6455a344.png" data-download-href="https://discuss.pytorch.org/uploads/default/2f6a50d339d7f551573b7e667173dbbf6455a344" title="23"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/2f6a50d339d7f551573b7e667173dbbf6455a344_2_690x336.png" alt="23" data-base62-sha1="6LsdJqZxnNVEst4ABuv8n9T9KYY" width="690" height="336" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/2f6a50d339d7f551573b7e667173dbbf6455a344_2_690x336.png, https://discuss.pytorch.org/uploads/default/original/2X/2/2f6a50d339d7f551573b7e667173dbbf6455a344.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/2/2f6a50d339d7f551573b7e667173dbbf6455a344.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">23</span><span class="informations">996×486 52.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>As a workaround, could you try using <code>img_name = os.path.join(self.root_dir, self.rotations.iloc[idx.item(), 0])</code>?<br>
I’ll try to dig into this issue.</p>

mariosfourn: <p>I’ve tried this already and it works. But If I create a DataLoader on the original dataset I will need a condition to check whether <code>idx</code> is a float or a tensor. Thanks for looking into it.</p>

lkhphuc: <p>I have the same problem happens in the pandas core indexing.<br>
<code>Type Error: object of type 'numpy.int64' has no len()</code><br>
Any update on this?</p>

praateekmahajan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> looks like <a class="mention" href="/u/mariosfourn">@mariosfourn</a> was correct while saying</p>
<blockquote>
<p><code>torch.utils.data.random_split()</code>  returns  <code>idx</code>  as  <code>torch.Tensor</code> rather than a float.</p>
</blockquote>
<p>As per the example in question, indexing <code>ants_dataset</code> would work correctly but an error would be raised if accessing an index for <code>train_dataset</code>.</p>
<p>This could be resolved by adding <code>idx = idx.item()</code> but this would make indexing <code>ants_dataset</code> not functional.</p>
<p>A quick hack would be to have</p>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        try:
          idx = idx.item()
        except:
          idx = idx
        return self.data.iloc[idx]
</code></pre>
<p>This way we’ll be able to access both <code>ants_dataset</code> as well as <code>train_dataset</code> using indices.</p>

ptrblck: <p>As far as I remember the reason for this behavior was a potential bug in pandas.<br>
There was also a <a href="https://github.com/pytorch/pytorch/pull/9237" rel="nofollow noopener">PR</a> which wasn’t merged due to potential performance issues.</p>
<p>Your workaround might work, so thanks for posting it here. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Ashima_Garg: <pre><code class="lang-auto">import torch
from torchvision.datasets import MNIST
transform = transforms.Compose([transforms.ToTensor(), 
                                        transforms.Normalize((0.5,), (0.5,))])
dataset = MNIST(root = './data', train = train, transform = transform, download=True)
train_set, val_set = torch.utils.data.random_split(dataset, [50000, 10000])
</code></pre>
<p>The function of random_split to split the dataset is not working. The size of train_set and val_set returned are both 60000 which is equal to the initial dataset size.</p>
<p>A similar error is also reported on stack overflow:<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/57789645/torch-utils-data-random-split-is-not-splitting-the-data" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/9625519/user9625519" target="_blank" rel="nofollow noopener">
    <img alt="user9625519" src="https://www.gravatar.com/avatar/b081342eb633703ad7be65384ca09295?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/57789645/torch-utils-data-random-split-is-not-splitting-the-data" target="_blank" rel="nofollow noopener">torch.utils.data.random_split() is not splitting the data</a>
</h4>

<div class="tags">
  <strong>deep-learning, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/9625519/user9625519" target="_blank" rel="nofollow noopener">
    user9625519
  </a>
  on <a href="https://stackoverflow.com/questions/57789645/torch-utils-data-random-split-is-not-splitting-the-data" target="_blank" rel="nofollow noopener">01:37PM - 04 Sep 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Please look into the issue.<br>
Thanks.</p>

ptrblck: <p>Your code snippet works for me (PyTorch <code>1.4.0.dev20191109</code>, torchvision <code>0.5.0a0+28003e9</code>) and yields:</p>
<pre><code class="lang-python">print(len(train_set))
&gt; 50000
print(len(val_set))
&gt; 10000
</code></pre>
<p>Which versions are you using?</p>

Ashima_Garg: <p>torch==1.3.1<br>
torchvision==0.4.2</p>

Ashima_Garg: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, the issue is resolved. It is working for me also. I was finding the length using:</p>
<pre><code class="lang-auto">print(len(train_set.dataset)) 
</code></pre>
<p>which gives the length of the parent dataset. I wanted to convert the object of Subset class to Dataset object.<br>
Is there a way to convert the Subset to Dataset object?</p>

ptrblck: <p><code>Subset</code> wraps the <code>Dataset</code> for the reason to apply the specified indices to get the subset of the samples.<br>
What is your use case that you want to revert it?<br>
You can pass the <code>Subset</code> to a <code>DataLoader</code>, if that’s an issue.</p>

Ashima_Garg: <p>I want to fetch the entire dataset in x and y. Dataset class has .data and .targets attributes which resolves the purpose.<br>
DataLoader with batch size set to len(dataset) will be used.</p>

ptrblck: <p>The <code>DataLoader</code> will use the length of the <code>Subset</code>, not the underlying <code>Dataset</code>.<br>
If you want to fetch underlying data to process it or for some other use case, I would recommend to use it before splitting.</p>
<p>Could you explain your use case a bit more, so that I understand, why you need to access the internal data after splitting?</p>

Ashima_Garg: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
My use case is to first divide the dataset into two different subsets, then for each subset,<br>
Each subset should have the <code>__getitem__</code> function such that, to load a batch of samples, the <code>__getitem__</code> function to return pair of samples and these pair of samples belong to the same class, i.e. batch of 4 would mean a total of 8 samples. These are paired samples belonging to the same class.</p>
<p>Example: from MNIST Dataset, a batch would mean (1, 1), (2, 2), (7, 7) and (9, 9).</p>
<p>Your post on <a href="https://discuss.pytorch.org/t/torch-utils-data-dataset-random-split/32209/3" class="inline-onebox">Torch.utils.data.dataset.random_split</a> resolves the issue of dividing the dataset into two subsets and using the <code>__getitem__</code> function for the individual subsets. But Can you help with the workaround of using index  in <code>__getitem__</code> to return the pairs from the same class.</p>
<p>Thanks.</p>

akschougule: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I had a same issue as Ashima. It seems I was checking the len(dataloader.dataset). However the dimensions still doesn’t look right. I am trying to split 200k rows into 160k of train and 40k of val.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fd3aa5b826d987b1d5f2729f21997b918cba0eb2.png" data-download-href="https://discuss.pytorch.org/uploads/default/fd3aa5b826d987b1d5f2729f21997b918cba0eb2" title="Screenshot from 2020-01-19 15-05-59"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd3aa5b826d987b1d5f2729f21997b918cba0eb2_2_690x469.png" alt="Screenshot from 2020-01-19 15-05-59" data-base62-sha1="A8arZ4J9lhAMGxNdn46IS5vlU2K" width="690" height="469" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd3aa5b826d987b1d5f2729f21997b918cba0eb2_2_690x469.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd3aa5b826d987b1d5f2729f21997b918cba0eb2_2_1035x703.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/d/fd3aa5b826d987b1d5f2729f21997b918cba0eb2.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-01-19 15-05-59</span><span class="informations">1100×749 75 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am not sure why I see 40k and 10k.</p>

Sarah_W: <p>Hi, I would like to split my dataset in a train and validation part, where both subset indices should be in range of 0 to len(train_data) or 0 to len(validation_data). Is there a method that I can use for this?</p>