Building custom dataset, how to return ids as well?

isalirezag: <p>so the format of a custom dataset should be like fllowing:</p>
<pre><code class="lang-auto">import torch
from torch.utils import data

class Dataset(data.Dataset):
  'Characterizes a dataset for PyTorch'
  def __init__(self, list_IDs, labels):
        'Initialization'
        self.labels = labels
        self.list_IDs = list_IDs

  def __len__(self):
        'Denotes the total number of samples'
        return len(self.list_IDs)

  def __getitem__(self, index):
        'Generates one sample of data'
        # Select sample
        ID = self.list_IDs[index]

        # Load data and get label
        X = torch.load('data/' + ID + '.pt')
        y = self.labels[ID]

        return X, y
</code></pre>
<p>I like to have have <code>ID</code> information in the output in addition to x and y. So i did <code>return X, y, ID</code><br>
, but now  when I do</p>
<pre><code class="lang-auto">data_loader = data.DataLoader(dataset, args.batch_size,
                                  num_workers=args.num_workers,
                                  shuffle=True )

batch_iterator = iter(data_loader)
images, targets, id  = next(batch_iterator)
</code></pre>
<p>I receive an error,<br>
anyone knows why?</p>

justusschock: <p>All data returned by a dataset needs to be a tensor, if you want to use the default collate_fn of the Dataloader. You have two options: write a custom collate function and pass it to the dataloader or wrap your ID inside a tensor (which is simpler I guess) and unwrap it outside the dataloader.</p>

isalirezag: <p>how can we wrap a string in a tensor? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=5" title=":thinking:" class="emoji" alt=":thinking:"></p>

justusschock: <p>Ah sorry, I implied your ID would be an integer. You cannot wrap a string to a tensor. I could think of some ways to achieve something like that, but it would not be very pytorch-like. If you are interested in these Ways you can PM me.</p>

JuanFMontesinos: <p>very weird way but u can just conver string into integers through an asci table and convert them back to an string calling a function.</p>
<p>got from internet</p>
<pre><code class="lang-auto">&gt;&gt;&gt; s = 'hi'
&gt;&gt;&gt; [ord(c) for c in s]
[104, 105]
</code></pre>

justusschock: <p>that’s what I thought about too. I also thought about wrapping the loader itself, but one would have to define a new iterator for this. I proposed another method, and if this method works (currently waiting for verification), I will post it here later on.</p>

justusschock: <aside class="quote no-group" data-username="justusschock" data-post="2" data-topic="22933">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/justusschock/48/4637_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/22933/2">On another topic</a></div>
<blockquote>
<p>While trying out my ideas, I think I came up with a very pytorch-like way:</p>
<p>Simply define a new <code>collate_fn</code> which calls the <code>default_collate</code> :</p>
<pre><code class="lang-auto">from torch.utils.data.dataloader import default_collate 


def id_collate(batch):
    new_batch = []
    ids = []
    for _batch in batch:
        new_batch.append(_batch[:-1])
        ids.append(_batch[-1])
    return default_collate(new_batch), ids
</code></pre>
<p>This could work If you pass it to your loader, since the ids (which have to be the last item in the tuple returned in the datasets <code>__getitem__</code> will be removed before calling the default collate and simply returned afterwards.</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/isalirezag">@isalirezag</a> reported this to work great.</p>

Igo312: <p>Sorry for late coming, but what I want to ask is does <code>collate_fn</code> can return a dict that some value is string now?</p>

FedericoCozziVM: <p>I think that another way to do this without building a custom collate function would be not to return the ID but directly the idx within the <strong>getitem</strong> implementation (which is numerical and can be treated in batches by the default collate function).  Something like:</p>
<pre><code class="lang-auto">class Dataset(data.Dataset):
  'Characterizes a dataset for PyTorch'
  def __init__(self, list_IDs, labels, retrun_idx: bool = False):
        'Initialization'
        self.labels = labels
        self.list_IDs = list_IDs
        self.return_idx = return_idx

  def __len__(self):
        'Denotes the total number of samples'
        return len(self.list_IDs)

  def __getitem__(self, index):
        'Generates one sample of data'
        # Select sample
        ID = self.list_IDs[index]

        # Load data and get label
        X = torch.load('data/' + ID + '.pt')
        y = self.labels[ID]

        if self.return_idx:
                return X, y, index
        return X, y
</code></pre>
<p>Then, you access externally to the list_IDs with the batch indexes</p>
<pre><code class="lang-auto">data_loader = data.DataLoader(dataset, args.batch_size,
                                  num_workers=args.num_workers,
                                  shuffle=True )

list_IDs = dataset.list_IDs

batch_iterator = iter(data_loader)
images, targets, idx  = next(batch_iterator)

ids = list_IDs[idx.cpu().numpy()]
</code></pre>