Dictionary in DataLoader

shivam2298: <p>if a Dataset return a dictionary in <strong>getitem</strong> function then how can I get batch of each of the dictionary item in my dataloader iterator loop? Is there any automatic way or do I have to extract manually each of the item of the dictionary for each of the sample in the batch.</p>

ptrblck: <p>If you return a <code>dict</code>, you would have to get all values using the specified keys in your loop:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.randn(10, 3, 24, 24)
        self.target = torch.randint(0, 10, (10,))
        
    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        
        return {'data': x, 'target': y}
    
    def __len__(self):
        return len(self.data)

dataset = MyDataset()
loader = DataLoader(
    dataset,
    batch_size=2,
    num_workers=2
)

for batch in loader:
    data = batch['data']
    target = batch['target']
    print(data.shape)
</code></pre>

shivam2298: <p>Thanks for the help.</p>

phucdoitoan: <p>Hi, is there any defined rules for the type of <strong>batch</strong> returned by DataLoader and the type of sample returned by <strong>getitem</strong>  in Dataset?<br>
For example: if <strong>getitem</strong> returns dict, <strong>batch</strong> will be dict of batches. If <strong>getitem</strong> returns tuple of items, <strong>batch</strong> will be tuple of batches …etc ???</p>

Sergius_Liu: <p>I am interested to know this also.<br>
There is a collador_fn function which decides how the loader handle different dataset.<br>
In the doc, default collador_fn only transfer np array to tensor without changing other format.<br>
…<br>
Dictionary seems to be collected as keys.<br>
No idea bout tuple or list.</p>
<p>Also want  to know.</p>

Minseok: <p>The item which is returned is dependent on the “return” code</p>

rui_zhang_331: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I wonder whether it is must needed to return the dict as this way?if we are doing something like below to directly return , is there problem? (each line is JSON string)</p>
<pre><code class="lang-auto">with open(current_file, mode='rb') as f:
            text = f.read().decode('utf-8')
            all_data.extend(text.split('\n'))

        json_data = []
        for line in all_data:
            try:
                json_data.append(json.loads(line))
            except:
                break
return json_data
</code></pre>

ptrblck: <p>Answered <a href="https://discuss.pytorch.org/t/how-does-dataloader-create-a-tensor-batch/55753/4">here</a>.</p>

Aldebaran: <p>I’ve got a strange behavior when using dictionary with Dataset and Dataloader.</p>
<pre><code class="lang-python">import torch
from torch.utils.data import DataLoader
from torchtext.data import Dataset
from transformers import AutoTokenizer 

    class MyDataset(Dataset):
        def __init__(self, tokenizer):
            self.tokenizer = tokenizer
            self.max_length = 8
            self.samples = [
                {"x1": "I lov 'd her soon , I lov 'd her late .", "x2": "I lov 'd her late , I lov 'd her soon"},
                {"x1": "I lov 'd her soon , I lov 'd her late .", "x2": "I lov 'd her late , I lov 'd her soon"},
                {"x1": "I lov 'd her soon , I lov 'd her late .", "x2": "I lov 'd her late , I lov 'd her soon"},
                {"x1": "I lov 'd her soon , I lov 'd her late .", "x2": "I lov 'd her late , I lov 'd her soon"}
            ]

        def _encode(self, sample):
            encoded = {
                "x1": self.tokenizer.encode(text=sample["x1"], max_length=self.max_length, pad_to_max_length=True),
                "x2": self.tokenizer.encode(text=sample["x2"], max_length=self.max_length, pad_to_max_length=True)
            }
            print(encoded)
            return encoded

        def __getitem__(self, idx):
            return self._encode(self.samples[idx])

        def __len__(self):
            return len(self.samples)

    tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
    dataset = MyDataset(tokenizer)
    loader = DataLoader(
        dataset,
        batch_size=2,
        num_workers=2
    )

    for batch in loader:
        x1 = batch['x1']
        print(x1)
        print(x1.shape)
</code></pre>
<p>which prints:</p>
<pre><code class="lang-python">Encoded Sample
 {'x1': [101, 1045, 8840, 2615, 1005, 1040, 2014, 102], 'x2': [101, 1045, 8840, 2615, 1005, 1040, 2014, 102]}

 {'x1': [101, 1045, 8840, 2615, 1005, 1040, 2014, 102], 'x2': [101, 1045, 8840, 2615, 1005, 1040, 2014, 102]}

 {'x1': [101, 1045, 8840, 2615, 1005, 1040, 2014, 102], 'x2': [101, 1045, 8840, 2615, 1005, 1040, 2014, 102]}

 {'x1': [101, 1045, 8840, 2615, 1005, 1040, 2014, 102], 'x2': [101, 1045, 8840, 2615, 1005, 1040, 2014, 102]}

batch x1
 [tensor([101, 101]), tensor([1045, 1045]), tensor([8840, 8840]), tensor([2615, 2615]), tensor([1005, 1005]), tensor([1040, 1040]), tensor([2014, 2014]), tensor([102, 102])]

Traceback (most recent call last):
  ...
print(x1.shape)
AttributeError: 'list' object has no attribute 'shape'
</code></pre>
<h4>Expected (or desired) behavior:</h4>
<pre><code class="lang-python"># first batch[x1]: torch.Size([2, 8])
tensor([
[101, 1045, 8840, 2615, 1005, 1040, 2014, 102],
[101, 1045, 8840, 2615, 1005, 1040, 2014, 102]
)]

# second batch[x1]: torch.Size([2, 8])
tensor([
[101, 1045, 8840, 2615, 1005, 1040, 2014, 102],
[101, 1045, 8840, 2615, 1005, 1040, 2014, 102]
])</code></pre>

ptrblck: <p>Could you change use random tensors for the code and update the snippet, please?<br>
I tried to reproduce it using this dummy <code>Dataset</code> and it seems to return the expected results:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.samples = [
            {"x1": torch.arange(10), 'x2': torch.arange(10, 20)},
            {"x1": torch.arange(10), 'x2': torch.arange(10, 20)},
            {"x1": torch.arange(10), 'x2': torch.arange(10, 20)},
            {"x1": torch.arange(10), 'x2': torch.arange(10, 20)},
        ]

    def __getitem__(self, idx):
        return self.samples[idx]

    def __len__(self):
        return len(self.samples)


dataset = MyDataset()
loader = DataLoader(
    dataset,
    batch_size=2,
    num_workers=0
)

for batch in loader:
    x1 = batch['x1']
    print(x1)
    print(x1.shape)
</code></pre>

Aldebaran: <p>I realize the difference now: your <code>samples</code> is a dictionary which values are tensors, not a list of integers as in my version.</p>

Z_Eamon: <p>there is a bug:<br>
def <strong>getitem</strong>(self, index):<br>
x = self.data[index]<br>
y = self.target[index]<br>
data_dict={‘data’: x, ‘target’: y}<br>
return data_dict<br>
then all item in the batch will be the same as the first item, which is wrong.<br>
I want to write this because sometime the data_dict is returned by a function, and it has so many keys.</p>

ptrblck: <p>I cannot reproduce the bug using some predefined tensors for <code>data</code> and <code>target</code>:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.arange(10).float().view(-1, 1)
        self.target = torch.arange(10).float().view(-1, 1) + 1
        
    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        data_dict={'data': x, 'target': y}
        return data_dict
    
    def __len__(self):
        return len(self.data)

dataset = MyDataset()
for batch in dataset:
    print(batch)

loader = DataLoader(dataset, batch_size=2, shuffle=True)
for batch in loader:
    print(batch)
</code></pre>
<p>Could you post an executable code snippet to reproduce this issue?</p>

aksg87: <p>Is there some magic happening in Dataset to make the return type of <code>dataset[1:2]</code> a <code>dict</code> and not <code>list</code>? I would think it would be a list!</p>

ptrblck: <p>The <code>default_collate</code> method uses some checks and in particular checks the input for <code>collections.abs.Mapping</code> <a href="https://github.com/pytorch/pytorch/blob/88a1e8eb0107dfa9bd0e63f5f7008459ac5d41a2/torch/utils/data/_utils/collate.py#L73-L74">here</a>, which then calls <code>default_collate</code> again on the values and returns a <code>dict</code>.</p>

aksg87: <p>Ah! Thanks. That’s very elegant <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

y-vectorfield: <p>I faced with <a href="https://discuss.pytorch.org/t/when-i-go-through-my-dataloader-i-find-nonetype-objects/131628/6">this problem</a>.<br>
Anyone knows the solution for this, please let me know.</p>