Can dataloader fetch me a batch of lists?

username: Ma_John
cooked: <p>The input to my model is images of varying length sequence, so I store them in a list.<br>
My question is, when the batch_size of my dataloader is 8, can it fetch me 8 lists of images?<br>
My dataset is defined as follows:</p>
<pre><code class="lang-auto">class mydataset(object):

    def __init__(self, root, train_dir, label_dir, img_num, transform, training=True):
        super(mydataset, self).__init__()
        self.root = root
        self.transform = transform
        self.train_dir = self.root + train_dir
        self.label_dir = self.root + label_dir
        self.img_num = img_num

        self.img_paths = []

        for i in range(1, img_num + 1):
            if not training:
                i = i + 530
            current_train_list = []
            current_train_dir_path = self.train_dir + '/' +  str(i) + '/'
            img_name_list = os.listdir(current_train_dir_path)
            for img_name in img_name_list:
                current_train_img_path = current_train_dir_path + img_name
                current_train_list.append(current_train_img_path)
                

            label_img_path = self.label_dir + "/" +str(i)+".JPG"


            self.img_paths.append({
                "train_img_path_list": current_train_list,
                "label_img_path": label_img_path
                })

    def __len__(self):
        return self.img_num

    def __getitem__(self, index):
        paths = self.img_paths[index]
        train_img_path_list = paths["train_img_path_list"]
        train_img_list = []

        for img_path in train_img_path_list:
            train_img_list.append(Image.open(img_path).convert('RGB'))

        label_img = Image.open(paths["label_img_path"]).convert('RGB')
        seed = np.random.randint(0, 2 ** 16)  # make a seed with numpy generator
        random.seed(seed)
        
        for i in range(len(train_img_list)):
            random.seed(seed)
            train_img_list[i] = self.transform(train_img_list[i])

        random.seed(seed)
        label_img = self.transform(label_img)

        sample = {"train_img_list": train_img_list, "label_img": label_img}
        return sample
</code></pre>
<p>data fetching:</p>
<pre><code class="lang-auto">for j, dataa in enumerate(trainloader):
    exposure_sequence_list, Label = dataa['train_img_list'], dataa['label_img']
</code></pre>
<p>when batch_size = 8, label is of shape [batch_size, channel, height, weight]<br>
and exposure_sequence is supposed to be 8 lists,</p>
<pre><code class="lang-auto">   print(type(exposure_sequence_list))
   print(len(exposure_sequence_list))
</code></pre>
<p>However, when I print them, type is <a>class:List</a>, and len is 7(one of the lists’ length)</p>

username: tom
cooked: <p>So what happens is that the default collate function will get a list <code>batch</code> of one list per batch item and the use <code>zip(*batch)</code> to get a list of batches. This is clearly not what you want, so you need to implement your own collate function and pass it to the dataloader.</p>
<p>You can use the <a href="https://github.com/pytorch/pytorch/blob/f6692146e771bd37d7e3483e2af58c236ffc96ca/torch/utils/data/_utils/collate.py#L42" rel="nofollow noopener"><code>default_collate</code> function</a> as inspiration for how to write one and handle the dicts.</p>
<p>Best regards</p>
<p>Thomas</p>