Dataset loader not loading entire dataset into one batch

username: ralph_20
cooked: <p>Hi, I’m trying to make a custom data loader that can apply transforms on my .npy image files. I’ve set the batch size to length of the array to get all the samples in one batch. However when I’m printing shape of the torch tensor after iterating throught the train loader, I’m getting the output as <code>torch.Size([3, 3, 64, 64])  </code> . I actually want it to be ‘torch.Size([61578, 3, 64, 64])’</p>
<pre><code class="lang-auto">import numpy as np
import torch
from torch.utils.data import Dataset, TensorDataset

import torchvision
import torchvision.transforms as transforms

from torchvision import datasets, transforms
from PIL import Image

import matplotlib.pyplot as plt



# Images
X_train = np.load('/content/galaxy_zoo/galaxy_zoo_train.npy')

X_train = torch.tensor(X_train)
X_train = torch.reshape(X_train, (len(X_train),3,64, 64))



class CustomTensorDataset(Dataset):
    """TensorDataset with support of transforms.
    """
    def __init__(self, tensors, transform=None):
        self.tensors = tensors
        self.transform = transform

    def __getitem__(self,index):
        x = self.tensors[index]

       
        x = self.transform(x)
        return x

    def __len__(self):
        return self.tensors[0].size(0)


# Let's add some transforms

so2_transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomAffine(90, translate=(0., 0.)),
    transforms.ToTensor(),
    #transforms.Normalize((0.1307,), (0.3081,))]  # mean and standard deviation, respectively, for normalization
])

se2_transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomAffine(90, translate=(0.25, 0.25)),
    transforms.ToTensor(),
    #transforms.Normalize((0.1307,), (0.3081,))]  # mean and standard deviation, respectively, for normalization
])
vanilla_transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.ToTensor(),
    #transforms.Normalize((0.1307,), (0.3081,))])
])


# Dataset w/o any tranformations
train_dataset_normal = CustomTensorDataset(tensors=X_train, transform=se2_transform)
train_loader = torch.utils.data.DataLoader(train_dataset_normal, batch_size=len(X_train))


</code></pre>
<blockquote>
<p>The dataset can be found here:</p>
</blockquote>
<pre><code class="lang-auto">!wget http://bergerlab-downloads.csail.mit.edu/spatial-vae/galaxy_zoo.tar.gz
!tar -xf /content/galaxy_zoo.tar.gz

</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc3d85ad94898cb0190b261bff2c56dfe9ae82fe.png" data-download-href="https://discuss.pytorch.org/uploads/default/bc3d85ad94898cb0190b261bff2c56dfe9ae82fe" title="Screenshot 2022-02-11 at 3.36.38 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc3d85ad94898cb0190b261bff2c56dfe9ae82fe_2_494x499.png" alt="Screenshot 2022-02-11 at 3.36.38 AM" data-base62-sha1="qRfzEI4KXFInQC7B9nE0GuWSHj8" width="494" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc3d85ad94898cb0190b261bff2c56dfe9ae82fe_2_494x499.png, https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc3d85ad94898cb0190b261bff2c56dfe9ae82fe.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc3d85ad94898cb0190b261bff2c56dfe9ae82fe.png 2x" data-dominant-color="2F2F2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-02-11 at 3.36.38 AM</span><span class="informations">702×710 36.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: nivek
cooked: <p>For your <code>__len__</code> method, should it be:<br>
<code>return self.tensors.size(0)</code>  # 61578<br>
instead of:<br>
<code>return self.tensors[0].size(0)</code> # 3</p>