AttributeError: 'CustomDataSet' object has no attribute 'size'

akib62: <p>I have images in a folder. So, I made a <code>custom dataset</code> to load the images.</p>
<pre><code class="lang-auto">class CustomDataSet(Dataset):
    def __init__(self, main_dir, transform=None):
        self.main_dir = main_dir
        self.transform = transform
        self.all_imgs = os.listdir(main_dir)

    def __len__(self):
        return len(self.all_imgs)

    def __getitem__(self, idx):
        img_loc = os.path.join(self.main_dir, self.all_imgs[idx])
        image = Image.open(img_loc).convert("RGB")
        tensor_image = self.transform(image)
        return tensor_image
</code></pre>
<p>Then, I am trying to make <code>TensorDataset</code> from <code>image_tensor and labels_tensor</code>. Finally, I want to make a <code>DataLoader</code> using this <code>TensorDataset</code></p>
<pre><code class="lang-auto">def load_train_data():
    # Some code to the directory path etc.

    # doing some preprocessing
    transform = transforms.Compose([
    transforms.Resize((256, 256)),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
    
    #loading images using CustomDataSet
    train_data_tensor = CustomDataSet(train_data_dir, transform=transform)
    
    # Making TnesorDataset using both train_data_tensor and train_label_price_tensor
    train_tensor = TensorDataset(train_data_tensor, train_label_price_tensor)
    
    # Making Train Dataloader
    train_loader = DataLoader(train_tensor, batch_size= 1, num_workers= 2, shuffle= True)
</code></pre>
<p>But, I am getting an error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "preprocess.py", line 86, in &lt;module&gt;
    load_train_data()
  File "preprocess.py", line 75, in load_train_data
    train_tensor = TensorDataset(train_data_tensor, train_label_price_tensor)
  File "/home/akib/.local/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 158, in __init__
    assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors)
  File "/home/akib/.local/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 158, in &lt;genexpr&gt;
    assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors)
AttributeError: 'CustomDataSet' object has no attribute 'size'
</code></pre>
<p>What wrong with my code?</p>

ptrblck: <p><code>TensorDataset</code> expects tensors as inputs not another <code>Dataset</code>.<br>
You could wrap the <code>train_data_tensor</code> (your <code>CustomDataSet</code>) directly in a <code>DataLoader</code>.</p>

akib62: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, here I am doing <code>transform</code>. Does it mean, images are converted into <code>tensor</code>, not the <code>CustomDataSet</code></p>
<aside class="quote no-group" data-username="akib62" data-post="1" data-topic="105601">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3bc359/48.png" class="avatar"> akib62:</div>
<blockquote>
<pre><code class="lang-auto">#loading images using CustomDataSet
    train_data_tensor = CustomDataSet(train_data_dir, transform=transform)
</code></pre>
</blockquote>
</aside>

ptrblck: <p>The passed <code>transform</code> will be applied in <code>CustomDataSet.__getitem__</code> in this line of code:</p>
<pre><code class="lang-python">tensor_image = self.transform(image)
</code></pre>
<p>and will thus be applied on the <code>PIL.Image</code> and <code>tensor_image</code> should be a tensor in this case.</p>

akib62: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>I have 100 images in a folder, but I am getting only one image location from the <code>__getitem__</code>.</p>
<pre><code class="lang-auto">def __init__(self, main_dir, label_full, transform):
  #Reading path and doing some operations
          #sorting images based on the name
        self.all_imgs = sorted(os.listdir(main_dir), key= lambda x : int(x.split("_")[0]))

def __getitem__(self, idx):
         img_loc = os.path.join(self.main_dir, self.all_imgs[idx])
        print(img_loc) # printing only 1 image location.
        image = Image.open(img_loc).convert("RGB")
</code></pre>
<p>Could you tell me why this is happening or I am doing some mistakes?</p>

ptrblck: <p>Could you check the length of <code>dataset.all_imgs</code> via <code>print(len(dataset.all_imgs))</code> and make sure more than one image is found?</p>

akib62: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, yes, it is showing the length is <code>100</code>!</p>
<pre><code class="lang-auto">    def __len__(self):
        print(len(self.all_imgs))
        return len(self.all_imgs)
</code></pre>
<p>I also checked it grom <code>__getitem__</code> and the result is same <code>100</code></p>

ptrblck: <p>Are you using a custom sampler or <code>collate_fn</code>?<br>
If the length of the dataset is given as 100, this code should work:</p>
<pre><code class="lang-python">for idx in range(100):
    batch = dataset[idx]
</code></pre>
<p>while using a <code>DataLoader</code> only returns a single sample?</p>
<pre><code class="lang-python">loader = DataLoader(dataset, batch_size=1)
for idx, batch in enumerate(loader):
    print(idx)
</code></pre>

akib62: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I am using <code>custom sampler</code>. I am getting this error (before and after applying your code).</p>
<p><code>ValueError: 1 is not in range</code></p>
<p>Full error post is <a href="https://discuss.pytorch.org/t/valueerror-1-is-not-in-range-customdataset/105681">here</a>.</p>
<p>I am not sure what is happening! Trying to create my <code>Custom Dataset</code> but not working at all!</p>

ptrblck: <p>In that case, make sure it’s passing <code>len(dataset)</code> indices to the <code>Dataset.__getitem__</code> method.</p>

akib62: <aside class="quote no-group" data-username="ptrblck" data-post="10" data-topic="105601">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>len(dataset)</p>
</blockquote>
</aside>
<p>How can I do it? Would you mind to give me any hints?</p>

ptrblck: <p>It depends on your custom sampler and I don’t know how you’ve implemented it.<br>
You could compare your implementation to the <a href="https://github.com/pytorch/pytorch/blob/9417e92722f1b6f75e162cbe661d3ed788d8d37a/torch/utils/data/sampler.py#L55-L70"><code>SequentialSampler</code></a>, which returns <code>iter(range(len(self.data_source)))</code> and thus indices in the range of the dataset.</p>