Why my Custom Dataset is not working?

akib62: <p>Hello,<br>
I have some images in a folder. So, I am trying to create a custom dataset with taking help from this <a href="https://discuss.pytorch.org/t/how-to-load-images-without-using-imagefolder/59999/2">post</a>. But, I am getting some errors.</p>
<p>My custom dataset class is given below:</p>
<pre><code class="lang-auto">class CustomDataSet(Dataset):
    def __init__(self, main_dir, transform):
        self.main_dir = main_dir
        self.transform = transform
        all_imgs = os.listdir(main_dir)
        self.total_imgs = natsort.natsorted(all_imgs) //Error-1

    def __len__(self):
        return len(self.all_imgs)

    def __getitem__(self, idx):
        img_loc = os.path.join(self.main_dir, self.all_imgs[idx])
        image = Image.open(img_loc).convert("RGB")
        tensor_image = self.transform(image)
        return tensor_image
</code></pre>
<p>My loader method</p>
<pre><code class="lang-auto"> train_data_dir = '/home/Houses-dataset-master'

  # Transformation
  transform = transforms.Compose([
  transforms.Resize((256, 256)),
  transforms.ToTensor(),
  transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

  # Giving the path 
  train_data_tensor = CustomDataSet(train_data_dir, transform=transform)

   # trying to  print the length of the train_data_tensor
   print(len(train_data_tensor)) //Getting Error-2

    # Converting labels into tensoe
    train_label_price_tensor = torch.tensor(label_price.values)
    print(train_label_price_tensor.size())
    print(train_data_tensor.size()) //getting Error-3
</code></pre>
<p>First I was getting an error for natsort</p>
<pre><code class="lang-auto">     Error-1: NameError: name 'natsort' is not defined
</code></pre>
<p>Then, I remove the <code>natsort</code> and replaced all <code>total_imgs</code> with <code>all_imags</code>. Getting</p>
<pre><code class="lang-auto">Error-2: AttributeError: 'CustomDataSet' object has no attribute 'all_imgs'
</code></pre>
<p>Finally, I removed the <code>error-2 code line</code>  from the program and getting another error at the time of printing the size of my <code>images tensor</code>.</p>
<pre><code class="lang-auto">Error-3: AttributeError: 'CustomDataSet' object has no attribute 'size'
</code></pre>
<p>Does it mean, my custom dataset is not working? Transformation is not working?</p>
<p>How can I solve these issues?</p>
<p>Thank you.</p>

pchandrasekaran: <p>You’ve missed the self for <code>all_imgs</code>.</p>
<p><code>all_imgs = os.listdir(main_dir)</code> to <code>self.all_imgs = os.listdir(main_dir)</code>.</p>
<p>Try out that first. It should fix it.</p>

akib62: <p><a class="mention" href="/u/pchandrasekaran">@pchandrasekaran</a> Thank you very much.</p>
<p>Your suggestion solve the Error-2 but, still I am getting Error-3</p>
<p><code> Error-3: AttributeError: 'CustomDataSet' object has no attribute 'size'</code></p>
<p>Could you tell me, why this is happening (transformation is not working)?</p>

pchandrasekaran: <p>That’s because the Dataset class doesn’t contain any implementation for size(). Do you need the size of the entire dataset as in [total_len, C, H, W] or are you simply trying to get the size() of the returned value?</p>

akib62: <p><a class="mention" href="/u/pchandrasekaran">@pchandrasekaran</a> thank you,</p>
<p>Yes, I was just simply trying to find the length of the <code>dataset</code>. Because I am using <code>ToTensor</code>  at the time of <code>transform</code>. It means, my <code>dataset</code> is converting into <code>Tensor</code> formate, right? If <code>ToTensor</code> worked then, I should get the <code>size</code>, right?</p>
<p>However, I need to add <code>data_tensor</code> and <code>label_tensor</code> both at the time of <code>dataloader</code>. Code is given below:</p>
<pre><code class="lang-auto">    # Making tensor using both image and labels tensor
    train_tensor = TensorDataset(train_data_tensor, train_label_price_tensor)
    
    # Making Train Dataloader
    train_loader = DataLoader(train_tensor, batch_size= 1, 
                              num_workers= 2, shuffle= True)
</code></pre>
<p>I am getting this error</p>
<p><code>AttributeError: 'CustomDataSet' object has no attribute 'size'</code></p>
<p>Could you tell me, how can I solve this problem?</p>

InnovArul: <aside class="quote no-group" data-username="akib62" data-post="1" data-topic="105558">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3bc359/48.png" class="avatar"> akib62:</div>
<blockquote>
<p><code>print(train_data_tensor.size()) //getting Error-3</code></p>
</blockquote>
</aside>
<p><code>Dataset</code> object does not have the attribute <code>size()</code> as the error suggests.<br>
To get the data out of <code>Dataset</code> object, you should use an index.<br>
For example,</p>
<pre><code class="lang-auto">img = train_data_tensor[0]
print(img.size()) # or print(img.shape)
</code></pre>

akib62: <p><a class="mention" href="/u/innovarul">@InnovArul</a> Thanks,</p>
<p>Then, how can I make a <code>dataloader</code> using this <code>dataset</code>? I am using this code</p>
<pre><code># Making tensor using both image and labels tensor
train_tensor = TensorDataset(train_data_tensor, train_label_price_tensor)

# Making Train Dataloader
train_loader = DataLoader(train_tensor, batch_size= 1, 
                          num_workers= 2, shuffle= True)
</code></pre>
<p>But getting this error</p>
<p><code>AttributeError: 'CustomDataSet' object has no attribute 'size'</code></p>
<p>Could you tell me, how can I solve this problem?</p>

akib62: <p><a class="mention" href="/u/innovarul">@InnovArul</a>, after applying your suggestions, I am getting the shape of a image.</p>

InnovArul: <p>It is confusing to me at this point.<br>
I see that you have own dataset class <code>CustomDataset</code>. I also see the use of the <code>TensorDataset</code>.</p>
<p>You can use any object of a <code>Dataset</code> class as input to <code>Dataloader</code>.</p>
<aside class="quote no-group" data-username="akib62" data-post="7" data-topic="105558">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3bc359/48.png" class="avatar"> akib62:</div>
<blockquote>
<pre><code class="lang-auto"># Making tensor using both image and labels tensor
train_tensor = TensorDataset(train_data_tensor, train_label_price_tensor)
</code></pre>
</blockquote>
</aside>
<p>Here, <code>train_data_tensor, train_label_price_tensor</code> are tensors? If there are tensors, I am not sure why the error talks about <code>CustomDataSet</code>.</p>

akib62: <p><a class="mention" href="/u/innovarul">@InnovArul</a>, I am using <code>transform</code> in my <code>CustomDataSet</code>. So, I am guessing that the code converting the dataset into <code>tensor</code> (I am not sure).</p>
<aside class="quote no-group" data-username="akib62" data-post="1" data-topic="105558">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3bc359/48.png" class="avatar"> akib62:</div>
<blockquote>
<pre><code class="lang-auto"># Giving the path 
  train_data_tensor = CustomDataSet(train_data_dir, transform=transform)
</code></pre>
</blockquote>
</aside>
<p><code>train_label_price_tensor</code> is <code>tensor</code>. I made it using</p>
<p><code>train_label_price_tensor = torch.tensor(label_price.values)</code></p>
<p>Is it possible to create a <code>CustomDataSet</code> into <code>Tensor</code>?</p>

InnovArul: <p><code>TensorDataset</code> accepts tensors as input. So, <code>CustomDataset</code> can’t be its parameter.<br>
Why can’t you pass labels to custom dataset and return image, label in <code>__getitem__</code>?<br>
and then pass the <code>CustomDataset</code> directly to dataloader?</p>

akib62: <p><a class="mention" href="/u/innovarul">@InnovArul</a> thanks.</p>
<p>Right now, I have done your suggested method (images and labels) from the <code>CustomDataSet</code>.</p>