Multiple errors while trying to load data

username: Pytorch_developer
cooked: <p><strong>This is my code:</strong><br>
import torch<br>
import torchvision<br>
from torchvision import transforms, datasets<br>
import torch.nn as nn<br>
import torch.nn.functional as f<br>
import torch.optim as optim<br>
import os<br>
import cv2<br>
import numpy as np<br>
from tqdm import tqdm</p>
<p>rebuild_data = True</p>
<p>class cats_and_dogs():<br>
image_size = 50<br>
cats = “PetImages/Cat”<br>
dogs = “PetImages/Dog”<br>
labels = {cats: 0, dogs: 1}</p>
<pre><code>training_data = []
catcount = 0
dogcount = 0

def make_training_data(self):
    for label in self.labels:
        print(label)
        for f in tqdm(os.listdir(label)):
            try:

                path = os.path.join(label, f)
                img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
                img = cv2.resize(img, (self.IMG_SIZE, self.IMG_SIZE))
                self.training_data.append([np.array(img), np.eye(2)[self.labels[label]]])

                if label == self.cats:
                    self.catcount += 1
                elif label == self.dogs:
                    self.dogcount += 1

            except Exception as e:
                pass
    
    np.random.shuffle(self.training_data)
    np.save("training_data.npy", self.training_data)

    print("cats: ", self.catcount)
    print("dogs: ", self.dogcount)
</code></pre>
<p>if rebuild_data == True:<br>
Dogs_and_Cats = cats_and_dogs()<br>
Dogs_and_Cats.make_training_data()<br>
<strong>and these are the errors:</strong><br>
PetImages/Cat<br>
Traceback (most recent call last):<br>
File “/Users/edenbrown/sample.ws45/new.py”, line 51, in <br>
Dogs_and_Cats.make_training_data()<br>
File “/Users/edenbrown/sample.ws45/new.py”, line 27, in make_training_data<br>
for f in tqdm(os.listdir(label)):<br>
FileNotFoundError: [Errno 2] No such file or directory: ‘PetImages/Cat’</p>

username: Matias_Vasquez
cooked: <p>Hi,</p>
<p>you might want to look at <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class" rel="noopener nofollow ugc">this</a> tutorial on how to create your own Dataset class.</p>
<p>Also, if your images are already divided by class, I would suggest using ImageFolder to create your Dataset more easily.</p>
<p><a href="https://pytorch.org/vision/main/generated/torchvision.datasets.ImageFolder.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/generated/torchvision.datasets.ImageFolder.html</a></p>
<p>However, your error seems to be that it does find that location.</p>
<p>Could you try putting an “r” before your strings for the cats and dogs variables?<br>
Like this:</p>
<pre><code class="lang-python">cats = r“PetImages/Cat”
</code></pre>
<p>Please let me know if this helps or if you still have this problem.</p>

username: Pytorch_developer
cooked: <p>The r before the string did not help and I am still confused even with the tutorial.</p>

username: Matias_Vasquez
cooked: <p>Ok,</p>
<p>let’s try it with ImageFolder. (I think this might help you, since it takes care of most of what you need to do.)</p>
<p>First of all, you need your Data divided into folders, where each folder contains only one class.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be2761eb709ffd06daa0bfef413ed7506fd2332c.png" alt="image" data-base62-sha1="r8b5X1rtDnvozSS0ovOIOykxw4Y" width="243" height="174"></p>
<p>Judging by your code, you already have this.</p>
<p>So in order to create your Dataset, you only need to pass the path to where the <strong>root</strong> directory of your classes is.</p>
<pre data-code-wrap="python"><code class="lang-python">from torchvision.datasets import ImageFolder
import torchvision.transforms as T

path = "[YOUR_PATH_TO_THE_ROOT]/root"
transform = T.ToTensor()

dataset = ImageFolder(root=path, transform=transform)
</code></pre>
<p>So, you only need to define your path until the root, where the folders for all of your classes are.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/7/470109acdb6493b96b5b4ab384da9e165526b5f9.png" alt="image" data-base62-sha1="a887G8tMGcfxRNIAQK3hjuHU8ml" width="243" height="226"></p>
<p>After this, you can access your dataset like this</p>
<pre data-code-wrap="python"><code class="lang-python">index = 0
img, lbl = dataset[index]
</code></pre>
<p>This is a format that can be handled by the DataLoader… but this comes later. For now try to do this, and if you get stuck please let me know.</p>

username: Pytorch_developer
cooked: <p><strong>So should my code look like this:</strong></p>
<p>from torchvision.datasets import ImageFolder<br>
import torchvision.transforms as T</p>
<p>path = "[PetImages/Cat, PetImages/Dog]/root<br>
transform = T.ToTensor()</p>
<p>dataset = ImageFolder(root=path, transform=transform)</p>
<p>index = 0<br>
img, lbl = dataset[index]</p>
<p><strong>and also what does lbl mean</strong></p>

username: Matias_Vasquez
cooked: <p>I have now put a random path of how it might look like. With “root”, I meant like in the image that I put. In your case, the “root” folder would be the “PetImages” one. But you still have to provide the full path so that python can find those files</p>
<pre><code class="lang-python">from torchvision.datasets import ImageFolder
import torchvision.transforms as T

path = "C:/Users/Matias/Desktop/PetImages"
# Do not include dogs or cats in the path, just up to PetImages
transform = T.ToTensor()

dataset = ImageFolder(root=path, transform=transform)

index = 0
img, lbl = dataset[index]
</code></pre>
<p>Your code might look something like this.</p>
<p>lbl is for the label of the image it belongs to.<br>
This means that the class cat might have the label 0 and dogs have the label 1. (or the other way around).</p>
<p>Let me know if it now works.</p>

username: Pytorch_developer
cooked: <p>This has worked but I would like to know how to shuffle the dataset so the image and label are not always the same.</p>

username: Matias_Vasquez
cooked: <p>Ok,</p>
<p>If you want your images to be in a random order, you can do something like this</p>
<pre><code class="lang-python">import random

dataset.samples = sorted(dataset.samples, key=lambda k:random.random())
</code></pre>
<p>now you can create a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">DataLoader</a>. This will give you batches of random images from your dataset.</p>
<p>You can do something like this:</p>
<pre><code class="lang-python">from torch.utils.data import DataLoader

dataloader = DataLoader(dataset, batch_size=32, shuffle=True)
</code></pre>
<p>You can look in the documentation that I have linked to get a better understanding.</p>
<p>This will be useful when you want to iterate the full dataset to train a model.</p>
<pre><code class="lang-python"># Dumb example of how you might use the dataloader
for i, (imgs, lbls) in enumerate(dataloader):
    print(i, lbls) 
    # this is not useful, you will do something better than just print 
    #   the batch number, and the labels for this batch
</code></pre>
<p>If you see the first <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#iterating-through-the-dataset" rel="noopener nofollow ugc">link</a> that I gave you, further down in the “Iterating through the dataset” section, they show how they use this.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

username: Pytorch_developer
cooked: <p><strong>I do not know if I did something wrong but I got this error:</strong></p>
<p>Traceback (most recent call last):<br>
File “/Users/edenbrown/sample.ws45/new.py”, line 15, in <br>
for i, (imgs, lbls) in enumerate(dataloader):<br>
File “/Users/edenbrown/opt/anaconda3/envs/env_pytorch3/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 530, in <strong>next</strong><br>
data = self._next_data()<br>
File “/Users/edenbrown/opt/anaconda3/envs/env_pytorch3/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 570, in _next_data<br>
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration<br>
File “/Users/edenbrown/opt/anaconda3/envs/env_pytorch3/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py”, line 52, in fetch<br>
return self.collate_fn(data)<br>
File “/Users/edenbrown/opt/anaconda3/envs/env_pytorch3/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”, line 172, in default_collate<br>
return [default_collate(samples) for samples in transposed]  # Backwards compatibility.<br>
File “/Users/edenbrown/opt/anaconda3/envs/env_pytorch3/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”, line 172, in <br>
return [default_collate(samples) for samples in transposed]  # Backwards compatibility.<br>
File “/Users/edenbrown/opt/anaconda3/envs/env_pytorch3/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”, line 138, in default_collate<br>
return torch.stack(batch, 0, out=out)<br>
RuntimeError: stack expects each tensor to be equal size, but got [3, 375, 500] at entry 0 and [3, 150, 200] at entry 2</p>
<p><strong>My code is:</strong></p>
<p>from torchvision.datasets import ImageFolder<br>
import torchvision.transforms as T<br>
import random<br>
from torch.utils.data import DataLoader</p>
<p>path = “/Users/edenbrown/Downloads/kagglecatsanddogs_3367a”<br>
transform = T.ToTensor()</p>
<p>dataset = ImageFolder(root=path, transform=transform)</p>
<p>dataset.samples = sorted(dataset.samples, key=lambda k:random.random())</p>
<p>dataloader = DataLoader(dataset, batch_size=32, shuffle=True)</p>
<p>for i, (imgs, lbls) in enumerate(dataloader):<br>
print(i, lbls)</p>

username: Pytorch_developer
cooked: <p>I think the error may be something to do with image size because some of the images are different sizes.</p>

username: Matias_Vasquez
cooked: <p>If you have images of different sizes, you can try resizing them within the transform when creating the dataset.</p>
<pre><code class="lang-python">transform = T.Compose([
    T.Resize((150, 200)),
    T.ToTensor()
])
</code></pre>

username: Pytorch_developer
cooked: <p>This has worked so thank you.</p>

username: Pytorch_developer
cooked: <p><strong>Also I do not believe the:</strong></p>
<p>import random</p>
<p>dataset.samples = sorted(dataset.samples, key=lambda k:random.random())</p>
<p><strong>is needed as the code still works perfectly without it.</strong></p>
<p><strong>Also the labels are the same for cats and dogs but I will figure out how to fix that in a different post.</strong></p>