How to load images from a folder whose names are stored in a csv file

raoashish10: <p>I have a csv file which has the names of the specific images which have to be loaded into a dataloader from a folder. This csv file also contains the labels/classes of the images. I tried using a dataset class. I had written the below code:</p>
<pre><code class="lang-auto"> class data(Dataset):
  def __init__(self,type='train',transform=None):
    self.imagecsv=''
    if type=='train':
      self.imagecsv=traincsv
    else:
      self.imagecsv=testcsv
    imagelabels=self.imagecsv[['image_id','Labels']]
    images=[]
    for i in imagelabels['image_id']:
      image=imread(os.path.join(path,i+'.jpg'))
      images.append(image)
    imagelabels['images']=images
    imagelabels.drop(columns='image_id')
      

  def __getitem__(self,idx):
    return imagelabels['images'][idx],imagelabels['Labels'][idx]
   
</code></pre>
<p>When I try to run this code, the system crashes everytime. I am running it on google colab with a GPU. I know I haven’t added an index parameter in the  <strong>getitem</strong>  function, it is because I need all the images whose names are specified in the training csv file. Is there a cleaner implementation which won’t crash my system? I also tried using Image.open but the system crashed again.</p>

Rohan_Kumar: <p>every time you index or call retrieve an item from your data class, your code processes all the images again. which i think would be a mistake while training.</p>
<p>shift the data retrieving part in <strong>init</strong>, and then just return what you have to return in  <strong>getitem</strong> .  and shouldnt it be def <strong>getitem</strong>( self, index) ?</p>

raoashish10: <p>I assume it is okay to drop the index parameter from the getitem function. I haven’t included because I wanted the whole dataset every time. Anyway thank you for your answer! I will try it</p>

raoashish10: <p>I tried this method, I’ll update the code. However, my system still tends to crash. Any other implementation ideas?</p>

Nikronic: <p>Hi,</p>
<p>I am not sure about your code, but there is easier ways to handle it. You can define your custom Dataset class and have all data in each iteration just by setting <code>batch_size=len(data)</code> and you no longer have to change your code for different purposes. And also, you have to implement <code>__len__</code> function too.</p>
<p>Here is conventional implementation of custom dataset class:</p>
<aside class="quote quote-modified" data-post="2" data-topic="55565">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/custom-dataset-labeling-from-csv/55565/2">Custom Dataset labeling from CSV</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hi, 
You can define your own custom dataset class easily to handle this kind of situations. 
Here is the top-level structure of the class your can implement: 

class PlacesDataset(Dataset):
  def __init__():
     # initialize variables such is path to csv file and images and transforms
  def __len__():
    # here you just need to return a single integer number as the length of your dataset, in your 
    #  case, number of images in your train folder or lines in csv file
  def __getitem__(): 
   …
  </blockquote>
</aside>

<p>After creating such a Dataset, you can use it for different purposes like this:</p>
<pre><code class="lang-auto">train_dataset = PlacesDataset(txt_path=args.txt,
                              img_dir=args.img,
                              transform=custom_transforms)

train_loader = DataLoader(dataset=train_dataset,
                          batch_size=args.bs,
                          shuffle=True,
                          num_workers=args.nw,
                          pin_memory=pin_memory)

</code></pre>
<p>bests</p>

raoashish10: <p>Thank you so much for helping me out! I followed the code instructions in the link you have provided and I must say a lot of concepts have been cleared for me now. This was very helpful. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

Nikronic: <p>You are welcome.<br>
I think best way to learn about best practices is to read the well-constructed codes which in this case, we can refer to implemented models in PyTorch iteslf. For instance, ResNet, VGG, etc have been implemented by PyTorch developers and they used most general and easy ways to do it, so by reading these codes, we can adopt and have a more reliable codes.</p>
<p>Good luck</p>

raoashish10: <p>Thanks for the advice! I will check out those codes to have a better understanding</p>