When I go through my dataloader, I find NoneType objects!

Aymen_Sekhri: <p><strong>Hello Everyone</strong><br>
I hope you are doing awesome, I am stuck on a big problem, I read lots of blogs about it but there isn’t a real solution.<br>
The problem is when I loop through my data loader (I am using <a href="https://stanfordmlgroup.github.io/competitions/chexpert/" rel="noopener nofollow ugc">Chexpert dataset</a>) I find NoneType objects instead of images.<br>
The structure of the dataset is<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18a391e15ad29ef59c3794bb5993564f5cdc21e9.png" alt="image" data-base62-sha1="3vXUaVbzRBUYl8ktQBxM6Icjt9v" width="255" height="213"></p>
<p>The root directory is CheXpert-v1.0-small, inside it, there are two folders contain images and two CSV files contain the path to the image and the label.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/8227452ded245b8d2d688de0c66d831eefef7a95.png" data-download-href="https://discuss.pytorch.org/uploads/default/8227452ded245b8d2d688de0c66d831eefef7a95" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/8227452ded245b8d2d688de0c66d831eefef7a95_2_690x186.png" alt="image" data-base62-sha1="izocQDGncN3Rw4cNqHmNMbEMdy5" width="690" height="186" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/8227452ded245b8d2d688de0c66d831eefef7a95_2_690x186.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/8227452ded245b8d2d688de0c66d831eefef7a95_2_1035x279.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/2/8227452ded245b8d2d688de0c66d831eefef7a95.png 2x" data-dominant-color="474747"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1242×336 18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>so I created a custom dataset:</p>
<pre><code class="lang-auto">class CXRDataset(Dataset):
  '''
  add some explication about this calss
  '''
  def __init__(self, root_dir = None, csv_file = None, transform=None):
    if root_dir is not None and csv_file is not None:
      self.root_dir = root_dir
      self.csv_file = csv_file
      self.cursor = 0
      self.annotations = pd.read_csv(os.path.join(self.root_dir, self.csv_file))
      self.transform = transform

  def __len__(self):
    return len(self.annotations)

  def __repr__(self): 
    return "Test root :% s csv :% s id :% s" % (self.root_dir, self.csv_file, self.cursor) 

  def __getitem__(self, index):
    self.cursor = index
    image_path = os.path.join(self.root_dir, self.annotations.iloc[index, 1])
    image = cv2.imread(image_path)

    y_label = torch.tensor(list(self.annotations.iloc[index, 2:]))
    
    if self.transform:
      # the transform should contain the initial pre-processing 
      image = self.transform(image)
    
    return image, y_label
</code></pre>
<p>and when I load the data:</p>
<pre><code class="lang-auto">train_dataset = CXRDataset(root_source, csv_train_source, transform)
valid_dataset = CXRDataset(root_source, csv_valid_source, transform)
</code></pre>
<p>then the data loader:</p>
<pre><code class="lang-auto">train_loader = DataLoader(dataset=train_dataset, batch_size=16, shuffle=True)
valid_loader = DataLoader(dataset=valid_dataset, batch_size=16, shuffle=True)
</code></pre>
<p>and now I want to check that my <code>train_loader</code> and <code>valid_loader</code>  that don’t have any <code>NoneType</code> objects.</p>
<p>I used this code, it does not work:</p>
<pre><code class="lang-auto">for dataset in [train_loader, valid_loader]:
  for batch in dataset:
    img_batch ,label_batch = batch
    for idx in range(len(label_batch)):
      image = img_batch[idx].squeeze()
      label = label_batch[idx]
      if image is None or not any(label) :
        print([dataset])
</code></pre>
<p>So please any advice to check my data before feed it to the NN?</p>
<p>Thanks in advance</p>

ptrblck: <p>I’m not sure what exactly “it does not work” mean regarding the last code snippet, i.e. if the check is not working or if you are seeing <code>None</code> samples and are unsure how to fix it.<br>
In any case, I would try to narrow down why <code>None</code> objects are returned and try to fix this afterwards.<br>
To do so, use a <code>batch_size</code> of <code>1</code> and disable shuffling, iterate the <code>DataLoader</code>, and check which index returns the <code>None</code> objects. Once found, index the <code>dataset</code> using this index and check the <code>__getitem__</code> further to isolate what’s failing.</p>

Aymen_Sekhri: <p>I got it, Thank you very much</p>

Nisarg_Doshi: <p>Hey, stuck at the same problem doing similar classification task. How did you solve it <a class="mention" href="/u/aymen_sekhri">@Aymen_Sekhri</a> ?<br>
Please help me.</p>

y-vectorfield: <p>Hello, I met the same problem.<br>
I did not solve this.</p>

y-vectorfield: <p>I would like to return the following dict type data from DataLoader.</p>
<pre data-code-wrap="python"><code class="lang-python">    def __getitem__(self, index):
        tmp_data = self.benchmark[index]

        return {"id": tmp_data["id"],
                "question": tmp_data["question"],
                "options": tmp_data["options"],
                "answer": tmp_data["answer"],
                "image": tmp_data["image"],
                "question_type": tmp_data["question_type"],
                "index2ans": tmp_data["index2ans"],
                "correct_choice": tmp_data["correct_choice"],
                "all_choices": tmp_data["all_choices"],
                "empty_prompt": tmp_data["empty_prompt"],
                "final_input_prompt": tmp_data["final_input_prompt"],
                "gt_content": tmp_data["gt_content"]}
</code></pre>
<p>However, If I implement the loader, this returns None type data.<br>
When I tried to print the return values of <strong>getitem</strong>, correct values were outputted…</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue using random data?</p>

y-vectorfield: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , thank you very much for your message.<br>
Acturally, I solved this problem.<br>
I tried a simple type of dict data, and the model was moved!<br>
The cause of this issue was the data collator func.<br>
I forgot to implement the return method into <code>__call__</code>.</p>