Dataloader/Dataset randomly gives out None

username: mnazaal
cooked: <p>Hi,<br>
Im doing an image segmentation task, and for that, within the Dataset, Im using a function which generates a stick model of a human based on the xy points of places of interest (head, joints etc). I have the xy points, and my Dataset class looks like the following.</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, json_file_dir, image_dir, transform=None):
        # json_file_dir :: String ; Path to json file with xy coordinates
        # image_dir     :: String ; Path to image data
        # transform     :: Optional transform for dataset
        
        self.json_file_dir = json_file_dir
        with open(json_file_dir) as f:
            self.json_file = json.loads(f.read())
        self.image_dir = image_dir
        self.transform = transform

    def __len__(self):
        return len(self.json_file)
 
    def __getitem__(self, index):
        # index :: index number for input,label pair
        
        image_name        = self.json_file[index]["filename"]  #needed to read the image
        image_coordinates = self.json_file[index]["keypoints"] #needed to generate the labelled image
        image             = cv2.imread(image_name)
                
        labelled_image = LabelMaker(image, image_coordinates, args="LINES") #outputs n channel label, n is the number of classes
        
        # image :: np.array 256x256x3
        # labelled_image :: np.array 256x256x4
        sample = {"image": image, "labelled_image": labelled_image}
        
        if self.transform:
            sample = self.transform(sample["image"], sample["labelled_image"])
        
        return sample

    
def transform(image, label):
    image = cv2.resize(image, (256, 256)).astype(np.float64)  # uint8 with RGB mode
    image -= rgb_mean  #[R, G, B] where R, G, B denote the mean values
    image = image.astype(np.float64)

    image = image.astype(float) / rgb_sd #[R, G, B] where R, G, B are the standard deviations of them

    image = torch.from_numpy(image).float()
    label = torch.from_numpy(label).float()
        
    return {"image": image, "labelled_image": label}    
</code></pre>
<p>After making an instance of this for the training dataset, I make a dataloader, with the parameters - batch size=5, shuffle=True and number of workers=4.</p>
<p><strong>However</strong><br>
During the training process, I get an error, which states</p>
<pre><code class="lang-auto">AttributeError: Traceback (most recent call last):
  File "dir/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 57, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "&lt;ipython-input-20-e1502ecaff7b&gt;", line 32, in __getitem__
    labelled_image = LabelMaker(image, image_coordinates, args="LINES")
  File "&lt;ipython-input-19-820c82037e68&gt;", line 23, in PointAndLineMaker
    height, width = image.shape[0], image.shape[1]
AttributeError: 'NoneType' object has no attribute 'shape'

Exception NameError: "global name 'FileNotFoundError' is not defined" in &lt;bound method _DataLoaderIter.__del__ of &lt;torch.utils.data.dataloader._DataLoaderIter object at 0x7f13cacc8c90&gt;&gt; ignored
</code></pre>
<p>Here, the LabelMaker is the function that takes an input image, and the required coordinates to generate the label. As you can see from my Dataset class, the input to this function is fed from that class after loading it from there.<br>
I checked with print statements going through the whole dataset trying to see whether anything is None but to no avail.<br>
Whats more annoying is that this error pops up randomly. For debugging purposes Im running 5 epochs, and it chooses to spit out this None at different times, and I also ran it for 5 epochs successfully without the error coming.</p>
<p>What am I doing wrong here?</p>
<p>EDIT:<br>
I have reason to believe this is a problem in openCV, specifically the imread function, since the problem is avoided after converting the input images to .npy files, and loading these into the dataset</p>

username: ptrblck
cooked: <p>Could you add a debug statement before the <code>LabelMarker(...)</code> line:</p>
<pre><code class="lang-python">if not image:
    print(image_name)
</code></pre>
<p>The next time it crashes, we would know, what file seems to be missing or corrupt.<br>
Also, could you set <code>shuffle=False</code> for the <code>DataLoader</code>? This would make sure to get the file deterministically.</p>

username: mnazaal
cooked: <p>I tried that, with shuffle=False as well, but the error still comes randomly.<br>
I also went through the whole dataset (which isnt too large luckily) thrice, and 1 time i got a None at a certain image file and when i checked it out it was not corrupted or missing. The other 2 runs, I didnt get a None. I even changed my workflow from Jupyter notebook to a more class-based IDE setup just in case Jupyter was messing with me.<br>
Also an important thing I may have not told originally in the first post (now I edited it there) was this message that I get</p>
<pre><code class="lang-auto">Exception NameError: "global name 'FileNotFoundError' is not defined" in &lt;bound method _DataLoaderIter.__del__ of &lt;torch.utils.data.dataloader._DataLoaderIter object at 0x7f13cacc8c90&gt;&gt; ignored
</code></pre>

username: lml
cooked: <p>Hello, any luck with this? I’m facing the same problem here.<br>
My <strong>getitem</strong> function:</p>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        name = self.name_list[idx]
        img_file = os.path.join(self.img_dir, name + '_leftImg8bit.png')
        image = cv2.imread(img_file) # it returns None
        .....
</code></pre>
<p>My Dataloader is using pin_memory=True, drop_last=True, shuffle=True</p>

username: Andrei_Cristea
cooked: <p>Could you add a couple of debug lines, just to make sure the file is actually there:</p>
<pre><code class="lang-auto">        name = self.name_list[idx]
        img_file = os.path.join(self.img_dir, name + '_leftImg8bit.png')
        if not os.path.isfile(img_file):
            print(f"file missing: {img_file}")
        image = cv2.imread(img_file) # it returns None
        if image is None:
            print(f"file none: {img_file}")
</code></pre>
<p>Apologies if you already did this, but it’s always good to first rule out all simple explanations <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>The first print would make sure the file is there, the second print will allow you to hand-check that file and make sure it isn’t corrupted.</p>
<p>(also, make sure none of the image files are open for viewing in any program by your OS, no idea if that could cause it but again just in the spirit of ruling out silly bugs!)</p>

username: lml
cooked: <p>Sure thing. I’m running this test. But yeah, I other tests like this one (try, except and print the name of the file. and after stopping the program, I can see the file is there, but I didn’t check whether or not this file was available at runtime. I’ll run this test and post here the results.)</p>