Dataset : IndexError: list index out of range

username: patawaitte
cooked: <p>Hi,<br>
I created custom dataset for classification of 8 channel images.<br>
I get an IndexError: list index out of range when I lanch the trainning.<br>
Do you see errors on my dataset?</p>
<pre><code class="lang-auto">LABEL_FILENAME="label.tif"

class Dataset(torch.utils.data.Dataset):
    def __init__(self, data_dir,):
        self.data_dir = data_dir#os.path.join(root_dir)
        self.samples = list()
        self.classes = [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]

        files = os.listdir(self.data_dir)
        for f in files:
            self.samples.append(f)

    def __len__(self):
        return len(self.samples)

    def __getitem__(self, idx):

        path = os.path.join(self.data_dir, self.samples[idx])
        print("p",self.samples[idx])
        label, profile = read(os.path.join(path,LABEL_FILENAME))
        label=label[0]

        label = torch.from_numpy(label)

        x = read(os.path.join(path, "MHCHD_22E01N.tif"))[0]
        x= np.array(x)
        x =torch.from_numpy(x)


        return x.float(), label.long()
</code></pre>

username: ptrblck
cooked: <p>Which line of code is throwing this error?<br>
Could it be that indexing <code>label</code> is not working or the <code>x = read(...)[0]</code> call?</p>

username: patawaitte
cooked: <p>Hi,<br>
Here the error :</p>
<p><code>File "/home/Projet/mydataset.py", line 58, in __getitem__     path = os.path.join(self.data_dir, self.samples[idx]) IndexError: list index out of range</code></p>

username: ptrblck
cooked: <p>Could you print the length of <code>self.samples</code> and the <code>idx</code>, which is causing this error?</p>

username: patawaitte
cooked: <p>Thanks for your help ptrblck!!</p>
<p>I get the name of the directory. In each directory I have 1 image (“MHCHD_22E01N.tif”), and one label image (label.tif).</p>
<blockquote>
<p>print(self.samples[idx]) :  dalle_238<br>
print(self.samples[idx]) :  dalle_268<br>
print(self.samples[idx]) :  dalle_195<br>
…</p>
</blockquote>

username: ptrblck
cooked: <p>Could you print the <code>idx</code> and try to index <code>samples</code> with this index outside of the training loop?</p>

username: patawaitte
cooked: <p>Hi,<br>
I’m not sure to understand what you mean by " try to index  <code>samples</code>  with this index"</p>
<p>If I print idx outside the training loop I get <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
`</p>
<blockquote>
<p>idx :  [0]<br>
self.samples :  [‘dalle_741’, ‘dalle_321’, ‘dalle_444’,(…)‘dalle_532’]<br>
len(self.samples) :  973</p>
</blockquote>

username: ptrblck
cooked: <p>Sorry for not being clear enough. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Print the <code>idx</code> in the training loop and note which <code>idx</code> is causing the error.<br>
Then use this particular value to check, if <code>dataset.samples[idx]</code> exists or what should be at this position in the list.</p>

username: patawaitte
cooked: <p>Ho thank you very much!<br>
I found my error!<br>
It was during the split of my datasets into batches!</p>

username: manhph2211
cooked: <p>Hi <a class="mention" href="/u/patawaitte">@patawaitte</a> , I meet the same problem <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> Can you tell me how to handle it?  Thanks in advance!</p>

username: Sehaba95
cooked: <p>While loading my CSV file, I have got this error " [IndexError: list index out of range]", I have solved this issue by deleting the column of index in my CSV file, and the project started working again</p>