Data Loader is not working

Flock1: <p>I have multiple <code>csv</code> files which contain 1D data and I want to use each row. Each file contains different number of rows. So I have written a dataloader like this:</p>
<pre><code class="lang-auto">class data_gen(torch.utils.data.Dataset):
    def __init__(self, files):
        
        self.files = files
        print("FILES: ", type(self.files))
        
    def __getitem__(self, i):

        print("GETite,")
        
        file1 = self.files[i]
        print("FILE1: ", file1)
        my_data = np.genfromtxt('/data/'+file1, delimiter=',')
        
        # file1 = np.reshape(file1,(1,len(file1)))
        # file1 = torch.from_numpy(file1).float()
        
        # return data
        print(len(my_data))
        return my_data

    def __len__(self): 
        
        return len(self.files)
</code></pre>
<p>However, when I call it like this:</p>
<pre><code class="lang-auto">train_dl_spec = data_gen(train_files[0])
</code></pre>
<p>I get the following output:</p>
<pre><code class="lang-auto">FILES:  &lt;class 'str'&gt;
</code></pre>
<p>It’s not processing <code>__getitem__</code> for some reason. What could be the reason?</p>

M_T: <p>You’re likely better off concatenating those CSV files prior to initializing the dataset object.</p>
<p>Handle all of that outside of the Dataset.</p>

Flock1: <p>I tried that. But the file becomes so big that it doesn’t load on RAM then.</p>

ptrblck: <aside class="quote no-group" data-username="Flock1" data-post="1" data-topic="123882">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/flock1/48/12220_2.png" class="avatar"> Flock1:</div>
<blockquote>
<p>It’s not processing <code>__getitem__</code> for some reason. What could be the reason?</p>
</blockquote>
</aside>
<p>I’m not sure I understand the issue correctly.<br>
The mentioned output is created in the <code>__init__</code>, so it seems the <code>Dataset</code> is initialized properly.<br>
What kind of issue are you seeing when calling <code>train_dl_spec[0]</code>?</p>

ejguan: <p>You are passing a <code>train_files[0]</code> that is a string to <code>self.files</code>. No sure if I fully understand your question. I guess you want <code>data_gen(train_files)[0]</code> to invoke <code>__getitem__</code> from data_gen instance.</p>

Flock1: <p>I think I was able to solve it. I first restarted the kernel and then, I edited the class:</p>
<pre><code class="lang-auto">class data_gen(torch.utils.data.Dataset):
    def __init__(self, files):
        
        self.files = files
        my_data = np.genfromtxt('/data/'+files, delimiter=',')
        self.dim = my_data.shape[1]
        self.data = []
        
    def __getitem__(self, i):

        file1 = self.files
        my_data = np.genfromtxt('/data/'+file1, delimiter=',')
        self.dim = my_data.shape[1]

        for j in range(my_data.shape[1]):
            tmp = np.reshape(my_data[:,j],(1,my_data.shape[0]))
            tmp = torch.from_numpy(tmp).float()
            self.data.append(tmp)        
        
        return self.data[i]

    def __len__(self): 
        
        return self.dim
</code></pre>
<p>And now, it’s working when I call<br>
<code>train_loader = torch.utils.data.DataLoader(         train_dl_spec, batch_size=128, shuffle=True, num_workers=8, pin_memory=True)</code></p>