How to create and handle the dataloaders for custom dataset (csv files)

username: c_t
cooked: <p>Hello,</p>
<p>I acquired a dataset with tweets where i did some preprocessing on it and now is the moment to load it in Pytorch in order to create and test some models.  It’s the first time that I will use a custom dataset and thus it’s the first time for me to manually handle the dataloaders and the Dataset class.</p>
<p>My questions are these:</p>
<p>First of all, what is the appropriate way to organise the csv files before feeding them to the dataloader? 1 file with all the tweets and their labels?  2 csv files with the tweets and their labels for train and test (train_data, test_data)?  or 4 files (train_x, train_y, test_x, test_y)?</p>
<p>Secondly, I am extremely confused on how I should create the dataloaders,  how to implement the Dataset class and how the text should be represented (in which form) etc.</p>
<p>Can anyone guide me or give me some good resources regarding that specific procedure?</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: sonnguyen
cooked: <p>Regarding csv files, I use pandas to read it. In your case, you should convert each tweet to a vector before loading it to dataloader.<br>
This tutorial explains how to make a dataloader from csv file.<br>
<a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a><br>
So basically, you have to define <em>init</em>_, <strong>len</strong>, and <strong>getitem</strong> functions for a data loader. In addition, your <strong>init</strong> should have the part of converting tweet to a vector.</p>

username: c_t
cooked: <p>Thank you for your answer!</p>
<p>Could you elaborate a little with me and tell me If I perceived anything not correctly?</p>
<p>I have a dataframe with 2 columns (sentiment and text) where the text is already preprocessed.</p>
<p>In the <strong>init()</strong> method of the Dataset class, the dataframe is given as an argument and then I should proceed with the vectorization process.  In the vectorization process, I need to create a vocabulary of each word contained in the dataset and map it to an integer, right?  Then I will create a third column called text_vectorized, where each sentece will be a list containing integers that correspond to the words of the sentence.</p>
<p>In the <strong>len</strong>() method i should return the size of the whole dataset, right?</p>
<p>Last but not least,  In the <strong>getitem</strong>() method, an index must be given as an argument in order to retrieve the corresponding vectorized text along with its label.</p>
<p>Have I missed something, or am I okay to go? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: sonnguyen
cooked: <p>In the <em>init()</em> method, you should make your data ready. I usually make it becomes <em>self.x</em>, and <em>self.t</em> for input vectors and target labels respectively. Yes, you have to create a vocabulary in the process of converting each text to vector. You can define a separate function for the converting to reduce the complexity of <em>init()</em> method.</p>
<p><em>len()</em> and <em>getitem()</em> methods tell the dataloader how you want it to return data in each minibatch. So it will return a bunch of samples, each with separate indexes <em>idx</em>. It will automatically allocate the <em>idx</em> between 0 and the <em>len()</em> of your dataset.</p>

username: c_t
cooked: <p>So the <strong>len</strong>() method should return the dataset’s length, right?</p>
<p>You kinda confused me with the <strong>getitem</strong>() method <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"> In the tutorial from the link you have provided me in your first reply, the <strong>getitem</strong>() returns a single “object” with its label.  Now you told me that it needs to return a bunch of samples.  Could you make it more clear please? Cheers</p>

username: sonnguyen
cooked: <p>Yes, <em>len()</em> method should return the dataset’s length.<br>
The <em>getitem()</em> return a single datapoint at index <em>idx</em> everytime it’s called. The dataloader will call it multiple times, depending on the provided batch_size.<br>
In the tutorial, they use batch_size = 4. shuffle = True tells it to randomly select the index between 0 and the <em>len()</em></p>
<blockquote>
<p>dataloader = DataLoader(transformed_dataset, batch_size=4,<br>
shuffle=True, num_workers=4)</p>
</blockquote>

username: sonnguyen
cooked: <p>In short, to create a dataloader, you need to define your data in <em>init()</em>, how you want it to return a single datapoint given an index <em>idx</em> in <em>getitem()</em>, and the range the <em>idx</em> should go ( between 0 and the <em>len()</em>). The dataloader will handle the rest on its own.</p>

username: c_t
cooked: <p>Thank you very much, it’s all clear now!</p>
<p>One last clarification! At this moment, what we discussed is what i have to implement for now.</p>
<p>Word embeddings, transformation to tensors, etc are later concerns, right?</p>

username: sonnguyen
cooked: <p>I suggest you make your data ready before put the data in dataloader.  The “ready data” means they are all vectors number with corresponding labels. So yes, word embeddings, transformations …<br>
The transformation to tensors can be handled later in training. The idea is that  if you are going to train it on GPU, you should put only the current mini-batch on GPU to save its memory. But if your GPU can handle all of your data at once then it’s fine to do the transformation right in the <em>init()</em> method.</p>