Best way to incrementally load data for LSTM using DataLoader

username: Ryohei_Namiki
cooked: <p>Currently, I have 500+ Pickle files that hold time-series data in the form of data frames, where each data frame represents a single day. Each of these data frames hold ~10,000 rows of data and ~500 features. I want to use the data and feed it through an LSTM model; however, loading the entire data set and doing a loop to create (input, output) tuples is too memory intensive. In addition, since 1 Pickle file does not represent a single LSTM input, I cannot simply load a single file in the <strong>get_item</strong> function as a solution.</p>
<p>How do I create a DataLoader for this situation? Would I have to load all of the data at once due to it being time-series data? Would I have to change the data format so that each file represents a single sample (which would also have the issue of just having too many files). Any help is appreciated!</p>

username: researchbuddy
cooked: <p>I think the first things it o cut your 500 features to a much shorter one. Feature scaling is definitely one way to look. What kind of dataset is this where you have 500 features? Curious to know. Is this medical related or simply your are trying to do a performance test. <img src="https://discuss.pytorch.org/images/emoji/apple/sunglasses.png?v=9" title=":sunglasses:" class="emoji" alt=":sunglasses:"></p>

username: Laura_Montalvo
cooked: <p>Hi there! Have you tried creating a custom dataset to create the dataloader??<br>
It’s a pytorch class, here are some examples:<br>
<a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a><br>
<a href="https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel" class="onebox" target="_blank" rel="noopener nofollow ugc">https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel</a><br>
<a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset</a><br>
It was really useful to me!</p>

username: Ryohei_Namiki
cooked: <p>Hi! Thanks for the resources. I’m still new to DataLoaders and a little confused about the specific use-case. So I see that the <strong>get_item</strong> function is supposed to return a single sample for the model to train on, and the tutorial does this by loading in a file for each sample; however, that would not be practical for me as I would have a couple million files if I created a file per sample. Is there any alternative use case for DataLoader where I wouldn’t have to have millions of files?</p>

username: Laura_Montalvo
cooked: <p>If your data are images try with the Image Folder class, if not I’m sorry I cant help. My data is all inclusive in a csv so I read the csv and then I feed the dataset class with the data and the labels so I get the data prepared for the dataloader</p>

username: shivammehta007
cooked: <p>You don’t have to manage for each <code>DataLoader </code>, will manage batches for you! You just have to manage the input and transformation for one item of the <code>Dataset</code>, the loader will manage the rest for the Batch. That is the point of having them in the first place.</p>