DataLoader - How to load the data in sequence

JalleNation: <p>Ladies and gentlemen, new to the world of ML and its great fun, however I’m slowly but surely going crazy trying to solve this.</p>
<p>I’ll do my best to explain my problem, but I think it’s the DataLoader that’s the issue or its a very easy fix but I cant see the forest because of all the threes.</p>
<p>Any kind soul(s) out there that can help, many thanks in advance!</p>
<p>Have used this as reference for my model:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/nklingen/Transformer-Time-Series-Forecasting">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/nklingen/Transformer-Time-Series-Forecasting" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/510afdd422fc0252935ec2ab7b5d515b5c4417c5f44ea854acbb8e75fe573ed2/nklingen/Transformer-Time-Series-Forecasting" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/nklingen/Transformer-Time-Series-Forecasting" target="_blank" rel="noopener nofollow ugc">GitHub - nklingen/Transformer-Time-Series-Forecasting</a></h3>

  <p>Contribute to nklingen/Transformer-Time-Series-Forecasting development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Everything is working well but when I’m trying to reuse the DataLoader &amp; inference for my timeseries data for predictions I´m just not getting the results I need. The DataSet is a timeseries where batches are linked to a Re-index. Let’s say that I have 5 batches / indexes in my data and I want a prediction made for each one in sequence. Since the DataLoader is pulling the index from <strong>getitem</strong> and that in turn pulls an index between 1 and len from the data, DataLoader set to shuffle can give me predictions of index 2,2,3,2,5 = 3 predictions when I want it to do 1,2,3,4,5 = 5 predictions.</p>
<p>Giving <strong>getitem</strong> a fixed index and turning of shuffle I get (1)5 predictions of the same index.</p>
<p>Surely not the right way to do it but I have tried to iterate in the <strong>getitem</strong> to achieve above but nothing is working. I saw some documentation on itterableDataset with start / end, but couldn’t get that to work either (I’m just to inexperienced at the moment to get my head around it).</p>
<p>Please help so I can continue my project.</p>

ptrblck: <p>I’m not sure I understand the use case completely, but</p>
<aside class="quote no-group" data-username="JalleNation" data-post="1" data-topic="134064">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/d07c76/48.png" class="avatar"> JalleNation:</div>
<blockquote>
<p>Since the DataLoader is pulling the index from <strong>getitem</strong> and that in turn pulls an index between 1 and len from the data,</p>
</blockquote>
</aside>
<p>that’s not the case. By default (unless you are creating your own <code>DataLoader</code>) the sampler will be used to create the batch indices and the <code>DataLoader</code> will grab these indices and pass it to <code>Dataset.__getitem__</code>. The <code>__getitem__</code> method is not creating the indices, but consuming them.<br>
Could you describe the issue a bit more, in particular where the indices are coming from?</p>

JalleNation: <p>Im probably not explaining very well, sorry about that.</p>
<p>Im using a CSV file with data that looks something like this:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Date</th>
<th>Name</th>
<th>Re-index</th>
<th>Data1</th>
<th>Data2</th>
<th>Data3</th>
<th>Data4</th>
</tr>
</thead>
<tbody>
<tr>
<td>2021-10-13</td>
<td>A</td>
<td>1</td>
<td>x</td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>2021-10-14</td>
<td>A</td>
<td>1</td>
<td>x</td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>2021-10-15</td>
<td>A</td>
<td>1</td>
<td>x</td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>2021-10-13</td>
<td>B</td>
<td>2</td>
<td>x</td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>2021-10-14</td>
<td>B</td>
<td>2</td>
<td>x</td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>2021-10-15</td>
<td>B</td>
<td>2</td>
<td>x</td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>2021-10-13</td>
<td>C</td>
<td>3</td>
<td>x</td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>2021-10-14</td>
<td>C</td>
<td>3</td>
<td>x</td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>2021-10-15</td>
<td>C</td>
<td>3</td>
<td>x</td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
</tbody>
</table>
</div><p>Every re-index number I want a prediction on in sequence. But I can only get the model to give me  either random predictions x times or x times predicitons of a specific re-index.</p>
<p>The CSV Data goes into the DataLoader and as I understand the <strong>getitem</strong> will take one of the Re-Index batches and send to the model. For predictions I use the inference tab where I can specify how many times it should iterate a prediction.  When I run DataLoader Shuffle True it gives random Re-index predictions x times, when shuffle = False it gives a prediction on max re-reindex x times and when I specify a Re-index at <strong>getitem</strong> idx=2 I will get a prediction on that Index x times.</p>
<p>In short it gives me predictions of Re-index at range(3) of either:<br>
DataLoader Shuffle = True - 2,1,2<br>
DataLoader Shuffle = False = Len / max - 3,3,3<br>
DataLoader Shuffle = Flase, idx Specific - 2,2,2<br>
I want - 1,2,3</p>
<p>Thanks for helping out, much appreciated!</p>

ptrblck: <p>Thanks for the update. If I understand your use case correctly you want to use a specific sampling strategy where each batch should contain a sample of each <code>Re-index</code> or at least they should not be repeated in a single batch.<br>
In that case you could create a custom sampler and implement the sampling logic there to make sure each batch contains different <code>Re-index</code> values.</p>

JalleNation: <p>No thank you for helping out!<br>
Correct, and the model is actually doing this at the moment, but its just not doing it in sequence but randomly or static. And I have tried to look at how to change the DataLoader to do this. But Im too green to make it happen, dont know how to make the <strong>getitem</strong> idx to go in sequence. If you can point me in the right direction with a bit of guidance that would be awesome!  Below is the full code for the DL.</p>
<p>import pandas as pd<br>
from torch.utils.data import Dataset, DataLoader<br>
from sklearn.preprocessing import MinMaxScaler<br>
import os<br>
import torch<br>
import numpy as np<br>
import random<br>
import matplotlib.pyplot as plt<br>
from joblib import dump<br>
from icecream import ic</p>
<p>class SensorDataset(Dataset):<br>
“”“Face Landmarks dataset.”""</p>
<pre><code>def __init__(self, csv_name, root_dir, training_length, forecast_window):
    """
    Args:
        csv_file (string): Path to the csv file.
        root_dir (string): Directory
    """
    
    # load raw data file
    csv_file = os.path.join(root_dir, csv_name)
    self.df = pd.read_csv(csv_file)
    self.root_dir = root_dir
    self.transform = MinMaxScaler()
    self.T = training_length
    self.S = forecast_window

def __len__(self):
    # return number of sensors
    return len(self.df.groupby(by=["reindexed_id"]))

# Will pull an index between 0 and __len__. 
def __getitem__(self, idx):
    
    # Sensors are indexed from 1
    idx = idx+1

    # np.random.seed(0)

    start = np.random.randint(0, len(self.df[self.df["reindexed_id"]==idx]) - self.T - self.S) 
    sensor_number = str(self.df[self.df["reindexed_id"]==idx][["sensor_id"]][start:start+1].values.item())
    index_in = torch.tensor([i for i in range(start, start+self.T)])
    index_tar = torch.tensor([i for i in range(start + self.T, start + self.T + self.S)])
    _input = torch.tensor(self.df[self.df["reindexed_id"]==idx][["humidity", "sin_hour", "cos_hour", "sin_day", "cos_day", "sin_month", "cos_month"]][start : start + self.T].values)
    target = torch.tensor(self.df[self.df["reindexed_id"]==idx][["humidity", "sin_hour", "cos_hour", "sin_day", "cos_day", "sin_month", "cos_month"]][start + self.T : start + self.T + self.S].values)

    # scalar is fit only to the input, to avoid the scaled values "leaking" information about the target range.
    # scalar is fit only for humidity, as the timestamps are already scaled
    # scalar input/output of shape: [n_samples, n_features].
    scaler = self.transform

    scaler.fit(_input[:,0].unsqueeze(-1))
    _input[:,0] = torch.tensor(scaler.transform(_input[:,0].unsqueeze(-1)).squeeze(-1))
    target[:,0] = torch.tensor(scaler.transform(target[:,0].unsqueeze(-1)).squeeze(-1))

    # save the scalar to be used later when inverse translating the data for plotting.
    dump(scaler, 'scalar_item.joblib')

    return index_in, index_tar, _input, target, sensor_number</code></pre>

ptrblck: <p>Take a look at <a href="https://discuss.pytorch.org/t/how-to-use-batchsampler-with-getitem-dataset/78788/4">this post</a> to check how the <code>BatchSampler</code> could work (it’ll pass a batch of indices to the <code>__getitem__</code> method). To do so, you could create a custom sampling strategy in a derived sampler class and create the batch of indices <a href="https://github.com/pytorch/pytorch/blob/871a31b9c444e52ba0cc6667fb317c11802ac4de/torch/utils/data/sampler.py#L227-L230">here</a>.<br>
Let me know, if you get stuck and I can try to follow up with an example code snippet.</p>

JalleNation: <p>Thank you very much for you help. Ill check it out!</p>