Correctly feeding LSTM with minibatch time sequence data

goffta: <p>Hi,</p>
<p>I’m having trouble with setting the correct tensor sizes for my research. I have about 400000 data points in the form: time, value. They are in a csv file. I would like to feed my LSTM in mini batches of 20 sequences of length 100 for each batch. I’m not sure how to that properly. Any advise appreciated.</p>

ptrblck: <p>From the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM">docs</a>:</p>
<blockquote>
<p><strong>input</strong> of shape (seq_len, batch, input_size): tensor containing the features of the input sequence. …</p>
</blockquote>
<p>How many features does each sample contain?<br>
Assuming it’s just a single feature, each batch should have the shape <code>[100, 20, 1]</code> using the default setup.<br>
However, if you specify <code>batch_first=True</code>, you would need to pass your data as <code>[20, 100, 1]</code>.</p>

goffta: <p>Each sample contains only 1 feature, paired with 1 label. The data is a time series. I’ve been reading the data using Pandas. I’ve attempted to use the DataSet and Dataloader. When using dataloader, to obtain the correct tensor, would it be as simple as specifying batch_size=20? It seems as this would only load a [20,1] tensor.</p>

ptrblck: <p>If you load a single sample in your <code>Dataset</code>'s <code>__getitem__</code> method in the shape <code>[seq_len, features]</code>, your <code>DataLoader</code> should return <code>[batch_size, seq_len, features]</code> using the default <code>collate_fn</code>.</p>
<p>In the training loop you could permute the dimensions to match <code>[seq_len, batch_size, features]</code> or just use <code>batch_size=First</code> in your LSTM.</p>

goffta: <p>First of all, thanks for the replies and advice! if I’m understanding correctly, my Dataset class should look like this:<br>
<code><br>
class OUDataset(Dataset):<br>
def <strong>init</strong>(self, csv_file):<br>
self.oudataframe = pd.read_csv(csv_file)<br>
self.seq_len = 100<br>
def <strong>len</strong>(self):<br>
return len(self.oudataframe)<br>
def <strong>getitem</strong>(self, idx):<br>
dy = self.oudataframe.iloc[idx, 0:]<br>
return self.seq_len, dy<br>
</code></p>
<p>However, I get an error when calling dataloader:<br>
<code><br>
TypeError: batch must contain tensors, numbers, dicts or lists; found &lt;class ‘pandas.core.series.Series’&gt;<br>
</code></p>
<p>My call to Dataloader looks like:<br>
<code><br>
dataSet = OUDataset(csv_file=filelocationhere)<br>
dataloader = Dataloader(dataSet, batch_size=20, shuffle=True)<br>
</code></p>
<p>Any idea what might be going on?</p>
<p>Edit:<br>
I changed the <code><strong>getitem</strong></code> method to:<br>
<code>dy = torch.tensor(self.oudataframe.iloc[idx,0:])</code><br>
and no longer get the same error. How can I verify that my tensors are the correct shape?</p>

ptrblck: <p>You could call check it via:</p>
<pre><code class="lang-python">data, target = next(iter(dataloader))
print(data.shape)
print(target.shape)
</code></pre>

goffta: <p>So, in the training loop I can do something l can loop for each epoch, and instead of enumerating on the dataloader, just use <code> data, target = next(iter(dataloader)) </code> and then do my calls to model, optimizer, and loss?</p>

ptrblck: <p>You could create an <code>iterator = iter(dataloader)</code> and then call <code>data, target = next(iterator)</code> inside your training loop, but you would meed to catch the <code>StopIteration</code> manually, so I would just stick to:</p>
<pre><code class="lang-python">for data, target in dataloder:
    ...
</code></pre>
<p>I’ve just used this line of code to get a single sample to check the shape.</p>

goffta: <p>Thanks. I must be doing something incorrect. With code that looks like this:<br>
<code><br>
dataloader = DataLoader(dataSet, batch_size=20, shuffle=True)</code></p>
<p>for t, dy in enumerate(dataloader):<br>
print(t)<br>
print(dy)<br>
print(dy[1])<br>
break<br>
<br>
I get:<br>
<code><br>
Testing dataset class …<br>
Moving to dataloader section …<br>
0<br>
[tensor([100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100,<br>
100, 100, 100, 100, 100, 100]), tensor([[ 1.7786e+04,  7.9729e-01],<br>
[ 1.2340e+04, -1.6030e-01],<br>
[ 9.1269e+03, -7.1534e-02],<br>
[ 2.9582e+04, -7.2178e-01],<br>
[ 1.2473e+04, -3.4016e-01],<br>
[ 1.1071e+04,  2.3349e-01],<br>
[ 3.2279e+04, -1.3710e+00],<br>
[ 1.4720e+03,  2.4033e-01],<br>
[ 2.0018e+04, -2.2206e-01],<br>
[ 1.4384e+04, -7.4043e-01],<br>
[ 3.5270e+02, -3.3346e-01],<br>
[ 1.1394e+04,  3.3650e-01],<br>
[ 2.2026e+04, -4.4804e-02],<br>
[ 2.3803e+04,  2.2828e-01],<br>
[ 1.6722e+04,  1.6180e-01],<br>
[ 2.7046e+04, -6.7901e-01],<br>
[ 2.9860e+04,  1.5923e-01],<br>
[ 2.4130e+04, -7.0021e-01],<br>
[ 1.1229e+04,  7.4529e-01],<br>
[ 3.6677e+04, -5.1128e-01]])]<br>
tensor([[ 1.7786e+04,  7.9729e-01],<br>
[ 1.2340e+04, -1.6030e-01],<br>
[ 9.1269e+03, -7.1534e-02],<br>
[ 2.9582e+04, -7.2178e-01],<br>
[ 1.2473e+04, -3.4016e-01],<br>
[ 1.1071e+04,  2.3349e-01],<br>
[ 3.2279e+04, -1.3710e+00],<br>
[ 1.4720e+03,  2.4033e-01],<br>
[ 2.0018e+04, -2.2206e-01],<br>
[ 1.4384e+04, -7.4043e-01],<br>
[ 3.5270e+02, -3.3346e-01],<br>
[ 1.1394e+04,  3.3650e-01],<br>
[ 2.2026e+04, -4.4804e-02],<br>
[ 2.3803e+04,  2.2828e-01],<br>
[ 1.6722e+04,  1.6180e-01],<br>
[ 2.7046e+04, -6.7901e-01],<br>
[ 2.9860e+04,  1.5923e-01],<br>
[ 2.4130e+04, -7.0021e-01],<br>
[ 1.1229e+04,  7.4529e-01],<br>
[ 3.6677e+04, -5.1128e-01]])<br>
</code><br>
This seems incorrect - while I seem to have 20 batches of 100, when printing dy (features, labels), I only seem to have 20 different points in the sequence, where I was expecting to have 100. What am I doing incorrectly? I don’t mind sharing the complete source if necessary.</p>
<p>Thanks in advance!</p>

ptrblck: <p>There might be a misunderstanding I’ve missed in your earlier post.<br>
When I said your <code>DataLoader</code> would return a batch of shape <code>[batch_size, seq_len, features]</code>, I was referring to the dimensions, e.g. <code>[20, 100, 2]</code>.</p>
<p>In your custom <code>Dataset</code> you are directly returning the value of <code>self.seq_len</code>.<br>
Instead you should create a data sample of the shape <code>[seq_len, features]</code>.<br>
I was assuming this format was already stored in your <code>oudataframe</code>.</p>

goffta: <p>So,</p>
<p>my getitem method in the custom Dataset should be:<br>
<code><br>
def <strong>getitem</strong>(self, idx):<br>
data = torch.zeros(100, 2)<br>
for i in range(0, 100):<br>
data[i] = torch.tensor(self.oudataframe.iloc[idx,0:])<br>
return data<br>
</code></p>

ptrblck: <p>This would create a tensor with 100 identical values, as you are indexing the dataframe with <code>[idx, 0:]</code>, which aren’t changed in the loop.</p>

goffta: <p>Ah, thanks for catching that silly error. It should be idx+i, then. Doing so should <em>hopefully</em> give me the correct shape. I must have a fundamental misunderstanding of what’s actually happening here. After making the change in the <strong>getitem</strong> method of the custom dataset to:<br>
<code><br>
def <strong>getitem</strong>(self, idx):<br>
data = torch.zeros(100, 2)<br>
for i in range(0, 100):<br>
data[i] = torch.tensor(self.oudataframe.iloc[idx+i,0:])<br>
return data<br>
</code><br>
and subsequently attempting to check shape via:<br>
<code><br>
dataloader = DataLoader(dataSet, batch_size=20, shuffle=False)<br>
data, target = next(iter(dataloader))<br>
print(data.shape)<br>
print(target.shape)<br>
</code><br>
I get:<br>
<code><br>
data, target = next(iter(dataloader))<br>
ValueError: too many values to unpack (expected 2)<br>
</code><br>
It seems that I am not actually returning what I think I am, or my syntax is incorrect.</p>
<p>Edit:</p>
<p>After setting a test variable to dataSet[0], the shape of the test variable is indeed [100,2].<br>
However, I don’t seem to be understanding how to iterate properly through the newly created dataloader object. Any insights?</p>

ptrblck: <p>You just forgot to return the target tensor, as you are currently using <code>return data</code>.<br>
Return the target as <code>return data, target</code> or just assign<br>
<code>data = next(iter(dataloader))</code>.</p>
<p>Note, that you are currently returning overlapping “windows” in your approach and you should take care of indexing out of bounds (by using <code>idx+100</code>).<br>
You could fix this by subtracting <code>100</code> from the overall length returned in <code>def __len__(self)</code>.</p>

goffta: <p>So, from what I seem to be understanding - I should have a seperate target tensor (i.e. the labels of each data point.) Should I seperate the (time, dy) components from the csv in the <strong>init</strong> method of the dataset?<br>
Edit:<br>
The first few lines of my csv are:<br>
<code><br>
0.0, 1.0<br>
0.1, 1.15478042212<br>
0.2, 1.31513637888<br>
0.30000000000000004, 1.12594414055<br>
0.4, 0.688184511979<br>
0.5, 0.249467795528<br>
0.6000000000000001, 0.140746339238<br>
0.7000000000000001, 0.0152185819711<br>
0.8, 0.260358790336<br>
0.9, -0.0533512260524<br>
</code><br>
where column 0 is a step in time, and column 1 is the corresponding value. I took to this to be that t is my data (feature), and dy is my target (label). Am I thinking incorrectly?</p>

ptrblck: <p>So your data should be the linearly increasing time steps, while your target is the (at first sight) random data?</p>
<p>I would split the data and target in <code>__getitem__</code> and just return both values, as this makes the training loop cleaner in my opinion.<br>
You could do it in <code>__init__</code> or <code>__getitem__</code> as it shouldn’t really matter.</p>

goffta: <p>Yes, my data is just linearly increasing timesteps, while the targets were generated based upon a predefined function, and adding in gaussian white noise. It’s a 1D model of an Ornstein Uhlenbeck process. So, I should be splitting the data, ‘stacking’ as I have in the loop on both columns, and returning the ordered pair as a 100x1,100x1 tensor?</p>

ptrblck: <p>Yes, that sounds right.<br>
Let me know, how the experiments went! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

goffta: <p>Will do. In the training loop I believe I should be able to do:</p>
<code>
for data, target in enumerate(dataloader):
<p>nn.LSTM(data)<br>
</p></code>
<p>And then use targets for my loss function… I think. With a data set of 400000 points, of this structure, I believe there should be around 2000 epochs to process all of the data. This also means that after I call DataLoader, len(dataloader) should return 2000?</p>

goffta: <p>This has been pretty odd. To actually access the correct data (t variable), I’ve had to only use the target. from the for data, target in enumerate(dataloader), data returns an integer, where target returns a tensor of [100,2] size. Is this something with how pandas handles things? The dataframe only has 2 indices, because trying to pull the target values specificially within the dataframe results in an out of bounds error.</p>
<p>Just for completeness here… I’m going to post my dataset method to avoid confusion.</p>
<pre><code class="lang-auto"># Data set formatting for easier input and processing by network.
class OUDataset(Dataset):

  def __init__(self, csv_file):
    self.oudataframe = pd.read_csv(csv_file)

  def __len__(self):
    return len(self.oudataframe) - 100
  
  def __getitem__(self, idx):
    data = torch.zeros(100, 1)
    target = torch.zeros(100,1)
    for i in range(0, 100):
      data[i] = torch.tensor(self.oudataframe.iloc[idx+i][0])
      target[i] = torch.tensor(self.oudataframe.iloc[idx+i][1])
    return [data, target]

######################################################################################

######################################################################################
# Function/Method definitions
######################################################################################

def main():
  k = 0 # iteration 0.
  print("Running!")
  print("Testing dataset class ...")
  dataSet = OUDataset(csv_file="../../trainingData/python/data/trainingDataOrnsteinUhlenbeck.dat")

  print("Moving to dataloader section ...")
  counter = 0 
  
  dataloader = DataLoader(dataSet, batch_size=20, shuffle=False)
  for data, target in enumerate(dataloader):
    print(data)
    print(target[0])
    print(target[1])
    break
</code></pre>