Help please! How to feed RNN with two time steps/inputs?

username: Laura_Montalvo
cooked: <p>Hello there! Here’s my problem:</p>
<p>I’m trying to use RNN for classification problem in which I have 25 classes. The dataset is conform of 1500 samples that are vectors of 800 columns, so each input has shape (1, 800).<br>
For each class I have a number of samples and because each element of the vector represents RSS each of the samples for each class is correlated.</p>
<p>What I want to do is to feed the RNN with two vectors each one with the respect label/target, so the hidden state can be shared between the first and the second so I take advantage of RNN’s sequential information.<br>
How ever I don’t know how to express that, Do I have to create the “sequences” in the dataset? Or it is something about feeding the model?</p>
<p>Thanks!!</p>

username: ptrblck
cooked: <p>RNNs in PyTorch expect the input to have a temporal dimension. The default input shape would be <code>[seq_len, batch_size, features]</code>, where <code>seq_len</code> defines the temporal sequence length.<br>
You could use <code>batch_first=True</code>, which would then expect an input of <code>[batch_size, seq_len, features]</code>.<br>
The RNN would execute the corresponding operations on the complete sequence length automatically.</p>

username: Laura_Montalvo
cooked: <p>Thanks, I get that, however I don’t know how to prepare my data to adopt that shape.<br>
I create the dataset with:</p>
<pre><code class="lang-auto">class WifiDataset(Dataset):
    def __init__(self, data,labels=None,transforms=None):
        self.X = data
        self.y = labels
        self.transforms = transforms    
    def __len__(self):
        return (len(np.asarray(self.X)))
    def __getitem__(self, i):
        data = self.X[i]        
        if self.transforms:
            data = self.transforms(data)
              if self.y is not None:
            return (data, self.y[i])
        else:
            return data
</code></pre>
<p>With this I create my dataset and then my dataloader, where my data,target in loader<br>
is like [batch_size, 1, input_size (=800)] and target = [batch_size].<br>
I dont know if is after I create the dataloader where I have to do something to turn inputs to [seq_len, batch_size, featers] or if it is in the dataset creation where I have to set the data to get seq_len data instead of jus one.<br>
Thanks!</p>

username: Laura_Montalvo
cooked: <p>I tried also coding like this (it’s an example to test how to make sequences):</p>
<pre><code class="lang-auto">class DataPrueba(Dataset):
    def __init__(self, data, seq, labels=None, transforms=None):
        self.X = data
        self.y = labels
        self.seq = seq
        self.transforms = transforms

    def __len__(self):
        return len(np.asarray(self.X))

    def __getitem__(self, i):
        if self.y is not None:
            if i + self.seq &gt; self.__len__():
                if self.transforms is not None:
                    item = []
                    item[:self.__len__()-i] = self.transforms(self.X[i:])
                    target_item = []
                    target_item[:self.__len__()-i] = self.transforms(self.y[i:])
                    data = item
                    labels = target_item
                else:
                    item = []
                    item[:self.__len__()-i] = self.X[i:]
                    data = item
                    target_item = []
                    target_item[:self.__len__()-i] = self.y[i:]
                    labels = target_item
            else:
                if self.transforms is not None:
                    data = self.transforms(self.X[i:i+self.seq])
                    labels = self.transforms(self.y[i:i+self.seq])
                else:
                    data = self.X[i:i+self.seq]
                    labels = self.y[i:i+self.seq]
            return data, labels
        else:
            if i + self.seq &gt; self.__len__():
                if self.transforms is not None:
                    item = []
                    item[:self.__len__()-i] = self.transforms(self.X[i:])
                    data = item
                else:
                    item = []
                    item[:self.__len__()-i] = self.X[i:]
                    data = item
            else:
                if self.transforms is not None:
                    data = self.transforms(self.X[i:i+self.seq])
                else:
                    data = self.X[i:i+self.seq]
            return data
        
        
x = torch.randn(6, 10)
labels = torch.randint(0, 6, (6,))

dataset = DataPrueba(x, seq=2, labels=labels, transforms=None)       
loader = DataLoader(dataset, batch_size=2)
in_size = x.shape[1]

for i, j in loader:
    print(i)
    print(i.shape)
    print(j)
    print(j.shape)
    break
        

model = nn.Sequential(nn.LSTM(input_size=10, hidden_size=35,num_layers=1, batch_first=True)),
                      nn.Linear(35, 6))

input_seq = i
output_seq, _ = model(torch.FloatTensor(input_seq))
last_output = output_seq[-1]

loss = nn.CrossEntropyLoss()
err = loss(last_output, j)
err.backward()
</code></pre>
<p>However if the Linear layeres is uncommented I get this error, that I guess t happens because the input is two sequences:</p>
<pre><code class="lang-auto">Traceback (most recent call last):

  File "/home/lauram/Desktop/RNN_TIMESEQ.py", line 110, in &lt;module&gt;
    output_seq, _ = model(torch.FloatTensor(input_seq))

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py", line 1688, in linear
    if input.dim() == 2 and bias is not None:

AttributeError: 'tuple' object has no attribute 'dim'
</code></pre>
<p>And when the Linear layer is commented then the CrossEntropyLoss cant be applied because as the input is two sequences is a 2D tensor, while CrossEntropyLoss expects just 1D tensor…</p>
<pre><code class="lang-auto">Traceback (most recent call last):

  File "/home/lauram/Desktop/RNN_TIMESEQ.py", line 114, in &lt;module&gt;
    err = loss(last_output, j)

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 961, in forward
    return F.cross_entropy(input, target, weight=self.weight,

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py", line 2468, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py", line 2264, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: 1D target tensor expected, multi-target not supported
</code></pre>
<p>Any idea on how to fix it? <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"></p>

username: ptrblck
cooked: <p>The idea of creating the sequences inside the <code>Dataset</code> is valid and you could either stick to your loop or have a look at e.g. <a href="https://discuss.pytorch.org/t/dataloader-for-a-lstm-model-with-a-sliding-window/22235/2">this simple post</a> to see, how to create “sliding windows”.</p>
<p>I would not recommend to put an <code>nn.LSTM</code> into an <code>nn.Sequential</code> container, since it’s working with multiple inputs and outputs.<br>
Create a custom <code>nn.Module</code> instead so that you could use e.g. the last time step of the lstm output to feed into the linear layer.</p>
<aside class="quote no-group" data-username="Laura_Montalvo" data-post="4" data-topic="113084">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/laura_montalvo/48/34770_2.png" class="avatar"> Laura_Montalvo:</div>
<blockquote>
<p>And when the Linear layer is commented then the CrossEntropyLoss cant be applied because as the input is two sequences is a 2D tensor, while CrossEntropyLoss expects just 1D tensor…</p>
</blockquote>
</aside>
<p>I assume you are using the complete hidden state as the model output, which won’t work in this case.<br>
If each sequence has a single target, you could e.g. use the last time step of the lstm output or reduce the temporal dimension in any other way (e.g. taking the mean).</p>

username: Laura_Montalvo
cooked: <p>I get it, thanks. I’m using some kind of “sliding windows” however, when using this sliding window, the last element of the (i,j) in loader does not have the same dimensions as the previous ones. I mean, when idx+seqlength (seqlength is like sliding window in my example) is &gt; than <code>self.__len__</code> then the last data returned by get item has just idx elements. I mean, imagine sequence length is 2, then if when creating the dataset:</p>
<ul>
<li>input size = 10</li>
<li>batch_size = 2</li>
<li>sequence_length = 2</li>
<li>nº of samples = 6<br>
then in the data, target in loader would return:</li>
<li>data.shape = [2,2,10] target shape [2,2] for the first iter</li>
<li>data.shape = [2,2,10] target shape [2,2] for the second iter<br>
and for the third iter then it will return an error like:<br>
RuntimeError: stack expects each tensor to be equal size, but got [2, 10] at entry 0 and [1, 10] at entry 1<br>
because the last sequence length condition is not satisfied. How could I solve that? My data are vectors with values 0 and target values 0 as well so I don’t think padding with zeros would be something reliable because when training the dataset it would be mistaking the padded sequences with some target and data. Below is the code I’m using (it’s a test code because my dataset is much larger so I rather prefer trying with something smaller first). Thanks!</li>
</ul>
<pre><code class="lang-auto">class DataPrueba(Dataset):
    def __init__(self, data, seq, labels=None, transforms=None):
        self.X = data
        self.y = labels
        self.seq = seq
        self.transforms = transforms

    def __len__(self):
        return len(np.asarray(self.X))

    def __getitem__(self, i):
        if self.y is not None:
            if i + self.seq &gt; self.__len__():
                if self.transforms is not None:
                    item = []
                    item[:self.__len__()-i] = self.transforms(self.X[i:])
                    target_item = []
                    target_item[:self.__len__()-i] = self.transforms(self.y[i:])
                    data = item
                    labels = target_item
                else:
                    data = torch.zeros_like(self.X[i:])
                    data[:self.__len__()-i] = self.X[i:]
                    labels = torch.zeros_like(self.y[i:])
                    labels[:self.__len__()-i] = self.y[i:]
            else:
                if self.transforms is not None:
                    data = self.transforms(self.X[i:i+self.seq])
                    labels = self.transforms(self.y[i:i+self.seq])
                else:
                    data = self.X[i:i+self.seq]
                    labels = self.y[i:i+self.seq]
            return torch.FloatTensor(data), labels
        else:
            if i + self.seq &gt; self.__len__():
                if self.transforms is not None:
                    item = []
                    item[:self.__len__()-i] = self.transforms(self.X[i:])
                    data = item
                else:
                    item = []
                    item[:self.__len__()-i] = self.X[i:]
                    data = item
            else:
                if self.transforms is not None:
                    data = self.transforms(self.X[i:i+self.seq])
                else:
                    data = self.X[i:i+self.seq]
            return torch.FloatTensor(data)
        
        
x = torch.randn(6, 10)
labels = torch.randint(0, 6, (6,))
num_classes = 6;
hidden_size = 35
batch_size = 2
seq = 2
num_layers=1

transform = transforms.Lambda(lambda x: listToTensor(x))
dataset = DataPrueba(x, seq=seq, labels=labels, transforms=None)       
loader = DataLoader(dataset, batch_size=batch_size, drop_last=True)
in_size = x.shape[1]

for idx, (i, j) in enumerate(loader):
    print(idx)
    print(str(i)+'\t'+str(i.shape))
    print(str(j)+'\t'+str(j.shape))
    
# Fully connected neural network with one hidden layer
class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(RNN, self).__init__()
        self.num_layers = num_layers
        self.hidden_size = hidden_size
        self.rnn = nn.RNN(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, num_classes)
        
    def forward(self, x):
        # Set initial hidden states (and cell states for RNN/LSTM)
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size)
        # Forward propagate RNN
        out, _ = self.rnn(x, h0)  
        out = self.fc(out)
        return out

model = RNN(in_size, hidden_size, num_layers, num_classes)
criterion = nn.CrossEntropyLoss()

for epoch in range(10):
    for i, (images, labels) in enumerate(loader):  
        # origin shape: [N, 1, 28, 28]
        # resized: [N, 28, 28]
        images = images.reshape(-1, seq, in_size)
        print(images.shape)
        # Forward pass
        outputs = model(images)
        loss = criterion(outputs.permute(0,2,1), labels)
</code></pre>

username: ptrblck
cooked: <p>In the posted code snippet I’ve reduced the length of the <code>Dataset</code> by the <code>window</code> so that only “full” windows would be returned. I think this would also work for you.</p>

username: Laura_Montalvo
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="113084">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>In the posted code snippet I’ve reduced the length of the <code>Dataset</code> by the <code>window</code> so that only “full” windows would be returned. I think this would also work for you</p>
</blockquote>
</aside>
<p>I tried doing that by replacing the sentence</p>
<pre data-code-wrap="def"><code class="lang-def">        return len(np.asarray(self.X))
</code></pre>
<p>with</p>
<pre data-code-wrap="def"><code class="lang-def">        return len(self.X) - self.seq
</code></pre>
<p>but this error comes if I do that:</p>
<pre><code class="lang-auto">  File "/home/lauram/Desktop/RNN_TIMESEQ.py", line 108, in &lt;module&gt;
    for idx, (i, j) in enumerate(loader):

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]

  File "/home/lauram/Desktop/RNN_TIMESEQ.py", line 57, in __getitem__
    data[:self.__len__()-i] = self.X[i:]

RuntimeError: The expanded size of the tensor (1) must match the existing size (3) at non-singleton dimension 0.  Target sizes: [1, 10].  Tensor sizes: [3, 10]
</code></pre>

username: ptrblck
cooked: <p>This error is expected now, since you’ve reduced the <code>__len__</code> of the dataset, and this slicing will thus fail:</p>
<pre><code class="lang-python">data = torch.zeros_like(self.X[i:])
data[:self.__len__()-i] = self.X[i:]
</code></pre>
<p>I’m currently unsure, why you are initializing <code>data</code> as a zero tensor in the shape of <code>X[i:]</code> and assign the values in the next line via a slicing operation.<br>
Wouldn’t <code>data = self.X[i:]</code> just work?</p>

username: Laura_Montalvo
cooked: <p>I changed it to:</p>
<pre><code class="lang-auto">class DataPrueba(Dataset):
    def __init__(self, data, seq, labels=None, transforms=None):
        self.X = data
        self.y = labels
        self.seq = seq
        self.transforms = transforms
    def __len__(self):
        return len(self.X) - self.seq
    def __getitem__(self, i):
        if self.y is not None:
            if i + self.seq &gt; self.__len__():
                if self.transforms is not None:
                    item = []
                    item[:self.__len__()-i] = self.transforms(self.X[i:])
                    target_item = []
                    target_item[:self.__len__()-i] = self.transforms(self.y[i:])
                    data = item
                    labels = target_item
                else:
                    data = self.X[i:]
                    labels = self.y[i:]
</code></pre>
<p>However I still get the error expected above:</p>
<pre><code class="lang-auto">  File "/home/lauram/Desktop/RNN_TIMESEQ.py", line 113, in &lt;module&gt;
    for idx, (i, j) in enumerate(loader):

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 83, in default_collate
    return [default_collate(samples) for samples in transposed]

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 83, in &lt;listcomp&gt;
    return [default_collate(samples) for samples in transposed]

  File "/home/lauram/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)

RuntimeError: stack expects each tensor to be equal size, but got [2, 10] at entry 0 and [3, 10] at entry 1
</code></pre>
<p>I also though data = self.X[i:] would work, however (maybe because of the slicing) the next error appears:</p>
<pre><code class="lang-auto">  File "/home/lauram/Desktop/RNN_TIMESEQ.py", line 57, in __getitem__
    data[:self.__len__()-i] = self.X[i:]

UnboundLocalError: local variable 'data' referenced before assignment 
</code></pre>
<p>However it should have been because of the slicing, because that error is not happening now.</p>