Many-to-One LSTM Input Shape

JapinderSandhu: <aside class="onebox allowlistedgeneric" data-onebox-src="https://wandb.ai/ayush-thakur/dl-question-bank/reports/One-to-Many-Many-to-One-and-Many-to-Many-LSTM-Examples-in-Keras--VmlldzoyMDIzOTM">
  <header class="source">
      <img src="https://cdn.wandb.ai/production/f88bac3f7/favicon.png" class="site-icon" width="32" height="32">

      <a href="https://wandb.ai/ayush-thakur/dl-question-bank/reports/One-to-Many-Many-to-One-and-Many-to-Many-LSTM-Examples-in-Keras--VmlldzoyMDIzOTM" target="_blank" rel="noopener nofollow ugc" title="07:10PM - 12 August 2020">W&amp;B – 12 Aug 20</a>
  </header>

  <article class="onebox-body">
    <img src="https://storage.googleapis.com/wandb-production.appspot.com/ayush-thakur/images/views/202393/preview.png?md5=9706f931d0c688f8c823fd987f242238" class="thumbnail onebox-avatar" width="300" height="300">

<h3><a href="https://wandb.ai/ayush-thakur/dl-question-bank/reports/One-to-Many-Many-to-One-and-Many-to-Many-LSTM-Examples-in-Keras--VmlldzoyMDIzOTM" target="_blank" rel="noopener nofollow ugc">One-to-Many, Many-to-One and Many-to-Many LSTM Examples in Keras</a></h3>

  <p>Use cases of LSTM for different deep learning tasks. Made by Ayush Thakur using Weights &amp; Biases</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>“One-to-many sequence problems are sequence problems where the input data has one time-step, and the output contains a vector of multiple values or multiple time-steps.”</p>
<p>I am trying to make a One-to-many LSTM based model in pytorch.</p>
<p>It is a binary classification problem there is only 2 classes. However, the labels should be a vector of 2 classes so for example:</p>
<pre><code class="lang-bash">LABEL VECTOR [array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([1., 0.]), array([1., 0.]), array([1., 0.]), array([1., 0.]), array([1., 0.]), array([0., 1.])]
</code></pre>
<p><strong>num_classes = 2</strong></p>
<pre><code class="lang-python">from torch import nn

#define Model
class LSTMClassifier(nn.Module):

    def __init__(self, input_size, lstm1_hidden_size,num_layers, num_classes):
        super(LSTMClassifier, self).__init__()

        #shape = (1,8192,16)
        self.lstm1 = nn.LSTM(input_size=input_size, hidden_size=lstm1_hidden_size, num_layers=num_layers, batch_first=True)
     
        self.classifier = nn.Linear(lstm1_hidden_size, num_classes)

        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        lstm_out, _ = self.lstm1(x) #hidden state &amp; cell state returned 

        pred = self.classifier(lstm_out)

        pred = self.sigmoid(pred)


        return pred

</code></pre>
<p>This model outputs a target shape of <strong>(1,num_segments,2)</strong></p>
<p>The shape of the data is:<br>
<strong>(1,num_segments,8192)</strong></p>
<p>The shape of the labels is:<br>
<strong>(1,num_segments,16,2)</strong></p>
<p>Again the labels look like the following:</p>
<ul>
<li>there are always fixed 16 labels, each having binary classification with 2 columns. Each column may be a 0 or 1. Thus 2 classes. So I want the output of the LSTM model to be a sequence of binary classifications.</li>
</ul>
<pre><code class="lang-bash">LABEL VECTOR [array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([0., 1.]), array([1., 0.]), array([1., 0.]), array([1., 0.]), array([1., 0.]), array([1., 0.]), array([0., 1.])]
</code></pre>
<p>Right now the error I am getting is:</p>
<p>ValueError: Target size (torch.Size([1, 7, 16, 2])) must be the same as input size (torch.Size([1, 7, 2]))</p>
<p>How can I structure this LSTM pytorch model to get an output as a vector of Binary Classification labels?</p>

thecho7: <p>The output of LSTM layer is <code>output, (h_n, c_n)</code> (see <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM" class="inline-onebox" rel="noopener nofollow ugc">LSTM — PyTorch 1.10.1 documentation</a>)</p>
<p>The hidden state of the last layer is usually utilized to do the binary classification.<br>
To do so,</p>
<pre><code class="lang-auto">#define Model
class LSTMClassifier(nn.Module):
    def __init__(self, input_size, lstm1_hidden_size,num_layers, num_classes):
        super(LSTMClassifier, self).__init__()
        #shape = (1,8192,16)
        self.lstm1 = nn.LSTM(input_size=input_size, hidden_size=lstm1_hidden_size, num_layers=num_layers, batch_first=True)
        self.classifier = nn.Linear(lstm1_hidden_size, num_classes)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        last_h, _ = self.lstm1(x)[1] #hidden state is returned
        # (tensor of shape (D * num_layers, N, H_out))
        last_h = last_h.permute(1, 0, 2).flatten(start_dim=1)
        pred = self.classifier(last_h)
        pred = self.sigmoid(pred)
        return pred
</code></pre>
<p>The above is an example code to help your understand.</p>