How to get output results from decoder RNN

lanka: <p>hi,<br>
i have trained encoder-decoder model for image captioning and need to test the results for input test data,suppose test data is <strong><code>features</code></strong> with  <strong><code>512</code></strong> size torch vector and <strong><code>hidden_size</code></strong> are also the same.</p>
<pre><code class="lang-auto">class DecoderRNN(nn.Module):
    def __init__(self, embed_size, hidden_size, vocab_size, num_layers, max_seq_length):
        """Set the hyper-parameters and build the layers."""
        super(DecoderRNN, self).__init__()
        self.embed = nn.Embedding(vocab_size, embed_size)
        self.lstm = nn.LSTM(embed_size, hidden_size, num_layers, batch_first=True)
        self.linear = nn.Linear(hidden_size, vocab_size)
        self.max_seg_length = max_seq_length
        
    def forward(self, features, captions, lengths):
        """Decode image feature vectors and generates captions."""
        embeddings = self.embed(captions)
        embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)
        packed = pack_padded_sequence(embeddings, lengths, batch_first=True) 
        hiddens, _ = self.lstm(packed)
        outputs = self.linear(hiddens[0])
        return outputs
    
    def sample(self, features, states=None):
        """Generate captions for given image features using greedy search."""
        sampled_ids = []
        inputs = features.unsqueeze(1)
        for i in range(self.max_seg_length):
            hiddens, states = self.lstm(inputs, states)          # hiddens: (batch_size, 1, hidden_size)
            outputs = self.linear(hiddens.squeeze(1))            # outputs:  (batch_size, vocab_size)
            _, predicted = outputs.max(1)                        # predicted: (batch_size)
            sampled_ids.append(predicted)
            inputs = self.embed(predicted)                       # inputs: (batch_size, embed_size)
            inputs = inputs.unsqueeze(1)                         # inputs: (batch_size, 1, embed_size)
        sampled_ids = torch.stack(sampled_ids, 1)                # sampled_ids: (batch_size, max_seq_length)
        return sampled_ids
</code></pre>
<p>so here is my input feature for the decoder, please let me know any suggestions for my approach. because I’m getting almost the same ouput sentence for every input;</p>
<pre><code class="lang-auto">decoder = DecoderRNN(embed_size, hidden_size, len(voc),num_layers,max_seq_length=20).to(device)
decoder.load_state_dict(torch.load(decoder_path))
img=Variable(image)
img_tensor =img.view(1,512)
img_tensor = img_tensor.to(device)
decoder.eval()
sampled_ids = decoder.sample(img_tensor)
sampled_ids = sampled_ids[0].cpu().numpy()
</code></pre>
<p>is this the correct way to generate LSTM output from decoderRNN?</p>

vdw: <p>Without the code of the encoder and just by looking a the code, it’s difficult to make a goo comment.</p>
<p>However, the line</p>
<pre><code class="lang-auto">outputs = self.linear(hiddens[0])
</code></pre>
<p>sure seems off. With <code>batch_first=True</code> the shape of <code>hiddens</code> is <code>(batch_size, seq_len, hidden_size)</code> (since you don’t use a bidirectional LSTM). So <code>hidden[0]</code> just considers to first sequence in your batch. I don’t think that’s what you want to do, and it’s a bit telling that your network is not throwing an error. So there might be other issues :).</p>

lanka: <p>suppose, the encoder just transfer learned features like VGG, so the encoder output 4096 and reduces to 512 (see <a href="https://discuss.pytorch.org/t/how-to-concatenate-cnn-features-and-reduce-size/172089" class="inline-onebox">How to concatenate CNN features and reduce size</a>) .then nothing new in encoder side.</p>