Combining CNN - LSTM - Research paper implementation

sravuri: <p>Hello,</p>
<p>I am trying to implement the methodology proposed in this paper <a href="https://arxiv.org/pdf/1808.05819.pdf" rel="noopener nofollow ugc">here</a> as the authors have not released the code yet.</p>
<p>The network architecture is a combination of a BaseCNN and a LSTM layer. The image below depicts the network architecture used by the authors taken from the paper mentioned above.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d5f86699fd588139c0aaafc82895d49e82ffd8be.png" data-download-href="https://discuss.pytorch.org/uploads/default/d5f86699fd588139c0aaafc82895d49e82ffd8be" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5f86699fd588139c0aaafc82895d49e82ffd8be_2_690x169.png" alt="image" data-base62-sha1="uwRTDKxKQCPg5doHdQq3S2gBKmq" width="690" height="169" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5f86699fd588139c0aaafc82895d49e82ffd8be_2_690x169.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5f86699fd588139c0aaafc82895d49e82ffd8be_2_1035x253.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5f86699fd588139c0aaafc82895d49e82ffd8be_2_1380x338.png 2x" data-dominant-color="E8E8D8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1600×392 73.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My questions are as follows:</p>
<ol>
<li>
<p>Considering my input raster size to be [B,3,350,350] with batch_size ‘B’ = 8 and my BaseCNN as mobilenet_v2, the output feature dimension is condensed to [B, 4096] feature vector after the dense layer as per the architecture shown in the above image (Fig 2)</p>
</li>
<li>
<p>The output of first FC layer is further reduced to [B,128] vector and then given as an input to a LSTM (Fig 3). The initial cell state is initialized to zero. <strong>Here is where I am confused on how can I give a 1D vector as an input to LSTM which expects input as (seq_len, batch, input_size) and  hidden state  of shape (num_layers * num_directions, batch, hidden_size)</strong></p>
</li>
</ol>
<p>PS: Section 3.3 in the paper explains the network architecture in detail.</p>
<p>How can I implement this ?</p>
<p>Thanks</p>

suraj.pt: <p>From section 3.2</p>
<blockquote>
<p>we train the model to output 2H-D vector, representing predicted x- and y-positions for each of H trajectory point</p>
</blockquote>
<p>it seems that the model output is <code>[H, 2H/3H]</code> where H is the no. of trajectory points in the time horizon. So I’d expect the LSTM to receive <code>[H, B, 128]</code> input tensor.</p>

sravuri: <p>Thank you for the reply!</p>
<p>Yes, the LSTM would expect input as [H,B,128]. But the output of the FC layer as can been seen in Fig 3 is of the shape[B,128]. How do I reshape it before feeding it to the LSTM layer ?</p>

vdw: <p>As far as I can tell – I only had a very quick look at the paper – the LSTM is used as a decoder similar to Seq2Seq models.</p>
<p>The figure indicates that the output of FC is used as the first initial state which has a shape of <code>(num_layers * num_directions, batch, hidden_size)</code>. Given that it’s used as a decoder <code>num_directions=1</code> and usually <code>num_layers=1</code> as well. So the expected shape is <code>(1,B,128)</code> So you can simple do <code>unsqueeze(0)</code> on the <code>(B,128)</code> tensor.</p>
<p>However, this clarifies only <code>h_0</code> and <code>c_0</code>. To get the decoder going also requires some initial input. For decoding texts, that’s usually a special Start-of-Sequence <code>&lt;SOS&gt;</code> token.</p>
<p>Alternatively, the papers state “[…] initial input is obtained by converting output of the FC layer of size 4,096 into a vector of size 128 with another FC layer”. If this really is the input but not the initial hidden state, then they are sloppy with the terminology.</p>
<p>If all fails, just send an email to the authors to clarify. Usually works. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

sravuri: <p>Thank you for the explanation, clarified most of my confusion. I’ll email the authors for more information.</p>
<p>Thanks again <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>