CNN LSTM implementation for video classification

IliasPap: <p>I have implemented a Cnn connected with an lstm to classify multi label videos with CTC Loss<br>
I have two implementations as followed and I don’t know which is better for the forward/bakward operations and if there is any impact in training the network.</p>
<pre><code class="lang-auto">class TimeDistributed_Subunet(nn.Module):
    def __init__(self,hidden_size,n_layers,dropt,bidirectional,N_classes ):
        super(TimeDistributed_Subunet, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = n_layers
        dim_feats = 4096
        self.cnn = models.alexnet(pretrained=True)
        self.cnn.classifier[-1] = Identity()
        self.rnn = nn.LSTM(
            input_size=dim_feats,
            hidden_size=self.hidden_size,
            num_layers=self.num_layers,
            dropout=dropt,
            bidirectional=bidirectional)
        self.n_cl = N_classes
        if (bidirectional):
            self.last_linear = nn.Linear(2 * self.hidden_size, self.n_cl)
        else:
            self.last_linear = nn.Linear(self.hidden_size, self.n_cl)

    def forward(self, x):

        batch_size, time_steps, C, H, W = x.size()
        output = torch.tensor([])
        output = output.to(x.get_device())
        for i in range(time_steps):

            cnn_out = self.cnn(x[:, i, :, :, :]).unsqueeze(0)

            rnn_out,(hidden,cell_state) = self.rnn(cnn_out)

            logits = self.last_linear(rnn_out)
            output = torch.cat((output,logits),0)

        return output







</code></pre>
<p>second implemenatation</p>
<pre><code class="lang-auto">class Identity(nn.Module):
    def __init__(self):
        super(Identity, self).__init__()
        
    def forward(self, x):
        return x








class SubUnet_orig(nn.Module):
    def __init__(self,hidden_size,n_layers,dropt,bi,N_classes):
        super(SubUnet_orig, self).__init__()

        self.hidden_size=hidden_size
        self.num_layers=n_layers

        dim_feats = 4096
  
        self.cnn=models.alexnet(pretrained=True)
        self.cnn.classifier[-1]=Identity()
        self.rnn = nn.LSTM(
            input_size=dim_feats,
            hidden_size=self.hidden_size,
            num_layers=self.num_layers,
            dropout=dropt,
            bidirectional=True)
        self.n_cl=N_classes
        if(True):
            self.last_linear = nn.Linear(2*self.hidden_size,self.n_cl)
        else:
            self.last_linear = nn.Linear(self.hidden_size,self.n_cl)
            

    def forward(self, x):

        batch_size, timesteps, C,H, W = x.size()
        c_in = x.view(batch_size * timesteps, C, H, W)
        
        c_out = self.cnn(c_in)

        r_out, (h_n, h_c) = self.rnn(c_out.view(-1,batch_size,4096))  

        r_out2 = self.last_linear(r_out)

        return r_out2

</code></pre>
<p>The main difference is at the rnn where in the first implementation the cnn output for each timestep is passed inside a for loop to the rnn and in the other case the outputs for all timesteps  are passed directly to the rnn</p>

Omar2017: <p>also, I need to know which one is correct? anyone know?</p>

IliasPap: <p>Up to now the second implementation works fine</p>

akashs: <p>I am also using the second approach. However, I have a confusion: as batch samples and timesteps are squashed, won’t it have any problem in rnn sequential learning? i.e when the sequence is reshaped again to (timesteps, samples/batch, output_size), will it retain the sequential (timesteps) features ordering for each sample/batch as it was before squashing?</p>

IliasPap: <p>view operation on tensors is supposed to keep the order of the features, I didn’t have any problem</p>

barloccia: <p>Can you please share your code about this implementation? I am facing similar structure but I have some doubts about the accuracy of the model! Thanks!</p>

IliasPap: <pre><code>def forward(self, x):

    batch_size, timesteps, C,H, W = x.size()
    c_in = x.view(batch_size * timesteps, C, H, W)
    
    c_out = self.cnn(c_in)

    r_out, (h_n, h_c) = self.rnn(c_out.view(-1,batch_size,c_out.shape[-1]))  

    logits = self.classifier(r_out)

    return logits</code></pre>

barloccia: <p>Do you have any idea how to visualize using an heatmap the activations that made the classification? A kind of grad cam method applied to this kind of network (CNN+LSTM). I would like to visualize the features at the final time step (or even at each time step) that have been activated during classification. Any suggestions?</p>

Julio: <p>Hello Ilias, I’m interested in RNN at the moment, and this example looks great. Do you have the complete code (with the training or testing parts too) uploaded somewhere and could you share it?</p>
<p>Thanks a lot</p>

rjrohit996: <p>I have multiple videos, each video have 30 frames. Need to extract features from a given video(30 frames) and feed it to LSTM. how do i extract single feature representation given a set of 30 images<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/6/16ec2a5ae4f743fd21774bea693acc9cce6c9690.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/16ec2a5ae4f743fd21774bea693acc9cce6c9690" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/16ec2a5ae4f743fd21774bea693acc9cce6c9690_2_690x320.jpeg" alt="image" data-base62-sha1="3gMua4BmfkEJ7BZ0KvEzarwmCRO" width="690" height="320" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/16ec2a5ae4f743fd21774bea693acc9cce6c9690_2_690x320.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/16ec2a5ae4f743fd21774bea693acc9cce6c9690_2_1035x480.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/6/16ec2a5ae4f743fd21774bea693acc9cce6c9690.jpeg 2x" data-dominant-color="DED7CE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1257×584 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Bipin_Jaiswal: <p>have you  solved it , if yes please share the code</p>