Adding an LSTM to a trained CNN ResNet during fine-tuning

catb: <p>I have a pretrained CNN ResNet model to detect keypoints in an image, and after training, and I realize that adding an LSTM layer would’ve likely improved predictions (the images are timeseries images). I don’t have access to a GPU anymore to retrain the model on all the data, but I was hoping to use the weights from the CNN ResNet and somehow finetune the last layer using an LSTM. Is the best way to do that to have two separate models where the output of the CNN ResNet feeds into the LSTM? Or is there a way that I can somehow during the fine-tuning process add in the sequence information. My original model is below, but so far in the new model the predictions have been worse off than without the sequence information (I’ve been feeding the CNN into the LSTM and training both progressively), so I think it might be unnecessarily complicated and not working right.</p>
<p>Current model architecture (what is the easiest way to add LSTM information during fine-tuning?)</p>
<pre><code class="lang-auto">import torch.nn as nn
</code></pre>
<pre><code class="lang-auto">import torch.nn.functional as F
</code></pre>
<pre><code class="lang-auto">import pretrainedmodels  
</code></pre>
<pre><code class="lang-auto">
</code></pre>
<pre><code class="lang-auto">class snowPoleResNet50(nn.Module):
</code></pre>
<pre><code class="lang-auto">def __init__(self, pretrained, requires_grad):
</code></pre>
<pre><code class="lang-auto">#def __init__(self, pretrained, requires_grad, input_size, hidden_size, num_layers, num_classes):
</code></pre>
<pre><code class="lang-auto">super(snowPoleResNet50, self).__init__()
</code></pre>
<pre><code class="lang-auto">if pretrained == True:
</code></pre>
<pre><code class="lang-auto">self.model = pretrainedmodels.__dict__['resnet50'](pretrained='imagenet')
</code></pre>
<pre><code class="lang-auto">else:
</code></pre>
<pre><code class="lang-auto">self.model = pretrainedmodels.__dict__['resnet50'](pretrained=None)
</code></pre>
<pre><code class="lang-auto">if requires_grad == True:
</code></pre>
<pre><code class="lang-auto">for param in self.model.parameters():
</code></pre>
<pre><code class="lang-auto">param.requires_grad = True
</code></pre>
<pre><code class="lang-auto">print('Training intermediate layer parameters...')
</code></pre>
<pre><code class="lang-auto">elif requires_grad == False:
</code></pre>
<pre><code class="lang-auto">for param in self.model.parameters():
</code></pre>
<pre><code class="lang-auto">param.requires_grad = False
</code></pre>
<pre><code class="lang-auto">print('Freezing intermediate layer parameters...')
</code></pre>
<pre><code class="lang-auto"># change the final layer
</code></pre>
<pre><code class="lang-auto">self.l0 = nn.Linear(2048, 4)  #### the second value is the number of points you want to predict
</code></pre>
<pre><code class="lang-auto">
</code></pre>
<pre><code class="lang-auto">def forward(self, x):
</code></pre>
<pre><code class="lang-auto"># get the batch size only, ignore (c, h, w)
</code></pre>
<pre><code class="lang-auto">batch, _, _, _ = x.shape
</code></pre>
<pre><code class="lang-auto">x = self.model.features(x)
</code></pre>
<pre><code class="lang-auto">x = F.adaptive_avg_pool2d(x, 1).reshape(batch, -1)
</code></pre>
<pre><code class="lang-auto">l0 = self.l0(x)
</code></pre>
<pre><code class="lang-auto">return l0
</code></pre>
<pre><code class="lang-auto">
</code></pre>