Combine Feature Extractor Model with main Prediction Model

Nidhi_Joshi: <p>Hey guys,<br>
A noob in pytorch here. My model planning for a task includes combining a feature extractor model which is a conv 1d model with multiple layers with a prediction model which is a stacked lstm layers. I am having a hard time understanding how to combine both these models while the initialization stages.<br>
One of the sample models I checked initialized the feature extraction model in the main prediction model itself and then picked the inputs for the prediction model from the feature extractor.<br>
Here is a noob attempt of how I am trying to do and probably not understanding if it is even right way to approach.</p>
<pre><code class="lang-auto">
class PredictionModel(nn.Module):
    def __init__(self, features, targets, nlayers,convlayers, hsize_pred,hsize_fe,dropout):
        super().__init__()
        FE = FeatureExtractor(features, convlayers, hsize_fe,dropout) # can I connect my FE model from # here in the main prediction model??
        new_features = FE.size()[2] # this is of course throwing an error but I just planned to pick new features like this. is it even right approach??
        layers = []
        for _ in range(nlayers):
            if len(layers) ==0:
                layers.append(nn.LSTM(new_features,hsize_pred))
                layers.append(nn.Dropout(dropout))
                layers.append(nn.Tanh())
                layers.append(nn.LSTM(hsize_pred,hsize_pred))
                layers.append(nn.Dropout(dropout))
                layers.append(nn.Tanh())
                layers.append(nn.LSTM(hsize_pred,hsize_pred))
                layers.append(nn.Dropout(dropout))
                layers.append(nn.Tanh())
            else:
                layers.append(nn.LSTM(hsize_pred,hsize_pred))
                layers.append(nn.Dropout(dropout))
                layers.append(nn.Tanh())
                layers.append(nn.LSTM(hsize_pred,hsize_pred))
                layers.append(nn.Dropout(dropout))
                layers.append(nn.Tanh())
                layers.append(nn.LSTM(hsize_pred,hsize_pred))
                layers.append(nn.Dropout(dropout))
                layers.append(nn.Tanh())
        layers.append(nn.Linear(445,targets))
        self.model = nn.Sequential(*layers)

    def forward(self,x):
        self.x = x
        return self.model(self.x)

class FeatureExtractor(nn.Module):
    def __init__(self, features, convlayers, hsize_fe,dropout):
        super().__init__()
        # self.features = features
        # self.targets = targets
        # self.layers = nlayers
        layers = []
        for _ in range(convlayers):
            if len(layers) == 0:
                layers.append(nn.Conv1d(features, hsize_fe, 3))
                layers.append(nn.BatchNorm1d(hsize_fe))
                layers.append(nn.Dropout(dropout))
                layers.append(nn.ReLU())
            else:
                layers.append(nn.Conv1d(hsize_fe, hsize_fe, 3))
                layers.append(nn.BatchNorm1d(hsize_fe))
                layers.append(nn.Dropout(dropout))
                layers.append(nn.ReLU())
        layers.append(nn.Linear(26,features))
        self.model = nn.Sequential(*layers)
        
    def forward(self,x):
        self.x = x
        return self.model(self.x)
</code></pre>
<p>Some questions are in the code comment section.<br>
How to proceed further in this quest.<br>
Any help is greatly helpful for me.</p>
<p>Thanks</p>

ptrblck: <pre><code class="lang-python">FE = FeatureExtractor(features, convlayers, hsize_fe,dropout) # can I connect my FE model from # here in the main prediction model??
</code></pre>
<p>The usual approach would be to use the feature extractor in the <code>forward</code> and pass its outputs to the classifier afterwards.</p>
<pre><code class="lang-python">new_features = FE.size()[2] # this is of course throwing an error but I just planned to pick new features like this. is it even right approach??
</code></pre>
<p>Your <code>LSTM</code> modules expect an input of <code>[seq_len, batch_size, features]</code> so you would have to check how these dimensions map to the output of your feature extractor and especially how the temporal dimension should be created.<br>
E.g. if your <code>FeatureExtractor</code> module returns an output of <code>[batch_size, out_channels, height, width]</code> you could try to use the spatial dimensions (height and width) as the temporal dimension and the <code>out_channels</code> as the features. Of course other approaches are possible and in the end it depends on your use case and your idea.</p>
<p>I would also assume that <code>self.model</code> will raise an error as you are wrapping <code>LSTM</code> modules in it which will return multiple outputs while <code>nn.Sequential</code> expects a single in- and output.<br>
There are ways to expand it, but I would probably just use an <code>nn.ModuleList</code> and call the layers in the <code>forward</code> separately in a loop.</p>

Nidhi_Joshi: <p>Yes, thanks you for the feature extraction approach. I planned to initiate both the models separately in the engine.py file and eventually supply inputs sequentially yo each model</p>
<p>And yes that list of layers did throw an error aa I missed out the two outputs given by lstm layers.<br>
Thank you again for second approach.</p>
<p>Will surely implement it from here.</p>
<p>Regards<br>
Nidhi</p>