Backward() and train a part of a model

amirhf: <p>Hi,</p>
<p>I’m trying to train some part of a model like below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0b775416d94b28a23a029f999c6d4b088729bbbd.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0b775416d94b28a23a029f999c6d4b088729bbbd" title="IMG_5BC31395F3B4-1"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0b775416d94b28a23a029f999c6d4b088729bbbd_2_690x450.jpeg" alt="IMG_5BC31395F3B4-1" data-base62-sha1="1DqUyGRRsR5D9X4DoT1bZul32ah" width="690" height="450" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0b775416d94b28a23a029f999c6d4b088729bbbd_2_690x450.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0b775416d94b28a23a029f999c6d4b088729bbbd_2_1035x675.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/0b775416d94b28a23a029f999c6d4b088729bbbd.jpeg 2x" data-dominant-color="F3F2ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_5BC31395F3B4-1</span><span class="informations">1349×880 733 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>input is high dimensional and I’m using a simple NN on a small partition of the data (<code>f0</code>) to reduce dimension and then concatenate with the rest of the data (<code>f1</code>) on which nothing has been applied. then I pass them to a frozen model <code>M</code> (pretrained Keras model) and then get the binary output and then I calculate loss. I just want to train the NN model and do backward on its parameters. Is this possible?</p>

ptrblck: <p>The workflow would be possible in PyTorch.<br>
However, if seems you would like to mix PyTorch with a pretrained Keras model afterwards, which won’t work (at least I’m not aware of such a workflow and if someone actually has used it before).</p>
<p>Would it be possible to port the Keras parameters to a PyTorch model?</p>

amirhf: <p>Thank you for the response.</p>
<p>So there are some layers in my Keras model that are custom defined and the existing libraries don’t support custom layers when converting. But, how would I do that if M was a pretrained PyTorch model?</p>
<p>Also, is it because the output of the Keras model is a numpy array and doesn’t keep the grad_fn and requires_grad that this is not possible?</p>

ptrblck: <p>A dummy approach would be:</p>
<pre data-code-wrap="python"><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc = nn.Linear(4, 2)
        self.base = models.resnet50()
        
    def forward(self, x):
        # split x to x0, x1
        x0, x1 = x[:, :4], x[:, 4:]
        x0 = self.fc(x0)
        # Concatenate
        x = torch.cat((x0, x1), 1)
        # reshape to fit resnet50 input shape
        x = x.view(x.size(0), 3, 224, 224)
        x = self.base(x)
        return x

model = MyModel()
x = torch.randn(1, 3*224*224 + 2)
output = model(x)
</code></pre>
<p>Note that I’ve used resnet50, which expects image tensors as the input, thus I had to reshape the concatenated tensor to <code>[batch_size, 3, 224, 224]</code>.<br>
If you are dealing with a pretrained model using only linear layers, you wouldn’t have to reshape it.</p>
<aside class="quote no-group" data-username="amirhf" data-post="3" data-topic="51973">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amirhf/48/5634_2.png" class="avatar"> amirhf:</div>
<blockquote>
<p>Also, is it because the output of the Keras model is a numpy array and doesn’t keep the grad_fn and requires_grad that this is not possible?</p>
</blockquote>
</aside>
<p>Yes, basically you would have to convert the concatenated PyTorch tensor to a numpy array, which will detach the computation graph, so that you won’t be able to calculate the gradients using the final loss.</p>

amirhf: <p>Thank you so much. This clarified a lot of things and helps me a lot!</p>

ptrblck: <p>You’re welcome.<br>
Let us know, if you need help porting the code to train the model end-to-end <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

amirhf: <p>Do you mean porting the Keras model? Do you think it’s possible to port a trained Keras model (with custom layers) to PyTorch?</p>

ptrblck: <p>It depends, what kind of custom layers there are, but as long as there are equivalent PyTorch methods, it should work.<br>
There might be some pitfalls, e.g. flipped kernels etc., but users in this forum ported successfully models before.</p>

amirhf: <p>Sounds like a good project. I’ll research a bit more (this forum and web) to see what I can do. Thanks for the help again. I’ll reach out if I needed help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>