Converting Keras VGG model from paper to Pytorch

martinodonnell: <p>I am trying to follow the guide from a scientific paper but can not work out how to convert the model from keras to VGG16<br>
The paper is found at <a href="http://cs230.stanford.edu/projects_spring_2019/reports/18681590.pdf" rel="nofollow noopener">http://cs230.stanford.edu/projects_spring_2019/reports/18681590.pdf</a>. The model is describe at the end of page 2 and start of 3.</p>
<p>I am taking the pretrainined VGG16 as the parameter and creating an output with 196 classes</p>
<pre><code class="lang-auto">class NetworkV1_4(nn.Module):#http://cs230.stanford.edu/projects_spring_2019/reports/18681590.pdf
    def __init__(self, base, num_classes):
        super().__init__()

        self.base = base

        self.base.classifier = nn.Sequential(
            nn.Linear(in_features=25088, out_features=25088,bias=True),
            nn.ReLU(inplace=True),
            nn.Linear(in_features=25088, out_features=512,bias=True),
            nn.ReLU(inplace=True),

            nn.Dropout(0.5),
            nn.Linear(512, num_classes),
        )

    def forward(self, x):
        fc = self.base(x)
        return fc
</code></pre>

mailcorahul: <p>Hi,<br>
May I know the problem you’re facing? Can you be more specific?</p>

martinodonnell: <p>HI, I am trying to replicate the result seen in the paper. They are doing fine-grain vehicle recognition. They find that they need to add modifications to the architecture of VGG16 to get better result and remove overfitting. There is a link to a github which follows the architecture but I don’t know who to convert it to pytorch from keras (<a href="https://github.com/Xiaotian-WANG/Fine-Tune-VGG-Networks-Based-on-Stanford-Cars/blob/master/fine_tune_model.py" rel="nofollow noopener">https://github.com/Xiaotian-WANG/Fine-Tune-VGG-Networks-Based-on-Stanford-Cars/blob/master/fine_tune_model.py</a>)</p>
<p>Thanks</p>

martinodonnell: <p>I found this github repo which is what I am looking for. Line 55-60(<a href="https://github.com/nnbenavides/Fine-Grained-Vehicle-Classification/blob/master/train_vgg_model.py" class="inline-onebox" rel="noopener nofollow ugc">Fine-Grained-Vehicle-Classification/train_vgg_model.py at master · nnbenavides/Fine-Grained-Vehicle-Classification · GitHub</a>) but I don’t know if I am doing it right</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8c7fc4dbd4af4083565ab1b19c17771654b0c84d.png" data-download-href="https://discuss.pytorch.org/uploads/default/8c7fc4dbd4af4083565ab1b19c17771654b0c84d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c7fc4dbd4af4083565ab1b19c17771654b0c84d_2_690x347.png" alt="image" data-base62-sha1="k2UAyzoBl024BxObC5bfggJ3kQJ" width="690" height="347" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c7fc4dbd4af4083565ab1b19c17771654b0c84d_2_690x347.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c7fc4dbd4af4083565ab1b19c17771654b0c84d_2_1035x520.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c7fc4dbd4af4083565ab1b19c17771654b0c84d_2_1380x694.png 2x" data-dominant-color="2A2A2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2650×1336 407 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The left side is their model and the right is mine.</p>
<p>I am using the code seen below</p>
<pre><code class="lang-auto">class NetworkV1_5(nn.Module):#http://cs230.stanford.edu/projects_spring_2019/reports/18681590.pdf Take 2
    def __init__(self, base, num_classes):#Define the layers
        super().__init__()

        self.base = base
        self.base.classifier = nn.Sequential(
            nn.Linear(in_features=25088, out_features=512, bias=True),
            nn.ReLU(inplace=True),
            nn.Dropout(0.7),
            nn.Linear(512, num_classes),
        )

    def forward(self, x):
        fc = self.base(x)
        return fc
</code></pre>
<p>The paper I quoted above says that the first fully connected layer is dropped from VGG16. Then the second layers dimensions are decreased to 512. A dropout later is then added and then the last layer with 196 outputs(Number of fine-grain annotations int he dataset)</p>
<p>Is this enough?</p>

mailcorahul: <p>Why are you using <code>nn.AdaptiveAvgPool2d</code> to get to spatial dims of <code>7x7</code>?<br>
I see the Keras model using maxpooling to bring 14x14 down to 7x7.</p>
<p>Apart from that, the rest seems fine. It’s better to start training your model and see how well it is performing on the benchmarking dataset. On a side note, it is still better to have a network architecture that deviates from the one mentioned in the paper. You may get better results, you never know <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

martinodonnell: <aside class="quote no-group" data-username="mailcorahul" data-post="5" data-topic="68409">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mailcorahul/48/15919_2.png" class="avatar"> mailcorahul:</div>
<blockquote>
<p>Keras model using maxpooling to bring 14x14 down to 7x7.</p>
<p>Apart from that, the rest seems fine. It’s better to start training your</p>
</blockquote>
</aside>
<p>Will give it a go. I do pull in the same model but in pytorch so it may be a little different. Thanks</p>