Correct way to build and get encodings from siamese using pretrained model

username: John_J_Watson
cooked: <p>I am trying to build a small siamese network (with an aim to get encodings from the last/pre-last layer) and would like to use a pretrained model + the extra layers needed to get the encodings.</p>
<p>I have something like this at the moment, but the results dont look great, so I now wonder if this is the correct way to build off a pretrained model.</p>
<pre><code class="lang-auto">class PretrainedSiamese(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.pt_model = torch.hub.load('pytorch/vision', 'some_pretrained_model', pretrained=True)
        for param in self.pt_model.parameters():
            param.requires_grad = False
        self.pt_model.classifier[1] = torch.nn.Linear(in_features=self.model_ft.classifier[1].in_features, out_features=2)
        self.linear1 = nn.Linear(in_features=2, out_features=256, bias=True)
        self.linear2 = nn.Linear(in_features=256, out_features=128, bias=True) # encoder layer
        self.linear3 = nn.Linear(in_features=128, out_features=2, bias=True)
</code></pre>
<p>The forwrad I have is simply:</p>
<pre><code class="lang-auto">            out = self.pt_model(out)
            out = x.view(out.shape[0], -1)
            out = self.linear1(out)
            out = F.relu(out)
            out = self.linear2(out)
..&lt; do siamese i.e loop twice&gt;
</code></pre>
<p>So, I pass the outputs (logits) from the pretrained classifier and attach a small network to this to get the encoding from the pre-final layer. Does this sound sensible? I am unsure now that I have terrible results <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>The code looks generally alright, assuming that <code>self.model_ft</code> is a typo and should be <code>self.pt_model</code> and that <code>.classifier[1]</code> is the last linear layer in the pretrained model.</p>
<p>If that’s the case then the number of output features for the first linear layer seem to be a bit small with only 2 neurons. You would create quite a strong bottleneck at this point and might lose a lot of information from the signal.</p>

username: John_J_Watson
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: thank you so much for your input. Yes, it was a typo, corrected it now.</p>
<p>I completely understand what you are saying about the bottleneck - but, I was wondering about this below question (and hence chose 2):<br>
[1] the siamese is <code>almost</code> a binary classifier with only two outputs (1/0 for same paid, different pair). The pretrained model is trained on a 1000 classes, but I have only 2.</p>
<p>Does this rational make any sense? In essense, the output from the pretrained model can at most have a 1000 classes (assuming 1000 classes is what the pt model is trained on?) If not, what should this number be? or is it another hyper paramter?</p>

username: ptrblck
cooked: <p>Thanks for the information.<br>
I didn’t realize that the 2 would represent the number of classes.</p>
<p>Usually you would use the penultimate layer from the pretrained model as the “feature” output and add a classifier on top of it. Currently <code>pt_model.classifier[1]</code> as well as <code>linear3</code> could be seen as a classification layer and the first bottleneck might be too aggressive.<br>
However, as always it all depends on your use case and your model might in fact work just fine. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: John_J_Watson
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for the suggestion. Infact, I did want to do what you are suggesting but couldnt find a way to do it: i.e how do I get the output of the penultimate layer and skip the classifier in the pretrained model? I can then attach my layers to this penultimate layer.</p>
<p>I would be really greatful if you could point me to any info on how to get the feature output from the penultimate layer.</p>

username: John_J_Watson
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: actually, it turned out to be quite simple. I just did:</p>
<pre><code class="lang-auto">
self. pt_model=self.pt_model.features
</code></pre>
<p>it seems to work!</p>

username: ptrblck
cooked: <p>This approach might work or alternatively you could also replace the last layer with <code>nn.Identity()</code> and would thus get the feature tensor as the output. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: John_J_Watson
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you again for this reply. Could you please point me to an example of how to use the Identity function to achieve this? Would like to try it. BTW: the last layer in say mobilenet is a classifier</p>
<pre><code class="lang-auto">&lt;.. features..&gt;
    (18): ConvBNReLU(
      (0): Conv2d(320, 1280, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (1): BatchNorm2d(1280, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU6(inplace=True)
    )
  )
  (classifier): Sequential(
    (0): Dropout(p=0.2, inplace=False)
    (1): Linear(in_features=1280, out_features=1000, bias=True)
  )
)
Sequential(
  (0): Dropout(p=0.2, inplace=False)
  (1): Linear(in_features=1280, out_features=1000, bias=True)
</code></pre>
<p>I presume I drop the <code>classifier</code> which is what I do with: <code>self. pt_model=self.pt_model.features</code>. Thus, the output is now from <code>(2): ReLU6(inplace=True)</code> - does this make sense? Would love to know how to do it via Identity method - just to be sure that it gives the same output.</p>
<p>Many thanks for all your help. REALLY appreciate it.</p>

username: ptrblck
cooked: <p>Your assignment might work, but I would always double check, if you really get the desired outputs.<br>
Since you are assigning the <code>features</code> block to <code>pt_model</code>, you assume that this block is executed sequentially in the original <code>forward</code> method, which might not always be the case.</p>
<p>To just reuse the original <code>forward</code> method, you could use:</p>
<pre><code class="lang-python">pt_model.classifier = nn.Identity()
</code></pre>
<p>to replace the <code>classifier</code> module with a simple identity method.</p>

username: John_J_Watson
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: when I try the Identity method you have suggested, my outputs are way smaller (1280 for mobilenet, which is the size of the classfier input) - but the output of the  <code>(2): ReLU6(inplace=True) </code> when I do <code>self. pt_model=self.pt_model.features</code> seems 20480. Bizzare!</p>
<p>May I ask what you mean by a feature vector? To me this <code>(2): ReLU6(inplace=True) </code> is a feature vector. No?</p>
<p>Also, for mobilenet, I see:</p>
<pre><code class="lang-auto">MobileNetV2(
  (features): Sequential(
...
</code></pre>
<p>so they are all sequential right?</p>
<p>In any case, I think the Identity method is better since it preserves the original instamce and I would prefer this if I can understand the size issue!</p>
<p>When I look at the mobilenet arch, towards the end, I seem to have:</p>
<pre><code class="lang-auto">    (18): ConvBNReLU(
      (0): Conv2d(320, 1280, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (1): BatchNorm2d(1280, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU6(inplace=True)
    )
  )
  (classifier): Sequential(
    (0): Dropout(p=0.2, inplace=False)
    (1): Linear(in_features=1280, out_features=1000, bias=True)
  )
)
Sequential(
  (0): Dropout(p=0.2, inplace=False)
  (1): Linear(in_features=1280, out_features=1000, bias=True)
)

</code></pre>
<p>By turning the classfier to Identity, my output at the end should be the size of <code>(2): ReLU6(inplace=True)</code> - is that not right?</p>
<p>Thanks again <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="John_J_Watson" data-post="10" data-topic="82984">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john_j_watson/48/16421_2.png" class="avatar"> John_J_Watson:</div>
<blockquote>
<p>when I try the Identity method you have suggested, my outputs are way smaller (1280 for mobilenet, which is the size of the classfier input) - but the output of the <code>(2): ReLU6(inplace=True) </code> when I do <code>self. pt_model=self.pt_model.features</code> seems 20480. Bizzare!</p>
</blockquote>
</aside>
<p>That’s exactly the difference between reusing a submodule (<code>features</code> in this case) and replacing just the classifier while keeping the <code>forward</code> method.<br>
As you can see in <a href="https://github.com/pytorch/vision/blob/650e3dbe2ca36f818b3fb121af774b01fa18d466/torchvision/models/mobilenet.py#L166">this line of code</a> a functional pooling is applied, which will reduce the spatial size of the activation and reshape it.</p>
<p>I cannot say which approach is the right one, so you might want to run some experiments with and without the pooling.</p>

username: John_J_Watson
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: Thank you fo this. Pardon my ignorance, but, when you say:</p>
<pre><code class="lang-auto">I cannot say which approach is the right one, so you might want to run some experiments with and without the pooling.
</code></pre>
<p>do you mean I modify the <code>mobilenet.py</code> file and run experiments with and without this line:</p>
<p><code>x = nn.functional.adaptive_avg_pool2d(x, 1).reshape(x.shape[0], -1)</code></p>
<p>Also, what am I looking for in the results? Do you mean to say, I should measure accuracy in the two cases (with and withut the line?) Sorry about the naive questions, have not done this kind of experimenting before. I suspect what you are saying is that the approach is problem dependent (with and without pooling)?</p>

username: ptrblck
cooked: <p>Your new use case seems to use the penultimate activation tensors as an extracted feature.<br>
While your approach would return the feature tensor before the pooling layer (which will thus be bigger), my proposed approach would apply the pooling and thus yield a smaller activation.</p>
<p>I don’t know, how your model (or overall use case) will use this tensor, so I cannot say that mu proposed approach is the “right one” for your use case.</p>
<aside class="quote no-group" data-username="John_J_Watson" data-post="12" data-topic="82984">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john_j_watson/48/16421_2.png" class="avatar"> John_J_Watson:</div>
<blockquote>
<p>Also, what am I looking for in the results? Do you mean to say, I should measure accuracy in the two cases (with and withut the line?)</p>
</blockquote>
</aside>
<p>Yes, I would suggest to use the validation loss (or accuracy) to compare both approaches.</p>

username: John_J_Watson
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: thank you VERY much for the detailed reply. I will run the experiments and report back. Thank you again - I would have never come this far withoutout your input!</p>

username: John_J_Watson
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: I am still doing experiments on these, but today, I saw this snippet with detectron2:<br>
<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>
<p>these lines:</p>
<pre><code class="lang-auto">
# load a pre-trained model for classification and return
# only the features
backbone = torchvision.models.mobilenet_v2(pretrained=True).features
# FasterRCNN needs to know the number of
# output channels in a backbone. For mobilenet_v2, it's 1280
# so we need to add it here
backbone.out_channels = 1280
</code></pre>
<p>means that instead of using Identity() as you suggested, they are using just the <code>.features</code>  but, I am perplexed how they are assigning the out_channels here as 1280, since the features as you have shown on this line:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/650e3dbe2ca36f818b3fb121af774b01fa18d466/torchvision/models/mobilenet.py#L166" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/650e3dbe2ca36f818b3fb121af774b01fa18d466/torchvision/models/mobilenet.py#L166" target="_blank" rel="nofollow noopener">pytorch/vision/blob/650e3dbe2ca36f818b3fb121af774b01fa18d466/torchvision/models/mobilenet.py#L166</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="156" style="counter-reset: li-counter 155 ;">
<li>                nn.init.zeros_(m.bias)</li>
<li>            elif isinstance(m, nn.Linear):</li>
<li>                nn.init.normal_(m.weight, 0, 0.01)</li>
<li>                nn.init.zeros_(m.bias)</li>
<li>
</li><li>    def _forward_impl(self, x):</li>
<li>        # This exists since TorchScript doesn't support inheritance, so the superclass method</li>
<li>        # (this one) needs to have a name other than `forward` that can be accessed in a subclass</li>
<li>        x = self.features(x)</li>
<li>        # Cannot use "squeeze" as batch-size can be 1 =&gt; must use reshape with x.shape[0]</li>
<li class="selected">        x = nn.functional.adaptive_avg_pool2d(x, 1).reshape(x.shape[0], -1)</li>
<li>        x = self.classifier(x)</li>
<li>        return x</li>
<li>
</li><li>    def forward(self, x):</li>
<li>        return self._forward_impl(x)</li>
<li>
</li><li>
</li><li>def mobilenet_v2(pretrained=False, progress=True, **kwargs):</li>
<li>    """</li>
<li>    Constructs a MobileNetV2 architecture from</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>…uses avg pooling to get to 1280. So, what is the detectron tutorial doing here? <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> Using just the features cannot result in a vector of size 1280. Would be nice to know what this tutorial is doing</p>

username: ptrblck
cooked: <p>This attribute is not directly used in <code>mobilenet_v2</code>, but in <code>FasterRCNN</code>.<br>
From the docs for <code>FasterRCNN</code>:</p>
<blockquote>
<p>backbone (nn.Module): the network used to compute the features for the model.<br>
It should contain a out_channels attribute, which indicates the number of output<br>
channels that each feature map has (and it should be the same for all feature maps).<br>
The backbone should return a single Tensor or and OrderedDict[Tensor].</p>
</blockquote>
<p>So you are just creating this attribute, so that it can be used in the <code>RPNHead</code>.</p>

username: John_J_Watson
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: thanks again for the explanation - makes sense <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: rzhang63
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I’m trying to build a DANN model from this <a href="http://sites.skoltech.ru/compvision/projects/grl/" rel="noopener nofollow ugc">paper</a>. The architecture is given in the figure below.</p>
<p>For the feature extractor (green part) I’d like to use a pretrained model, e.g. resnet50. I’m just wondering if my code is correct.</p>
<pre><code class="lang-auto">class ReverseLayerF(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, alpha):
        ctx.alpha = alpha
        return x.view_as(x)

    @staticmethod
    def backward(ctx, grad_output):
        output = grad_output.neg() * ctx.alpha
        return output, None


class DANN(torch.nn.Module):
    def __init__(self,architecture,pretrained=True):
        super(DANN, self).__init__()
        assert architecture in ['shuffle05', 'mobile', 'resnet50']
        if architecture == 'shuffle05':
            model = models.shufflenet_v2_x0_5(pretrained=pretrained)
        elif architecture == 'mobile':
            model = models.mobilenet_v2(pretrained=pretrained)
        else:
            model = models.resnet50(pretrained=pretrained)
        self.feature_extractor = torch.nn.Sequential(*list(model.children())[:-1])

        self.label_classifier = torch.nn.Sequential(
            torch.nn.Linear(2048,128)
            ,torch.nn.BatchNorm1d(128)
            ,torch.nn.Dropout2d()
            ,torch.nn.ReLU(True)
            ,torch.nn.Linear(128,64)
            ,torch.nn.BatchNorm1d(64)
            ,torch.nn.ReLU(True)
            ,torch.nn.Linear(64,1)
        )

        self.domain_classifier = torch.nn.Sequential(
            torch.nn.Linear(2048, 128)
            ,torch.nn.BatchNorm1d(128)
            ,torch.nn.ReLU(True)
            ,torch.nn.Linear(128,1)
        )

    def forward(self,x,alpha):
        feature = self.feature_extractor(x)
        feature = feature.view(feature.shape[0],-1)
        reverse_feature = ReverseLayerF.apply(feature, alpha)
        class_logit = self.label_classifier(feature)
        domain_logit = self.domain_classifier(reverse_feature)
        return class_logit,domain_logit

my_model = DANN('resnet50')
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88e79e1510e08145c66d4cdf4c889534d0ef8423.png" data-download-href="https://discuss.pytorch.org/uploads/default/88e79e1510e08145c66d4cdf4c889534d0ef8423" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88e79e1510e08145c66d4cdf4c889534d0ef8423_2_517x225.png" alt="image" data-base62-sha1="jx7aNE0MKz4YoCOa2Axeevkxm3V" width="517" height="225" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88e79e1510e08145c66d4cdf4c889534d0ef8423_2_517x225.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88e79e1510e08145c66d4cdf4c889534d0ef8423_2_775x337.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/8/88e79e1510e08145c66d4cdf4c889534d0ef8423.png 2x" data-dominant-color="EAEAE6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">980×428 123 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The code looks alright. Do you see any errors or issues during training?</p>
<p>You should be a bit careful with wrapping all child modules into an <code>nn.Sequential</code> container:</p>
<pre><code class="lang-python">self.feature_extractor = torch.nn.Sequential(*list(model.children())[:-1])
</code></pre>
<p>While this would work for simple models, it could break for models which are using functional API calls inside their <code>forward</code> method.<br>
If you want to remove the last layer, you could replace it with an <code>nn.Identity()</code> module instead.</p>

username: you_Tian
cooked: <p>Hi <a class="mention" href="/u/rzhang63">@rzhang63</a>, can you please tell me what the alpha meaning in class ReverseLayerF, doing something like parameter changing, but not quite sure alpha meaning here <a class="mention" href="/u/ptrblck">@ptrblck</a></p>