Beginner questions about using PyTorch's AlexNet model from torch.hub

seberino: <p>Here is sample code to use PyTorch’s built in AlexNet model for CIFAR-10 dataset classifier (10 classes):</p>
<pre><code class="lang-auto">model               = torch.hub.load("pytorch/vision", 
                                     "alexnet", 
                                     weights = "DEFAULT")
model.classifier[1] = torch.nn.Linear(9216,4096)
model.classifier[4] = torch.nn.Linear(4096,1024)
model.classifier[6] = torch.nn.Linear(1024,10)
</code></pre>
<p>Why were these three Linear layers added?  My understanding is that AlexNet<br>
ended with two 4096 size linear layers and a 1000 size linear layer.<br>
(See <a href="https://www.mdpi.com/remotesensing/remotesensing-09-00848/article_deploy/html/images/remotesensing-09-00848-g001.png" rel="noopener nofollow ugc">https://www.mdpi.com/remotesensing/remotesensing-09-00848/article_deploy/html/images/remotesensing-09-00848-g001.png</a>)</p>
<p>Also, are these Linear layers <em>appended</em> to AlexNet or do they <em>modify</em> existing AlexNet<br>
layers?</p>
<p>Thanks,</p>
<p>Chris</p>

ptrblck: <aside class="quote no-group" data-username="seberino" data-post="1" data-topic="188410">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/df788c/48.png" class="avatar"> seberino:</div>
<blockquote>
<p>Why were these three Linear layers added?</p>
</blockquote>
</aside>
<p>I guess the author wanted to reinitialize the first linear layer, as it’s using the same config, and create a smaller feature space with only 10 outputs for the CIFAR10 dataset, but you might want to ask the author of this code about their intention.</p>
<aside class="quote no-group" data-username="seberino" data-post="1" data-topic="188410">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/df788c/48.png" class="avatar"> seberino:</div>
<blockquote>
<p>Also, are these Linear layers <em>appended</em> to AlexNet or do they <em>modify</em> existing AlexNet<br>
layers?</p>
</blockquote>
</aside>
<p>The layers are modified and the original <code>.classifier</code> module is defined as:</p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.hub.load("pytorch/vision", 
                       "alexnet", 
                       weights="DEFAULT")

print(model.classifier)
# Sequential(
#   (0): Dropout(p=0.5, inplace=False)
#   (1): Linear(in_features=9216, out_features=4096, bias=True)
#   (2): ReLU(inplace=True)
#   (3): Dropout(p=0.5, inplace=False)
#   (4): Linear(in_features=4096, out_features=4096, bias=True)
#   (5): ReLU(inplace=True)
#   (6): Linear(in_features=4096, out_features=1000, bias=True)
# )
</code></pre>

seberino: <p>ptrblck</p>
<p>Thanks!  It someone wanted to be less ambitious, and just <em>append</em> one new layer<br>
to knock down the number of classes, would they replace those 3 Linear layers lines<br>
with just this one modification?..</p>
<p><code>model.classifier[7] = torch.nn.Linear(1000,10)</code></p>
<p>From your <code>print(model.classifier)</code> output, it looks like this would be a clean way to just<br>
tweak the number of AlexNet outputs without messing with AlexNet’s architecture no?</p>
<p>Sincerely,</p>
<p>Chris</p>

ptrblck: <p>No, this won’t work as you will get an indexing error.<br>
You could <code>append</code> layers to <code>model.classifier</code> instead.<br>
However, note that appending a new classification layer after the original one is not a common approach as you would rather replace the last classification layer with your own one.<br>
The reason to do so is to be able to use the features to train the model for your use case instead of using the logits for 1000 classes to train your 10 classes.</p>

seberino: <p>Thanks a lot.  That makes sense.</p>
<p>cs</p>