PyTorch Transfer learning with Densenet

username: boomcan90
cooked: <p>So, I’ve been trying to modify <a href="http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a> to work with DenseNet instead of ResNet, but I can’t seem to figure out what to change the fc layer to.</p>
<p>This is what I currently have:</p>
<pre><code class="lang-auto">model_ft = models.densenet169(pretrained=True)
num_features = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_features, len(CLASS_NAMES))
</code></pre>
<p>but running the code gives me this error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "pytorch_densenet.py", line 154, in &lt;module&gt;
    num_features = model_ft.fc.in_features
  File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 398, in __getattr__
    type(self).__name__, name))
AttributeError: 'DenseNet' object has no attribute 'fc'
</code></pre>
<p>So, uh… How do I change the last layer of DenseNet to work with Transfer Learning?</p>
<p>Thanks a lot.</p>

username: austin
cooked: <p>Looking at the model <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/densenet.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/densenet.py</a> It looks like they assigned all of the conv layers to <code>self.features</code> and all of the and all of the classification layers to <code>self.classifier</code>.</p>
<p>In your case, <code>model.features</code> should give you the feature extractor that you want but not all of the torchvision models follow the same api. In the future you will just have to look at the model in that repo and figure out where the last conv layer is.</p>

username: Mona_Jalal
cooked: <p>Did you figure how to use densenet instead of resnet? I cannot find an example that does so.</p>
<p>I get the following error:</p>
<pre><code class="lang-auto">/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torchvision-0.2.1-py3.6.egg/torchvision/models/densenet.py:212: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.
Downloading: "https://download.pytorch.org/models/densenet161-8d451a50.pth" to /home/grad3/jalal/.torch/models/densenet161-8d451a50.pth
100%|██████████| 115730790/115730790 [00:04&lt;00:00, 24886091.87it/s]

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
&lt;ipython-input-290-2e17f45b78dc&gt; in &lt;module&gt;()
     12 
     13 
---&gt; 14 num_ftrs = model_ft.fc.in_features
     15 model_ft.fc = nn.Linear(num_ftrs, 9)
     16 

/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    516                 return modules[name]
    517         raise AttributeError("'{}' object has no attribute '{}'".format(
--&gt; 518             type(self).__name__, name))
    519 
    520     def __setattr__(self, name, value):

AttributeError: 'DenseNet' object has no attribute 'fc'
</code></pre>
<p>for the following code:</p>
<pre><code class="lang-auto">######################################################################
# Finetuning the convnet
# ----------------------
#
# Load a pretrained model and reset final fully connected layer.
#

class_weights = torch.FloatTensor(weight).cuda()
#model_ft = models.resnet18(pretrained=True)
###model_ft = models.resnet50(pretrained=True)
model_ft = models.densenet161(pretrained=True)


num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 9)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss(weight=class_weights)

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)
###optim.Adam(amsgrad=True)
# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)
</code></pre>

username: Mona_Jalal
cooked: <p>actually I replaced these two lines and seems the training is happening but I am not sure if replacing resnet50 with densenet161 would only need these two line replacements:</p>
<blockquote>
<p>##<span class="hashtag-raw">#num_ftrs</span> = model_ft.fc.in_features<br>
num_ftrs = model_ft.classifier.in_features<br>
##<span class="hashtag-raw">#model_ft</span>.fc = nn.Linear(num_ftrs, 9)<br>
model_ft.classifier = nn.Linear(num_ftrs, 9)</p>
</blockquote>

username: rusty
cooked: <p>Resnet uses the name <code>fc</code> for its last layer while Densenet uses the name <code>classifier</code> for its last layer. You may see these naming and indexing by printing out the model</p>
<pre><code class="lang-auto">model = models.densenet161(pretrained=True)
print(model)
</code></pre>

username: fgtoralesch
cooked: <p>This is well documented in pytorch tutorials. Here an extract with your solution (agreeing with what <a class="mention" href="/u/rusty">@rusty</a> said above):</p>
<p><code>Preformatted text`To reshape the network, we reinitialize the classifier’s linear layer as model.classifier = nn.Linear(1024, num_classes)</code></p>
<p>Source: <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html</a></p>

username: Sadaf
cooked: <p>To just add some extra information, if you want to use each block of DenseNet in your network, you can use the following:<br>
pretrained_model = densenet121()<br>
self.features = pretrained_model._modules[‘features’]<br>
self.block = self.features._modules[‘denseblock1’]</p>

username: Imre_Kovacs
cooked: <p>You can add a customized classifier as follows:</p>
<ol>
<li>Check the architecture of your model, in this case it is a Densenet-161. Printing it yields and displaying here the last layers:<br>
…<br>
)<br>
(denselayer24): _DenseLayer(<br>
(norm1): BatchNorm2d(2160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu1): ReLU(inplace=True)<br>
(conv1): Conv2d(2160, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)<br>
(norm2): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu2): ReLU(inplace=True)<br>
(conv2): Conv2d(192, 48, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
)<br>
)<br>
(norm5): BatchNorm2d(2208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
(classifier): Linear(in_features=2208, out_features=1000, bias=True)<br>
)</li>
</ol>
<p>So you can see that “norm5” has an output size of 2208, that is the output size of the Densenet-161 without classifier.</p>
<ol start="2">
<li>You can verifiy the number features (equals output size) by</li>
</ol>
<pre><code class="lang-auto">num_ftrs = model_transfer.classifier.in_features
num_ftrs
</code></pre>
<p>This prints again 2208.</p>
<ol start="3">
<li>Now you can  add your classifier to the network:</li>
</ol>
<pre><code class="lang-auto">model_transfer.classifier = nn.Sequential(
                        nn.Linear(num_ftrs, 256),  
                        nn.ReLU(), 
                        nn.Dropout(0.4),
                        nn.Linear(256, n_classes),                   
                        nn.Softmax(dim=1))
</code></pre>
<ol start="4">
<li>Verify by printing model with</li>
</ol>
<pre><code class="lang-auto">model_transfer
</code></pre>
<p>yields:<br>
…<br>
(norm5): BatchNorm2d(2208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
(classifier): Sequential(<br>
(0): Linear(in_features=2208, out_features=256, bias=True)<br>
(1): ReLU()<br>
(2): Dropout(p=0.4, inplace=False)<br>
(3): Linear(in_features=256, out_features=133, bias=True)<br>
(4): LogSoftmax()<br>
)<br>
)</p>
<p>Now you are ready to use your own Densenet-161!</p>