Extract the 2048 vector of a fine-tuned Inception V3 on test set

Mona_Jalal: <p>I have already finetuned an Inception V3 model on my dataset which was already pretrained on ImageNet.</p>
<p>Now, I need to load the save pt model and run it on test patches and extract the 2048 feature vector from Inception V3.</p>
<p>I am not sure what code snippet I should use for extracting feature vectors of size 2048 on an inception v3 model that is already finetuned on my own data (train data) on the test set.</p>
<pre><code class="lang-auto">import torch
model = torch.load('pt/model_ft_100e_best_acc.pt')
feature_extractor = torch.nn.Sequential(*(list(model.modules())[:-2]))
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

batch_size = 1

input_size = 299 # for Inception V3

data_transforms = {
    'train': transforms.Compose([
        transforms.Resize((input_size, input_size)),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.6964, 0.4764, 0.6246], [0.1896, 0.2210, 0.1864]) 

    ]),
    'val': transforms.Compose([
        transforms.Resize((input_size, input_size)),
        transforms.ToTensor(),
        transforms.Normalize([0.6964, 0.4764, 0.6246], [0.1896, 0.2210, 0.1864]) 
    ]),

    'test': transforms.Compose([
        transforms.Resize((input_size, input_size)),
        transforms.ToTensor(),
        transforms.Normalize([0.6964, 0.4764, 0.6246], [0.1896, 0.2210, 0.1864]) 
    ])
}


data_dir = "/projectnb/ivcgroup/jalal/data/fold1/" # for running on SCC
# Create training and validation datasets
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms[x]) for x in ['test']}
# Create training and validation dataloaders

dataloaders_dict = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=batch_size, shuffle=True, num_workers=4) for x in ['test']}


sample_fnames_labels = dataloaders_dict['test'].dataset.samples
sample_large_images = {}


test_large_images = {}
test_loss = 0.0
test_acc = 0


with torch.no_grad():
    test_running_loss = 0.0
    test_running_corrects = 0
    print(len(dataloaders_dict['test']))
    for i, (inputs, labels) in enumerate(dataloaders_dict['test']):
        print(i)
        test_input = inputs.to(device)
        test_label = labels.to(device)
        test_output = feature_extractor(test_input) # here I need to get feature vectors of size 2048
        print("test feature size is: ", test_output.shape)
</code></pre>

Mona_Jalal: <p>In my model, I do see that avgpool layer shows 2048. However, when I try this solution from <a class="mention" href="/u/ptrblck">@ptrblck</a>  it doesn’t recognize the syntax.</p>
<pre><code class="lang-auto">
      (branch3x3dbl_3b): BasicConv2d(
        (conv): Conv2d(384, 384, kernel_size=(3, 1), stride=(1, 1), padding=(1, 0), bias=False)
        (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
      )
      (branch_pool): BasicConv2d(
        (conv): Conv2d(2048, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
    (dropout): Dropout(p=0.5, inplace=False)
    (fc): Linear(in_features=2048, out_features=2, bias=True)
  )
)&gt;
</code></pre>
<p>I tried:<br>
<code>model.avgpool.register_forward_hook(get_activation('avgpool'))</code></p>
<aside class="quote quote-modified" data-post="6" data-topic="17254">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">How can l load my best model as a feature extractor/evaluator?</a> 
  </div>
  <blockquote>
    I’ve created a small code snippet using a forward hook to store one activation from fc2: 
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.cl1 = nn.Linear(25, 60)
        self.cl2 = nn.Linear(60, 16)
        self.fc1 = nn.Linear(16, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
        
    def forward(self, x):
        x = F.relu(self.cl1(x))
        x = F.relu(self.cl2(x))
        x = F.relu(self.fc1(x))
  …
  </blockquote>
</aside>

<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a38a45d87d9bae945fc7933edfd2a1fd45bc546d.png" data-download-href="https://discuss.pytorch.org/uploads/default/a38a45d87d9bae945fc7933edfd2a1fd45bc546d" title="Screen Shot 2022-05-24 at 12.47.39 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a38a45d87d9bae945fc7933edfd2a1fd45bc546d_2_690x282.png" alt="Screen Shot 2022-05-24 at 12.47.39 PM" data-base62-sha1="nkK4HCMuYJ31f6XhPlP2bBYN9F3" width="690" height="282" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a38a45d87d9bae945fc7933edfd2a1fd45bc546d_2_690x282.png, https://discuss.pytorch.org/uploads/default/original/3X/a/3/a38a45d87d9bae945fc7933edfd2a1fd45bc546d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/3/a38a45d87d9bae945fc7933edfd2a1fd45bc546d.png 2x" data-dominant-color="222222"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-05-24 at 12.47.39 PM</span><span class="informations">962×394 26.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The error you are seeing is unrelated to PyTorch and most likely caused by <code>Jupyter Notebook Renderers Extension</code> as described e.g. <a href="https://stackoverflow.com/questions/68736618/error-loading-preloads-could-not-find-renderer">here</a>.</p>

Mona_Jalal: <p>Thank you for pointing me to that link. However, it also seems that get_activation is not recognized. Is the code in cell <span class="hashtag">#12</span> correct for inception v3 in your opinion?</p>

ptrblck: <p>Yes it looks correct and also works fine:</p>
<pre><code class="lang-python">activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

model = models.inception_v3()
model.avgpool.register_forward_hook(get_activation("avgpool"))

x = torch.randn(2, 3, 299, 299)
out = model(x)

print(activation['avgpool'].shape)
# torch.Size([2, 2048, 1, 1])
</code></pre>

Mona_Jalal: <p>you were correct that the specific renderer error was on VSCode Jupyter notebook side. I had the extension installed but I uninstalled it and installed it back again and the problem is gone.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/41ec2c242db37ecbb505e84b6a87de03eab8019c.png" data-download-href="https://discuss.pytorch.org/uploads/default/41ec2c242db37ecbb505e84b6a87de03eab8019c" title="Screen Shot 2022-05-24 at 11.21.37 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41ec2c242db37ecbb505e84b6a87de03eab8019c_2_690x120.png" alt="Screen Shot 2022-05-24 at 11.21.37 PM" data-base62-sha1="9pb21Ai7zZFodY7jExw8jwqiIcQ" width="690" height="120" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41ec2c242db37ecbb505e84b6a87de03eab8019c_2_690x120.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41ec2c242db37ecbb505e84b6a87de03eab8019c_2_1035x180.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41ec2c242db37ecbb505e84b6a87de03eab8019c_2_1380x240.png 2x" data-dominant-color="282D31"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-05-24 at 11.21.37 PM</span><span class="informations">1784×312 103 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>^ this is the version I had initially. I also uninstalled and installed the Jupyter extension.</p>
<p>Here’s the error I am getting:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b42318061da262c95cd15cd4d2af9fdb8ce53807.png" data-download-href="https://discuss.pytorch.org/uploads/default/b42318061da262c95cd15cd4d2af9fdb8ce53807" title="Screen Shot 2022-05-24 at 11.22.51 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b42318061da262c95cd15cd4d2af9fdb8ce53807_2_690x186.png" alt="Screen Shot 2022-05-24 at 11.22.51 PM" data-base62-sha1="pHz8eQdbcIXQUGvnogqYf6vbHAb" width="690" height="186" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b42318061da262c95cd15cd4d2af9fdb8ce53807_2_690x186.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b42318061da262c95cd15cd4d2af9fdb8ce53807_2_1035x279.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b42318061da262c95cd15cd4d2af9fdb8ce53807_2_1380x372.png 2x" data-dominant-color="212525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-05-24 at 11.22.51 PM</span><span class="informations">1534×415 72.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<blockquote>
<p>AttributeError: ‘DataParallel’ object has no attribute ‘avgpool’</p>
</blockquote>
<p>From the link you shared, I used this answer: <a href="https://stackoverflow.com/a/69575712/2414957" class="inline-onebox" rel="noopener nofollow ugc">python - Error loading preloads: Could not find renderer - Stack Overflow</a></p>

Mona_Jalal: <p>I get an error using your exact code copied:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/285cf7894476e2e1b885b6efc0bea923620bad6f.png" data-download-href="https://discuss.pytorch.org/uploads/default/285cf7894476e2e1b885b6efc0bea923620bad6f" title="Screen Shot 2022-05-24 at 11.26.23 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/285cf7894476e2e1b885b6efc0bea923620bad6f_2_690x308.png" alt="Screen Shot 2022-05-24 at 11.26.23 PM" data-base62-sha1="5L4hkE4QkgDIKXiK7bfjyo4BuN1" width="690" height="308" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/285cf7894476e2e1b885b6efc0bea923620bad6f_2_690x308.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/285cf7894476e2e1b885b6efc0bea923620bad6f_2_1035x462.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/285cf7894476e2e1b885b6efc0bea923620bad6f_2_1380x616.png 2x" data-dominant-color="222424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-05-24 at 11.26.23 PM</span><span class="informations">1595×712 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>&gt; AttributeError: ‘DataParallel’ object has no attribute ‘avgpool’</strong></p>

Mona_Jalal: <p>fixed it using <code>module</code> as suggested by you here</p><aside class="quote" data-post="2" data-topic="14842">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/fine-tuning-resnet-dataparallel-object-has-no-attribute-fc/14842/2">Fine tuning resnet: 'DataParallel' object has no attribute 'fc'</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    When using DataParallel your nn.Module will be in .module: 
model = Net()
model.fc

model_parallel = nn.DataParallel(model)
model_parallel.module.fc
  </blockquote>
</aside>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f4ff2fc368c94302053f52e8209e5dc5bcc92cf8.png" data-download-href="https://discuss.pytorch.org/uploads/default/f4ff2fc368c94302053f52e8209e5dc5bcc92cf8" title="Screen Shot 2022-05-24 at 11.43.30 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4ff2fc368c94302053f52e8209e5dc5bcc92cf8_2_690x137.png" alt="Screen Shot 2022-05-24 at 11.43.30 PM" data-base62-sha1="yXleIYk7VK6v5NVPMZf5n4P9jFm" width="690" height="137" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4ff2fc368c94302053f52e8209e5dc5bcc92cf8_2_690x137.png, https://discuss.pytorch.org/uploads/default/original/3X/f/4/f4ff2fc368c94302053f52e8209e5dc5bcc92cf8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/4/f4ff2fc368c94302053f52e8209e5dc5bcc92cf8.png 2x" data-dominant-color="282927"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-05-24 at 11.43.30 PM</span><span class="informations">936×186 24.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>