Extracting features from pre-trained CNN, but got same value of any frame

username: Sally
cooked: <p>Hi, I’m new in PyTorch. I tried to get the inner layer of ResNet-101 and also Vgg (both pretrained) as features of images. But when I took a look at the features, the output from ResNet are all having same value, and also output from Vgg…</p>
<p>Is there anyone knows the problem? Thank you.</p>

username: ptrblck
cooked: <p>What is your use case and when do the features look the same?<br>
Also how did you visualize them?</p>

username: Sally
cooked: <p>I tried to extract features for frames in videos. So I first split video into frames with 32fps. And then I use the following code to extract pool 5 layer as features (save to .pt).</p>
<pre><code class="lang-auto">def Preprocess(input_image):
    transform = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ])
    input_tensor = transform(input_image)
    batch = input_tensor.unsqueeze(0) # create a mini-batch as expected by the model
    return batch

input_image = Image.open(inputFile)
input_batch = Preprocess(input_image)
model = models.resnet101(pretrained=True)

res101_conv = nn.Sequential(*list(model.children())[:-1])

for param in res101_conv.parameters():
     param.requires_grad = False
outputs = res101_conv(input_batch)
result = outputs[0].flatten()
</code></pre>
<p>And I check the output of each frames, frame 1 feature is similar to frame 1000. I used consine similarity to check. this is the code.</p>
<pre><code class="lang-auto">cosine = nn.CosineSimilarity(dim=1, eps=1e-6)
print(cosine(frame_one.view(1, -1), frame_two.view(1, -1)))
</code></pre>

username: ptrblck
cooked: <p>What are these frames showing, what is the prediction, and which features do you expect?<br>
Is the prediction wrong for both frames or why are you concerned?</p>

username: Sally
cooked: <p>Hi I extract these features just to use them for representing a video (I want to use these feature to analyze videos), not to classify the object or images. But the extracted features similarity is 0.9 up. And I think it is weird that actually the video has different frames and each frame are having different content, the extracted features should not be that similar. Or am I wrong with this concept?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Sally" data-post="5" data-topic="83935">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/d6d6ee/48.png" class="avatar"> Sally:</div>
<blockquote>
<p>And I think it is weird that actually the video has different frames and each frame are having different content, the extracted features should not be that similar. Or am I wrong with this concept?</p>
</blockquote>
</aside>
<p>It depends, what the input frames are showing and what the model predicts.<br>
E.g. if the model predicts the same class for all input frames, I would assume that the extracted features are quite equal.<br>
Depending on your current use case and input, the model might predict a single random class or just completely random output, since it wasn’t trained on your dataset.</p>
<p>Did you check, what the model prediction is for these frames? If the prediction is equally wrong, the model isn’t able to extract useful features from your frames.</p>

username: Sally
cooked: <p>Sorry for the late reply.</p>
<p>Actually the frames are extracted from a random video and there is no label for the object classification of each frame. I just tried to use the deep features to see if I can find the similarity between frames and find the boundary of videos. Just like you said, the model is not trained on my dataset so I think maybe the pre-trained resnet is not suitable in my case. Thank you so much for helping me to figure out the reason.</p>

username: ptrblck
cooked: <p>This might be a possible reason, but you could still run some sanity checks with your current video data.<br>
E.g. if your video contains some frames with ImageNet objects, such as dogs, I would recommend to try to classify them using your current code. If the model fails completely, you might have some bugs e.g. in the preprocessing.</p>

username: Sally
cooked: <p>Ok I’ll try! Thank you a lot!</p>