Extract features from parts

username: mathwseg
cooked: <p>if I have an image that is split into several parts and I need to extract features from them as a whole image…  I mean, for example, the original image is a dog at the beach … this picture is divided into three parts “beach, sea, dog”. is there any way to extract features for those parts as one image? does concatenate features will be the same feature as the original image</p>

username: Andrei_Cristea
cooked: <p>I think <em>summing</em> is what you want, and here’s a quick experiment that shows that it probably works, but it may not be trivial:</p>
<pre><code class="lang-auto">import torch, torchvision
from torchvision.transforms import ToTensor, FiveCrop, Normalize, ToPILImage
img = Image.open("cat.jpg").resize((224, 224))
device = torch.device("cuda")

n = lambda x: Normalize(0.50, 0.25)(x)
tensor_big = n(ToTensor()(img)).to(device)  # the overall image
tensors_parts = [n(ToTensor()(x)).to(device) for x in FiveCrop(224//2)(img)[:-1]]  # image divided into 4

net = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True)
net.fc = torch.nn.Sequential()  # eliminate classification layer
net.to(device)
</code></pre>
<pre><code class="lang-auto">ToPILImage()(torchvision.utils.make_grid(tensor_big))
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c78f467d2bdb30145970735eda29b4ffc72d0330.png" alt="image" data-base62-sha1="stnYVfzCfq2pF1hIzQrPrRLmA36" width="168" height="168"></p>
<pre><code class="lang-auto">ToPILImage()(torchvision.utils.make_grid(tensors_parts))
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec46d5efe1ba1a5af76172bd4a96b8e5a5f70445.png" alt="image" data-base62-sha1="xIcrLk8lZwQJuaVAyCcmUiiCw4t" width="343" height="87"></p>
<p>Let’s analyze how correlated the following two features are:</p>
<ul>
<li>features extracted from the overall image</li>
<li>features extracted from each of the quarters of the image, and then summed</li>
</ul>
<pre><code class="lang-auto">features_big = net(tensor_big.unsqueeze(0)).squeeze()
features_parts = net(torch.stack(tensors_parts))
x = features_parts.sum(dim=0).cpu().detach().numpy()
y = features_big.cpu().detach().numpy()
plt.scatter(x, y)
plt.title(f"correlation = {100 * np.corrcoef(x, y)[0, 1]:.0f}%")
plt.xlabel("features of the parts, summed")
plt.ylabel("features of the overall image")
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/2/42f128c29840a2074bc32bcc8bb6610c05d76bb2.png" alt="image" data-base62-sha1="9ycc2RHY1hLeoEKfgsWVtUoECJQ" width="289" height="208"></p>
<p>94% correlation seems pretty high, however this could be driven by network weights or something. Let’s run an experiment where we generate random images and see how much correlation those yield, so we can opine on whether 94% is especially high or not:</p>
<pre><code class="lang-auto">n_simulations = 1000
corrcoeffs = []
for _ in range(n_simulations):
    z = net(torch.randn(3, 224, 224).unsqueeze(0).to(device)).squeeze().cpu().detach().numpy()
    corrcoeffs.append(np.corrcoef(z, y)[0, 1])
plt.hist(corrcoeffs, bins=20);
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e0bc6e97f6cc2475e7b38ec3b3ecebd165d7d55d.png" alt="image" data-base62-sha1="w46FmiJLucX7aQRHXHTw9e5m20R" width="282" height="188"></p>
<p>This suggests that, at least as far as images generated from random pixels are concerned, we have the following results:</p>
<ul>
<li>the “baseline” correlation between noise-generated features and our features is also very high, at around 90%</li>
<li>however, 94% is significantly stronger than this baseline, the maximum achieved in 1000 random sims being &lt; 91%</li>
</ul>
<p>So I would weakly conclude from this that summing the features extracted from parts is a promising approach.</p>
<p>Would be curious if others have thoughts on this.</p>

username: mathwseg
cooked: <p>great explanation , thanks a lot</p>