What is additional 256 features in ViT and an image restoration

timosy: <p>I’m useing ViT via vit_pytorch, a model is below,</p>
<pre><code class="lang-auto">ViT(
  (to_patch_embedding): Sequential(
    (0): Rearrange('b c (h p1) (w p2) -&gt; b (h w) (p1 p2 c)', p1=16, p2=16)
    (1): Linear(in_features=768, out_features=1024, bias=True)
  )
  (dropout): Dropout(p=0.1, inplace=False)
  (transformer): Transformer(
    (layers): ModuleList(

</code></pre>
<p>I input an image: torch.Size([1, 3, 128, 128]) and set a patch-size to 8 (16x16 with RGB image=768),<br>
and  I can get output:</p>
<pre><code class="lang-auto">torch.Size([1, 64, 1024]) after to_patch_embedding
torch.Size([1, 65, 1024]) after transformer
</code></pre>
<p>What is additional 256 features geneatred via Linear module in “to_patch_embedding”?</p>
<p>I’d like to restore/reconstruct a 2d image using the last torch.Size([1, 65, 1024]) , lets say it is a kind of  feaure 2d image. Is it possible ?</p>

ptrblck: <aside class="quote no-group" data-username="timosy" data-post="1" data-topic="161103">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/f04885/48.png" class="avatar"> timosy:</div>
<blockquote>
<p>What is additional 256 features geneatred via Linear module in “to_patch_embedding”?</p>
</blockquote>
</aside>
<p>Which 256 features are you referring to? Could you describe this question in more details, please?</p>

timosy: <p><strong>Thanks for the kind responce.</strong></p>
<p>I input an image: torch.Size([1, 3, 128, 128]) and set a patch-size to 8 (where <span class="hashtag">#of</span> features is 16x16 with RGB image=768), and  I can get output:</p>
<pre><code class="lang-auto">torch.Size([1, 64, 1024]) after "to_patch_embedding"
torch.Size([1, 65, 1024]) after "transformer"
</code></pre>
<p>When I look in the output of “to_patch_embedding” additional 256 features is added to the initial 768 features. Does this mean that “Linear” module just increases the amount of features using, for instance, Multi Layer Parceptron?</p>
<p><strong>The above one was my first question, and second question is below</strong></p>
<p>In the end of “transformer”, I can get the feature of “torch.Size([1, 65, 1024])”, the first array [:,0:1,:] means class token if I understand correctly, and the rest of the array is the feature of the image I input.</p>
<p>Here, what I’d like to do is to restore/reconstruct a 2d image using the last output “torch.Size([1, 65, 1024])”, Is it possible?</p>
<p>If my question is not a proper one in this forum, I’ll close the question.</p>

ptrblck: <aside class="quote no-group" data-username="timosy" data-post="3" data-topic="161103">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/f04885/48.png" class="avatar"> timosy:</div>
<blockquote>
<p>Does this mean that “Linear” module just increases the amount of features using, for instance, Multi Layer Parceptron?</p>
</blockquote>
</aside>
<p>Yes, the linear layer maps the <code>768</code> features from the input activation (i.e. the output of the <code>Rearrange</code> module) to <code>1024</code> features:</p>
<pre data-code-wrap="python"><code class="lang-python"> Linear(in_features=768, out_features=1024, bias=True)
</code></pre>
<aside class="quote no-group" data-username="timosy" data-post="3" data-topic="161103">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/f04885/48.png" class="avatar"> timosy:</div>
<blockquote>
<p>Here, what I’d like to do is to restore/reconstruct a 2d image using the last output “torch.Size([1, 65, 1024])”, Is it possible?</p>
</blockquote>
</aside>
<p>I don’t know how the image is related to these output features, but assuming the features were created from an image I could imagine that restoring them could work. However, it also highly depends on the model architecture, the input image resolution etc. You could check some approaches to restore images using e.g. autoencoders.</p>

timosy: <p>Thank you for your kind comments</p>
<blockquote>
<p>Yes, the linear layer maps the <code>768</code> features from the input activation</p>
</blockquote>
<p>I understand that the additional 256 features which are added via “Linear module” are unknown.</p>
<blockquote>
<p>I don’t know how the image is related to these output features, …</p>
</blockquote>
<p>Yes, one approach can be to put decoder after the transformer, and feed the output features to the decorder. Before doing it, I wantted to ask experts if other approach is possible to restore the input image.</p>
<p>Best regards,</p>

ptrblck: <aside class="quote no-group" data-username="timosy" data-post="5" data-topic="161103">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/f04885/48.png" class="avatar"> timosy:</div>
<blockquote>
<p>I understand that the additional 256 features which are added via “Linear module” are unknown.</p>
</blockquote>
</aside>
<p>No, these features are neither “unknown” nor are they added. The linear layer uses a linear transformation using its trainable parameters.<br>
Here is a small example showing it with just the <code>weight</code> matrix (the <code>bias</code> was skipped to simplify the example):</p>
<pre data-code-wrap="python"><code class="lang-python">input_features = torch.arange(5).view(1, -1).float()
print(input_features)
# tensor([[0., 1., 2., 3., 4.]])

lin = nn.Linear(in_features=5, out_features=7, bias=False)
print(lin.weight)
# Parameter containing:
# tensor([[-0.1611, -0.4197, -0.0630,  0.0092, -0.1258],
#         [-0.1206,  0.3807,  0.0975, -0.0231,  0.1823],
#         [-0.1722, -0.4393, -0.2412, -0.4215, -0.2296],
#         [-0.4187,  0.2990,  0.1310, -0.3879,  0.0489],
#         [ 0.3792, -0.3314, -0.3503, -0.3338, -0.2754],
#         [-0.0981,  0.3740, -0.1645,  0.2752, -0.1942],
#         [ 0.1042,  0.1736,  0.0024,  0.2128, -0.0113]], requires_grad=True)

out = lin(input_features)
print(out)
# tensor([[-1.0211,  1.2358, -3.1047, -0.4071, -3.1350,  0.0937,  0.7719]],
#        grad_fn=&lt;MmBackward0&gt;)
</code></pre>
<p>As you can see, the linear layer performs a matrix multiplication using the <code>input_feature</code> and its <code>weight</code> matrix to create the output features. No features are just "added` to the inputs or so, since the input is transformed.</p>

timosy: <p>Thank you for your kind comment again with an exsample!</p>