RuntimeError: The size of tensor a (65537) must match the size of tensor b (50177) at non-singleton dimension 1

username: szahan
cooked: <p>I am trying to extract part of the pretrained mvit model but it’s giving me the following error RuntimeError: The size of tensor a (65537) must match the size of tensor b (50177) at non-singleton dimension 1. If I use x3d_l, it works just fine. Not sure what I am doing wrong.</p>
<pre><code class="lang-auto">class VideoModel(nn.Module):
            def __init__(self):
                super(VideoModel, self).__init__()
                model_name = 'mvit_base_32x3'#'mvit_base_32x3'   X3D_L - 77.44% and mvit_base_32x3 - 80.30 on kinetics400 
                original_model =  torch.hub.load('facebookresearch/pytorchvideo', model_name, pretrained=True)
                self.features = nn.Sequential(
                    *list(original_model.children())[:-1] #mvit
                    #*list(original_model.blocks.children())[:-1] #x3d
                )
                
            def forward(self, x):
                x = self.features(x)
                return x

model = VideoModel()
x = torch.randn(1, 3,32,256,256)
pred = model(x)
</code></pre>
<p>If I change the size of x to (1,3,32,224,224) I get NotImplementedError but the following code works and produces the [1, 400] prediction probability.</p>
<pre><code class="lang-auto">model_name = 'mvit_base_32x3'#'mvit_base_32x3'   X3D_L - 77.44% and mvit_base_32x3 - 80.30 on kinetics400 
original_model =  torch.hub.load('facebookresearch/pytorchvideo', model_name, pretrained=True)
x = torch.randn(1, 3,32,224,224)
pred = original_model(x)
</code></pre>
<p>Any suggestion will be a great help. Thanks.</p>

username: Matias_Vasquez
cooked: <p>Here you can see what can happen when you use <code>nn.Sequential</code>.</p>
<aside class="quote quote-modified" data-post="2" data-topic="151700">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matias_vasquez/48/47847_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/wrapping-submodules-in-sequential/151700/2">Wrapping submodules in Sequential</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Maybe the easiest way to understand this is with a little example. 
Here I have defined two Modules that perform functional stuff. The SimpleBlock multiplies by 2. The MainModel uses this SimpleBlock in two different places and also performs some functional stuff (multiply by 3 and add 3). 
This can, of course, be something more complex like using layers, batch normalization, flatten or whatever you want to do. 
class SimpleBlock(torch.nn.Module):
    def __init__(self):
        super().__init__…
  </blockquote>
</aside>

<p>If you inspect <a href="https://github.com/facebookresearch/pytorchvideo/blob/825e8aa9430e6dc431c9b6548355820ba08dbf09/pytorchvideo/models/hub/x3d.py#L132" rel="noopener nofollow ugc">here</a> how the <code>x3d</code> is created, you will see that it uses <code>nn.Modules</code>. If you see my other answer then you will see why this works.</p>
<p>Now in contrast, MViT is using functional API on the <code>forward</code> pass. You can inspect the source code <a href="https://github.com/facebookresearch/pytorchvideo/blob/825e8aa9430e6dc431c9b6548355820ba08dbf09/pytorchvideo/models/vision_transformers.py#L168" rel="noopener nofollow ugc">here</a>.</p>
<p>If you use the <code>original_model</code>, then, as you said, you have to input the video in the shape <code>BxCxTxHxW</code> where <code>C=3</code>, <code>H=224</code> and <code>W=224</code>.</p>
<h2><a name="not-changing-anything-1" class="anchor" href="#not-changing-anything-1"></a>Not changing anything</h2>
<p>This will work.</p>
<pre data-code-wrap="python"><code class="lang-python">model_name = 'mvit_base_32x3'
original_model =  torch.hub.load('facebookresearch/pytorchvideo', model_name, pretrained=True)

input_vid = torch.randn(1, 3,32,224,224)
original_model(input_vid).shape
</code></pre>
<pre data-code-wrap="python"><code class="lang-python"># Output
torch.Size([1, 400])
</code></pre>
<h2><a name="using-nnsequential-2" class="anchor" href="#using-nnsequential-2"></a>Using nn.Sequential</h2>
<p>Above I have linked another answer as to why it may not work with <code>nn.Sequential</code>.</p>
<pre data-code-wrap="python"><code class="lang-python">features = torch.nn.Sequential(*list(original_model.children())[:-1])
features(input_vid)
</code></pre>
<p>As you said, this will raise the following error</p>
<pre data-code-wrap="python"><code class="lang-python">--&gt; 201     raise NotImplementedError
</code></pre>
<h2><a name="modifying-the-head-3" class="anchor" href="#modifying-the-head-3"></a>Modifying the head</h2>
<p>If you print the original model, you can see that you can access the head like this</p>
<pre data-code-wrap="python"><code class="lang-python">original_model.head
</code></pre>
<pre data-code-wrap="python"><code class="lang-python"># Output
VisionTransformerBasicHead(
  (sequence_pool): SequencePool()
  (dropout): Dropout(p=0.5, inplace=False)
  (proj): Linear(in_features=768, out_features=400, bias=True)
)
</code></pre>
<p>Here you can see that <code>original_model.head.proj</code> is a <code>Linear</code> layer. You can modify this to fit your needs.</p>
<p>For example</p>
<pre data-code-wrap="python"><code class="lang-python">model_name = 'mvit_base_32x3'
original_model =  torch.hub.load('facebookresearch/pytorchvideo', model_name, pretrained=True)

N_CLASSES = 10
original_model.head.proj = torch.nn.Linear(768, N_CLASSES)

input_vid = torch.randn(1, 3,32,224,224)
original_model(input_vid).shape
</code></pre>
<p>This will work.</p>
<pre data-code-wrap="python"><code class="lang-python"># Output
torch.Size([1, 10])
</code></pre>
<p>If, for example you wanted to get rid of this <code>Linear</code> layer you could use <code>nn.Identity()</code> instead.</p>
<p>This will give you the output up until before the last <code>Linear</code> layer.</p>

username: szahan
cooked: <aside class="quote no-group" data-username="Matias_Vasquez" data-post="2" data-topic="151666">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matias_vasquez/48/47847_2.png" class="avatar"> Matias_Vasquez:</div>
<blockquote>
<p><code>nn.Identity()</code></p>
</blockquote>
</aside>
<p>Thank you for your detailed explanation. I had no idea of this property.</p>