Import pytorchvideo transformer model

username: szahan
cooked: <p>Hi</p>
<p>I am trying to import the last MViT model from model zoo with pretrained weights</p>
<p>link: <a href="https://pytorchvideo.readthedocs.io/en/latest/model_zoo.html" class="inline-onebox" rel="noopener nofollow ugc">Model Zoo and Benchmarks — PyTorchVideo documentation</a></p>
<p>there are many examples for slow_r50/ slowfast_r50 but I could not find any for MViT</p>
<p>for example “x3d s” model can be loaded using the following code</p>
<pre><code class="lang-auto">model_name = 'x3d_s'
model = torch.hub.load('facebookresearch/pytorchvideo', model_name, pretrained=True)
</code></pre>
<p>I found this example here <a href="https://pytorch.org/hub/facebookresearch_pytorchvideo_x3d/" class="inline-onebox" rel="noopener nofollow ugc">X3D | PyTorch</a></p>
<p>but how can I load MViT? I have tried using the combination of arch and depth as is the case with many models (though not all) but did not work.</p>
<p>plus what will be the input shape.</p>
<p>Could you please help? Thanks</p>

username: Matias_Vasquez
cooked: <p>You can guide yourself with the github repository to see how it is done for <code>x3d_s</code> and look how it should be done for the transformer.</p>
<p>This is for the <code>x3d_s</code>. As you can see, you can load any model from this file by using the names that are used in the <code>def</code>.</p>
<ul>
<li>x3d_s: line 68</li>
<li>x3d_m: line 100</li>
<li>x3d_l: line 132</li>
<li>etc.</li>
</ul>
<aside class="onebox githubblob" data-onebox-src="https://github.com/facebookresearch/pytorchvideo/blob/5e585415bde879756f60b8864b78e87c102c6abc/pytorchvideo/models/hub/x3d.py#L68">
  <header class="source">

      <a href="https://github.com/facebookresearch/pytorchvideo/blob/5e585415bde879756f60b8864b78e87c102c6abc/pytorchvideo/models/hub/x3d.py#L68" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/facebookresearch/pytorchvideo/blob/5e585415bde879756f60b8864b78e87c102c6abc/pytorchvideo/models/hub/x3d.py#L68" target="_blank" rel="noopener nofollow ugc">facebookresearch/pytorchvideo/blob/5e585415bde879756f60b8864b78e87c102c6abc/pytorchvideo/models/hub/x3d.py#L68</a></h4>



    <pre class="onebox">      <code class="lang-py">
        <ol class="start lines" start="58" style="counter-reset: li-counter 57 ;">
            <li>    return _x3d(</li>
            <li>        pretrained=pretrained,</li>
            <li>        progress=progress,</li>
            <li>        checkpoint_path=checkpoint_paths["x3d_xs"],</li>
            <li>        input_clip_length=4,</li>
            <li>        input_crop_size=160,</li>
            <li>        **kwargs,</li>
            <li>    )</li>
            <li>
            </li>
<li>
            </li>
<li class="selected">def x3d_s(</li>
            <li>    pretrained: bool = False,</li>
            <li>    progress: bool = True,</li>
            <li>    **kwargs,</li>
            <li>):</li>
            <li>    """</li>
            <li>    X3D-XS model architecture [1] trained on the Kinetics dataset.</li>
            <li>    Model with pretrained weights has top1 accuracy of 73.33.</li>
            <li>
            </li>
<li>    [1] Christoph Feichtenhofer, "X3D: Expanding Architectures for</li>
            <li>    Efficient Video Recognition." https://arxiv.org/abs/2004.04730</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>If you now go to the transformer file, you can look for the definitions</p>
<ul>
<li>mvit_base_16x4: line 57</li>
<li>mvit_base_32x3: line 92</li>
<li>mvit_base_16: line 127</li>
<li>etc</li>
</ul>
<aside class="onebox githubblob" data-onebox-src="https://github.com/facebookresearch/pytorchvideo/blob/main/pytorchvideo/models/hub/vision_transformers.py">
  <header class="source">

      <a href="https://github.com/facebookresearch/pytorchvideo/blob/main/pytorchvideo/models/hub/vision_transformers.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/facebookresearch/pytorchvideo/blob/main/pytorchvideo/models/hub/vision_transformers.py" target="_blank" rel="noopener nofollow ugc">facebookresearch/pytorchvideo/blob/main/pytorchvideo/models/hub/vision_transformers.py</a></h4>


      <pre><code class="lang-py"># Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.

from typing import Any

import torch.nn as nn
from pytorchvideo.models.vision_transformers import (
    create_multiscale_vision_transformers,
)

from .utils import MODEL_ZOO_ROOT_DIR, hub_model_builder


checkpoint_paths = {
    "mvit_base_16x4": "{}/kinetics/MVIT_B_16x4.pyth".format(MODEL_ZOO_ROOT_DIR),
    "mvit_base_32x3": "{}/kinetics/MVIT_B_32x3_f294077834.pyth".format(
        MODEL_ZOO_ROOT_DIR
    ),
    "mvit_base_16": "{}/imagenet/MVIT_B_16_f292487636.pyth".format(MODEL_ZOO_ROOT_DIR),
}

</code></pre>



  This file has been truncated. <a href="https://github.com/facebookresearch/pytorchvideo/blob/main/pytorchvideo/models/hub/vision_transformers.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>So now you only need to choose the one you want and do the same thing in your example code.</p>
<pre><code class="lang-python"># For example
model_name = "mvit_base_32x3"
model = torch.hub.load('facebookresearch/pytorchvideo', model_name, pretrained=True)
</code></pre>

username: szahan
cooked: <aside class="quote no-group" data-username="Matias_Vasquez" data-post="2" data-topic="148229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matias_vasquez/48/47847_2.png" class="avatar"> Matias_Vasquez:</div>
<blockquote>
<pre><code class="lang-auto">model_name = "mvit_base_32x3"
model = torch.hub.load('facebookresearch/pytorchvideo', model_name, pretrained=True)
</code></pre>
</blockquote>
</aside>
<p>Thank you so much for your detailed reply. It’s just not gonna solve this one but also will help me in future.</p>