Model summary for slowfast_r50 model

username: Renuraj
cooked: <p>In the below script of pytorch video for classification using pretrained model slowfast_r50, I am not able to get the model summary rather I am getting model architecture blocks only;<br>
import torch<br>
import json<br>
from torchsummary import summary<br>
from torchvision.transforms import Compose, Lambda<br>
from torchvision.transforms._transforms_video import (<br>
CenterCropVideo,<br>
NormalizeVideo,<br>
)<br>
from pytorchvideo.data.encoded_video import EncodedVideo<br>
from pytorchvideo.transforms import (<br>
ApplyTransformToKey,<br>
ShortSideScale,<br>
UniformTemporalSubsample,<br>
UniformCropVideo<br>
)<br>
from typing import Dict</p>
<h1><a name="p-455353-device-on-which-to-run-the-model-1" class="anchor" href="#p-455353-device-on-which-to-run-the-model-1"></a>Device on which to run the model</h1>
<h1><a name="p-455353-set-to-cuda-to-load-on-gpu-2" class="anchor" href="#p-455353-set-to-cuda-to-load-on-gpu-2"></a>Set to cuda to load on GPU</h1>
<p>device = “cpu”</p>
<h1><a name="p-455353-pick-a-pretrained-model-and-load-the-pretrained-weights-3" class="anchor" href="#p-455353-pick-a-pretrained-model-and-load-the-pretrained-weights-3"></a>Pick a pretrained model and load the pretrained weights</h1>
<p>model_name = “slowfast_r50”<br>
model = torch.hub.load(“facebookresearch/pytorchvideo”, model=model_name, pretrained=True)<br>
print(model)<br>
slow_input_size=[1,3,8,256,256]<br>
fast_input_size=[1,3,32,256,256]</p>
<p>input_size=[slow_input_size,fast_input_size]</p>
<h1><a name="p-455353-model_summarysummarymodelinput_size-4" class="anchor" href="#p-455353-model_summarysummarymodelinput_size-4"></a>model_summary=summary(model,input_size)</h1>
<h1><a name="p-455353-printmodel_summary-5" class="anchor" href="#p-455353-printmodel_summary-5"></a>print(model_summary)</h1>
<h1><a name="p-455353-save-the-entire-model-including-the-architecture-6" class="anchor" href="#p-455353-save-the-entire-model-including-the-architecture-6"></a>Save the entire model (including the architecture)</h1>
<p>model_save=torch.save(model, “slowfast_r50_full_model.pth”)<br>
<span class="hashtag-raw">#print</span>(model_save)</p>
<h1><a name="p-455353-set-to-eval-mode-and-move-to-desired-device-7" class="anchor" href="#p-455353-set-to-eval-mode-and-move-to-desired-device-7"></a>Set to eval mode and move to desired device</h1>
<p>model = model.to(device)<br>
model = model.eval()<br>
with open(“/home/mantra/Documents/Projects/Video/pytorchvideo_tutorial/kinetics_classnames.json”, “r”) as f:<br>
kinetics_classnames = json.load(f)</p>
<h1><a name="p-455353-create-an-id-to-label-name-mapping-8" class="anchor" href="#p-455353-create-an-id-to-label-name-mapping-8"></a>Create an id to label name mapping</h1>
<p>kinetics_id_to_classname = {}<br>
for k, v in kinetics_classnames.items():<br>
kinetics_id_to_classname[v] = str(k).replace(‘"’, “”)</p>
<p>side_size = 256<br>
mean = [0.45, 0.45, 0.45]<br>
std = [0.225, 0.225, 0.225]<br>
crop_size = 256<br>
num_frames = 32<br>
sampling_rate = 2<br>
frames_per_second = 30<br>
alpha = 4</p>
<p>class PackPathway(torch.nn.Module):<br>
“”"<br>
Transform for converting video frames as a list of tensors.<br>
“”"<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()</p>
<pre><code>def forward(self, frames: torch.Tensor):
    fast_pathway = frames
    # Perform temporal sampling from the fast pathway.
    slow_pathway = torch.index_select(
        frames,
        1,
        torch.linspace(
            0, frames.shape[1] - 1, frames.shape[1] // alpha
        ).long(),
    )
    frame_list = [slow_pathway, fast_pathway]
    return frame_list
</code></pre>
<p>transform =  ApplyTransformToKey(<br>
key=“video”,<br>
transform=Compose(<br>
[<br>
UniformTemporalSubsample(num_frames),<br>
Lambda(lambda x: x/255.0),<br>
NormalizeVideo(mean, std),<br>
ShortSideScale(<br>
size=side_size<br>
),<br>
CenterCropVideo(crop_size),<br>
PackPathway()<br>
]<br>
),<br>
)</p>
<h1><a name="p-455353-the-duration-of-the-input-clip-is-also-specific-to-the-model-9" class="anchor" href="#p-455353-the-duration-of-the-input-clip-is-also-specific-to-the-model-9"></a>The duration of the input clip is also specific to the model.</h1>
<p>clip_duration = (num_frames * sampling_rate)/frames_per_second<br>
print(clip_duration)</p>
<h1><a name="p-455353-load-the-example-video-10" class="anchor" href="#p-455353-load-the-example-video-10"></a>Load the example video</h1>
<p>video_path = “/home/mantra/Documents/Projects/Video/pytorchvideo_tutorial/-1HT31BzADs_000118_000128.mp4”</p>
<h1><a name="p-455353-select-the-duration-of-the-clip-to-load-by-specifying-the-start-and-end-duration-11" class="anchor" href="#p-455353-select-the-duration-of-the-clip-to-load-by-specifying-the-start-and-end-duration-11"></a>Select the duration of the clip to load by specifying the start and end duration</h1>
<h1><a name="p-455353-the-start_sec-should-correspond-to-where-the-action-occurs-in-the-video-12" class="anchor" href="#p-455353-the-start_sec-should-correspond-to-where-the-action-occurs-in-the-video-12"></a>The start_sec should correspond to where the action occurs in the video</h1>
<p>start_sec = 0<br>
end_sec = start_sec + clip_duration</p>
<h1><a name="p-455353-initialize-an-encodedvideo-helper-class-13" class="anchor" href="#p-455353-initialize-an-encodedvideo-helper-class-13"></a>Initialize an EncodedVideo helper class</h1>
<p>video = EncodedVideo.from_path(video_path)</p>
<h1><a name="p-455353-load-the-desired-clip-14" class="anchor" href="#p-455353-load-the-desired-clip-14"></a>Load the desired clip</h1>
<p>video_data = video.get_clip(start_sec=start_sec, end_sec=end_sec)<br>
print(‘video_data’)<br>
print(type(video_data[‘video’]))<br>
print(video_data[‘video’].shape)</p>
<p>print(type(video_data[‘audio’]))<br>
print(video_data[‘audio’].shape)</p>
<p>print(‘video data keys’)<br>
for key, value in video_data.items() :<br>
print(key)</p>
<h1><a name="p-455353-apply-a-transform-to-normalize-the-video-input-15" class="anchor" href="#p-455353-apply-a-transform-to-normalize-the-video-input-15"></a>Apply a transform to normalize the video input</h1>
<p>video_data = transform(video_data)<br>
print(video_data)<br>
print(video_data[‘video’][0])</p>
<h1><a name="p-455353-move-the-inputs-to-the-desired-device-16" class="anchor" href="#p-455353-move-the-inputs-to-the-desired-device-16"></a>Move the inputs to the desired device</h1>
<p>inputs = video_data[“video”]<br>
inputs = [i.to(device)[None, …] for i in inputs]<br>
print(inputs[0].shape)<br>
print(inputs[1].shape)</p>
<h1><a name="p-455353-pass-the-input-clip-through-the-model-17" class="anchor" href="#p-455353-pass-the-input-clip-through-the-model-17"></a>Pass the input clip through the model</h1>
<p>preds = model(inputs)</p>
<h1><a name="p-455353-torchsize1-3-8-256-256-18" class="anchor" href="#p-455353-torchsize1-3-8-256-256-18"></a>torch.Size([1, 3, 8, 256, 256])</h1>
<h1><a name="p-455353-torchsize1-3-32-256-256-19" class="anchor" href="#p-455353-torchsize1-3-32-256-256-19"></a>torch.Size([1, 3, 32, 256, 256])</h1>
<h1><a name="p-455353-model_summarysummarymodel1-3-8-256-256-20" class="anchor" href="#p-455353-model_summarysummarymodel1-3-8-256-256-20"></a>model_summary=summary(model,(1, 3, 8, 256, 256))</h1>
<p>model_summary=summary(model,(([[1, 3, 8, 256, 256],[1, 3, 32, 256, 256]])))<br>
print(model_summary)</p>
<p>print(preds)<br>
print(preds.shape)</p>
<h1><a name="p-455353-get-the-predicted-classes-21" class="anchor" href="#p-455353-get-the-predicted-classes-21"></a>Get the predicted classes</h1>
<p>post_act = torch.nn.Softmax(dim=1)<br>
preds = post_act(preds)<br>
print(preds)<br>
pred_classes = preds.topk(k=5).indices<br>
print(pred_classes)<br>
print(pred_classes.shape)<br>
print(preds[0][253])</p>
<h1><a name="p-455353-printpreds0151-22" class="anchor" href="#p-455353-printpreds0151-22"></a>print(preds[0][151])</h1>
<h1><a name="p-455353-printpreds0182-23" class="anchor" href="#p-455353-printpreds0182-23"></a>print(preds[0][182])</h1>
<h1><a name="p-455353-printpreds0367-24" class="anchor" href="#p-455353-printpreds0367-24"></a>print(preds[0][367])</h1>
<h1><a name="p-455353-printpreds0160-25" class="anchor" href="#p-455353-printpreds0160-25"></a>print(preds[0][160])</h1>
<h1><a name="p-455353-map-the-predicted-classes-to-the-label-names-26" class="anchor" href="#p-455353-map-the-predicted-classes-to-the-label-names-26"></a>Map the predicted classes to the label names</h1>
<p>pred_class_names = [kinetics_id_to_classname[int(i)] for i in pred_classes[0]]<br>
print(“Predicted labels: %s” % ", ".join(pred_class_names))</p>
<p>I am getting the following error while running above script, please anyone help me to reslove this issue;</p>
<p>Traceback (most recent call last):<br>
File “/home/mantra/Documents/Projects/Video/pytorchvideo_tutorial/slowfast_arch.py”, line 14, in <br>
summary(model, input_sizes )<br>
File “/home/mantra/miniconda3/envs/video/lib/python3.12/site-packages/torchsummary/torchsummary.py”, line 72, in summary<br>
model(*x)<br>
File “/home/mantra/miniconda3/envs/video/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/home/mantra/miniconda3/envs/video/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1562, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
TypeError: Net.forward() takes 2 positional arguments but 3 were given</p>