Video Classification using UCF-101 dataset

username: yakhyo
cooked: <p>I am trying to use video classifcation from torchvision models. The official code uses <code>kinetics</code> dataset however when I try to use UCF-101 dataset I am getting these <strong>runtime errors</strong>. Link to the train code:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/vision/blob/main/references/video_classification/train.py">
  <header class="source">

      <a href="https://github.com/pytorch/vision/blob/main/references/video_classification/train.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/main/references/video_classification/train.py" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/main/references/video_classification/train.py</a></h4>


      <pre><code class="lang-py">import datetime
import os
import time
import warnings

import presets
import torch
import torch.utils.data
import torchvision
import torchvision.datasets.video_utils
import utils
from torch import nn
from torch.utils.data.dataloader import default_collate
from torchvision.datasets.samplers import DistributedSampler, UniformClipSampler, RandomClipSampler


def train_one_epoch(model, criterion, optimizer, lr_scheduler, data_loader, device, epoch, print_freq, scaler=None):
    model.train()
    metric_logger = utils.MetricLogger(delimiter="  ")
    metric_logger.add_meter("lr", utils.SmoothedValue(window_size=1, fmt="{value}"))
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/vision/blob/main/references/video_classification/train.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Thanks in advace</p>
<pre data-code-wrap="cmd"><code class="lang-nohighlight">torch              1.12.0+cu113
torchaudio         0.12.0+cu113
torchvision        0.13.0+cu113
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/pyler/PycharmProjects/res-ufc101/train.py", line 388, in &lt;module&gt;
    main(args)
  File "/home/pyler/PycharmProjects/res-ufc101/train.py", line 287, in main
    train_one_epoch(model, criterion, optimizer, lr_scheduler, data_loader, device, epoch, args.print_freq, scaler)
  File "/home/pyler/PycharmProjects/res-ufc101/train.py", line 24, in train_one_epoch
    for video, target in metric_logger.log_every(data_loader, print_freq, header):
  File "/home/pyler/PycharmProjects/res-ufc101/utils.py", line 127, in log_every
    for obj in iterable:
  File "/home/pyler/Python/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 652, in __next__
    data = self._next_data()
  File "/home/pyler/Python/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1347, in _next_data
    return self._process_data(data)
  File "/home/pyler/Python/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1373, in _process_data
    data.reraise()
  File "/home/pyler/Python/envs/torch/lib/python3.8/site-packages/torch/_utils.py", line 461, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/pyler/Python/envs/torch/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/pyler/Python/envs/torch/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/pyler/Python/envs/torch/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/pyler/Python/envs/torch/lib/python3.8/site-packages/torchvision/datasets/ucf101.py", line 128, in __getitem__
    video = self.transform(video)
  File "/home/pyler/PycharmProjects/res-ufc101/presets.py", line 26, in __call__
    return self.transforms(x)
  File "/home/pyler/Python/envs/torch/lib/python3.8/site-packages/torchvision/transforms/transforms.py", line 94, in __call__
    img = t(img)
  File "/home/pyler/Python/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/pyler/Python/envs/torch/lib/python3.8/site-packages/torchvision/transforms/transforms.py", line 269, in forward
    return F.normalize(tensor, self.mean, self.std, self.inplace)
  File "/home/pyler/Python/envs/torch/lib/python3.8/site-packages/torchvision/transforms/functional.py", line 360, in normalize
    return F_t.normalize(tensor, mean=mean, std=std, inplace=inplace)
  File "/home/pyler/Python/envs/torch/lib/python3.8/site-packages/torchvision/transforms/functional_tensor.py", line 959, in normalize
    tensor.sub_(mean).div_(std)
RuntimeError: The size of tensor a (240) must match the size of tensor b (3) at non-singleton dimension 1
</code></pre>

username: ptrblck
cooked: <p>Based on this error:</p>
<pre><code class="lang-python">RuntimeError: The size of tensor a (240) must match the size of tensor b (3) at non-singleton dimension 1
</code></pre>
<p>I would guess you might be passing the input tensors in a channels-last format while channels first <code>[batch_size, channels, height, width]</code> is expected. Could you check if this is the case?</p>

username: yakhyo
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your answer but it could not resolve my issue.<br>
I only changed the dataset and some of its parameters. But the same problem still exists. Original code  for kinetics dataset:</p>
<pre><code class="lang-auto">        dataset = torchvision.datasets.Kinetics(
            args.data_path,
            frames_per_clip=args.clip_len,
            num_classes=args.kinetics_version,
            split="train",
            step_between_clips=1,
            transform=transform_train,
            frame_rate=args.frame_rate,
            extensions=(
                "avi",
                "mp4",
            ),
            output_format="TCHW",
        )
</code></pre>
<p>I changed it to the following:</p>
<pre><code class="lang-auto">dataset = torchvision.datasets.UCF101(
            args.data_path,
            annotation_path='../../Datasets/UCF-101/annotations',
            frames_per_clip=args.clip_len,
            train=True,
            step_between_clips=1,
            transform=transform_train,
            frame_rate=args.frame_rate,
            output_format="TCHW",
        )
</code></pre>

username: ptrblck
cooked: <p>Print the shape of <code>x</code> in line 26 in <code>/home/pyler/PycharmProjects/res-ufc101/presets.py</code> and check what it’s returning. I would still guess that the memory format might be wrong and thus the transformation fails.</p>

username: yakhyo
cooked: <p>Yes, the problem is coming that transformation <code>x = {Tensor: (16, 240, 320, 3)}}</code><br>
. However I specified the <code>output_format</code> for the dataset as <code>TCHW</code>  while creating a dataset but did not work.</p>

username: ptrblck
cooked: <p>I don’t think the <code>output_format</code> would fix the issue, as the transformation is expected to work on <code>[T, H, W, C]</code> frames as seen in the <a href="https://pytorch.org/vision/main/generated/torchvision.datasets.UCF101.html">docs</a>:</p>
<blockquote>
<p><strong>transform</strong> (<em>callable</em> <em>,</em> <em>optional</em>) – A function/transform that takes in a TxHxWxC video and returns a transformed version.</p>
</blockquote>
<p>If your transformation doesn’t support it, you could permute the data inside the transform via:</p>
<pre data-code-wrap="python"><code class="lang-python">...
transforms.Lambda(lambda x: x.permute(0, 3, 1, 2)),
...
</code></pre>

username: yakhyo
cooked: <p>Thank you so much, it worked for me.</p>