Is there an off-the-shelf video data loader to use?

username: zhuyi490
cooked: <p>Hi all,</p>
<p>I was wondering is there an off-the-shelf video data loader to use? I mean, if the structure of the dataset is like:</p>
<p><code>A root folder containing multiple class directories, and each class contain multiple video clip directories, and each video contain a list of continuous frames.</code></p>
<p>This dataset structure is quite common for most academic video dataset. If there is no such data loader, could anyone point out which files should I look at and modify? Thank you very much.</p>

username: smth
cooked: <p>you can copy over this file and modify the image loading to video loading:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py" target="_blank" rel="nofollow noopener">pytorch/vision/blob/master/torchvision/datasets/folder.py</a></h4>
<pre><code class="lang-py">import torch.utils.data as data

from PIL import Image
import os
import os.path

IMG_EXTENSIONS = ['.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm']


def is_image_file(filename):
    """Checks if a file is an image.

    Args:
        filename (string): path to a file

    Returns:
        bool: True if the filename ends with a known image extension
    """
    filename_lower = filename.lower()
    return any(filename_lower.endswith(ext) for ext in IMG_EXTENSIONS)
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: chihyaoma
cooked: <p>As <a class="mention" href="/u/smth">@smth</a> pointed out, you should start with <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py" rel="nofollow noopener">folder.py</a></p>
<hr>
<p>Below are the major steps that I think you might need to go through.</p>
<ol>
<li>
<strong>make_dataset()</strong><br>
You probably need to make some changes to the <em>make_dataset()</em> function, which is basically listing all the video frame files. If you need to group a number of video frames and tie them with a video, you probably want to do that in here (when listing the video frames)</li>
<li>
<strong>_ _ getitem _ _</strong><br>
The second part will be how you collect the items. This can be straight forward or can be quite complicated. The easiest way to do is: given a video id (obtained by using <em>index</em> and the list you just collected in step one), grab all the video frames associated with that particular video id. Note that, in this way, your data loading time will be considerably longer, and your system memory might explode because you are loading tens or hundreds of images per video into system memory before you pass it to GPU for training.</li>
<li>
<strong>YourOwn_collate_fn()</strong><br>
By default, the items collected in _ _ getitem _ _ will be combined into batch by <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L82" rel="nofollow noopener">default_collate</a>. This might no longer be the case when you have a list or stack of video frames. The easiest way I can think of is to create your own collate function and reformating the data you collected so that you can call default_collate within your own collate function.</li>
</ol>
<hr>
<p>These are just some of my thoughts and I am also very interested in having an off-the-shelf video dataloader and having some example codes so that people can refer to.</p>

username: pipehappy1
cooked: <p>Any update on this one? It would be nice to have a built-in video data loader instead of converting them into images first…</p>

username: kvrd18
cooked: <p>Are there any updates on this? Quick data reading is essential for video processing.</p>

username: rwightman
cooked: <p>It would certainly be a nice to have, but it’s actually quite challenging to do this right and do it efficiently so you’re not spinning 16 cores at 100% feeding one model and consuming 100s of GB of memory. I have a lot of experience storing, streaming video with other synchronized streams in a past life and it’s a rather big pain in the ass to say the least.</p>
<p>You would ideally want to leverage the video decoders on the Nvidia card, and even more ideally keep the decoded buffers on the card, a zero-copy to Tensors ideally. It looks like Nvidia has done the basics of something like that with their DeepStream SDK but I have’t looked closely enough to see the details, they may have made shortcuts given that they claim to be feeding standard image models that don’t have a time axis.</p>
<p>Aside from leveraging hardware decoding, the timestamping of video, and corresponding streams that you may want to keep in synch such as audio, text, other metadata can be quite challenging. At a codec level, advance video codecs have their frames out of order, the amount of video you have to decode to get a sequence of frames in the correct order varies depending on codec and chosen encode parameters. It can be quite significant which could cause problems with buffering and memory usage. Most surveillance systems get around this by enforcing maximum keyframe intervals based on a users desired latency to storage usage tradeoff, but typical media from streaming sources or disks like blu-rays, Netflix, etc have very long keyframe intervals to reduce bandwidth since random access is minimal and people are okay waiting a few seconds for the stream to resume after changing position. If you are doing this constantly to get fixed time windows, it would be a huge overhead.</p>
<p>Then you’d have to figure out how to represent your targets, what are they? how are they represented relative to a stream of images with a time axis. I assume there would be a lot of possible variety and different wants here.</p>
<p>And I thought about all this because I looked at doing it, but it will have to wait until a possible project is viable that requires this as it is a significant undertaking <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: asif
cooked: <p>I am trying to make a video dataloader in the <a href="https://github.com/pytorch/tutorials/issues/78" rel="nofollow noopener">https://github.com/pytorch/tutorials/issues/78</a> approach but I am getting this error</p>
<p>Traceback (most recent call last):<br>
File “<a href="http://dataload.py" rel="nofollow noopener">dataload.py</a>”, line 59, in <br>
for batch_idx, (data,target) in enumerate(train_loader):<br>
File “/home/acp15maj/.conda/envs/pytorch/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 201, in <strong>next</strong><br>
return self._process_next_batch(batch)<br>
File “/home/acp15maj/.conda/envs/pytorch/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 221, in _process_next_batch<br>
raise batch.exc_type(batch.exc_msg)<br>
RuntimeError: Traceback (most recent call last):<br>
File “/home/acp15maj/.conda/envs/pytorch/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 40, in _worker_loop<br>
samples = collate_fn([dataset[i] for i in batch_indices])<br>
File “/home/acp15maj/.conda/envs/pytorch/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 109, in default_collate<br>
return [default_collate(samples) for samples in transposed]<br>
File “/home/acp15maj/.conda/envs/pytorch/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 109, in <br>
return [default_collate(samples) for samples in transposed]<br>
File “/home/acp15maj/.conda/envs/pytorch/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 95, in default_collate<br>
return torch.stack([torch.from_numpy(b) for b in batch], 0)<br>
File “/home/acp15maj/.conda/envs/pytorch/lib/python3.5/site-packages/torch/functional.py”, line 64, in stack<br>
return torch.cat(inputs, dim)<br>
RuntimeError: inconsistent tensor sizes at /opt/conda/conda-bld/pytorch_1503968623488/work/torch/lib/TH/generic/THTensorMath.c:2709</p>
<p>My code is</p>
<pre><code>class CustomDataset(Dataset):
    def __init__(self,csv_path,transform=None):
        # TODO
        # 1. Initialize file path or list of file names.
        # full file path is given in the .csv file
        temp_df = pd.read_csv(csv_path)
        self.label=MultiLabelBinarizer()
        self.transform=transform
        self.X_train=temp_df['video_file']
        self.Y_train=self.label.fit_transform(temp_df['label'].to_string().split()).astype(np.float32)

    def __getitem__(self, index):
        # numpy array of video file (50,3,200,112) shape
        video_file=cvstream.get_video(self.X_train[index]) 
        video_tensor=torch.from_numpy(video_file)
        if self.transform is not None:
            video_file=self.transform(video_file)
        target_labels=torch.from_numpy(self.Y_train[index])
        print(target_labels)
        return video_tensor,target_labels

    def __len__(self):
        return len(self.X_train.index)



if __name__ == '__main__':
    # Then, you can just use prebuilt torch's data loader.
    custom_dataset = CustomDataset('trainlist01.csv',transform=None)
    train_loader = torch.utils.data.DataLoader(dataset=custom_dataset,
                                               batch_size=3,
                                               shuffle=True,
                                               num_workers=3
                                                )
    for batch_idx, (data,target) in enumerate(train_loader):
        data, target = data.cuda(async=True), target.cuda(async=True)
</code></pre>
<p>Please pardon me if it is a silly question. <img src="//discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=5" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

username: isalirezag
cooked: <p>how exactly? any good/understanding example?</p>

username: raivokoot
cooked: <p>Late, but this repository right here offers exactly what this thread is asking for: off-the-shelf PyTorch Video Dataset Loading <a href="https://github.com/RaivoKoot/Video-Dataset-Loading-Pytorch" rel="noopener nofollow ugc">https://github.com/RaivoKoot/Video-Dataset-Loading-Pytorch</a></p>