How upload sequence of image on video-classification

Ales: <p>Hi I’m new with Pytorch and I want to know how create class dataset that load a sequence of image take on a folder<br>
I need to do this because then I will use LSTM to train my sequence of frame and classificate my video<br>
Thanks to answer</p>

ptrblck: <p>Do you have video data or already images?<br>
In the former case, you could have a look at <a href="https://github.com/NVIDIA/nvvl">nvvl</a> to load sequences from your video.<br>
In the latter case, you could write your own <code>Dataset</code> and return sequences.<br>
I would need a bit more information about your data and use case, e.g. do you have one label for a whole sequence or does each frame have a label? How are the frames separated etc.</p>

Ales: <p>Thank for the answer, my dataset is divided in 3 folder : 1 to training-set, 1 to test-set and another for validation-set, in the training set folder I have 6 different action folder (I divided it because I have to keep labels for dataset and I don’t know how is doing also that) that contains another 25 folder that describe 25 people doing action and in every 25 folder there are sequence frame(.jpg) , now I want to train a LSTM model but I need to load this images together because I need to mantain sequence for my task of video classification<br>
train-----boxing,jogging etc------- person1_boxing-------0001.jpg</p>

ptrblck: <p>Let’s focus first on the training folder and use the same approach to test and eval, if possible.<br>
So in your training folder you have 6 different action folders, which represent different classes.<br>
For example, boxing would be class0, and jogging class1.<br>
In each of these “action (class) folders” you have 25 subfolders with frames from different persons performing the current action.<br>
As far as I understand you don’t want to mix up the frames of different persons, i.e. you would like to get sequences of a single action from a single person. The next sequence might have another single action from another single person.<br>
Is this correct?</p>
<p>EDIT: Do you want each sequence to have the same length, e.g. 10 images?<br>
If so, do you want a sliding window approach, i.e.:</p>
<pre><code class="lang-auto">batch0: box_person0_image0, box_person0_image1, box_person0_image2, ... box_person0_image9
batch1: box_person0_image1, box_person0_image2, box_person0_image3, ... box_person0_image10
</code></pre>
<p>or rather:</p>
<pre><code class="lang-auto">batch0: box_person0_image0, box_person0_image1, box_person0_image2, ... box_person0_image9
batch1: box_person0_image10, box_person0_image11, box_person0_image12, ... box_person0_image19
</code></pre>

Ales: <p>In the folder of for ex. ‘boxing’ I have the folder of different people that describe the same action boxing. In a folder person that playing box I have different frame but in sequence.<br>
I don’t know how load my data in the dataset.py (my class Dataset).Could you show me an example?<br>
I know that I should have the images loaded sequentially, but how can I load them in blocks of images?</p>

Ales: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a9b63933a7bc84ebba2278f374fb55c060ce1f1d.png" data-download-href="https://discuss.pytorch.org/uploads/default/a9b63933a7bc84ebba2278f374fb55c060ce1f1d" title="10"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a9b63933a7bc84ebba2278f374fb55c060ce1f1d_2_624x500.png" alt="10" data-base62-sha1="odl6trNsUoLqPBIpCLnBTDgveix" width="624" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a9b63933a7bc84ebba2278f374fb55c060ce1f1d_2_624x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/a/a9b63933a7bc84ebba2278f374fb55c060ce1f1d_2_936x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/a9b63933a7bc84ebba2278f374fb55c060ce1f1d.png 2x" data-dominant-color="333333"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">10</span><span class="informations">996×798 62.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Is that correct?</p>

ptrblck: <p>To create a code sample I would need to know some more information mentioned in my last post.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="24865">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>As far as I understand you don’t want to mix up the frames of different persons, i.e. you would like to get sequences of a single action from a single person. The next sequence might have another single action from another single person.<br>
Is this correct?</p>
</blockquote>
</aside>
<p>Also do you want the sliding window approach or neighboring windows?</p>

Ales: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="24865">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>EDIT: Do you want each sequence to have the same length, e.g. 10 images?<br>
If so, do you want a sliding window approach, i.e.:</p>
</blockquote>
</aside>
<p>I’m sorry, I explained myself badly, I’m interested only in associating with each frame sequence the correct labels like ‘running’ etc …<br>
I don’t care if individual people are recognized. I just need the people folder to keep the frame block single.<br>
I’m only interested in processing frames in sequence to allow the algorithm to recognize the ‘running’ action for example.<br>
I think is better a Sliding window approach.</p>

ptrblck: <p>Assuming your folder structure looks like this:</p>
<pre><code class="lang-auto">root/
    - boxing/
        -person0/
            -image00.png
            -image01.png
            - ...
        -person1
            - image00.png
            - image01.png
            - ...
    - jogging
        -person0/
            -image00.png
            -image01.png
            - ...
        -person1
            - image00.png
            - image01.png
            - ...
</code></pre>
<p>You could first get all image paths and the corresponding target.<br>
Then we would have to take care of the invalid indices, i.e. images from different persons, as this might be problematic for the training.<br>
Using a sampler, we can get all valid indices for the current sequence length.<br>
Here is a code sample I adapted to your use case:</p>
<pre><code class="lang-python">
import os
import glob

import torch
from torch.utils.data import Dataset, DataLoader
import torchvision.transforms as transforms

from PIL import Image


class MySampler(torch.utils.data.Sampler):
    def __init__(self, end_idx, seq_length):        
        indices = []
        for i in range(len(end_idx)-1):
            start = end_idx[i]
            end = end_idx[i+1] - seq_length
            indices.append(torch.arange(start, end))
        indices = torch.cat(indices)
        self.indices = indices
        
    def __iter__(self):
        indices = self.indices[torch.randperm(len(self.indices))]
        return iter(indices.tolist())
    
    def __len__(self):
        return len(self.indices)


class MyDataset(Dataset):
    def __init__(self, image_paths, seq_length, transform, length):
        self.image_paths = image_paths
        self.seq_length = seq_length
        self.transform = transform
        self.length = length
        
    def __getitem__(self, index):
        start = index
        end = index + self.seq_length
        print('Getting images from {} to {}'.format(start, end))
        indices = list(range(start, end))
        images = []
        for i in indices:
            image_path = self.image_paths[i][0]
            image = Image.open(image_path)
            if self.transform:
                image = self.transform(image)
            images.append(image)
        x = torch.stack(images)
        y = torch.tensor([self.image_paths[start][1]], dtype=torch.long)
        
        return x, y
    
    def __len__(self):
        return self.length


root_dir = './video_data_test/'
class_paths = [d.path for d in os.scandir(root_dir) if d.is_dir]

class_image_paths = []
end_idx = []
for c, class_path in enumerate(class_paths):
    for d in os.scandir(class_path):
        if d.is_dir:
            paths = sorted(glob.glob(os.path.join(d.path, '*.png')))
            # Add class idx to paths
            paths = [(p, c) for p in paths]
            class_image_paths.extend(paths)
            end_idx.extend([len(paths)])

end_idx = [0, *end_idx]
end_idx = torch.cumsum(torch.tensor(end_idx), 0)
seq_length = 10

sampler = MySampler(end_idx, seq_length)
transform = transforms.Compose([
    transforms.Resize((32, 32)),
    transforms.ToTensor()
])

dataset = MyDataset(
    image_paths=class_image_paths,
    seq_length=seq_length,
    transform=transform,
    length=len(sampler))

loader = DataLoader(
    dataset,
    batch_size=1,
    sampler=sampler
)

for data, target in loader:
    print(data.shape)
</code></pre>
<p>If you use the <code>Dataset</code> without the provided sampler, you will get invalid sequences, e.g. one part might come from person0 while the other from person1.</p>

a.par.s: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="24865">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>neighboring windows</p>
</blockquote>
</aside>
<p>Would you please provide the code with neighboring windows?</p>

ravitejajnv: <p>Say my sequence length is 10, and say one of my video is of length 55 frames, I’d get 5 batches of 10 frames each, but how are the last 5 frames handled.</p>
<p>Also can someone point me out to detailed explanation of sampler?</p>

ptrblck: <p>Note that the posted code was for a very specifc use case, where different actions as well as persons were used.<br>
Are you using a similar work flow or how is your data structured?</p>

ravitejajnv: <p>Yeah, a very similar work flow. I’m using a CNN + LSTM architecture with this kind of file structure. This dataloder does get me frames in sequence, but my doubt was, as I explained what if I’ve some 33 frames and my sequence length is 10. This sequence length yields me three sets of 10 frames each, but the last three frames are they discarded or repeated?</p>
<p>Also can you please point me towards some more detailed and up to date explanation of the sampler.</p>
<pre><code class="lang-auto">train/
    - action1/
        -vid1/
            -image00.png
            -image01.png
            - ...
        -vid2
            - image00.png
            - image01.png
            - ...
    - action2
        -vid1/
            -image00.png
            -image01.png
            - ...
        -vid2
            - image00.png
            - image01.png
            - ...
val/
    - action1/
        -vid1/
            -image00.png
            -image01.png
            - ...
        -vid2
            - image00.png
            - image01.png
            - ...
    - action2
        -vid1/
            -image00.png
            -image01.png
            - ...
        -vid2
            - image00.png
            - image01.png
            - ...
</code></pre>

ravitejajnv: <p>Is it because we are discarding a few indices when we do</p>
<pre><code class="lang-auto">for i in range(len(end_idx)-1):
            start = end_idx[i]
            end = end_idx[i+1] - seq_length
            print(start, end)
            indices.append(torch.arange(start, end))
        indices = torch.cat(indices)
        self.indices = indices
</code></pre>
<p>since if I’ve 55 frames in action1-&gt;vid1 then end_idx would be [0, 55, …] so when appending indices we append everything between 0-45, in indices.append(torch.arange(start, end)), because my end is 55-seq_length = 45, and thus I’m discarding last few frames, and thus my indices would only have [0,1,…,44, 55, 56, …]</p>
<p>but what happens when <strong>geitem</strong> , got an index of 44. then list(range(start , end)) would give [44, 45, 46, 47, 48, 49, 50, 51, 52, 53], but these indices are not there in self.indices in the sampler. but image_path = self.image_paths[i][0], we would get the image from the next video, right, because here 45 means the 46th entry in the image_paths folder?</p>
<pre><code class="lang-auto">def __getitem__(self, index):
        start = index
        end = index + self.seq_length
        print('Getting images from {} to {}'.format(start, end))
        indices = list(range(start, end))
        images = []
        for i in indices:
            image_path = self.image_paths[i][0]
            image = Image.open(image_path)
            if self.transform:
                image = self.transform(image)
            images.append(image)
        x = torch.stack(images)
        y = torch.tensor([self.image_paths[start][1]], dtype=torch.long)
        
        return x, y
</code></pre>

ravitejajnv: <p>I got it. Sorry. Since we didn’t append 45-54 in our indices, when we give list(range(start , end)) which would give [44, 45, 46, 47, 48, 49, 50, 51, 52, 53] are all still valid frames.</p>

Ajmunnisa_KP: <p>hi, i have to give a set of 8 video frames as input to the deep learning model. Can I do this in a similar way.?  The output of 1st convolutional layer has to be of size  (8*[112<em>112</em>64]). Can you hep me. I am new in this field</p>

raivokoot: <aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="15" height="15">
      <a href="https://github.com/RaivoKoot/Video-Dataset-Loading-Pytorch" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars.githubusercontent.com/u/29647087?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="59" height="59">

<h3><a href="https://github.com/RaivoKoot/Video-Dataset-Loading-Pytorch" target="_blank" rel="noopener nofollow ugc">RaivoKoot/Video-Dataset-Loading-Pytorch</a></h3>


  <p><span class="label1">Generic PyTorch Dataset Implementation for Loading, Preprocessing and Augmenting Video Datasets</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>This will help you load multiple video frames at once.</p>

Goli: <p>Hey <span class="mention">@ptrblck_de</span></p>
<p>Thanks for your help in most of the questions that I read to solve my problem in the past week.</p>
<p>I have a dataset from medical images that are extremely large (each of them almost [8900 * 8700* 3]), therefore we split them manually into some smaller patches ( ~ [300 * 300 * 3] ) with respect to some meannginful medical properties.</p>
<p>Now I have a separate folder for each patient that contains patches that are split from the original image. ( Folder 1 [image 0, image 1 ,…] folder 2 [image 0, image 1 ,… ])</p>
<p>Now I want to build a data loader and then pass my input to a CNN, but  I confused after reading a lot of questions here.</p>
<p>Would you please help me with these questions?</p>
<ol>
<li>I should use ImageDataFolder or iterative Dataloder like you write here? Your above code can load my data?</li>
<li>my patches are not in the exact same size, I should transform them after load in <em>init</em> function?</li>
</ol>
<p>Many thanks for your help in advance.</p>

ptrblck: <ol>
<li>
<p>I would probably not use the <code>ImageFolder</code> dataset, as it would assign a new class label to each subfolder. If I understand your use case correctly, the folders contain the image patches, so you should implement the loading logic in a custom <code>Dataset</code> instead.</p>
</li>
<li>
<p>You could keep these patches in the current shape and transform them in the <code>__getitem__</code> of your custom <code>Dataset</code>. Of course, if you could recreate these patches in a constant shape, you could save the processing time during loading, but it depends on your actual use case and if you want to rerun the offline data creation step.</p>
</li>
</ol>

Goli: <p>So you think that I should define iterative data loader?<br>
How can I reconstruct the meaningful images from input images?<br>
How do I separate patches from each other? For example patches of image 1 from image 2 and so on …</p>