Video Classification using Transfer Learning (ResNet 3D) Pytorch

Chamroukhi: <p>Dear all,<br>
i have .npz files in my dataset which every file represent sequence of image(15frames) “X” and its target “Y”:<br>
video1: [array(img1, img2, img3, …, img10)], [Y1]<br>
video2: [array(img1, img2,img3, …, img10)], [Y2]<br>
…<br>
and i search how to load this custom data with DataLoader if it is possible?<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

a_d: <p>Hey -<br>
This could be done in the following way -</p>
<pre><code class="lang-auto">class VideoDataset(torch.utils.data.Dataset):
    def __init__(self, path_to_npy_files):
        self.video_files = os.listdir(path_to_npy_files)
        '''
       Anything else here
        '''

    def __getitem__(self, idx):
        video_frames = []

        video_file = self.video_files[idx]
        video = numpy.load(video_file)

        image_tensor = torch.from_numpy(video[0]).permute(0, 3, 1, 2) # I am assuming video is of shape D, H, W, C. If not so, please change accordingly
         
        label = torch.tensor(video[1], dtype = torch.long) # I assume you are classifying video frames
        return video_tensor, label    

    def __len__(self):
        return len(self.video_files)

</code></pre>
<p>I am not sure what exactly is in the npz file. From what I understood, npz file stores an array, who’s first entry is an array which contains the images and second is the class, and I have wriiten the dataset accordingly, If this is not the case, then please change accordingly. only the <code>video[0]</code> and <code>video[1]</code> part would change.</p>
<p>The dataloader can be created as follows -</p>
<pre><code class="lang-auto">def GetDataloader(path_to_npy_files, batch_size, num_workers):
    dataset = VideoDataset(path_to_npy_files)
    dataloader = torch.utils.data.DataLoader(dataset = dataset, batch_size=batch_size, num_workers = num_workers, shuffle=True)
</code></pre>
<p>PS - Please excuse any indentation errors, I have directly typed the code here.</p>

Chamroukhi: <p>But i need to use .npz file, each npz file represent (X , label):<br>
it’s correct like this ?</p>
<pre><code class="lang-auto">def npy_loader(path):
    with np.load(path) as train_data:
        train_examples=train_data['x']
        train_labels=train_data['y']
    X = torch.from_numpy(train_examples)
    Y = torch.from_numpy(train_labels)
    return X,Y
    

dataset = datasets.DatasetFolder(
    root='data/npz_file',
    loader=npy_loader,
    extensions='.npz'
)```</code></pre>

a_d: <p>Oh yeah this works too, I forgot you already have an array of frames <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"><br>
So I believe this should work.<br>
are you facing any trouble with this approach ?</p>

Chamroukhi: <p>with ResNet 3D ?<br>
not yet I am in the first task (data preparation)</p>

a_d: <p>I am not familiar with ResNet3D , but if it has 3D convolutions, then the input to the network should be in the shape - <code>[batch, num_channels, num_frames/depth, H, W]</code><br>
that is the shape that would get loaded with my dataloader. I am not so familiar with <code>DatasetFolder</code>.<br>
If this is the shape getting loaded, then you are good to go</p>