Dimension issue in loss function with per frame logits [I3D]

username: grizzlycoder
cooked: <p>Hello,</p>
<p>I am working on trying to run the training module from (traini3d.py) : <a href="https://github.com/piergiaj/pytorch-i3d" rel="noopener nofollow ugc">https://github.com/piergiaj/pytorch-i3d</a> on my own dataset of videos (they do it for Charades).</p>
<p>With some help, I have successfully gotten a Dataset module functioning to extract features. Now, in the training module,  I have a dimension issue here:</p>
<pre><code class="lang-auto">                per_frame_logits = i3d(inputs)
                # upsample to input size
                print(per_frame_logits)
                per_frame_logits = F.upsample(per_frame_logits, t, mode='linear')
                # compute localization loss
                loc_loss = F.binary_cross_entropy_with_logits(per_frame_logits, labels)
</code></pre>
<p>The labels are of dimension 1 but the per frame logits are of dimension 1, num_classes, frames per second (so in my case 1,2,24 [just for testing purposes I am using 2 classes]. I am unsure as to how to change the dimension of the per frame logits. Here is the Dataset module I use:</p>
<pre><code class="lang-auto">import numpy

from pathlib import Path
import torch.utils.data as data_utl

class customVideoDataset(data_utl.Dataset):
    def __init__(self, path, frame_count, transforms = None):
        self.videos = []
        self.labels = []
        self.transforms = transforms
        self.frames = frame_count
        folder = Path(path)
        for label in sorted(os.listdir(folder)):
            for fname in os.listdir(os.path.join(folder, label)):
                self.videos.append(os.path.join(folder, label, fname))
                self.labels.append(label)

        #for fname in sorted(os.listdir(folder)):
        #  self.videos.append(os.path.join(folder, fname))
        self.label2index = {label: index for index, label in enumerate(sorted(set(self.labels)))}
        self.index2label = {index: label for index, label in enumerate(sorted(set(self.labels)))}
        self.label_array = [self.label2index[label] for label in self.labels]

    def __getitem__(self, idx):
        video = cv2.VideoCapture(self.videos[idx])
        stacked_frames = numpy.empty(shape=(self.frames, 224, 224, 3),
                                     dtype=numpy.dtype('float32'))  # as frame would have shape h,w,channels
        frame_count = 0
        while video.isOpened() and frame_count&lt;self.frames:
            ret, frame = video.read()
            if not ret:
                break
            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            frame = cv2.resize(frame, (224, 224))
            stacked_frames[frame_count] = frame
            frame_count += 1
        video.release()
        if self.transforms:
          stacked_frames = self.transforms(stacked_frames)
        stacked_frames = torch.from_numpy(stacked_frames.transpose((3, 0, 1, 2)))
        return stacked_frames, self.label_array[idx]

    def __len__(self):
        length = len(self.videos)
        return length
    
    def namer(self):
      return self.index2label
      
    def get_vids(self):
      names = []
      for name in self.videos:
        names.append(name)
      return names
</code></pre>
<p>Does someone know where the dimension issue arises from?</p>

username: ptrblck
cooked: <p>Depending on your use case you could either change the model to output a prediction for the whole sequence or alternatively change your data loading logic to return labels for each time step in the sequence.</p>
<p>Currently you are trying to mix both approaches, i.e. you are using a single label for a model output for a sequence of frames.</p>

username: grizzlycoder
cooked: <p>Okay i see! This makes sense, thank you!</p>