Use convolutional model (S3D) for training multi-class binary for video classification?

Lukens: <p>I am using this model:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/kylemin/S3D">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/kylemin/S3D" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/8d9753af1ccc41f18d6fdd2b9bb6a52c0ffd73c4b846f1967d0a5710977d4adf/kylemin/S3D" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/kylemin/S3D" target="_blank" rel="noopener nofollow ugc">GitHub - kylemin/S3D: Release of the pretrained S3D Network in PyTorch (ECCV...</a></h3>

  <p>Release of the pretrained S3D Network in PyTorch (ECCV 2018) - GitHub - kylemin/S3D: Release of the pretrained S3D Network in PyTorch (ECCV 2018)</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I should train the model to recognize a series of (volcanic) videos which can belong to 8 different classes (all or only some of them) For example:</p>
<p>-video 1 belongs to classes 1,4,6<br>
-video 2 belongs to class 3<br>
-video 3 belongs to classes 2,5<br>
etc…<br>
Each class has a binary label 1 (belongs) or 0 (does not belong).</p>
<p>I thought of calculating the loss for each of these 8 classes by specifying in each of these the weight of that specific class.<br>
This is my class_weight = [5.56, 51.5, 3.67, 1.39, 5.36, 2.82, 5.18, 1.76]</p>
<pre><code class="lang-auto">def binary_loss_fn(outputs, targets):
      t1, t2, t3, t4, t5, t6, t7, t8 = targets
      o1, o2, o3, o4, o5, o6, o7, o8 = outputs
      
      l1 = nn.BCEWithLogitsLoss(pos_weight=class_weight[0], reduction='none')(o1, t1)
      l2 = nn.BCEWithLogitsLoss(pos_weight=class_weight[1], reduction='none')(o2, t2)
      l3 = nn.BCEWithLogitsLoss(pos_weight=class_weight[2], reduction='none')(o3, t3)
      l4 = nn.BCEWithLogitsLoss(pos_weight=class_weight[3], reduction='none')(o4, t4)
      l5 = nn.BCEWithLogitsLoss(pos_weight=class_weight[4], reduction='none')(o5, t5)
      l6 = nn.BCEWithLogitsLoss(pos_weight=class_weight[5], reduction='none')(o6, t6)
      l7 = nn.BCEWithLogitsLoss(pos_weight=class_weight[6], reduction='none')(o7, t7)
      l8 = nn.BCEWithLogitsLoss(pos_weight=class_weight[7], reduction='none')(o8, t8)
      return (l1 + l2 + l3 + l4 + l5 + l6 + l7 + l8) / 8
</code></pre>
<p>My dataset is structured as follows:<br>
210 videos of 300 frames each, of these I took one every five frames and combined in 29 different combinations each of 32 frames:<br>
frames 1,2,3,4,5,6 … 300 → frames 1,6,11,16,21 … (total 60 frames):</p>
<ul>
<li>first combination: first 32 frames (1-32)</li>
<li>second combination: second 32 frames shifted by 1 (2-33)</li>
<li>third combination: third 32 frames shifted by 1 (3-34)<br>
etc…</li>
</ul>
<p>total array 210x29x32</p>
<p>The input of the s3d model are these 32 frames concatenated with pytorch and resized:<br>
<strong>torch.Size ([2, 3, 32, 224, 224])</strong><br>
associated with a list of 8 targets each relating to a class (1 or 0)</p>
<pre><code class="lang-auto">def fit(model, train_dataloader):
    print('Training')
    model.train()
    train_running_loss = 0.0
    train_running_correct = 0
    for i, data in tqdm(enumerate(train_dataloader), total=int(len(train_data)/train_dataloader.batch_size)):
        #data, target, path = data[0].to(device), data[1].to(device), data[2]
        # extract the features and labels
        video = data['video'].to(device)
        target1 = data['label1'].to(device)
        target2 = data['label2'].to(device)
        target3 = data['label3'].to(device)
        target4 = data['label4'].to(device)
        target5 = data['label5'].to(device)
        target6 = data['label6'].to(device)
        target7 = data['label7'].to(device)
        target8 = data['label8'].to(device)
        path = data['path']


        optimizer.zero_grad()
        outputs = model(video)
        targets = (target1, target2, target3, target4, target5, target6, target7, target8)
        outputs = torch.sigmoid(outputs)
        loss = binary_loss_fn(outputs, targets)
          
        train_running_loss += loss.item()

        preds = torch.tensor([[1 if i &gt;= 0.5 else 0 for i in j] for j in outputs]).to(device)
        train_running_correct = (preds == target).sum().item()

        loss.backward()
        optimizer.step()
        
    train_loss = train_running_loss/len(train_dataloader.dataset)
    train_accuracy = 100. * train_running_correct/len(train_dataloader.dataset)
    return train_loss, train_accuracy
</code></pre>
<p>This is my fit function where I am going to pass my targets and my model’s output inside binary_loss_fn. I specified the number of classes when I initialize the model<br>
<code>model = S3D (8) .to (device)</code><br>
<strong>But i can’t understand what kind of output I get on these s3d because i can’t calculate individually my 8 loss functions.</strong></p>
<p>This is a type model’s output with batch equal to 2:</p>
<pre><code class="lang-auto">torch.tensor ([
[0.4150, 0.6833, 0.4165, 0.4718, 0.4183, 0.5799, 0.4773, 0.3026],
[0.4923, 0.5274, 0.4894, 0.5042, 0.4440, 0.4746, 0.4580, 0.4449]
])
</code></pre>
<p>If it can be useful this is my dataset function</p>
<pre><code class="lang-auto">class ImageDataset(Dataset):
    def __init__(self, images, labels=None):
        self.X = images
        self.y = labels
         
    def __len__(self):
        rows = len(self.X)
        columns = len(self.X[0])
        total_length = rows * columns
        return (total_length)

    def __getitem__(self, i):
        col = len(self.X[0])
        a = i//col
        b = (i%col)-1
        image = self.X[a][b]
        tensor_sequence = []
        for idx in range(0, len(image)):  
          path_img = image[idx]
          tensor_img = tensor[path_img]
          tensor_sequence.append(tensor_img)
        video = torch.cat(tensor_sequence, 1)
        x = path_img.split("data/videos/")
        path = x[1].split(".avi/EMOT")
        path=path[0]
        col = len(self.y[0])
        a1 = i//col
        b1 = (i%col)-1
        label = self.y[a1][b1]
        target = [int(i) for i in label[0].split(",")]
        
        label1 = torch.tensor(target[0], dtype=torch.float32)
        label2 = torch.tensor(target[1], dtype=torch.float32)
        label3 = torch.tensor(target[2], dtype=torch.float32)
        label4 = torch.tensor(target[3], dtype=torch.float32)
        label5 = torch.tensor(target[4], dtype=torch.float32)
        label6 = torch.tensor(target[5], dtype=torch.float32)
        label7 = torch.tensor(target[6], dtype=torch.float32)
        label8 = torch.tensor(target[7], dtype=torch.float32)
        return {
            'video': video,
            'label1': label1,
            'label2': label2,
            'label3': label3,
            'label4': label4,
            'label5': label5,
            'label6': label6,
            'label7': label7,
            'label8': label8,
            'path': path
        }

#Defining the Training and Validation Data Loaders
train_data = ImageDataset(xtrain, ytrain)
val_data = ImageDataset(xval, yval)
test_data = ImageDataset(xtest, ytest)

# dataloaders
trainloader = DataLoader(train_data, batch_size=batch_size, shuffle=True)
valloader = DataLoader(val_data, batch_size=batch_size, shuffle=False)
testloader = DataLoader(test_data, batch_size=batch_size, shuffle=False)
</code></pre>
<p>This is my csv file where for each image path I have its associated binarized target:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9f35b97da5eaa6692b31b13fd58e97f15440266.png" alt="image" data-base62-sha1="qwZIRQiY4MdrJ0pQ1kWbRJBAYOa" width="208" height="59"></p>
<p>How could I fix this? Thank you</p>

ptrblck: <p>The output looks as if it’s containing the logits for each class in the shape <code>[batch_size, nb_classes]</code>. I’m unsure why you would need the 8 separate loss functions, as you should be able to provide a tensor containing the weights for all classes to <code>pos_weight</code>.</p>

Lukens: <p>The output seems to have the right shape but when I pass it to the binary_loss_fn function it expects 8 elements (o1, o2, o3, o4, o5, o6, o7, o8 = outputs) but finds 2 because the first dimension is referred to the batch size. However, can I use an array as a parameter on pos_weight using only a loss function? I hope I have understood correctly</p>

ptrblck: <p>Yes, you can pass a tensor in the shape <code>[nb_classes]</code> as the argument to <code>pos_weight</code>.<br>
In case you want to split the output in <code>dim1</code>, you could use <code>output.split(1, dim=1)</code>.</p>

Lukens: <p>Ok perfect this is great and i solved using <code>output.split(1, dim=1)</code> but there is another problem:</p>
<aside class="quote no-group" data-username="Lukens" data-post="1" data-topic="126938">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lukens/48/40355_2.png" class="avatar"> Lukens:</div>
<blockquote>
<pre><code class="lang-auto">targets = (target1, target2, target3, target4, target5, target6, target7, target8)
outputs = torch.sigmoid(outputs)
loss = binary_loss_fn(outputs, targets)
</code></pre>
</blockquote>
</aside>
<p>size of outuputs and targets is not the same when i call the function binary_loss_fn, I read this error:</p>
<p><code>ValueError: Target size (torch.Size([2])) must be the same as input size (torch.Size([2, 1]))</code></p>
<p>this is my targets:<br>
<code>(tensor([0., 0.], device='cuda:0'), tensor([0., 0.], device='cuda:0'), tensor([0., 0.], device='cuda:0'), tensor([0., 0.], device='cuda:0'), tensor([0., 0.], device='cuda:0'), tensor([0., 0.], device='cuda:0'), tensor([0., 1.], device='cuda:0'), tensor([1., 0.], device='cuda:0'))</code></p>
<p>this is my outputs after the split:</p>
<pre><code class="lang-auto">(tensor([[0.5201],
        [0.5200]], device='cuda:0', grad_fn=&lt;SplitBackward&gt;), tensor([[0.3892],
        [0.3740]], device='cuda:0', grad_fn=&lt;SplitBackward&gt;), tensor([[0.4612],
        [0.4714]], device='cuda:0', grad_fn=&lt;SplitBackward&gt;), tensor([[0.5387],
        [0.5210]], device='cuda:0', grad_fn=&lt;SplitBackward&gt;), tensor([[0.5536],
        [0.5366]], device='cuda:0', grad_fn=&lt;SplitBackward&gt;), tensor([[0.4580],
        [0.4908]], device='cuda:0', grad_fn=&lt;SplitBackward&gt;), tensor([[0.5311],
        [0.5242]], device='cuda:0', grad_fn=&lt;SplitBackward&gt;), tensor([[0.5169],
        [0.5242]], device='cuda:0', grad_fn=&lt;SplitBackward&gt;))
</code></pre>

ptrblck: <p>You would either have to add <code>dim1</code> to the target via <code>unsqueeze(1)</code> or remove <code>dim1</code> in the output.<br>
I would prefer the former, but both should work.</p>

Lukens: <p>thank you, its work!! <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"> <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>

Lukens: <p>About the loss result of my function I have something like that:<br>
<code>tensor([1.0996, 0.9478], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)</code></p>
<p>I have a tensor as large as my chosen batch (2), but I should have a single value to calculate the loss or am I wrong?</p>
<p>It might work in this way? <code>loss = torch.sum(loss)</code></p>

ptrblck: <p>Yes, the <code>sum</code> or <code>mean</code> could should work.</p>
<aside class="quote no-group" data-username="Lukens" data-post="8" data-topic="126938">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lukens/48/40355_2.png" class="avatar"> Lukens:</div>
<blockquote>
<p>I have a tensor as large as my chosen batch (2), but I should have a single value to calculate the loss or am I wrong?</p>
</blockquote>
</aside>
<p>Since you are using <code>reduction='none'</code>, an unreduced loss is expected for each criterion, so you could use the default <code>mean</code> reduction instead.</p>