Behavioral classifier using skeletal features

ahmed: <p>I am using the Keypoint RCNN to predict keypoints. Using those keypoints, I am able to create a skeletal for detected objects (please see example in image below). What I want to do now is collect the keypoints of the detected object over a number of frames. I want to then classify these frames/keypoints as whether the pedestrian is going to cross the road or not. I hope my explanation makes sense.</p>
<p>I am thinking to feed in only the keypoints into the model without the corresponding images. I want the network to be dependent on just the keypoints for prediction. Would this be something that could work?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed682750b7fa96b62a59dd08c3d96eb1246fabdb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ed682750b7fa96b62a59dd08c3d96eb1246fabdb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed682750b7fa96b62a59dd08c3d96eb1246fabdb_2_690x388.jpeg" alt="image" data-base62-sha1="xScj7SOgVd5xkNEd0XEngvLITEv" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed682750b7fa96b62a59dd08c3d96eb1246fabdb_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed682750b7fa96b62a59dd08c3d96eb1246fabdb.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed682750b7fa96b62a59dd08c3d96eb1246fabdb.jpeg 2x" data-dominant-color="8A888A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">960×540 110 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The issue I am having is how to feed the list of keypoints into the model. I have some experience working with image classification, but not with 3D inputs (i.e. the list of keypoints). Do I use <code>torch.nn.Conv3d</code> for this task?</p>
<p>Any advise would be greatly appreciated.</p>

ahmed: <p>I made a dummy model just to have a play around with</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv3d(in_channels=1, out_channels=64, kernel_size=(1,7,7), stride=(1, 2, 2), padding=(0, 3, 3), bias=False)
        self.bn1 = nn.BatchNorm3d(64, track_running_stats=True)
        self.relu = nn.ReLU(inplace=True)
        self.pool = nn.MaxPool3d(kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1))
        self.conv2 = nn.Conv3d(in_channels=64, out_channels=32, kernel_size=(1,7,7), stride=(1, 2, 2), padding=(0, 3, 3), bias=False)
        self.fc1 = nn.Linear(320, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 1)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = self.bn1(x)
        x = self.relu(x)
        x = self.pool(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = self.pool(x)
        x = x.reshape(x.size(0), -1) 
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
    
net = Net().cuda()

import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2): 

    running_loss = 0.0
    optimizer.zero_grad()
    inputs = torch.stack(kypnts_list, dim=1).unsqueeze(0).to("cuda")
    outputs = net(inputs)
    loss = criterion(inputs.squeeze(0), intentions.cuda())
    loss.backward()
    optimizer.step()
</code></pre>
<p>But when I try to train the model, I get the following error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-184-78c208c097a2&gt; in &lt;module&gt;
     45 
     46     outputs = net(inputs)
---&gt; 47     loss = criterion(inputs.squeeze(0), intentions.cuda())
     48     loss.backward()
     49     optimizer.step()

~\miniconda3\envs\torch_env\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~\miniconda3\envs\torch_env\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
    914     def forward(self, input, target):
    915         return F.cross_entropy(input, target, weight=self.weight,
--&gt; 916                                ignore_index=self.ignore_index, reduction=self.reduction)
    917 
    918 

~\miniconda3\envs\torch_env\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2019     if size_average is not None or reduce is not None:
   2020         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2022 
   2023 

~\miniconda3\envs\torch_env\lib\site-packages\torch\nn\functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
-&gt; 1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1841     else:
   1842         # dim == 3 or dim &gt; 4

RuntimeError: 1only batches of spatial targets supported (non-empty 3D tensors) but got targets of size: : [1]
</code></pre>
<p>The label that I am feed in here is <code>0</code>, where <code>0</code> means the pedestrian will not be crossing and <code>1</code> where the pedestrian will cross.</p>

ptrblck: <p>For a binary classification use case with <code>nn.CrossEntropyLoss</code> the number of output values should be 2, so you would need to use <code>self.fc3 = nn.Linear(84, 2)</code>.<br>
Alternatively, you could keep the single output unit and use <code>nn.BCEWithLogitsLoss</code> as the criterion.</p>

ahmed: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you very much for your suggestion. I will have a play around with it and see what sort of results I get.</p>

ahmed: <p>I have tried to implement the suggestions that <a class="mention" href="/u/ptrblck">@ptrblck</a> made in the previous post, however, the loss for my model is <code>0.00</code> from start to finish and doesn’t change at all during the training phase.</p>
<p>This is my model and training phase:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv3d(in_channels=1, out_channels=64, kernel_size=(1,7,7), stride=(1, 2, 2), padding=(0, 3, 3), bias=False)
        self.bn1 = nn.BatchNorm3d(64, track_running_stats=True)
        self.relu = nn.ReLU(inplace=True)
        self.pool = nn.MaxPool3d(kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1))
        self.conv2 = nn.Conv3d(in_channels=64, out_channels=32, kernel_size=(1,7,7), stride=(1, 2, 2), padding=(0, 3, 3), bias=False)
        self.fc1 = nn.Linear(64000, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 2)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = self.bn1(x)
        x = self.relu(x)
        x = self.pool(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = self.pool(x)
        x = x.reshape(x.size(0), -1) 
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        x = x.unsqueeze(dim=1)
        return x
    
net = Net().to("cuda")

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(10): 
    running_loss = 0.0
    optimizer.zero_grad()
    
    x = [torch.mean(i,0) for i in kypnts_list]
    x1 = (torch.stack(x, dim=0))
    x1 = F.pad(x1, pad=(0, 0, 0, 0, 0, 300))
    x1 = (x1.unsqueeze(dim=0)).unsqueeze(1)
    x1 = x1.to("cuda")

    labels = torch.tensor(intentions_data[0]['intent'][0]).to("cuda")
    outputs = net(x1)
    loss = criterion(outputs.squeeze(dim=1), labels.long())
    loss.backward()
    optimizer.step()
        
    running_loss += loss.item()
    print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000))
    running_loss = 0.0

print('Finished Training')
</code></pre>
<p>I have a feeling it’s due to my generic model that I made, but I thought I would see some attempt at the model “learning” (i.e. some changes in <code>loss</code> values)</p>

ahmed: <p>I just noticed I made a mistake for <code>self.fc3 = nn.Linear(84, 1)</code>, where I forgot to change it to <code>self.fc3 = nn.Linear(84, 2)</code>, and it seems to be training now.</p>