Change model output shape

username: ahmed
cooked: <p>Is there a way that I can have my model output a specific shape? Say if my input (ground truth) has a shape of <code>[15, 3]</code>, however, the output (<code>x_kp</code>) size from the model is <code>[15, 1]</code>.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, num_classes, batch_size):
        super(Model, self).__init__()
        self.conv1 = nn.Conv3d(in_channels=256, out_channels=512, kernel_size=(3, 3, 3), stride=(1, 1, 1))
        self.conv2 = nn.Conv3d(in_channels=512, out_channels=512, kernel_size=(3, 3, 3), stride=(1, 1, 1))
        self.conv3 = nn.Conv3d(in_channels=512, out_channels=512, kernel_size=(3, 3, 3), stride=(1, 1, 1))
        self.conv4 = nn.Conv3d(in_channels=512, out_channels=512, kernel_size=(3, 3, 3), stride=(1, 1, 1))
        self.conv5 = nn.Conv3d(in_channels=512, out_channels=512, kernel_size=(3, 3, 3), stride=(1, 1, 1))
        self.conv6 = nn.Conv3d(in_channels=512, out_channels=512, kernel_size=(3, 3, 3), stride=(1, 1, 1))
        self.conv7 = nn.Conv3d(in_channels=512, out_channels=512, kernel_size=(3, 3, 3), stride=(1, 1, 1))
        self.conv8 = nn.Conv3d(in_channels=512, out_channels=512, kernel_size=(3, 3, 3), stride=(1, 1, 1))
        self.cls_score = nn.Linear(471040, num_classes)
        self.kp_score = nn.Linear(471040, 17*batch_size)
        
    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        x = F.relu(self.conv4(x))
        x = F.relu(self.conv5(x))
        x = F.relu(self.conv6(x))
        x = F.relu(self.conv7(x))
#         x = F.relu(self.conv8(x))
        x = x.flatten(start_dim=1)
        x_label = F.relu(self.cls_score(x))
        x_kp = F.relu(self.kp_score(x))
        return x_label, x_kp
    
model = Model(num_classes=2, batch_size=batch_size).to(device)
</code></pre>
<p>My criterion and loss is calculated using the following:</p>
<pre><code class="lang-auto">pred_label, pred_kp = model(cat_features)

kp_criterion = nn.SmoothL1Loss()
label_criterion = nn.CrossEntropyLoss()

kp_loss = kp_criterion(torch.mean(pred_kp), torch.mean(cat_kp).to(device))
label_loss = label_criterion(pred_label, crossing_labels)
</code></pre>

username: ptrblck
cooked: <p>The output shape of <code>[15, 1]</code> is a bit weird, since it should be <code>[batch_size, 17*batch_size]</code> based on your model definition.</p>
<p>You can define the output shape via the <code>out_features</code> of the linear layer.</p>
<p>That being said, it’s also unusual to define a specific shape relative to the batch size, as the model definition is independent of the batch size in a standard scenario.</p>

username: ahmed
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="96676">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The output shape of <code>[15, 1]</code> is a bit weird, since it should be <code>[batch_size, 17*batch_size]</code> based on your model definition</p>
</blockquote>
</aside>
<p>I’m probably doing something wrong here. But this is my workflow:</p>
<ol>
<li>Extract features <code>features = pose_estimator.backbone(img_list.tensor.float())</code>. The shape of the extracted features is <code>torch.Size([1, 256, 272, 480])</code> via <code>print(features[0].shape)</code>. Each image is added appended to a list called <code>features_list</code>.</li>
<li>Then I concatenate the images using:<br>
<code>cat_features = torch.cat([i[0].cpu() for i in features_list], 0)</code>. The shape is <code>torch.Size([3, 256, 34, 60])</code>, where the <code>3</code> represents that 3 image features were concatenated (i.e. batch size).</li>
<li>Then I apply <code>cat_features = cat_features.transpose(1, 0).unsqueeze(0).to(device)</code> to get in the format I need: <code>torch.Size([1, 256, 3, 272, 480])</code>, where now the batch is <code>1</code> is one as I want the concatenated features to be seen as a single batch by the model, and <code>3</code> represents the depth (i.e. 3 images).</li>
</ol>
<p>I hope that so far it’s all making sense.</p>
<p>What I would like to do is use this image to predict future actions of pedestrians. So, for that, I also obtain <code>keypoints</code> using the <code>keypoint_rcnn model</code>. Each <code>keypoint</code> has a shape of <code>torch.Size([1, 17, 3])</code> and when concatenating the <code>keypoints</code> of the 3 images as I did with the images in <code>step 2</code>, <code>cat_kp = torch.cat(kypnt_instances[i]['pred_keypoints'], 0).to(device)</code>, I get a shape of <code>torch.Size([51, 3])</code>.</p>
<p>When I run my model, <code>pred_label, pred_kp = model(cat_features)</code>, the shape of <code>pred_kp</code> is <code>torch.Size([51, 1])</code>, so when I try to calculate the loss <code>kp_loss = kp_criterion(pred_kp, cat_kp)</code>, I get an error saying the <code>red_kp and cat_kp</code> are not the same size.</p>
<p>I think the issue is, that I want the model to see it as a batch size of 1, but output the keypoints as a batch of 3, 1 for each of the images. Which doesn’t make sense now that I think about it. Is there a way that I can combine/reshape <code>cat_kp</code> so that it is <code>torch.Size([51, 1])</code> instead of <code>torch.Size([51, 3])</code> without losing too much feature quality?</p>
<p>Sorry for the long post and I hope that it all makes some sense.</p>

username: ptrblck
cooked: <p>Based on your description, the error is raised since <code>nn.SmoothL1Loss</code> expects the output and target to have the same shape.<br>
If you want to use a batch size of 1 and use 3 different images with 17 points each, you could make sure the output and target have the shape <code>[batch_size, 3, 17]</code> or you could also flatten the points to <code>[batch_size, 51]</code>. The calculated loss would be the same.</p>
<aside class="quote no-group" data-username="ahmed" data-post="3" data-topic="96676">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/8c91f0/48.png" class="avatar"> ahmed:</div>
<blockquote>
<p>Is there a way that I can combine/reshape <code>cat_kp</code> so that it is <code>torch.Size([51, 1])</code> instead of <code>torch.Size([51, 3])</code> without losing too much feature quality?</p>
</blockquote>
</aside>
<p>I don’t know, as I’m not familiar with your use case.<br>
However, I would recommend to fix the shape issues and keep the model “idea” unchanged for now.</p>

username: ahmed
cooked: <p>Thank you for the quick reply.</p>
<blockquote>
<p>If you want to use a batch size of 1 and use 3 different images with 17 points each, you could make sure the output and target have the shape  <code>[batch_size, 3, 17]</code>.</p>
</blockquote>
<p>Sorry, but probably a silly question, but how would I make sure that the output is <code>[batch_size, 3, 17]</code>? What I mean to say is, how would I change <code>self.kp_score = nn.Linear(66078720, 17, 3)</code> to get the output that I require? I hope that that is what you meant with your suggestion.</p>

username: ahmed
cooked: <blockquote>
<p>you could also flatten the points to `[batch_size, 51]</p>
</blockquote>
<p>So, I followed your advice to flatten the keypoints since I couldn’t figure out how to do the initial suggestion <code>you could make sure the output and target have the shape [batch_size, 3, 17]</code> (probably as misunderstanding on my part for how to do it).</p>
<p>I did it by setting <code>self.kp_score = nn.Linear(66078720, 17*3*3)</code>, where 17 are the keypoints, 3 is number of images and the other 3 is to match the predicted keypoints shape <code>[3, 51]</code>. I flattened <code>cat_kp = cat_kp.flatten(start_dim=0)</code> and <code>pred_kp = pred_kp.flatten(start_dim=0)</code>.</p>
<p>However, there are 2 “issues” that I am having now:</p>
<ol>
<li>The <code>kp_loss</code> for is very high (over 200). Not sure if that’s an issue due to problem number 2, as I can’t begin to train the model.</li>
<li>I get an issue saying the GPU is out of memory. I have an RTX 2080 Ti 11GB. I thought this would be able to handle this. Is it because I am using <code>conv3D</code>, because it works fine to training for detection with a batch size of 8 images.</li>
</ol>

username: ptrblck
cooked: <ol>
<li>
<p>Could you flatten the tensors starting in <code>dim1</code>? This might not change the high loss value (depends on the used criterion), but would most likely fit the expected input shape of <code>[batch_size, *]</code>. I would recommend to check some outputs and targets manually and make sure that the values are in the expected range and that no unwanted permutation was appiled somewhere in the code.</p>
</li>
<li>
<p>Your model might create large intermediate tensors, which would create this OOM issue. The number of input features is also relatively large in the linear layer, which would also point towards a high memory usage.</p>
</li>
</ol>

username: ahmed
cooked: <p>Thank you again for your suggestions.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="96676">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you flatten the tensors starting in <code>dim1</code> ?</p>
</blockquote>
</aside>
<p>This doesn’t seem to make a difference in terms to the high loss value. However, I have tried changing the <code>optimizer</code> from <code>SGD</code> to <code>Adam</code>, and the loss values begin to drop (although still really high).</p>
<aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="96676">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I would recommend to check some outputs and targets manually and make sure that the values are in the expected range and that no unwanted permutation was appiled somewhere in the code.</p>
</blockquote>
</aside>
<p>When I view the outputs and targets, they look like they’re within the expected range (I don’t see any large variations in the predicted keypoints when comparing to the ground truth values).</p>
<aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="96676">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Your model might create large intermediate tensors, which would create this OOM issue. The number of input features is also relatively large in the linear layer, which would also point towards a high memory usage.</p>
</blockquote>
</aside>
<p>You were right, I adjusted my model by adding some more convolutional layers etc, and now the model is working.</p>