Combine two models; right use of CrossEntropyLoss (Classification) and SmoothL1Loss (regressor)

username: MikeTensor
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/573bb311fd0b619993556d2bc6f383d056602853.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/573bb311fd0b619993556d2bc6f383d056602853" title="the setup"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/573bb311fd0b619993556d2bc6f383d056602853_2_690x299.jpeg" alt="Alt text" title="the setup" width="690" height="299" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/573bb311fd0b619993556d2bc6f383d056602853_2_690x299.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/573bb311fd0b619993556d2bc6f383d056602853_2_1035x448.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/7/573bb311fd0b619993556d2bc6f383d056602853.jpeg 2x" data-dominant-color="DBDDD8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">the setup</span><span class="informations">1377×597 175 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<a href="https://openaccess.thecvf.com/content_ICCV_2019/papers/Zhu_Learning_Object-Specific_Distance_From_a_Monocular_Image_ICCV_2019_paper.pdf" rel="noopener nofollow ugc">Source | Paper</a></p>
<p>ModelA: <code>nn.CrossEntropyLoss()</code> → Is used for <strong>Classification</strong>.<br>
ModelB: <code>nn.SmoothL1Loss(reduction='mean')</code> → Is used for <strong>regression</strong>.</p>
<p>I am having two questions:<br>
<strong>1) How to combine the two <code>model</code>s?</strong><br>
<strong>2) How should the <code>distance_regressor</code> look like together with the <code>loss_dist_fn</code></strong></p>
<p>Referring to 1) Combining two <code>model</code>s</p>
<pre><code class="lang-auto">class_dist_parameters = list(classifier.parameters()) + list(distance_regressor.parameters())
optimizer = torch.optim.Adam(class_dist_parameters, lr=0.001)
</code></pre>
<p>Within my training_step I do everything serialized:<br>
Like:</p>
<pre><code class="lang-auto">classifier.train()
distance_regressor.train()

#within the for_loop:
  for batch_out, (image_dl, rois_dl, class_label_dl, distance_label_dl, df_train_dl) in enumerate(dataloader):


            distance_label_dl = distance_label_dl.reshape(distance_label_dl.shape[0], 1)

            output_class_pred = classifier(rois_out)
            roi_output_distance_pred = distance_regressor(rois_out)

            output_distance_pred = softplus(roi_output_distance_pred)

            # 2. Calculate  and accumulate loss
            #size mismatch (got input: [9], target: [8]) → fixed with reshape.
            loss_class = loss_fn_class(output_class_pred, class_label_dl)
            train_loss_class += loss_class.item()

            loss_distance = loss_fn_distance(output_distance_pred), distance_label_dl)
            train_loss_distance += loss_distance.item()

            # 3. Optimizer zero grad
            optimizer.zero_grad()

            # 4. Loss backward
            #loss.backward()

            loss_class.backward()
            loss_distance.backward()

            # 5. Optimizer step
            optimizer.step()

        # Calculate and accumulate accuracy metric across all batches
        y_pred_class_no = torch.argmax(torch.softmax(output_class_pred, dim=1), dim=1)

        #######################################
        # Not sure how to continue with the distance_regressor
        #######################################
        y_pred_distance_no = output_distance_pred
        #y_pred_distance_no = torch.argmax(torch.softplus(output_distance_pred), dim=1)

        ### Classifier seems to work like this.
        train_acc_class += (y_pred_class_no == class_label_dl).sum().item()/len(output_class_pred)

        ### 
        train_rmse_acc_distance += ((torch.pow((y_pred_distance_no - distance_label_dl), 2)).sum())/len(output_distance_pred)



</code></pre>
<p>Anything I forgot, or special about combining the two models?</p>
<p>Referring to<br>
<strong>2) How should the <code>distance_regressor</code> look like together with the <code>loss_dist_fn</code></strong></p>
<pre><code class="lang-auto">def build_distance_regressor(in_feature_size, first_layer_size, second_layer_size, out_feature_size):
    distance_regressor = nn.Sequential(
        nn.Flatten(1, -1),
        nn.Linear(in_features=in_feature_size, out_features=first_layer_size),
        nn.Linear(in_features=first_layer_size, out_features=second_layer_size),
        nn.Linear(in_features=second_layer_size, out_features=out_feature_size),
    )
    return distance_regressor.to(device)
</code></pre>
<p>with</p>
<pre><code class="lang-auto">dist_regressor= build_distance_regressor(in_feature_size=214016, first_layer_size=2048, second_layer_size=512, out_feature_size=1)
</code></pre>
<p>But …<br>
Right now I don’t get any improvements, regarding to <code>distance_regressor</code>.<br>
model_output is a big negative value; which means that <code>softplus</code> is making zeros out of it, and even after 20 Epochs; nothing changes.</p>
<p><code>classifier</code> seems to work in single use (simple Linear-Layer).<br>
So I am thinking about to normalize the distance_targets with <code>min–max</code>.</p>
<p>Any ideas how to get the <code>distance_regressor</code> working?</p>
<p>Maybe you have an idea? <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>P.S. I already consulted this topic: <a href="https://discuss.pytorch.org/t/a-model-with-multiple-outputs/10440/" class="inline-onebox">A model with multiple outputs</a></p>