fasterRCNN getting both predictions and loss during evaluation

username: almog
cooked: <p>Hello,<br>
I am using fasterRCNN and would like to evaluate my model by checking the loss, IoU, and accuracy.<br>
When the model is on train() mode, it returns only the loss, and when on eval() mode it returns only the predictions (relevant for calculating accuracy and IoU).</p>
<p><strong>I have seen few posts about it in this forum, but none have found a solution.</strong></p>
<p>How can I return from the model both loss and predictions during evaluation?<br>
Thank you</p>

username: emcp
cooked: <p>are you working with a specific model tutorial or example?</p>
<p>Hard to answer without knowing more details but if you’re using the example pytorch tutorial I believe the evaluation function uses a inference (predictions) step then takes your inference and runs it by some COCO evaluation metrics…</p>
<p>I have my best attempt at splitting up the two from one another in this post</p>
<aside class="quote quote-modified" data-post="3" data-topic="124556">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/db5fbb/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/object-detection-tutorial-with-cocodataset-giving-empty-mask-predictions/124556/3">Object detection tutorial with COCODataset giving empty mask predictions</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I whittled down a simpler example of the original example and got a (I think) proper score function… only to realize the T.toTensor function does some strange things… and I need to rewrite this for inference… correct?  don’t I need to transform all images to a tensor or ? 
so far I have this 
@torch.no_grad()
def score(paths_to_images):
    # train on the GPU or on the CPU, if a GPU is not available
    device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

    num_…
  </blockquote>
</aside>


username: almog
cooked: <p>Thank you for your reply <a class="mention" href="/u/emcp">@emcp</a> !</p>
<p>I am using something like that: <a href="https://blog.francium.tech/object-detection-with-faster-rcnn-bc2e4295bf49" class="inline-onebox" rel="noopener nofollow ugc">Object Detection with Faster RCNN | by Arun Prakash | Francium Tech</a><br>
This is the training part from the tutorial:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/7/275419ec2874615e2458a53e8f36a48e2a219098.png" data-download-href="https://discuss.pytorch.org/uploads/default/275419ec2874615e2458a53e8f36a48e2a219098" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/275419ec2874615e2458a53e8f36a48e2a219098_2_605x500.png" alt="image" data-base62-sha1="5BUO3TDlgKwvQusDBGQioVtGBzG" width="605" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/275419ec2874615e2458a53e8f36a48e2a219098_2_605x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/275419ec2874615e2458a53e8f36a48e2a219098_2_907x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/7/275419ec2874615e2458a53e8f36a48e2a219098.png 2x" data-dominant-color="2F302A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1050×867 164 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My problem is that the model returns either the loss <strong>or</strong> the predictions so I want to find a way to get both for evaluation without iterating over my entire data twice every epoch (it takes me 24 minutes for one iteration over train set- 16,000 images with batch_size=32)</p>
<p>I am not familiar with an evaluation function in pytorch tutorial. I would appreciate if you can direct me to it if it is relevant for what I want</p>
<p>Thank you!</p>

username: emcp
cooked: <blockquote>
<p>I am not familiar with an evaluation function in pytorch tutorial. I would appreciate if you can direct me to it if it is relevant for what I want</p>
</blockquote>
<p>I thought you’d been looking at the pytorch torchvision tutorial and it’s source code</p>
<p><a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>
<p>The main training function</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/vision/blob/main/references/detection/train.py#L203-L224">
  <header class="source">

      <a href="https://github.com/pytorch/vision/blob/main/references/detection/train.py#L203-L224" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/main/references/detection/train.py#L203-L224" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/main/references/detection/train.py#L203-L224</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="203" style="counter-reset: li-counter 202 ;">
          <li># Data loading code</li>
          <li>print("Loading data")</li>
          <li></li>
          <li>dataset, num_classes = get_dataset(is_train=True, args=args)</li>
          <li>dataset_test, _ = get_dataset(is_train=False, args=args)</li>
          <li></li>
          <li>print("Creating data loaders")</li>
          <li>if args.distributed:</li>
          <li>    train_sampler = torch.utils.data.distributed.DistributedSampler(dataset)</li>
          <li>    test_sampler = torch.utils.data.distributed.DistributedSampler(dataset_test, shuffle=False)</li>
          <li>else:</li>
          <li>    train_sampler = torch.utils.data.RandomSampler(dataset)</li>
          <li>    test_sampler = torch.utils.data.SequentialSampler(dataset_test)</li>
          <li></li>
          <li>if args.aspect_ratio_group_factor &gt;= 0:</li>
          <li>    group_ids = create_aspect_ratio_groups(dataset, k=args.aspect_ratio_group_factor)</li>
          <li>    train_batch_sampler = GroupedBatchSampler(train_sampler, group_ids, args.batch_size)</li>
          <li>else:</li>
          <li>    train_batch_sampler = torch.utils.data.BatchSampler(train_sampler, args.batch_size, drop_last=True)</li>
          <li></li>
      </ol>
    </code></pre>


  This file has been truncated. <a href="https://github.com/pytorch/vision/blob/main/references/detection/train.py#L203-L224" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The <code>train_one_epoch</code> piece</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/vision/blob/main/references/detection/engine.py#L13-L55">
  <header class="source">

      <a href="https://github.com/pytorch/vision/blob/main/references/detection/engine.py#L13-L55" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/main/references/detection/engine.py#L13-L55" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/main/references/detection/engine.py#L13-L55</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="13" style="counter-reset: li-counter 12 ;">
          <li>model.train()</li>
          <li>metric_logger = utils.MetricLogger(delimiter="  ")</li>
          <li>metric_logger.add_meter("lr", utils.SmoothedValue(window_size=1, fmt="{value:.6f}"))</li>
          <li>header = f"Epoch: [{epoch}]"</li>
          <li></li>
          <li>lr_scheduler = None</li>
          <li>if epoch == 0:</li>
          <li>    warmup_factor = 1.0 / 1000</li>
          <li>    warmup_iters = min(1000, len(data_loader) - 1)</li>
          <li></li>
          <li>    lr_scheduler = torch.optim.lr_scheduler.LinearLR(</li>
          <li>        optimizer, start_factor=warmup_factor, total_iters=warmup_iters</li>
          <li>    )</li>
          <li></li>
          <li>for images, targets in metric_logger.log_every(data_loader, print_freq, header):</li>
          <li>    images = list(image.to(device) for image in images)</li>
          <li>    targets = [{k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in t.items()} for t in targets]</li>
          <li>    with torch.cuda.amp.autocast(enabled=scaler is not None):</li>
          <li>        loss_dict = model(images, targets)</li>
          <li>        losses = sum(loss for loss in loss_dict.values())</li>
      </ol>
    </code></pre>


  This file has been truncated. <a href="https://github.com/pytorch/vision/blob/main/references/detection/engine.py#L13-L55" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: almog
cooked: <p>Thank you <a class="mention" href="/u/emcp">@emcp</a><br>
Sorry, I did see this tutorial. My training procedure is similar to this one.<br>
However, in this tutorial they don’t calculate both loss and IoU for evaluation- only IoU.<br>
I have also looked at their evaluate() function which runs over test_loader, but it only calculates IoU and AP, not loss.<br>
I want to also calculate the loss on the test_loader in order to make sure that I don’t overfit over train_loader while training.</p>
<p>Thank you</p>

username: KouWeiBin
cooked: <p>In my opinion, you maybe copy the loss related code from train function to evaluate function.</p>