Best practices for collecting metrics

username: Viktor_Todosijevic
cooked: <p>Hi,</p>
<p>I’ve been struggling with collecting a lot of different metrics in PyTorch’s train/val loops. I always end up with a lot of boilerplate code that looks a lot like this:</p>
<pre><code class="lang-python">for epoch in range(num_epochs):

  metric1_train = []
  metric2_train = []
  metric3_train = []
    ...
  
  model.train()
  for batch in train_loader:
      ...
      metirc1_train.append(...)
      metric2_train.append(...)
      metric3_train.append(...)
      ...
  
  wandb.log({
    "train/metric1": np.mean(metirc1_train),
    ...
  })

  model.eval()
  with torch.no_grad():
      metric1_val = []
      metric2_val = []
      metric3_val = []
      ...
      for batch in val_loader:
        ...
        metirc1_val.append(...)
        metric2_val.append(...)
        metric3_val.append(...)
        ...

  wandb.log({
    "val/metric1": np.mean(metirc1_val),
    ...
  })

</code></pre>
<p>I want to not have to repeat all the code in the validation part. I tried tackling this in a OOP way but it didn’t end up pretty. I’d also like it to be easy to add new metrics, this I couldn’t handle well.</p>
<p>I wanna ask if there is a standard, best practice way to do something like this that handles 10s of metrics?</p>
<p>Thanks!</p>

username: avihu111
cooked: <p>I think you might be interested in “PyTorch Lightning” and “TorchMetrics”, which wrap PyTorch:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.pytorchlightning.ai/index.html">
  <header class="source">
      <img src="https://uploads-ssl.webflow.com/5f76c986da6f6011315a6c45/5f7b485594e5335f56984a3c_Lightning_FavIcon.png" class="site-icon" width="32" height="32">

      <a href="https://www.pytorchlightning.ai/index.html" target="_blank" rel="noopener nofollow ugc">pytorchlightning.ai</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/388;"><img src="https://uploads-ssl.webflow.com/5f76c986da6f6011315a6c45/5f87362356e77f2cff6acf25_Title_Purple_2.png" class="thumbnail" width="690" height="388"></div>

<h3><a href="https://www.pytorchlightning.ai/index.html" target="_blank" rel="noopener nofollow ugc">PyTorch Lightning</a></h3>

  <p>The ultimate PyTorch research framework. Scale your models, without the boilerplate.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p><a href="https://torchmetrics.readthedocs.io/en/latest/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://torchmetrics.readthedocs.io/en/latest/</a></p>
<p>In case you want to use vanilla PyTorch, I would use a separate dictionaries for:</p>
<ul>
<li>input data</li>
<li>model outputs</li>
<li>losses</li>
<li>metrics<br>
By doing so, you can prevent a lot of duplicative lines by using dictionary comprehension.</li>
</ul>
<pre><code class="lang-auto">for batch_dict in train_loader:
   preds_dict = model(**batch_dict)
   loss_dict = loss_fn(**preds_dict, **batch_dict)
   metrics_dict = metric_fn(**pred_dict, **batch_dict)
   wandb.log(metrics_dict)
</code></pre>
<p>I like this format of writing since it automatically unpacks only the relevant inputs for me, the rest of the data is left in <code>**kwargs</code> in the function signature.<br>
Hope this helps!</p>