Object detection fine tuning: interpreting printout of train_one_epoch?

cortical_iv: <p>I’m custom training a mask-rcnn using the helpful tutorial:</p>
<p><a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>
<p>Things are basically working, and I’m at the stage where I’m walking through really trying to understand and tweak things. Currently I’m confused by the <code>MetricLogger</code> outputs when I run <code>engine.train_one_epoch()</code> as follows:</p>
<pre><code>metric_logger = train_one_epoch(model, optimizer, data_loader, device, epoch=0, print_freq=1);
</code></pre>
<p>When I run that (focusing on <code>loss_classifier</code>), for the eight batches, I get the following:</p>
<pre><code>Epoch: [0]  [0/8]  loss_classifier: 0.6603 (0.6603) 
Epoch: [0]  [1/8]  loss_classifier: 0.6557 (0.6580)  
Epoch: [0]  [2/8]  loss_classifier: 0.6557 (0.5927)  
Epoch: [0]  [3/8]  loss_classifier: 0.4622 (0.5132)  
Epoch: [0]  [4/8]  loss_classifier: 0.6557 (0.5601)  
Epoch: [0]  [5/8]  loss_classifier: 0.6557 (0.5904)  
Epoch: [0]  [6/8]  loss_classifier: 0.6557 (0.5562)  
Epoch: [0]  [7/8]  loss_classifier: 0.5268 (0.5525)
</code></pre>
<p>I <em>thought</em> the loss was supposed to be described for each batch, with the running average in parentheses. But in the third row, the bit in parenthesis goes below all the values in the batches so far, so it seems it <em>can’t</em> be a running average. The class does have this <code>SmoothedValue</code> attribute so I think I’m just misunderstanding how things work at a basic level.</p>
<p>Also, my goal is to save values from each run of <code>train_one_epoch()</code>,  and I’m not sure what is the best thing to pull at the end as a full summary of each epoch when I save checkpoints and gather summary statistics, or what the right syntax is for getting the values.</p>
<p>Since I started with this tutorial, I feel like I might be missing some cache of discussion/documentation on this MetricLogger class – sorry for all these questions. Any link to a discussion or information anyone can give to any docs would be greatly appreciated! I’m just piecing things together as best I can <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=12" title=":grin:" class="emoji" alt=":grin:" loading="lazy" width="20" height="20"> I am coming over from tensorflow to pytorch – so far it has been great!</p>

ptrblck: <aside class="quote no-group" data-username="cortical_iv" data-post="1" data-topic="193496">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cortical_iv/48/66172_2.png" class="avatar"> cortical_iv:</div>
<blockquote>
<p>But in the third row, the bit in parenthesis goes below all the values in the batches so far, so it seems it <em>can’t</em> be a running average.</p>
</blockquote>
</aside>
<p>Why wouldn’t it be possible if a lot of intermediate values were lower than the current loss value (i.e. <code>0.6557</code>)?</p>

cortical_iv: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for replying: by setting <code>print_freq</code> to <code>1</code> I was explicitly trying to include every batch, so I wouldn’t miss values and the running mean would just equal the mean of all rows thus far in the epoch.  I guess I’m missing something in how the calculations are being performed, as it sounds like the behavior I reported is expected even with <code>print_freq</code> set to 1.</p>
<p>Incidentally, I’m planning to just use <code>loss_classifier = metric_logger.meters['loss_classifier'].avg</code> in my logger when saving checkpoints for validation.</p>

ptrblck: <p>You might not miss anything as I didn’t realize the <code>print_freq=1</code> setting and would also expect the same result of a running average per step.</p>
<p>EDIT: the <code>print_freq</code> argument seems to be passed to <code>log_every</code> <a href="https://github.com/pytorch/vision/blob/7e9e7844e7652bf7b90eef2e9b9397bd20c00e4c/references/detection/utils.py#L145">here</a>, which seems to print the epoch time and eta, if I’m not mistaken.<br>
So still unsure where exactly the losses are printed. I’m not at my workstation so cannot just run the example. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>