Average time per epoch calculation on pytorch ignite

username: Tanya_Boone
cooked: <p>Hello,</p>
<p>I need to save a file with the average elapsed time per epoch and I found this function on the documentation:</p>
<p><a href="https://pytorch.org/ignite/_modules/ignite/handlers/timing.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/ignite/_modules/ignite/handlers/timing.html</a></p>
<pre><code class="lang-auto"> t = Timer(average=True)
for _ in range(10):
       work()
       idle()
       t.step()
</code></pre>
<pre><code class="lang-auto">def save_time(engine):
     torch.save(t.value(),'time.pt´)

trainer.add_event_handler(Events.EPOCH_STARTED(every=3), save_time)
</code></pre>
<p>this works but It shows the elapsed time in 3 epochs and I suppose the file overwrites every 3 epochs but <strong>what I want is to calculate the average time just once for the first epoch and stop the timer</strong>. I don’t want to divide timer value between 3, also I don’t want to generate one file per epoch. One option is to run my code with  just one epoch but I am running several models and I would like to generate this just once every time I am going to train.</p>
<p>How can I achieve this? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>pd: the documentation show an example doing it by iteration, but I am unable to translate this example to epoch. I believe the documentation should extend to epoch example as well.</p>
<p>Regards,</p>

username: sdesrozis
cooked: <p><a class="mention" href="/u/tanya_boone">@Tanya_Boone</a> Thank you for this question.</p>
<p>I think that several things could help you.</p>
<p>First, you can use <code>Timer</code> with filtered event. I think that in your case, you should try the following code to reset the timer every 3 epochs (started)</p>
<pre><code class="lang-python">timer.attach(trainer,
             start=Events.EPOCH_STARTED(every=3),
             resume=Events.ITERATION_STARTED,
             pause=Events.ITERATION_COMPLETED)
</code></pre>
<p>An other option is to use directly <code>Timer</code> (without <code>step</code> to do not average)</p>
<pre><code class="lang-auto">timer = Timer()

@trainer.on(Events.STARTED)
def time_start():
    # reset the timer at the beginning
    timer.reset()

@trainer.on(Events.EPOCH_STARTED(every=3))
def time_start():
    # use value() to get the time
    print("timer of 3 epochs :", timer.value())
    # reset the timer from now for 3 next epochs
    timer.reset()
</code></pre>
<p>Last point I would like to mention, we recently added epoch timers in <code>trainer.state.timers</code>. That is another option which should help you to mesure epoch times</p>
<pre><code class="lang-python">@trainer.on(Events.EPOCH_COMPLETED)
def print_timers():
    # trainer.state.timers is a dict
    print("timers=", trainer.state.timers)
</code></pre>
<p>If you need more support, don’t hesitate to ask, it would be a pleasure to help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>HTH</p>

username: vfdev-5
cooked: <p>Thanks Sylvain !</p>
<p><a class="mention" href="/u/tanya_boone">@Tanya_Boone</a> please let us know if Sylvain’s answer solves your problem. Thanks !</p>

username: Tanya_Boone
cooked: <aside class="quote no-group" data-username="sdesrozis" data-post="2" data-topic="82506">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sdesrozis/48/24628_2.png" class="avatar"> sdesrozis:</div>
<blockquote>
<pre><code class="lang-auto">timer.attach(trainer,
             start=Events.EPOCH_STARTED(every=3),
             resume=Events.ITERATION_STARTED,
             pause=Events.ITERATION_COMPLETED)
</code></pre>
</blockquote>
</aside>
<p>Hello,</p>
<p>I am trying with this method</p>
<pre><code class="lang-auto">timer.attach(trainer,start=Events.EPOCH_STARTED(every=1),
    resume=Events.ITERATION_STARTED,
    pause=Events.ITERATION_COMPLETED)
</code></pre>
<p>After I want to print the value</p>
<p>print(“timers=”, trainer.state.timers)</p>
<p>Engine run is terminating due to exception: ‘State’ object has no attribute 'timers". I changed to timer but I am getting the same error <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

username: sdesrozis
cooked: <p>This feature is not in stable 0.3.0. You have to use git version or nightly version. Sorry to miss that point.</p>

username: vfdev-5
cooked: <p>To install nightly release: <a href="https://github.com/pytorch/ignite#nightly-releases" rel="nofollow noopener">https://github.com/pytorch/ignite#nightly-releases</a></p>

username: gnufabio
cooked: <p>Hi all, I am sorry to bum an old conversation, but my question is related to the question of the OP.</p>
<p>I’m trying to measure the average wall-clock time per training epoch with Ignite’s Timer. I then want to log it to wandb when the training is completed or terminated (e.g. due to a Ctrl+C exception).</p>
<p>Right now I am using this code:</p>
<pre><code class="lang-auto">timer = Timer(average=True)
timer.attach(
    train_engine,
    start=Events.STARTED,
    resume=Events.EPOCH_STARTED,
    pause=Events.EPOCH_COMPLETED,
    step=Events.EPOCH_COMPLETED,
)

def log_timer(engine: Engine):
    print(f"Average time per epoch: {self.timer.value():.2f}")
    wandb.log({"avg_epoch_time": timer.value()})
    timer.reset()

train_engine.add_event_handler(Events.COMPLETED | Events.TERMINATE | Events.EXCEPTION_RAISED, log_timer)
</code></pre>
<p>However,  the logged time-per-epoch seems unreasonably low (like 0.01, while I was expecting something like 2-4 seconds). Am I doing something wrong in the definition of the Timer?</p>

username: vfdev-5
cooked: <p><a class="mention" href="/u/gnufabio">@gnufabio</a> hard to see what can go wrong without some code.<br>
It may depend on whether you do some work on EPOCH_COMPLETED or EPOCH_STARTED events and attached the timer first before attaching these time consuming handlers ?<br>
Maybe, as you average the times, you have something less expected ?<br>
Alternatively, you can try to use <code>engine.state.times[Events.EPOCH_COMPLETED.name]</code>, <a href="https://pytorch.org/ignite/generated/ignite.engine.events.State.html#ignite.engine.events.State" class="inline-onebox" rel="noopener nofollow ugc">State — PyTorch-Ignite v0.5.1 Documentation</a> for that.</p>

username: gnufabio
cooked: <p><a class="mention" href="/u/vfdev-5">@vfdev-5</a>  thanks for your answer.</p>
<p>I think I found the problem - it was due to log_timer being called both for Events.TERMINATE and Events.COMPLETE. timer.value() was low because I was resetting the timer after Events.TERMINATE was fired. I solved by removing the log_timer handler after its first call.</p>
<p>That being said, I am not sure why Events.TERMINATE and Events.COMPLETE are both fired, since <a href="https://pytorch.org/ignite/generated/ignite.engine.events.Events.html" rel="noopener nofollow ugc">in the official documentation</a> they seem to be mutually exclusive.<br>
Just as a reference, I have a bunch of other handlers attached to the engine:</p>
<ul>
<li>Handler to run the validation loop at the end of each epoch</li>
<li>ModelCheckpoint used to save the checkpoint and restore the weights when trainer fires the TERMINATE or COMPLETE event</li>
<li>EarlyStopping fired on evaluator’s Events.COMPLETE</li>
<li>Handler to catch Ctrl+C and gracefully terminate the training</li>
</ul>

username: vfdev-5
cooked: <p>Glad you could find the problem and hope it works now as expected. Feel free to ask other questions here or on our discord.</p>
<blockquote>
<p>That being said, I am not sure why Events.TERMINATE and Events.COMPLETE are both fired, since <a href="https://pytorch.org/ignite/generated/ignite.engine.events.Events.html" rel="noopener nofollow ugc">in the official documentation</a> they seem to be mutually exclusive.</p>
</blockquote>
<p>We provided some info about that here: <a href="https://pytorch.org/ignite/generated/ignite.engine.engine.Engine.html#ignite.engine.engine.Engine.terminate" class="inline-onebox" rel="noopener nofollow ugc">Engine — PyTorch-Ignite v0.5.1 Documentation</a></p>
<p>It was a question whether we should skip or not <code>Events.COMPLETE</code> when <code>terminate()</code> was called and it was decided to keep it. Maybe, it could be a feature to add a flag to <code>terminate(skip_complete_event=True)</code> to skip <code>Events.COMPLETE</code>.</p>

username: gnufabio
cooked: <aside class="quote no-group" data-username="vfdev-5" data-post="10" data-topic="82506">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vfdev-5/48/3535_2.png" class="avatar"> vfdev-5:</div>
<blockquote>
<p>Maybe, it could be a feature to add a flag to <code>terminate(skip_complete_event=True)</code> to skip <code>Events.COMPLETE</code>.</p>
</blockquote>
</aside>
<p>I think that would make things more intuitive and flexible. While from the Engine’s docs it is clear that <code>trainer.terminate()</code> fires both <code>Events.COMPLETE</code> and <code>Events.TERMINATE</code>, in the <a href="https://pytorch.org/ignite/generated/ignite.engine.events.Events.html" rel="noopener nofollow ugc">Events’ docs</a> there is a table that looks misleading…</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2e5346d8b3cf5b4c60c9bad033bebfbcc19868f.png" data-download-href="https://discuss.pytorch.org/uploads/default/a2e5346d8b3cf5b4c60c9bad033bebfbcc19868f" title=""><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2e5346d8b3cf5b4c60c9bad033bebfbcc19868f_2_690x239.png" alt="" data-base62-sha1="nf2pZuVJGLRXaeHa7zj2lEiciKz" width="690" height="239" role="presentation" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2e5346d8b3cf5b4c60c9bad033bebfbcc19868f_2_690x239.png, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2e5346d8b3cf5b4c60c9bad033bebfbcc19868f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2e5346d8b3cf5b4c60c9bad033bebfbcc19868f.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename"></span><span class="informations">831×288 11.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Or perhaps I am misinterpreting it? Anyways, thank you for your kind support!</p>

username: vfdev-5
cooked: <p>You are right, the table needs to be updated to make it clear.</p>
<p>PS: if you wish to make this contribution to the project, you are very welcome!</p>