Epoch_length wrong number of iterations

xen0f0n: <p>I want to iterate over the training samples (172 samples) a fixed number of times (e.g. 512)</p>
<p><code>trainer.run(train_loader, max_epochs=500, epoch_length=512)</code></p>
<p>and in my Dataset subclass implementation I’ve  defined <code>__len__</code>  to return the number of samples (172 samples).<br>
But epoch_length does not stop loading at 512, but at 688 samples (4x172). I’ve noticed this behavior by printing a counter when <code>__getitem__</code> is called.<br>
Any thoughts?</p>

vfdev-5: <p>Hi <a class="mention" href="/u/xen0f0n">@xen0f0n</a>, what is the expected behaviour for you ?<br>
I think what happens is a normal behaviour:<br>
Your data provider has 172 samples, when engine is asked to measure an epoch as 512 iterations it does not mean that data provider should be restarted after 512 iterations.</p>
<pre><code class="lang-auto">data
|-----|-----|-----|-----|-----|-----|-----|-----|
epoch
|----------|----------|----------|----------|
</code></pre>
<p>So, trainer asks for data from the data provider and on 512th iteration, data loader has been restarted previously twice and provides 168th sample, then on 513th iteration (and epoch + 1), provided data corresponds to 169th sample from data loader.</p>
<p>What do you think ?</p>

xen0f0n: <p><a class="mention" href="/u/vfdev-5">@vfdev-5</a>  Well, it would make sense if on 513th I was on epoch + 1.<br>
But I don’t think that’s the case. At the completion of each epoch I log some metrics and this takes place on iteration 688 and not before. Hence, that’s when the epoch ends, right?</p>

vfdev-5: <p>OK, we need to check versions and the code. Here is my snippet</p>
<pre><code class="lang-python">import ignite
print(ignite.__version__)

from ignite.engine import Engine

trainer = Engine(lambda e, b: print("{} - {} : {}".format(e.state.epoch, e.state.iteration, b)))

data = list(range(172))
trainer.run(data, epoch_length=512, max_epochs=10)
</code></pre>
<p>with the output</p>
<pre><code class="lang-auto">0.4.0.dev20200302
1 - 1 : 0
1 - 2 : 1
1 - 3 : 2
1 - 4 : 3
1 - 5 : 4
1 - 6 : 5
1 - 7 : 6
1 - 8 : 7
1 - 9 : 8
1 - 10 : 9
1 - 11 : 10
...
1 - 510 : 165
1 - 511 : 166
1 - 512 : 167
2 - 513 : 168
2 - 514 : 169
2 - 515 : 170
2 - 516 : 171
2 - 517 : 0
2 - 518 : 1
2 - 519 : 2
2 - 520 : 3
2 - 521 : 4
2 - 522 : 5
2 - 523 : 6
...
10 - 5112 : 123
10 - 5113 : 124
10 - 5114 : 125
10 - 5115 : 126
10 - 5116 : 127
10 - 5117 : 128
10 - 5118 : 129
10 - 5119 : 130
10 - 5120 : 131
</code></pre>

xen0f0n: <p><a class="mention" href="/u/vfdev-5">@vfdev-5</a> I got the same output. But I still can’t explain that the evaluation happening on EPOCH_COMPLETED<br>
<code>@trainer.on(Events.EPOCH_COMPLETED)</code><br>
takes place NOT on iteration 512 but on 688. I’ve also tried this for <code>epoch_length=3</code>, and <code>engine.state.iteration</code> is 172 when logging.</p>
<pre><code class="lang-auto">
@trainer.on(Events.EPOCH_COMPLETED)
    def evaluate(trainer):
        with evaluator.add_event_handler(Events.COMPLETED, log_metrics, "train"):
            evaluator.run(train_loader)
</code></pre>
<p>It would make more sense to me if the both <code>Events.ITERATION_COMPLETED(every=512)</code> and <code>Events.EPOCH_COMPLETED(every=1)</code> fired the “same time”.</p>

xen0f0n: <p><a class="mention" href="/u/vfdev-5">@vfdev-5</a> it seems I’ve made a mistake… Trainer accepts <code>epoch_length</code> as argument, BUT I’ve created an evaluator engine  with <code>create_supervised_evaluator</code> and although <code>trainer.state</code> is correct, <code>evaluator.state</code> doesn’t have the same <code>epoch</code> and <code>iteration</code>.</p>

vfdev-5: <p>So, could you find where is a problem in your code ?</p>
<blockquote>
<p>although <code>trainer.state</code> is correct, <code>evaluator.state</code> doesn’t have the same <code>epoch</code> and <code>iteration</code> .</p>
</blockquote>
<p><code>evaluator.state</code> normally contains nothing interesting except metrics. As we usually run it<br>
like <code>evaluator.run(train_loader)</code> it makes 1 epoch defined by <code>len(train_loader)</code> and its should contain according to your example, the following:</p>
<pre><code class="lang-auto">iteration: 172
epoch: 1
epoch_length: 172
max_epochs: 1
</code></pre>
<p>And it is correct, evaluator did 1 epoch and 172 iterations as asked. And it does not contain any training information.</p>

xen0f0n: <p><a class="mention" href="/u/vfdev-5">@vfdev-5</a> I am a bit confused. I use the same evaluator for both <code>train_loader</code> and <code>val_loader</code>. Are the metrics calculated during training, or does the evaluator iterate the set again after the training epoch (with other random transforms) , just without updating the weights?</p>
<p>On the train set, the metrics should take into account the samples of the whole epoch (512 samples) and not just the <code>len(train_loader)</code>. Having <code>batch_size=1</code>,  I should have 512 CE losses and calculate the mean. And IoU should be averaged for 512 samples (not 172).</p>

vfdev-5: <blockquote>
<p>I use the same evaluator for both <code>train_loader</code> and <code>val_loader</code> . Are the metrics calculated during training, or does the evaluator iterate the set again after the training epoch (with other random transforms) , just without updating the weights?</p>
</blockquote>
<p>Evaluator does not compute metrics during the training. Yes, evaluator iterates the set again and metrics are computed with “fixed” model (without updating the weights).</p>
<blockquote>
<p>On the train set, the metrics should take into account the samples of the whole epoch (512 samples)</p>
</blockquote>
<p>in this case you need to just set <code>epoch_length=512</code> in <code>evaluator.run(train_loader, epoch_length=512)</code>.</p>