Additional logs when update pytorch version to 1.7 and using Distribute Training

PistonY: <p>Hi all, I found <code>logger.info</code> will print additional info when using Distribute training (DDP).<br>
When using pytorch version of 1.6.Calling <code>logger.info</code> may give:</p>
<pre><code class="lang-auto">Epoch 1, Node 0, GPU 3, Iter 300, Top1 Accuracy:0.083602, Loss:5.4438, 132 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 6, Iter 300, Top1 Accuracy:0.086197, Loss:5.4218, 129 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 1, Iter 300, Top1 Accuracy:0.084302, Loss:5.4295, 86 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 2, Iter 300, Top1 Accuracy:0.083498, Loss:5.4405, 130 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 0, Iter 300, Top1 Accuracy:0.087469, Loss:5.4242, 131 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 5, Iter 300, Top1 Accuracy:0.083731, Loss:5.4297, 127 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 7, Iter 300, Top1 Accuracy:0.0856, Loss:5.4234, 130 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 4, Iter 300, Top1 Accuracy:0.08355, Loss:5.4322, 86 samples/s. lr: 0.64553.
</code></pre>
<p>But when I update pytorch version to 1.7</p>
<pre><code class="lang-auto">Epoch 1, Node 0, GPU 3, Iter 300, Top1 Accuracy:0.083602, Loss:5.4438, 132 samples/s. lr: 0.64553.
INFO:Distribute training logs.:Epoch 1, Node 0, GPU 3, Iter 300, Top1 Accuracy:0.083602, Loss:5.4438, 132 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 6, Iter 300, Top1 Accuracy:0.086197, Loss:5.4218, 129 samples/s. lr: 0.64553.
INFO:Distribute training logs.:Epoch 1, Node 0, GPU 6, Iter 300, Top1 Accuracy:0.086197, Loss:5.4218, 129 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 1, Iter 300, Top1 Accuracy:0.084302, Loss:5.4295, 86 samples/s. lr: 0.64553.
INFO:Distribute training logs.:Epoch 1, Node 0, GPU 1, Iter 300, Top1 Accuracy:0.084302, Loss:5.4295, 86 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 2, Iter 300, Top1 Accuracy:0.083498, Loss:5.4405, 130 samples/s. lr: 0.64553.
INFO:Distribute training logs.:Epoch 1, Node 0, GPU 2, Iter 300, Top1 Accuracy:0.083498, Loss:5.4405, 130 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 0, Iter 300, Top1 Accuracy:0.087469, Loss:5.4242, 131 samples/s. lr: 0.64553.
INFO:Distribute training logs.:Epoch 1, Node 0, GPU 0, Iter 300, Top1 Accuracy:0.087469, Loss:5.4242, 131 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 5, Iter 300, Top1 Accuracy:0.083731, Loss:5.4297, 127 samples/s. lr: 0.64553.
INFO:Distribute training logs.:Epoch 1, Node 0, GPU 5, Iter 300, Top1 Accuracy:0.083731, Loss:5.4297, 127 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 7, Iter 300, Top1 Accuracy:0.0856, Loss:5.4234, 130 samples/s. lr: 0.64553.
INFO:Distribute training logs.:Epoch 1, Node 0, GPU 7, Iter 300, Top1 Accuracy:0.0856, Loss:5.4234, 130 samples/s. lr: 0.64553.
Epoch 1, Node 0, GPU 4, Iter 300, Top1 Accuracy:0.08355, Loss:5.4322, 86 samples/s. lr: 0.64553.
INFO:Distribute training logs.:Epoch 1, Node 0, GPU 4, Iter 300, Top1 Accuracy:0.08355, Loss:5.4322, 86 samples/s. lr: 0.64553.
</code></pre>
<p>I’m sure I only update pytorch version and do not change any package.<br>
Who print the addition info and how to stop this.</p>

osalpekar: <p><a class="mention" href="/u/pistony">@PistonY</a> Do you mind sharing the part of the training script that prints these logs? I don’t see any code in PyTorch itself that would generate these extra logs.</p>

PistonY: <p><a class="mention" href="/u/osalpekar">@osalpekar</a> hi, thx for reply.<br>
I use <a href="https://github.com/PistonY/ModelZoo.pytorch/blob/master/scripts/distribute_train_script.py" rel="noopener nofollow ugc">this</a> script and <a href="https://github.com/PistonY/ModelZoo.pytorch/blob/eb4cc16bfbb6bdf1c021f5f6faba7fbfc8c04612/scripts/distribute_train_script.py#L339" rel="noopener nofollow ugc">this</a> line will print the info.</p>

pritamdamania87: <p><a class="mention" href="/u/pistony">@PistonY</a> Looking at the additional logs, isn’t it coming from here: <a href="https://github.com/PistonY/ModelZoo.pytorch/blob/fd85403bb1430ce591a97586a097c989630aa82b/scripts/utils.py#L41" rel="noopener nofollow ugc">https://github.com/PistonY/ModelZoo.pytorch/blob/fd85403bb1430ce591a97586a097c989630aa82b/scripts/utils.py#L41</a> instead of PyTorch?</p>

PistonY: <p>hi, <a class="mention" href="/u/pritamdamania87">@pritamdamania87</a> thanks for reply, but this line don’t print anything when using pytorch version &lt;= 1.7.0, I didn’t modify any code but update pytorch version, do you know how to stop it?</p>

pritamdamania87: <p>I think the upgrade to PyTorch might be unrelated here and something else is changing that is causing additional logging. These are log messages generated by your application and not PyTorch, probably double checking the logging configuration might reveal the reason for this.</p>

PistonY: <p>I don’t think it’s my issues.<br>
I config <code>logs</code> as bellow.</p>
<pre><code class="lang-auto">def get_logger(file_path):
    streamhandler = logging.StreamHandler()
    logger = logging.getLogger('Distribute training logs.')
    logger.setLevel(logging.INFO)
    logger.addHandler(streamhandler)
    return logger
</code></pre>
<p>I tried this still has additional logs.<br>
Could you please give some suggests <a class="mention" href="/u/osalpekar">@osalpekar</a>?</p>

PistonY: <p>I remove the name of <code>logging.getLogger</code> and no additional logs print.<br>
But print this.<br>
<code>Reducer buckets have been rebuilt in this iteration.</code><br>
I don’t know what happened, but finally logs is clear,</p>