DDP training log issue

hkz: <p>Hi there.<br>
I am playing with ImageNet training in Pytorch following <a href="https://github.com/pytorch/examples/tree/master/imagenet" rel="noopener nofollow ugc">official examples</a>. To log things in DDP training, I write a function <code>get_logger</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">import logging
import os
import sys


class NoOp:
    def __getattr__(self, *args):
        def no_op(*args, **kwargs):
            """Accept every signature by doing non-operation."""
            pass

        return no_op


def get_logger(log_dir, log_name=None, resume="", is_rank0=True):
    """Get the program logger.

    Args:
        log_dir (str): The directory to save the log file.
        log_name (str, optional): The log filename. If None, it will use the main
            filename with ``.log`` extension. Default is None.
        resume (str): If False, open the log file in writing and reading mode.
            Else, open the log file in appending and reading mode; Default is "".
        is_rank0 (boolean): If True, create the normal logger; If False, create the null
           logger, which is useful in DDP training. Default is True.
    """
    if is_rank0:
        logger = logging.getLogger(__name__)
        logger.setLevel(level=logging.INFO)

        # StreamHandler
        stream_handler = logging.StreamHandler(sys.stdout)
        stream_handler.setLevel(level=logging.INFO)
        logger.addHandler(stream_handler)

        # FileHandler
        mode = "w+" if resume == "False" else "a+"
        if log_name is None:
            log_name = os.path.basename(sys.argv[0]).split(".")[0] + (".log")
        file_handler = logging.FileHandler(os.path.join(log_dir, log_name), mode=mode)
        file_handler.setLevel(level=logging.INFO)
        logger.addHandler(file_handler)
    else:
        logger = NoOp()

    return logger
</code></pre>
<p>The logger should log to a file and print to stdout only in rank0. This works with <code>torch.__version__</code>&lt;=1.6.0:</p>
<pre><code class="lang-auto">Use GPU: 0 for training
=&gt; creating model 'resnet50'
Epoch: [0][  0/153]     Time  3.652 ( 3.652)    Data  1.754 ( 1.754)    Loss 7.1175e+00 (7.1175e+00)    Acc@1   0.00 (  0.00)   Acc@5   0.00 (  0.00)
Epoch: [0][ 10/153]     Time  0.339 ( 0.632)    Data  0.000 ( 0.160)    Loss 8.2664e+00 (1.3576e+01)    Acc@1   2.35 (  2.67)   Acc@5  11.76 ( 15.29)
Epoch: [0][ 20/153]     Time  0.340 ( 0.493)    Data  0.000 ( 0.089)    Loss 5.9911e+00 (1.0709e+01)    Acc@1  10.59 (  3.75)   Acc@5  21.18 ( 17.03)
Epoch: [0][ 30/153]     Time  0.343 ( 0.444)    Data  0.000 ( 0.064)    Loss 4.9582e+00 (9.2672e+00)    Acc@1   0.00 (  3.49)   Acc@5  16.47 ( 16.96)
Epoch: [0][ 40/153]     Time  0.340 ( 0.419)    Data  0.000 ( 0.051)    Loss 4.5358e+00 (8.3598e+00)    Acc@1   5.88 (  3.64)   Acc@5  15.29 ( 16.96)
Epoch: [0][ 50/153]     Time  0.342 ( 0.404)    Data  0.000 ( 0.043)    Loss 3.4166e+00 (7.5119e+00)    Acc@1   7.06 (  3.62)   Acc@5  22.35 ( 17.02)
</code></pre>
<p>However, when <code>torch.__version__</code>&gt;1.6.0, a repeated INFO exists in stdout while not exists in the log file:</p>
<pre><code class="lang-auto">Use GPU: 0 for training
=&gt; creating model 'resnet50'
Epoch: [0][  0/153]     Time  3.884 ( 3.884)    Data  1.946 ( 1.946)    Loss 7.0728e+00 (7.0728e+00)    Acc@1   0.00 (  0.00)   Acc@5   0.00 (  0.00)
Epoch: [0][ 10/153]     Time  0.338 ( 0.655)    Data  0.000 ( 0.177)    Loss 1.1332e+01 (1.1690e+01)    Acc@1   3.53 (  4.06)   Acc@5  12.94 ( 15.08)
INFO:log:Epoch: [0][ 10/153]    Time  0.338 ( 0.655)    Data  0.000 ( 0.177)    Loss 1.1332e+01 (1.1690e+01)    Acc@1   3.53 (  4.06)   Acc@5  12.94 ( 15.08)
Epoch: [0][ 20/153]     Time  0.340 ( 0.505)    Data  0.000 ( 0.098)    Loss 7.3043e+00 (9.7744e+00)    Acc@1  10.59 (  3.98)   Acc@5  25.88 ( 16.75)
INFO:log:Epoch: [0][ 20/153]    Time  0.340 ( 0.505)    Data  0.000 ( 0.098)    Loss 7.3043e+00 (9.7744e+00)    Acc@1  10.59 (  3.98)   Acc@5  25.88 ( 16.75)
Epoch: [0][ 30/153]     Time  0.341 ( 0.452)    Data  0.000 ( 0.069)    Loss 5.5403e+00 (8.6561e+00)    Acc@1   4.71 (  3.72)   Acc@5  15.29 ( 16.81)
INFO:log:Epoch: [0][ 30/153]    Time  0.341 ( 0.452)    Data  0.000 ( 0.069)    Loss 5.5403e+00 (8.6561e+00)    Acc@1   4.71 (  3.72)   Acc@5  15.29 ( 16.81)
Epoch: [0][ 40/153]     Time  0.341 ( 0.424)    Data  0.000 ( 0.055)    Loss 4.7685e+00 (7.6899e+00)    Acc@1   4.71 (  3.85)   Acc@5  22.35 ( 17.27)
INFO:log:Epoch: [0][ 40/153]    Time  0.341 ( 0.424)    Data  0.000 ( 0.055)    Loss 4.7685e+00 (7.6899e+00)    Acc@1   4.71 (  3.85)   Acc@5  22.35 ( 17.27)
Epoch: [0][ 50/153]     Time  0.340 ( 0.408)    Data  0.000 ( 0.046)    Loss 3.9618e+00 (6.9825e+00)    Acc@1   0.00 (  3.46)   Acc@5  18.82 ( 17.07)
INFO:log:Epoch: [0][ 50/153]    Time  0.340 ( 0.408)    Data  0.000 ( 0.046)    Loss 3.9618e+00 (6.9825e+00)    Acc@1   0.00 (  3.46)   Acc@5  18.82 ( 17.07)
</code></pre>
<p>The log file:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74b87196561f87c93b6cd43b8242b07ceabefb2f.png" data-download-href="https://discuss.pytorch.org/uploads/default/74b87196561f87c93b6cd43b8242b07ceabefb2f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74b87196561f87c93b6cd43b8242b07ceabefb2f_2_690x90.png" alt="image" data-base62-sha1="gEyyqRXcdCVS2DzSSMnYhYcf8x9" width="690" height="90" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74b87196561f87c93b6cd43b8242b07ceabefb2f_2_690x90.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74b87196561f87c93b6cd43b8242b07ceabefb2f_2_1035x135.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74b87196561f87c93b6cd43b8242b07ceabefb2f_2_1380x180.png 2x" data-dominant-color="2B2E31"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1805×238 76.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I provided a working example in <a href="https://gist.github.com/hkunzhe/7be531cd4f7b5d3d721f62687dc09aad" rel="noopener nofollow ugc">my gist</a>.</p>

pritamdamania87: <p>Can you confirm this is the case for 1.9.0 as well?</p>

hkz: <p>Yes, this weird behavior exists in the latest version, i.e., 1.9.0.</p>

ibro45: <p>similar to this? <a href="https://discuss.pytorch.org/t/distributed-1-8-0-logging-twice-in-a-single-process-same-code-works-properly-in-1-7-0/114103/7" class="inline-onebox">Distributed 1.8.0 logging twice in a single process, same code works properly in 1.7.0 - #7 by ibro45</a><br>
PyTorch sets up the loggers somewhere, rebuilding the log handers it as mentioned solves the problem. Personally, I went for <a href="https://github.com/Delgan/loguru/pulse" rel="noopener nofollow ugc">loguru</a> as it’s even easier to do that with it.</p>

hkz: <p>Yes! I met the same problem. It would have saved me a lot of time if I could have searched this post:). And thanks for your suggestion, I will look into loguru for an alternative solution.</p>

pritamdamania87: <p><a class="mention" href="/u/hkz">@hkz</a> I tested this locally and the issue does seem to be in 1.8.0, but it seems like it should be resolved in 1.9.0 (by <a href="https://github.com/pytorch/pytorch/pull/52757" class="inline-onebox" rel="noopener nofollow ugc">[Resubmit] Fix for incorrect usage of logging in torch/distributed/distributed_c10d.py by szmigacz · Pull Request #52757 · pytorch/pytorch · GitHub</a>). I see the double log in 1.8.0, but not in 1.9.0. Can you double check you PyTorch install using `python -c ‘import torch; print(torch.<strong>version</strong>)’?</p>

hkz: <p>Thanks for your kind reply! I create two conda environments, <code>torch1.6</code> and <code>torch1.9</code> respectively.<br>
Running the <a href="https://gist.github.com/hkunzhe/7be531cd4f7b5d3d721f62687dc09aad" rel="noopener nofollow ugc">working example</a>, I can confirm the logging issue exists in <code>torch1.9</code> while not in <code>torch1.6</code>.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c0e8ae2cd851242b75e51d038c3c10840cd99fbd.png" data-download-href="https://discuss.pytorch.org/uploads/default/c0e8ae2cd851242b75e51d038c3c10840cd99fbd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0e8ae2cd851242b75e51d038c3c10840cd99fbd_2_690x157.png" alt="image" data-base62-sha1="rwybLis5srFUg4kKqaVMTAN1Szz" width="690" height="157" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0e8ae2cd851242b75e51d038c3c10840cd99fbd_2_690x157.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0e8ae2cd851242b75e51d038c3c10840cd99fbd_2_1035x235.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0e8ae2cd851242b75e51d038c3c10840cd99fbd_2_1380x314.png 2x" data-dominant-color="0D0D0B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1805×411 77.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d2a22b93a20151f83f38d3d025e0b9fe0724d5d.png" data-download-href="https://discuss.pytorch.org/uploads/default/4d2a22b93a20151f83f38d3d025e0b9fe0724d5d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d2a22b93a20151f83f38d3d025e0b9fe0724d5d_2_690x219.png" alt="image" data-base62-sha1="b0D2vlTsQV36HzaCvXXOZ1kV5Qp" width="690" height="219" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d2a22b93a20151f83f38d3d025e0b9fe0724d5d_2_690x219.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d2a22b93a20151f83f38d3d025e0b9fe0724d5d_2_1035x328.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d2a22b93a20151f83f38d3d025e0b9fe0724d5d_2_1380x438.png 2x" data-dominant-color="141412"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2111×671 174 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
If I replace <code>logging</code> with <code>loguru</code> as suggested by ibro45, the weird behavior disappeared in <code>torch1.9</code>.</p>
<p>I wonder if we are using the same test case. Can you share your test files?</p>

pritamdamania87: <p>I was using a simpler test case, this is the script I was running:</p>
<pre><code class="lang-auto">import logging
import os
import sys
import torch.distributed as dist
import torch
import torch.nn.parallel
import torch.nn as nn

os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '12355'
dist.init_process_group('nccl', rank=0, world_size=1)
print(logging.root.hasHandlers())
model = torch.nn.parallel.DistributedDataParallel(nn.Linear(10, 10).cuda(),  device_ids=[0])
input = torch.rand(10, 10).cuda()
model(input)

class NoOp:
    def __getattr__(self, *args):
        def no_op(*args, **kwargs):
            """Accept every signature by doing non-operation."""
            pass

        return no_op


def get_logger(log_dir, log_name=None, resume="", is_rank0=True):
    """Get the program logger.

    Args:
        log_dir (str): The directory to save the log file.
        log_name (str, optional): The log filename. If None, it will use the main
            filename with ``.log`` extension. Default is None.
        resume (str): If False, open the log file in writing and reading mode.
            Else, open the log file in appending and reading mode; Default is "".
        is_rank0 (boolean): If True, create the normal logger; If False, create the null
           logger, which is useful in DDP training. Default is True.
    """
    if is_rank0:
        logger = logging.getLogger(__name__)
        logger.setLevel(level=logging.INFO)

        # StreamHandler
        stream_handler = logging.StreamHandler(sys.stdout)
        stream_handler.setLevel(level=logging.INFO)
        logger.addHandler(stream_handler)

        # FileHandler
        mode = "w+" if resume == "False" else "a+"
        if log_name is None:
            log_name = os.path.basename(sys.argv[0]).split(".")[0] + (".log")
        file_handler = logging.FileHandler(os.path.join(log_dir, log_name), mode=mode)
        file_handler.setLevel(level=logging.INFO)
        logger.addHandler(file_handler)
    else:
        logger = NoOp()

    return logger

logger = get_logger('/tmp/', 'foo')

logger.info('hello')
</code></pre>

hkz: <p>Thanks for sharing! I dive into my test code and find the function <code>.backdward()</code> in the training loop (&gt;=2) is the cause of this weird behavior. Based on your minimal test code, I write the following scripts:</p>
<pre><code class="lang-auto">import logging
import os
import sys
import torch.distributed as dist
import torch
import torch.nn.parallel
import torch.nn as nn

os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '12355'
dist.init_process_group('nccl', rank=0, world_size=1)
print(logging.root.hasHandlers())
model = torch.nn.parallel.DistributedDataParallel(nn.Linear(10, 10).cuda(),  device_ids=[0])
criterion = nn.CrossEntropyLoss().cuda()

class NoOp:
    def __getattr__(self, *args):
        def no_op(*args, **kwargs):
            """Accept every signature by doing non-operation."""
            pass

        return no_op


def get_logger(log_dir, log_name=None, resume="", is_rank0=True):
    """Get the program logger.

    Args:
        log_dir (str): The directory to save the log file.
        log_name (str, optional): The log filename. If None, it will use the main
            filename with ``.log`` extension. Default is None.
        resume (str): If False, open the log file in writing and reading mode.
            Else, open the log file in appending and reading mode; Default is "".
        is_rank0 (boolean): If True, create the normal logger; If False, create the null
           logger, which is useful in DDP training. Default is True.
    """
    if is_rank0:
        logger = logging.getLogger(__name__)
        logger.setLevel(level=logging.INFO)

        # StreamHandler
        stream_handler = logging.StreamHandler(sys.stdout)
        stream_handler.setLevel(level=logging.INFO)
        logger.addHandler(stream_handler)

        # FileHandler
        mode = "w+" if resume == "False" else "a+"
        if log_name is None:
            log_name = os.path.basename(sys.argv[0]).split(".")[0] + (".log")
        file_handler = logging.FileHandler(os.path.join(log_dir, log_name), mode=mode)
        file_handler.setLevel(level=logging.INFO)
        logger.addHandler(file_handler)
    else:
        logger = NoOp()

    return logger

logger = get_logger('/tmp/', 'foo')

logger.info('hello')

num_epochs = 2
backward = False
logger.info('num_epochs: {}, backward: {}'.format(num_epochs, backward))
for i in range(num_epochs):
    input = torch.rand(10, 10).cuda()
    target = torch.tensor([1]*10).cuda()
    output = model(input)
    loss = criterion(output, target)
    if backward:
        loss.backward()
    logger.info('training...')
</code></pre>
<ul>
<li>pytorch version 1.9.0, num_epochs&gt;=2, backward=True:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/00afc98ce948d5bfc1c2be79c81ea1511dc0181d.png" data-download-href="https://discuss.pytorch.org/uploads/default/00afc98ce948d5bfc1c2be79c81ea1511dc0181d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00afc98ce948d5bfc1c2be79c81ea1511dc0181d_2_690x219.png" alt="image" data-base62-sha1="64CB98FzbAQSDkfViwC0djzqC9" width="690" height="219" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00afc98ce948d5bfc1c2be79c81ea1511dc0181d_2_690x219.png, https://discuss.pytorch.org/uploads/default/original/3X/0/0/00afc98ce948d5bfc1c2be79c81ea1511dc0181d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/0/00afc98ce948d5bfc1c2be79c81ea1511dc0181d.png 2x" data-dominant-color="0A0C09"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">948×302 37.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
<li>pytorch version 1.9.0, num_epochs&gt;=2, backward=False:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d145c4fcdab174dee4903fcd71ac6b3dba05c3f1.png" data-download-href="https://discuss.pytorch.org/uploads/default/d145c4fcdab174dee4903fcd71ac6b3dba05c3f1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d145c4fcdab174dee4903fcd71ac6b3dba05c3f1_2_690x203.png" alt="image" data-base62-sha1="tRjgQCbzp7Ef6BKeCd1yT02yQq5" width="690" height="203" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d145c4fcdab174dee4903fcd71ac6b3dba05c3f1_2_690x203.png, https://discuss.pytorch.org/uploads/default/original/3X/d/1/d145c4fcdab174dee4903fcd71ac6b3dba05c3f1.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/1/d145c4fcdab174dee4903fcd71ac6b3dba05c3f1.png 2x" data-dominant-color="0A0D09"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">946×279 35.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
<li>pytorch version 1.9.0, num_epochs=1, backward=True:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/d/dd35a05a5c61fab9d59b29650819ce9bd87a2759.png" data-download-href="https://discuss.pytorch.org/uploads/default/dd35a05a5c61fab9d59b29650819ce9bd87a2759" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd35a05a5c61fab9d59b29650819ce9bd87a2759_2_690x183.png" alt="image" data-base62-sha1="vyUpJ5CefGS6rNT63a8ImXTLWm5" width="690" height="183" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd35a05a5c61fab9d59b29650819ce9bd87a2759_2_690x183.png, https://discuss.pytorch.org/uploads/default/original/3X/d/d/dd35a05a5c61fab9d59b29650819ce9bd87a2759.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/d/dd35a05a5c61fab9d59b29650819ce9bd87a2759.png 2x" data-dominant-color="0A0E0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">950×253 34.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
<li>pytorch version 1.6.0, num_epochs&gt;=2, backward=True:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/977010272b16c9292d50353cce90f8ba9f7d0203.png" data-download-href="https://discuss.pytorch.org/uploads/default/977010272b16c9292d50353cce90f8ba9f7d0203" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/977010272b16c9292d50353cce90f8ba9f7d0203_2_690x226.png" alt="image" data-base62-sha1="lBGbUuUWFmEsTdMEKgze6dS5frB" width="690" height="226" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/977010272b16c9292d50353cce90f8ba9f7d0203_2_690x226.png, https://discuss.pytorch.org/uploads/default/original/3X/9/7/977010272b16c9292d50353cce90f8ba9f7d0203.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/7/977010272b16c9292d50353cce90f8ba9f7d0203.png 2x" data-dominant-color="0A0E0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">856×281 34.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ul>