Pytorch 1.8 distributed mode will disable python logging module

coincheung: <p>Text code is like this:</p>
<pre><code class="lang-auto">import sys
import argparse
import logging
import torch.distributed as dist

def parse_args():
    parse = argparse.ArgumentParser()
    parse.add_argument('--local_rank', dest='local_rank', type=int, default=-1,)
    return parse.parse_args()

args = parse_args()
torch.cuda.set_device(args.local_rank)
dist.init_process_group(
    backend='nccl',
    #  init_method='tcp://127.0.0.1:{}'.format(cfg.port),
    init_method='env://',
    world_size=torch.cuda.device_count(),
    rank=args.local_rank
)


def set_logger():
    logfile = 'log.txt'
    log_level = logging.INFO
    FORMAT = '%(levelname)s %(filename)s(%(lineno)d): %(message)s'
    logging.basicConfig(level=log_level, format=FORMAT, filename=logfile)
    logging.root.addHandler(logging.StreamHandler(sys.stdout))

set_logger()
logger = logging.getLogger()
logger.info('acb')
</code></pre>
<p>If I launch the code like this:</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 test.py
</code></pre>
<p>I will see no log message. This is not same as pytorch 1.7.</p>

ptrblck: <p>Might be related to <a href="https://github.com/pytorch/pytorch/issues/51428">this issue</a> with a potential <a href="https://github.com/pytorch/pytorch/pull/52757">fix</a>.</p>

coincheung: <p>Hi, Is there any temporary method to bypass this problem in the user side until the upstream is fixed ?</p>

ptrblck: <p>You could either cherry-pick the mentioned PR and build from source or, I think, in your code base you could use <code>logging.basicConfig(..., force=True)</code>.</p>

coincheung: <p>adding force does not work for me, I use python3.7, should I update to python 3.8 ?</p>

ptrblck: <p>Yes, I think so. From the <a href="https://docs.python.org/3/library/logging.html">docs</a>:</p>
<blockquote>
<p>Changed in version 3.8: The <em>force</em> argument was added.</p>
</blockquote>

ani0075: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>This issue still exists though it seems fixes were pushed in the issues you cited.<br>
I am using Pytorch 1.9.0 CUDA 10.2 build and Python 3.7 on Linux.</p>
<p>Do you know how and where to initialize the logger so that it works correctly?</p>
<p>Thanks.</p>
<p>EDIT: Seems like I found an implementation which works. I have been scouring for a solution where logging works with torch.distributed and couldn’t seem to make it work. I also saw some other users facing issues with the same thing where the logger wouldn’t print anything to console/file.</p>
<p>I looked at the logger implementation here</p><aside class="onebox githubblob" data-onebox-src="https://github.com/stanford-futuredata/pytorch-distributed/blob/c8c25e82b4f49ff8a4139283d1d47a435ede04c6/training/logger.py">
  <header class="source">

      <a href="https://github.com/stanford-futuredata/pytorch-distributed/blob/c8c25e82b4f49ff8a4139283d1d47a435ede04c6/training/logger.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/stanford-futuredata/pytorch-distributed/blob/c8c25e82b4f49ff8a4139283d1d47a435ede04c6/training/logger.py" target="_blank" rel="noopener nofollow ugc">stanford-futuredata/pytorch-distributed/blob/c8c25e82b4f49ff8a4139283d1d47a435ede04c6/training/logger.py</a></h4>


    <pre><code class="lang-py">from tensorboardX import SummaryWriter
import torch
import time    
  
class TensorboardLogger:
  def __init__(self, output_dir, is_master=False):
    self.output_dir = output_dir
    self.current_step = 0
    if is_master: self.writer = SummaryWriter(self.output_dir)
    else: self.writer = NoOp()
    self.log('first', time.time())

  def log(self, tag, val):
    """Log value to tensorboard (relies on global_example_count being set properly)"""
    if not self.writer: return
    self.writer.add_scalar(tag, val, self.current_step)

  def update_step_count(self, batch_total):
    self.current_step += batch_total

</code></pre>


  This file has been truncated. <a href="https://github.com/stanford-futuredata/pytorch-distributed/blob/c8c25e82b4f49ff8a4139283d1d47a435ede04c6/training/logger.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>and this works. I am not sure what I was doing wrong but I will investigate further if I have time. It seems like there are subtle differences in which process and where (before or after dist.init_process_group) you initialize your logger.</p>