The PyTorch compiler hijacks my logger and floods it with DEBUG messages

vgoklani: <p>This is my logger.py file:</p>
<pre><code class="lang-python">import sys
import datetime
import pytz
import logging

eastern = pytz.timezone('US/Eastern')


class StdErrFilter(logging.Filter):
    def filter(self, rec):
        return rec.levelno in (logging.ERROR,)


class StdOutFilter(logging.Filter):
    def filter(self, rec):
        return rec.levelno in (logging.DEBUG, logging.INFO, logging.WARNING, logging.NOTSET)


def timetz(*args):
    return datetime.datetime.now(tz=eastern).timetuple()


def get_logger(caller, level=None, filename: str = None):
    logger = logging.getLogger(caller)

    format_str = '%(process)s - %(asctime)s - %(name)s - %(levelname)s - %(message)s'
    formatter = logging.Formatter(format_str)
    logging.Formatter.converter = timetz

    h1 = logging.StreamHandler(sys.stdout)
    h1.setLevel(logging.INFO)
    h1.setFormatter(formatter)
    h1.addFilter(StdOutFilter())

    h2 = logging.StreamHandler(sys.stderr)
    h2.setLevel(logging.ERROR)
    h2.setFormatter(formatter)
    h2.addFilter(StdErrFilter())

    logging.basicConfig(level=logging.NOTSET, format=format_str, handlers=[h1, h2])

    if filename:
        file_handler = logging.FileHandler(filename)
        file_handler.setLevel(logging.INFO)
        file_handler.setFormatter(formatter)
        logger.addHandler(file_handler)

    return logger

</code></pre>
<p>Which is instantiated like this:</p>
<pre><code class="lang-python">log = get_logger(__name__)
</code></pre>
<p>Unfortunately the pytorch compiler is flooding my logger with DEBUG messages. What’s the best way for us to manage this situation? I would like to capture useful messages (i.e. errors) from the compiler, but I don’t need the DEBUG messages. Thanks!</p>

ptrblck: <p>A feature request to add more control in the PyTorch logging util. is tracked <a href="https://github.com/pytorch/pytorch/issues/94788">here</a> so you might want to check if this request would also cover your use case.</p>

ezyang: <p>I’ll need to play around with your example code to give exact guidance, but PyTorch library is supposed to install its own handler which will prevent our logs from getting to the root logger by default, so figuring out why that didn’t work would have been my first step.</p>

vgoklani: <p><a class="mention" href="/u/ezyang">@ezyang</a> The issue here is that both <code>torch._dynamo</code> and <code>torch._inductor</code> are writing to the root logger. I haven’t seen pytorch-eager logs in the root logger. One thing to note, we weren’t having this issue prior to Feb 1, something changed in the nightly builds. For the code above, if we change the <code>level=logging.INFO</code> in the basicConfig, then we just get the INFO messages from the compiler, which are useful. But I would prefer to have control of our own DEBUG messages. thanks!</p>

ptrblck: <p>Since the issue you are seeing began at Feb 1st, <a href="https://github.com/pytorch/pytorch/pull/93252">this PR</a> could be related.</p>

mlazos: <p>This <a href="https://github.com/pytorch/pytorch/pull/94858" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/pull/94858</a> should fix this issue, let me know if you continue seeing it after this PR is merged.</p>