Repository: DataBiosphere/toil

Issue:
Suppressing toil logs at all levels so that pipeline-specific logs can get to leader
I am trying to figure out how I can have my toil pipelines log only statements I produce.

If I set `logLevel=INFO`, then my logs will get hugely spammed from toil internal logs. So, for my purposes, I set `loglevel=WARNING`. This means that when I write my own log messages, if I want the leader to report them even when a job does not fail, I have to set the `logLevel` higher than `INFO`. But then this means that I will be potentially misleading when reporting success messages at a level like `WARNING` or `ERROR`.

Is there a way to separate the loggers so that I can have my pipeline-level logs go to a different handler?



Discussion:
First of all, define "hugely spammed". Have you ever seen the `--logDebug` logs? I wonder what term you would use for those ;-) At INFO level it should still be possible to just grep the logs for the interesting bits.

Anyhow, the logging framework can be configured through a separate config file. Thresholds can be set per logger and each module in Toil uses a different logger. I think that should get you there. We will have to add a little bit of support for that in Toil as it currently invokes `logging.basicConfig` instead of `logging.config.fileConfig`.

Possible complications: Keep in mind that unless you use logToMaster() or --logDebug, worker logs from successful jobs are completely suppressed.

Hugely spammed is largely the result of the fact that I am running a dozen Toil pipelines concurrently, all logging to the same place (the same logging handlers instantiated by my Luigi wrapper).

I am trying to create a logging scenario that will allow me to provide informative logging to a user without the need to grep. One challenge here is that since this is wrapped in Luigi, Luigi already uses the `fileConfig` process to ingest a logging config file. I guess I will need to manually mark each Toil handler within this config file and set the levels individually there? Right now, I control the Toil log level with the root handler.

> I guess I will need to manually mark each Toil handler within this config file and set the levels individually there?

Since each module's logger uses the module's package path as the logger name, you should just be able to specify a level for the `"toil"` logger. That should affect the `"toil.foo"` and `"toil.bar"` child loggers. I have only found scant evidence for this, though. The docs speak repeatedly about a "logger hierarchy" but never defines how that hierarchy is set up.

I will do some testing and let you know.

On Thu, Aug 25, 2016 at 2:03 PM Hannes Schmidt notifications@github.com
wrote:

> I guess I will need to manually mark each Toil handler within this config
> file and set the levels individually there?
> 
> Since each module's logger uses the module's package path as the logger
> name, you should just be able to specify a level for the "toil" logger.
> That should affect the "toil.foo" and "toil.bar" child loggers. I have
> only found scant evidence for this, though. The docs speak repeatedly about
> a "logger hierarchy" but never defines how that hierarchy is set up.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> https://github.com/BD2KGenomics/toil/issues/1120#issuecomment-242539867,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AHdLXY-UWtINKSfRGeBxyk0dd7AsrB1mks5qjgM0gaJpZM4JtYmX
> .

A quick test so far. Setting a `toil` logging module does propagate the new log level to lower levels, but specifying a lower level does not override the higher one. Here is the config file I just tested with:

``` [formatters]
keys: default

[handlers]
keys: console, logfile

[loggers]
keys: root, cat, generate_hints_db, luigi-interface, toil, toil-batchSystems, toil-common

[formatter_default]
format: %(asctime)s %(name)-5s %(levelname)-5s %(message)s
datefmt: %m-%d %H:%M:%S

[handler_console]
class: StreamHandler
args: []
formatter: default

[handler_logfile]
class: FileHandler
args: ['log.txt']
formatter: default

[logger_root]
level: ERROR
handlers: console
qualname: root

[logger_cat]
level: INFO
handlers: console
qualname: cat
propagate: 0

[logger_generate_hints_db]
level: INFO
handlers: console
qualname: generate_hints_db
propagate: 0

[logger_luigi-interface]
level: WARNING
handlers: console
qualname: luigi-interface
propagate: 1

[logger_toil]
level: INFO
handlers: console
qualname: toil

[logger_toil-batchSystems]
level: ERROR
handlers: console
qualname: toil-batchSystems

[logger_toil-common]
level: ERROR
handlers: console
qualname: toil-common
```

And a snippet of log:

```
08-25 14:10:52 toil.leader INFO  Starting the main loop
08-25 14:10:52 toil.leader INFO  Starting the main loop
08-25 14:10:52 toil.batchSystems.singleMachine INFO  Executing command: '_toil_worker /hive/users/ifiddes/Comparative-Annotation-Toolkit/CAT/test_work_v2/toil/transcript_alignment/monDom5/jobStore F/T/jobSRhQwv'.
08-25 14:10:52 toil.batchSystems.singleMachine INFO  Executing command: '_toil_worker /hive/users/ifiddes/Comparative-Annotation-Toolkit/CAT/test_work_v2/toil/transcript_alignment/monDom5/jobStore F/T/jobSRhQwv'.
```

Removing those last two loggers has no effect on the output. Removing the `toil` logger causes Toil to default to the `root` logger, which is at level `ERROR`.

Are you sure it's `toil-batchSystems` and not `toil.batchSystems`?

If you use a period, you get an error. I got the dash syntax from the luigi config, which you can see in the example config file they provide.

Must be a quirk of the configuration file syntax:

```
Python 2.7.10 (default, Oct 23 2015, 19:19:21) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
In[2]: import logging as l
In[3]: l.basicConfig()
In[4]: p=l.getLogger('a')
In[5]: c=l.getLogger('a.b')
In[6]: g=l.getLogger('a.b.c')
In[7]: p.info('info')
In[8]: c.info('info')
In[9]: g.info('info')
In[10]: p.setLevel(l.INFO)
In[11]: p.info('info')
INFO:a:info
In[12]: c.info('info')
INFO:a.b:info
In[13]: g.info('info')
INFO:a.b.c:info
In[14]: c.setLevel(l.WARNING)
In[15]: p.info('info')
INFO:a:info
In[16]: c.info('info')
In[18]: g.info('info')
```

Interesting. Here is the traceback if I try to use a period:

```
08-25 14:09:40 luigi-interface ERROR Uncaught exception in luigi
Traceback (most recent call last):
  File "/cluster/home/ifiddes/anaconda2/lib/python2.7/site-packages/luigi/retcodes.py", line 61, in run_with_retcodes
    worker = luigi.interface._run(argv)['worker']
  File "/cluster/home/ifiddes/anaconda2/lib/python2.7/site-packages/luigi/interface.py", line 238, in _run
    return _schedule_and_run([cp.get_task_obj()], worker_scheduler_factory)
  File "/cluster/home/ifiddes/anaconda2/lib/python2.7/site-packages/luigi/interface.py", line 168, in _schedule_and_run
    setup_interface_logging(logging_conf)
  File "/cluster/home/ifiddes/anaconda2/lib/python2.7/site-packages/luigi/interface.py", line 60, in setup_interface_logging
    logging.config.fileConfig(conf_file, disable_existing_loggers=False)
  File "/cluster/home/ifiddes/anaconda2/lib/python2.7/logging/config.py", line 86, in fileConfig
    _install_loggers(cp, handlers, disable_existing_loggers)
  File "/cluster/home/ifiddes/anaconda2/lib/python2.7/logging/config.py", line 228, in _install_loggers
    qn = cp.get(sectname, "qualname")
  File "/cluster/home/ifiddes/anaconda2/lib/python2.7/ConfigParser.py", line 607, in get
    raise NoSectionError(section)
NoSectionError: No section: 'logger_toil.batchSystems'
```

`logger_toil-batchSystems` occurs twice in the file. Did you change both?

Oops, you are right. It no longer crashes, and it works! I can successfully suppress the batchSystems log. I now get a warning about the realtime logger

`08-25 14:44:46 toil.realtimeLogger INFO  Real-time logging disabled`

I also get messages from `toil.lib.bioio` that appear contradictory:

`08-25 14:44:46 toil.lib.bioio INFO  Logging set at level: WARNING`

The former is not a warning and is to be expected if you don't enable real time logging via the command line. The latter is indeed confusing.


Resolving patch:
diff --git a/src/toil/lib/bioio.py b/src/toil/lib/bioio.py
--- a/src/toil/lib/bioio.py
+++ b/src/toil/lib/bioio.py
@@ -33,9 +33,13 @@
 defaultLogLevel = logging.INFO
 logger = logging.getLogger(__name__)
 rootLogger = logging.getLogger()
+toilLogger = logging.getLogger('toil')
 
-def getLogLevelString():
-    return logging.getLevelName(rootLogger.getEffectiveLevel())
+
+def getLogLevelString(logger=None):
+    if logger is None:
+        logger = rootLogger
+    return logging.getLevelName(logger.getEffectiveLevel())
 
 __loggingFiles = []
 def addLoggingFileHandler(fileName, rotatingLogging=False):
@@ -49,11 +53,14 @@ def addLoggingFileHandler(fileName, rotatingLogging=False):
     rootLogger.addHandler(handler)
     return handler
 
-def setLogLevel(level, logger=rootLogger):
+
+def setLogLevel(level, logger=None):
     """
     Sets the log level to a given string level (like "INFO"). Operates on the
     root logger by default, but another logger can be specified instead.
     """
+    if logger is None:
+        logger = rootLogger
     level = level.upper()
     if level == "OFF": level = "CRITICAL"
     # Note that getLevelName works in both directions, numeric to textual and textual to numeric
@@ -137,10 +144,12 @@ def setLoggingFromOptions(options):
     rootLogger.setLevel(defaultLogLevel)
     if options.logLevel is not None:
         setLogLevel(options.logLevel)
-    logger.info("Logging set at level: %s" % getLogLevelString())
+    logger.info("Root logger is at level '%s', 'toil' logger at level '%s'.",
+                getLogLevelString(logger=rootLogger), getLogLevelString(logger=toilLogger))
     if options.logFile is not None:
-        addLoggingFileHandler(options.logFile, options.logRotating)
-        logger.info("Logging to file: %s" % options.logFile)
+        addLoggingFileHandler(options.logFile, rotatingLogging=options.logRotating)
+        logger.info("Logging to file '%s'." % options.logFile)
+
 
 def system(command):
     """


Test patch: