Repository: DataBiosphere/toil

Issue:
Move logging message for internal Toil job operations to DEBUG
cc @arkal @hannes-ucsc 

This PR is spun out of the discussion in #1048. In CWL runs we see a large number of logging messages devoted to internal Toil processes output to INFO:

```
Got message from job at time 07-13-2016 10:15:21: Successfully deleted local copies of file with ID 'h/y/tmpMJXEcB.tmp'.
Got message from job at time 07-13-2016 10:15:21: Successfully deleted local copies of file with ID 'j/W/tmp2cwAzR.tmp'.
Got message from job at time 07-13-2016 10:15:21: Successfully deleted local copies of file with ID 'G/2/tmpGT1jJU.tmp'.
```

From the discussion, the work plan is to only report these messages in INFO if called through the Toil API, and if called from internal Toil (or Toil/CWL) processes report them as DEBUG.


Discussion:
Proposed solutions:
- add `=quiet=False` kwarg to the fileStore methods (@arkal)
- add wrappers, e.g. `deleteGlobalFile()` makes the log entry and calls `_quietDeleteGlobalFile()` which does the real work but doesn't log (@hannes_ucsc)
- add wrappers dynamically with magic methods (@hannes_ucsc)
- add command line flag, e.g. --fileStoreLogLevel (@arkal)
- use `inspect` to determine whether calling frame belongs to Toil (@hannes_ucsc)
- use a counter to track entering/leaving Toil code, change log verbosity accordingly (@hannes_ucsc)

Hannes, Christopher and all;
I wanted to follow up on this discussion now that we've gotten CWL up to date with Toil. I'm doing testing on real runs and still struggle with keeping track of pipeline progress at a high level. For a cwltoil job run, the logging around an operation looks like:
```
kunkel 2016-12-23 11:57:04,043 MainThread INFO toil.leader: Starting the main loop
kunkel 2016-12-23 11:57:05,555 Thread-42 INFO toil.statsAndLogging: Got message from job at time 12-23-2016 11:57:05: Job 'CWLWorkflow' J/8/job1rtEsQ used 0.00% (0.0 B [0B] used, 2.0 GB [2147483648B] requested) at the end of its run.
kunkel 2016-12-23 11:57:07,069 Thread-42 INFO toil.statsAndLogging: Got message from job at time 12-23-2016 11:57:07: Job 'CWLScatter' e/p/jobFqHxl3 used 0.00% (0.0 B [0B] used, 2.0 GB [2147483648B] requested) at the end of its run.
kunkel 2016-12-23 11:57:07,071 Thread-42 INFO toil.statsAndLogging: Got message from job at time 12-23-2016 11:57:07: Job 'CWLScatter' Z/b/jobRxjcX2 used 0.00% (0.0 B [0B] used, 2.0 GB [2147483648B] requested) at the end of its run.
kunkel 2016-12-23 11:57:16,687 Thread-42 INFO toil.statsAndLogging: Got message from job at time 12-23-2016 11:57:16: Job 'file:///home/chapmanb/drive/work/cwl/test_bcbio_cwl/run_info-cwl-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl L/N/jobjopV6K used 0.00% (104.0 KB [106496B] used, 2.0 GB [2147483648B] requested) at the end of its run.
kunkel 2016-12-23 11:57:16,687 Thread-42 INFO toil.statsAndLogging: Got message from job at time 12-23-2016 11:57:16: Successfully deleted local copies of file with ID 'q/F/tmpIRJx_G.tmp'.
kunkel 2016-12-23 11:57:16,687 Thread-42 INFO toil.statsAndLogging: Got message from job at time 12-23-2016 11:57:16: Successfully deleted local copies of file with ID 't/U/tmpUHV6fy.tmp'.
kunkel 2016-12-23 11:57:16,687 Thread-42 INFO toil.statsAndLogging: Got message from job at time 12-23-2016 11:57:16: Successfully deleted local copies of file with ID 'i/5/tmpItZXEN.tmp'.
kunkel 2016-12-23 11:57:16,687 Thread-42 INFO toil.statsAndLogging: Got message from job at time 12-23-2016 11:57:16: Successfully deleted local copies of file with ID '9/V/tmpI2FCLB.tmp'.
kunkel 2016-12-23 11:57:16,688 Thread-42 INFO toil.statsAndLogging: Got message from job at time 12-23-2016 11:57:16: Successfully deleted local copies of file with ID 'P/l/tmpQ2aA5b.tmp'.
kunkel 2016-12-23 11:57:16,688 Thread-42 INFO toil.statsAndLogging: Got message from job at time 12-23-2016 11:57:16: Successfully deleted local copies of file with ID 'z/c/tmpPTk5Bb.tmp'.
kunkel 2016-12-23 11:57:16,688 Thread-42 INFO toil.statsAndLogging: Got message from job at time 12-23-2016 11:57:16: Successfully deleted local copies of file with ID 'U/U/tmpokIXW4.tmp'.
```
So there are two kinds of non-helpful messages. The first are internal CWL calls:
```
kunkel 2016-12-23 11:57:05,555 Thread-42 INFO toil.statsAndLogging: Got message from job at time 12-23-2016 11:57:05: Job 'CWLWorkflow' J/8/job1rtEsQ used 0.00% (0.0 B [0B] used, 2.0 GB 
```
the second are deletion messages:
```
kunkel 2016-12-23 11:57:16,688 Thread-42 INFO toil.statsAndLogging: Got message from job at time 12-23-2016 11:57:16: Successfully deleted local copies of file with ID 'U/U/tmpokIXW4.tmp'.
```
What I'd like to do is remove these so we only see the one informative line about `prep_samples` finishing. Ideally we'd also have a logging message when this job starts.

Do you have tips/clues about how best to dig into this? I'm happy to work on but welcome suggestions about how best to do it the preferred Toil way so I can working within your plans. Thanks much.
@chapmanb I'm in favor of moving the two non-helpful log statements to DEBUG as long as we get a WARNING level message when a job uses over 100% of the disk for the first message.

It actually might be difficult to log every time a job starts just because of Toil's logging mechanisms. The job is run on the worker & logs are usually only propagated when it finishes. The logToMaster mechanism is likely more what we would want, but I think that would be racy since it is independent from the way a job's success/failure is reported to the leader. We might end up with messages indicating failure before we get the message saying it has failed.

Resolving patch:
diff --git a/src/toil/fileStore.py b/src/toil/fileStore.py
--- a/src/toil/fileStore.py
+++ b/src/toil/fileStore.py
@@ -491,7 +491,7 @@ def open(self, job):
                                            disk=diskUsed,
                                            humanRequestedDisk=bytes2human(jobReqs),
                                            requestedDisk=jobReqs))
-            self.logToMaster(logString, level=logging.INFO)
+            self.logToMaster(logString, level=logging.DEBUG)
             if diskUsed > jobReqs:
                 self.logToMaster("Job used more disk than requested. Please reconsider modifying "
                                  "the user script to avoid the chance  of failure due to "
@@ -628,7 +628,7 @@ def readGlobalFile(self, fileStoreID, userPath=None, cache=True, mutable=None):
         # userPath. Cache operations can only occur on local files.
         with self.cacheLock() as lockFileHandle:
             if fileIsLocal and self._fileIsCached(fileStoreID):
-                logger.info('CACHE: Cache hit on file with ID \'%s\'.' % fileStoreID)
+                logger.debug('CACHE: Cache hit on file with ID \'%s\'.' % fileStoreID)
                 assert not os.path.exists(localFilePath)
                 if mutable:
                     shutil.copyfile(cachedFileName, localFilePath)
@@ -719,10 +719,10 @@ def readGlobalFileStream(self, fileStoreID):
 
         # If fileStoreID is in the cache provide a handle from the local cache
         if self._fileIsCached(fileStoreID):
-            logger.info('CACHE: Cache hit on file with ID \'%s\'.' % fileStoreID)
+            logger.debug('CACHE: Cache hit on file with ID \'%s\'.' % fileStoreID)
             return open(self.encodedFileID(fileStoreID), 'r')
         else:
-            logger.info('CACHE: Cache miss on file with ID \'%s\'.' % fileStoreID)
+            logger.debug('CACHE: Cache miss on file with ID \'%s\'.' % fileStoreID)
             return self.jobStore.readFileStream(fileStoreID)
 
     def deleteLocalFile(self, fileStoreID):
@@ -797,9 +797,9 @@ def deleteLocalFile(self, fileStoreID):
                         os.remove(cachedFile)
                         cacheInfo.cached -= fileSize
                 self.logToMaster('Successfully deleted cached copy of file with ID '
-                                 '\'%s\'.' % fileStoreID)
+                                 '\'%s\'.' % fileStoreID, level=logging.DEBUG)
             self.logToMaster('Successfully deleted local copies of file with ID '
-                             '\'%s\'.' % fileStoreID)
+                             '\'%s\'.' % fileStoreID, level=logging.DEBUG)
 
     def deleteGlobalFile(self, fileStoreID):
         jobStateIsPopulated = False
@@ -823,7 +823,7 @@ def deleteGlobalFile(self, fileStoreID):
         # Add the file to the list of files to be deleted once the run method completes.
         self.filesToDelete.add(fileStoreID)
         self.logToMaster('Added file with ID \'%s\' to the list of files to be' % fileStoreID +
-                         ' globally deleted.')
+                         ' globally deleted.', level=logging.DEBUG)
 
     # Cache related methods
     @contextmanager
@@ -1029,11 +1029,11 @@ def addToCache(self, localFilePath, jobStoreFileID, callingFunc, mutable=None):
                     self.returnFileSize(jobStoreFileID, localFilePath, lockFileHandle,
                                         fileAlreadyCached=False)
                 if callingFunc == 'read':
-                    logger.info('CACHE: Read file with ID \'%s\' from the cache.' %
-                                jobStoreFileID)
+                    logger.debug('CACHE: Read file with ID \'%s\' from the cache.' %
+                                 jobStoreFileID)
                 else:
-                    logger.info('CACHE: Added file with ID \'%s\' to the cache.' %
-                                jobStoreFileID)
+                    logger.debug('CACHE: Added file with ID \'%s\' to the cache.' %
+                                 jobStoreFileID)
 
     def returnFileSize(self, fileStoreID, cachedFileSource, lockFileHandle,
                        fileAlreadyCached=False):
@@ -1603,7 +1603,7 @@ def open(self, job):
                                            disk=diskUsed,
                                            humanRequestedDisk=bytes2human(jobReqs),
                                            requestedDisk=jobReqs))
-            self.logToMaster(logString, level=logging.INFO)
+            self.logToMaster(logString, level=logging.DEBUG)
             if diskUsed > jobReqs:
                 self.logToMaster("Job used more disk than requested. Cconsider modifying the user "
                                  "script to avoid the chance of failure due to incorrectly "
diff --git a/src/toil/leader.py b/src/toil/leader.py
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -25,6 +25,7 @@
 from collections import namedtuple
 
 from bd2k.util.expando import Expando
+from bd2k.util.humanize import bytes2human
 
 from toil import resolveEntryPoint
 from toil.jobStores.abstractJobStore import NoSuchJobException
@@ -140,6 +141,10 @@ def __init__(self, config, batchSystem, provisioner, jobStore, rootJob, jobCache
         self.potentialDeadlockedJobs = None
         self.potentialDeadlockTime = 0
 
+        # internal jobs we should not expose at top level debugging
+        self.debugJobNames = ("CWLJob", "CWLWorkflow", "CWLScatter", "CWLGather",
+                              "ResolveIndirect")
+
     def run(self):
         """
         This runs the leader process to issue and manage jobs.
@@ -420,11 +425,12 @@ def innerLoop(self):
                                 "for job %s", jobID)
                 else:
                     if result == 0:
-                        logger.debug('Batch system is reporting that the job %s ended successfully',
-                                     updatedJob)
+                        cur_logger = (logger.debug if str(updatedJob.jobName).startswith(self.debugJobNames)
+                                      else logger.info)
+                        cur_logger('Job ended successfully: %s', updatedJob)
                     else:
-                        logger.warn('Batch system is reporting that the job %s failed with exit value %i',
-                                    updatedJob, result)
+                        logger.warn('Job failed with exit value %1: %s',
+                                    result, updatedJob)
                     self.processFinishedJob(jobID, result, wallTime=wallTime)
 
             else:
@@ -514,10 +520,12 @@ def issueJob(self, jobNode):
                                     self.jobStoreLocator, jobNode.jobStoreID))
         jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
         self.jobBatchSystemIDToIssuedJob[jobBatchSystemID] = jobNode
-        logger.debug("Issued job with job store ID: %s and job batch system ID: "
-                     "%s and cores: %.2f, disk: %.2f, and memory: %.2f",
-                     jobNode.jobStoreID, str(jobBatchSystemID), jobNode.cores,
-                     jobNode.disk, jobNode.memory)
+        cur_logger = (logger.debug if jobNode.jobName.startswith(self.debugJobNames)
+                      else logger.info)
+        cur_logger("Issued job %s with job batch system ID: "
+                   "%s and cores: %s, disk: %s, and memory: %s",
+                   jobNode, str(jobBatchSystemID), int(jobNode.cores),
+                   bytes2human(jobNode.disk), bytes2human(jobNode.memory))
 
     def issueJobs(self, jobs):
         """
@@ -896,4 +904,4 @@ def _updatePredecessorStatus(self, jobStoreID):
                     self.toilState.updatedJobs.add((predecessorJob, 0))
 
                     logger.debug('Job %s has all its non-service successors completed or totally '
-                                 'failed', predecessorJob)
\ No newline at end of file
+                                 'failed', predecessorJob)


Test patch: