Repository: DataBiosphere/toil

Issue:
Excessive logging in clusters
The logs from rna seq run contain too many repeated prints of the following message:

```
toil.batchSystems.mesos.batchSystem: Although there are queued jobs, none of them could be run with theoffers extended to the framework.
```

This is being logged once for every offer and therefore node in the cluster. Preventing this will cut down on excessive logging. 

Manifest:

```
#   Edit this manifest to include information pertaining to each sample to be run.
#   There are 4 tab-separated columns: filetype, paired/unpaired, UUID, URL(s) to sample
#
#   filetype    Filetype of the sample. Options: "tar" or "fq", for tarball/tarfile or fastq/fastq.gz
#   paired      Indicates whether the data is paired or single-ended. Options:  "paired" or "single"
#   UUID        This should be a unique identifier for the sample to be processed
#   URL         A URL ['http://', 'file://', 's3://', 'ftp://', 'gnos://'] pointing to the sample
#
#   If sample is being submitted as a fastq pair, provide two URLs separated by a comma.
#
#   Examples of several combinations are provided below. Lines beginning with # are ignored.
#
#   tar paired  UUID_1  file:///path/to/sample.tar
#   fq  paired  UUID_2  file:///path/to/R1.fq.gz,file:///path/to/R2.fq.gz
#   tar single  UUID_3  http://sample-depot.com/single-end-sample.tar
#   tar paired  UUID_4  s3://my-bucket-name/directory/paired-sample.tar.gz
#   fq  single  UUID_5  s3://my-bucket-name/directory/single-end-file.fq
#
#   Place your samples below, one per line.
tar     paired  DTB-137-Progression     https://s3-us-west-2.amazonaws.com/cgl-driver-projects-encrypted/ucsf-pnoc/ucsf_issue53_input/DTB-137-Progression.tar
tar     paired  DTB-159_Baseline        https://s3-us-west-2.amazonaws.com/cgl-driver-projects-encrypted/ucsf-pnoc/ucsf_issue53_input/DTB-159_Baseline.tar
tar     paired  DTB-181-Baseline        https://s3-us-west-2.amazonaws.com/cgl-driver-projects-encrypted/ucsf-pnoc/ucsf_issue53_input/DTB-181-Baseline.tar
```

Config:

```
# RNA-seq CGL Pipeline configuration file
# This configuration file is formatted in YAML. Simply write the value (at least one space) after the colon.
# Edit the values in this configuration file and then rerun the pipeline: "toil-rnaseq run"
# Just Kallisto or STAR/RSEM can be run by supplying only the inputs to those tools
# Local inputs follow the URL convention: file:///full/path/to/input
# Comments (beginning with #) do not need to be removed. Optional parameters may be left blank.
##############################################################################################################
# Required: URL ['http://', 'file://', 's3://', 'ftp://', 'gnos://'] to index tarball used by STAR
star-index: s3://cgl-pipeline-inputs/rnaseq_cgl/starIndex_hg38_no_alt.tar.gz

# Required: URL ['http://', 'file://', 's3://', 'ftp://', 'gnos://'] to kallisto index file.
kallisto-index: s3://cgl-pipeline-inputs/rnaseq_cgl/kallisto_hg38.idx

# Required: URL ['http://', 'file://', 's3://', 'ftp://', 'gnos://'] to reference tarball used by RSEM
rsem-ref: s3://cgl-pipeline-inputs/rnaseq_cgl/rsem_ref_hg38_no_alt.tar.gz

# NOTE: Pipeline requires at least one output option
# Optional: Provide a full file path (/path/to/output-dir) where results will appear
output-dir:

# Optional: Provide an s3 path (s3://bucket/dir) where results will appear
s3-output-dir: s3://cgl-driver-projects/wcdt/medbook_UCSF_RNA-seq.20160714.53.cgl2

# Optional: If true, will preprocess samples with cutadapt using adapter sequences.
cutadapt: true

# Optional: If true, will run FastQC and include QC in sample output
fastqc: true

# Adapter sequence to trim. Defaults set for Illumina
fwd-3pr-adapter: AGATCGGAAGAG

# Adapter sequence to trim (for reverse strand). Defaults set for Illumina
rev-3pr-adapter: AGATCGGAAGAG

# Optional: Provide a full path to a 32-byte key used for SSE-C Encryption in Amazon
ssec: /home/mesosbox/shared/shared/master.key

# Optional: Provide a full path to a CGHub Key used to access GNOS hosted data
gtkey:

# Optional: If true, saves the wiggle file (.bg extension) output by STAR
wiggle: true

# Optional: If true, saves the aligned bam (by coordinate) produced by STAR
# You must also specify an ssec key if you want to upload to the s3-output-dir
save-bam: true

# Optional: If true, uses resource requirements appropriate for continuous integration
ci-test:

```

running with toil-script v2.0.1

See attached [log file](https://github.com/BD2KGenomics/toil-scripts/files/384865/log.txt).



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/mesos/batchSystem.py b/src/toil/batchSystems/mesos/batchSystem.py
--- a/src/toil/batchSystems/mesos/batchSystem.py
+++ b/src/toil/batchSystems/mesos/batchSystem.py
@@ -123,6 +123,13 @@ def __init__(self, config, maxCores, maxMemory, maxDisk, masterAddress,
         self.unusedJobID = itertools.count()
         self.lastReconciliation = time.time()
         self.reconciliationPeriod = 120
+
+        # These control how frequently to log a message that would indicate if no jobs are
+        # currently able to run on the offers given. This can happen if the cluster is busy
+        # or if the nodes in the cluster simply don't have enough resources to run the jobs
+        self.lastTimeOfferLogged = 0
+        self.logPeriod = 30  # seconds
+
         self._startDriver()
 
     def issueBatchJob(self, command, memory, cores, disk, preemptable):
@@ -343,6 +350,7 @@ def resourceOffers(self, driver, offers):
             self._declineAllOffers(driver, offers)
             return
 
+        unableToRun = True
         # Right now, gives priority to largest jobs
         for offer in offers:
             runnableTasks = []
@@ -393,16 +401,23 @@ def resourceOffers(self, driver, offers):
                 runnableTasks.extend(runnableTasksOfType)
             # Launch all runnable tasks together so we only call launchTasks once per offer
             if runnableTasks:
+                unableToRun = False
                 driver.launchTasks(offer.id, runnableTasks)
                 for task in runnableTasks:
                     self._updateStateToRunning(offer, task)
                     log.info('Launched Mesos task %s.', task.task_id.value)
             else:
-                log.info('Although there are queued jobs, none of them could be run with offer %s '
-                         'extended to the framework. Enable debug logging to see details '
-                         'about tasks queued and offers made.', offer.id)
+                log.debug('Although there are queued jobs, none of them could be run with offer %s '
+                          'extended to the framework.', offer.id)
                 driver.declineOffer(offer.id)
 
+        if unableToRun and time.time() > (self.lastTimeOfferLogged + self.logPeriod):
+            self.lastTimeOfferLogged = time.time()
+            log.info('Although there are queued jobs, none of them were able to run in '
+                     'any of the offers extended to the framework. There are currently '
+                     '%i jobs running. Enable debug level logging to see more details about '
+                     'job types and offers received.', len(self.runningJobMap))
+
     def _trackOfferedNodes(self, offers):
         for offer in offers:
             nodeAddress = socket.gethostbyname(offer.hostname)


Test patch: