Repository: DataBiosphere/toil

Issue:
--stats is missing certain job functions
_From @jvivian on December 14, 2015 19:29_

Stats output:
http://pastebin.com/piy7dWSz

Script being run:
https://github.com/BD2KGenomics/toil-scripts/blob/master/rnaseq_cgl/rnaseq_cgl_pipeline.py

So in this case, STAR and RSEM functions didn't show up in the output.  Given the times, you can tell certain jobs got "merged".

_Copied from original issue: BD2KGenomics/toil-scripts#70_



Discussion:
_From @cgcloud-jenkins on December 15, 2015 2:18_

Okay, looking now.

On Mon, Dec 14, 2015 at 11:29 AM, John Vivian notifications@github.com
wrote:

> Stats output:
> http://pastebin.com/piy7dWSz
> 
> Script being run:
> 
> https://github.com/BD2KGenomics/toil-scripts/blob/master/rnaseq_cgl/rnaseq_cgl_pipeline.py
> 
> So in this case, STAR and RSEM functions didn't show up in the output.
> Given the times, you can tell certain jobs got "merged".
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/BD2KGenomics/toil-scripts/issues/70.
> 
> ## 
> 
> You received this message because you are subscribed to the Google Groups
> "cgloud-jenkins" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cgloud-jenkins+unsubscribe@googlegroups.com.
> To post to this group, send email to cgloud-jenkins@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cgloud-jenkins/BD2KGenomics/toil-scripts/issues/70%40github.com
> https://groups.google.com/d/msgid/cgloud-jenkins/BD2KGenomics/toil-scripts/issues/70%40github.com?utm_medium=email&utm_source=footer
> .
> For more options, visit https://groups.google.com/d/optout.

_From @benedictpaten on December 15, 2015 6:25_

So this is a bug created in the switch from xml to json for the stats logging. I'm assigning to CJ to fix as he is most familiar with the new code and also created the bug ;)

Previously in Job._execute we had the lines:

if stats != None:
            stats = ET.SubElement(stats, "job")
            stats.attrib["time"] = str(time.time() - startTime)
            ....

Note that we created a subelement (sorry for the sloppy reassignment of the variable), "job", and create attributes for it. This allows _execute to be run multiple times for the same stats object to contain a list of "job" elements, each representing a job run by the worker.

In the new code using the MagicExpando we have:

if stats != None:
            stats.jobs.time = str(time.time() - startTime)
            ....

Note there is only a single "jobs" attribute, and this is overwritten by the second and subsequent jobs, so that only the last job run on the worker gets reported.

One fix would be for "jobs" to be an array, with multiple values, one for each job containing a dictionary of the stats for the job. I could do this, but not sure about fixing the toilStats class.


Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -1301,11 +1301,15 @@ def _execute(self, jobWrapper, stats, localTempDir, jobStore, fileStore):
             os.chdir(baseDir)
         #Finish up the stats
         if stats != None:
-            stats.jobs.time = str(time.time() - startTime)
             totalCpuTime, totalMemoryUsage = getTotalCpuTimeAndMemoryUsage()
-            stats.jobs.clock = str(totalCpuTime - startClock)
-            stats.jobs.class_name = self._jobName()
-            stats.jobs.memory = str(totalMemoryUsage)
+            stats.jobs.append(
+                Expando(
+                    time=str(time.time() - startTime),
+                    clock=str(totalCpuTime - startClock),
+                    class_name=self._jobName(),
+                    memory=str(totalMemoryUsage)
+                )
+            )
 
     def _jobName(self):
         """
diff --git a/src/toil/utils/toilStats.py b/src/toil/utils/toilStats.py
--- a/src/toil/utils/toilStats.py
+++ b/src/toil/utils/toilStats.py
@@ -557,6 +557,7 @@ def processData(config, stats, options):
     # Add worker info
     worker = filter(None, stats.workers)
     jobs = filter(None, stats.jobs)
+    jobs = [item for sublist in jobs for item in sublist]
 
     def fn4(job):
         try:
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -199,6 +199,7 @@ def badWorker():
 
     workerFailed = False
     statsDict = MagicExpando()
+    statsDict.jobs = []
     messages = []
     blockFn = lambda : True
     cleanCacheFn = lambda x : True


Test patch:
diff --git a/src/toil/test/utils/utilsTest.py b/src/toil/test/utils/utilsTest.py
--- a/src/toil/test/utils/utilsTest.py
+++ b/src/toil/test/utils/utilsTest.py
@@ -26,6 +26,8 @@
 from toil.lib.bioio import system
 from toil.test import ToilTest
 from toil.test.sort.sortTest import makeFileToSort
+from toil.utils.toilStats import getStats, processData
+from toil.common import loadJobStore
 
 
 class UtilsTest(ToilTest):
@@ -167,9 +169,32 @@ def testUnicodeSupport(self):
         options.logLevel = 'debug'
         Job.Runner.startToil(Job.wrapFn(printUnicodeCharacter), options)
 
+    def testMultipleJobsPerWorkerStats(self):
+        """
+        Tests case where multiple jobs are run on 1 worker to insure that all jobs report back their data
+        """
+        options = Job.Runner.getDefaultOptions(self._getTestJobStorePath())
+        options.clean = 'never'
+        options.stats = True
+        Job.Runner.startToil(RunTwoJobsPerWorker(), options)
+
+        jobStore = loadJobStore(options.jobStore)
+        stats = getStats(options)
+        collatedStats =  processData(jobStore.config, stats, options)
+        self.assertTrue(len(collatedStats.job_types)==2,"Some jobs are not represented in the stats")
 
 def printUnicodeCharacter():
     # We want to get a unicode character to stdout but we can't print it directly because of
     # Python encoding issues. To work around this we print in a separate Python process. See
     # http://stackoverflow.com/questions/492483/setting-the-correct-encoding-when-piping-stdout-in-python
     check_call([sys.executable, '-c', "print '\\xc3\\xbc'"])
+
+class RunTwoJobsPerWorker(Job):
+    """
+    Runs child job with same resources as self in an attempt to chain the jobs on the same worker
+    """
+    def __init__(self):
+        Job.__init__(self)
+
+    def run(self, fileStore):
+        self.addChildFn(printUnicodeCharacter)