Repository: DataBiosphere/toil

Issue:
Memory usage stats are incorrect
I just discovered a copy-paste bug in jobTree's stats this morning, and lo and behold, it's still in toil as well.

https://github.com/BD2KGenomics/toil/blob/43484f818126155f20dfe4f7b173f0e97c19d457/src/toil/lib/bioio.py#L179

```python
    me = resource.getrusage(resource.RUSAGE_SELF)
    childs = resource.getrusage(resource.RUSAGE_CHILDREN)
    totalCPUTime = me.ru_utime+me.ru_stime+childs.ru_utime+childs.ru_stime
    totalMemoryUsage = me.ru_maxrss+ me.ru_maxrss
```

That last line should obviously read `me.ru_maxrss + childs.ru_maxrss`. The stats basically reflected twice the worker python process RSS, rather than the sum of its RSS and the RSS of any terminated child processes. I'm testing my fix right now, to make sure this actually does vaguely what is advertised. Will make a PR in a bit.
Memory usage stats are incorrect
I just discovered a copy-paste bug in jobTree's stats this morning, and lo and behold, it's still in toil as well.

https://github.com/BD2KGenomics/toil/blob/43484f818126155f20dfe4f7b173f0e97c19d457/src/toil/lib/bioio.py#L179

```python
    me = resource.getrusage(resource.RUSAGE_SELF)
    childs = resource.getrusage(resource.RUSAGE_CHILDREN)
    totalCPUTime = me.ru_utime+me.ru_stime+childs.ru_utime+childs.ru_stime
    totalMemoryUsage = me.ru_maxrss+ me.ru_maxrss
```

That last line should obviously read `me.ru_maxrss + childs.ru_maxrss`. The stats basically reflected twice the worker python process RSS, rather than the sum of its RSS and the RSS of any terminated child processes. I'm testing my fix right now, to make sure this actually does vaguely what is advertised. Will make a PR in a bit.


Discussion:

Resolving patch:
diff --git a/src/toil/lib/bioio.py b/src/toil/lib/bioio.py
--- a/src/toil/lib/bioio.py
+++ b/src/toil/lib/bioio.py
@@ -171,12 +171,14 @@ def system(command):
     subprocess.check_call(command, shell=isinstance(command, string_types), bufsize=-1)
 
 def getTotalCpuTimeAndMemoryUsage():
-    """Gives the total cpu time and memory usage of itself and its children.
+    """
+    Gives the total cpu time of itself and all its children, and the maximum RSS memory usage of
+    itself and its single largest child.
     """
     me = resource.getrusage(resource.RUSAGE_SELF)
     childs = resource.getrusage(resource.RUSAGE_CHILDREN)
-    totalCPUTime = me.ru_utime+me.ru_stime+childs.ru_utime+childs.ru_stime
-    totalMemoryUsage = me.ru_maxrss+ me.ru_maxrss
+    totalCPUTime = me.ru_utime + me.ru_stime + childs.ru_utime + childs.ru_stime
+    totalMemoryUsage = me.ru_maxrss + childs.ru_maxrss
     return totalCPUTime, totalMemoryUsage
 
 def getTotalCpuTime():
@@ -185,7 +187,7 @@ def getTotalCpuTime():
     return getTotalCpuTimeAndMemoryUsage()[0]
 
 def getTotalMemoryUsage():
-    """Gets the amount of memory used by the process and its children.
+    """Gets the amount of memory used by the process and its largest child.
     """
     return getTotalCpuTimeAndMemoryUsage()[1]
 
diff --git a/src/toil/utils/toilStats.py b/src/toil/utils/toilStats.py
--- a/src/toil/utils/toilStats.py
+++ b/src/toil/utils/toilStats.py
@@ -387,11 +387,10 @@ def reportPrettyData(root, worker, job, job_types, options):
     """
     out_str = "Batch System: %s\n" % root.batch_system
     out_str += ("Default Cores: %s  Default Memory: %s\n"
-                "Max Cores: %s  Max Threads: %s\n" % (
+                "Max Cores: %s\n" % (
         reportNumber(get(root, "default_cores"), options),
         reportMemory(get(root, "default_memory"), options, isBytes=True),
         reportNumber(get(root, "max_cores"), options),
-        reportNumber(get(root, "max_threads"), options),
         ))
     out_str += ("Total Clock: %s  Total Runtime: %s\n" % (
         reportTime(get(root, "total_clock"), options),
@@ -460,7 +459,7 @@ def assertNonnegative(i,name):
 
     itemWaits=[]
     for index in range(0,len(itemTimes)):
-        itemWaits.append(itemClocks[index]-itemTimes[index])
+        itemWaits.append(itemTimes[index] - itemClocks[index])
 
     itemWaits.sort()
     itemTimes.sort()


Test patch: