Repository: DataBiosphere/toil

Issue:
AttributeError: 'Expando' object has no attribute 'workers'
When I run:

```
$ rm -rf toilWorkflow/; python src/toil/wdl/wdltoil.py
```

I get this:

```
INFO:toil.lib.bioio:Logging set at level: INFO
INFO:toil.common:Using the single machine batch system
INFO:toil.jobStores.fileJobStore:Jobstore directory is: /Users/sfrazer/projects/toil/toilWorkflow
WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (8).
WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (8589934592).
INFO:toil.batchSystems.singleMachine:Setting up the thread pool with 80 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 8.
INFO:toil.common:Written the environment for the jobs to the environment file
INFO:toil.leader:Checked batch system has no running jobs and no updated jobs
INFO:toil.leader:Found 1 jobs to start and 0 jobs with successors to run
INFO:toil.leader:Starting the main loop
INFO:toil.batchSystems.singleMachine:Executing command: '/Users/sfrazer/venv/wdl-toil/bin/_toil_worker /Users/sfrazer/projects/toil/toilWorkflow b/V/jobUulNcW'.
INFO:toil.batchSystems.singleMachine:Executing command: '/Users/sfrazer/venv/wdl-toil/bin/_toil_worker /Users/sfrazer/projects/toil/toilWorkflow X/J/joby_OmSG'.
INFO:toil.batchSystems.singleMachine:Executing command: '/Users/sfrazer/venv/wdl-toil/bin/_toil_worker /Users/sfrazer/projects/toil/toilWorkflow k/r/jobSMCrWI'.
INFO:toil.batchSystems.singleMachine:Executing command: '/Users/sfrazer/venv/wdl-toil/bin/_toil_worker /Users/sfrazer/projects/toil/toilWorkflow b/V/jobUulNcW'.
Process Process-1:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/sfrazer/projects/toil/src/toil/leader.py", line 67, in statsAndLoggingAggregatorProcess
    if jobStore.readStatsAndLogging(callback) == 0:
  File "/Users/sfrazer/projects/toil/src/toil/jobStores/fileJobStore.py", line 238, in readStatsAndLogging
    callback(fH)
  File "/Users/sfrazer/projects/toil/src/toil/leader.py", line 47, in callback
    workers = stats.workers
AttributeError: 'Expando' object has no attribute 'workers'
INFO:toil.leader:Only failed jobs and their dependents (0 total) are remaining, so exiting.
INFO:toil.leader:Finished the main loop
INFO:toil.leader:Waiting for stats and logging collator process to finish
INFO:toil.leader:Stats/logging finished collating in 0.000979900360107 seconds
None
```

It was actually working for a little while and then suddenly it started giving back this error and I don't quite know why.  Any ideas?

AttributeError: 'Expando' object has no attribute 'workers'
When I run:

```
$ rm -rf toilWorkflow/; python src/toil/wdl/wdltoil.py
```

I get this:

```
INFO:toil.lib.bioio:Logging set at level: INFO
INFO:toil.common:Using the single machine batch system
INFO:toil.jobStores.fileJobStore:Jobstore directory is: /Users/sfrazer/projects/toil/toilWorkflow
WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (8).
WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (8589934592).
INFO:toil.batchSystems.singleMachine:Setting up the thread pool with 80 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 8.
INFO:toil.common:Written the environment for the jobs to the environment file
INFO:toil.leader:Checked batch system has no running jobs and no updated jobs
INFO:toil.leader:Found 1 jobs to start and 0 jobs with successors to run
INFO:toil.leader:Starting the main loop
INFO:toil.batchSystems.singleMachine:Executing command: '/Users/sfrazer/venv/wdl-toil/bin/_toil_worker /Users/sfrazer/projects/toil/toilWorkflow b/V/jobUulNcW'.
INFO:toil.batchSystems.singleMachine:Executing command: '/Users/sfrazer/venv/wdl-toil/bin/_toil_worker /Users/sfrazer/projects/toil/toilWorkflow X/J/joby_OmSG'.
INFO:toil.batchSystems.singleMachine:Executing command: '/Users/sfrazer/venv/wdl-toil/bin/_toil_worker /Users/sfrazer/projects/toil/toilWorkflow k/r/jobSMCrWI'.
INFO:toil.batchSystems.singleMachine:Executing command: '/Users/sfrazer/venv/wdl-toil/bin/_toil_worker /Users/sfrazer/projects/toil/toilWorkflow b/V/jobUulNcW'.
Process Process-1:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/sfrazer/projects/toil/src/toil/leader.py", line 67, in statsAndLoggingAggregatorProcess
    if jobStore.readStatsAndLogging(callback) == 0:
  File "/Users/sfrazer/projects/toil/src/toil/jobStores/fileJobStore.py", line 238, in readStatsAndLogging
    callback(fH)
  File "/Users/sfrazer/projects/toil/src/toil/leader.py", line 47, in callback
    workers = stats.workers
AttributeError: 'Expando' object has no attribute 'workers'
INFO:toil.leader:Only failed jobs and their dependents (0 total) are remaining, so exiting.
INFO:toil.leader:Finished the main loop
INFO:toil.leader:Waiting for stats and logging collator process to finish
INFO:toil.leader:Stats/logging finished collating in 0.000979900360107 seconds
None
```

It was actually working for a little while and then suddenly it started giving back this error and I don't quite know why.  Any ideas?



Discussion:
I'm getting this error as well with toil version 3.1. It also seems as if now I am only running a single job at a time on my local machine when it should have enough resources allocated/requested to be running several

As this is a bug in a release, we should fix this ASAP.

I'm getting this error as well with toil version 3.1. It also seems as if now I am only running a single job at a time on my local machine when it should have enough resources allocated/requested to be running several

As this is a bug in a release, we should fix this ASAP.


Resolving patch:
diff --git a/src/toil/leader.py b/src/toil/leader.py
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -44,20 +44,23 @@ def statsAndLoggingAggregatorProcess(jobStore, stop):
 
     def callback(fileHandle):
         stats = json.load(fileHandle, object_hook=Expando)
-        workers = stats.workers
         try:
-            logs = workers.log
+            logs = stats.workers.log
         except AttributeError:
             # To be expected if there were no calls to logToMaster()
             pass
         else:
             for message in logs:
                 logger.log(int(message.level),
-                           "Got message from job at time: %s : %s",
-                           time.strftime("%m-%d-%Y %H:%M:%S"), message.text)
-
-        for log in stats.logs:
-            logger.info("%s:     %s", log.jobStoreID, log.text)
+                           'Got message from job at time %s: %s',
+                           time.strftime('%m-%d-%Y %H:%M:%S'), message.text)
+        try:
+            logs = stats.logs
+        except AttributeError:
+            pass
+        else:
+            for log in logs:
+                logger.info("%s:    %s", log.jobStoreID, log.text)
 
     while True:
         # This is a indirect way of getting a message to the process to exit
@@ -488,11 +491,13 @@ def mainLoop(config, batchSystem, jobStore, rootJobWrapper):
     ##########################################
     #Finish up the stats/logging aggregation process
     ##########################################
-    logger.info("Waiting for stats and logging collator process to finish")
+    logger.info('Waiting for stats and logging collator process to finish ...')
     startTime = time.time()
     stopStatsAndLoggingAggregatorProcess.put(True)
     worker.join()
-    logger.info("Stats/logging finished collating in %s seconds", time.time() - startTime)
+    if worker.exitcode != 0:
+        raise RuntimeError('Stats/logging collator failed with exit code %d.' % worker.exitcode)
+    logger.info('... finished collating stats and logs. Took %s seconds', time.time() - startTime)
     # in addition to cleaning on exceptions, onError should clean if there are any failed jobs
 
     #Parse out the return value from the root job


Test patch: