Repository: DataBiosphere/toil

Issue:
It is possible to open a file that was never written to in worker.py
I ran into this error in `toil/src/toil/worker.py` while debugging a unit test: `toil.test.cwl.CWLTest.test_run_revsort_debug_worker`:

```
> Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/home/ben/git/updateDependencies/toil/src/toil/test/cwl/cwlTest.py", line 106, in test_run_revsort_debug_worker
    u'sha1$b9214658cc453331b62c2282b772a5c063dbd284'}})
  File "/home/ben/git/updateDependencies/toil/src/toil/test/cwl/cwlTest.py", line 58, in _debug_worker_tester
    os.path.join(rootDir, jobfile)], stdout=st)
  File "/home/ben/git/updateDependencies/toil/src/toil/cwl/cwltoil.py", line 1065, in main
    outobj = toil.start(wf1)
  File "/home/ben/git/updateDependencies/toil/src/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/ben/git/updateDependencies/toil/src/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/ben/git/updateDependencies/toil/src/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/ben/git/updateDependencies/toil/src/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/ben/git/updateDependencies/toil/src/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/ben/git/updateDependencies/toil/src/toil/leader.py", line 415, in _processReadyJob
    self._runJobSuccessors(jobGraph)
  File "/home/ben/git/updateDependencies/toil/src/toil/leader.py", line 333, in _runJobSuccessors
    self.issueJobs(successors)
  File "/home/ben/git/updateDependencies/toil/src/toil/leader.py", line 614, in issueJobs
    self.issueJob(job)
  File "/home/ben/git/updateDependencies/toil/src/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/ben/git/updateDependencies/toil/src/toil/batchSystems/singleMachine.py", line 246, in issueBatchJob
    self.worker(self.inputQueue)
  File "/home/ben/git/updateDependencies/toil/src/toil/batchSystems/singleMachine.py", line 203, in worker
    self._runWorker(jobCommand, jobID, environment)
  File "/home/ben/git/updateDependencies/toil/src/toil/batchSystems/singleMachine.py", line 156, in _runWorker
    redirectOutputToLogFile=not self.debugWorker) # Call the worker
  File "/home/ben/git/updateDependencies/toil/src/toil/worker.py", line 457, in workerScript
    with open(tempWorkerLogPath, "r") as f:
IOError: [Errno 2] No such file or directory: u'/tmp/toil-2aa7792a-bde3-49ff-aa48-b67cfd9b15f1-236cd0b5778c41cd8e2c50a641f24218/tmp0fqFvG/worker_log.txt'
```

You can only write this file (`tempWorkerLogPath`) when `redirectOutputToLogFile` is true. However, the file is attempted to be read only after checking that `workerFailed` is true.

This can be turned off:
https://github.com/DataBiosphere/toil/blob/master/src/toil/worker.py#L186

But it always checks here:
https://github.com/DataBiosphere/toil/blob/master/src/toil/worker.py#L457

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-316)
┆Issue Number: TOIL-316



Discussion:

Resolving patch:
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -451,7 +451,7 @@ def badWorker():
     # Now our file handles are in exactly the state they were in before.
 
     #Copy back the log file to the global dir, if needed
-    if workerFailed:
+    if workerFailed and redirectOutputToLogFile:
         jobGraph.logJobStoreFileID = jobStore.getEmptyFileStoreID(jobGraph.jobStoreID)
         jobGraph.chainedJobs = listOfJobs
         with jobStore.updateFileStream(jobGraph.logJobStoreFileID) as w:


Test patch: