Repository: DataBiosphere/toil

Issue:
Put workdir in mesos executor sandboxes by default
Feature request by @adamnovak



Discussion:
OK, I ran my big run and besides some errors from the Azure job store it seems to have worked fine.

Those errors looked like this:

```
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:       Exception in thread Thread-13:
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:       Traceback (most recent call last):
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           self.run()
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/lib/python2.7/threading.py", line 754, in run
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           self.__target(*self.__args, **self.__kwargs)
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1371, in asyncUpdate
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           self.jobStore.update(self.jobWrapper)
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/azureJobStore.py", line 248, in update
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           entity=job.toItem(chunkSize=self.jobChunkSize))
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/azureJobStore.py", line 665, in f
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           return function(**kwargs)
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/local/lib/python2.7/dist-packages/azure/storage/table/tableservice.py", line 496, in update_entity
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           response = self._perform_request(request)
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py", line 171, in _perform_request
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           resp = self._filter(request)
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/local/lib/python2.7/dist-packages/azure/storage/table/tableservice.py", line 664, in _perform_request_worker
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           return self._httpclient.perform_request(request)
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/local/lib/python2.7/dist-packages/azure/storage/_http/httpclient.py", line 181, in perform_request
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           self.send_request_body(connection, request.body)
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/local/lib/python2.7/dist-packages/azure/storage/_http/httpclient.py", line 143, in send_request_body
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           connection.send(request_body)
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/local/lib/python2.7/dist-packages/azure/storage/_http/requestsclient.py", line 81, in send
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           self.response = self.session.request(self.method, self.uri, data=request_body, headers=self.headers, timeout=self.timeout)
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           resp = self.send(prep, **send_kwargs)
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           r = adapter.send(request, **kwargs)
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:         File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 387, in send
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:           raise Timeout(e)
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:       Timeout: HTTPSConnectionPool(host='hgvm.table.core.windows.net', port=443): Read timed out.
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:       
WARNING:toil.leader:1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57:       WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 1e4f311e_4c5f_4b51_93e7_9b8b0b99cd57 to 0
```

They probably deserve their own ticket to add retry logic around whatever is timing out. This ticket here I think is fixed.


Resolving patch:
diff --git a/src/toil/batchSystems/mesos/executor.py b/src/toil/batchSystems/mesos/executor.py
--- a/src/toil/batchSystems/mesos/executor.py
+++ b/src/toil/batchSystems/mesos/executor.py
@@ -49,6 +49,10 @@ def __init__(self):
         self.workerCleanupInfo = None
         Resource.prepareSystem()
         self.address = None
+        # Setting this value at this point will ensure that the toil workflow directory will go to
+        # the mesos sandbox if the user hasn't specified --workDir on the command line.
+        if not os.getenv('TOIL_WORKDIR'):
+            os.environ['TOIL_WORKDIR'] = os.getcwd()
 
     def registered(self, driver, executorInfo, frameworkInfo, slaveInfo):
         """


Test patch: