Repository: databiosphere/toil
Issue: Azure JobStore doesn't retry when it gets a "server is busy" error

I had my master running against the Azure jobstore crash overnight with this log:

```
INFO:toil.batchSystems.mesos.batchSystem:Preparing to launch Mesos task 9468 using offer 20151124-210426-83886090-5050-35140-O28977...
INFO:toil.batchSystems.mesos.batchSystem:...launching Mesos task 9468
INFO:toil.batchSystems.mesos.batchSystem:Preparing to launch Mesos task 9469 using offer 20151124-210426-83886090-5050-35140-O28978...
INFO:toil.batchSystems.mesos.batchSystem:Stopping Mesos driver
I1125 01:15:24.215514 61380 sched.cpp:1589] Asked to stop the driver
INFO:toil.batchSystems.mesos.batchSystem:Joining Mesos driver
INFO:toil.batchSystems.mesos.batchSystem:Joined Mesos driver
Traceback (most recent call last):
  File "scripts/parallelMappingEvaluation.py", line 1453, in <module>
    sys.exit(main(sys.argv))
  File "scripts/parallelMappingEvaluation.py", line 1443, in main
    failed_jobs = Job.Runner.startToil(root_job,  options)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 438, in startToil
    return mainLoop(config, batchSystem, jobStore, rootJob)
  File "/usr/local/lib/python2.7/dist-packages/toil/leader.py", line 460, in mainLoop
    jobBatcher.processFinishedJob(jobBatchSystemID, result)
  File "/usr/local/lib/python2.7/dist-packages/toil/leader.py", line 223, in processFinishedJob
    jobWrapper = self.jobStore.load(jobStoreID)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/azureJobStore.py", line 146, in load
    jobEntity = self.jobItems.get_entity(row_key=jobStoreID)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/azureJobStore.py", line 521, in get_entity
    return self.__getattr__('get_entity')(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/azureJobStore.py", line 515, in f
    return function(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/tableservice.py", line 361, in get_entity
    response = self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py", line 179, in _perform_request
    _storage_error_handler(ex)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/__init__.py", line 1174, in _storage_error_handler
    return _general_error_handler(http_error)
  File "/usr/local/lib/python2.7/dist-packages/azure/__init__.py", line 551, in _general_error_handler
    http_error.respbody.decode('utf-8-sig'))
azure.WindowsAzureError: Unknown error (Service Unavailable)
<?xml version="1.0" encoding="utf-8"?><error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><code>ServerBusy</code><message xml:lang="en-US">The server is busy.
RequestId:6552fd81-0002-0063-361e-275738000000
Time:2015-11-25T01:15:24.4344591Z</message></error>
INFO:toil.batchSystems.mesos.batchSystem:...launching Mesos task 9469
INFO:toil.batchSystems.mesos.batchSystem:Preparing to launch Mesos task 9480 using offer 20151124-210426-83886090-5050-35140-O28979...
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/leader.py", line 67, in statsAndLoggingAggregatorProcess
    if jobStore.readStatsAndLogging(statsAndLoggingCallBackFn) == 0:
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/azureJobStore.py", line 277, in readStatsAndLogging
    for entity in self.statsFileIDs.query_entities():
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/azureJobStore.py", line 515, in f
    return function(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/tableservice.py", line 401, in query_entities
    response = self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py", line 179, in _perform_request
    _storage_error_handler(ex)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/__init__.py", line 1174, in _storage_error_handler
    return _general_error_handler(http_error)
  File "/usr/local/lib/python2.7/dist-packages/azure/__init__.py", line 551, in _general_error_handler
    http_error.respbody.decode('utf-8-sig'))
WindowsAzureError: Unknown error (Service Unavailable)
<?xml version="1.0" encoding="utf-8"?><error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><code>ServerBusy</code><message xml:lang="en-US">The server is busy.
RequestId:5b65500a-0002-00e4-0f1e-270217000000
Time:2015-11-25T01:15:24.3709488Z</message></error>
INFO:toil.batchSystems.mesos.batchSystem:...launching Mesos task 9480
INFO:toil.batchSystems.mesos.batchSystem:Preparing to launch Mesos task 9481 using offer 20151124-210426-83886090-5050-35140-O28980...
Failed to call scheduler's resourceOffer
sys.excepthook is missing
lost sys.stderr
I1125 01:15:24.473640 61398 sched.cpp:1623] Asked to abort the driver
I1125 01:15:24.474150 61398 sched.cpp:831] Stopping framework '20151124-210426-83886090-5050-35140-0006'
```

It looks like when Azure is temporarily overloaded it throws a `WindowsAzureError` with the message "Unknown error (Service Unavailable)" and a bunch of XML actually explaining that the server is busy--presumably too busy to handle the request now, but perhaps it will be less busy later.

Toil should re-try its request in this situation, in hopes of reaching the server when it is less busy.


Changed files:
src/toil/jobStores/azureJobStore.py

diff --git a/src/toil/jobStores/azureJobStore.py b/src/toil/jobStores/azureJobStore.py
index 7f42843b..577d6889 100644
--- a/src/toil/jobStores/azureJobStore.py
+++ b/src/toil/jobStores/azureJobStore.py
@@ -613,7 +613,9 @@ class AzureJob(JobWrapper):
 
 def retryOnAzureTimeout(exception):
     timeoutMsg = "could not be completed within the specified time"
-    return isinstance(exception, WindowsAzureError) and timeoutMsg in str(exception)
+    busyMsg = "Service Unavailable"
+    return isinstance(exception, WindowsAzureError) and (timeoutMsg in str(exception)
+        or busyMsg in str(exception))
 
 
 def retry_on_error(num_tries=5, retriable_exceptions=(socket.error, socket.gaierror,