Repository: DataBiosphere/toil

Issue:
Missing resource when removing table entries from Azure JobStore
After inducing the Azure JobStore to start up on the Mesos workers, I observed the following behavior at the end of my test script, in the stdout of the last mesos task.

```
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 408, in <module>
    main()
  File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 401, in main
    jobStore.delete(f)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/azureJobStore.py", line 151, in delete
    self.jobItems.delete_entity(row_key=jobStoreID)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/azureJobStore.py", line 491, in f
    return function(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/tableservice.py", line 562, in delete_entity
    self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py", line 179, in _perform_request
    _storage_error_handler(ex)
  File "/usr/local/lib/python2.7/dist-packages/azure/storage/__init__.py", line 1174, in _storage_error_handler
    return _general_error_handler(http_error)
  File "/usr/local/lib/python2.7/dist-packages/azure/__init__.py", line 546, in _general_error_handler
    _ERROR_NOT_FOUND.format(str(http_error)))
azure.WindowsAzureMissingResourceError: Not found (Not Found)
```

I wrapped the `self.jobItems.delete_entity(row_key=jobStoreID)` line in a try-except-pass for the given error, and my script runs successfully. Maybe that's the right solution?

However, even after the script successfully completes, I can't re-use the Azure job store name; I get a message saying it already exists.



Discussion:
Looks like a jobs can be deleted twice in some cases--the try-except-pass should work fine. Will fix later tonight.

For the second issue, I think replacing:

```
        self.registryTable.update_entity(row_key=self.namePrefix,
                                         entity={'exists': 'False'})
```

with:

```
        self.registryTable.delete_entity(row_key=self.namePrefix)
```

in azureJobStore.py should work. I'll submit a PR but it may take a little while to get merged.


Resolving patch:
diff --git a/src/toil/jobStores/azureJobStore.py b/src/toil/jobStores/azureJobStore.py
--- a/src/toil/jobStores/azureJobStore.py
+++ b/src/toil/jobStores/azureJobStore.py
@@ -148,15 +148,18 @@ def update(self, job):
                                     entity=job.toItem(chunkSize=self.jobChunkSize))
 
     def delete(self, jobStoreID):
-        self.jobItems.delete_entity(row_key=jobStoreID)
+        try:
+            self.jobItems.delete_entity(row_key=jobStoreID)
+        except WindowsAzureMissingResourceError:
+            # Job deletion is idempotent, and this job has been deleted already
+            return
         filterString = "PartitionKey eq '%s'" % jobStoreID
         for fileEntity in self.jobFileIDs.query_entities(filter=filterString):
             jobStoreFileID = fileEntity.RowKey
             self.deleteFile(jobStoreFileID)
 
     def deleteJobStore(self):
-        self.registryTable.update_entity(row_key=self.namePrefix,
-                                         entity={'exists': 'False'})
+        self.registryTable.delete_entity(row_key=self.namePrefix)
         self.jobItems.delete_table()
         self.jobFileIDs.delete_table()
         self.files.delete_container()


Test patch: