Repository: DataBiosphere/toil

Issue:
Remove registry from Azure job store
The AWS job store maintains a registry due to eventual consistency issues. @joelarmstrong do you recall why Azure also maintains a registry? If there does not exist a compelling reason it may be worth forgoing the cost of writing to a reading from an extra table in Azure. 

Instead the existence of a job store can be determined by probing for some component of it that is eventually consistent.  



Discussion:
No compelling reason at all, I kept the registry only for symmetry with the AWS jobStore, in case there was a use case for querying the registry to see what jobStores existed. The same functionality could be achieved by checking for the existence of the config pickle or something.

Thanks for the info @joelarmstrong. @hannes-ucsc with this in mind would you feel more comfortable removing the registry in #916 while I'm already touching that part of the code? Or shall we contend with it in a different PR?

I'm ok with removing it in #916.

Just make sure that the last storage component you remove is the one that you detect job store existence with. This applies to all job stores.


Resolving patch:
diff --git a/src/toil/jobStores/azureJobStore.py b/src/toil/jobStores/azureJobStore.py
--- a/src/toil/jobStores/azureJobStore.py
+++ b/src/toil/jobStores/azureJobStore.py
@@ -139,12 +139,8 @@ def __init__(self, accountName, namePrefix, config=None,
         self.tableService = TableService(account_key=self.account_key, account_name=accountName)
         self.blobService = BlobService(account_key=self.account_key, account_name=accountName)
 
-        # Register our job-store in the global table for this storage account
-        self.registryTable = self._getOrCreateTable('toilRegistry')
-        exists = self.registryTable.get_entity(row_key=self.namePrefix)
+        exists = self._jobStoreExists()
         self._checkJobStoreCreation(config is not None, exists, accountName + ":" + self.namePrefix)
-        self.registryTable.insert_or_replace_entity(row_key=self.namePrefix,
-                                                    entity={'exists': True})
 
         # Serialized jobs table
         self.jobItems = self._getOrCreateTable(self.qualify('jobs'))
@@ -225,13 +221,28 @@ def delete(self, jobStoreID):
             self.deleteFile(jobStoreFileID)
 
     def deleteJobStore(self):
-        self.registryTable.delete_entity(row_key=self.namePrefix)
         self.jobItems.delete_table()
         self.jobFileIDs.delete_table()
         self.files.delete_container()
         self.statsFiles.delete_container()
         self.statsFileIDs.delete_table()
 
+    def _jobStoreExists(self):
+        """
+        Checks if job store exists by querying the existence of the statsFileIDs table. Note that
+        this is the last component that is deleted in deleteJobStore.
+        """
+        for attempt in retry_azure():
+            with attempt:
+                try:
+                    table = self.tableService.query_tables(table_name=self.qualify('statsFileIDs'))
+                    return table is not None
+                except AzureMissingResourceHttpError as e:
+                    if e.status_code == 404:
+                        return False
+                    else:
+                        raise
+
     def getEnv(self):
         return dict(AZURE_ACCOUNT_KEY=self.account_key)
 


Test patch:
diff --git a/src/toil/test/jobStores/jobStoreTest.py b/src/toil/test/jobStores/jobStoreTest.py
--- a/src/toil/test/jobStores/jobStoreTest.py
+++ b/src/toil/test/jobStores/jobStoreTest.py
@@ -936,6 +936,11 @@ def testLargeJob(self):
         self.assertIsNot(job1, job2)
         self.assertEqual(job2.command, command)
 
+    def testJobStoreExists(self):
+        self.assertTrue(self.master._jobStoreExists())
+        self.master.deleteJobStore()
+        self.assertFalse(self.master._jobStoreExists())
+
     def _prepareTestFile(self, containerName, size=None):
         from toil.jobStores.azureJobStore import _fetchAzureAccountKey
         from azure.storage.blob import BlobService