Repository: DataBiosphere/toil

Issue:
`toil clean` doesn't clean up job stores on Azure
I've been running jobs with Azure job stores, and I've been wanting to restart them from scratch. I'm supposed to be able to do this with `toil clean`, but it doesn't work.

```
anovak@amn2master1:~/hgvm-graph-bakeoff-evaluations$ toil clean azure:hgvm:amntree
amn2master1 2016-08-29 20:12:58,971 MainThread INFO toil.lib.bioio: Logging set at level: INFO
amn2master1 2016-08-29 20:12:58,971 MainThread INFO toil.utils.toilClean: Attempting to delete the job store
amn2master1 2016-08-29 20:12:59,034 MainThread INFO toil.utils.toilClean: Successfully deleted the job store
anovak@amn2master1:~/hgvm-graph-bakeoff-evaluations$ scripts/parallelMappingEvaluation.py azure:hgvm:amntree graph_urls.tsv azure:hgvm:pilot-input/reads azure:hgvm:pilot-output/low-coverage/aug16/out1 --batchSystem=mesos --mesosMaster 10.0.0.5:5050 --kmer_size=16 --edge_max=3 --index_mode gcsa-mem --include_primary --sample_limit 10 --disableCaching --workDir /tmp 2>&1 | tee log.txt
INFO:toil.lib.bioio:Logging set at level: INFO
INFO:toil.lib.bioio:Logging set at level: INFO
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): hgvm.table.core.windows.net
Traceback (most recent call last):
  File "scripts/parallelMappingEvaluation.py", line 1170, in <module>
    sys.exit(main(sys.argv))
  File "scripts/parallelMappingEvaluation.py", line 1160, in main
    failed_jobs = Job.Runner.startToil(root_job,  options)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 540, in startToil
    with Toil(options) as toil:
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 501, in __enter__
    jobStore.initialize(config)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/azureJobStore.py", line 152, in initialize
    raise JobStoreExistsException(self.locator)
toil.jobStores.abstractJobStore.JobStoreExistsException: The job store 'hgvm:amntree' already exists. Use --restart to resume the workflow, or remove the job store with 'toil clean' to start the workflow from scratch
anovak@amn2master1:~/hgvm-graph-bakeoff-evaluations$ 
```

Thinking it could have been a timing thing with Azure, I've tried waiting a few minutes between the clean command and trying to re-use the job store name, but it hasn't helped. Clean is still reporting success, and Toil is still telling me that I need to run clean.

`toil clean` doesn't clean up job stores on Azure
I've been running jobs with Azure job stores, and I've been wanting to restart them from scratch. I'm supposed to be able to do this with `toil clean`, but it doesn't work.

```
anovak@amn2master1:~/hgvm-graph-bakeoff-evaluations$ toil clean azure:hgvm:amntree
amn2master1 2016-08-29 20:12:58,971 MainThread INFO toil.lib.bioio: Logging set at level: INFO
amn2master1 2016-08-29 20:12:58,971 MainThread INFO toil.utils.toilClean: Attempting to delete the job store
amn2master1 2016-08-29 20:12:59,034 MainThread INFO toil.utils.toilClean: Successfully deleted the job store
anovak@amn2master1:~/hgvm-graph-bakeoff-evaluations$ scripts/parallelMappingEvaluation.py azure:hgvm:amntree graph_urls.tsv azure:hgvm:pilot-input/reads azure:hgvm:pilot-output/low-coverage/aug16/out1 --batchSystem=mesos --mesosMaster 10.0.0.5:5050 --kmer_size=16 --edge_max=3 --index_mode gcsa-mem --include_primary --sample_limit 10 --disableCaching --workDir /tmp 2>&1 | tee log.txt
INFO:toil.lib.bioio:Logging set at level: INFO
INFO:toil.lib.bioio:Logging set at level: INFO
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): hgvm.table.core.windows.net
Traceback (most recent call last):
  File "scripts/parallelMappingEvaluation.py", line 1170, in <module>
    sys.exit(main(sys.argv))
  File "scripts/parallelMappingEvaluation.py", line 1160, in main
    failed_jobs = Job.Runner.startToil(root_job,  options)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 540, in startToil
    with Toil(options) as toil:
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 501, in __enter__
    jobStore.initialize(config)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/azureJobStore.py", line 152, in initialize
    raise JobStoreExistsException(self.locator)
toil.jobStores.abstractJobStore.JobStoreExistsException: The job store 'hgvm:amntree' already exists. Use --restart to resume the workflow, or remove the job store with 'toil clean' to start the workflow from scratch
anovak@amn2master1:~/hgvm-graph-bakeoff-evaluations$ 
```

Thinking it could have been a timing thing with Azure, I've tried waiting a few minutes between the clean command and trying to re-use the job store name, but it hasn't helped. Clean is still reporting success, and Toil is still telling me that I need to run clean.



Discussion:
Job store initialization and reinitialization in Azure and Google is currently broken.

@hannes-ucsc How soon might you guys be working on this issue? Right now the hgvm azure storage account is very cluttered with garbage jobstore blobs and I can't just clean the entire storage account since there are some jobstores which I am not the owner of. 

It's not planned for the next three months.

Support for Azure and Google is advertised on the front page of the docs,
so I would imagine this would need to be either fixed or hacked around with
some sort of not-supported-on-this-platform warning in `toil clean` before
the next release, right?

Is there someone on the actual Toil team who is responsible for the Azure
support, or is it all contributors who are supposed to be maintaining it?

On Tue, Oct 25, 2016 at 1:17 PM, Hannes Schmidt notifications@github.com
wrote:

> It's not planned for the next three months.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> https://github.com/BD2KGenomics/toil/issues/1125#issuecomment-256163140,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AE0_Xy2wt63jEQpJYtVCpZykXBhd1lJWks5q3mPwgaJpZM4Jv2no
> .

@hannes-ucsc I don't remember if it was toil version 3.1.0 or something close to that version, but there was a previous version of toil where `toil clean` actually did work in azure. It was only until I had updated to 3.3.0 and later where I noticed the functionality was no longer there.

> Is there someone on the actual Toil team who is responsible for the Azure support, or is it all contributors who are supposed to be maintaining it?

Once a PR is merged the Toil team is responsible for maintaining the code. However, maintenance is subject to prioritization with the implementation and maintenance of other features.

A few months ago, the Azure tests were failing sporadically to a degree that was affecting our team's throughput. Basically every build was failing due to the Azure job store tests and no PR could be merged. So we decided to not maintain Azure for a while. This decision was desperate but was supported by @benedictpaten. We disabled the tests and later made changes to the other job stores, but didn't work them into the Azure implementation. Those changes are what's breaking `toil clean`.

This is low on our list of priorities. If you want it done earlier you have to do it yourself.

Job store initialization and reinitialization in Azure and Google is currently broken.

@hannes-ucsc How soon might you guys be working on this issue? Right now the hgvm azure storage account is very cluttered with garbage jobstore blobs and I can't just clean the entire storage account since there are some jobstores which I am not the owner of. 

It's not planned for the next three months.

Support for Azure and Google is advertised on the front page of the docs,
so I would imagine this would need to be either fixed or hacked around with
some sort of not-supported-on-this-platform warning in `toil clean` before
the next release, right?

Is there someone on the actual Toil team who is responsible for the Azure
support, or is it all contributors who are supposed to be maintaining it?

On Tue, Oct 25, 2016 at 1:17 PM, Hannes Schmidt notifications@github.com
wrote:

> It's not planned for the next three months.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> https://github.com/BD2KGenomics/toil/issues/1125#issuecomment-256163140,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AE0_Xy2wt63jEQpJYtVCpZykXBhd1lJWks5q3mPwgaJpZM4Jv2no
> .

@hannes-ucsc I don't remember if it was toil version 3.1.0 or something close to that version, but there was a previous version of toil where `toil clean` actually did work in azure. It was only until I had updated to 3.3.0 and later where I noticed the functionality was no longer there.

> Is there someone on the actual Toil team who is responsible for the Azure support, or is it all contributors who are supposed to be maintaining it?

Once a PR is merged the Toil team is responsible for maintaining the code. However, maintenance is subject to prioritization with the implementation and maintenance of other features.

A few months ago, the Azure tests were failing sporadically to a degree that was affecting our team's throughput. Basically every build was failing due to the Azure job store tests and no PR could be merged. So we decided to not maintain Azure for a while. This decision was desperate but was supported by @benedictpaten. We disabled the tests and later made changes to the other job stores, but didn't work them into the Azure implementation. Those changes are what's breaking `toil clean`.

This is low on our list of priorities. If you want it done earlier you have to do it yourself.


Resolving patch:
diff --git a/src/toil/jobStores/azureJobStore.py b/src/toil/jobStores/azureJobStore.py
--- a/src/toil/jobStores/azureJobStore.py
+++ b/src/toil/jobStores/azureJobStore.py
@@ -165,6 +165,7 @@ def resume(self):
         super(AzureJobStore, self).resume()
 
     def destroy(self):
+        self._bind()
         for name in 'jobItems', 'jobFileIDs', 'files', 'statsFiles', 'statsFileIDs':
             resource = getattr(self, name)
             if resource is not None:


Test patch: