Repository: DataBiosphere/toil

Issue:
NoSuchBucket error during Toil context manager exit
I have the feeling this happens while `__exit__` is invoked with a primary exception. The primary exception is shadowed by the secondary exception that results from not being able to delete the job store. The robust deletion PR might improve this but we should also log the primary exception eagerly in `__exit__`.

NoSuchBucket error during Toil context manager exit
I have the feeling this happens while `__exit__` is invoked with a primary exception. The primary exception is shadowed by the secondary exception that results from not being able to delete the job store. The robust deletion PR might improve this but we should also log the primary exception eagerly in `__exit__`.



Discussion:
@hannes-ucsc Yes, robust clean will ignore any exceptions of this nature completely. How about logging any exceptions caught in `AbstractJobStore._deleteJobStore`(the main body of each `cleanJobStore` and `deleteJobStore` methods) at the debug level?

Yes, secondary exceptions that you swallow should be logged at debug level.

But I am concerned about the primary exception. Maybe there should be a general `except:` clause around `Toil.__exit__` that catches any exception, and depending on whether `__exit__` was called with a primary exception, logs it or reraises it. It should log the secondary exception if a primary exception was passed in. It should reraise the secondary exception if no primary exception was passed in.

If the robust clean PR addresses this, this issue should be referenced in that PR's title and in the body. Then Waffle will stack the two.

@JakeNarkizian, did you see [my previous comment](https://github.com/BD2KGenomics/toil/issues/921#issuecomment-226684843)?

@hannes-ucsc Yes, thanks for the reminder. I haven't addressed this in #900, yet. 

Robust clean ignores exceptions raised due to corrupted job stores components, and it is reasonably implied what components were present at the time of cleaning in the log, but specific exceptions like `NoSuchBucket` aren't logged explicitly. The Toil context manager also doesn't log anything differently. 

I definitely can resolve this issue as well if you think it is better since I am touching code around there anyway, but I'm slightly hesitant to take on anymore because of how big this PR is already. 

It currently addresses 7 tracked issues, and a few others that don't have tickets (like using the general retry logic you've implemented for google, refactoring job store string to job store locator, I'm sure I'm blanking on more than one at the moment...). It also refactors quite a bit of each of the four currently supported job stores because of the need for class methods for each job store component. I've been delayed a few times by merge conflicts due to the sheer amount of code I've had to change in the job stores. 

That said, it might be worth addressing this issue as well since--like i mentioned above--the code's already touched. What do you think? 

**tl;dr** 

I haven't addressed it yet.
It might be wise to since I'm touching the code in common, but the PR is already really big.
what do you think?

You probably mean #916, not #900. Anyways, I thought that #916 would solve this automatically. Maybe I'm not understanding this correctly.

So #916 is not going to make the cut for 3.2.0. Can you break #916 up into smaller PRs? 

Yes, sorry #916. 
I can break #916 into separate PRs but each branch is going to have to build off the last in a somewhat messy way since its changes are very closely related, I'd like to run it by you before I start the process.

Did you had a chance to glance at #916's loadBucket and createBucket methods in AWS? Perhaps we can discuss this when we talk about those methods. 

Punting for next sprint.

@hannes-ucsc Yes, robust clean will ignore any exceptions of this nature completely. How about logging any exceptions caught in `AbstractJobStore._deleteJobStore`(the main body of each `cleanJobStore` and `deleteJobStore` methods) at the debug level?

Yes, secondary exceptions that you swallow should be logged at debug level.

But I am concerned about the primary exception. Maybe there should be a general `except:` clause around `Toil.__exit__` that catches any exception, and depending on whether `__exit__` was called with a primary exception, logs it or reraises it. It should log the secondary exception if a primary exception was passed in. It should reraise the secondary exception if no primary exception was passed in.

If the robust clean PR addresses this, this issue should be referenced in that PR's title and in the body. Then Waffle will stack the two.

@JakeNarkizian, did you see [my previous comment](https://github.com/BD2KGenomics/toil/issues/921#issuecomment-226684843)?

@hannes-ucsc Yes, thanks for the reminder. I haven't addressed this in #900, yet. 

Robust clean ignores exceptions raised due to corrupted job stores components, and it is reasonably implied what components were present at the time of cleaning in the log, but specific exceptions like `NoSuchBucket` aren't logged explicitly. The Toil context manager also doesn't log anything differently. 

I definitely can resolve this issue as well if you think it is better since I am touching code around there anyway, but I'm slightly hesitant to take on anymore because of how big this PR is already. 

It currently addresses 7 tracked issues, and a few others that don't have tickets (like using the general retry logic you've implemented for google, refactoring job store string to job store locator, I'm sure I'm blanking on more than one at the moment...). It also refactors quite a bit of each of the four currently supported job stores because of the need for class methods for each job store component. I've been delayed a few times by merge conflicts due to the sheer amount of code I've had to change in the job stores. 

That said, it might be worth addressing this issue as well since--like i mentioned above--the code's already touched. What do you think? 

**tl;dr** 

I haven't addressed it yet.
It might be wise to since I'm touching the code in common, but the PR is already really big.
what do you think?

You probably mean #916, not #900. Anyways, I thought that #916 would solve this automatically. Maybe I'm not understanding this correctly.

So #916 is not going to make the cut for 3.2.0. Can you break #916 up into smaller PRs? 

Yes, sorry #916. 
I can break #916 into separate PRs but each branch is going to have to build off the last in a somewhat messy way since its changes are very closely related, I'd like to run it by you before I start the process.

Did you had a chance to glance at #916's loadBucket and createBucket methods in AWS? Perhaps we can discuss this when we talk about those methods. 

Punting for next sprint.


Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -490,15 +490,21 @@ def __enter__(self):
 
     def __exit__(self, exc_type, exc_val, exc_tb):
         """
-        Clean up after a workflow invocaton. Depending on the configuration, delete the job store.
+        Clean up after a workflow invocation. Depending on the configuration, delete the job store.
         """
+        try:
+            if (exc_type is not None and self.config.clean == "onError" or
+                            exc_type is None and self.config.clean == "onSuccess" or
+                        self.config.clean == "always"):
+                logger.info("Attempting to delete the job store")
+                self._jobStore.deleteJobStore()
+                logger.info("Successfully deleted the job store")
+        except Exception as e:
+            if exc_type is None:
+                raise
+            else:
+                logger.exception('The following error was raised during clean up:')
         self._inContextManager = False
-        if (exc_type is not None and self.config.clean == "onError" or
-                        exc_type is None and self.config.clean == "onSuccess" or
-                    self.config.clean == "always"):
-            logger.info("Attempting to delete the job store")
-            self._jobStore.deleteJobStore()
-            logger.info("Successfully deleted the job store")
         return False  # let exceptions through
 
     def start(self, rootJob):


Test patch: