internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataBiosphere/toil

Issue:
Wonky try clause in ServiceJob.run
[This try](https://github.com/BD2KGenomics/toil/blob/658395fa2e8621ea45ac76020ef02fe0ca832523/src/toil/job.py#L1574) should be after service.start() such that if start() fails, stop() isn't called

Wonky try clause in ServiceJob.run
[This try](https://github.com/BD2KGenomics/toil/blob/658395fa2e8621ea45ac76020ef02fe0ca832523/src/toil/job.py#L1574) should be after service.start() such that if start() fails, stop() isn't called



Discussion:

Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -1571,10 +1571,9 @@ def run(self, fileStore):
         #Unpickle the service
         userModule = self._loadUserModule(self.serviceModule)
         service = self._unpickle( userModule, BytesIO( self.pickledService ) )
+        #Start the service
+        startCredentials = service.start(fileStore)
         try:
-            #Start the service
-            startCredentials = service.start(fileStore)
-            
             #The start credentials  must be communicated to processes connecting to
             #the service, to do this while the run method is running we
             #cheat and set the return value promise within the run method


Test patch:
Repository: DataBiosphere/toil

Issue:
Existing job store error should mention `toil clean`



Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/abstractJobStore.py b/src/toil/jobStores/abstractJobStore.py
--- a/src/toil/jobStores/abstractJobStore.py
+++ b/src/toil/jobStores/abstractJobStore.py
@@ -195,9 +195,10 @@ def _checkJobStoreCreation(create, exists, jobStoreString):
         :raise JobStoreCreationException:  if create == exists
         """
         if create and exists:
-            raise JobStoreCreationException("The job store '%s' already exists. "
-                                            "Use --restart or 'toil restart' to resume this jobStore, "
-                                            "else remove it to start from scratch" % jobStoreString)
+            raise JobStoreCreationException("The job store '%s' already exists. Use --restart or "
+                                            "'toil restart' to resume the workflow, or remove the "
+                                            "job store with 'toil clean' to start the workflow "
+                                            "from scratch" % jobStoreString)
         if not create and not exists:
             raise JobStoreCreationException("The job store '%s' does not exist, so there "
                                             "is nothing to restart." % jobStoreString)


Test patch:
Repository: DataBiosphere/toil

Issue:
CWL conformance tests fail on PR #778
Connected to #778

http://jenkins.cgcloud.info/job/toil-pull-requests/1068/testReport/src.toil.test.cwl.cwlTest/CWLTest/test_run_conformance/
http://jenkins.cgcloud.info/job/toil-pull-requests/1073/testReport/junit/src.toil.test.cwl.cwlTest/CWLTest/test_run_conformance/

@tetron, any idea what this could be?



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -39,7 +39,7 @@
         'encryption': [
             'pynacl==0.3.0'],
         'cwl': [
-            'cwltool==1.0.20160411194133']},
+            'cwltool==1.0.20160413143011']},
     package_dir={'': 'src'},
     packages=find_packages('src', exclude=['*.test']),
     entry_points={


Test patch:
Repository: DataBiosphere/toil

Issue:
Info level log before and after cleaning up the jobStore



Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -485,8 +485,10 @@ def __exit__(self, exc_type, exc_val, exc_tb):
         if (exc_type is not None and self.config.clean == "onError" or
             exc_type is None and self.config.clean == "onSuccess" or
              self.config.clean == "always"):
-
+            
+            logger.info("Attempting to delete the job store")
             self._jobStore.deleteJobStore()
+            logger.info("Successfully deleted the job store")
         return False  # let exceptions through
 
     def start(self, rootJob):
diff --git a/src/toil/lib/bioio.py b/src/toil/lib/bioio.py
--- a/src/toil/lib/bioio.py
+++ b/src/toil/lib/bioio.py
@@ -13,6 +13,8 @@
 # limitations under the License.
 
 from __future__ import absolute_import
+
+import socket
 import sys
 import os
 import logging
@@ -29,11 +31,6 @@
 from xml.dom import minidom  # For making stuff pretty
 
 defaultLogLevel = logging.INFO
-
-# TODO: looks like this can be removed
-
-loggingFormatter = logging.Formatter('%(asctime)s %(levelname)s %(lineno)s %(message)s')
-
 logger = logging.getLogger(__name__)
 rootLogger = logging.getLogger()
 
@@ -134,7 +131,8 @@ def setLoggingFromOptions(options):
     """
     Sets the logging from a dictionary of name/value options.
     """
-    logging.basicConfig(format="%(asctime)s %(levelname)s:%(name)s: %(message)s")
+    formatStr = socket.gethostname() + ': %(asctime)s %(levelname)s: %(name)s: %(message)s'
+    logging.basicConfig(format=formatStr)
     rootLogger.setLevel(defaultLogLevel)
     if options.logLevel is not None:
         setLogLevel(options.logLevel)
diff --git a/src/toil/utils/toilClean.py b/src/toil/utils/toilClean.py
--- a/src/toil/utils/toilClean.py
+++ b/src/toil/utils/toilClean.py
@@ -55,8 +55,9 @@ def main():
         logger.info("The specified JobStore does not exist, it may have already been deleted")
         sys.exit(0)
 
-    logger.info("Deleting the JobStore")
+    logger.info("Attempting to delete the job store")
     jobStore.deleteJobStore()
+    logger.info("Successfully deleted the job store")
 
 
 def _test():


Test patch:
Repository: DataBiosphere/toil

Issue:
Too many requests error from Google job store
See Jenkins build [here](http://jenkins.cgcloud.info/job/toil-pull-requests/1326/).



Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/googleJobStore.py b/src/toil/jobStores/googleJobStore.py
--- a/src/toil/jobStores/googleJobStore.py
+++ b/src/toil/jobStores/googleJobStore.py
@@ -63,6 +63,7 @@ def __init__(self, namePrefix, projectID=None, config=None):
         self._encryptedHeaders = self.headerValues
 
         self.uri = boto.storage_uri(self.gsBucketURL, GOOGLE_STORAGE)
+        self.files = None
 
         exists = True
         try:
@@ -74,7 +75,7 @@ def __init__(self, namePrefix, projectID=None, config=None):
         self._checkJobStoreCreation(create, exists, projectID+':'+namePrefix)
 
         if not exists:
-            self.files = self.uri.create_bucket(headers=self.headerValues)
+            self.files = self._retryCreateBucket(self.uri, self.headerValues)
 
         super(GoogleJobStore, self).__init__(config=config)
         self.sseKeyPath = self.config.sseKey
@@ -315,6 +316,19 @@ def readStatsAndLogging(self, callback, readAll=False):
 
         return filesRead
 
+    @staticmethod
+    def _retryCreateBucket(uri, headers):
+        bucket = None
+        while not bucket:
+            try:
+                bucket = uri.create_bucket(headers=headers)
+            except boto.exception.GSResponseError as e:
+                if e.status == 429:
+                    time.sleep(10)
+                else:
+                    raise
+        return bucket
+
     @staticmethod
     def _newID(isFile=False, jobStoreID=None):
         if isFile and jobStoreID:  # file associated with job


Test patch:
diff --git a/src/toil/test/jobStores/jobStoreTest.py b/src/toil/test/jobStores/jobStoreTest.py
--- a/src/toil/test/jobStores/jobStoreTest.py
+++ b/src/toil/test/jobStores/jobStoreTest.py
@@ -694,8 +694,10 @@ def _hashUrl(url):
 
     @staticmethod
     def _createExternalStore():
-        uri = "gs://import-export-test-%s" % str(uuid.uuid4())
-        return boto.storage_uri(uri).create_bucket(headers=GoogleJobStoreTest.headers)
+        from toil.jobStores.googleJobStore import GoogleJobStore
+        uriString = "gs://import-export-test-%s" % str(uuid.uuid4())
+        uri = boto.storage_uri(uriString)
+        return GoogleJobStore._retryCreateBucket(uri=uri, headers=GoogleJobStoreTest.headers)
 
     @staticmethod
     def _cleanUpExternalStore(url):
Repository: DataBiosphere/toil

Issue:
Nonetype object has no attribute "terminate"
An exception generated on interpreter shutdown while running SortTest. Seems benign but the exception should still be either avoided, caught, or documented. See http://stackoverflow.com/questions/1745232/solving-thread-cleanup-on-paramiko for more on this exception



Discussion:
@cket, can you provide more info? I can only vaguely remember what this one was about.

Updated the description with more details

Better explanation found here: http://stackoverflow.com/questions/20596918/python-exception-in-thread-thread-1-most-likely-raised-during-interpreter-shutd

This is caused by our badWorker daemon thread. Seemingly dependent on what stage of shutdown the interpreter is in, we will get either get a traceback for this error or just a printed statement informing us there was an error.

Traceback:

```
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
  File "/Users/cj/toil/src/toil/worker.py", line 102, in badWorker
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'kill'
```

Unhelpful printed message:

```
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
```

Changing badWorker to a non daemon thread appears to fix the problem


Resolving patch:
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -102,7 +102,14 @@ def badWorker():
             os.kill(os.getpid(), signal.SIGKILL) #signal.SIGINT)
             #TODO: FIX OCCASIONAL DEADLOCK WITH SIGINT (tested on single machine)
         t = Thread(target=badWorker)
-        t.daemon = True
+        # Ideally this would be a daemon thread but that causes an intermittent (but benign)
+        # exception similar to the one described here:
+        # http://stackoverflow.com/questions/20596918/python-exception-in-thread-thread-1-most-likely-raised-during-interpreter-shutd
+        # Our exception is:
+        #    Exception in thread Thread-1 (most likely raised during interpreter shutdown):
+        #    <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'kill'
+        # This attribute error is caused by the call os.kill() and apparently unavoidable with a
+        # daemon
         t.start()
 
     ##########################################


Test patch:
Repository: DataBiosphere/toil

Issue:
Add more helpful error message if caching unsupported batch system is used
Stems from #928



Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -660,7 +660,9 @@ def createBatchSystem(config, jobStore=None, userScript=None):
             raise RuntimeError('Unrecognised batch system: %s' % config.batchSystem)
 
         if not (config.disableSharedCache or batchSystemClass.supportsWorkerCleanup()):
-            raise RuntimeError('%s currently does not support shared caching.' % config.batchSystem)
+            raise RuntimeError('%s currently does not support shared caching.  Use the '
+                               '--disableSharedCache option when running Toil with this batch '
+                               'system.' % config.batchSystem)
         logger.info('Using the %s' %
                     re.sub("([a-z])([A-Z])", "\g<1> \g<2>", batchSystemClass.__name__).lower())
 


Test patch:
Repository: DataBiosphere/toil

Issue:
getJobStoreClasses log message is not descriptive enough
If a job store class cannot be imported the following messaged is logged: 
`toil.jobStores.abstractJobStore: Unable to import 'toil.jobStores.azureJobStore'`

This message should be more descriptive.



Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/abstractJobStore.py b/src/toil/jobStores/abstractJobStore.py
--- a/src/toil/jobStores/abstractJobStore.py
+++ b/src/toil/jobStores/abstractJobStore.py
@@ -94,10 +94,10 @@ def getJobStoreClasses():
     jobStoreClassNames = (
         "toil.jobStores.azureJobStore.AzureJobStore",
         "toil.jobStores.fileJobStore.FileJobStore",
+        "toil.jobStores.googleJobStore.GoogleJobStore",
         "toil.jobStores.aws.jobStore.AWSJobStore",
         "toil.jobStores.abstractJobStore.JobStoreSupport"
     )
-
     jobStoreClasses = []
     for className in jobStoreClassNames:
         moduleName, className = className.rsplit('.', 1)
@@ -105,7 +105,8 @@ def getJobStoreClasses():
         try:
             module = import_module(moduleName)
         except ImportError:
-            logger.info("Unable to import '%s'" % moduleName)
+            logger.info("Unable to import '%s'. This may be because Toil was not installed with the"
+                        " relevant extra." % moduleName)
         else:
             jobStoreClasses.append(getattr(module, className))
     return jobStoreClasses


Test patch:
Repository: DataBiosphere/toil

Issue:
retry_azure is broken
There is a missing return statement. [See](https://github.com/BD2KGenomics/toil/blob/master/src/toil/jobStores/azureJobStore.py#L789)



Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/azureJobStore.py b/src/toil/jobStores/azureJobStore.py
--- a/src/toil/jobStores/azureJobStore.py
+++ b/src/toil/jobStores/azureJobStore.py
@@ -786,4 +786,4 @@ def defaultRetryPredicate(exception):
 
 
 def retry_azure(delays=(0, 1, 1, 4, 16, 64), timeout=300, predicate=defaultRetryPredicate):
-    retry(delays=delays, timeout=timeout, predicate=predicate)
+    return retry(delays=delays, timeout=timeout, predicate=predicate)


Test patch:
Repository: DataBiosphere/toil

Issue:
SSLError during SDB requests
http://jenkins.cgcloud.info/job/toil-pull-requests/1462/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsMesos/

Needs retry.



Discussion:
This has exactly the same stacktrace
http://jenkins.cgcloud.info/job/toil-pull-requests/1465/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsMesos/

and this one is similar, but crashes with the same SSLError
http://jenkins.cgcloud.info/job/toil-pull-requests/1465/testReport/junit/src.toil.test.src.jobCacheTest/AwsJobStoreCacheTest/testWriteNonLocalFileToJobStore/

I fear SDB is going bonkers.


Resolving patch:
diff --git a/src/toil/jobStores/aws/utils.py b/src/toil/jobStores/aws/utils.py
--- a/src/toil/jobStores/aws/utils.py
+++ b/src/toil/jobStores/aws/utils.py
@@ -20,6 +20,8 @@
 import types
 
 import errno
+from ssl import SSLError
+
 from boto.exception import (SDBResponseError,
                             BotoServerError,
                             S3ResponseError,
@@ -202,8 +204,16 @@ def no_such_sdb_domain(e):
             and e.error_code.endswith('NoSuchDomain'))
 
 
+def retryable_ssl_error(e):
+    # https://github.com/BD2KGenomics/toil/issues/978
+    return isinstance(e, SSLError) and e.reason == 'DECRYPTION_FAILED_OR_BAD_RECORD_MAC'
+
+
 def retryable_sdb_errors(e):
-    return sdb_unavailable(e) or no_such_sdb_domain(e) or connection_reset(e)
+    return (sdb_unavailable(e)
+            or no_such_sdb_domain(e)
+            or connection_reset(e)
+            or retryable_ssl_error(e))
 
 
 def retry_sdb(delays=default_delays, timeout=default_timeout, predicate=retryable_sdb_errors):
@@ -223,21 +233,21 @@ def retry_s3(delays=default_delays, timeout=default_timeout, predicate=retryable
     return retry(delays=delays, timeout=timeout, predicate=predicate)
 
 
-def region_to_bucket_location( region ):
+def region_to_bucket_location(region):
     if region == 'us-east-1':
         return ''
     else:
         return region
 
 
-def bucket_location_to_region( location ):
+def bucket_location_to_region(location):
     if location == '':
         return 'us-east-1'
     else:
         return location
 
 
-def bucket_location_to_http_url( location ):
+def bucket_location_to_http_url(location):
     if location:
         return 'https://s3-' + location + '.amazonaws.com'
     else:


Test patch:
Repository: DataBiosphere/toil

Issue:
Conflict error on AWS cache test
http://jenkins.cgcloud.info/job/toil-pull-requests/1514/testReport/src.toil.test.src.jobCacheTest/AwsJobStoreCacheTest/testRemoveLocalImmutablyReadFile/



Discussion:
@JakeNarkizian, I edited the description. Please don't use 'see' as the anchor for all your links. That's unorthodox and not very readable.

I think this can be fixed with a retry during bucket creation.


Resolving patch:
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -545,8 +545,9 @@ def _getOrCreateBucket(self, bucket_name, versioning=False):
                         location = region_to_bucket_location(self.region)
                         bucket = self.s3.create_bucket(bucket_name, location=location)
                     except S3CreateError as e:
-                        if e.error_code == 'BucketAlreadyOwnedByYou':
+                        if e.error_code in ('BucketAlreadyOwnedByYou', 'OperationAborted'):
                             # https://github.com/BD2KGenomics/toil/issues/955
+                            # https://github.com/BD2KGenomics/toil/issues/995
                             log.warn('Got %s, retrying.', e)
                             # and loop
                         else:


Test patch:
Repository: DataBiosphere/toil

Issue:
cwltoil's writeFile uses root logger
https://github.com/BD2KGenomics/toil/pull/867#issuecomment-227446745

cwltoil's writeFile uses root logger
https://github.com/BD2KGenomics/toil/pull/867#issuecomment-227446745



Discussion:

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -161,7 +161,7 @@ def writeFile(writeFunc, index, x):
         try:
             index[x] = (writeFunc(rp), os.path.basename(x))
         except Exception as e:
-            logging.error("Got exception '%s' while writing '%s'", e, x)
+            cwllogger.error("Got exception '%s' while copying '%s'", e, x)
             raise
     return index[x]
 


Test patch:
Repository: DataBiosphere/toil

Issue:
Installation of 3.2.0 with `aws` extra fails due to missing futures dependency
This will go into 3.2.1.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -32,7 +32,8 @@
             'psutil==3.0.1'],
         'aws': [
             botoRequirement,
-            'cgcloud-lib==1.4a1.dev195' ],
+            'cgcloud-lib==1.4a1.dev195',
+            'futures==3.0.5'],
         'azure': [
             'azure==1.0.3'],
         'encryption': [
diff --git a/src/toil/version.py b/src/toil/version.py
--- a/src/toil/version.py
+++ b/src/toil/version.py
@@ -12,4 +12,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-version = '3.2.0'
+version = '3.2.1a1'


Test patch:
Repository: DataBiosphere/toil

Issue:
Spark master IP address is captured in constructor
And then used in the `start` method. The constructor and `start` method are not guaranteed to run on the same note.



Discussion:

Resolving patch:
diff --git a/src/toil/lib/spark.py b/src/toil/lib/spark.py
--- a/src/toil/lib/spark.py
+++ b/src/toil/lib/spark.py
@@ -178,10 +178,7 @@ def __init__(self,
         if cores is None:
             cores = multiprocessing.cpu_count()
 
-        if overrideLeaderIP is None:
-            self.hostname = subprocess.check_output(["hostname", "-f",])[:-1]
-        else:
-            self.hostname = overrideLeaderIP
+        self.hostname = overrideLeaderIP
 
         Job.Service.__init__(self, memory=memory, cores=cores, disk=disk)
 
@@ -193,6 +190,9 @@ def start(self, fileStore):
         :param fileStore: Unused
         """
 
+        if self.hostname is None:
+            self.hostname = subprocess.check_output(["hostname", "-f",])[:-1]
+
         _log.info("Started Spark master container.")
         self.sparkContainerID = _docker_call(no_rm = True,
                                              work_dir = os.getcwd(),


Test patch:
Repository: DataBiosphere/toil

Issue:
Installation of 3.2.0 with `aws` extra fails due to missing futures dependency
This will go into 3.2.1.

Spark master IP address is captured in constructor
And then used in the `start` method. The constructor and `start` method are not guaranteed to run on the same note.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -32,7 +32,8 @@
             'psutil==3.0.1'],
         'aws': [
             botoRequirement,
-            'cgcloud-lib==1.4a1.dev195' ],
+            'cgcloud-lib==1.4a1.dev195',
+            'futures==3.0.5'],
         'azure': [
             'azure==1.0.3'],
         'encryption': [
diff --git a/src/toil/lib/spark.py b/src/toil/lib/spark.py
--- a/src/toil/lib/spark.py
+++ b/src/toil/lib/spark.py
@@ -178,10 +178,7 @@ def __init__(self,
         if cores is None:
             cores = multiprocessing.cpu_count()
 
-        if overrideLeaderIP is None:
-            self.hostname = subprocess.check_output(["hostname", "-f",])[:-1]
-        else:
-            self.hostname = overrideLeaderIP
+        self.hostname = overrideLeaderIP
 
         Job.Service.__init__(self, memory=memory, cores=cores, disk=disk)
 
@@ -193,6 +190,9 @@ def start(self, fileStore):
         :param fileStore: Unused
         """
 
+        if self.hostname is None:
+            self.hostname = subprocess.check_output(["hostname", "-f",])[:-1]
+
         _log.info("Started Spark master container.")
         self.sparkContainerID = _docker_call(no_rm = True,
                                              work_dir = os.getcwd(),


Test patch:
Repository: DataBiosphere/toil

Issue:
Give attributes to writeGlobalFile's return value
Start with size. So we can use it in promise lambdas



Discussion:
This needs some explanation. Why are we pursuing this?

I wanted to see if this works before I added a bunch of documentation to it. I don't want to bake the size of files into my pipelines (i.e. saying run_bwa returns a tuple of (fsID of bam file, fsID of bai file, size of bam file). instead, in the lambdas we can just do something like `disk=PromisedRequirement(lambda x: x.size+30000, A.rv())`


Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -3046,9 +3046,4 @@ def __init__(self, fileStoreID, size):
 
     @classmethod
     def forPath(cls, fileStoreID, filePath):
-        size = cls._resolveFileSize(filePath)
-        return cls(fileStoreID, size)
-
-    @staticmethod
-    def _resolveFileSize(filePath):
-        return os.stat(filePath).st_size
+        return cls(fileStoreID, os.stat(filePath).st_size)


Test patch: