internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataBiosphere/toil

Issue:
setuptools 8.0.0 or later is now required for PEP 440 version support
python 2.7.9
setuptools 5.5.1
```
Traceback (most recent call last):
  File "version_template.py", line 137, in <module>
    _main()
  File "version_template.py", line 133, in _main
    sys.stdout.write(expand_(*sys.argv[1:]))
  File "version_template.py", line 128, in expand_
    return resolve(name)
  File "version_template.py", line 122, in resolve
    v = v()
  File "version_template.py", line 61, in distVersion
    if build_number is not None and parse_version(baseVersion).is_prerelease:
AttributeError: 'tuple' object has no attribute 'is_prerelease'
```

setuptools 8.0.0 or later is now required for PEP 440 version support
python 2.7.9
setuptools 5.5.1
```
Traceback (most recent call last):
  File "version_template.py", line 137, in <module>
    _main()
  File "version_template.py", line 133, in _main
    sys.stdout.write(expand_(*sys.argv[1:]))
  File "version_template.py", line 128, in expand_
    return resolve(name)
  File "version_template.py", line 122, in resolve
    v = v()
  File "version_template.py", line 61, in distVersion
    if build_number is not None and parse_version(baseVersion).is_prerelease:
AttributeError: 'tuple' object has no attribute 'is_prerelease'
```



Discussion:
I don't want to drop the requirement but I'll make it more explicit. Thank you for catching this.

I don't want to drop the requirement but I'll make it more explicit. Thank you for catching this.


Resolving patch:
diff --git a/version_template.py b/version_template.py
--- a/version_template.py
+++ b/version_template.py
@@ -22,7 +22,7 @@
 
 # Note to maintainers:
 #
-#  - don't import at module level unless you intend for the import to be included in the output
+#  - don't import at module level unless you want the imported value to be included in the output
 #  - only import from the Python standard run-time library (you can't have any dependencies)
 
 baseVersion = '3.5.0a1'
@@ -58,7 +58,12 @@ def distVersion():
     """
     from pkg_resources import parse_version
     build_number = buildNumber()
-    if build_number is not None and parse_version(baseVersion).is_prerelease:
+    parsedBaseVersion = parse_version(baseVersion)
+    if isinstance(parsedBaseVersion, tuple):
+        raise RuntimeError("Setuptools version 8.0 or newer required. Update by running "
+                           "'pip install setuptools --upgrade'")
+
+    if build_number is not None and parsedBaseVersion.is_prerelease:
         return baseVersion + '.dev' + build_number
     else:
         return baseVersion


Test patch:
Repository: DataBiosphere/toil

Issue:
All jobs should have access to the underlying fileStore
From #1198, [comment](https://github.com/BD2KGenomics/toil/pull/1198#discussion_r84343466). Right now, only `ServiceJob` and `JobFunctionWrappingJob` set `self.fileStore`. We should make this consistent across jobs.


Discussion:

Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -293,7 +293,7 @@ def __init__(self, memory=None, cores=None, disk=None, preemptable=None, unitNam
         # entire return value.
         self._rvs = collections.defaultdict(list)
         self._promiseJobStore = None
-        self.fileStore = None
+        self._fileStore = None
 
     def run(self, fileStore):
         """
@@ -671,9 +671,9 @@ def defer(self, function, *args, **kwargs):
 
         :param dict kwargs: The keyword arguments to the function
         """
-        require( self.fileStore is not None, 'A deferred function may only be registered with a '
+        require(self._fileStore is not None, 'A deferred function may only be registered with a '
                                              'job while that job is running.')
-        self.fileStore._registerDeferredFunction(DeferredFunction.create(function, *args, **kwargs))
+        self._fileStore._registerDeferredFunction(DeferredFunction.create(function, *args, **kwargs))
 
 
     ####################################################
@@ -1280,7 +1280,9 @@ def _runner(self, jobGraph, jobStore, fileStore):
                filestore
         :return:
         """
-        # Run the job
+        # Make fileStore available as an attribute during run() ...
+        self._fileStore = fileStore
+        # ... but also pass it to run() as an argument for backwards compatibility.
         returnValues = self._run(jobGraph, fileStore)
         # Serialize the new jobs defined by the run method to the jobStore
         self._serialiseExistingJob(jobGraph, jobStore, returnValues)
@@ -1391,9 +1393,13 @@ class JobFunctionWrappingJob(FunctionWrappingJob):
     instance (see :func:`toil.job.Job.run`), it is made a variable of the wrapping job \
     called fileStore.
     """
+
+    @property
+    def fileStore(self):
+        return self._fileStore
+
     def run(self, fileStore):
         userFunction = self._getUserFunction()
-        self.fileStore = fileStore
         rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
         return rValue
 
@@ -1537,11 +1543,11 @@ def __init__(self, service):
         # the job is run. It is used to access the start and terminate flags.
         self.jobGraph = None
 
-    def run(self, fileStore):
-
-        # we need access to the filestore from underneath the service job
-        self.fileStore = fileStore
+    @property
+    def fileStore(self):
+        return self._fileStore
 
+    def run(self, fileStore):
         # Unpickle the service
         logger.debug('Loading service module %s.', self.serviceModule)
         userModule = self._loadUserModule(self.serviceModule)


Test patch:
Repository: DataBiosphere/toil

Issue:
toil/resource.py throws uninformative assert
It'd be useful to know what the `dirPath` is, at least:

```
Traceback (most recent call last):
  File "/root/toil-scripts/.env/bin/toil-bwa", line 11, in <module>
    load_entry_point('toil-scripts==2.1.0a1', 'console_scripts', 'toil-bwa')()
  File "build/bdist.linux-x86_64/egg/toil_scripts/bwa_alignment/bwa_alignment.py", line 292, in main
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 390, in wrapJobFn
    return JobFunctionWrappingJob(fn, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1246, in __init__
    self.userFunctionModule = ModuleDescriptor.forModule(userFunction.__module__).globalize()
  File "/usr/local/lib/python2.7/dist-packages/toil/resource.py", line 381, in forModule
    assert os.path.isdir(dirPath)
AssertionError
```


Discussion:
But the assert firing is a bug in its own right, correct? What triggered this?

No, I think the assert fired because I installed some `.egg-links` into a Virtualenv, which we officially don't support. That being said, I can't confirm that without fixing this first.

If an unsuspecting user can mess this up, maybe this is a case for `require()`.

Yeah, it should definitely be a `require()`, and with an error message that relates the bad path.


Resolving patch:
diff --git a/src/toil/resource.py b/src/toil/resource.py
--- a/src/toil/resource.py
+++ b/src/toil/resource.py
@@ -394,7 +394,9 @@ def forModule(cls, name):
                 assert dirPathTail == package
             dirPath = os.path.sep.join(filePath)
         log.debug("Module dir is %s", dirPath)
-        assert os.path.isdir(dirPath)
+        require(os.path.isdir(dirPath),
+                'Bad directory path %s for module %s. Note that hot-deployment does not support \
+                .egg-link files yet, or scripts located in the root directory.', dirPath, name)
         fromVirtualEnv = inVirtualEnv() and dirPath.startswith(sys.prefix)
         return cls(dirPath=dirPath, name=name, fromVirtualEnv=fromVirtualEnv)
 


Test patch:
Repository: DataBiosphere/toil

Issue:
Improve logging of config
@cket and I think it would be better to just log the config as one long line rather than multiple lines.

Also, http://jenkins.cgcloud.info/job/toil-pull-requests/2299/testReport/junit/src.toil.test.src.hotDeploymentTest/HotDeploymentTest/testDeferralWithFailureAndEncapsulation/ shows that config seems to logged twice. Search for `x/l/jobsv6IB5    ` in above console output.
Improve logging of config
@cket and I think it would be better to just log the config as one long line rather than multiple lines.

Also, http://jenkins.cgcloud.info/job/toil-pull-requests/2299/testReport/junit/src.toil.test.src.hotDeploymentTest/HotDeploymentTest/testDeferralWithFailureAndEncapsulation/ shows that config seems to logged twice. Search for `x/l/jobsv6IB5    ` in above console output.


Discussion:

Resolving patch:
diff --git a/src/toil/__init__.py b/src/toil/__init__.py
--- a/src/toil/__init__.py
+++ b/src/toil/__init__.py
@@ -17,7 +17,6 @@
 import logging
 import os
 import sys
-from pprint import pformat
 
 from subprocess import check_output
 
@@ -141,9 +140,10 @@ def lookupEnvVar(name, envName, defaultValue):
         return value
 
 
-def logProcessContext(config, logger):
+def logProcessContext(config):
     # toil.version.version (string) canont be imported at top level because it conflicts with
     # toil.version (module) and Sphinx doesn't like that.
     from toil.version import version
-    logger.info("Running Toil version %s.", version)
-    logger.info("Configuration:\n %s", pformat(config.__dict__))
+    log.info("Running Toil version %s.", version)
+    log.debug("Configuration: %s", config.__dict__)
+
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -912,7 +912,7 @@ def _runMainLoop(self, rootJob):
         :param toil.job.Job rootJob: The root job for the workflow.
         :rtype: Any
         """
-        logProcessContext(self.config, logger)
+        logProcessContext(self.config)
 
         with RealtimeLogger(self._batchSystem,
                             level=self.options.logLevel if self.options.realTimeLogging else None):
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -186,13 +186,6 @@ def badWorker():
     #Close the descriptor we used to open the file
     os.close(logFh)
 
-    for handler in list(logger.handlers): #Remove old handlers
-        logger.removeHandler(handler)
-    
-    #Add the new handler. The sys.stderr stream has been redirected by swapping
-    #the file descriptor out from under it.
-    logger.addHandler(logging.StreamHandler(sys.stderr))
-
     debugging = logging.getLogger().isEnabledFor(logging.DEBUG)
     ##########################################
     #Worker log file trapped from here on in
@@ -215,7 +208,7 @@ def badWorker():
         logger.debug("Next available file descriptor: {}".format(
             nextOpenDescriptor()))
 
-        logProcessContext(config, logger)
+        logProcessContext(config)
 
         ##########################################
         #Load the jobGraph


Test patch:
Repository: DataBiosphere/toil

Issue:
Improve toil cluster utils CLI
Instead of requiring the -p flag make it default to 'aws'. It would also be great to make the cluster name optional -- if the user only has 1 cluster running it should use that by default, else raise an error asking for an explicit argument


Discussion:

Resolving patch:
diff --git a/src/toil/utils/__init__.py b/src/toil/utils/__init__.py
--- a/src/toil/utils/__init__.py
+++ b/src/toil/utils/__init__.py
@@ -5,12 +5,12 @@
 
 from toil.provisioners.aws import getCurrentAWSZone
 
-logger = logging.getLogger( __name__ )
+logger = logging.getLogger(__name__)
 
 
 def addBasicProvisionerOptions(parser):
     parser.add_argument("--version", action='version', version=version)
-    parser.add_argument('-p', "--provisioner", dest='provisioner', choices=['aws'], required=True,
+    parser.add_argument('-p', "--provisioner", dest='provisioner', choices=['aws'], required=False, default="aws",
                         help="The provisioner for cluster auto-scaling. Only aws is currently "
                              "supported")
     currentZone = getCurrentAWSZone()


Test patch:
diff --git a/src/toil/test/utils/utilsTest.py b/src/toil/test/utils/utilsTest.py
--- a/src/toil/test/utils/utilsTest.py
+++ b/src/toil/test/utils/utilsTest.py
@@ -83,8 +83,10 @@ def statusCommand(self):
     def testAWSProvisionerUtils(self):
         clusterName = 'cluster-utils-test' + str(uuid.uuid4())
         try:
-            system([self.toilMain, 'launch-cluster', '--nodeType=t2.micro', '--keyPairName=jenkins@jenkins-master',
-                    clusterName, '--provisioner=aws'])
+            # --provisioner flag should default to aws, so we're not explicitly
+            # specifying that here
+            system([self.toilMain, 'launch-cluster', '--nodeType=t2.micro',
+                    '--keyPairName=jenkins@jenkins-master', clusterName])
         finally:
             system([self.toilMain, 'destroy-cluster', '--provisioner=aws', clusterName])
         try:
Repository: DataBiosphere/toil

Issue:
Install screen in appliance



Discussion:

Resolving patch:
diff --git a/docker/Dockerfile.py b/docker/Dockerfile.py
--- a/docker/Dockerfile.py
+++ b/docker/Dockerfile.py
@@ -27,7 +27,8 @@
                          'wget',
                          'curl',
                          'mesos=1.0.0-2.0.89.ubuntu1404',
-                         'rsync'])
+                         'rsync',
+                         'screen'])
 
 
 def heredoc(s):
@@ -76,7 +77,7 @@ def heredoc(s):
     # Install statically linked version of docker client
     RUN wget -O /usr/bin/docker https://get.docker.com/builds/Linux/x86_64/docker-1.10.3 \
         && chmod +x /usr/bin/docker
-    
+
     # Fix for Mesos interface dependency missing on ubuntu
     RUN pip install protobuf==3.0.0
 
@@ -86,6 +87,9 @@ def heredoc(s):
     # Fix for https://issues.apache.org/jira/browse/MESOS-3793
     ENV MESOS_LAUNCHER=posix
 
+    # Fix for `screen` (https://github.com/BD2KGenomics/toil/pull/1386#issuecomment-267424561)
+    ENV TERM linux
+
     # An appliance may need to start more appliances, e.g. when the leader appliance launches the
     # worker appliance on a worker node. To support this, we embed a self-reference into the image:
     ENV TOIL_APPLIANCE_SELF {applianceSelf}


Test patch:
Repository: DataBiosphere/toil

Issue:
Rename toil appliance containers used in AWS provisioner
Right now they are just called 'leader' and 'worker. They should be called 'toil_leader' and 'toil_worker'


Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/aws/__init__.py b/src/toil/provisioners/aws/__init__.py
--- a/src/toil/provisioners/aws/__init__.py
+++ b/src/toil/provisioners/aws/__init__.py
@@ -280,7 +280,7 @@ def _get_spot_history(ctx, instance_type):
             -v /var/lib/mesos:/var/lib/mesos \
             -v /var/lib/docker:/var/lib/docker \
             -v /var/lib/toil:/var/lib/toil \
-            --name={role} \
+            --name=toil_{role} \
             {image} \
             {args}
 """
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -124,7 +124,7 @@ def _sshAppliance(cls, leaderIP, *args, **kwargs):
         args = map(pipes.quote, args)
         ttyFlag = '-t' if tty else ''
         commandTokens = ['ssh', '-o', "StrictHostKeyChecking=no", '-t', 'core@%s' % leaderIP,
-                         'docker', 'exec', '-i', ttyFlag, 'leader'] + args
+                         'docker', 'exec', '-i', ttyFlag, 'toil_leader'] + args
         inputString = kwargs.pop('input', None)
         if inputString is not None:
             kwargs['stdin'] = subprocess.PIPE
@@ -148,7 +148,7 @@ def _sshInstance(cls, leaderIP, *args):
     def rsyncLeader(cls, clusterName, args):
         leader = cls._getLeader(clusterName)
         sshCommand = 'ssh -o "StrictHostKeyChecking=no"'  # Skip host key checking
-        remoteRsync = "docker exec -i leader rsync"  # Access rsync inside appliance
+        remoteRsync = "docker exec -i toil_leader rsync"  # Access rsync inside appliance
         parsedArgs = []
         hostInserted = False
         # Insert remote host address
@@ -200,7 +200,7 @@ def _waitForAppliance(cls, ip_address):
         logger.info('Waiting for leader Toil appliance to start...')
         while True:
             output = cls._sshInstance(ip_address, 'docker', 'ps')
-            if 'leader' in output:
+            if 'toil_leader' in output:
                 logger.info('...Toil appliance started')
                 break
             else:


Test patch:
Repository: DataBiosphere/toil

Issue:
Scaler thread shutdown error
File "/home/rnaenv/bin/toil-rnaseq", line 11, in <module>
    sys.exit(main())
  File "/home/rnaenv/local/lib/python2.7/site-packages/toil_rnaseq/rnaseq_cgl_pipeline.py", line 573, in main
    Job.Runner.startToil(Job.wrapJobFn(map_job, download_sample, samples, config), args)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 738, in startToil
    return toil.start(job)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 655, in start
    return self._runMainLoop(rootJobGraph)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 963, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/usr/local/lib/python2.7/dist-packages/toil/leader.py", line 170, in run
    self.clusterScaler.shutdown()
  File "/usr/local/lib/python2.7/dist-packages/toil/provisioners/clusterScaler.py", line 262, in shutdown
    self.scaler.join()
AttributeError: 'NoneType' object has no attribute 'join'


Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/clusterScaler.py b/src/toil/provisioners/clusterScaler.py
--- a/src/toil/provisioners/clusterScaler.py
+++ b/src/toil/provisioners/clusterScaler.py
@@ -259,7 +259,7 @@ def shutdown(self):
         self.stop = True
         for scaler in self.preemptableScaler, self.scaler:
             if scaler is not None:
-                self.scaler.join()
+                scaler.join()
 
     def addCompletedJob(self, job, wallTime):
         """


Test patch:
Repository: DataBiosphere/toil

Issue:
Node preemptability is determined inconsistently
The batch system looks for a text value for preemptability while the cluster scaler code looks for a scalar value.


Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/mesos/batchSystem.py b/src/toil/batchSystems/mesos/batchSystem.py
--- a/src/toil/batchSystems/mesos/batchSystem.py
+++ b/src/toil/batchSystems/mesos/batchSystem.py
@@ -437,7 +437,7 @@ def _trackOfferedNodes(self, offers):
             preemptable = False
             for attribute in offer.attributes:
                 if attribute.name == 'preemptable':
-                    preemptable = bool(attribute.scalar.value)
+                    preemptable = strict_bool(attribute.text.value)
             if preemptable:
                 try:
                     self.nonPreemptibleNodes.remove(offer.slave_id.value)


Test patch:
Repository: DataBiosphere/toil

Issue:
Add missing abstract method in LSF batchsystem

Add missing abstract method in LSF batchsystem



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/lsf.py b/src/toil/batchSystems/lsf.py
--- a/src/toil/batchSystems/lsf.py
+++ b/src/toil/batchSystems/lsf.py
@@ -137,6 +137,9 @@ def supportsWorkerCleanup(cls):
     def supportsHotDeployment(cls):
         return False
 
+    def shutdown(self):
+        pass
+
     def __init__(self, config, maxCores, maxMemory, maxDisk):
         super(LSFBatchSystem, self).__init__(config, maxCores, maxMemory, maxDisk)
         self.lsfResultsFile = self._getResultsFileName(config.jobStore)


Test patch:
Repository: DataBiosphere/toil

Issue:
If zone is not set the cluster utils fail very slowly

If zone is not set the cluster utils fail very slowly



Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/aws/__init__.py b/src/toil/provisioners/aws/__init__.py
--- a/src/toil/provisioners/aws/__init__.py
+++ b/src/toil/provisioners/aws/__init__.py
@@ -18,6 +18,8 @@
 import datetime
 from cgcloud.lib.util import std_dev, mean
 
+from toil.test import runningOnEC2
+
 logger = logging.getLogger(__name__)
 
 ZoneTuple = namedtuple('ZoneTuple', ['name', 'price_deviation'])
@@ -50,7 +52,7 @@ def _getCurrentAWSZone(spotBid=None, nodeType=None, ctx=None):
             zone = boto.config.get('Boto', 'ec2_region_name')
             if zone is not None:
                 zone += 'a'  # derive an availability zone in the region
-        if not zone:
+        if not zone and runningOnEC2():
             try:
                 zone = get_instance_metadata()['placement']['availability-zone']
             except KeyError:


Test patch:
Repository: DataBiosphere/toil

Issue:
Don't spin down EC2 nodes prematurely
Since we pay for EC2 nodes by the hour it doesn't make sense to terminate them until soon before their next billing cycle.
Don't spin down EC2 nodes prematurely
Since we pay for EC2 nodes by the hour it doesn't make sense to terminate them until soon before their next billing cycle.


Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/abstractProvisioner.py b/src/toil/provisioners/abstractProvisioner.py
--- a/src/toil/provisioners/abstractProvisioner.py
+++ b/src/toil/provisioners/abstractProvisioner.py
@@ -194,6 +194,9 @@ def _removeNodes(self, instances, numNodes, preemptable=False, force=False):
             nodesToTerminate.sort(key=lambda (instance, nodeInfo): (
                 nodeInfo.workers if nodeInfo else 1,
                 self._remainingBillingInterval(instance)))
+            if not force:
+                # don't terminate nodes that still have > 15% left in their allocated (prepaid) time
+                nodesToTerminate = [nodeTuple for nodeTuple in nodesToTerminate if self._remainingBillingInterval(nodeTuple[0]) <= 0.15]
             nodesToTerminate = nodesToTerminate[:numNodes]
             if log.isEnabledFor(logging.DEBUG):
                 for instance, nodeInfo in nodesToTerminate:


Test patch:
Repository: DataBiosphere/toil

Issue:
Handle race between finding jobState file and loading it
http://jenkins.cgcloud.info/job/toil-branches/branch/releases%252F3.5.x/13/testReport/src.toil.test.src.fileStoreTest/NonCachingFileStoreTestWithFileJobStore/testToilIsNotBroken/
Handle race between finding jobState file and loading it
http://jenkins.cgcloud.info/job/toil-branches/branch/releases%252F3.5.x/13/testReport/src.toil.test.src.fileStoreTest/NonCachingFileStoreTestWithFileJobStore/testToilIsNotBroken/


Discussion:

Resolving patch:
diff --git a/src/toil/fileStore.py b/src/toil/fileStore.py
--- a/src/toil/fileStore.py
+++ b/src/toil/fileStore.py
@@ -1783,7 +1783,14 @@ def _getAllJobStates(workflowDir):
                 if filename == '.jobState':
                     jobStateFiles.append(os.path.join(root, filename))
         for filename in jobStateFiles:
-            yield NonCachingFileStore._readJobState(filename)
+            try:
+                yield NonCachingFileStore._readJobState(filename)
+            except IOError as e:
+                if e.errno == 2:
+                    # job finished & deleted its jobState file since the jobState files were discovered
+                    continue
+                else:
+                    raise
 
     @staticmethod
     def _readJobState(jobStateFileName):


Test patch: