internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataBiosphere/toil

Issue:
Sort example in documentation writes output with restrictive permissions
When the sort example is run, it produces an output file with 444. If the example is run again, it will fail as the file is set to deny write permissions to the owner. Output should be set to 644 at minimum.

(cc #1303)


Discussion:

Resolving patch:
diff --git a/docs/running/toil-sort-example.py b/docs/running/toil-sort-example.py
--- a/docs/running/toil-sort-example.py
+++ b/docs/running/toil-sort-example.py
@@ -78,6 +78,7 @@ def cleanup(job, temp_output_id, output_file):
     """
     tempFile = job.fileStore.readGlobalFile(temp_output_id)
     shutil.copy(tempFile, output_file)
+    os.chmod(output_file, 0o644)
     job.fileStore.logToMaster("Finished copying sorted file to output: %s" % output_file)
 
 


Test patch:
Repository: DataBiosphere/toil

Issue:
aws provisioner role name can conflict with newer versions of cgcloud
The provisioner's role name suffix, `-toil`:

https://github.com/BD2KGenomics/toil/blob/a50d65147dd8c24bd3ab79ad1d1a080b4674e4ec/src/toil/provisioners/aws/awsProvisioner.py#L600

is incompatible with versions of cgcloud after this commit:

https://github.com/BD2KGenomics/cgcloud/commit/7dafcde84d61c6e9fd992f13449c79975fb48814

The error will look something like this:
```
Traceback (most recent call last):
  File "/home/joel/cactus/venv/bin/toil", line 11, in <module>
    load_entry_point('toil', 'console_scripts', 'toil')()
  File "/home/joel/toil/src/toil/utils/toilMain.py", line 33, in main
    module.main()
  File "/home/joel/toil/src/toil/utils/toilLaunchCluster.py", line 53, in main
    keyName=config.keyPairName, spotBid=spotBid, zone=config.zone)
  File "/home/joel/toil/src/toil/provisioners/aws/awsProvisioner.py", line 340, in launchCluster
    profileARN = cls._getProfileARN(ctx)
  File "/home/joel/toil/src/toil/provisioners/aws/awsProvisioner.py", line 601, in _getProfileARN
    iamRoleName = ctx.setup_iam_ec2_role(role_name=roleName, policies=policy)
  File "/home/joel/cactus/venv/local/lib/python2.7/site-packages/cgcloud/lib/context.py", line 651, in setup_iam_ec2_role
    aws_role_name = self.to_aws_name( role_name )
  File "/home/joel/cactus/venv/local/lib/python2.7/site-packages/cgcloud/lib/context.py", line 379, in to_aws_name
    name = self.absolute_name( name )
  File "/home/joel/cactus/venv/local/lib/python2.7/site-packages/cgcloud/lib/context.py", line 297, in absolute_name
    raise self.InvalidPathError( result )
cgcloud.lib.context.InvalidPathError: Invalid path '/jcarmstr/cactus/cluster/-toil'
```

The setup.py specifies the cgcloud dependency a version *prior* to this commit, but if (like me) you installed a different version of cgcloud in your virtualenv, I think the newer version is kept.

Obviously this isn't a big deal since it's basically user error (using a wrong version of a dependency). But maybe the role name suffix could be changed to just `toil` (no hyphen) to avoid this?


Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -608,7 +608,7 @@ def groupNotFound(e):
     def _getProfileARN(cls, ctx):
         def addRoleErrors(e):
             return e.status == 404
-        roleName = '-toil'
+        roleName = 'toil'
         policy = dict(iam_full=iamFullPolicy, ec2_full=ec2FullPolicy,
                       s3_full=s3FullPolicy, sbd_full=sdbFullPolicy)
         iamRoleName = ctx.setup_iam_ec2_role(role_name=roleName, policies=policy)


Test patch:
Repository: DataBiosphere/toil

Issue:
Add option to disable strict CWL evaluation
@chapmanb any opinions on this? It seems like it would be helpful to be able to control this in CWLToil
Add option to disable strict CWL evaluation
@chapmanb any opinions on this? It seems like it would be helpful to be able to control this in CWLToil


Discussion:
CJ -- I don't know how to do this but if it's helpful for some use case it makes sense to explore. What cases are you running into where the CWL fails evaluation but will still run correctly that we want to enable?
I was given a CWL workflow that raised the following error:
```
Object `test.cwl` is not valid because tried `CommandLineTool` but 
unrecognized extension field xxxx and strict is True.  Did you include a $schemas section?
```
When I changed CWLToil to use strict=False everything worked OK
CJ -- I don't know how to do this but if it's helpful for some use case it makes sense to explore. What cases are you running into where the CWL fails evaluation but will still run correctly that we want to enable?
I was given a CWL workflow that raised the following error:
```
Object `test.cwl` is not valid because tried `CommandLineTool` but 
unrecognized extension field xxxx and strict is True.  Did you include a $schemas section?
```
When I changed CWLToil to use strict=False everything worked OK

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -619,6 +619,7 @@ def main(args=None, stdout=sys.stdout):
     # user to select jobStore or get a default from logic one below.
     parser.add_argument("--jobStore", type=str)
     parser.add_argument("--conformance-test", action="store_true")
+    parser.add_argument("--not-strict", action="store_true")
     parser.add_argument("--no-container", action="store_true")
     parser.add_argument("--quiet", dest="logLevel", action="store_const", const="ERROR")
     parser.add_argument("--basedir", type=str)
@@ -648,9 +649,10 @@ def main(args=None, stdout=sys.stdout):
     if options.logLevel:
         cwllogger.setLevel(options.logLevel)
 
+    useStrict = not options.not_strict
     try:
         t = cwltool.load_tool.load_tool(options.cwltool, cwltool.workflow.defaultMakeTool,
-                                        resolver=cwltool.resolver.tool_resolver)
+                                        resolver=cwltool.resolver.tool_resolver, strict=useStrict)
     except cwltool.process.UnsupportedRequirement as e:
         logging.error(e)
         return 33


Test patch:
Repository: DataBiosphere/toil

Issue:
Non-preemptable jobs can be run on preemptable nodes
I'm about 90% sure this is due to a mistake in the "job chain" functionality, I think have a fix and am testing overnight.
Non-preemptable jobs can be run on preemptable nodes
I'm about 90% sure this is due to a mistake in the "job chain" functionality, I think have a fix and am testing overnight.


Discussion:

Resolving patch:
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -383,6 +383,9 @@ def recursiveDelete(jobGraph2):
             if successorJobNode.disk > jobGraph.disk:
                 logger.debug("We need more disk for the next job, so finishing")
                 break
+            if successorJobNode.preemptable != jobGraph.preemptable:
+                logger.debug("Preemptability is different for the next job, returning to the leader")
+                break
             if successorJobNode.predecessorNumber > 1:
                 logger.debug("The jobGraph has multiple predecessors, we must return to the leader.")
                 break


Test patch:
Repository: DataBiosphere/toil

Issue:
Look into root volume deletion 

Look into root volume deletion 



Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -450,11 +450,16 @@ def expectedShutdownErrors(e):
     @classmethod
     def _terminateInstances(cls, instances, ctx):
         instanceIDs = [x.id for x in instances]
+        # get all volume ids now. After termination the block device mapping attr
+        # will not be populated
+        ebsIDs = [x.block_device_mapping["/dev/xvda"].volume_id for x in instances]
+
         logger.info('Terminating instance(s): %s', instanceIDs)
         cls._terminateIDs(instanceIDs, ctx)
         logger.info('... Waiting for instance(s) to shut down...')
         for instance in instances:
             wait_transition(instance, {'running', 'shutting-down'}, 'terminated')
+        cls._deleteRootEBS(ebsIDs, ctx)
         logger.info('Instance(s) terminated.')
 
     @classmethod
@@ -465,6 +470,13 @@ def _terminateIDs(cls, instanceIDs, ctx):
                 ctx.ec2.terminate_instances(instance_ids=instanceIDs)
         logger.info('Instance(s) terminated.')
 
+    @classmethod
+    def _deleteRootEBS(cls, ebsIDs, ctx):
+        for volumeID in ebsIDs:
+            for attempt in retry(predicate=AWSProvisioner.throttlePredicate):
+                with attempt:
+                    ctx.ec2.delete_volume(volumeID)
+
     def _logAndTerminate(self, instances):
         self._terminateInstances(instances, self.ctx)
 


Test patch:
diff --git a/src/toil/test/provisioners/aws/awsProvisionerTest.py b/src/toil/test/provisioners/aws/awsProvisionerTest.py
--- a/src/toil/test/provisioners/aws/awsProvisionerTest.py
+++ b/src/toil/test/provisioners/aws/awsProvisionerTest.py
@@ -17,6 +17,10 @@
 from abc import abstractmethod
 from inspect import getsource
 from textwrap import dedent
+
+from boto.ec2.blockdevicemapping import BlockDeviceType
+from boto.exception import EC2ResponseError
+
 from toil.provisioners.aws.awsProvisioner import AWSProvisioner
 
 from uuid import uuid4
@@ -152,7 +156,22 @@ def _test(self, spotInstances=False, fulfillableBid=True):
 
         self.sshUtil(checkStatsCommand)
 
+        ctx = AWSProvisioner._buildContext(self.clusterName)
+        assert isinstance(self.leader.block_device_mapping["/dev/xvda"], BlockDeviceType)
+        volumeID = self.leader.block_device_mapping["/dev/xvda"].volume_id
+        ctx.ec2.get_all_volumes(volume_ids=[volumeID])
         AWSProvisioner.destroyCluster(self.clusterName)
+        self.leader.update()
+        try:
+            ctx.ec2.get_all_volumes(volume_ids=[volumeID])
+        except EC2ResponseError as e:
+            if e.status == 400 and 'InvalidVolume.NotFound' in e.code:
+                pass
+            else:
+                raise
+        else:
+            self.fail('Volume with ID %s was not cleaned up properly' % volumeID)
+
         assert len(self.getMatchingRoles(self.clusterName)) == 0
Repository: DataBiosphere/toil

Issue:
Add option to disable hot-deployment
If the User script has non-python dependencies in the package, they will not be hot-deployed. Allow the user to disable hot-deployment .

Related to #1570 


Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -69,6 +69,7 @@ def __init__(self):
 
         #Batch system options
         self.batchSystem = "singleMachine"
+        self.disableHotDeployment = False
         self.scale = 1
         self.mesosMasterAddress = 'localhost:5050'
         self.parasolCommand = "parasol"
@@ -199,6 +200,7 @@ def parseJobStore(s):
 
         #Batch system options
         setOption("batchSystem")
+        setOption("disableHotDeployment")
         setOption("scale", float, fC(0.0))
         setOption("mesosMasterAddress")
         setOption("parasolCommand")
@@ -338,6 +340,10 @@ def _addOptions(addGroupFn, config):
     addOptionFn("--batchSystem", dest="batchSystem", default=None,
                       help=("The type of batch system to run the job(s) with, currently can be one "
                             "of singleMachine, parasol, gridEngine, lsf or mesos'. default=%s" % config.batchSystem))
+    addOptionFn("--disableHotDeployment", dest="disableHotDeployment", action='store_true', default=None,
+                help=("Should hot-deployment of the user script be deactivated? If True, the user "
+                      "script/package should be present at the same location on all workers. "
+                      "default=%s" % config.disableHotDeployment))
     addOptionFn("--scale", dest="scale", default=None,
                 help=("A scaling factor to change the value of all submitted tasks's submitted cores. "
                       "Used in singleMachine batch system. default=%s" % config.scale))
@@ -839,7 +845,8 @@ def _setupHotDeployment(self, userScript=None):
                 logger.info('User script %s belongs to Toil. No need to hot-deploy it.', userScript)
                 userScript = None
             else:
-                if self._batchSystem.supportsHotDeployment():
+                if (self._batchSystem.supportsHotDeployment() and
+                        not self.config.disableHotDeployment):
                     # Note that by saving the ModuleDescriptor, and not the Resource we allow for
                     # redeploying a potentially modified user script on workflow restarts.
                     with self._jobStore.writeSharedFileStream('userScript') as f:


Test patch:
Repository: DataBiosphere/toil

Issue:
effectiveRequirements and PromisedRequirement are not explained well in the documentation
What are they? How do I effectively use them?

The best I could find is this badly-formatted example:

http://toil.readthedocs.io/en/releases-3.6.x/toilAPI.html?#toil.job.PromisedRequirement.__init__

But is there a way to involve the size of a file in the FileStore? Or values coming from multiple jobs?

And is there a way to attach the requirement determination logic to the job being scheduled, somehow, and not the job scheduling it?


Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/abstractJobStore.py b/src/toil/jobStores/abstractJobStore.py
--- a/src/toil/jobStores/abstractJobStore.py
+++ b/src/toil/jobStores/abstractJobStore.py
@@ -199,7 +199,7 @@ def createRootJob(self, *args, **kwargs):
         """
         Create a new job and set it as the root job in this job store
 
-        :rtype : toil.jobGraph.JobGraph
+        :rtype: toil.jobGraph.JobGraph
         """
         rootJob = self.create(*args, **kwargs)
         self.setRootJob(rootJob.jobStoreID)
@@ -272,7 +272,7 @@ def importFile(self, srcUrl, sharedFileName=None):
 
         :param str sharedFileName: Optional name to assign to the imported file within the job store
 
-        :return The jobStoreFileId of the imported file or None if sharedFileName was given
+        :return: The jobStoreFileId of the imported file or None if sharedFileName was given
         :rtype: FileID|None
         """
         # Note that the helper method _importFile is used to read from the source and write to


Test patch:
Repository: DataBiosphere/toil

Issue:
Toil Appliance should set /var/lib/toil as the default workdir 

Toil Appliance should set /var/lib/toil as the default workdir 



Discussion:

Resolving patch:
diff --git a/docker/Dockerfile.py b/docker/Dockerfile.py
--- a/docker/Dockerfile.py
+++ b/docker/Dockerfile.py
@@ -103,6 +103,10 @@ def heredoc(s):
     # worker appliance on a worker node. To support this, we embed a self-reference into the image:
     ENV TOIL_APPLIANCE_SELF {applianceSelf}
 
+    RUN mkdir /var/lib/toil
+
+    ENV TOIL_WORKDIR /var/lib/toil
+
     # This component changes most frequently and keeping it last maximizes Docker cache hits.
     COPY {sdistName} .
     RUN pip install {sdistName}[aws,mesos,encryption,cwl]


Test patch:
diff --git a/src/toil/test/utils/utilsTest.py b/src/toil/test/utils/utilsTest.py
--- a/src/toil/test/utils/utilsTest.py
+++ b/src/toil/test/utils/utilsTest.py
@@ -136,6 +136,9 @@ def testAWSProvisionerUtils(self):
                 self.fail('The remote command failed silently where it should have '
                           'raised an error')
 
+            AWSProvisioner.sshLeader(clusterName=clusterName,
+                                     args=['python', '-c', "import os; assert os.environ['TOIL_WORKDIR']=='/var/lib/toil'"])
+
             # `toil rsync-cluster`
             # Testing special characters - string.punctuation
             fname = '!"#$%&\'()*+,-.;<=>:\ ?@[\\\\]^_`{|}~'
Repository: DataBiosphere/toil

Issue:
toil launch-cluster ignores AWS AZ
The AWS provisioner always requires you to specify an AZ. If you just say 'us-west-2', for example, you get an error:

```
raven 2017-04-04 12:53:41,028 MainThread INFO toil.utils.toilLaunchCluster: Using aws provisioner.
Traceback (most recent call last):
  File "/home/joel/cactus/venv/bin/toil", line 11, in <module>
    load_entry_point('toil==3.7.0a1', 'console_scripts', 'toil')()
  File "/home/joel/cactus/venv/local/lib/python2.7/site-packages/toil/utils/toilMain.py", line 33, in main
    module.main()
  File "/home/joel/cactus/venv/local/lib/python2.7/site-packages/toil/utils/toilLaunchCluster.py", line 77, in main
    vpcSubnet=config.vpcSubnet)
  File "/home/joel/cactus/venv/local/lib/python2.7/site-packages/toil/provisioners/aws/awsProvisioner.py", line 385, in launchCluster
    ctx = cls._buildContext(clusterName=clusterName, zone=zone)
  File "/home/joel/cactus/venv/local/lib/python2.7/site-packages/toil/provisioners/aws/awsProvisioner.py", line 135, in _buildContext
    return Context(availability_zone=zone, namespace=cls._toNameSpace(clusterName))
  File "/home/joel/cactus/venv/local/lib/python2.7/site-packages/cgcloud/lib/context.py", line 135, in __init__
    % availability_zone )
ValueError: Can't extract region from availability zone 'us-west-2'
```

But if you do provide an AZ, the provisioner ignores it. For example, I specified us-west-2b when launching a cluster (I have an EBS volume there I want to attach) and got an instance in us-west-2c.


Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -386,7 +386,8 @@ def launchCluster(cls, instanceType, keyName, clusterName, spotBid=None, userTag
         kwargs = {'key_name': keyName, 'security_group_ids': [sg.id for sg in sgs],
                   'instance_type': instanceType,
                   'user_data': userData, 'block_device_map': bdm,
-                  'instance_profile_arn': profileARN}
+                  'instance_profile_arn': profileARN,
+                  'placement': zone}
         if vpcSubnet:
             kwargs["subnet_id"] = vpcSubnet
         if not spotBid:
@@ -541,7 +542,8 @@ def _addNodes(self, instances, numNodes, preemptable=False):
                   'instance_type': self.instanceType.name,
                   'user_data': userData,
                   'block_device_map': bdm,
-                  'instance_profile_arn': arn}
+                  'instance_profile_arn': arn,
+                  'placement': getCurrentAWSZone()}
         kwargs["subnet_id"] = self._getClusterInstance(self.instanceMetaData).subnet_id
 
         instancesLaunched = []


Test patch:
Repository: DataBiosphere/toil

Issue:
no TOIL_LSF_ARGS
Needed to set queue and other LSF options


Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/lsf.py b/src/toil/batchSystems/lsf.py
--- a/src/toil/batchSystems/lsf.py
+++ b/src/toil/batchSystems/lsf.py
@@ -38,6 +38,9 @@ def prepareBsub(cpu, mem):
     mem = '' if mem is None else '-R "select[type==X86_64 && mem > ' + str(int(mem/ 1000000)) + '] rusage[mem=' + str(int(mem/ 1000000)) + ']" -M' + str(int(mem/ 1000000)) + '000'
     cpu = '' if cpu is None else '-n ' + str(int(cpu))
     bsubline = ["bsub", mem, cpu,"-cwd", ".", "-o", "/dev/null", "-e", "/dev/null"]
+    lsfArgs = os.getenv('TOIL_LSF_ARGS')
+    if lsfArgs:
+        bsubline.extend(lsfArgs.split())
     return bsubline
 
 def bsub(bsubline):


Test patch:
Repository: DataBiosphere/toil

Issue:
Improve autoscaling documentation
The autoscaling documentation would benefit from a list of steps for getting started. Also many terms such as "Toil Appliance" could be better defined and use links to other parts of documentation.


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -186,7 +186,7 @@ def setup(app):
 # The name of an image file (within the static path) to use as favicon of the
 # docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
 # pixels large.
-html_favicon = "_static/shortcut.png"
+html_favicon = "_static/favicon.ico"
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
diff --git a/src/toil/utils/toilLaunchCluster.py b/src/toil/utils/toilLaunchCluster.py
--- a/src/toil/utils/toilLaunchCluster.py
+++ b/src/toil/utils/toilLaunchCluster.py
@@ -40,14 +40,14 @@ def main():
     parser.add_argument("--keyPairName", dest='keyPairName', required=True,
                         help="The name of the AWS key pair to include on the instance")
     parser.add_argument("-t", "--tag", metavar='NAME=VALUE', dest='tags', default=[], action='append',
-                        help="Tags are added to the AWS cluster for this node and all of its"
-                             "children. Tags are of the form: "
-                             " -t key1=value1 --tag key2=value2 "
+                        help="Tags are added to the AWS cluster for this node and all of its "
+                             "children. Tags are of the form:\n"
+                             " --t key1=value1 --tag key2=value2\n"
                              "Multiple tags are allowed and each tag needs its own flag. By "
                              "default the cluster is tagged with "
-                             " {"
-                             "      \"Name\": clusterName,"
-                             "      \"Owner\": IAM username"
+                             " {\n"
+                             "      \"Name\": clusterName,\n"
+                             "      \"Owner\": IAM username\n"
                              " }. ")
     parser.add_argument("--vpcSubnet",
                         help="VPC subnet ID to launch cluster in. Uses default subnet if not specified."


Test patch:
Repository: DataBiosphere/toil

Issue:
toil clean can take a while on AWS
For large job stores, `toil clean` can take tens of minutes to hours, at least on AWS. I think that may mean it's deleting every file individually. Maybe in these situations the job store could just drop the bucket / domain?
toil clean can take a while on AWS
For large job stores, `toil clean` can take tens of minutes to hours, at least on AWS. I think that may mean it's deleting every file individually. Maybe in these situations the job store could just drop the bucket / domain?


Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -778,7 +778,10 @@ def _bindDomain(self, domain_name, create=False, block=True):
                 retry timeout expires.
         """
         log.debug("Binding to job store domain '%s'.", domain_name)
-        for attempt in retry_sdb(predicate=lambda e: no_such_sdb_domain(e) or sdb_unavailable(e)):
+        retryargs = dict(predicate=lambda e: no_such_sdb_domain(e) or sdb_unavailable(e))
+        if not block:
+            retryargs['timeout'] = 15
+        for attempt in retry_sdb(**retryargs):
             with attempt:
                 try:
                     return self.db.get_domain(domain_name)


Test patch: