internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataBiosphere/toil

Issue:
Global name 'config' not defined for getting azure job store
`global name 'config' is not defined` error found in `toil/common.py` on line 617 in `getJobStore` when running toil version 3.3.1 on an Azure cluster. Error does not show up for toil version 3.2.1 and the toil script runs to completion. File `.toilAzureCredentials` is set properly.

Toil script command:
`./vg_evaluation_pipeline.py --batchSystem mesos --mesosMaster 10.0.0.5:5050 --realTimeLogging --logError --logDebug --edge_max 5 --kmer_size 16 --index_mode gcsa-mem --include_primary 'azure:hgvm:hgvmeval-jobstore33' --path_name 'ref' --path_size 81189 '/home/cmarkello/debug_eval_input/BRCA1.vg' '/home/cmarkello/debug_eval_input/BRCA1/NA12877/NA12877.bam.fq' 'NA12877' '/home/cmarkello/debug_eval_output' 'azure:hgvm:hgvmdebugtest-input' 'azure:hgvm:hgvmdebugtest-output'`

Toil script `main(args)` function:
`

```
if len(args) == 2 and args[1] == "--test":
    # Run the tests
    return doctest.testmod(optionflags=doctest.NORMALIZE_WHITESPACE)

RealTimeLogger.start_master()
options = parse_args(args) # This holds the nicely-parsed options object

with Toil(options) as toil:
    if not toil.options.restart:

        # Upload local files to the remote IO Store
        inputGraphFileID = toil.importFile('file://'+options.vg_graph)
        inputReadsFileID = toil.importFile('file://'+options.sample_reads)
        uploadList = [[inputGraphFileID, options.vg_graph], [inputReadsFileID, options.sample_reads]]

        # Make a root job
        root_job = Job.wrapJobFn(run_upload, options, uploadList, cores=2, memory="5G", disk="5G")

        # Run the job and store the returned list of output files to download
        outputFileIDList = toil.start(root_job)
    else:
        outputFileIDList = toil.restart()

    # Download output files to the local machine that runs this script
    run_download(toil, options, outputFileIDList)

print("All jobs completed successfully")

RealTimeLogger.stop_master()`
```



Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -633,7 +633,7 @@ def getJobStore(locator):
         elif name == 'azure':
             from toil.jobStores.azureJobStore import AzureJobStore
             account, namePrefix = rest.split(':', 1)
-            return AzureJobStore(account, namePrefix, config=config)
+            return AzureJobStore(rest)
 
         elif name == 'google':
             from toil.jobStores.googleJobStore import GoogleJobStore


Test patch:
Repository: DataBiosphere/toil

Issue:
Handle relative paths for --workDir



Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -158,6 +158,11 @@ def parseJobStore(s):
         setOption("jobStore", parsingFn=parseJobStore)
         #TODO: LOG LEVEL STRING
         setOption("workDir")
+        if self.workDir is not None:
+            self.workDir = os.path.abspath(self.workDir)
+            if not os.path.exists(self.workDir):
+                raise RuntimeError("The path provided to --workDir (%s) does not exist."
+                                   % self.workDir)
         setOption("stats")
         setOption("cleanWorkDir")
         setOption("clean")


Test patch:
Repository: DataBiosphere/toil

Issue:
Incorporate fix for tests leaking IAM roles and instance profiles
https://github.com/BD2KGenomics/cgcloud/issues/219
http://jenkins.cgcloud.info/job/cgcloud/372/



Discussion:

Resolving patch:
diff --git a/src/toil/version.py b/src/toil/version.py
--- a/src/toil/version.py
+++ b/src/toil/version.py
@@ -14,7 +14,7 @@
 
 version = '3.5.0a1'
 
-cgcloudVersion = '1.6.0a1.dev370'
+cgcloudVersion = '1.6.0a1.dev378'
 
 if __name__ == '__main__':
     print version


Test patch:
Repository: DataBiosphere/toil

Issue:
Implement FileIDs in NonCachingFileStore



Discussion:
If you need me to look at something, let me know!


Resolving patch:
diff --git a/src/toil/fileStore.py b/src/toil/fileStore.py
--- a/src/toil/fileStore.py
+++ b/src/toil/fileStore.py
@@ -1449,7 +1449,7 @@ def open(self, job):
     def writeGlobalFile(self, localFileName, cleanup=False):
         absLocalFileName = self._abspath(localFileName)
         cleanupID = None if not cleanup else self.jobWrapper.jobStoreID
-        return self.jobStore.writeFile(absLocalFileName, cleanupID)
+        return FileID.forPath(self.jobStore.writeFile(absLocalFileName, cleanupID), absLocalFileName)
 
     def readGlobalFile(self, fileStoreID, userPath=None, cache=True, mutable=None):
         if userPath is not None:


Test patch:
diff --git a/src/toil/test/src/promisedRequirementTest.py b/src/toil/test/src/promisedRequirementTest.py
--- a/src/toil/test/src/promisedRequirementTest.py
+++ b/src/toil/test/src/promisedRequirementTest.py
@@ -74,7 +74,7 @@ def testConcurrencyStatic(self):
                 _, maxValue = batchSystemTest.getCounters(counterPath)
                 self.assertEqual(maxValue, self.cpuCount / coresPerJob)
 
-        def getOptions(self, tempDir):
+        def getOptions(self, tempDir, caching=True):
             """
             Configures options for Toil workflow and makes job store.
             :param str tempDir: path to test directory
@@ -88,6 +88,8 @@ def getOptions(self, tempDir):
             # defaultCores defaults to 1 - this is coincidentally the core requirement relied upon by this
             # test, so we change defaultCores to 2 to make the test more strict
             options.defaultCores = 2
+            if not caching:
+                options.disableCaching = True
             return options
 
         def getCounterPath(self, tempDir):
@@ -106,7 +108,7 @@ def getCounterPath(self, tempDir):
         def testJobConcurrency(self):
             pass
 
-        def testPromisesWithJobStoreFileObjects(self):
+        def testPromisesWithJobStoreFileObjects(self, caching=True):
             """
             Check whether FileID objects are being pickled properly when used as return
             values of functions.  Then ensure that lambdas of promised FileID objects can be
@@ -124,8 +126,10 @@ def testPromisesWithJobStoreFileObjects(self):
             F1.addChild(F2)
             F2.addChild(G)
 
-            Job.Runner.startToil(F1, self.getOptions(self._createTempDir('testFiles')))
+            Job.Runner.startToil(F1, self.getOptions(self._createTempDir('testFiles'), caching=caching))
 
+        def testPromisesWithNonCachingFileStore(self):
+            self.testPromisesWithJobStoreFileObjects(caching=False)
 
         def testPromiseRequirementRaceStatic(self):
             """
Repository: DataBiosphere/toil

Issue:
Import files from FTP
It looks like there's a [fake JobStore that reads from http and https for file import](https://github.com/BD2KGenomics/toil/blob/master/src/toil/jobStores/abstractJobStore.py#L963) using urllib2. Since urllib2 can also do FTP, it would be easy to just add "ftp" to the list of protocols that this JobStore supports, to enable importing files from FTP.

Since many bioinformatics data sets live on FTP (like 1000 Genomes and the GRC reference assembly), it would be nice to just be able to point Toil at those URLs and have those resources available.



Discussion:
+1

Pinging @JakeNarkizian, who's baby this is.

I've reached the point where I actually want this to work now. I guess I will try to do it.


Resolving patch:
diff --git a/src/toil/jobStores/abstractJobStore.py b/src/toil/jobStores/abstractJobStore.py
--- a/src/toil/jobStores/abstractJobStore.py
+++ b/src/toil/jobStores/abstractJobStore.py
@@ -960,7 +960,7 @@ class JobStoreSupport(AbstractJobStore):
 
     @classmethod
     def _supportsUrl(cls, url, export=False):
-        return url.scheme.lower() in ('http', 'https') and not export
+        return url.scheme.lower() in ('http', 'https', 'ftp') and not export
 
     @classmethod
     def _readFromUrl(cls, url, writable):


Test patch:
diff --git a/src/toil/test/jobStores/jobStoreTest.py b/src/toil/test/jobStores/jobStoreTest.py
--- a/src/toil/test/jobStores/jobStoreTest.py
+++ b/src/toil/test/jobStores/jobStoreTest.py
@@ -460,6 +460,12 @@ def testImportHttpFile(self):
             with self.master.readFileStream(self.master.importFile(srcUrl)) as f:
                 self.assertEqual(hashlib.md5(f.read()).hexdigest(), srcHash)
 
+        def testImportFtpFile(self):
+            srcUrl, srcHash = ('ftp://speedtest.tele2.net/512KB.zip',
+                               '59071590099d21dd439896592338bf95')
+            with self.master.readFileStream(self.master.importFile(srcUrl)) as f:
+                self.assertEqual(hashlib.md5(f.read()).hexdigest(), srcHash)
+
         def testFileDeletion(self):
             """
             Intended to cover the batch deletion of items in the AWSJobStore, but it doesn't hurt
Repository: DataBiosphere/toil

Issue:
Missing -t flag for AWS provisioner ssh-cluster util
Without the flag the util is functional but very ugly



Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -19,6 +19,7 @@
 import time
 from contextlib import contextmanager
 
+import sys
 from boto.ec2.blockdevicemapping import BlockDeviceMapping, BlockDeviceType
 from boto.exception import BotoServerError, EC2ResponseError
 from cgcloud.lib.ec2 import (ec2_instance_types, retry_ec2, wait_spot_requests_active, a_short_time,
@@ -82,16 +83,18 @@ def getNodeShape(self, preemptable=False):
     def sshLeader(cls, clusterName):
         leader = cls._getLeader(clusterName)
         logger.info('SSH ready')
-        cls._sshAppliance(leader.ip_address, 'bash')
+        tty = sys.stdin.isatty()
+        cls._sshAppliance(leader.ip_address, 'bash', tty=tty)
 
     @classmethod
-    def _sshAppliance(cls, leaderIP, command):
-        command = "ssh -o \"StrictHostKeyChecking=no\" -t core@%s \"docker exec -i leader %s\"" % (leaderIP, command)
+    def _sshAppliance(cls, leaderIP, command, tty=False):
+        ttyFlag = 't' if tty else ''
+        command = 'ssh -o "StrictHostKeyChecking=no" -t core@%s "docker exec -i%s leader %s"' % (leaderIP, ttyFlag, command)
         return subprocess.check_call(command, shell=True)
 
     @classmethod
     def _sshInstance(cls, leaderIP, command):
-        command = "ssh -o \"StrictHostKeyChecking=no\" -t core@%s \"%s\"" % (leaderIP, command)
+        command = 'ssh -o "StrictHostKeyChecking=no" -t core@%s "%s"' % (leaderIP, command)
         ouput = subprocess.check_output(command, shell=True)
         return ouput
 


Test patch:
Repository: DataBiosphere/toil

Issue:
Toil appliance must start docker after mounting volumes
@cket and I ran into this bug this afternoon when testing out the autoscaler. On machines with multiple ephemeral volumes, the Toil appliance JBODs the volumes together and mounts them to `/var/lib/docker`, `/var/lib/toil`, etc. If this happens after the Docker daemon launches, docker fails sporadically when writing to `/var/lib/docker`, and the Toil appliance does an infinite failure loop.



Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/aws/__init__.py b/src/toil/provisioners/aws/__init__.py
--- a/src/toil/provisioners/aws/__init__.py
+++ b/src/toil/provisioners/aws/__init__.py
@@ -91,10 +91,11 @@
         [Unit]
         Description=mounts ephemeral volumes & bind mounts toil directories
         Author=cketchum@ucsc.edu
-        After=docker.service
+        Before=docker.service
 
         [Service]
-        Restart=on-failure
+        Type=oneshot
+        Restart=no
         ExecStart=/usr/bin/bash /home/core/volumes.sh
 
     - name: "toil-{role}.service"


Test patch:
Repository: DataBiosphere/toil

Issue:
Add documentation for new cluster related utils



Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -342,15 +342,13 @@ def _addOptionFn(*name, **kwargs):
 
         _addOptionFn('nodeType', metavar='TYPE',
                      help="Node type for {non-|}preemptable nodes. The syntax depends on the "
-                          "provisioner used. For the cgcloud provisioner this is the name of an "
-                          "EC2 instance type{|, followed by a colon and the price in dollar to "
-                          "bid for a spot instance}, for example 'c3.8xlarge{|:0.42}'. The AWS provisioner "
-                          "is the name of the EC2 instance type followed by a colon and the price "
-                          "in dollars, for example: 'm3.medium:0.10'")
+                          "provisioner used. For the cgcloud and AWS provisioners this is the name "
+                          "of an EC2 instance type{|, followed by a colon and the price in dollar "
+                          "to bid for a spot instance}, for example 'c3.8xlarge{|:0.42}'.")
         _addOptionFn('nodeOptions', metavar='OPTIONS',
                      help="Provisioning options for the {non-|}preemptable node type. The syntax "
-                          "depends on the provisioner used. The CGCloud provisioner doesn't "
-                          "currently support any node options.")
+                          "depends on the provisioner used. Neither the CGCloud nor the AWS "
+                          "provisioner support any node options.")
         for p, q in [('min', 'Minimum'), ('max', 'Maximum')]:
             _addOptionFn(p, 'nodes', default=None, metavar='NUM',
                          help=q + " number of {non-|}preemptable nodes in the cluster, if using "


Test patch:
Repository: DataBiosphere/toil

Issue:
Throw user friendly exception when TOIL_APPLIANCE_SELF is not set



Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -40,8 +40,6 @@
 
 class AWSProvisioner(AbstractProvisioner, BaseAWSProvisioner):
 
-    dockerInfo = os.environ.get('TOIL_APPLIANCE_SELF')
-
     def __init__(self, config, batchSystem):
         self.instanceMetaData = get_instance_metadata()
         self.clusterName = self.instanceMetaData['security-groups']
@@ -86,6 +84,18 @@ def sshLeader(cls, clusterName):
         tty = sys.stdin.isatty()
         cls._sshAppliance(leader.ip_address, 'bash', tty=tty)
 
+    @classmethod
+    def dockerInfo(cls):
+        try:
+            return os.environ['TOIL_APPLIANCE_SELF']
+        except KeyError:
+            raise RuntimeError('Please set TOIL_APPLIANCE_SELF environment variable to the '
+                               'image of the Toil Appliance you wish to use. For example: '
+                               "'quay.io/ucsc_cgl/toil:3.5.0a1--80c340c5204bde016440e78e84350e3c13bd1801'. "
+                               'See https://quay.io/repository/ucsc_cgl/toil-leader?tab=tags '
+                               'for a full list of available versions.')
+
+
     @classmethod
     def _sshAppliance(cls, leaderIP, command, tty=False):
         ttyFlag = 't' if tty else ''
@@ -192,7 +202,7 @@ def launchCluster(cls, instanceType, keyName, clusterName, spotBid=None):
         # the security group name is used as the cluster identifier
         cls._createSecurityGroup(ctx, clusterName)
         bdm = cls._getBlockDeviceMapping(ec2_instance_types[instanceType])
-        dockerLeaderData = cls.dockerInfo.rsplit(':', 1)
+        dockerLeaderData = cls.dockerInfo().rsplit(':', 1)
         leaderRepo = dockerLeaderData[0]
         leaderTag = dockerLeaderData[1]
         leaderData = {'role': 'leader', 'tag': leaderTag,
@@ -263,7 +273,7 @@ def _addNodes(self, instancesToLaunch, preemptable=False):
         bdm = self._getBlockDeviceMapping(self.instanceType)
         arn = self._getProfileARN(self.ctx)
         # quay.io/toil-leader:tag
-        workerData = self.dockerInfo.rsplit(':', 1)
+        workerData = self.dockerInfo().rsplit(':', 1)
         workerRepo = workerData[0].rsplit('-', 1)[0] + '-worker'
         workerTag = workerData[1]
         workerData = {'role': 'worker', 'tag': workerTag,


Test patch:
Repository: DataBiosphere/toil

Issue:
Workaround pip issue 4059
Symptom:

http://jenkins.cgcloud.info/job/toil-pull-requests/2029/testReport/src.toil.test.src.resourceTest/ResourceTest/testVirtualEnv/

Upstream ticket:

https://github.com/pypa/pip/issues/4059

We should prevent the latest pip from being installed in the virtualenv.


Discussion:

Resolving patch:
diff --git a/src/toil/leader.py b/src/toil/leader.py
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -31,7 +31,6 @@
 
 from toil import resolveEntryPoint
 from toil.jobStores.abstractJobStore import NoSuchJobException
-from toil.jobStores.aws.jobStore import AWSJobStore
 from toil.lib.bioio import getTotalCpuTime, logStream
 from toil.provisioners.clusterScaler import ClusterScaler
 
@@ -283,7 +282,8 @@ def processRemovedJob(jobStoreID):
             try:
                 jobWrapper = self.jobStore.load(jobStoreID)
             except NoSuchJobException:
-                if isinstance(self.jobStore, AWSJobStore):
+                # Avoid importing AWSJobStore as the corresponding extra might be missing
+                if self.jobStore.__class__.__name__ == 'AWSJobStore':
                     # We have a ghost job - the job has been deleted but a stale read from
                     # SDB gave us a false positive when we checked for its existence.
                     # Process the job from here as any other job removed from the job store.


Test patch:
diff --git a/src/toil/test/src/resourceTest.py b/src/toil/test/src/resourceTest.py
--- a/src/toil/test/src/resourceTest.py
+++ b/src/toil/test/src/resourceTest.py
@@ -67,7 +67,8 @@ def _testExternal(self, moduleName, pyFiles, virtualenv=False):
         dirPath = self._createTempDir()
         if virtualenv:
             self.assertTrue(inVirtualEnv())
-            check_call(['virtualenv', dirPath])
+            # Passing --never-download prevents silent upgrades to pip, wheel and setuptools
+            check_call(['virtualenv', '--never-download', dirPath])
             sitePackages = os.path.join(dirPath, 'lib', 'python2.7', 'site-packages')
             # tuple assignment is necessary to make this line immediately precede the try:
             oldPrefix, sys.prefix, dirPath = sys.prefix, dirPath, sitePackages
Repository: DataBiosphere/toil

Issue:
Tests should be run serially by default

Log config at start
Log every attribute in the config object once the leader and worker processes have fully populated it.  Also log the Toil version.


Discussion:
… and without test reports.

Basically this entails going back to the previous method of running the tests, i.e. running tests with pytest directly and not going through run_tests.py. The current `test` target in the Makefile should be renamed to `test-parallel` and this serial test target should become `test`.

The Config object should be logged at INFO level. Every attribute of the Config object should be should be logged, ideally one entry per line, but not one log entry per line. The current version can be imported from `toil.version`.


Resolving patch:
diff --git a/src/toil/__init__.py b/src/toil/__init__.py
--- a/src/toil/__init__.py
+++ b/src/toil/__init__.py
@@ -17,6 +17,7 @@
 import logging
 import os
 import sys
+from pprint import pformat
 
 from subprocess import check_output
 
@@ -138,3 +139,11 @@ def lookupEnvVar(name, envName, defaultValue):
     else:
         log.info('Overriding %s of %s with %s from %s.', name, defaultValue, value, envName)
         return value
+
+
+def logProcessContext(config, logger):
+    # toil.version.version (string) canont be imported at top level because it conflicts with
+    # toil.version (module) and Sphinx doesn't like that.
+    from toil.version import version
+    logger.info("Running Toil version %s.", version)
+    logger.info("Configuration:\n %s", pformat(config.__dict__))
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -26,6 +26,7 @@
 from bd2k.util.exceptions import require
 from bd2k.util.humanize import bytes2human
 
+from toil import logProcessContext
 from toil.lib.bioio import addLoggingOptions, getLogLevelString, setLoggingFromOptions
 from toil.realtimeLogger import RealtimeLogger
 
@@ -873,6 +874,8 @@ def _runMainLoop(self, rootJob):
         :param toil.job.Job rootJob: The root job for the workflow.
         :rtype: Any
         """
+        logProcessContext(self.config, logger)
+
         with RealtimeLogger(self._batchSystem,
                             level=self.options.logLevel if self.options.realTimeLogging else None):
             # FIXME: common should not import from leader
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -30,6 +30,7 @@
 from bd2k.util.expando import Expando, MagicExpando
 from toil.common import Toil
 from toil.fileStore import FileStore
+from toil import logProcessContext
 import signal
 
 logger = logging.getLogger( __name__ )
@@ -209,6 +210,8 @@ def badWorker():
         logger.debug("Next available file descriptor: {}".format(
             nextOpenDescriptor()))
 
+        logProcessContext(config, logger)
+
         ##########################################
         #Load the jobWrapper
         ##########################################


Test patch:
Repository: DataBiosphere/toil

Issue:
Minor copy-paste error in worker.py
I just stumbled across this:

https://github.com/BD2KGenomics/toil/blob/5ec9a2cd1fb081bd59a93a7aa0023d19cab6750b/src/toil/worker.py#L496

``` python
    #Close redirected stderr and replace with the original standard error.
    os.dup2(origStdOut, 2)
```

That should read `os.dup2(origStdErr, 2)` if you want to actually restore the original stderr to fd 2. I can't think of a way that this would ever cause a problem, though.

Minor copy-paste error in worker.py
I just stumbled across this:

https://github.com/BD2KGenomics/toil/blob/5ec9a2cd1fb081bd59a93a7aa0023d19cab6750b/src/toil/worker.py#L496

``` python
    #Close redirected stderr and replace with the original standard error.
    os.dup2(origStdOut, 2)
```

That should read `os.dup2(origStdErr, 2)` if you want to actually restore the original stderr to fd 2. I can't think of a way that this would ever cause a problem, though.



Discussion:

Resolving patch:
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -493,7 +493,7 @@ def recursiveDelete(jobWrapper2):
     os.dup2(origStdOut, 1)
     
     #Close redirected stderr and replace with the original standard error.
-    os.dup2(origStdOut, 2)
+    os.dup2(origStdErr, 2)
     
     #sys.stdout and sys.stderr don't need to be modified at all. We don't need
     #to call redirectLoggerStreamHandlers since they still log to sys.stderr


Test patch: