internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataBiosphere/toil

Issue:
mesosMasterAddress should default to leader's private IP, not localhost
When starting a workflow on the leader node of a cluster for autoscaling one needs to specify the option `mesosMasterAddress=<my_local_IP_address>:5050` because the default (localhost:5050) is not useful at all and causes the workflow to fail more or less ambiguously. 

To get the local IP address without any external dependencies the following code hunk from [here](http://stackoverflow.com/a/24196955) seems to be the most robust and straight forward.

```
import socket
import fcntl
import struct

def get_ip_address(ifname):
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    return socket.inet_ntoa(fcntl.ioctl(
        s.fileno(),
        0x8915,  # SIOCGIFADDR
        struct.pack('256s', ifname[:15])
    )[20:24])

get_ip_address('eth0')
```


Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -20,6 +20,7 @@
 import sys
 import tempfile
 import time
+import socket
 from argparse import ArgumentParser
 from threading import Thread
 
@@ -34,6 +35,7 @@
 from toil.lib.bioio import addLoggingOptions, getLogLevelString, setLoggingFromOptions
 from toil.realtimeLogger import RealtimeLogger
 
+
 logger = logging.getLogger(__name__)
 
 # This constant is set to the default value used on unix for block size (in bytes) when
@@ -71,7 +73,8 @@ def __init__(self):
         self.batchSystem = "singleMachine"
         self.disableHotDeployment = False
         self.scale = 1
-        self.mesosMasterAddress = 'localhost:5050'
+        # may return localhost on some systems (not osx and coreos) https://stackoverflow.com/a/166520
+        self.mesosMasterAddress = '%s:5050' % 'localhost' # socket.gethostbyname(socket.gethostname())
         self.parasolCommand = "parasol"
         self.parasolMaxBatches = 10000
         self.environment = {}
@@ -1114,3 +1117,5 @@ def getFileSystemSize(dirPath):
     freeSpace = diskStats.f_frsize * diskStats.f_bavail
     diskSize = diskStats.f_frsize * diskStats.f_blocks
     return freeSpace, diskSize
+
+


Test patch:
Repository: DataBiosphere/toil

Issue:
Service jobs are hardcoded to be nonpreemptable
Thanks to @fnothaft for finding this


Discussion:

Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -1100,7 +1100,8 @@ def processService(serviceJob, depth):
             # Create the service job tuple
             j = ServiceJobNode(jobStoreID=serviceJobGraph.jobStoreID,
                                memory=serviceJobGraph.memory, cores=serviceJobGraph.cores,
-                               disk=serviceJobGraph.disk, startJobStoreID=serviceJobGraph.startJobStoreID,
+                               disk=serviceJobGraph.disk, preemptable=serviceJobGraph.preemptable,
+                               startJobStoreID=serviceJobGraph.startJobStoreID,
                                terminateJobStoreID=serviceJobGraph.terminateJobStoreID,
                                errorJobStoreID=serviceJobGraph.errorJobStoreID,
                                jobName=serviceJobGraph.jobName, unitName=serviceJobGraph.unitName,
@@ -1509,9 +1510,9 @@ def getUserScript(self):
 
 
 class ServiceJobNode(JobNode):
-    def __init__(self, jobStoreID, memory, cores, disk, startJobStoreID, terminateJobStoreID,
+    def __init__(self, jobStoreID, memory, cores, disk, preemptable, startJobStoreID, terminateJobStoreID,
                  errorJobStoreID, unitName, jobName, command, predecessorNumber):
-        requirements = dict(memory=memory, cores=cores, disk=disk, preemptable=False)
+        requirements = dict(memory=memory, cores=cores, disk=disk, preemptable=preemptable)
         super(ServiceJobNode, self).__init__(unitName=unitName, jobName=jobName,
                                              requirements=requirements,
                                              jobStoreID=jobStoreID,


Test patch:
Repository: DataBiosphere/toil

Issue:
Avoid copying on import to job store when possible
When using a file job store most of the time we can hard link files into the job store rather than create entire new copies


Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/fileJobStore.py b/src/toil/jobStores/fileJobStore.py
--- a/src/toil/jobStores/fileJobStore.py
+++ b/src/toil/jobStores/fileJobStore.py
@@ -165,15 +165,21 @@ def jobs(self):
     def _importFile(self, otherCls, url, sharedFileName=None):
         if issubclass(otherCls, FileJobStore):
             if sharedFileName is None:
-                fd, absPath = self._getTempFile()
-                shutil.copyfile(self._extractPathFromUrl(url), absPath)
+                fd, absPath = self._getTempFile()  # use this to get a valid path to write to in job store
                 os.close(fd)
+                os.unlink(absPath)
+                try:
+                    os.link(self._extractPathFromUrl(url), absPath)
+                except OSError:
+                    shutil.copyfile(self._extractPathFromUrl(url), absPath)
                 return FileID(self._getRelativePath(absPath), os.stat(absPath).st_size)
             else:
                 self._requireValidSharedFileName(sharedFileName)
-                with self.writeSharedFileStream(sharedFileName) as writable:
-                    with open(self._extractPathFromUrl(url), 'r') as readable:
-                        shutil.copyfileobj(readable, writable)
+                path = self._getSharedFilePath(sharedFileName)
+                try:
+                    os.link(self._extractPathFromUrl(url), path)
+                except:
+                    shutil.copyfile(self._extractPathFromUrl(url), path)
                 return None
         else:
             return super(FileJobStore, self)._importFile(otherCls, url,
@@ -290,11 +296,14 @@ def readFileStream(self, jobStoreFileID):
     # with specific jobs.
     ##########################################  
 
+    def _getSharedFilePath(self, sharedFileName):
+        return os.path.join(self.jobStoreDir, sharedFileName)
+
     @contextmanager
     def writeSharedFileStream(self, sharedFileName, isProtected=None):
         # the isProtected parameter has no effect on the fileStore
         assert self._validateSharedFileName( sharedFileName )
-        with open( os.path.join( self.jobStoreDir, sharedFileName ), 'w' ) as f:
+        with open(self._getSharedFilePath(sharedFileName), 'w') as f:
             yield f
 
     @contextmanager


Test patch:
Repository: DataBiosphere/toil

Issue:
"Pip show" doesn't contain information about Toil's license



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -29,6 +29,8 @@ def runSetup():
         author='Benedict Paten',
         author_email='benedict@soe.usc.edu',
         url="https://github.com/BD2KGenomics/toil",
+        classifiers=["License :: OSI Approved :: Apache Software License"],
+        license="Apache License v2.0",
         install_requires=[
             'bd2k-python-lib>=1.14a1.dev35',
             'dill==0.2.5',


Test patch:
Repository: DataBiosphere/toil

Issue:
cwltoil.py fails in resolve_indirect_inner for Workflow if substep is a workflow and some inputs are optional and empty
I have a Workflow job that uses workflow for the substep. If some inputs for the substep are optional and empty, it fails resolve_indirect_inner() in cwltoil.py:

    def resolve_indirect_inner(d):
        if isinstance(d, IndirectDict):
            r = {}
            for k, v in d.items():
                if isinstance(v, MergeInputs):
                    r[k] = v.resolve()
                else:
                    r[k] = v[1][v[0]]
            return r
        else:
            return d

The problem is at Line 103:

             r[k] = v[1][v[0]]



Discussion:

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -93,14 +93,14 @@ def do_eval(self, inputs, ctx):
         return cwltool.expression.do_eval(self.expr, inputs, self.req,
                                           None, None, {}, context=ctx)
 
-def resolve_indirect_inner(d):
+def _resolve_indirect_inner(d):
     if isinstance(d, IndirectDict):
         r = {}
         for k, v in d.items():
             if isinstance(v, MergeInputs):
                 r[k] = v.resolve()
             else:
-                r[k] = v[1][v[0]]
+                r[k] = v[1].get(v[0])
         return r
     else:
         return d
@@ -114,7 +114,7 @@ def resolve_indirect(d):
             needEval = True
         else:
             inner[k] = v
-    res = resolve_indirect_inner(inner)
+    res = _resolve_indirect_inner(inner)
     if needEval:
         ev = {}
         for k, v in iteritems(d):


Test patch:
Repository: DataBiosphere/toil

Issue:
Add job name to command string
So that it will be easier to see which processes are associated with which jobs.
Add job name to command string
So that it will be easier to see which processes are associated with which jobs.


Discussion:

Resolving patch:
diff --git a/src/toil/leader.py b/src/toil/leader.py
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -47,8 +47,7 @@
 from toil.provisioners.clusterScaler import ClusterScaler
 from toil.serviceManager import ServiceManager
 from toil.statsAndLogging import StatsAndLogging
-from toil.jobGraph import JobNode
-from toil.job import ServiceJobNode
+from toil.job import JobNode, ServiceJobNode
 from toil.toilState import ToilState
 
 logger = logging.getLogger( __name__ )
@@ -533,7 +532,7 @@ def issueJob(self, jobNode):
         Add a job to the queue of jobs
         """
         jobNode.command = ' '.join((resolveEntryPoint('_toil_worker'),
-                                    self.jobStoreLocator, jobNode.jobStoreID))
+                                    jobNode.jobName, self.jobStoreLocator, jobNode.jobStoreID))
         jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
         self.jobBatchSystemIDToIssuedJob[jobBatchSystemID] = jobNode
         if jobNode.preemptable:
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -98,11 +98,9 @@ def main():
     #Input args
     ##########################################
     
-    jobStoreLocator = sys.argv[1]
-    jobStoreID = sys.argv[2]
-    # we really want a list of job names but the ID will suffice if the job graph can't
-    # be loaded. If we can discover the name, we will replace this initial entry
-    listOfJobs = [jobStoreID]
+    listOfJobs = [sys.argv[1]]
+    jobStoreLocator = sys.argv[2]
+    jobStoreID = sys.argv[3]
     
     ##########################################
     #Load the jobStore/config file


Test patch:
Repository: DataBiosphere/toil

Issue:
make docker is failing as of today
Looks like a old setuptools version is having trouble parsing complex requirements in the newest schema-salad version.

```
[snip]
Collecting cgcloud-lib==1.6.0a1.dev393 (from toil==3.12.0a1)
  Downloading cgcloud-lib-1.6.0a1.dev393.tar.gz (44kB)
Collecting futures==3.0.5 (from toil==3.12.0a1)
  Downloading futures-3.0.5-py2-none-any.whl
Collecting cwltool==1.0.20170822192924 (from toil==3.12.0a1)
  Downloading cwltool-1.0.20170822192924.tar.gz (277kB)
Collecting schema-salad<3,>=2.6 (from toil==3.12.0a1)
  Downloading schema-salad-2.6.20170927145003.tar.gz (311kB)
    Complete output from command python setup.py egg_info:
    error in schema-salad setup command: Invalid environment marker: python_version>="3"
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-8j0tUs/schema-salad/
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c pip install toil-3.12.0a1.tar.gz[aws,mesos,encryption,cwl]' returned a non-zero code: 1
Makefile:198: recipe for target 'docker' failed
make: *** [docker] Error 1
```


Discussion:

Resolving patch:
diff --git a/docker/Dockerfile.py b/docker/Dockerfile.py
--- a/docker/Dockerfile.py
+++ b/docker/Dockerfile.py
@@ -77,7 +77,10 @@ def heredoc(s):
     RUN chmod 777 /usr/bin/waitForKey.sh
 
     # The stock pip is too old and can't install from sdist with extras
-    RUN pip install --upgrade pip==8.1.2
+    RUN pip install --upgrade pip==9.0.1
+
+    # Default setuptools is too old
+    RUN pip install --upgrade setuptools==36.5.0
 
     # Include virtualenv, as it is still the recommended way to deploy pipelines
     RUN pip install --upgrade virtualenv==15.0.3


Test patch:
Repository: DataBiosphere/toil

Issue:
static provisioning blocked



Discussion:

Resolving patch:
diff --git a/src/toil/leader.py b/src/toil/leader.py
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -145,7 +145,9 @@ def __init__(self, config, batchSystem, provisioner, jobStore, rootJob, jobCache
         self.provisioner = provisioner
  
         # Create cluster scaling thread if the provisioner is not None
-        self.clusterScaler = None if self.provisioner is None else ClusterScaler(self.provisioner, self, self.config)
+        self.clusterScaler = None
+        if self.provisioner is not None and len(self.provisioner.nodeTypes) > 0:
+            self.clusterScaler = ClusterScaler(self.provisioner, self, self.config)
 
         # A service manager thread to start and terminate services
         self.serviceManager = ServiceManager(jobStore, self.toilState)


Test patch:
Repository: DataBiosphere/toil

Issue:
Refactor ssh and rsync awsProvisioner code into AbstractProvisioner (fixes #1971)



Discussion:
Build finished. 8 tests run, 0 skipped, 2 failed.

Build finished. 8 tests run, 0 skipped, 1 failed.

Build finished. 717 tests run, 23 skipped, 0 failed.

[I don't think this failure is related](http://jenkins.cgcloud.info/job/toil-pull-request-integration-tests/1750/). This shouldn't be so messy, but I'm having problems running the integration tests on my machine... 

Jenkins, test this please
Build finished. 717 tests run, 23 skipped, 0 failed.

Build finished. 8 tests run, 0 skipped, 0 failed.

Build finished. 717 tests run, 23 skipped, 0 failed.


Resolving patch:
diff --git a/src/toil/provisioners/abstractProvisioner.py b/src/toil/provisioners/abstractProvisioner.py
--- a/src/toil/provisioners/abstractProvisioner.py
+++ b/src/toil/provisioners/abstractProvisioner.py
@@ -23,7 +23,7 @@
 from future.utils import with_metaclass
 
 from bd2k.util.retry import never
-from toil.lib.ec2 import a_short_time
+a_short_time = 5
 
 log = logging.getLogger(__name__)
 


Test patch:
Repository: DataBiosphere/toil

Issue:
Toil not putting enough jobs on i3.8xlarge nodes
I'm seeing fewer jobs than I expect on each of my i3.8xlarge nodes.  From the log, my nodes look like 

`Running bin packing for node shapes [_Shape(wallTime=3600, memory=261885630873.6, cores=32, disk=8160437862400, preemptable=True)] and 357 job(s).`

And my jobs:

`Issued job 'run_construct_region_graph' ee16af3b85fa-41b3-a876-8146e640f3d9 with job batch system ID: 96 and cores: 1, disk: 32.0 G, and memory: 32.0 G`

From @joelarmstrong 
>yeah, it looks like the i3 series drives are at a different location than usual, because they're NVMe and not SSD they are at /dev/nvme0n* rather than /dev/sd* so toil doesn't mount them properly right now

So it seems there's an issue mounting these nodes, though the log above does show Toil reading 8TB for it. 


Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/aws/__init__.py b/src/toil/provisioners/aws/__init__.py
--- a/src/toil/provisioners/aws/__init__.py
+++ b/src/toil/provisioners/aws/__init__.py
@@ -213,7 +213,7 @@ def _get_spot_history(ctx, instance_type):
         ephemeral_count=0
         drives=""
         directories="toil mesos docker"
-        for drive in /dev/xvd{{b..z}}; do
+        for drive in /dev/xvd{{b..z}} /dev/nvme*n*; do
             echo checking for $drive
             if [ -b $drive ]; then
                 echo found it
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -383,7 +383,7 @@ def _buildContext(cls, clusterName, zone=None):
     @memoize
     def _discoverAMI(cls, ctx):
         def descriptionMatches(ami):
-            return ami.description is not None and 'stable 1235.4.0' in ami.description
+            return ami.description is not None and 'stable 1632.2.1' in ami.description
         coreOSAMI = os.environ.get('TOIL_AWS_AMI')
         if coreOSAMI is not None:
             return coreOSAMI


Test patch:
Repository: DataBiosphere/toil

Issue:
Add citation.  (#2010)
This just adds an explicit reference to the toil citation to the beginning of the toil docs in addition to the paper link.  Hopefully this induces more people to cite toil:

Vivian, J., Rao, A. A., Nothaft, F. A., Ketchum, C., Armstrong, J., Novak, A., … Paten, B. (2017). Toil enables reproducible, open source, big biomedical data analyses. Nature Biotechnology, 35(4), 314–316. http://doi.org/10.1038/nbt.3772
  


Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/torque.py b/src/toil/batchSystems/torque.py
--- a/src/toil/batchSystems/torque.py
+++ b/src/toil/batchSystems/torque.py
@@ -184,6 +184,7 @@ def prepareQsub(self, cpu, mem, jobID):
 
             if cpu is not None and math.ceil(cpu) > 1:
                 reqline.append('ncpus=' + str(int(math.ceil(cpu))))
+                reqline.append('nodes=1:ppn=' + str(int(math.ceil(cpu))))
 
             # Other resource requirements can be passed through the environment (see man qsub)
             reqlineEnv = os.getenv('TOIL_TORQUE_REQS')


Test patch:
Repository: DataBiosphere/toil

Issue:
Expose the cwl option "--no-match-user" in toil-cwl-runner.


┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-307)
┆Epic: TOIL: Integrate WES 
┆Issue Number: TOIL-307



Discussion:
➤ Lon commented:

Other options would be cool, but the “\-\-no-match-user” option is the only one that is important for now.


Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -477,8 +477,7 @@ def run(self, fileStore):
             'tmp_outdir_prefix': tmp_outdir_prefix,
             'tmpdir_prefix': fileStore.getLocalTempDir(),
             'make_fs_access': functools.partial(ToilFsAccess, fileStore=fileStore),
-            'toil_get_file': functools.partial(toilGetFile, fileStore, index, existing),
-            'no_match_user': False})
+            'toil_get_file': functools.partial(toilGetFile, fileStore, index, existing)})
         del opts['job_order']
 
         # Run the tool
@@ -948,6 +947,8 @@ def main(args=None, stdout=sys.stdout):
                         default="tmp")
     parser.add_argument("--force-docker-pull", action="store_true", default=False, dest="force_docker_pull",
                         help="Pull latest docker image even if it is locally present")
+    parser.add_argument("--no-match-user", action="store_true", default=False,
+                        help="Disable passing the current uid to `docker run --user`")
 
     # mkdtemp actually creates the directory, but
     # toil requires that the directory not exist,
@@ -1058,7 +1059,8 @@ def setSecondary(fileobj):
                                   'use_container': use_container,
                                   'tmpdir': os.path.realpath(outdir),
                                   'job_script_provider': job_script_provider,
-                                  'force_docker_pull': options.force_docker_pull})
+                                  'force_docker_pull': options.force_docker_pull,
+                                  'no_match_user': options.no_match_user})
 
                 (wf1, wf2) = makeJob(**make_opts)
             except cwltool.process.UnsupportedRequirement as e:


Test patch:
Repository: DataBiosphere/toil

Issue:
Set LSF log level to error when jobs fail
Some time ago I submitted a #2062 to make sure we log the LSF jobID to `error` if the pipeline fails, but PR wasn't merged and seems we didn't do it. 

The following lines should log to `error` instead of `debug`:

https://github.com/DataBiosphere/toil/blob/master/src/toil/batchSystems/lsf.py#L113
https://github.com/DataBiosphere/toil/blob/master/src/toil/batchSystems/lsf.py#L117

I'm submitting #2359, I think these changes are very useful and inoffensive. Seeking feedback from @mr-c.

┆Issue is synchronized with this [JIRA Story](https://ucsc-cgl.atlassian.net/browse/TOIL-341)
┆Issue Number: TOIL-341



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/lsf.py b/src/toil/batchSystems/lsf.py
--- a/src/toil/batchSystems/lsf.py
+++ b/src/toil/batchSystems/lsf.py
@@ -110,11 +110,11 @@ def getJobExitCode(self, lsfJobID):
                 elif "Exited with exit code" in line:
                     exit = int(line[line.find("Exited with exit code ")+22:]
                                .split('.')[0])
-                    logger.debug("bjobs detected job exit code: "
+                    logger.error("bjobs detected job exit code: "
                                  "{}".format(exit))
                     return exit
                 elif "Completed <exit>" in line:
-                    logger.debug("bjobs detected job failed for job: "
+                    logger.error("bjobs detected job failed for job: "
                                  "{}".format(job))
                     return 1
                 elif line.find("Started on ") > -1:


Test patch:
Repository: DataBiosphere/toil

Issue:
Add recognition of Conda virtual environments
Hi,
we used Conda virtual environments instead of virtualenv and ran into the issue that toil does not recognize that it is running inside a Conda virtual environment. _toil_worker could thus not be found, without explicitely setting the PATH variable to point to its location in .bash_profile.

Extending the check for virtual environments inVirtualEnv() in src/toil/__init__.py to
    return hasattr(sys, 'real_prefix') or 'CONDA_DEFAULT_ENV' in os.environ
solved the problem for us. Would this change make sense, and should I submit a pull request?

Thanks, Ilja

┆Issue is synchronized with this [JIRA Story](https://ucsc-cgl.atlassian.net/browse/TOIL-377)
┆Issue Number: TOIL-377



Discussion:
@ibebio Yes, that sounds like a sensible change.  Please do submit a PR!  Thanks for posting this.  :)

Resolving patch:
diff --git a/src/toil/__init__.py b/src/toil/__init__.py
--- a/src/toil/__init__.py
+++ b/src/toil/__init__.py
@@ -48,7 +48,10 @@ def toilPackageDirPath():
 
 
 def inVirtualEnv():
-    return hasattr(sys, 'real_prefix')
+    """
+    Returns whether we are inside a virtualenv or Conda virtual environment.
+    """
+    return hasattr(sys, 'real_prefix') or 'CONDA_DEFAULT_ENV' in os.environ
 
 
 def resolveEntryPoint(entryPoint):


Test patch: