internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataBiosphere/toil

Issue:
Update psutil version
`psutil` is pinned to version 3.0.1, which causes dependency conflicts with other packages like `dask`, which requires version >=5.0. All `psutil` calls are in **src/toil/batchSystems/mesos/executor.py** and I confirmed that on the latest version (5.6.3), every `psutil` call works the same:

```
In [1]: import psutil

In [2]: psutil.cpu_percent()
Out[2]: 8.9

In [3]: psutil.cpu_count()
Out[3]: 8

In [4]: psutil.virtual_memory()
Out[4]: svmem(total=17179869184, available=5836066816, percent=66.0, used=9086025728, free=72404992, active=5766565888, inactive=5725958144, wired=3319459840)

In [5]: psutil.__version__
Out[5]: '5.6.3'
```

Happy to file a PR if that's helpful.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-414)
┆Issue Number: TOIL-414



Discussion:
Maybe change it to `psutil >= 3.0.1, <6` ?

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -25,7 +25,7 @@ def runSetup():
     futures = 'futures==3.1.1'
     pycryptodome = 'pycryptodome==3.5.1'
     pymesos = 'pymesos==0.3.7'
-    psutil = 'psutil==3.0.1'
+    psutil = 'psutil >= 3.0.1, <6'
     azureCosmosdbTable = 'azure-cosmosdb-table==0.37.1'
     azureAnsible = 'ansible[azure]==2.5.0a1'
     azureStorage = 'azure-storage==0.35.1'


Test patch:
Repository: DataBiosphere/toil

Issue:
CWL failure when passing an array with default values
It looks like PR #2785 introduced a bug that throws the following exception at runtime for particular inputs:

```
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 346, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1392, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1329, in _run
    return self.run(fileStore)
  File "/usr/local/lib/python2.7/dist-packages/toil/cwl/cwltoil.py", line 952, in run
    jobobj[key][0]] is None:
TypeError: tuple indices must be integers, not unicode
```

CWL input to reproduce:

```
cwlVersion: v1.0
class: Workflow
requirements:
  SubworkflowFeatureRequirement: {}
inputs:
  strings:
    type: string[]
    default: ['hello', 'world']
steps:
  step1:
    run:
      class: Workflow
      inputs:
        strings: string[]
      outputs: []
      steps:
        step_inline:
          in:
            strings: strings
          out: []
          run:
            class: CommandLineTool
            baseCommand: [echo]
            outputs: []
            inputs:
              strings:
                default: ['hello']
                type: string[]
                inputBinding:
                  position: 1
    in:
      strings: strings
    out: []
outputs: []
```

I've verified that the above input works fine when the PR is reverted. It also works fine if `default: ['hello']` is removed from the CWL file.



┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-432)
┆Issue Number: TOIL-432



Discussion:

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -947,8 +947,10 @@ def run(self, file_store):
                                         jobobj[key] = DefaultWithSource(
                                             d, jobobj[key])
                                     else:
-                                        if jobobj[key][1][
-                                                jobobj[key][0]] is None:
+                                        if (isinstance(jobobj[key][1], tuple) and
+                                                jobobj[key][1][0][jobobj[key][0]]
+                                                is None) or (jobobj[key][1][
+                                                    jobobj[key][0]] is None):
                                             d = copy.copy(inp["default"])
                                             jobobj[key] = (
                                                 "default", {"default": d})


Test patch:
Repository: DataBiosphere/toil

Issue:
Example script duplicated
`examples/hello.py` and `src/toil/test/docs/scripts/tutorial_arguments.py` are duplicate scripts with the same contents.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-443)
┆Issue Number: TOIL-443



Discussion:

Resolving patch:
diff --git a/examples/hello.py b/examples/hello.py
deleted file mode 100644
--- a/examples/hello.py
+++ /dev/null
@@ -1,19 +0,0 @@
-from toil.common import Toil
-from toil.job import Job
-
-class HelloWorld(Job):
-    def __init__(self, message):
-        Job.__init__(self,  memory="1G", cores=2, disk="2G")
-        self.message = message
-
-    def run(self, fileStore):
-        return "Hello, world!, here's a message: %s" % self.message
-
-if __name__=="__main__":
-    parser = Job.Runner.getDefaultArgumentParser()
-    options = parser.parse_args()
-
-    hello_job = HelloWorld("Woot")
-
-    with Toil(options) as toil:
-        print(toil.start(hello_job))


Test patch:
Repository: DataBiosphere/toil

Issue:
Kubernetes job prefix might spook Kubernetes
An extremely long user name, or a user name containing such spooky and terrifying characters as *uppercase letters*, could make us generate a job name that Kubernetes won't accept.

See https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-445)
┆Issue Number: TOIL-445



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/kubernetes.py b/src/toil/batchSystems/kubernetes.py
--- a/src/toil/batchSystems/kubernetes.py
+++ b/src/toil/batchSystems/kubernetes.py
@@ -35,6 +35,7 @@
 import os
 import pickle
 import pytz
+import string
 import subprocess
 import sys
 import uuid
@@ -72,15 +73,20 @@ def __init__(self, config, maxCores, maxMemory, maxDisk):
 
         # Find all contexts and the active context.
         # The active context gets us our namespace.
-        contexts, active_context = kubernetes.config.list_kube_config_contexts()
+        contexts, activeContext = kubernetes.config.list_kube_config_contexts()
         if not contexts:
             raise RuntimeError("No Kubernetes contexts available in ~/.kube/config or $KUBECONFIG")
             
         # Identify the namespace to work in
-        self.namespace = active_context.get('context', {}).get('namespace', 'default')
+        self.namespace = activeContext.get('context', {}).get('namespace', 'default')
+
+        # Make a Kubernetes-acceptable version of our username: not too long,
+        # and all lowercase letters, numbers, or - or .
+        acceptableChars = set(string.ascii_lowercase + string.digits + '-.')
+        safeUsername = ''.join([c for c in getpass.getuser().lower() if c in acceptableChars])[:100]
 
         # Create a prefix for jobs, starting with our username
-        self.jobPrefix = '{}-toil-{}-'.format(getpass.getuser(), uuid.uuid4())
+        self.jobPrefix = '{}-toil-{}-'.format(safeUsername, uuid.uuid4())
         
         # Instead of letting Kubernetes assign unique job names, we assign our
         # own based on a numerical job ID. This functionality is managed by the


Test patch:
Repository: DataBiosphere/toil

Issue:
Have a way to increase the disk requirement of failed jobs when retrying them
In #2876 @glennhickey complained that, with Kubernetes now enforcing disk limits, you can get in trouble if your workflow has a job with a too-low disk limit, because there's no way to increase it once the workflow has started.

There should be a way to recover a workflow that has a job that can't complete with the amount of disk it thinks it needs, instead of starting over completely.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-465)
┆Issue Number: TOIL-465



Discussion:

Resolving patch:
diff --git a/src/toil/jobGraph.py b/src/toil/jobGraph.py
--- a/src/toil/jobGraph.py
+++ b/src/toil/jobGraph.py
@@ -117,8 +117,13 @@ def setupJobAfterFailure(self, config):
         # batch system)
         if self.memory < config.defaultMemory:
             self._memory = config.defaultMemory
-            logger.warn("We have increased the default memory of the failed job %s to %s bytes",
+            logger.warn("We have increased the memory of the failed job %s to the default of %s bytes",
                         self, self.memory)
+            
+        if self.disk < config.defaultDisk:
+            self._disk = config.defaultDisk
+            logger.warn("We have increased the disk of the failed job %s to the default of %s bytes",
+                        self, self.disk)
 
     def restartCheckpoint(self, jobStore):
         """Restart a checkpoint after the total failure of jobs in its subtree.


Test patch:
Repository: DataBiosphere/toil

Issue:
Python 3.6 no longer available via ppa:jonathonf/python-3.6
Recently announced: https://launchpad.net/~jonathonf

I think we can use `ppa:deadsnakes/ppa` instead and things should work fine again.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-470)
┆Issue Number: TOIL-470



Discussion:

Resolving patch:
diff --git a/docker/Dockerfile.py b/docker/Dockerfile.py
--- a/docker/Dockerfile.py
+++ b/docker/Dockerfile.py
@@ -77,7 +77,7 @@ def heredoc(s):
         > /etc/apt/sources.list.d/nodesource.list \
         && apt-key adv --keyserver keyserver.ubuntu.com --recv 68576280
 
-    RUN add-apt-repository -y ppa:jonathonf/python-3.6
+    RUN add-apt-repository -y ppa:deadsnakes/ppa
     
     RUN apt-get -y update --fix-missing && \
         DEBIAN_FRONTEND=noninteractive apt-get -y upgrade && \


Test patch:
Repository: DataBiosphere/toil

Issue:
We call jobStore.getGlobalFileSize when we mean jobStore.getFileSize
https://github.com/DataBiosphere/toil/blob/7471248edc8569e37ec3d4c6161c65e9330b54e9/src/toil/fileStores/abstractFileStore.py#L306

The actual implemented method is `getFileSize`:

https://github.com/DataBiosphere/toil/blob/7471248edc8569e37ec3d4c6161c65e9330b54e9/src/toil/jobStores/abstractJobStore.py#L912

I don't think Cactus will work with the caching file store the way it is right now.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-482)
┆Issue Number: TOIL-482



Discussion:

Resolving patch:
diff --git a/src/toil/fileStores/abstractFileStore.py b/src/toil/fileStores/abstractFileStore.py
--- a/src/toil/fileStores/abstractFileStore.py
+++ b/src/toil/fileStores/abstractFileStore.py
@@ -303,7 +303,7 @@ def getGlobalFileSize(self, fileStoreID):
         if size is None:
             # It fell off
             # Someone is mixing FileStore and JobStore file APIs, or serializing FileIDs as strings.
-            size = self.jobStore.getGlobalFileSize(fileStoreID)
+            size = self.jobStore.getFileSize(fileStoreID)
 
         return size
 


Test patch:
Repository: DataBiosphere/toil

Issue:
Python 3.6 doesn't work well in a Toil Appliance
When I use `TOIL_APPLIANCE_SELF=quay.io/ucsc_cgl/toil:3.24.1a1-3475cb8e60da9e064296dd96b6460562aebb0356 toil launch-cluster...`

I get something with python3 defaulting to `3.5.2`.  It is also where Toil seems to get installed.  Python3 support in Cactus requires Python 3.6.  This means I can' use (non singleMachine) batch systems with Cactus under Python3

@adamnovak mentions 

> Hm, maybe Toil is installing in 3.5 instead of 3.6. We should change https://github.com/DataBiosphere/toil/blob/3475cb8e60da9e064296dd96b6460562aebb0356/docker/Dockerfile.py#L25 to use something like https://github.com/DataBiosphere/toil/blob/3475cb8e60da9e064296dd96b6460562aebb0356/version_template.py#L67-L75 but to select the major and minor version of pip to use.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-507)
┆Issue Number: TOIL-507



Discussion:

Resolving patch:
diff --git a/docker/Dockerfile.py b/docker/Dockerfile.py
--- a/docker/Dockerfile.py
+++ b/docker/Dockerfile.py
@@ -22,7 +22,7 @@
 
 # Make sure to install packages into the pip for the version of Python we are
 # building for.
-pip = 'pip{}'.format(sys.version_info[0])
+pip = 'python{}.{} -m pip'.format(sys.version_info[0], sys.version_info[1])
 
 
 dependencies = ' '.join(['libffi-dev',  # For client side encryption for extras with PyNACL
@@ -125,7 +125,7 @@ def heredoc(s):
     RUN {pip} install --upgrade virtualenv==15.0.3
 
     # Install s3am (--never-download prevents silent upgrades to pip, wheel and setuptools)
-    RUN virtualenv --python python3 --never-download /home/s3am \
+    RUN virtualenv --python python3.6 --never-download /home/s3am \
         && /home/s3am/bin/pip install s3am==2.0 \
         && ln -s /home/s3am/bin/s3am /usr/local/bin/
 
diff --git a/version_template.py b/version_template.py
--- a/version_template.py
+++ b/version_template.py
@@ -67,13 +67,7 @@ def python():
     appliance is only built for particular Python versions, and we would like
     workflows to work with a variety of leader Python versions.
     """
-    import sys
-    
-    if sys.version_info[0] == 3:
-        # Ignore minor version
-        return 'python3'
-    else:
-        return exactPython() 
+    return exactPython()
 
 
 def _pythonVersionSuffix():
@@ -86,10 +80,7 @@ def _pythonVersionSuffix():
 
     # For now, we assume all Python 3 releases are intercompatible.
     # We also only tag the Python 2 releases specially, since Python 2 is old and busted.
-    if sys.version_info[0] == 3:
-        return ''
-    else:
-        return '-py{}'.format(sys.version_info[0])
+    return '-py{}.{}'.format(sys.version_info[0], sys.version_info[1])
 
       
 def dockerTag():


Test patch:
Repository: DataBiosphere/toil

Issue:
python3 exception when logging to mtailProc
Exception:
```
 Traceback (most recent call last):
   File "/usr/local/bin/toil-cwl-runner", line 11, in <module>
     sys.exit(main())
   File "/usr/local/lib/python3.6/dist-packages/toil/cwl/cwltoil.py", line 1509, in main
     outobj = toil.start(wf1)
   File "/usr/local/lib/python3.6/dist-packages/toil/common.py", line 818, in start
     return self._runMainLoop(rootJobGraph)
   File "/usr/local/lib/python3.6/dist-packages/toil/common.py", line 1089, in _runMainLoop
     jobCache=self._jobCache).run()
   File "/usr/local/lib/python3.6/dist-packages/toil/leader.py", line 213, in run
     self.innerLoop()
   File "/usr/local/lib/python3.6/dist-packages/toil/leader.py", line 535, in innerLoop
     self._processReadyJobs()
   File "/usr/local/lib/python3.6/dist-packages/toil/leader.py", line 459, in _processReadyJobs
     self._processReadyJob(jobGraph, resultStatus)
   File "/usr/local/lib/python3.6/dist-packages/toil/leader.py", line 409, in _processReadyJob
     self.issueJob(JobNode.fromJobGraph(jobGraph))
   File "/usr/local/lib/python3.6/dist-packages/toil/leader.py", line 619, in issueJob
     self.toilMetrics.logIssuedJob(jobNode)
   File "/usr/local/lib/python3.6/dist-packages/toil/common.py", line 1269, in logIssuedJob
     self.log("issued_job %s" % jobType)
   File "/usr/local/lib/python3.6/dist-packages/toil/common.py", line 1253, in log
     self.mtailProc.stdin.write(message + "\n") #self.mtailProc.stdin.write((message + "\n").encode('utf-8'))
 TypeError: a bytes-like object is required, not 'str'
```

Also, looks like the new subprocess module doesn't immediately flush the log stats like before, so stats take a while to update. Adding `flush()` should do the trick since it's not too high volume.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-514)
┆Issue Number: TOIL-514



Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -1231,7 +1231,8 @@ def requestPredicate(e):
 
     def log(self, message):
         if self.mtailProc:
-            self.mtailProc.stdin.write(message + "\n")
+            self.mtailProc.stdin.write((message + "\n").encode("utf-8"))
+            self.mtailProc.stdin.flush()
 
     # Note: The mtail configuration (dashboard/mtail/toil.mtail) depends on these messages
     # remaining intact


Test patch:
Repository: DataBiosphere/toil

Issue:
Mesos' frameworkMessage no longer working in py3
The [`frameworkMessage`](https://github.com/DataBiosphere/toil/blob/e193fdce9c7a5b36498f1d40475aff04abc35495/src/toil/batchSystems/mesos/batchSystem.py#L659) method seems to be longer invoked since py3 update. This results in `executor.nodeInfo` being null, which ultimately results in instances [not getting terminated](https://github.com/DataBiosphere/toil/blob/9f8f1fa0b934aa1af8b7aac2d314948f58c5e0cf/src/toil/provisioners/clusterScaler.py#L703) when downscaling as the number of running tasks cannot be determined. Note that `resourceOffers` and other functions are still called.

I don't yet know the underlying reason for this.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-515)
┆Issue Number: TOIL-515



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/mesos/executor.py b/src/toil/batchSystems/mesos/executor.py
--- a/src/toil/batchSystems/mesos/executor.py
+++ b/src/toil/batchSystems/mesos/executor.py
@@ -138,7 +138,7 @@ def _sendFrameworkMessage(self, driver):
                                         memoryTotal=psutil.virtual_memory().total,
                                         workers=len(self.runningTasks))
             log.debug("Send framework message: %s", message)
-            driver.sendFrameworkMessage(encode_data(repr(message)))
+            driver.sendFrameworkMessage(encode_data(repr(message).encode('utf-8')))
             # Prevent workers launched together from repeatedly hitting the leader at the same time
             time.sleep(random.randint(45, 75))
 


Test patch:
Repository: DataBiosphere/toil

Issue:
Remove Sphinx as a dependency
As mentioned in https://github.com/DataBiosphere/toil/pull/3011#issuecomment-601349848 we depend on Sphinx, but we only use it for generating documentation. We never use it at runtime.

We should remove the Sphinx dependency, and just make sure it is installed by the `Makefile` when running the `docs` target. Maybe it could be one of the things installed by `make prepare`.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-528)
┆Issue Number: TOIL-528



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -53,7 +53,6 @@ def runSetup():
     docker = 'docker==2.5.1'
     dateutil = 'python-dateutil'
     addict = 'addict<=2.2.0'
-    sphinx = 'sphinx>=2.4.4, <3'
     pathlib2 = 'pathlib2==2.3.2'
 
     core_reqs = [
@@ -65,7 +64,6 @@ def runSetup():
         dateutil,
         psutil,
         addict,
-        sphinx,
         pathlib2,
         pytz]
 


Test patch:
Repository: DataBiosphere/toil

Issue:
SGE batch system doesn't always honor TOIL_GRIDENGINE_PE
the TOIL_GRIDENGINE_PE environment parameter is only used for when number of job CPUs is great than one.  Some configurations require -pe  even with one CPU.  Also, if multiple CPUs are requested and TOIL_GRIDENGINE_PE is not set, it defaults to 'shm'.  There is no requirement to have such a PE.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-541)
┆Issue Number: TOIL-541



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/gridengine.py b/src/toil/batchSystems/gridengine.py
--- a/src/toil/batchSystems/gridengine.py
+++ b/src/toil/batchSystems/gridengine.py
@@ -134,9 +134,12 @@ def prepareQsub(self, cpu, mem, jobID):
                     if arg.startswith(("vf=", "hvmem=", "-pe")):
                         raise ValueError("Unexpected CPU, memory or pe specifications in TOIL_GRIDGENGINE_ARGs: %s" % arg)
                 qsubline.extend(sgeArgs)
-            if cpu is not None and math.ceil(cpu) > 1:
-                peConfig = os.getenv('TOIL_GRIDENGINE_PE') or 'shm'
-                qsubline.extend(['-pe', peConfig, str(int(math.ceil(cpu)))])
+            if os.getenv('TOIL_GRIDENGINE_PE') is not None:
+                peCpu = int(math.ceil(cpu)) if cpu is not None else 1
+                qsubline.extend(['-pe', os.getenv('TOIL_GRIDENGINE_PE'), str(peCpu)])
+            elif (cpu is not None) and (cpu > 1):
+                raise RuntimeError("must specify PE in TOIL_GRIDENGINE_PE environment variable when using multiple CPUs. "
+                                   "Run qconf -spl and your local documentation for possible values")
 
             stdoutfile = self.boss.formatStdOutErrPath(jobID, 'gridengine', '$JOB_ID', 'std_output')
             stderrfile = self.boss.formatStdOutErrPath(jobID, 'gridengine', '$JOB_ID', 'std_error')


Test patch:
Repository: DataBiosphere/toil

Issue:
Debug logging with single machine is now extremely spammy because it logs on every scheduling loop.
Trying to run Cactus with --logDebug on single machine gives me:

```
Tried to start job 41 and got: None
Not enough cores to run job 38
Tried to start job 38 and got: None
Not enough cores to run job 41
Tried to start job 41 and got: None
Not enough cores to run job 38
Tried to start job 38 and got: None
Not enough cores to run job 41
Tried to start job 41 and got: None
...
```

I get many messages like this per second.

This is my own dang fault for spinning so fast/logging so much in the inner loop of the single machine scheduler. I think we should just remove these particular debug log lines as we aren't currently trying to debug the single machine scheduler itself.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-555)
┆Issue Number: TOIL-555



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/singleMachine.py b/src/toil/batchSystems/singleMachine.py
--- a/src/toil/batchSystems/singleMachine.py
+++ b/src/toil/batchSystems/singleMachine.py
@@ -207,8 +207,6 @@ def daddy(self):
                         result = self._startChild(jobCommand, jobID,
                             coreFractions, jobMemory, jobDisk, environment)
 
-                        log.debug('Tried to start job %s and got: %s', jobID, str(result))
-                        
                         if result is None:
                             # We did not get the resources to run this job.
                             # Requeue last, so we can look at the next job.
@@ -361,9 +359,6 @@ def _setSchedulingStatusMessage(self, message):
         """
         
         self.schedulingStatusMessage = message
-        
-        # Report the message in the debug log too.
-        log.debug(message)
 
     def _startChild(self, jobCommand, jobID, coreFractions, jobMemory, jobDisk, environment):
         """


Test patch: