internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataBiosphere/toil

Issue:
AttributeError: 'Expando' object has no attribute 'workers'
Both with  current master 044970558318d28654d2f79d1c9ccf31c4dfab81 and 3.18.0:
```
$ toil stats scratch/workflow/22/job
DEBUG:past.translation:Running find_module: avro...
DEBUG:past.translation:Running load_module for avro...
DEBUG:past.translation:Autoconverting avro ...
DEBUG:past.translation:Running find_module: avro.schema...
DEBUG:past.translation:Running load_module for avro.schema...
DEBUG:past.translation:Autoconverting avro.schema ...
Traceback (most recent call last):
  File "/home/gijs/Work/buis/.venv/bin/toil", line 11, in <module>
    load_entry_point('toil', 'console_scripts', 'toil')()
  File "/home/gijs/Work/toil/src/toil/utils/toilMain.py", line 35, in main
    module.main()
  File "/home/gijs/Work/toil/src/toil/utils/toilStats.py", line 604, in main
    collatedStatsTag = processData(jobStore.config, stats)
  File "/home/gijs/Work/toil/src/toil/utils/toilStats.py", line 554, in processData
    worker = [_f for _f in stats.workers if _f]
AttributeError: 'Expando' object has no attribute 'workers'
```


Discussion:
starting the workflow with 
`$ toil-cwl-runner --jobStore=scratch/workflow/22/job --workDir=scratch/workflow/22/work workflow.cwl job.yaml`

and workflow looks like:
```
cwlVersion: v1.0
class: CommandLineTool
baseCommand: sleep

inputs:
  seconds:
    type: int
    inputBinding:
      position: 1

outputs: {}
```
looks related to https://github.com/DataBiosphere/toil/issues/2188

i don't get it, I thought I had this working. Maybe the `workers` en `jobs` fields are not writting to the config with such a simple workflow.
ah wait, this CWL file only has one step, that seems to be the problem.

Resolving patch:
diff --git a/src/toil/utils/toilStats.py b/src/toil/utils/toilStats.py
--- a/src/toil/utils/toilStats.py
+++ b/src/toil/utils/toilStats.py
@@ -535,7 +535,8 @@ def processData(config, stats):
     """
     Collate the stats and report
     """
-    if 'total_time' not in stats:  # toil job not finished yet
+    if 'total_time' not in stats or 'total_clock' not in stats:
+        # toil job not finished yet
         stats.total_time = [0.0]
         stats.total_clock = [0.0]
 
@@ -551,8 +552,8 @@ def processData(config, stats):
                                )
 
     # Add worker info
-    worker = [_f for _f in stats.workers if _f]
-    jobs = [_f for _f in stats.jobs if _f]
+    worker = [_f for _f in getattr(stats, 'workers', []) if _f]
+    jobs = [_f for _f in getattr(stats, 'jobs', []) if _f]
     jobs = [item for sublist in jobs for item in sublist]
 
     def fn4(job):
@@ -563,7 +564,7 @@ def fn4(job):
 
     buildElement(collatedStatsTag, worker, "worker")
     createSummary(buildElement(collatedStatsTag, jobs, "jobs"),
-                  stats.workers, "worker", fn4)
+                  getattr(stats, 'workers', []), "worker", fn4)
     # Get info for each job
     jobNames = set()
     for job in jobs:


Test patch:
Repository: DataBiosphere/toil

Issue:
Intermediate files in a CWL workflow that should be copied to the workdir fail completely when the input data is on a separate filesystem from the jobstore.
See #2534.


Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/fileJobStore.py b/src/toil/jobStores/fileJobStore.py
--- a/src/toil/jobStores/fileJobStore.py
+++ b/src/toil/jobStores/fileJobStore.py
@@ -387,14 +387,14 @@ def readFile(self, jobStoreFileID, localFilePath, symlink=False):
                         raise
             else:
                 try:
-                    os.link(jobStoreFilePath, localFilePath)
+                    os.link(os.path.realpath(jobStoreFilePath), localFilePath)
                 except OSError as e:
                     if e.errno == errno.EEXIST:
                         # Overwrite existing file, emulating shutil.copyfile().
                         os.unlink(localFilePath)
                         # It would be very unlikely to fail again for same reason but possible
                         # nonetheless in which case we should just give up.
-                        os.link(jobStoreFilePath, localFilePath)
+                        os.link(os.path.realpath(jobStoreFilePath), localFilePath)
                     else:
                         logger.critical('jobStoreFilePath: ' + jobStoreFilePath + ' ' + str(os.path.exists(jobStoreFilePath)))
                         logger.critical('localFilePath: ' + localFilePath + ' ' + str(os.path.exists(localFilePath)))


Test patch:
Repository: DataBiosphere/toil

Issue:
Env value of TOIL_DOCKER_REGISTRY is not honored in the docker image
When a docker image is build with a custom `TOIL_DOCKER_REGISTRY`, the built docker image still looks at `quay.io/ucsc_cgl` for pulling related images (e.g. `mtail`, `grafana`, etc). The fix is to put the actual registry in `version_template.py`.


Discussion:

Resolving patch:
diff --git a/version_template.py b/version_template.py
--- a/version_template.py
+++ b/version_template.py
@@ -28,7 +28,6 @@
 
 baseVersion = '3.20.0a1'
 cgcloudVersion = '1.6.0a1.dev393'
-dockerRegistry = 'quay.io/ucsc_cgl'
 dockerName = 'toil'
 
 
@@ -115,6 +114,11 @@ def currentCommit():
     return str(output)
 
 
+def dockerRegistry():
+    import os
+    return os.getenv('TOIL_DOCKER_REGISTRY', 'quay.io/ucsc_cgl')
+
+
 def dirty():
     from subprocess import call
     try:


Test patch:
Repository: DataBiosphere/toil

Issue:
Caching is disabled by default and no way to turn it on
Caching is currently disabled by default ( from https://github.com/DataBiosphere/toil/commit/2e54371ea316fe2803ce53d1477a51fadcfb5d2e ) and there is no way to turn it on.

Options:
* Change the default value of `disableCaching` back to False
* Add an `enableCaching` option to preserve the current default behavior.


Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -519,7 +519,8 @@ def _addOptions(addGroupFn, config):
     # Misc options
     #
     addOptionFn = addGroupFn("Toil Miscellaneous Options", "Miscellaneous Options")
-    addOptionFn('--disableCaching', dest='disableCaching', action='store_true', default=True,
+    addOptionFn('--disableCaching', dest='disableCaching',
+                type='bool', nargs='?', const=True, default=True,
                 help='Disables caching in the file store. This flag must be set to use '
                      'a batch system that does not support caching such as Grid Engine, Parasol, '
                      'LSF, or Slurm')
@@ -593,6 +594,7 @@ def addOptions(parser, config=Config()):
         def addGroup(headingString, bodyString):
             return parser.add_argument_group(headingString, bodyString).add_argument
 
+        parser.register("type", "bool", lambda v: v.lower() == "true")  # Custom type for arg=True/False.
         _addOptions(addGroup, config)
     else:
         raise RuntimeError("Unanticipated class passed to addOptions(), %s. Expecting "


Test patch:
Repository: DataBiosphere/toil

Issue:
S3 jobstore deletion is slow for very large jobs
Very large jobs can have 10k+ files in the s3 jobstore bucket. Currently, the deletion is done one file at a time in a single thread (a separate API call is made for each file) ([code](https://github.com/DataBiosphere/toil/blob/bda6f311d4f1a8f66635728c2f2447049701fe2f/src/toil/jobStores/aws/jobStore.py#L1291)). The alternative is to use the `delete_keys` method that makes one API call for each 1000 files, which dramatically speeds up the cleanup process.
 


Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -1287,8 +1287,10 @@ def _delete_bucket(self, bucket):
                 try:
                     for upload in bucket.list_multipart_uploads():
                         upload.cancel_upload()
+                    keys = list()
                     for key in bucket.list_versions():
-                        bucket.delete_key(key.name, version_id=key.version_id)
+                        keys.append((key.name, key.version_id))
+                    bucket.delete_keys(keys, quiet=True)
                     bucket.delete()
                 except S3ResponseError as e:
                     if e.error_code == 'NoSuchBucket':


Test patch:
Repository: DataBiosphere/toil

Issue:
Intermediate files in a CWL workflow that should be copied to the workdir fail completely when the input data is on a separate filesystem from the jobstore.
See #2534.


Discussion:
I found the error and this was likely introduced with the feature of creating symlinks for jobstore files  (#1687). The fileJobStore checks if the jobstore file is on the same device as the workdir before creating a hard link. If the file is a symlink it will be resolved during the device check but not when creating the hard link. If the symlink is on a different device from the workdir the hard link will fail.

Resolving patch:
diff --git a/src/toil/jobStores/fileJobStore.py b/src/toil/jobStores/fileJobStore.py
--- a/src/toil/jobStores/fileJobStore.py
+++ b/src/toil/jobStores/fileJobStore.py
@@ -387,14 +387,14 @@ def readFile(self, jobStoreFileID, localFilePath, symlink=False):
                         raise
             else:
                 try:
-                    os.link(os.path.realpath(jobStoreFilePath), localFilePath)
+                    os.link(jobStoreFilePath, localFilePath)
                 except OSError as e:
                     if e.errno == errno.EEXIST:
                         # Overwrite existing file, emulating shutil.copyfile().
                         os.unlink(localFilePath)
                         # It would be very unlikely to fail again for same reason but possible
                         # nonetheless in which case we should just give up.
-                        os.link(os.path.realpath(jobStoreFilePath), localFilePath)
+                        os.link(jobStoreFilePath, localFilePath)
                     else:
                         logger.critical('jobStoreFilePath: ' + jobStoreFilePath + ' ' + str(os.path.exists(jobStoreFilePath)))
                         logger.critical('localFilePath: ' + localFilePath + ' ' + str(os.path.exists(localFilePath)))


Test patch:
Repository: DataBiosphere/toil

Issue:
Python 3 tests can't run because python-dev and python3.6-dev now conflict
```
Step 7/28 : RUN apt-get -y install libffi-dev python3.6 python3.6-dev python-dev python3-dev python-pip python3-pip libcurl4-openssl-dev libssl-dev wget curl openssh-server mesos=1.0.1-2.0.94.ubuntu1604 nodejs rsync screen && apt-get clean && rm -rf /var/lib/apt/lists/*
 ---> Running in 20f21279c094
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 python3.6 : Breaks: python3-dev (< 3.6.5~rc1-1) but 3.5.1-3 is to be installed
E: Unable to correct problems, you have held broken packages.
The command '/bin/sh -c apt-get -y install libffi-dev python3.6 python3.6-dev python-dev python3-dev python-pip python3-pip libcurl4-openssl-dev libssl-dev wget curl openssh-server mesos=1.0.1-2.0.94.ubuntu1604 nodejs rsync screen && apt-get clean && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100
Makefile:202: recipe for target 'docker' failed
make: *** [docker] Error 100
```

We can't have both these packages installed at the same time apparently, at least with the versions currently in the repos.

See also maybe: #2408

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-357)
┆Issue Number: TOIL-357



Discussion:

Resolving patch:
diff --git a/docker/Dockerfile.py b/docker/Dockerfile.py
--- a/docker/Dockerfile.py
+++ b/docker/Dockerfile.py
@@ -24,7 +24,6 @@
                          'python3.6',
                          'python3.6-dev',
                          'python-dev',  # For installing Python packages with native code
-                         'python3-dev',
                          'python-pip',  # Bootstrap pip, but needs upgrading, see below
                          'python3-pip',
                          'libcurl4-openssl-dev',


Test patch:
Repository: DataBiosphere/toil

Issue:
Docker build allows apt-get updates to come from cache and apt-get upgrades/installs to run based on them
It's possible that the `apt-get update` commands in our Dockerfile don't actually run before the apt-get upgrade commands they are supposed to do work for, because the image they produce might be put in the Docker cache and then reused.

We should always apt-get update on the same line as any other apt operation.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-361)
┆Issue Number: TOIL-361



Discussion:

Resolving patch:
diff --git a/docker/Dockerfile.py b/docker/Dockerfile.py
--- a/docker/Dockerfile.py
+++ b/docker/Dockerfile.py
@@ -60,9 +60,7 @@ def heredoc(s):
 print(heredoc('''
     FROM ubuntu:16.04
     
-    RUN apt-get -y update && apt-get -y upgrade
-
-    RUN apt-get -y update --fix-missing && apt-get -y install apt-transport-https ca-certificates software-properties-common
+    RUN apt-get -y update --fix-missing && apt-get -y upgrade && apt-get -y install apt-transport-https ca-certificates software-properties-common && apt-get clean && rm -rf /var/lib/apt/lists/*
 
     RUN echo "deb http://repos.mesosphere.io/ubuntu/ xenial main" \
         > /etc/apt/sources.list.d/mesosphere.list \
@@ -73,9 +71,7 @@ def heredoc(s):
 
     RUN add-apt-repository -y ppa:jonathonf/python-3.6
     
-    RUN apt-get -y update
-
-    RUN apt-get -y install {dependencies} && apt-get clean && rm -rf /var/lib/apt/lists/*
+    RUN apt-get -y update --fix-missing && apt-get -y upgrade && apt-get -y install {dependencies} && apt-get clean && rm -rf /var/lib/apt/lists/*
 
     RUN mkdir /root/.ssh && \
         chmod 700 /root/.ssh


Test patch:
Repository: DataBiosphere/toil

Issue:
runningOnEC2 function is broken on c5/m5 instances
The current method of checking [`/sys/hypervisor/uuid`](https://github.com/DataBiosphere/toil/blob/master/src/toil/test/__init__.py#L310) does not work on new c5/m5 instances (see https://serverfault.com/a/903599 )
The proper fix is to check `http://169.254.169.254/latest/dynamic/instance-identity/` according to [AWS doc](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html).

Also, that method is for some reason under `test`, but it's used when checking for zones in the main code [here](https://github.com/DataBiosphere/toil/blob/master/src/toil/provisioners/aws/__init__.py#L53), so it should probably be moved to `provisioners/aws/__init__.py`

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-392)
┆Issue Number: TOIL-392



Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/aws/__init__.py b/src/toil/provisioners/aws/__init__.py
--- a/src/toil/provisioners/aws/__init__.py
+++ b/src/toil/provisioners/aws/__init__.py
@@ -19,14 +19,31 @@
 import datetime
 from toil.lib.misc import std_dev, mean
 from six import string_types
-
-from toil.test import runningOnEC2
+from six.moves.urllib.request import urlopen
+from six.moves.urllib.error import URLError
 
 logger = logging.getLogger(__name__)
 
 ZoneTuple = namedtuple('ZoneTuple', ['name', 'price_deviation'])
 
 
+def runningOnEC2():
+    def file_begins_with(path, prefix):
+        with open(path) as f:
+            return f.read(len(prefix)) == prefix
+
+    hv_uuid_path = '/sys/hypervisor/uuid'
+    if os.path.exists(hv_uuid_path) and file_begins_with(hv_uuid_path, 'ec2'):
+        return True
+    # Some instances do not have the /sys/hypervisor/uuid file, so check the identity document instead.
+    # See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-identity-documents.html
+    try:
+        urlopen('http://169.254.169.254/latest/dynamic/instance-identity/document', timeout=1)
+        return True
+    except URLError:
+        return False
+
+
 def zoneToRegion(zone):
     """Get a region (e.g. us-west-2) from a zone (e.g. us-west-1c)."""
     from toil.lib.context import Context


Test patch:
diff --git a/src/toil/test/__init__.py b/src/toil/test/__init__.py
--- a/src/toil/test/__init__.py
+++ b/src/toil/test/__init__.py
@@ -38,6 +38,7 @@
 from toil.lib.iterables import concat
 from toil.lib.threading import ExceptionalThread
 from toil.lib.misc import mkdir_p
+from toil.provisioners.aws import runningOnEC2
 from toil import subprocess
 from toil import which
 from toil import toilPackageDirPath, applianceSelf
@@ -302,16 +303,6 @@ def travis_test(test_item):
         return test_item
 
 
-def file_begins_with(path, prefix):
-    with open(path) as f:
-        return f.read(len(prefix)) == prefix
-
-
-def runningOnEC2():
-    hv_uuid_path = '/sys/hypervisor/uuid'
-    return os.path.exists(hv_uuid_path) and file_begins_with(hv_uuid_path, 'ec2')
-
-
 def needs_google(test_item):
     """
     Use as a decorator before test classes or methods to only run them if Google Storage usable.
Repository: DataBiosphere/toil

Issue:
CWL tmpdir is not unique in docker runs
While debugging a strange failure in a pipeline, I noticed that the CWL tmpdir is actually the same for all docker runs in a worker (i.e. the mapped volume is `--volume=/var/lib/toil:/tmp:rw` if `--tmpdir-prefix /var/lib/toil` is used). The same is true without `--tmpdir-prefix` (with the difference that everything would be mapped to `tmp` within the current folder, but all tasks share the same `tmp` directory).

This can cause data corruption if jobs assume that the `/tmp` directory is unique (within the docker context) as jobs would overwrite data from others running in the same worker.

Searching a bit, it looks like setting [`runtime_context.tmpdir`](https://github.com/DataBiosphere/toil/blob/1460f7371068019410f5058c11f3c1c6628c8db9/src/toil/cwl/cwltoil.py#L1231) is not necessary and actually causes issues as it essentially [overrides `tmpdir_prefix`](https://github.com/common-workflow-language/cwltool/blob/master/cwltool/command_line_tool.py#L542). So, I think just setting `tmpdir_prefix` should be enough on the Toil side and would result in the correct behavior.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-400)
┆Issue Number: TOIL-400



Discussion:

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -542,8 +542,7 @@ def run(self, file_store):
                 cwljob.pop(inp_id)
 
         # Exports temporary directory for batch systems that reset TMPDIR
-        os.environ["TMPDIR"] = os.path.realpath(
-            self.runtime_context.tmpdir or file_store.getLocalTempDir())
+        os.environ["TMPDIR"] = os.path.realpath(file_store.getLocalTempDir())
         outdir = os.path.join(file_store.getLocalTempDir(), "out")
         os.mkdir(outdir)
         # Just keep the temporary output prefix under the job's local temp dir,
@@ -1128,7 +1127,6 @@ def main(args=None, stdout=sys.stdout):
 
     outdir = os.path.abspath(options.outdir)
     tmp_outdir_prefix = os.path.abspath(options.tmp_outdir_prefix)
-    tmpdir_prefix = os.path.abspath(options.tmpdir_prefix)
 
     fileindex = {}
     existing = {}
@@ -1228,7 +1226,6 @@ def set_secondary(fileobj):
 
             try:
                 runtime_context.use_container = use_container
-                runtime_context.tmpdir = os.path.realpath(tmpdir_prefix)
                 runtime_context.tmp_outdir_prefix = os.path.realpath(
                     tmp_outdir_prefix)
                 runtime_context.job_script_provider = job_script_provider


Test patch:
Repository: DataBiosphere/toil

Issue:
--default-container option not available in toil-cwl-runner
Hello,
It would useful to have a --default-container when using toil-cwl-runner, as we do in cwltool.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-415)
┆Issue Number: TOIL-415



Discussion:

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -1080,7 +1080,9 @@ def main(args=None, stdout=sys.stdout):
         "--relax-path-checks", action="store_true",
         default=False, help="Relax requirements on path names to permit "
         "spaces and hash characters.", dest="relax_path_checks")
-
+    parser.add_argument("--default-container",
+                        help="Specify a default docker container that will be "
+                        "used if the workflow fails to specify one.")
     if args is None:
         args = sys.argv[1:]
 


Test patch:
Repository: DataBiosphere/toil

Issue:
Healthy workers being ignored by Toil
There's a bug when ignoring nodes (for termination) that results in healthy workers being ignored by the batch system. This manifests in long running jobs where a worker that is previously ignored by the autoscaler (i.e. when scaling down) appears with the same IP later on (i.e. when scaling up).

* There is a [bug when removing nodes from the 'ignored' list](https://github.com/DataBiosphere/toil/blob/b37cc3f478249a98b5bb84a4e8939a9a9e079715/src/toil/provisioners/clusterScaler.py#L674) if the worker is being terminated instantly. Easy fix: should be `if node.privateIP in self.ignoredNodes`
* There is also a more subtle bug: a worker is added to the ignored list, but cannot be terminated as tasks are still running in the worker. The worker is then preempted, which gets removed from [autoscaler's ignored list](https://github.com/DataBiosphere/toil/blob/b37cc3f478249a98b5bb84a4e8939a9a9e079715/src/toil/provisioners/clusterScaler.py#L688). However, it's still in the batch system's ignored list (as `unignore` is not called). This results in the batch system still having the worker's IP in its ignored list even if it shows up later when scaling up. The solution is to also remove the ignored worker from the batch system.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-416)
┆Issue Number: TOIL-416



Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/clusterScaler.py b/src/toil/provisioners/clusterScaler.py
--- a/src/toil/provisioners/clusterScaler.py
+++ b/src/toil/provisioners/clusterScaler.py
@@ -671,7 +671,7 @@ def _removeNodes(self, nodeToNodeInfo, nodeType, numNodes, preemptable=False, fo
         logger.debug('Terminating %i instance(s).', len(nodeToNodeInfo))
         if nodeToNodeInfo:
             for node in nodeToNodeInfo:
-                if node in self.ignoredNodes:
+                if node.privateIP in self.ignoredNodes:
                     self.ignoredNodes.remove(node.privateIP)
                     self.leader.batchSystem.unignoreNode(node.privateIP)
             self.provisioner.terminateNodes(nodeToNodeInfo)
@@ -685,7 +685,10 @@ def _terminateIgnoredNodes(self):
         #Remove any nodes that have already been terminated from the list
         # of ignored nodes
         allNodeIPs = [node.privateIP for node in nodeToNodeInfo]
-        self.ignoredNodes = set([ip for ip in self.ignoredNodes if ip in allNodeIPs])
+        terminatedIPs = set([ip for ip in self.ignoredNodes if ip not in allNodeIPs])
+        for ip in terminatedIPs:
+            self.ignoredNodes.remove(ip)
+            self.leader.batchSystem.unignoreNode(ip)
 
         logger.debug("There are %i nodes being ignored by the batch system, "
                     "checking if they can be terminated" % len(self.ignoredNodes))


Test patch: