internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataBiosphere/toil

Issue:
Write tutorial for kubernetes launch cluster
The existing kubernetes tutorial assumes a preexisting cluster, Toil can create kubernetes clusters, we just don't tell anyone how.



┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-987)
┆friendlyId: TOIL-987



Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/abstractProvisioner.py b/src/toil/provisioners/abstractProvisioner.py
--- a/src/toil/provisioners/abstractProvisioner.py
+++ b/src/toil/provisioners/abstractProvisioner.py
@@ -827,6 +827,7 @@ def getKubernetesValues(self, architecture: str = 'amd64'):
         """
         Returns a dict of Kubernetes component versions and paths for formatting into Kubernetes-related templates.
         """
+        cloud_provider = self.getKubernetesCloudProvider()
         return dict(
             ARCHITECTURE=architecture,
             CNI_VERSION="v0.8.2",
@@ -849,7 +850,7 @@ def getKubernetesValues(self, architecture: str = 'amd64'):
             METRICS_API_VERSION="v0.3.7",
             CLUSTER_NAME=self.clusterName,
             # YAML line that tells the Kubelet to use a cloud provider, if we need one.
-            CLOUD_PROVIDER_SPEC=('cloud-provider: ' + self.getKubernetesCloudProvider()) if self.getKubernetesCloudProvider() else ''
+            CLOUD_PROVIDER_SPEC=('cloud-provider: ' + cloud_provider) if cloud_provider else ''
         )
 
     def addKubernetesServices(self, config: InstanceConfiguration, architecture: str = 'amd64'):


Test patch:
Repository: DataBiosphere/toil

Issue:
GPU support broken on SingleMachine?
This came up here: https://github.com/ComparativeGenomicsToolkit/cactus/issues/1062

The error is 
```
 File "/home/cactus/cactus_env/lib/python3.8/site-packages/toil/batchSystems/singleMachine.py", line 604, in _startChild
    snapshot = accelerator_set.get_free_snapshot()
AttributeError: 'ResourcePool' object has no attribute 'get_free_snapshot'
```
and seems due to the fact that accelerator_set is a `ResourcePool` when the method in qestion is a member of `ResourceSet`.  

This issue occurs in v5.10.0 and v5.11.0 but not v5.9.0.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1351)
┆Issue Number: TOIL-1351



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/singleMachine.py b/src/toil/batchSystems/singleMachine.py
--- a/src/toil/batchSystems/singleMachine.py
+++ b/src/toil/batchSystems/singleMachine.py
@@ -601,7 +601,7 @@ def _startChild(self, jobCommand, jobID, coreFractions, jobMemory, jobDisk, job_
         if job_accelerators:
             # Try and find some accelerators to use.
             # Start with all the accelerators that are free right now
-            accelerator_set : ResourceSet = self.resource_sources[3]
+            accelerator_set : ResourceSet = self.resource_sources[-1]
             snapshot = accelerator_set.get_free_snapshot()
             # And build a plan of the ones we want
             accelerators_needed, problem = self._identify_sufficient_accelerators(job_accelerators, snapshot)


Test patch:
Repository: DataBiosphere/toil

Issue:
WES ignores host in production
When trying to run `toil server --host 0.0.0.0`, I noticed that it would always only listen on `127.0.0.1` no matter what `--host` is set to but running with `--debug` didn't have this problem.

```
❯ toil server --host 0.0.0.0
...
[2022-11-11 16:50:46 +0000] [7173] [INFO] Starting gunicorn 20.1.0
[2022-11-11 16:50:46 +0000] [7173] [INFO] Listening at: http://127.0.0.1:8000
...
```
vs
```
❯ toil server --host 0.0.0.0 --debug
...
INFO:werkzeug:WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:8080
...
```

I tracked the problem down to [this line](https://github.com/DataBiosphere/toil/blob/master/src/toil/server/wsgi_app.py#L44). It appears to be overwriting the settings taken from the command line with Gunicorn's defaults before checking to see if anything has been set which `bind` won't be as it's been set to `None` in the merge.

Swapping the dictionaries around seems to have fixed it.
```python
        for key, value in {**vars(env_args), **self.options}.items():
```

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1242)
┆Issue Number: TOIL-1242



Discussion:

Resolving patch:
diff --git a/src/toil/server/wsgi_app.py b/src/toil/server/wsgi_app.py
--- a/src/toil/server/wsgi_app.py
+++ b/src/toil/server/wsgi_app.py
@@ -41,7 +41,7 @@ def load_config(self) -> None:
 
         # TODO: also read from the Gunicorn config file?
 
-        for key, value in {**self.options, **vars(env_args)}.items():
+        for key, value in {**vars(env_args), **self.options}.items():
             if key in self.cfg.settings and value is not None:
                 self.cfg.set(key.lower(), value)
 


Test patch:
Repository: DataBiosphere/toil

Issue:
CI lint fails to run
The [CI lint test](https://ucsc-ci.com/databiosphere/toil/-/jobs/46292) is failing with the errors:
```
src/toil/lib/aws/session.py:40:1: error: Module "botocore.credentials" has no
attribute "JSONFileCache"  [attr-defined]
    from botocore.credentials import JSONFileCache
    ^
src/toil/lib/aws/iam.py:10:1: error: Module "mypy_boto3_iam.type_defs" has no
attribute "AttachedPolicyTypeDef"  [attr-defined]
    from mypy_boto3_iam.type_defs import AttachedPolicyTypeDef
    ^
Found 2 errors in 2 files (checked 67 source files)
```

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1370)
┆Issue Number: TOIL-1370



Discussion:

Resolving patch:
diff --git a/src/toil/lib/aws/iam.py b/src/toil/lib/aws/iam.py
--- a/src/toil/lib/aws/iam.py
+++ b/src/toil/lib/aws/iam.py
@@ -7,7 +7,7 @@
 
 import boto3
 from mypy_boto3_iam import IAMClient
-from mypy_boto3_iam.type_defs import AttachedPolicyTypeDef
+from mypy_boto3_iam.type_defs import AttachedPolicyOutputTypeDef
 from mypy_boto3_sts import STSClient
 
 from toil.lib.aws import zone_to_region
@@ -145,7 +145,7 @@ def get_actions_from_policy_document(policy_doc: Dict[str, Any]) -> AllowedActio
                             allowed_actions[resource][key].append(statement[key])
 
     return allowed_actions
-def allowed_actions_attached(iam: IAMClient, attached_policies: List[AttachedPolicyTypeDef]) -> AllowedActionCollection:
+def allowed_actions_attached(iam: IAMClient, attached_policies: List[AttachedPolicyOutputTypeDef]) -> AllowedActionCollection:
     """
     Go through all attached policy documents and create an AllowedActionCollection representing granted permissions.
 
diff --git a/src/toil/lib/aws/session.py b/src/toil/lib/aws/session.py
--- a/src/toil/lib/aws/session.py
+++ b/src/toil/lib/aws/session.py
@@ -37,7 +37,7 @@
 import botocore
 from boto3 import Session
 from botocore.client import Config
-from botocore.credentials import JSONFileCache
+from botocore.utils import JSONFileCache
 from botocore.session import get_session
 
 logger = logging.getLogger(__name__)


Test patch:
Repository: DataBiosphere/toil

Issue:
If `--batchLogsDir` is set to a nonexistent directory, Slurm jobs all fail
Slurm won't run a job if you tell it to send the output to a file in a nonexistent directory. But Toil doesn't make sure the `--batchLogsDir` directory exists before running Slurm jobs, or even warn about it.

The workaround is to manually make the directory, but Toil ought to do it automatically, on the assumption that it is on a shared filesystem (or at least a place Toil can access on the leader node).

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1423)
┆Issue Number: TOIL-1423



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/abstractBatchSystem.py b/src/toil/batchSystems/abstractBatchSystem.py
--- a/src/toil/batchSystems/abstractBatchSystem.py
+++ b/src/toil/batchSystems/abstractBatchSystem.py
@@ -37,6 +37,7 @@
 from toil.deferred import DeferredFunctionManager
 from toil.fileStores.abstractFileStore import AbstractFileStore
 from toil.job import JobDescription, ParsedRequirement, Requirer
+from toil.lib.memoize import memoize
 from toil.resource import Resource
 
 logger = logging.getLogger(__name__)
@@ -392,6 +393,7 @@ def set_message_bus(self, message_bus: MessageBus) -> None:
         # We do in fact send messages to the message bus.
         self._outbox = message_bus.outbox()
 
+    @memoize
     def get_batch_logs_dir(self) -> str:
         """
         Get the directory where the backing batch system should save its logs.
@@ -404,6 +406,9 @@ def get_batch_logs_dir(self) -> str:
         """
         if self.config.batch_logs_dir:
             # Use what is specified
+            if not os.path.isdir(self.config.batch_logs_dir):
+                # But if it doesn't exist, make it exist
+                os.makedirs(self.config.batch_logs_dir, exist_ok=True)
             return self.config.batch_logs_dir
         # And if nothing is specified use the workDir.
         return Toil.getToilWorkDir(self.config.workDir)


Test patch:
diff --git a/contrib/slurm-test/slurm_test.sh b/contrib/slurm-test/slurm_test.sh
--- a/contrib/slurm-test/slurm_test.sh
+++ b/contrib/slurm-test/slurm_test.sh
@@ -16,7 +16,7 @@ docker exec ${LEADER} sudo apt install python3-pip -y
 docker exec ${LEADER} pip3 install "git+https://github.com/DataBiosphere/toil.git@${GIT_COMMIT}"
 docker exec ${LEADER} sinfo -N -l
 # Test 1: A really basic workflow to check Slurm is working correctly
-docker exec ${LEADER} python3 /home/admin/toil_workflow.py file:my-job-store --batchSystem slurm --disableCaching --retryCount 0
+docker exec ${LEADER} python3 /home/admin/toil_workflow.py file:my-job-store --batchSystem slurm --disableCaching --retryCount 0 --batchLogsDir ./nonexistent/paths
 docker cp ${LEADER}:/home/admin/output.txt output_Docker.txt
 # Test 2: Make sure that "sort" workflow runs under slurm
 docker exec ${LEADER} python3 /home/admin/sort.py file:my-job-store --batchSystem slurm --disableCaching --retryCount 0
Repository: DataBiosphere/toil

Issue:
None of the batch system tests run anymore
The line that actually runs the batch system tests is commented out here:

https://github.com/DataBiosphere/toil/blob/15763af711b2b5d652c0fd9a46dad9dcb7e98feb/.gitlab-ci.yml#L195

We need to run those tests overall even if Mesos is not installed.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1433)
┆Issue Number: TOIL-1433

3.8, 3.9, 3.11 appliance builds failing on CI
For 3.8 and 3.9, I think the updated urllib library is causing dependency conflicts. What works for 3.9 doesn't seem to work with 3.10 and up. It also looks like the 3.11 build is trying to install mesos, which isn't supported.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1434)
┆Issue Number: TOIL-1434



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -49,13 +49,13 @@ def run_setup():
     # to how wheels work, so it is not included in all and
     # must be explicitly installed as an extra
     all_reqs = []
+
     non_htcondor_extras = [
         "aws",
         "cwl",
         "encryption",
         "google",
         "kubernetes",
-        "mesos",
         "wdl",
         "server"
     ]
@@ -64,8 +64,9 @@ def run_setup():
         all_reqs += extras_require[extra]
     # We exclude htcondor from "all" because it can't be on Mac
     extras_require['htcondor:sys_platform!="darwin"'] = get_requirements("htcondor")
+    extras_require['mesos'] = get_requirements("mesos")
+    all_reqs += get_requirements("mesos")
     extras_require["all"] = all_reqs
-
     setup(
         name='toil',
         version=version.distVersion,


Test patch:
Repository: DataBiosphere/toil

Issue:
Toil should stop accounting for CWL's `outdirSize` when using `--bypass-file-store`
If you use `toil-cwl-runner` to run a CWL workflow with `--bypass-file-store`, the `outdirSize` for each task is still included in the Toil job's disk requirement.

https://github.com/DataBiosphere/toil/blob/836944ff7dc934ab4a748652c31c8403dc918734/src/toil/cwl/cwltoil.py#L2271-L2274

But when bypassing the file store, we store all the outputs for all the tasks forever, in storage not controlled by Toil. So each job should then only request the space it needs for temp files, not for temp files plus outputs, assuming the job actually understands [CWL's completely separate temporary and output directories](https://www.commonwl.org/v1.2/CommandLineTool.html#ResourceRequirement).

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1451)
┆Issue Number: TOIL-1451



Discussion:

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -2265,13 +2265,20 @@ def __init__(
                 )
             preemptible = parsed_value
 
+        # We always need space for the temporary files for the job
+        total_disk = cast(int, req["tmpdirSize"]) * (2**20)
+        if not getattr(runtime_context, "bypass_file_store", False):
+            # If using the Toil file store, we also need space for the output
+            # files, which may need to be stored locally and copied off the
+            # node.
+            total_disk += cast(int, req["outdirSize"]) * (2**20)
+        # If not using the Toil file store, output files just go directly to
+        # their final homes their space doesn't need to be accounted per-job.
+
         super().__init__(
             cores=req["cores"],
             memory=int(req["ram"] * (2**20)),
-            disk=int(
-                (cast(int, req["tmpdirSize"]) * (2**20))
-                + (cast(int, req["outdirSize"]) * (2**20))
-            ),
+            disk=int(total_disk),
             accelerators=accelerators,
             preemptible=preemptible,
             tool_id=self.cwltool.tool["id"],


Test patch:
Repository: DataBiosphere/toil

Issue:
toil-wdl-runner does not strip leading whitespace in an evaluated command template
toil-wdl-runner returns IndentationError if common leading whitespace exists within a code block like so:
```wdl
version 1.1

workflow wf {
    input {}
    call t {}
    output {}
}

task t {
    input {}
    command <<<
    python <<CODE
    print("hello")  # indentation error
    CODE
    >>>
    output {}
    runtime {
        docker: "python"
    }
}
```
But the workflow will run if all leading indentation after the `<<CODE` specifier is removed:
```wdl
...
    command <<<
    python <<CODE
print("hello")
CODE
    >>>
...
```
I think the spec mentions a [similar case](https://github.com/openwdl/wdl/blob/main/versions/1.1/SPEC.md#stripping-leading-whitespace). I'm not sure if that workflow is runnable or not as miniwdl doesn't seem to run it, but this seems to be the issue.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1453)
┆Issue Number: TOIL-1453



Discussion:
@stxue1 Can you open a bug against MiniWDL for this? Maybe Mike Lin will helpfully fix it for us.
I think I phrased the miniwdl part wrong. Miniwdl is able to run the example I wrote, but not the example in the spec, which I don't think is a working example. I think the issue might be on Toil's side, but it could be possible for miniwdl to automatically strip whitespace when the command string is passed off to miniwdl.

Resolving patch:
diff --git a/src/toil/wdl/wdltoil.py b/src/toil/wdl/wdltoil.py
--- a/src/toil/wdl/wdltoil.py
+++ b/src/toil/wdl/wdltoil.py
@@ -48,7 +48,7 @@
 import WDL.Error
 import WDL.runtime.config
 from configargparse import ArgParser, SUPPRESS
-from WDL._util import byte_size_units
+from WDL._util import byte_size_units, strip_leading_whitespace
 from WDL.CLI import print_error
 from WDL.runtime.backend.docker_swarm import SwarmContainer
 from WDL.runtime.backend.singularity import SingularityContainer
@@ -1638,9 +1638,12 @@ def patched_run_invocation(*args: Any, **kwargs: Any) -> List[str]:
 
             # Make a new standard library for evaluating the command specifically, which only deals with in-container paths and out-of-container paths.
             command_library = ToilWDLStdLibTaskCommand(file_store, task_container)
+            
+            # Work around wrong types from MiniWDL. See <https://github.com/chanzuckerberg/miniwdl/issues/665>
+            dedent = cast(Callable[[str], Tuple[int, str]], strip_leading_whitespace)
 
             # Work out the command string, and unwrap it
-            command_string: str = evaluate_named_expression(self._task, "command", WDL.Type.String(), self._task.command, contained_bindings, command_library).coerce(WDL.Type.String()).value
+            command_string: str = dedent(evaluate_named_expression(self._task, "command", WDL.Type.String(), self._task.command, contained_bindings, command_library).coerce(WDL.Type.String()).value)[1]
 
             # Grab the standard out and error paths. MyPy complains if we call
             # them because in the current MiniWDL version they are untyped.


Test patch:
Repository: DataBiosphere/toil

Issue:
Options module not found with a built docker image
When I tried to run toil-wdl-runner in an AWS cluster with a custom docker image, I got this:
```
Traceback (most recent call last):
  File "/usr/local/bin/toil-wdl-runner", line 5, in <module>
    from toil.wdl.wdltoil import main
  File "/usr/local/lib/python3.11/dist-packages/toil/wdl/wdltoil.py", line 57, in <module>
    from toil.common import Toil, addOptions, check_and_create_default_config_file
  File "/usr/local/lib/python3.11/dist-packages/toil/common.py", line 56, in <module>
    from toil.options.common import add_base_toil_options, JOBSTORE_HELP
ModuleNotFoundError: No module named 'toil.options'
```
I think the options folder in src/toil never had an `__init__.py` file, so something went wrong in the imports.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1470)
┆Issue Number: TOIL-1470



Discussion:

Resolving patch:
diff --git a/src/toil/options/__init__.py b/src/toil/options/__init__.py
new file mode 100644


Test patch:
Repository: DataBiosphere/toil

Issue:
Assertion error with mutually exclusive argument groups
Suppressed mutually exclusive argument groups triggers an argparse bug: https://github.com/python/cpython/issues/62090. This causes issues when running `--help` on toil-wdl-runner.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1471)
┆Issue Number: TOIL-1471



Discussion:

Resolving patch:
diff --git a/src/toil/options/cwl.py b/src/toil/options/cwl.py
--- a/src/toil/options/cwl.py
+++ b/src/toil/options/cwl.py
@@ -38,7 +38,10 @@ def add_cwl_options(parser: ArgumentParser, suppress: bool = True) -> None:
         default="",
         help=suppress_help or "Log your tools stdout/stderr to this location outside of container",
     )
-    dockergroup = parser.add_mutually_exclusive_group()
+    # this is as a result of suppressed help statements not working well with mutually_exclusive_groups, which will
+    # cause an assertion error
+    # https://github.com/python/cpython/issues/62090
+    dockergroup = parser.add_mutually_exclusive_group() if not suppress_help else parser.add_argument_group()
     dockergroup.add_argument(
         "--user-space-docker-cmd",
         help=suppress_help or "(Linux/OS X only) Specify a user space docker command (like "
@@ -208,7 +211,8 @@ def add_cwl_options(parser: ArgumentParser, suppress: bool = True) -> None:
         default=False,
         help=suppress_help or SUPPRESS,
     )
-    checkgroup = parser.add_mutually_exclusive_group()
+    # same workaround as dockergroup
+    checkgroup = parser.add_mutually_exclusive_group() if not suppress_help else parser.add_argument_group()
     checkgroup.add_argument(
         "--compute-checksum",
         action="store_true",


Test patch:
Repository: DataBiosphere/toil

Issue:
Toil is marking its inputs executable
I think that Toil is marking input files from the local disk executable. It is resulting in weird changes to the file permissions in my Git tree when I run Toil tests.



┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1462)
┆Issue Number: TOIL-1462



Discussion:
➤ Adam Novak commented:

stxue thinks that the caching file store is doing this, and it is happening because we are “mixing up soft links and hard links when it gets the permissions”. Maybe we are getting symlink Christmas tree permissions assigned to normal files? But why to input files?

Resolving patch:
diff --git a/src/toil/fileStores/__init__.py b/src/toil/fileStores/__init__.py
--- a/src/toil/fileStores/__init__.py
+++ b/src/toil/fileStores/__init__.py
@@ -40,7 +40,7 @@ def __init__(self, fileStoreID: str, size: int, executable: bool = False) -> Non
 
     def pack(self) -> str:
         """Pack the FileID into a string so it can be passed through external code."""
-        return f'{self.size}:{int(self.executable)}:{self}'
+        return f'{self.size}:{"1" if self.executable else "0"}:{self}'
 
     @classmethod
     def forPath(cls, fileStoreID: str, filePath: str) -> 'FileID':
@@ -54,7 +54,7 @@ def unpack(cls, packedFileStoreID: str) -> 'FileID':
         vals = packedFileStoreID.split(':', 2)
         # Break up the packed value
         size = int(vals[0])
-        executable = bool(vals[1])
+        executable = (vals[1] == "1")
         value = vals[2]
         # Create the FileID
         return cls(value, size, executable)


Test patch:
diff --git a/src/toil/test/cwl/cwlTest.py b/src/toil/test/cwl/cwlTest.py
--- a/src/toil/test/cwl/cwlTest.py
+++ b/src/toil/test/cwl/cwlTest.py
@@ -17,6 +17,7 @@
 import os
 import re
 import shutil
+import stat
 import subprocess
 import sys
 import unittest
@@ -136,6 +137,8 @@ def run_conformance_tests(
             "--statusWait=10",
             "--retryCount=2",
             "--relax-path-checks",
+            # Defaults to 20s but we can't start hundreds of nodejs processes that fast on our CI potatoes
+            "--eval-timeout=600",
             f"--caching={caching}"
         ]
 
@@ -254,6 +257,13 @@ def _tester(self, cwlfile, jobfile, expect, main_args=[], out_name="output", out
         out.get(out_name, {}).pop("nameroot", None)
         self.assertEqual(out, expect)
 
+        for k, v in expect.items():
+            if isinstance(v, dict) and "class" in v and v["class"] == "File" and "path" in v:
+                # This is a top-level output file.
+                # None of our output files should be executable.
+                self.assertTrue(os.path.exists(v["path"]))
+                self.assertFalse(os.stat(v["path"]).st_mode & stat.S_IXUSR)
+
     def _debug_worker_tester(self, cwlfile, jobfile, expect):
         from toil.cwl import cwltoil