internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataBiosphere/toil

Issue:
Debian deleted the deb we need
Looks like we may have made a mistake trying to hackily install Debian's Singularity packages after all. They increased their glibc version:

```
#10 3.075 dpkg: singularity-container: dependency problems, but configuring anyway as you requested:
#10 3.075  singularity-container depends on libc6 (>= 2.33); however:
#10 3.075   Version of libc6:amd64 on system is 2.23-0ubuntu11.3.
```

But Ubuntu hasn't, so now their binary won't start:

```
#10 3.125 /usr/local/libexec/toil/singularity-real: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/local/libexec/toil/singularity-real)
```

We'd need to be on Ubuntu 21.04 or later to have glibc 2.33: https://packages.ubuntu.com/hirsute/libc6

I think we need to dig up the old deb files and maybe just host them ourselves, or find a stable archive of the particular Debian version they belong to.

_Originally posted by @adamnovak in https://github.com/DataBiosphere/toil/issues/3971#issuecomment-1012434048_

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-1129)
┆friendlyId: TOIL-1129



Discussion:

Resolving patch:
diff --git a/docker/Dockerfile.py b/docker/Dockerfile.py
--- a/docker/Dockerfile.py
+++ b/docker/Dockerfile.py
@@ -103,10 +103,11 @@ def heredoc(s):
         apt-get clean && \
         rm -rf /var/lib/apt/lists/*
 
-    # Install Singularity from a newer Debian.
+    # Install a particular old Debian Sid Singularity from somewhere.
     # The dependencies it thinks it needs aren't really needed and aren't
     # available here.
-    RUN wget https://debian.osuosl.org/debian/pool/main/s/singularity-container/$(curl -sSL 'https://debian.osuosl.org/debian/pool/main/s/singularity-container/' | grep -o "singularity-container_3[^\\"]*$(if [ $TARGETARCH = amd64 ] ; then echo amd64 ; else echo arm64 ; fi).deb" | head -n1) && \
+    ADD singularity-sources.tsv /etc/singularity/singularity-sources.tsv
+    RUN wget "$(cat /etc/singularity/singularity-sources.tsv | grep "^$TARGETARCH" | cut -f3)" && \
         (dpkg -i singularity-container_3*.deb || true) && \
         dpkg --force-depends --configure -a && \
         sed -i 's/containernetworking-plugins, //' /var/lib/dpkg/status && \


Test patch:
Repository: DataBiosphere/toil

Issue:
We're getting the new Werkzeug without asking for it
CI for the merge https://github.com/DataBiosphere/toil/commit/0e256d63cb974a87b8f6b807bf7d23bc9a12fb76 failed at the lint stage, because the merge commit ends up installing a different Werkzeug than the PR's test run did, and the new one has type hints, which upsets MyPy because we now have an unused ignore.

This is because the `connexion` devs finally got access to the `connexion` PyPI package again, and published the current release there. So we started picking up connexion 2.10 instead of 2.5, which is now compatible with Flask 2. So we started installing Flask 2 and Werkzeug 2.

If we're going to import out of Werkzeug, we need to depend on a particular major version of it, so it can't be changed put from under us by pip.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-1130)
┆friendlyId: TOIL-1130



Discussion:

Resolving patch:
diff --git a/src/toil/server/wes/abstract_backend.py b/src/toil/server/wes/abstract_backend.py
--- a/src/toil/server/wes/abstract_backend.py
+++ b/src/toil/server/wes/abstract_backend.py
@@ -9,7 +9,7 @@
 from urllib.parse import urldefrag
 
 import connexion  # type: ignore
-from werkzeug.utils import secure_filename  # type: ignore
+from werkzeug.utils import secure_filename
 
 logger = logging.getLogger(__name__)
 


Test patch:
Repository: DataBiosphere/toil

Issue:
Any reason why "futures" is a requirement?
It's listed in [requirements-aws.txt](https://github.com/DataBiosphere/toil/blob/95982020dd263ed2b3ad5c41cd99fee19a6bdb30/requirements-aws.txt#L4)
```
futures>=3.1.1, <4
```
And seems to be behind this [Cactus CI error](https://ucsc-ci.com/comparativegenomicstoolkit/cactus/-/jobs/8808) that I'm getting all of the sudden (via Cactus's [Dockerfile](https://github.com/ComparativeGenomicsToolkit/cactus/blob/4259ad72fce9c1e81c5c7f13b2dd3bb8619eb4dc/Dockerfile#L46-L48)):
```
ERROR: Could not find a version that satisfies the requirement futures<4,>=3.1.1; extra == "all" (from toil[all]) (from versions: 0.2.python3, 0.1, 0.2, 1.0, 2.0, 2.1, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.2.0, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.1.0, 3.1.1)
```
Where it seems that pip just stopped wanting to make wheels of it altogether.  I'm guessing this may have something to do with [futures](https://pypi.org/project/futures/) being python 2-specific?

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1139)
┆friendlyId: TOIL-1139



Discussion:

Resolving patch:
diff --git a/src/toil/lib/aws/utils.py b/src/toil/lib/aws/utils.py
--- a/src/toil/lib/aws/utils.py
+++ b/src/toil/lib/aws/utils.py
@@ -76,9 +76,7 @@ def delete_iam_instance_profile(
     iam_resource = cast(IAMServiceResource, session.resource("iam", region_name=region))
     instance_profile = iam_resource.InstanceProfile(instance_profile_name)
     if instance_profile.roles is not None:
-        # The type stubs think this is somehow just one role, when it's a
-        # collection. See <https://github.com/vemel/mypy_boto3_builder/issues/122>.
-        for role in instance_profile.roles: #type: ignore
+        for role in instance_profile.roles:
             printq(f'Now dissociating role: {role.name} from instance profile {instance_profile_name}', quiet)
             instance_profile.remove_role(RoleName=role.name)
     instance_profile.delete()


Test patch:
Repository: DataBiosphere/toil

Issue:
WES: Example `docker-compose` file for minimum viable Internet deployment
Add example to the docs for securely opening WES to the Internet. 

> ... we could also show a compose file with [Traefik](https://github.com/traefik/traefik#overview) in front of the Toil server, doing [SSL with LetsEncrypt like this](https://doc.traefik.io/traefik/user-guides/docker-compose/acme-http/) and [HTTP Basic Authentication like this](https://jensknipper.de/blog/basic-authentication-with-traefik/), to protect it with a username and password, which would be a minimum viable Internet deployment.

(https://github.com/DataBiosphere/toil/pull/3779#discussion_r732953473)

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1066)
┆friendlyId: TOIL-1066



Discussion:

Resolving patch:
diff --git a/src/toil/server/app.py b/src/toil/server/app.py
--- a/src/toil/server/app.py
+++ b/src/toil/server/app.py
@@ -76,9 +76,7 @@ def create_app(args: argparse.Namespace) -> "connexion.FlaskApp":
         CORS(flask_app.app, resources={r"/ga4gh/*": {"origins": args.cors_origins}})
 
     # add workflow execution service (WES) API endpoints
-    backend = ToilBackend(work_dir=args.work_dir,
-                          options=args.opt,
-                          base_url=f"http://{get_public_ip()}:{args.port}")
+    backend = ToilBackend(work_dir=args.work_dir, options=args.opt)
 
     flask_app.add_api('workflow_execution_service.swagger.yaml',
                       resolver=connexion.Resolver(backend.resolve_operation_id))  # noqa
diff --git a/src/toil/server/wes/toil_backend.py b/src/toil/server/wes/toil_backend.py
--- a/src/toil/server/wes/toil_backend.py
+++ b/src/toil/server/wes/toil_backend.py
@@ -21,6 +21,7 @@
 from typing import Optional, List, Dict, Any, overload, Generator, Tuple
 
 from flask import send_from_directory
+from flask.globals import request as flask_request
 from werkzeug.utils import redirect
 from werkzeug.wrappers.response import Response
 
@@ -128,10 +129,9 @@ class is responsible for validating and executing submitted workflows.
     in the filesystem to store and retrieve data associated with the runs.
     """
 
-    def __init__(self, work_dir: str, options: List[str], base_url: str) -> None:
+    def __init__(self, work_dir: str, options: List[str]) -> None:
         super(ToilBackend, self).__init__(options)
-        self.work_dir = work_dir
-        self.base_url = base_url
+        self.work_dir = os.path.abspath(work_dir)
 
         self.supported_versions = {
             "py": ["3.6", "3.7", "3.8", "3.9"],
@@ -273,8 +273,8 @@ def get_run_log(self, run_id: str) -> Dict[str, Any]:
         stdout = ""
         stderr = ""
         if os.path.isfile(os.path.join(run.work_dir, 'stdout')):
-            stdout = f"{self.base_url}/toil/wes/v1/logs/{run_id}/stdout"
-            stderr = f"{self.base_url}/toil/wes/v1/logs/{run_id}/stderr"
+            stdout = f"{flask_request.host_url}toil/wes/v1/logs/{run_id}/stdout"
+            stderr = f"{flask_request.host_url}toil/wes/v1/logs/{run_id}/stderr"
 
         exit_code = run.fetch("exit_code")
 


Test patch:
Repository: DataBiosphere/toil

Issue:
Boto3 typing has been fixed and so CI no longer passes
The type stubs for Boto3 have been updated, fixing some bugs we were working around. This makes the workarounds fail MyPy.

```
/builds/databiosphere/toil/contrib/admin/mypy-with-ignore.py
src/toil/lib/aws/utils.py:252: error: Unused "type: ignore" comment
            info = s3_client.head_bucket(Bucket=bucket_name)  # type: igno...
            ^
src/toil/lib/aws/utils.py: note: In function "get_bucket_region":
src/toil/lib/aws/utils.py:253:16: error: Redundant cast to "str" 
[redundant-cast]
            return cast(str, info['ResponseMetadata']['HTTPHeaders']['x-am...
                   ^
Found 2 errors in 1 file (checked 59 source files)
```

We should fix the MyPy type checking, and pin versions of the stubs so this can't happen again to formerly-working commits.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1176)
┆friendlyId: TOIL-1176



Discussion:

Resolving patch:
diff --git a/src/toil/lib/aws/utils.py b/src/toil/lib/aws/utils.py
--- a/src/toil/lib/aws/utils.py
+++ b/src/toil/lib/aws/utils.py
@@ -248,9 +248,8 @@ def attempt_head_bucket() -> Optional[str]:
         # for an 'x-amz-bucket-region' header on the response, which can tell
         # us where the bucket is. See
         # <https://github.com/aws/aws-sdk-cpp/issues/844#issuecomment-383747871>
-        # This won't typecheck until https://github.com/youtype/mypy_boto3_builder/issues/147 is fixed.
-        info = s3_client.head_bucket(Bucket=bucket_name)  # type: ignore
-        return cast(str, info['ResponseMetadata']['HTTPHeaders']['x-amz-bucket-region'])
+        info = s3_client.head_bucket(Bucket=bucket_name)
+        return info['ResponseMetadata']['HTTPHeaders']['x-amz-bucket-region']
 
     # Compose a list of strategies we want to try in order, which may work.
     # None is an acceptable return type that actually means something.


Test patch:
Repository: DataBiosphere/toil

Issue:
Performance testing for Graviton instance


┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-973)
┆epic: Graviton2 Summer 2021 Project
┆friendlyId: TOIL-973



Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -1472,7 +1472,7 @@ def __init__(self, provisioner: Optional["AbstractProvisioner"] = None) -> None:
                      "-v", "/proc:/host/proc",
                      "-v", "/sys:/host/sys",
                      "-v", "/:/rootfs",
-                     "quay.io/prometheus/node-exporter:0.15.2",
+                     "quay.io/prometheus/node-exporter:v1.3.1",
                      "-collector.procfs", "/host/proc",
                      "-collector.sysfs", "/host/sys",
                      "-collector.filesystem.ignored-mount-points",
diff --git a/src/toil/provisioners/abstractProvisioner.py b/src/toil/provisioners/abstractProvisioner.py
--- a/src/toil/provisioners/abstractProvisioner.py
+++ b/src/toil/provisioners/abstractProvisioner.py
@@ -704,7 +704,7 @@ def addNodeExporterService(self, config: InstanceConfiguration):
                 -v /:/rootfs \\
                 --name node-exporter \\
                 --restart always \\
-                quay.io/prometheus/node-exporter:v0.15.2 \\
+                quay.io/prometheus/node-exporter:v1.3.1 \\
                 --path.procfs /host/proc \\
                 --path.sysfs /host/sys \\
                 --collector.filesystem.ignored-mount-points ^/(sys|proc|dev|host|etc)($|/)


Test patch:
Repository: DataBiosphere/toil

Issue:
Add Python 3.10 testing.


┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1043)
┆friendlyId: TOIL-1043



Discussion:
Gonna just wait for python packages to work better with 3.10 so checking back in a month

Resolving patch:
diff --git a/docker/Dockerfile.py b/docker/Dockerfile.py
--- a/docker/Dockerfile.py
+++ b/docker/Dockerfile.py
@@ -31,7 +31,7 @@
                          'python3.7-distutils' if python == 'python3.7' else '',
                          'python3.8-distutils' if python == 'python3.8' else '',
                          'python3.9-distutils' if python == 'python3.9' else '',
-                         # 'python3.9-venv' if python == 'python3.9' else '',
+                         'python3.10-distutils' if python == 'python3.10' else '',
                          'python3-pip',
                          'libssl-dev',
                          'wget',
@@ -139,7 +139,7 @@ def heredoc(s):
     RUN chmod 777 /usr/bin/waitForKey.sh && chmod 777 /usr/bin/customDockerInit.sh && chmod 777 /usr/local/bin/singularity
 
     # The stock pip is too old and can't install from sdist with extras
-    RUN {pip} install --upgrade pip==21.3.1
+    RUN curl -sS https://bootstrap.pypa.io/get-pip.py | {python}
 
     # Default setuptools is too old
     RUN {pip} install --upgrade setuptools==59.7.0
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -84,6 +84,7 @@ def run_setup():
           'Programming Language :: Python :: 3.7',
           'Programming Language :: Python :: 3.8',
           'Programming Language :: Python :: 3.9',
+          'Programming Language :: Python :: 3.10',
           'Topic :: Scientific/Engineering',
           'Topic :: Scientific/Engineering :: Bio-Informatics',
           'Topic :: Scientific/Engineering :: Astronomy',


Test patch:
Repository: DataBiosphere/toil

Issue:
make `--export=ALL` the default for TOIL_SLURM_ARGS?
Or at least document that it may be required 

From https://gitter.im/bd2k-genomics-toil/Lobby?at=619398fecd4972068bac9921

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1086)
┆friendlyId: TOIL-1086



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/slurm.py b/src/toil/batchSystems/slurm.py
--- a/src/toil/batchSystems/slurm.py
+++ b/src/toil/batchSystems/slurm.py
@@ -272,6 +272,23 @@ def prepareSbatch(self,
             if job_environment:
                 environment.update(job_environment)
 
+            # "Native extensions" for SLURM (see DRMAA or SAGA)
+            nativeConfig = os.getenv('TOIL_SLURM_ARGS')
+
+            # --export=[ALL,]<environment_toil_variables>
+            set_exports = "--export=ALL"
+
+            if nativeConfig is not None:
+                logger.debug("Native SLURM options appended to sbatch from TOIL_SLURM_ARGS env. variable: %s", nativeConfig)
+
+                for arg in nativeConfig.split():
+                    if arg.startswith("--mem") or arg.startswith("--cpus-per-task"):
+                        raise ValueError(f"Some resource arguments are incompatible: {nativeConfig}")
+                    # repleace default behaviour by the one stated at TOIL_SLURM_ARGS
+                    if arg.startswith("--export"):
+                        set_exports = arg
+                sbatch_line.extend(nativeConfig.split())
+
             if environment:
                 argList = []
 
@@ -279,7 +296,10 @@ def prepareSbatch(self,
                     quoted_value = quote(os.environ[k] if v is None else v)
                     argList.append(f'{k}={quoted_value}')
 
-                sbatch_line.append('--export=' + ','.join(argList))
+                set_exports += ',' + ','.join(argList)
+
+            # add --export to the sbatch
+            sbatch_line.append(set_exports)
 
             parallel_env = os.getenv('TOIL_SLURM_PE')
             if cpu and cpu > 1 and parallel_env:
@@ -295,15 +315,6 @@ def prepareSbatch(self,
             stderrfile: str = self.boss.formatStdOutErrPath(jobID, '%j', 'err')
             sbatch_line.extend(['-o', stdoutfile, '-e', stderrfile])
 
-            # "Native extensions" for SLURM (see DRMAA or SAGA)
-            nativeConfig = os.getenv('TOIL_SLURM_ARGS')
-            if nativeConfig is not None:
-                logger.debug("Native SLURM options appended to sbatch from TOIL_SLURM_ARGS env. variable: %s", nativeConfig)
-                if ("--mem" in nativeConfig) or ("--cpus-per-task" in nativeConfig):
-                    raise ValueError(f"Some resource arguments are incompatible: {nativeConfig}")
-
-                sbatch_line.extend(nativeConfig.split())
-
             return sbatch_line
 
         def parse_elapsed(self, elapsed):


Test patch:
Repository: DataBiosphere/toil

Issue:
Private IP support no longer works (presumably code regression issue)
Private IP support was supposedly fixed for issue [2659](https://github.com/DataBiosphere/toil/pull/2569) but due to likely code regression it no longer [works](https://github.com/DataBiosphere/toil/blob/master/src/toil/provisioners/aws/awsProvisioner.py#L391). Likely fix is modify line 391 in awsProvisioner.py to check for private_ip_address in addition to public_ip_address (i.e. only fail if both are None).

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1244)
┆Issue Number: TOIL-1244



Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -389,7 +389,11 @@ def launchCluster(self,
         leader.reload()
 
         if leader.public_ip_address is None:
-            raise RuntimeError("AWS did not assign a public IP to the cluster leader! Leader is lost!")
+            # Sometimes AWS just fails to assign a public IP when we really need one.
+            # But sometimes people intend to use private IPs only in Toil-managed clusters.
+            # TODO: detect if we have a route to the private IP and fail fast if not.
+            logger.warning("AWS did not assign a public IP to the cluster leader. If you aren't "
+                           "connected to the private subnet, cluster setup will fail!")
 
         # Remember enough about the leader to let us launch workers in its
         # cluster.


Test patch:
Repository: DataBiosphere/toil

Issue:
Please include /usr/bin/time -v in Toil appliance
This has been on my wishlist for cactus for a while, and maybe if the appliance is being changed for #4282 this would be a good time?

Cactus has an option to run things with `/usr/bin/time -v` then to write the peak memory usage in the logs.  This works everywhere but on toil appliances.  

I think it's just a matter of `apt install time` when making the image, which doesn't cost too much space
```
After this operation, 109 kB of additional disk space will be used.
```

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1248)
┆Issue Number: TOIL-1248



Discussion:

Resolving patch:
diff --git a/docker/Dockerfile.py b/docker/Dockerfile.py
--- a/docker/Dockerfile.py
+++ b/docker/Dockerfile.py
@@ -53,6 +53,7 @@
                          'less',
                          'vim',
                          'git',
+                         'time',
                          # Dependencies for Mesos which the deb doesn't actually list
                          'libsvn1',
                          'libcurl4-nss-dev',


Test patch:
Repository: DataBiosphere/toil

Issue:
Deleting Kubernetes jobs when batch delete fails
Right now we try to claen up Kubernetes jobs when Toil shuts down with a batch delete:

https://github.com/DataBiosphere/toil/blob/98dbf33147ed029800dd43a73ee5c64e83feda7b/src/toil/batchSystems/kubernetes.py#L979

If that fails (for example, because we don't have permission to do batch deletes), we go through all the *pods*, inspect them, and try to delete them:

https://github.com/DataBiosphere/toil/blob/98dbf33147ed029800dd43a73ee5c64e83feda7b/src/toil/batchSystems/kubernetes.py#L1002

We should be deleting the *jobs* individually, and not the pods. If there are any remaining jobs, deleting the pod will leave the job around. Hopefully the job will be in a failed state because the pod died, but I'm not sure the job won't immediately try to recreate the missing pod.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-769)
┆Issue Number: TOIL-769



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/kubernetes.py b/src/toil/batchSystems/kubernetes.py
--- a/src/toil/batchSystems/kubernetes.py
+++ b/src/toil/batchSystems/kubernetes.py
@@ -21,12 +21,10 @@
 Docker
 """
 import datetime
-import functools
 import logging
 import math
 import os
 import string
-import subprocess
 import sys
 import tempfile
 import time
@@ -41,9 +39,7 @@
     Iterator,
     List,
     Literal,
-    NamedTuple,
     Optional,
-    Sequence,
     Tuple,
     Type,
     TypeVar,
@@ -114,7 +110,6 @@
 from toil.lib.misc import get_user_name, slow_down, utc_now
 from toil.lib.retry import ErrorCondition, retry
 from toil.resource import Resource
-from toil.statsAndLogging import configure_root_logger, set_log_level
 
 logger = logging.getLogger(__name__)
 retryable_kubernetes_errors: List[Union[Type[Exception], ErrorCondition]] = [
@@ -1512,7 +1507,18 @@ def shutdown(self) -> None:
                 # Anything other than a 404 is weird here.
                 logger.error("Exception when calling BatchV1Api->delete_collection_namespaced_job: %s" % e)
 
-            # aggregate all pods and check if any pod has failed to cleanup or is orphaned.
+            # If batch delete fails, try to delete all remaining jobs individually.
+            logger.debug('Deleting Kubernetes jobs individually for toil_run=%s', self.run_id)
+            for jobID in self._getIssuedNonLocalBatchJobIDs():
+                jobName = f'{self.job_prefix}{jobID}'
+                logger.debug(f'Deleting Kubernetes job {jobName}')
+                self._api('batch', errors=[]).delete_namespaced_job(
+                    jobName,
+                    self.namespace,
+                    propagation_policy='Background'
+                )
+
+            # Aggregate all pods and check if any pod has failed to cleanup or is orphaned.
             ourPods = self._ourPodObject()
 
             for pod in ourPods:


Test patch: