internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataBiosphere/toil

Issue:
Remove mentions of coreOS and replace with flatcar.


┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-787)
┆Issue Number: TOIL-787



Discussion:

Resolving patch:
diff --git a/src/toil/provisioners/abstractProvisioner.py b/src/toil/provisioners/abstractProvisioner.py
--- a/src/toil/provisioners/abstractProvisioner.py
+++ b/src/toil/provisioners/abstractProvisioner.py
@@ -867,7 +867,7 @@ def addKubernetesLeader(self, config: InstanceConfiguration):
             chown $(id -u):$(id -g) $HOME/.kube/config
 
             # Install network
-            kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/{FLANNEL_VERSION}/Documentation/kube-flannel.yml
+            kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/{FLANNEL_VERSION}/Documentation/kube-flannel.yml
 
             # Deploy rubber stamp CSR signing bot
             kubectl apply -f https://raw.githubusercontent.com/kontena/kubelet-rubber-stamp/release/{RUBBER_STAMP_VERSION}/deploy/service_account.yaml
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -566,8 +566,7 @@ def _buildContext(self):
     @memoize
     def _discoverAMI(self) -> str:
         """
-        :return: The AMI ID (a string like 'ami-0a9a5d2b65cce04eb') for CoreOS
-                 or a compatible replacement like Flatcar.
+        :return: The AMI ID (a string like 'ami-0a9a5d2b65cce04eb') for Flatcar.
         :rtype: str
         """
 
@@ -576,9 +575,7 @@ def _discoverAMI(self) -> str:
         if ami is not None:
             return ami
 
-        # CoreOS is dead, long live Flatcar
-
-        # Flatcar images, however, only live for 9 months.
+        # Flatcar images only live for 9 months.
         # Rather than hardcode a list of AMIs by region that will die, we use
         # their JSON feed of the current ones.
         JSON_FEED_URL = 'https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_ami_all.json'


Test patch:
Repository: DataBiosphere/toil

Issue:
Restarting a workflow started on a batch system that supports hot deployment will fail on a batch system that doesn't support hot deployment
https://github.com/DataBiosphere/toil/blob/1f0930b7b9ecc31ca556d15ab07ff836ba85eb23/src/toil/common.py#L979-L994

Nowhere in the restart branch here do we check to see if the batch system we are restarting on supports hot deployment. We just try and hot deploy if there is a user script in the job store.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-826)
┆Issue Number: TOIL-826



Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -978,13 +978,16 @@ def _setupAutoDeployment(self, userScript=None):
                     userScript = None
         else:
             # This branch is hit on restarts
-            from toil.jobStores.abstractJobStore import NoSuchFileException
-            try:
-                with self._jobStore.readSharedFileStream('userScript') as f:
-                    userScript = safeUnpickleFromStream(f)
-            except NoSuchFileException:
-                logger.debug('User script neither set explicitly nor present in the job store.')
-                userScript = None
+            if (self._batchSystem.supportsAutoDeployment() and
+                not self.config.disableAutoDeployment):
+                # We could deploy a user script
+                from toil.jobStores.abstractJobStore import NoSuchFileException
+                try:
+                    with self._jobStore.readSharedFileStream('userScript') as f:
+                        userScript = safeUnpickleFromStream(f)
+                except NoSuchFileException:
+                    logger.debug('User script neither set explicitly nor present in the job store.')
+                    userScript = None
         if userScript is None:
             logger.debug('No user script to auto-deploy.')
         else:


Test patch:
Repository: DataBiosphere/toil

Issue:
Kubernetes batch system shutdown is missing a propagation policy, orphaning pods
https://github.com/DataBiosphere/toil/blob/0eb51d9b50c793733fcd0500f043cdbd1c7bf664/src/toil/batchSystems/kubernetes.py#L1050-L1053

This call doesn't set a Foreground or even Background propagation policy, so pods are left behind running or waiting to run even when their jobs have been deleted by this batch deletion.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-841)
┆Issue Number: TOIL-841



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/kubernetes.py b/src/toil/batchSystems/kubernetes.py
--- a/src/toil/batchSystems/kubernetes.py
+++ b/src/toil/batchSystems/kubernetes.py
@@ -1049,7 +1049,8 @@ def shutdown(self):
         try:
             self._try_kubernetes_expecting_gone(self._api('batch').delete_collection_namespaced_job,
                                                             self.namespace,
-                                                            label_selector="toil_run={}".format(self.runID))
+                                                            label_selector="toil_run={}".format(self.runID),
+                                                            propagation_policy='Background')
             logger.debug('Killed jobs with delete_collection_namespaced_job; cleaned up')
         except ApiException as e:
             if e.status != 404:


Test patch:
Repository: DataBiosphere/toil

Issue:
pip install toil-cwl-runner should work


┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-879)
┆Issue Number: TOIL-879



Discussion:
I made https://pypi.org/project/toil-cwl-runner/ to prevent typo-squatting. Happy to add @DailyDreaming @adamnovak as owners and then you can remove me or demote me to a maintainer. 

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -101,8 +101,8 @@ def run_setup():
         name='toil',
         version=version.distVersion,
         description='Pipeline management software for clusters.',
-        author='Benedict Paten',
-        author_email='benedict@soe.usc.edu',
+        author='Benedict Paten and the Toil community',
+        author_email='toil-community@googlegroups.com',
         url="https://github.com/DataBiosphere/toil",
         classifiers=[
           'Development Status :: 5 - Production/Stable',


Test patch:
Repository: DataBiosphere/toil

Issue:
Add type hints to _nacl.py
Add type hints to src/toil/lib/encryption/_nacl.py so it can be checked under mypy during linting.

Refers to #3568.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-880)
┆Issue Number: TOIL-880



Discussion:

Resolving patch:
diff --git a/contrib/admin/mypy-with-ignore.py b/contrib/admin/mypy-with-ignore.py
--- a/contrib/admin/mypy-with-ignore.py
+++ b/contrib/admin/mypy-with-ignore.py
@@ -8,6 +8,7 @@
 import subprocess
 import sys
 
+os.environ['MYPYPATH'] = 'contrib/typeshed'
 pkg_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))  # noqa
 sys.path.insert(0, pkg_root)  # noqa
 
@@ -114,7 +115,7 @@ def main():
         'src/toil/lib/objects.py',
         'src/toil/lib/io.py',
         'src/toil/lib/docker.py',
-        'src/toil/lib/encryption/_nacl.py',
+        # 'src/toil/lib/encryption/_nacl.py',
         'src/toil/lib/encryption/_dummy.py',
         'src/toil/lib/encryption/conftest.py',
         'src/toil/lib/encryption/__init__.py',
diff --git a/src/toil/lib/encryption/_nacl.py b/src/toil/lib/encryption/_nacl.py
--- a/src/toil/lib/encryption/_nacl.py
+++ b/src/toil/lib/encryption/_nacl.py
@@ -19,7 +19,7 @@
 overhead = 16 + SecretBox.NONCE_SIZE
 
 
-def encrypt(message, keyPath):
+def encrypt(message: bytes, keyPath: str) -> bytes:
     """
     Encrypts a message given a path to a local file containing a key.
 
@@ -57,7 +57,7 @@ def encrypt(message, keyPath):
     return bytes(sb.encrypt(message, nonce))
 
 
-def decrypt(ciphertext, keyPath):
+def decrypt(ciphertext: bytes, keyPath: str) -> bytes:
     """
     Decrypts a given message that was encrypted with the encrypt() method.
 


Test patch:
Repository: DataBiosphere/toil

Issue:
Add type hints to resources.py
Add type hints to src/toil/lib/encryption/resources.py so it can be checked under mypy during linting.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-885)
┆Issue Number: TOIL-885



Discussion:

Resolving patch:
diff --git a/contrib/admin/mypy-with-ignore.py b/contrib/admin/mypy-with-ignore.py
--- a/contrib/admin/mypy-with-ignore.py
+++ b/contrib/admin/mypy-with-ignore.py
@@ -40,13 +40,11 @@ def main():
         'src/toil/wdl/utils.py',
         'src/toil/wdl/wdl_types.py',
         'src/toil/wdl/wdl_synthesis.py',
-        # 'src/toil/wdl/__init__.py',
         'src/toil/wdl/wdl_analysis.py',
         'src/toil/wdl/wdl_functions.py',
         'src/toil/wdl/toilwdl.py',
         'src/toil/wdl/versions/draft2.py',
         'src/toil/wdl/versions/v1.py',
-        # 'src/toil/wdl/versions/__init__.py',
         'src/toil/wdl/versions/dev.py',
         'src/toil/provisioners/clusterScaler.py',
         'src/toil/provisioners/abstractProvisioner.py',
@@ -105,7 +103,6 @@ def main():
         'src/toil/utils/toilClean.py',
         'src/toil/utils/toilLaunchCluster.py',
         'src/toil/lib/memoize.py',
-        'src/toil/lib/resources.py',
         'src/toil/lib/throttle.py',
         'src/toil/lib/humanize.py',
         'src/toil/lib/compatibility.py',
diff --git a/src/toil/lib/resources.py b/src/toil/lib/resources.py
--- a/src/toil/lib/resources.py
+++ b/src/toil/lib/resources.py
@@ -15,10 +15,10 @@
 import os
 import resource
 
-from typing import List
+from typing import List, Tuple
 
 
-def get_total_cpu_time_and_memory_usage():
+def get_total_cpu_time_and_memory_usage() -> Tuple[float, int]:
     """
     Gives the total cpu time of itself and all its children, and the maximum RSS memory usage of
     itself and its single largest child.
@@ -30,7 +30,7 @@ def get_total_cpu_time_and_memory_usage():
     return total_cpu_time, total_memory_usage
 
 
-def get_total_cpu_time():
+def get_total_cpu_time() -> float:
     """Gives the total cpu time, including the children."""
     me = resource.getrusage(resource.RUSAGE_SELF)
     childs = resource.getrusage(resource.RUSAGE_CHILDREN)


Test patch:
Repository: DataBiosphere/toil

Issue:
AWS job store can't clean up a job where the overlargeID is missing
I observed some AWS jobs ready to be deleted that had lost their `overlargeID` fields. This made the job store unable to delete them; it would crash complaining the job was malformed instead.

We should be able to delete even malformed or corrupted jobs.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-886)
┆Issue Number: TOIL-886



Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -250,15 +250,25 @@ def _registered(self, value):
                         registry_domain.put_attributes(item_name=self.namePrefix,
                                                        attributes=attributes)
 
-    def _checkItem(self, item):
+    def _checkItem(self, item, enforce: bool = True):
+        """
+        Make sure that the given SimpleDB item actually has the attributes we think it should.
+        
+        Throw otherwise.
+        
+        If enforce is false, log but don't throw.
+        """
+        
         if "overlargeID" not in item:
-            raise RuntimeError("overlargeID attribute isn't present: you are restarting"
-                               " an old, incompatible jobstore, or the jobstore logic is"
-                               " incorrect.")
+            logger.error("overlargeID attribute isn't present: either SimpleDB entry is "
+                         "corrupt or jobstore is from an extremely old Toil: %s", item)
+            if enforce:
+                raise RuntimeError("encountered SimpleDB entry missing required attribute "
+                                   "'overlargeID'; is your job store ancient?")
 
     def _awsJobFromItem(self, item):
         self._checkItem(item)
-        if item["overlargeID"]:
+        if item.get("overlargeID", None):
             assert self.fileExists(item["overlargeID"])
             # This is an overlarge job, download the actual attributes
             # from the file store
@@ -365,8 +375,11 @@ def delete(self, jobStoreID):
         for attempt in retry_sdb():
             with attempt:
                 item = self.jobsDomain.get_attributes(compat_bytes(jobStoreID), consistent_read=True)
-        self._checkItem(item)
-        if item["overlargeID"]:
+        # If the overlargeID has fallen off, maybe we partially deleted the
+        # attributes of the item? Or raced on it? Or hit SimpleDB being merely
+        # eventually consistent? We should still be able to get rid of it.
+        self._checkItem(item, enforce = False)
+        if item.get("overlargeID", None):
             logger.debug("Deleting job from filestore")
             self.deleteFile(item["overlargeID"])
         for attempt in retry_sdb():


Test patch:
Repository: DataBiosphere/toil

Issue:
Add an override for the stored work dir to make local debugging easier for single jobs
If you want to debug a job with `toil debug-job`, you can get in trouble if the job store specifies a workdir that isn't available on your system.

There should be a way to override the workdir that is stored, for when you want to bring a job from a cluster onto your local machine for debugging.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-887)
┆Issue Number: TOIL-887



Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -1068,7 +1068,7 @@ def getToilWorkDir(configWorkDir: Optional[str] = None) -> str:
         :return: Path to the Toil work directory, constant across all machines
         :rtype: str
         """
-        workDir = configWorkDir or os.getenv('TOIL_WORKDIR') or tempfile.gettempdir()
+        workDir = os.getenv('TOIL_WORKDIR_OVERRIDE') or configWorkDir or os.getenv('TOIL_WORKDIR') or tempfile.gettempdir()
         if not os.path.exists(workDir):
             raise RuntimeError(f'The directory specified by --workDir or TOIL_WORKDIR ({workDir}) does not exist.')
         return workDir


Test patch:
Repository: DataBiosphere/toil

Issue:
Add type hints to conversions.py
Add type hints to src/toil/lib/encryption/conversions so it can be checked under mypy during linting.

Refers to #3568.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-888)
┆Issue Number: TOIL-888



Discussion:

Resolving patch:
diff --git a/contrib/admin/mypy-with-ignore.py b/contrib/admin/mypy-with-ignore.py
--- a/contrib/admin/mypy-with-ignore.py
+++ b/contrib/admin/mypy-with-ignore.py
@@ -109,7 +109,7 @@ def main():
         'src/toil/lib/iterables.py',
         'src/toil/lib/bioio.py',
         'src/toil/lib/ec2.py',
-        'src/toil/lib/conversions.py',
+        # 'src/toil/lib/conversions.py',
         'src/toil/lib/ec2nodes.py',
         'src/toil/lib/misc.py',
         'src/toil/lib/expando.py',
diff --git a/src/toil/lib/conversions.py b/src/toil/lib/conversions.py
--- a/src/toil/lib/conversions.py
+++ b/src/toil/lib/conversions.py
@@ -40,7 +40,7 @@ def bytes_in_unit(unit: str = 'B') -> int:
 
 def convert_units(num: float,
                   src_unit: str,
-                  dst_unit: Optional[str] = 'B') -> float:
+                  dst_unit: str = 'B') -> float:
     """Returns a float representing the converted input in dst_units."""
     assert src_unit.lower() in VALID_PREFIXES, f"{src_unit} not a valid unit, valid units are {VALID_PREFIXES}."
     assert dst_unit.lower() in VALID_PREFIXES, f"{dst_unit} not a valid unit, valid units are {VALID_PREFIXES}."


Test patch:
Repository: DataBiosphere/toil

Issue:
Add type hints to misc.py
Adds type hints to src/toil/lib/encryption/misc.py so it can be checked under mypy during linting.

Refers to #3568.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-889)
┆Issue Number: TOIL-889



Discussion:

Resolving patch:
diff --git a/contrib/admin/mypy-with-ignore.py b/contrib/admin/mypy-with-ignore.py
--- a/contrib/admin/mypy-with-ignore.py
+++ b/contrib/admin/mypy-with-ignore.py
@@ -111,7 +111,7 @@ def main():
         'src/toil/lib/ec2.py',
         'src/toil/lib/conversions.py',
         'src/toil/lib/ec2nodes.py',
-        'src/toil/lib/misc.py',
+        # 'src/toil/lib/misc.py',
         'src/toil/lib/expando.py',
         'src/toil/lib/threading.py',
         'src/toil/lib/exceptions.py',
diff --git a/src/toil/lib/misc.py b/src/toil/lib/misc.py
--- a/src/toil/lib/misc.py
+++ b/src/toil/lib/misc.py
@@ -4,16 +4,19 @@
 import shutil
 import subprocess
 import sys
+import typing
+
+from typing import Iterator, Union, List, Optional
 
 logger = logging.getLogger(__name__)
 
 
-def printq(msg, quiet: bool):
+def printq(msg: str, quiet: bool) -> None:
     if not quiet:
         print(msg)
 
 
-def truncExpBackoff():
+def truncExpBackoff() -> Iterator[float]:
     # as recommended here https://forums.aws.amazon.com/thread.jspa?messageID=406788#406788
     # and here https://cloud.google.com/storage/docs/xml-api/reference-status
     yield 0
@@ -29,7 +32,7 @@ def truncExpBackoff():
 class CalledProcessErrorStderr(subprocess.CalledProcessError):
     """Version of CalledProcessError that include stderr in the error message if it is set"""
 
-    def __str__(self):
+    def __str__(self) -> str:
         if (self.returncode < 0) or (self.stderr is None):
             return str(super())
         else:
@@ -37,7 +40,8 @@ def __str__(self):
             return "Command '%s' exit status %d: %s" % (self.cmd, self.returncode, err)
 
 
-def call_command(cmd, *, input=None, timeout=None, useCLocale=True, env=None):
+def call_command(cmd: List[str], *args: str, input: Optional[str] = None, timeout: Optional[float] = None,
+                useCLocale: bool = True, env: Optional[typing.Dict[str, str]] = None) -> Union[str, bytes]:
     """Simplified calling of external commands.  This always returns
     stdout and uses utf- encode8.  If process fails, CalledProcessErrorStderr
     is raised.  The captured stderr is always printed, regardless of


Test patch:
Repository: DataBiosphere/toil

Issue:
Add type hints to toilMain.py
Add type hints to src/toil/utils/toilMain.py so it can be checked under mypy during linting.

Refers to #3568.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-891)
┆Issue Number: TOIL-891



Discussion:

Resolving patch:
diff --git a/contrib/admin/mypy-with-ignore.py b/contrib/admin/mypy-with-ignore.py
--- a/contrib/admin/mypy-with-ignore.py
+++ b/contrib/admin/mypy-with-ignore.py
@@ -93,7 +93,6 @@ def main():
         'src/toil/utils/toilDebugFile.py',
         'src/toil/utils/toilStatus.py',
         'src/toil/utils/toilStats.py',
-        'src/toil/utils/toilMain.py',
         'src/toil/utils/__init__.py',
         'src/toil/utils/toilLaunchCluster.py',
         'src/toil/lib/memoize.py',
@@ -103,9 +102,7 @@ def main():
         'src/toil/lib/iterables.py',
         'src/toil/lib/bioio.py',
         'src/toil/lib/ec2.py',
-        # 'src/toil/lib/conversions.py',
         'src/toil/lib/ec2nodes.py',
-        # 'src/toil/lib/misc.py',
         'src/toil/lib/expando.py',
         'src/toil/lib/threading.py',
         'src/toil/lib/exceptions.py',
diff --git a/src/toil/utils/toilMain.py b/src/toil/utils/toilMain.py
--- a/src/toil/utils/toilMain.py
+++ b/src/toil/utils/toilMain.py
@@ -2,13 +2,14 @@
 import re
 import sys
 import textwrap
+import types
 
 import pkg_resources
 
 from toil.version import version
+from typing import Any, Dict
 
-
-def main():
+def main() -> None:
     modules = loadModules()
 
     if len(sys.argv) < 2 or sys.argv[1] == '--help':
@@ -27,9 +28,20 @@ def main():
         sys.exit(1)
 
     del sys.argv[1]
-    module.main()
+    get_or_die(module, 'main')()
+
+def get_or_die(module: types.ModuleType, name: str) -> Any:
+    """
+    Get an object from a module or complain that it is missing.
+    """
+    if hasattr(module, name):
+        return getattr(module, name)
+    else:
+        sys.stderr.write(f'Internal Toil error!\nToil utility module '
+                         f'{module.__name__} is missing required attribute {name}\n')
+        sys.exit(1)
 
-def loadModules():
+def loadModules() -> Dict[str, types.ModuleType]:
     # noinspection PyUnresolvedReferences
     from toil.utils import (toilClean,
                             toilDebugFile,
@@ -44,9 +56,9 @@ def loadModules():
     return {"-".join([i.lower() for i in re.findall('[A-Z][^A-Z]*', name)]): module for name, module in locals().items()}
 
 
-def printHelp(modules):
+def printHelp(modules: Dict[str, types.ModuleType]) -> None:
     name = os.path.basename(sys.argv[0])
-    descriptions = '\n        '.join(f'{cmd} - {mod.__doc__.strip()}' for cmd, mod in modules.items() if mod)
+    descriptions = '\n        '.join(f'{cmd} - {get_or_die(mod, "__doc__").strip()}' for cmd, mod in modules.items() if mod)
     print(textwrap.dedent(f"""
         Usage: {name} COMMAND ...
                {name} --help
@@ -58,7 +70,7 @@ def printHelp(modules):
         """[1:]))
 
 
-def printVersion():
+def printVersion() -> None:
     try:
         print(pkg_resources.get_distribution('toil').version)
     except:


Test patch:
Repository: DataBiosphere/toil

Issue:
Add type hints to toilKill.py
Add type hints to src/toil/utils/toilKill.py so it can be checked under mypy during linting.

Refers to #3568.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-892)
┆Issue Number: TOIL-892



Discussion:

Resolving patch:
diff --git a/contrib/admin/mypy-with-ignore.py b/contrib/admin/mypy-with-ignore.py
--- a/contrib/admin/mypy-with-ignore.py
+++ b/contrib/admin/mypy-with-ignore.py
@@ -95,7 +95,6 @@ def main():
         'src/toil/utils/toilStats.py',
         'src/toil/utils/toilSshCluster.py',
         'src/toil/utils/toilMain.py',
-        'src/toil/utils/toilKill.py',
         'src/toil/utils/__init__.py',
         'src/toil/utils/toilDestroyCluster.py',
         'src/toil/utils/toilDebugJob.py',
diff --git a/src/toil/utils/toilKill.py b/src/toil/utils/toilKill.py
--- a/src/toil/utils/toilKill.py
+++ b/src/toil/utils/toilKill.py
@@ -22,7 +22,7 @@
 logger = logging.getLogger(__name__)
 
 
-def main():
+def main() -> None:
     parser = parser_with_common_options()
     options = parser.parse_args()
     set_logging_from_options(options)


Test patch: