Repository: DataBiosphere/toil

Issue:
HTCondor was never updated with environment support, causing "ValueError: too many values to unpack (expected 6)", and also can't handle the HTCondor schedd being busy
Following up from https://github.com/DataBiosphere/toil/issues/4315, I managed to successfully submit a Cactus job to HTCondor, using the following job descriptor:

```
Executable = /data/share/cactus_conda/bin/cactus
Arguments = " ./jobStore genomes.txt genomes.hal --logFile cactus.log --realTimeLogging --batchSystem htcondor --consCores 32 --disableCaching "
Log = cactus.$(process).log
Output = cactus.$(process).out
Error = cactus.$(process).err
request_cpus = 1
Queue 1
```

However, this failed with the errors from the title. The relevant output is below:

```
[2023-02-04T08:26:06+0100] [MainThread] [W] [toil.common] Batch system does not support auto-deployment. The user script ModuleDescriptor(dirPath='/data/share/cactus_conda/lib/python3.10/site-packages', name='cactus.progressive.cactus_progressive', fromVirtualEnv=False) will have to be present at the same location on every worker.
[2023-02-04T08:26:06+0100] [MainThread] [I] [toil.job] Saving graph of 1 jobs, 1 new
[2023-02-04T08:26:06+0100] [MainThread] [I] [toil.job] Processing job 'progressive_workflow' kind-progressive_workflow/instance-fgnhd28s v0
[2023-02-04T08:26:06+0100] [MainThread] [I] [toil] Running Toil version 5.9.0-8155e0a981f4d728762a7cbc920b7ed544fe4ae7 on host exec-02.novalocal.
[2023-02-04T08:26:06+0100] [MainThread] [I] [toil.realtimeLogger] Starting real-time logging.
[2023-02-04T08:26:06+0100] [MainThread] [I] [toil.leader] Issued job 'progressive_workflow' kind-progressive_workflow/instance-fgnhd28s v1 with job batch system ID: 0 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2023-02-04T08:26:07+0100] [Thread-2  ] [E] [toil.batchSystems.abstractGridEngineBatchSystem] GridEngine like batch system failure
Traceback (most recent call last):
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 257, in run
    while self._runStep():
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 246, in _runStep
    activity |= self.createJobs(newJob)
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/htcondor.py", line 45, in createJobs
    jobID, cpu, memory, disk, jobName, command = self.waitingJobs.pop(0)
ValueError: too many values to unpack (expected 6)
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/data/share/cactus_conda/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 257, in run
    while self._runStep():
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 246, in _runStep
    activity |= self.createJobs(newJob)
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/htcondor.py", line 45, in createJobs
    jobID, cpu, memory, disk, jobName, command = self.waitingJobs.pop(0)
ValueError: too many values to unpack (expected 6)
/data/share/cactus_conda/lib/python3.10/site-packages/htcondor/_deprecation.py:119: FutureWarning: The "requirements" argument of xquery() will be renamed to "constraint" in a future release. For now, it will accept both names.
  warnings.warn(
[2023-02-04T08:26:08+0100] [MainThread] [I] [toil.leader] 0 jobs are running, 1 jobs are issued and waiting to run
[2023-02-04T09:26:09+0100] [MainThread] [I] [toil.leader] 0 jobs are running, 1 jobs are issued and waiting to run
```

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1284)
┆Issue Number: TOIL-1284



Discussion:
An update: Looks like the error comes from here: https://github.com/DataBiosphere/toil/blob/master/src/toil/batchSystems/htcondor.py#L45

I added the following lines before the line mentioned above:

```
                temp = self.waitingJobs.pop(0)
                logger.warning("Type of temp: %s", type(temp))
                logger.warning("Length of temp: %r", len(temp))
                logger.warning("Value of temp: %s", temp)
```

and this was the output:

```
[2023-02-07T12:20:51+0100] [Thread-2  ] [W] [toil.batchSystems.htcondor] Type of temp: <class 'tuple'>
[2023-02-07T12:20:51+0100] [Thread-2  ] [W] [toil.batchSystems.htcondor] Length of temp: 7
[2023-02-07T12:20:51+0100] [Thread-2  ] [W] [toil.batchSystems.htcondor] Value of temp: (0, 1, 2147483648, 2147483648, 'progressive_workflow', '_toil_worker progressive_workflow file:/data/share/Cipri-Ilx-NGS/de-novo/cactus-RO-POL-ASP-PAR__STAR_HTCondor/jobStore kind-progressive_workflow/instance-buwtqt6i --context gASVzgAAAAAAAACMIXRvaWwuYmF0Y2hTeXN0ZW1zLmNsZWFudXBfc3VwcG9ydJSMFFdvcmtlckNsZWFudXBDb250ZXh0lJOUKYGUfZSMEXdvcmtlckNsZWFudXBJbmZvlIwldG9pbC5iYXRjaFN5c3RlbXMuYWJzdHJhY3RCYXRjaFN5c3RlbZSMEVdvcmtlckNsZWFudXBJbmZvlJOUKE5OjCRiMzlmNmY0Yi03Y2U3LTRlZmMtOTE3Ni1hZmY2MmZkZjhmMzSUjAZhbHdheXOUdJSBlHNiLg==', {'OMP_NUM_THREADS': '1'})
```

It looks like after the command string there is an unexpected 7th element with the value `{'OMP_NUM_THREADS': '1'}`. 

Another update: simply including a seventh variable e.g. `omp_threads` into the list that is [being created here](https://github.com/DataBiosphere/toil/blob/master/src/toil/batchSystems/htcondor.py#L45) got rid of the problem mentioned above.

Now, the job crashes due to a new error: `htcondor.HTCondorIOError: Failed to connect to schedd`. A bit of googling revealed an older discussion that seems to imply this issue is related to multithreading:

https://www-auth.cs.wisc.edu/lists/htcondor-users/2018-January/msg00118.shtml

This issue is on a level above what I'm confidently able to deal with. @adamnovak or @Hexotical could you offer any suggestions on how to deal with this?

Below is the full stack trace after I enabled debug logs in Toil:

```
[2023-02-08T08:46:44+0100] [MainThread] [I] [toil.leader] Issued job 'progressive_workflow' kind-progressive_workflow/instance-_704rlh6 v1 with job batch system ID: 0 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2023-02-08T08:46:44+0100] [MainThread] [D] [toil.bus] Notifying toil.bus.JobIssuedMessage with message: JobIssuedMessage(job_type='progressive_workflow', job_id='kind-progressive_workflow/instance-_704rlh6', toil_batch_id=0)
[2023-02-08T08:46:44+0100] [MainThread] [D] [toil.bus] Notifying toil.bus.QueueSizeMessage with message: QueueSizeMessage(queue_size=1)
[2023-02-08T08:46:45+0100] [Thread-2  ] [D] [toil.batchSystems.abstractGridEngineBatchSystem] No activity, sleeping for 1s
[2023-02-08T08:46:45+0100] [Thread-2  ] [D] [toil.batchSystems.htcondor] Submitting 'executable = /bin/sh\ntransfer_executable = False\narguments = "-c \'_toil_worker progressive_workflow file:/data/share/Cipri-Ilx-NGS/de-novo/cactus-RO-POL-ASP-PAR__STAR_HTCondor/jobStore kind-progressive_workflow/instance-_704rlh6 --context gASVzgAAAAAAAACMIXRvaWwuYmF0Y2hTeXN0ZW1zLmNsZWFudXBfc3VwcG9ydJSMFFdvcmtlckNsZWFudXBDb250ZXh0lJOUKYGUfZSMEXdvcmtlckNsZWFudXBJbmZvlIwldG9pbC5iYXRjaFN5c3RlbXMuYWJzdHJhY3RCYXRjaFN5c3RlbZSMEVdvcmtlckNsZWFudXBJbmZvlJOUKE5OjCQyMjM4MjEyZC1jMzQ5LTQ1ZjEtYTQxNy0xZDg1ZmZkOTk5ZjeUjAZhbHdheXOUdJSBlHNiLg==\'"\nenvironment = "TOIL_RT_LOGGING_ADDRESS=\'10.252.254.166:2836\' TOIL_RT_LOGGING_LEVEL=\'Debug\'"\ngetenv = True\nshould_transfer_files = Yes\noutput = /var/lib/condor/execute/dir_34978/toil_2238212d-c349-45f1-a417-1d85ffd999f7.0.$(cluster).out.log\nerror = /var/lib/condor/execute/dir_34978/toil_2238212d-c349-45f1-a417-1d85ffd999f7.0.$(cluster).err.log\nlog = /var/lib/condor/execute/dir_34978/toil_2238212d-c349-45f1-a417-1d85ffd999f7.0.$(cluster).events.log\nrequest_cpus = 1\nrequest_memory = 2097152.000KB\nrequest_disk = 2097152.000KB\nleave_in_queue = (JobStatus == 4)\nMY.IsToilJob = True\nMY.ToilJobID = 0\nMY.ToilJobName = "progressive_workflow"\nMY.ToilJobKilled = False\n'
[2023-02-08T08:46:45+0100] [Thread-2  ] [D] [toil.batchSystems.htcondor] Connecting to HTCondor Schedd on local machine
[2023-02-08T08:46:46+0100] [MainThread] [D] [toil.batchSystems.htcondor] Connecting to HTCondor Schedd on local machine
/data/share/cactus_conda/lib/python3.10/site-packages/htcondor/_deprecation.py:119: FutureWarning: The "requirements" argument of xquery() will be renamed to "constraint" in a future release. For now, it will accept both names.
  warnings.warn(
[2023-02-08T08:46:50+0100] [Thread-2  ] [E] [toil.batchSystems.abstractGridEngineBatchSystem] GridEngine like batch system failure
Traceback (most recent call last):
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 257, in run
    while self._runStep():
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 246, in _runStep
    activity |= self.createJobs(newJob)
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/htcondor.py", line 52, in createJobs
    batchJobID = self.submitJob(submitObj)
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/htcondor.py", line 123, in submitJob
    with schedd.transaction() as txn:
  File "/data/share/cactus_conda/lib/python3.10/site-packages/htcondor/_lock.py", line 70, in wrapper
    rv = func(*args, **kwargs)
htcondor.HTCondorIOError: Failed to connect to schedd.
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/data/share/cactus_conda/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 257, in run
    while self._runStep():
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 246, in _runStep
    activity |= self.createJobs(newJob)
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/htcondor.py", line 52, in createJobs
    batchJobID = self.submitJob(submitObj)
  File "/data/share/cactus_conda/lib/python3.10/site-packages/toil/batchSystems/htcondor.py", line 123, in submitJob
    with schedd.transaction() as txn:
  File "/data/share/cactus_conda/lib/python3.10/site-packages/htcondor/_lock.py", line 70, in wrapper
[2023-02-08T08:46:50+0100] [MainThread] [I] [toil.leader] 0 jobs are running, 1 jobs are issued and waiting to run
    rv = func(*args, **kwargs)
htcondor.HTCondorIOError: Failed to connect to schedd.
[2023-02-08T08:47:16+0100] [MainThread] [D] [toil.batchSystems.htcondor] Connecting to HTCondor Schedd on local machine
```

The problem with the new last item looks to be that, in https://github.com/DataBiosphere/toil/commit/81ca29c797bb802e57088b02717cf6695d9bdc2f#diff-0f2719dd6171193d6a36d8c3f0ec35139002dfc70cd665f9e3e6b83d3ec0ded2R110, we changed the type of the argument to `createJobs`, but we never made the corresponding change in the HTCondor batch system code. And either we have turned off type checking for that file, or the `Any` type in the argument means we didn't catch that this change breaks the HTCondor batch system.

As for the threading, it does indeed look like we have 2 threads that want to talk to htcondor at the same time. It looks like if one thread is doing something, the connection from the other thread might fail, which we don't handle.

But I'm not sure that that that is really the cause of the HTCondorIOError here; it looks like *neither* thread is managing to even connect, and I think we only have 2 threads here (the main one and the one shipping new jobs to HTCondor). Maybe doing some backoff or locking around the connect calls would help anyway?
Hi @adamnovak, thanks for looking into this. I tried [the following changes](https://github.com/DataBiosphere/toil/compare/master...amizeranschi:toil:master), but these had no effect on the error mentioned above (`Failed to connect to schedd`). 


Resolving patch:
diff --git a/src/toil/batchSystems/abstractGridEngineBatchSystem.py b/src/toil/batchSystems/abstractGridEngineBatchSystem.py
--- a/src/toil/batchSystems/abstractGridEngineBatchSystem.py
+++ b/src/toil/batchSystems/abstractGridEngineBatchSystem.py
@@ -17,23 +17,34 @@
 from datetime import datetime
 from queue import Empty, Queue
 from threading import Lock, Thread
-from typing import Any, Dict, List, Optional, Union
+from typing import Any, Dict, List, Optional, Tuple, Union
 
 from toil.batchSystems.abstractBatchSystem import (BatchJobExitReason,
                                                    UpdatedBatchJobInfo)
 from toil.batchSystems.cleanup_support import BatchSystemCleanupSupport
 from toil.bus import ExternalBatchIdMessage
+from toil.job import JobDescription, AcceleratorRequirement
 from toil.lib.misc import CalledProcessErrorStderr
 
 logger = logging.getLogger(__name__)
 
 
+# Internally we throw around these flat tuples of random important things about a job.
+# Assigned ID
+# Required cores
+# Required memory
+# Command to run
+# Unit name of the job
+# Environment dict for the job
+# Accelerator requirements for the job
+JobTuple = Tuple[int, float, int, str, str, Dict[str, str], List[AcceleratorRequirement]]
+
 class AbstractGridEngineBatchSystem(BatchSystemCleanupSupport):
     """
     A partial implementation of BatchSystemSupport for batch systems run on a
     standard HPC cluster. By default auto-deployment is not implemented.
     """
-
+    
     class Worker(Thread, metaclass=ABCMeta):
 
         def __init__(self, newJobsQueue: Queue, updatedJobsQueue: Queue, killQueue: Queue, killedJobsQueue: Queue, boss: 'AbstractGridEngineBatchSystem') -> None:
@@ -57,7 +68,7 @@ def __init__(self, newJobsQueue: Queue, updatedJobsQueue: Queue, killQueue: Queu
             self.updatedJobsQueue = updatedJobsQueue
             self.killQueue = killQueue
             self.killedJobsQueue = killedJobsQueue
-            self.waitingJobs: List[Any] = list()
+            self.waitingJobs: List[JobTuple] = list()
             self.runningJobs = set()
             self.runningJobsLock = Lock()
             self.batchJobIDs = dict()
@@ -92,13 +103,11 @@ def forgetJob(self, jobID):
                 self.runningJobs.remove(jobID)
             del self.batchJobIDs[jobID]
 
-        def createJobs(self, newJob: Any) -> bool:
+        def createJobs(self, newJob: JobTuple) -> bool:
             """
-            Create a new job with the Toil job ID.
+            Create a new job with the given attributes.
 
             Implementation-specific; called by AbstractGridEngineWorker.run()
-
-            :param string newJob: Toil job ID
             """
             activity = False
             # Load new job id if present:
diff --git a/src/toil/batchSystems/htcondor.py b/src/toil/batchSystems/htcondor.py
--- a/src/toil/batchSystems/htcondor.py
+++ b/src/toil/batchSystems/htcondor.py
@@ -16,15 +16,35 @@
 import math
 import os
 import time
-from typing import Any, Dict, Optional
+from contextlib import contextmanager
+from threading import Lock
+from typing import Any, Dict, List, Optional, Tuple
 
 import htcondor
 
 from toil.batchSystems.abstractGridEngineBatchSystem import \
     AbstractGridEngineBatchSystem
 
+from toil.job import AcceleratorRequirement
+from toil.lib.retry import retry
+
 logger = logging.getLogger(__name__)
 
+# Internally we throw around these flat tuples of random important things about a job. These are *different* than the AbstractGridEngineBatchSystem ones!
+# MyPy can't deal with that but we will deal with MyPy later.
+#
+# Assigned ID
+# Required *whole* cores
+# Required memory
+# *Required disk*
+# *Unit name of the job* (swapped with command)
+# *Command to run* (swapped with unit name)
+# Environment dict for the job
+# Accelerator requirements for the job
+JobTuple = Tuple[int, int, int, int, str, str, Dict[str, str], List[AcceleratorRequirement]]
+
+# We have one global lock to control access to the HTCondor scheduler
+schedd_lock = Lock()
 
 class HTCondorBatchSystem(AbstractGridEngineBatchSystem):
     # When using HTCondor, the Schedd handles scheduling
@@ -33,7 +53,7 @@ class Worker(AbstractGridEngineBatchSystem.Worker):
 
         # Override the createJobs method so that we can use htcondor.Submit objects
         # and so that we can get disk allocation requests and ceil the CPU request.
-        def createJobs(self, newJob: Any) -> bool:
+        def createJobs(self, newJob: JobTuple) -> bool:
             activity = False
 
             if newJob is not None:
@@ -42,10 +62,13 @@ def createJobs(self, newJob: Any) -> bool:
             # Queue jobs as necessary:
             while len(self.waitingJobs) > 0 and len(self.runningJobs) < int(self.boss.config.maxLocalJobs):
                 activity = True
-                jobID, cpu, memory, disk, jobName, command = self.waitingJobs.pop(0)
+                jobID, cpu, memory, disk, jobName, command, environment, accelerators = self.waitingJobs.pop(0)
+
+                if accelerators:
+                    logger.warning('Scheduling job %s without enforcing accelerator requirement', jobID)
 
                 # Prepare the htcondor.Submit object
-                submitObj: htcondor.Submit = self.prepareSubmission(cpu, memory, disk, jobID, jobName, command)
+                submitObj: htcondor.Submit = self.prepareSubmission(cpu, memory, disk, jobID, jobName, command, environment)
                 logger.debug("Submitting %r", submitObj)
 
                 # Submit job and get batch system ID (i.e. the ClusterId)
@@ -63,7 +86,8 @@ def createJobs(self, newJob: Any) -> bool:
 
             return activity
 
-        def prepareSubmission(self, cpu: int, memory: int, disk: int, jobID: int, jobName: str, command: str) -> htcondor.Submit:
+        def prepareSubmission(self, cpu: int, memory: int, disk: int, jobID: int, jobName: str, command: str, environment: Dict[str, str]) -> htcondor.Submit:
+            # Note that we don't yet take the accelerators here.
 
             # Convert resource requests
             cpu = int(math.ceil(cpu)) # integer CPUs
@@ -87,8 +111,8 @@ def prepareSubmission(self, cpu: int, memory: int, disk: int, jobID: int, jobNam
             submit_parameters = {
                 'executable': '/bin/sh',
                 'transfer_executable': 'False',
-                'arguments': f'''"-c '{command}'"'''.encode(),    # Workaround for HTCondor Python bindings Unicode conversion bug
-                'environment': self.getEnvString(),
+                'arguments': f'''"-c '{self.duplicate_quotes(command)}'"'''.encode(),    # Workaround for HTCondor Python bindings Unicode conversion bug
+                'environment': self.getEnvString(environment),
                 'getenv': 'True',
                 'should_transfer_files': 'Yes',   # See note above for stdoutfile, stderrfile
                 'output': stdoutfile,
@@ -119,12 +143,13 @@ def prepareSubmission(self, cpu: int, memory: int, disk: int, jobID: int, jobNam
             # Return the Submit object
             return htcondor.Submit(submit_parameters)
 
+        @retry(errors=[htcondor.HTCondorIOError])
         def submitJob(self, submitObj):
 
             # Queue the job using a Schedd transaction
-            schedd = self.connectSchedd()
-            with schedd.transaction() as txn:
-                batchJobID = submitObj.queue(txn)
+            with self.connectSchedd() as schedd:
+                with schedd.transaction() as txn:
+                    batchJobID = submitObj.queue(txn)
 
             # Return the ClusterId
             return batchJobID
@@ -134,9 +159,9 @@ def getRunningJobIDs(self):
             # Get all Toil jobs that are running
             requirements = '(JobStatus == 2) && (IsToilJob)'
             projection = ['ClusterId', 'ToilJobID', 'EnteredCurrentStatus']
-            schedd = self.connectSchedd()
-            ads = schedd.xquery(requirements = requirements,
-                                    projection = projection)
+            with self.connectSchedd() as schedd:
+                ads = schedd.xquery(requirements = requirements,
+                                        projection = projection)
 
             # Only consider the Toil jobs that are part of this workflow
             batchJobIDs = [batchJobID for (batchJobID, task) in self.batchJobIDs.values()]
@@ -158,9 +183,9 @@ def killJob(self, jobID):
             logger.debug(f"Killing HTCondor job {batchJobID}")
 
             # Set the job to be killed when its exit status is checked
-            schedd = self.connectSchedd()
-            job_spec = f'(ClusterId == {batchJobID})'
-            schedd.edit(job_spec, 'ToilJobKilled', 'True')
+            with self.connectSchedd() as schedd:
+                job_spec = f'(ClusterId == {batchJobID})'
+                schedd.edit(job_spec, 'ToilJobKilled', 'True')
 
         def getJobExitCode(self, batchJobID):
             logger.debug(f"Getting exit code for HTCondor job {batchJobID}")
@@ -179,74 +204,114 @@ def getJobExitCode(self, batchJobID):
             projection = ['JobStatus', 'ToilJobKilled', 'ExitCode',
                               'HoldReason', 'HoldReasonSubCode']
 
-            schedd = self.connectSchedd()
-            ads = schedd.xquery(requirements = requirements,  projection = projection)
+            with self.connectSchedd() as schedd:
+                ads = schedd.xquery(requirements = requirements,  projection = projection)
 
-            # Make sure a ClassAd was returned
-            try:
+                # Make sure a ClassAd was returned
                 try:
-                    ad = next(ads)
-                except TypeError:
-                    ad = ads.next()
-            except StopIteration:
-                logger.error(
-                    f"No HTCondor ads returned using constraint: {requirements}")
-                raise
-
-            # Make sure only one ClassAd was returned
-            try:
+                    try:
+                        ad = next(ads)
+                    except TypeError:
+                        ad = ads.next()
+                except StopIteration:
+                    logger.error(
+                        f"No HTCondor ads returned using constraint: {requirements}")
+                    raise
+
+                # Make sure only one ClassAd was returned
                 try:
-                    next(ads)
-                except TypeError:
-                    ads.next()
-            except StopIteration:
-                pass
-            else:
-                logger.warning(
-                    f"Multiple HTCondor ads returned using constraint: {requirements}")
+                    try:
+                        next(ads)
+                    except TypeError:
+                        ads.next()
+                except StopIteration:
+                    pass
+                else:
+                    logger.warning(
+                        f"Multiple HTCondor ads returned using constraint: {requirements}")
 
-            if ad['ToilJobKilled']:
-                logger.debug(f"HTCondor job {batchJobID} was killed by Toil")
+                if ad['ToilJobKilled']:
+                    logger.debug(f"HTCondor job {batchJobID} was killed by Toil")
 
-                # Remove the job from the Schedd and return 1
-                job_spec = f'ClusterId == {batchJobID}'
-                schedd.act(htcondor.JobAction.Remove, job_spec)
-                return 1
+                    # Remove the job from the Schedd and return 1
+                    job_spec = f'ClusterId == {batchJobID}'
+                    schedd.act(htcondor.JobAction.Remove, job_spec)
+                    return 1
 
-            elif status[ad['JobStatus']] == 'Completed':
-                logger.debug("HTCondor job {} completed with exit code {}".format(
-                    batchJobID, ad['ExitCode']))
+                elif status[ad['JobStatus']] == 'Completed':
+                    logger.debug("HTCondor job {} completed with exit code {}".format(
+                        batchJobID, ad['ExitCode']))
 
-                # Remove the job from the Schedd and return its exit code
-                job_spec = f'ClusterId == {batchJobID}'
-                schedd.act(htcondor.JobAction.Remove, job_spec)
-                return int(ad['ExitCode'])
+                    # Remove the job from the Schedd and return its exit code
+                    job_spec = f'ClusterId == {batchJobID}'
+                    schedd.act(htcondor.JobAction.Remove, job_spec)
+                    return int(ad['ExitCode'])
 
-            elif status[ad['JobStatus']] == 'Held':
-                logger.error("HTCondor job {} was held: '{} (sub code {})'".format(
-                    batchJobID, ad['HoldReason'], ad['HoldReasonSubCode']))
+                elif status[ad['JobStatus']] == 'Held':
+                    logger.error("HTCondor job {} was held: '{} (sub code {})'".format(
+                        batchJobID, ad['HoldReason'], ad['HoldReasonSubCode']))
 
-                # Remove the job from the Schedd and return 1
-                job_spec = f'ClusterId == {batchJobID}'
-                schedd.act(htcondor.JobAction.Remove, job_spec)
-                return 1
+                    # Remove the job from the Schedd and return 1
+                    job_spec = f'ClusterId == {batchJobID}'
+                    schedd.act(htcondor.JobAction.Remove, job_spec)
+                    return 1
 
-            else: # Job still running or idle or doing something else
-                logger.debug("HTCondor job {} has not completed (Status: {})".format(
-                    batchJobID, status[ad['JobStatus']]))
-                return None
+                else: # Job still running or idle or doing something else
+                    logger.debug("HTCondor job {} has not completed (Status: {})".format(
+                        batchJobID, status[ad['JobStatus']]))
+                    return None
 
 
         """
         Implementation-specific helper methods
         """
 
+        @contextmanager
         def connectSchedd(self):
-            '''Connect to HTCondor Schedd and return a Schedd object'''
+            """
+            Connect to HTCondor Schedd and yield a Schedd object.
+
+            You can only use it inside the context.
+            Handles locking to make sure that only one thread is trying to do this at a time.
+            """
+
+            # Find the scheduler
+            schedd_ad = self._get_schedd_address()
+
+            with schedd_lock:
+                schedd = self._open_schedd_connection(schedd_ad)
+
+                # Ping the Schedd to make sure it's there and responding
+                try:
+                    self._ping_scheduler(schedd)
+                except RuntimeError:
+                    logger.error("Could not connect to HTCondor Schedd")
+                    raise
+
+                yield schedd
+
+        @retry(errors=[RuntimeError])
+        def _ping_scheduler(self, schedd: Any) -> None:
+            """
+            Ping the scheduler, or fail if it persistently cannot be contacted.
+            """
+            schedd.xquery(limit = 0)
+
+        @retry(errors=[htcondor.HTCondorIOError])
+        def _get_schedd_address(self) -> Optional[str]:
+            """
+            Get the HTCondor scheduler to connect to, or None for the local machine.
+
+            Retries if necessary.
+            """
+            # TODO: Memoize? Or is the collector meant to field every request?
 
             condor_host = os.getenv('TOIL_HTCONDOR_COLLECTOR')
             schedd_name = os.getenv('TOIL_HTCONDOR_SCHEDD')
 
+            # Get the scheduler's address, if not local
+            schedd_ad: Optional[str] = None
+
             # If TOIL_HTCONDOR_ variables are set, use them to find the Schedd
             if condor_host and schedd_name:
                 logger.debug(
@@ -263,24 +328,39 @@ def connectSchedd(self):
                     logger.error(
                         f"Could not find HTCondor Schedd with name {schedd_name}")
                     raise
-                else:
-                    schedd = htcondor.Schedd(schedd_ad)
-
-            # Otherwise assume the Schedd is on the local machine
             else:
+                # Otherwise assume the Schedd is on the local machine
                 logger.debug("Connecting to HTCondor Schedd on local machine")
-                schedd = htcondor.Schedd()
 
-            # Ping the Schedd to make sure it's there and responding
-            try:
-                schedd.xquery(limit = 0)
-            except RuntimeError:
-                logger.error("Could not connect to HTCondor Schedd")
-                raise
+            return schedd_ad
+
+        @retry(errors=[htcondor.HTCondorIOError])
+        def _open_schedd_connection(self, schedd_ad: Optional[str] = None) -> Any:
+            """
+            Open a connection to the htcondor schedd.
 
-            return schedd
+            Assumes that the necessary lock is already held. Retries if needed.
+
+            :param schedd_ad: Where the scheduler is. If not set, use the local machine.
+            """
+
+            # TODO: Don't hold the lock while retrying?
+
+            if schedd_ad is not None:
+                return htcondor.Schedd(schedd_ad)
+            else:
+                return htcondor.Schedd()
+
+        def duplicate_quotes(self, value: str) -> str:
+            """
+            Escape a string by doubling up all single and double quotes.
+
+            This is used for arguments we pass to htcondor that need to be
+            inside both double and single quote enclosures.
+            """
+            return value.replace("'", "''").replace('"', '""')
 
-        def getEnvString(self):
+        def getEnvString(self, overrides: Dict[str, str]) -> str:
             """
             Build an environment string that a HTCondor Submit object can use.
 
@@ -289,15 +369,17 @@ def getEnvString(self):
             """
 
             env_items = []
-            if self.boss.environment:
-                for key, value in self.boss.environment.items():
+            if self.boss.environment or overrides:
+                combined = dict(self.boss.environment)
+                combined.update(overrides)
+                for key, value in combined.items():
 
                     # Each variable should be in the form of <key>='<value>'
                     env_string = key + "="
 
                     # The entire value should be encapsulated in single quotes
                     # Quote marks (single or double) that are part of the value should be duplicated
-                    env_string += "'" + value.replace("'", "''").replace('"', '""') + "'"
+                    env_string += "'" + self.duplicate_quotes(value) + "'"
 
                     env_items.append(env_string)
 
@@ -316,8 +398,8 @@ def issueBatchJob(self, jobNode, job_environment: Optional[Dict[str, str]] = Non
             jobID = self.getNextJobID()
             self.currentJobs.add(jobID)
 
-            # Add the jobNode.disk and jobNode.jobName to the job tuple
+            # Construct our style of job tuple
             self.newJobsQueue.put((jobID, jobNode.cores, jobNode.memory, jobNode.disk, jobNode.jobName, jobNode.command,
-                                   job_environment))
+                                   job_environment or {}, jobNode.accelerators))
             logger.debug("Issued the job command: %s with job id: %s ", jobNode.command, str(jobID))
         return jobID


Test patch:
diff --git a/src/toil/test/batchSystems/batchSystemTest.py b/src/toil/test/batchSystems/batchSystemTest.py
--- a/src/toil/test/batchSystems/batchSystemTest.py
+++ b/src/toil/test/batchSystems/batchSystemTest.py
@@ -537,7 +537,7 @@ def test_preemptability_constraints(self):
          'toleration_seconds': None,
          'value': 'true'}]
         """).strip(), str(spot_spec.tolerations))
-        
+
     def test_label_constraints(self):
         """
         Make sure we generate the right preemptability constraints.