internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataBiosphere/toil

Issue:
Stop warning about `XDG_RUNTIME_DIR`
Our Slurm cluster will give you a nonexistent `XDG_RUNTIME_DIR` and isn’t going to stop. But the warning shows up in everyone’s logs, and whenever anyone has a problem they mention the warning to me, when it’s not plausibly the cause of their problem. So we should stop warning because it is always a red herring.



┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1454)
┆Issue Number: TOIL-1454



Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -1263,12 +1263,6 @@ def get_toil_coordination_dir(cls, config_work_dir: Optional[str], config_coordi
                  deleted.
         """
 
-        if 'XDG_RUNTIME_DIR' in os.environ and not os.path.exists(os.environ['XDG_RUNTIME_DIR']):
-            # Slurm has been observed providing this variable but not keeping
-            # the directory live as long as we run for.
-            logger.warning('XDG_RUNTIME_DIR is set to nonexistent directory %s; your environment may be out of spec!',
-                           os.environ['XDG_RUNTIME_DIR'])
-
         # Go get a coordination directory, using a lot of short-circuiting of
         # or and the fact that and returns its second argument when it
         # succeeds.


Test patch:
Repository: DataBiosphere/toil

Issue:
maxLogFileSize is too small and gets hit in practice
The default `--maxLogFileSize` is only 64,000 bytes. If jobs try to mount thousands of files into a container, and fail because of it, we might get a log that's all the list of the files the failed job tried to access, without the actual error message.

We should increase the default limit up to what would actually start to pose a problem to hold in memory or ship around on disk (megabytes or hundreds of megabytes).

https://github.com/DataBiosphere/toil/blob/de0c0053d38cb8cd0efc04607e36f23cbdb5e620/src/toil/options/common.py#L597

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1494)
┆Issue Number: TOIL-1494



Discussion:

Resolving patch:
diff --git a/src/toil/options/common.py b/src/toil/options/common.py
--- a/src/toil/options/common.py
+++ b/src/toil/options/common.py
@@ -594,12 +594,12 @@ def __call__(self, parser: Any, namespace: Any, values: Any, option_string: Any
         title="Toil log management options.",
         description="Options for how Toil should manage its logs."
     )
-    log_options.add_argument("--maxLogFileSize", dest="maxLogFileSize", default=64000, type=h2b,
+    log_options.add_argument("--maxLogFileSize", dest="maxLogFileSize", default=100 * 1024 * 1024, type=h2b,
                              action=make_open_interval_action(1),
                              help=f"The maximum size of a job log file to keep (in bytes), log files larger than "
                                   f"this will be truncated to the last X bytes. Setting this option to zero will "
                                   f"prevent any truncation. Setting this option to a negative value will truncate "
-                                  f"from the beginning.  Default={bytes2human(64000)}")
+                                  f"from the beginning.  Default={bytes2human(100 * 1024 * 1024)}")
     log_options.add_argument("--writeLogs", dest="writeLogs", nargs='?', action='store', default=None,
                              const=os.getcwd(), metavar="OPT_PATH",
                              help="Write worker logs received by the leader into their own files at the specified "


Test patch:
Repository: DataBiosphere/toil

Issue:
Add more debug logging to single machine batch system
I've been informed that the single machine batch system and file job store, used together, have been observed taking a minute or two between issuing a job and seeing the message from the resulting worker about redirecting its logging. This is happening at the scale of tens of jobs. It's apparently intermittent, which to me suggests a slow filesystem, but it would be nice to know for sure. It could be that there's too much synchronization and contention around the single machine daddy thread, which does all its thinking by using the issued job queue as a delay line.

It might be filesystem-slowness-related or it might not, but we don't currently have debug logging at e.g. the point where the child process is launched, or the point where the worker first starts running, to see where the time is going.



┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-718)
┆Issue Number: TOIL-718



Discussion:
Just to confirm, the problem of slowness to handover the logging was certainly the cluster filesystem.  Thanks, @adamnovak 
 for the help. However, indeed, a debug logging on this level at Toil would facilitate the investigation where the time is going.

For instance the 2-minutes of no info below:

```
[2020-10-30T16:24:17+0000] [MainThread] [I] [toil.leader] Issued job 'Job' kind-Job/instance-mksb3x09 with job batch system ID: 64 and cores: 0, disk: 100.0 M, and memory: 512.0 M

[2020-10-30T16:26:47+0000] [MainThread] [D] [toil.common] Obtained node ID d7499de5-16c9-4f1d-81b6-6c7a0b8ff866 from file /proc/sys/kernel/random/boot_id
```

Resolving patch:
diff --git a/src/toil/batchSystems/singleMachine.py b/src/toil/batchSystems/singleMachine.py
--- a/src/toil/batchSystems/singleMachine.py
+++ b/src/toil/batchSystems/singleMachine.py
@@ -655,6 +655,7 @@ def _startChild(self, jobCommand, jobID, coreFractions, jobMemory, jobDisk, job_
             # and all its children together. We assume that the
             # process group ID will equal the PID of the process we
             # are starting.
+            logger.debug("Attempting to run job command: %s", jobCommand)
             popen = subprocess.Popen(jobCommand,
                                      shell=True,
                                      env=child_environment,
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -159,6 +159,8 @@ def workerScript(
     if config.colored_logs:
         install_log_color()
 
+    logger.debug("Worker started for job %s...", job_name)
+
     ##########################################
     #Create the worker killer, if requested
     ##########################################
@@ -749,7 +751,6 @@ def in_contexts(contexts: List[str]) -> Iterator[None]:
 def main(argv: Optional[List[str]] = None) -> None:
     if argv is None:
         argv = sys.argv
-
     # Parse our command line
     options = parse_args(argv)
 


Test patch:
Repository: DataBiosphere/toil

Issue:
Caching isn't able to be enabled on Slurm
Following on from #2551, we should now be able to turn on caching on Slurm if we want to, but right now it is locked out, I think because we don't have the cleanup support base class inherited in.

We should change it so that caching can be turned on on Slurm, but defaults to off. Then we can speed test it and think about #4673.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1489)
┆Issue Number: TOIL-1489



Discussion:
This might be a duplicate of #928.

Resolving patch:
diff --git a/src/toil/batchSystems/abstractGridEngineBatchSystem.py b/src/toil/batchSystems/abstractGridEngineBatchSystem.py
--- a/src/toil/batchSystems/abstractGridEngineBatchSystem.py
+++ b/src/toil/batchSystems/abstractGridEngineBatchSystem.py
@@ -391,10 +391,6 @@ def __init__(self, config, maxCores, maxMemory, maxDisk):
         self._getRunningBatchJobIDsTimestamp = None
         self._getRunningBatchJobIDsCache = {}
 
-    @classmethod
-    def supportsWorkerCleanup(cls):
-        return False
-
     @classmethod
     def supportsAutoDeployment(cls):
         return False


Test patch:
Repository: DataBiosphere/toil

Issue:
Toil WDL input file is supposed to be `--inputs` in the examples in the docs, but `--input` is actually implemented
Right now on https://toil.readthedocs.io/en/latest/wdl/running.html in the example of moving a Cromwell command to Toil, we show the Cromwell `--inputs` option being used with Toil. But Toil actually names this option `--input` and can't currently accept `--inputs`.

We should fix the documentation to always use `--input` with Toil, and adjust the implementation to tolerate `--inputs`.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1557)
┆Issue Number: TOIL-1557



Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -713,7 +713,7 @@ def check_arguments(typ: str) -> None:
                             help="WDL document URI")
         parser.add_argument("inputs_uri", type=str, nargs='?',
                             help="WDL input JSON URI")
-        parser.add_argument("--input", "-i", dest="inputs_uri", type=str,
+        parser.add_argument("--input", "--inputs", "-i", dest="inputs_uri", type=str,
                             help="WDL input JSON URI")
         check_arguments(typ="wdl")
 


Test patch:
Repository: DataBiosphere/toil

Issue:
Coordination directory moves with the work directory when that's not advisable
For a few reasons, on our Slurm cluster, Toil is putting the coordination directory at the location of the work directory, since it can't use any of the available tmpfs locations.

But then when a user has a workflow that can't fit its scratch in node-local scratch, and moves the work directory to our Ceph share, the coordination directory goes with it. Then the aLastProcessStandingArena will hammer Ceph with O(n^2) file locking attempts in rapid succession as workers try and see if they are the last one alive on their node, and this can hang the Ceph MDS, since the Ceph MDS apparently isn't *quite* correct in its internal locking setup, and in any case isn't designed for this workload.

The workaround is to set `TOIL_COORDINATION_DIR=/data/tmp` whenever you set the work directory onto shared storage.

The fix is to not make the coordination directory follow the work directory, and to work harder to keep it local to the node. We should maybe just use `/tmp` if it's there, even if it's not technically the selected temp directory.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1543)
┆Issue Number: TOIL-1543



Discussion:

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -1306,6 +1306,9 @@ def get_toil_coordination_dir(cls, config_work_dir: Optional[str], config_coordi
                     os.path.join(os.environ['XDG_RUNTIME_DIR'], 'toil'))) or
                 # Try under /run/lock. It might be a temp dir style sticky directory.
                 try_path('/run/lock') or
+                # Before trying the workdir, try the /tmp directory as tmp directories
+                # should be local to the node and not shared like a work directoy might be
+                try_path('/tmp/toil_coordination') or
                 # Finally, fall back on the work dir and hope it's a legit filesystem.
                 cls.getToilWorkDir(config_work_dir)
         )


Test patch:
Repository: boto/boto

Issue:
RDS call modify_dbinstance with multi_az = True doesn't actually set an instance to MultiAZ
Making a call to a non-multiaz instance with multi_az=True doesn't actually switch the parameter. I assume this is also true for creating one from scratch, but I haven't tested that yet.



Discussion:
Are you specifying "apply_immediately=True" when you call modify_dbinstance?

I am, but even if I wasn't, shouldn't that at least be marked as a pending value? In the console, when I mark a single az rds instance multiaz without apply immediately, it's added as a pending value. 

Is there some kind of debugging output that would be helpful? Maybe I could make the call with the loglevel at DEBUG and there's some output in there that would help?

Try doing the following:

<pre>
>>> import boto
>>> boto.set_stream_logging('rds')
>>> rds = boto.connect_rds(debug=2)
</pre>

and then run your command and send me the output (mitch at garnaat dot com).  If you want to post it here, that's fine, too but I would edit the output to remove an Signatures that are displayed in the output.

Any more info on this?  I've looked through the code and compared it to the RDS docs and it looks like boto is doing the right thing.

I've tried both creating an RDS from scratch and modifying an existing one (with apply_immediately=True) and can confirm that the multi_az param does not work.  I enabled logging and can see "MultiAZ=true" as one of the params and that there's no AvailabilityZone param set.

From what I can see, boto conforms to the online AWS documentation, but something is amiss somewhere.


Resolving patch:
diff --git a/boto/emr/connection.py b/boto/emr/connection.py
--- a/boto/emr/connection.py
+++ b/boto/emr/connection.py
@@ -274,7 +274,7 @@ def _build_instance_args(self, ec2_keyname, availability_zone, master_instance_t
         if ec2_keyname:
             params['Instances.Ec2KeyName'] = ec2_keyname
         if availability_zone:
-            params['Placement'] = availability_zone
+            params['Placement.AvailabilityZone'] = availability_zone
 
         return params
 


Test patch:
Repository: boto/boto

Issue:
Support OtherSelectionText in Mechanical Turk Response
The "other" free text response from a _**SelectionAnswer**_ answer specification is not provided back. It appears this is due to not providing a marshalling rule for the _**OtherSelectionText**_ element within the _**QuestionFormAnswer**_ class. I've added a comment with the simple change below.  I will also fork and provide the fix through a corresponding pull request.

``` python
@@ -908,7 +908,7 @@
     def endElement(self, name, value, connection):
         if name == 'QuestionIdentifier':
             self.qid = value
-        elif name in ['FreeText', 'SelectionIdentifier'] and self.qid:
+        elif name in ['FreeText', 'SelectionIdentifier', 'OtherSelectionText'] and self.qid:
             self.fields.append((self.qid,value))
         elif name == 'Answer':
             self.qid = None

```



Discussion:

Resolving patch:
diff --git a/boto/mturk/connection.py b/boto/mturk/connection.py
--- a/boto/mturk/connection.py
+++ b/boto/mturk/connection.py
@@ -908,7 +908,7 @@ def __init__(self, connection):
     def endElement(self, name, value, connection):
         if name == 'QuestionIdentifier':
             self.qid = value
-        elif name in ['FreeText', 'SelectionIdentifier'] and self.qid:
+        elif name in ['FreeText', 'SelectionIdentifier', 'OtherSelectionText'] and self.qid:
             self.fields.append((self.qid,value))
         elif name == 'Answer':
             self.qid = None


Test patch:
Repository: boto/boto

Issue:
Close() function in connections.py
I think the self.connection in the close function of connections.py should be self._connection. My code was throwing errors until I changed it.



Discussion:

Resolving patch:
diff --git a/boto/connection.py b/boto/connection.py
--- a/boto/connection.py
+++ b/boto/connection.py
@@ -800,7 +800,7 @@ def close(self):
         and making a new request will open a connection again."""
 
         boto.log.debug('closing all HTTP connections')
-        self.connection = None  # compat field
+        self._connection = None  # compat field
 
 class AWSQueryConnection(AWSAuthConnection):
 


Test patch:
Repository: boto/boto

Issue:
s3.Key.open_read() should read current with Key's version_id if any
boto.s3.Key.get_file() makes sure to check self.version_id and adds the versionId string to the query.

getting a specific version of a key like:

```
for line in mybucket.get_key( 'my/key/name.whatever', version_id='SOMELONGSTRINGOFCRAP' ):
    print line.strip()
```

should let me stream out a file at a specific version_id ... instead, this code gives me the latest version of the file. 



Discussion:

Resolving patch:
diff --git a/boto/s3/key.py b/boto/s3/key.py
--- a/boto/s3/key.py
+++ b/boto/s3/key.py
@@ -156,6 +156,10 @@ def open_read(self, headers=None, query_args=None,
         if self.resp == None:
             self.mode = 'r'
 
+            if self.version_id:
+                query_args = query_args or []
+                query_args.append('versionId=%s' % self.version_id)
+
             provider = self.bucket.connection.provider
             self.resp = self.bucket.connection.make_request(
                 'GET', self.bucket.name, self.name, headers,


Test patch:
Repository: boto/boto

Issue:
CloudFormation Bug in Parsing Describe Stacks Response
Currently the boto.cloudformation.CloudFormationConnection.describe__stacks() call does _not_ correctly parse the Capabilities section of the XML Response. 

This makes the parser assume the Capabilities section is part of the main Stack element. This means that, for example, if a Stack does contain a Capabilities section and an Outputs section, the Outputs section will not correctly parse either, and instead, the Outputs member element will get parsed as top level Stack elements.

This is an issue with the 2.3.0 release.



Discussion:

Resolving patch:
diff --git a/boto/cloudformation/stack.py b/boto/cloudformation/stack.py
--- a/boto/cloudformation/stack.py
+++ b/boto/cloudformation/stack.py
@@ -11,6 +11,7 @@ def __init__(self, connection=None):
         self.notification_arns = []
         self.outputs = []
         self.parameters = []
+        self.capabilities = []
         self.stack_id = None
         self.stack_status = None
         self.stack_name = None
@@ -24,6 +25,9 @@ def startElement(self, name, attrs, connection):
         elif name == "Outputs":
             self.outputs = ResultSet([('member', Output)])
             return self.outputs
+        elif name == "Capabilities":
+            self.capabilities = ResultSet([('member', Capability)])
+            return self.capabilities
         else:
             return None
 
@@ -165,6 +169,20 @@ def endElement(self, name, value, connection):
     def __repr__(self):
         return "Output:\"%s\"=\"%s\"" % (self.key, self.value)
 
+class Capability:
+    def __init__(self, connection=None):
+        self.connection = None
+        self.value = None
+
+    def startElement(self, name, attrs, connection):
+        return None
+
+    def endElement(self, name, value, connection):
+        self.value = value
+
+    def __repr__(self):
+        return "Capability:\"%s\"" % (self.value)
+
 class StackResource:
     def __init__(self, connection=None):
         self.connection = connection


Test patch:
Repository: boto/boto

Issue:
Allow DeleteOnTermination to be changed via ModifyInstanceAttribute
See:

http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ApiReference-query-ModifyInstanceAttribute.html



Discussion:
gathering all information I can find about this:
- dupe of #820
- http://itsecureadmin.com/2011/06/aws-instance-ebs-volume-delete-on-termination/
- https://groups.google.com/forum/?fromgroups=#!topic/boto-users/21EYYnB4EgA

the example from boto-users group still doesn't work.

btw, the milestone looks very outdated. boto is in v2.6 now


Resolving patch:
diff --git a/boto/ec2/connection.py b/boto/ec2/connection.py
--- a/boto/ec2/connection.py
+++ b/boto/ec2/connection.py
@@ -853,6 +853,7 @@ def modify_instance_attribute(self, instance_id, attribute, value):
             * userData - Base64 encoded String (None)
             * disableApiTermination - Boolean (true)
             * instanceInitiatedShutdownBehavior - stop|terminate
+            * blockDeviceMapping - List of strings - ie: ['/dev/sda=false']
             * sourceDestCheck - Boolean (true)
             * groupSet - Set of Security Groups or IDs
             * ebsOptimized - Boolean (false)
@@ -882,6 +883,12 @@ def modify_instance_attribute(self, instance_id, attribute, value):
                 if isinstance(sg, SecurityGroup):
                     sg = sg.id
                 params['GroupId.%s' % (idx + 1)] = sg
+        elif attribute.lower() == 'blockdevicemapping':
+            for idx, kv in enumerate(value):
+                dev_name, _, flag = kv.partition('=')
+                pre = 'BlockDeviceMapping.%d' % (idx + 1)
+                params['%s.DeviceName' % pre] = dev_name
+                params['%s.Ebs.DeleteOnTermination' % pre] = flag or 'true'
         else:
             # for backwards compatibility handle lowercase first letter
             attribute = attribute[0].upper() + attribute[1:]


Test patch:
Repository: boto/boto

Issue:
s3: setting metadata item named 'date' causes SignatureDoesNotMatch error
This was surprising: I can't set a metadata item named 'date' for an S3 object. 

After looking at utils.py, I see that this is because metadata shares its namespace with other header items, and 'date is magical and not usable as my own metadata key. But it still seems wrong that setting it causes a SignatureDoesNotMatch error when the object is put. How about a mention of the magical behavior in the docs?



Discussion:
This is a bug in boto.  You should still be able to set a metadata item called "date" because that should get transformed to "x-amz-date" which does not conflict with the actual Date header.  Thanks for reporting this, I'll fix it.

this is still broken


Resolving patch:
diff --git a/boto/utils.py b/boto/utils.py
--- a/boto/utils.py
+++ b/boto/utils.py
@@ -172,7 +172,7 @@ def merge_meta(headers, metadata, provider=None):
     for k in metadata.keys():
         if k.lower() in ['cache-control', 'content-md5', 'content-type',
                          'content-encoding', 'content-disposition',
-                         'date', 'expires']:
+                         'expires']:
             final_headers[k] = metadata[k]
         else:
             final_headers[metadata_prefix + k] = metadata[k]


Test patch:
diff --git a/tests/integration/s3/test_key.py b/tests/integration/s3/test_key.py
--- a/tests/integration/s3/test_key.py
+++ b/tests/integration/s3/test_key.py
@@ -32,7 +32,7 @@
 from boto.exception import S3ResponseError
 
 
-class S3KeyTest (unittest.TestCase):
+class S3KeyTest(unittest.TestCase):
     s3 = True
 
     def setUp(self):
@@ -373,3 +373,13 @@ def test_website_redirect_with_bad_value(self):
         with self.assertRaises(key.provider.storage_response_error):
             # Must start with a / or http
             key.set_redirect('')
+
+    def test_setting_date(self):
+        key = self.bucket.new_key('test_date')
+        # This should actually set x-amz-meta-date & not fail miserably.
+        key.set_metadata('date', '20130524T155935Z')
+        key.set_contents_from_string('Some text here.')
+
+        check = self.bucket.get_key('test_date')
+        self.assertEqual(check.get_metadata('date'), u'20130524T155935Z')
+        self.assertTrue('x-amz-meta-date' in check._get_remote_metadata())