internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: aws/aws-cli

Issue:
S3 - RequestTimeout during large files
I'm trying to upload a large file (9 GB) and getting a RequestTimeout error using `aws s3 mv ...`

I haven't fully tested it yet, but it seems like if I run the command over and over it will eventually work.

Here's the debug log from a failed attempt: https://s3.amazonaws.com/nimbus-public/s3_backup.log

I'll post back if I determine that retrying the command several times works or not.

aws version: aws-cli/1.1.2 Python/2.7.3 Windows/2008ServerR2



Discussion:
After multiple retries the command does eventually work on these large files (7-11GB).  But sometimes takes dozens of retries.

BTW, I'm running the command on an EC2 instance - shouldn't be any latency or network issues.

Looking into this, I believe I know what's causing the issue.

Note, I'm having similar reliability issues moving even larger files ~175Gb to S3.  We've tried mv, sync, and copy with various results.  We're running the following:
aws s3 --version
aws-cli/1.1.2 Python/2.6.6 Linux/2.6.32-358.el6.x86_64

Note this is a single (large) file.  It's already compressed.

We often see:
A client error (RequestTimeout) occurred: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.

Because of the large file size, retrying is extremely expensive.  The larger the file size, the less luck we're having.

Is there a better tool that can be used linux command line for moving big data to S3?


Resolving patch:
diff --git a/awscli/customizations/s3/utils.py b/awscli/customizations/s3/utils.py
--- a/awscli/customizations/s3/utils.py
+++ b/awscli/customizations/s3/utils.py
@@ -216,6 +216,10 @@ def read(self, amount=None):
             self._amount_read += actual_amount
             return data
 
+    def seek(self, where):
+        self._fileobj.seek(self._start_byte + where)
+        self._amount_read = where
+
     def close(self):
         self._fileobj.close()
 


Test patch:
diff --git a/tests/unit/customizations/s3/test_utils.py b/tests/unit/customizations/s3/test_utils.py
--- a/tests/unit/customizations/s3/test_utils.py
+++ b/tests/unit/customizations/s3/test_utils.py
@@ -108,6 +108,16 @@ def test_read_with_amount_size(self):
         self.assertEqual(chunk.read(1), b'r')
         self.assertEqual(chunk.read(1), b'')
 
+    def test_reset_stream_emulation(self):
+        filename = os.path.join(self.tempdir, 'foo')
+        f = open(filename, 'wb')
+        f.write(b'onetwothreefourfivesixseveneightnineten')
+        f.flush()
+        chunk = ReadFileChunk(filename, start_byte=11, size=4)
+        self.assertEqual(chunk.read(), b'four')
+        chunk.seek(0)
+        self.assertEqual(chunk.read(), b'four')
+
     def test_read_past_end_of_file(self):
         filename = os.path.join(self.tempdir, 'foo')
         f = open(filename, 'wb')
Repository: aws/aws-cli

Issue:
VIMPAGER error
I realize that it is a bit of an aside, but it would be great to support alternative pagers.

```
~ $ echo $MANPAGER
/bin/sh -c "col -bx | vim -c 'set ft=man' -"
~ $ python --version                                                                                                                    1
Python 2.7.5
~ $ pip --version
pip 1.4.1 from /Users/carl/.virtualenv/lib/python2.7/site-packages (python 2.7)
~ $ aws --version
aws-cli/1.1.0 Python/2.7.5 Darwin/12.5.0
~ $ aws help
-bx: -c: line 0: unexpected EOF while looking for matching `"'
-bx: -c: line 1: syntax error: unexpected end of file
```



Discussion:
On this note, it would also be nice to list help just on the command line (without dropping into a pager)


Resolving patch:
diff --git a/awscli/help.py b/awscli/help.py
--- a/awscli/help.py
+++ b/awscli/help.py
@@ -14,6 +14,7 @@
 import logging
 import os
 import platform
+import shlex
 from subprocess import Popen, PIPE
 
 from docutils.core import publish_string
@@ -79,7 +80,7 @@ def get_pager_cmdline(self):
             pager = os.environ['MANPAGER']
         elif 'PAGER' in os.environ:
             pager = os.environ['PAGER']
-        return pager.split()
+        return shlex.split(pager)
 
     def render(self, contents):
         man_contents = publish_string(contents, writer=manpage.Writer())


Test patch:
diff --git a/tests/unit/test_help.py b/tests/unit/test_help.py
--- a/tests/unit/test_help.py
+++ b/tests/unit/test_help.py
@@ -86,3 +86,9 @@ def test_no_groff_exists(self):
         with self.assertRaisesRegexp(ExecutableNotFoundError,
                                      'Could not find executable named "groff"'):
             renderer.render('foo')
+
+    def test_shlex_split_for_pager_var(self):
+        pager_cmd = '/bin/sh -c "col -bx | vim -c \'set ft=man\' -"'
+        os.environ['PAGER'] = pager_cmd
+        self.assertEqual(self.renderer.get_pager_cmdline(),
+                         ['/bin/sh', '-c', "col -bx | vim -c 'set ft=man' -"])
Repository: aws/aws-cli

Issue:
Using --ip-permissions with --group-id causes validation error
If you attempt to use the --ip-permissions option with `aws ec2 authorize-security-group-*` subcommands while using a `--group-id` option to specify a VPC group, you receive this validation error:

```
ValueError: The --group_id option is not compatible with the --ip-permissions option
```

This should be allowed, though.  The workaround is to use the simplified, scalar options to set the permissions.



Discussion:

Resolving patch:
diff --git a/awscli/customizations/ec2secgroupsimplify.py b/awscli/customizations/ec2secgroupsimplify.py
--- a/awscli/customizations/ec2secgroupsimplify.py
+++ b/awscli/customizations/ec2secgroupsimplify.py
@@ -48,12 +48,11 @@ def _check_args(parsed_args, **kwargs):
     # raise an error.
     arg_dict = vars(parsed_args)
     if arg_dict['ip_permissions']:
-        for key in ('protocol', 'group_id', 'port', 'cidr',
+        for key in ('protocol', 'port', 'cidr',
                     'source_group', 'group_owner'):
             if arg_dict[key]:
-                msg = ('Mixing the --ip-permissions option '
-                       'with the simple, scalar options is '
-                       'not recommended.')
+                msg = ('The --%s option is not compatible '
+                       'with the --ip-permissions option ') % key
                 raise ValueError(msg)
 
 def _add_docs(help_command, **kwargs):


Test patch:
diff --git a/tests/unit/ec2/test_security_group_operations.py b/tests/unit/ec2/test_security_group_operations.py
--- a/tests/unit/ec2/test_security_group_operations.py
+++ b/tests/unit/ec2/test_security_group_operations.py
@@ -68,6 +68,17 @@ def test_ip_permissions(self):
                   'IpPermissions.1.IpRanges.1.CidrIp': '192.168.100.0/24'}
         self.assert_params_for_cmd(args_list, result)
 
+    def test_ip_permissions_with_group_id(self):
+        json = """[{"FromPort":8000,"ToPort":9000,"IpProtocol":"tcp","IpRanges":[{"CidrIp":"192.168.100.0/24"}]}]"""
+        args = ' --group-id sg-12345678 --ip-permissions %s' % json
+        args_list = (self.prefix + args).split()
+        result = {'GroupId': 'sg-12345678',
+                  'IpPermissions.1.FromPort': '8000',
+                  'IpPermissions.1.ToPort': '9000',
+                  'IpPermissions.1.IpProtocol': 'tcp',
+                  'IpPermissions.1.IpRanges.1.CidrIp': '192.168.100.0/24'}
+        self.assert_params_for_cmd(args_list, result)
+
     def test_both(self):
         captured = cStringIO()
         json = """[{"FromPort":8000,"ToPort":9000,"IpProtocol":"tcp","IpRanges":[{"CidrIp":"192.168.100.0/24"}]}]"""
Repository: aws/aws-cli

Issue:
Expires timestamp in bundle-instance policy is incorrect
If you do not supply a policy to `aws ec2 bundle-instance` the command will generate a policy for you which includes an expiration of the policy in 24 hours.  The ISO timestamp included in the policy, however, needs to include the `Z` suffix to denote UTC time.



Discussion:

Resolving patch:
diff --git a/awscli/customizations/ec2bundleinstance.py b/awscli/customizations/ec2bundleinstance.py
--- a/awscli/customizations/ec2bundleinstance.py
+++ b/awscli/customizations/ec2bundleinstance.py
@@ -119,7 +119,8 @@ def _generate_policy(params):
     # Creates a policy that provides access for 24 hours.
     delta = datetime.timedelta(hours=24)
     expires = datetime.datetime.utcnow() + delta
-    policy = POLICY.format(expires=expires.isoformat(),
+    expires_iso = expires.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
+    policy = POLICY.format(expires=expires_iso,
                            bucket=params['Bucket'],
                            prefix=params['Prefix'])
     params['UploadPolicy'] = policy


Test patch:
diff --git a/tests/unit/ec2/test_bundle_instance.py b/tests/unit/ec2/test_bundle_instance.py
--- a/tests/unit/ec2/test_bundle_instance.py
+++ b/tests/unit/ec2/test_bundle_instance.py
@@ -25,11 +25,12 @@ class TestBundleInstance(BaseAWSCommandParamsTest):
 
     prefix = 'ec2 bundle-instance'
 
-    Base64Policy = ('eyJleHBpcmF0aW9uIjogIjIwMTMtMDgtMTBUMDA6MDA6MDAiL'
-                    'CJjb25kaXRpb25zIjogW3siYnVja2V0IjogIm15YnVja2V0In'
-                    '0seyJhY2wiOiAiZWMyLWJ1bmRsZS1yZWFkIn0sWyJzdGFydHM'
-                    'td2l0aCIsICIka2V5IiwgImZvb2JhciJdXX0=')
-    PolicySignature = '0Wr6cr2Je//jinxyiuL4qMs51Lk='
+    Base64Policy = ('eyJleHBpcmF0aW9uIjogIjIwMTMtMDgtMTBUMDA6MDA6MDAuM'
+                    'DAwMDAwWiIsImNvbmRpdGlvbnMiOiBbeyJidWNrZXQiOiAibXl'
+                    'idWNrZXQifSx7ImFjbCI6ICJlYzItYnVuZGxlLXJlYWQifSxbI'
+                    'nN0YXJ0cy13aXRoIiwgIiRrZXkiLCAiZm9vYmFyIl1dfQ==')
+
+    PolicySignature = 'ynxybUMv9YuGbPl7HZ8AFJW/2t0='
     
     def setUp(self):
         super(TestBundleInstance, self).setUp()
@@ -62,6 +63,7 @@ def test_no_policy_provided(self):
                    'Storage.S3.UploadPolicy': self.Base64Policy,
                    'Storage.S3.UploadPolicySignature': self.PolicySignature}
         self.assert_params_for_cmd(args_list, result)
+        
 
     def test_policy_provided(self):
         policy = '{"notarealpolicy":true}'
Repository: aws/aws-cli

Issue:
documentation: Show reduced redundancy option in "aws s3 cp help"
When a user types `aws s3 cp help` it describes this option:

```
--storage-class The type of storage to use for the object. Defaults  to
'STANDARD'
```

It would be super-helpful to list the string the user should specify here if they want reduced redundancy storage (i.e., `'REDUCED_REDUNDANCY'`).



Discussion:
+1 - Thanks for the tip.


Resolving patch:
diff --git a/awscli/customizations/s3/description.py b/awscli/customizations/s3/description.py
--- a/awscli/customizations/s3/description.py
+++ b/awscli/customizations/s3/description.py
@@ -107,6 +107,7 @@ def add_param_descriptions(params_dict):
 
     params_dict['storage-class']['documents'] = (
         "The type of storage to use for the object. "
+        "Valid choices are: STANDARD | REDUCED_REDUNDANCY. "
         "Defaults to 'STANDARD'")
 
     params_dict['website-redirect']['documents'] = (


Test patch:
Repository: aws/aws-cli

Issue:
typo in s3api list-objects documentation
The documentation for the s3api list-objects --max-items parameter says that a `NextMarker` will be provided, while the --starting-token parameter refers to this as `NextToken` which is the actual name of the returned token in JSON.

So in short I think that the `NextMarker` should really say `NextToken` to prevent any confusion.



Discussion:

Resolving patch:
diff --git a/awscli/customizations/paginate.py b/awscli/customizations/paginate.py
--- a/awscli/customizations/paginate.py
+++ b/awscli/customizations/paginate.py
@@ -36,7 +36,7 @@
 MAX_ITEMS_HELP = """
 <p>The total number of items to return.  If the total number
 of items available is more than the value specified in
-max-items then a <code>NextMarker</code> will
+max-items then a <code>NextToken</code> will
 be provided in the output that you can use to resume pagination.
 """
 


Test patch:
Repository: aws/aws-cli

Issue:
Docs: s3api restore-object
Regarding the documentation here: http://docs.aws.amazon.com/cli/latest/reference/s3api/restore-object.html

I am having a hard time figuring out what the different arguments are for and what their values should be. These docs just look a little sparse. I'll see if I can find some time to help add to them where I can.

Thanks,
-Seth



Discussion:
I think this might be a bug in our documentation generation process.  It looks like the restore-request takes a `Days` key: `aws s3api restore-object --bucket bucket-name --key key-name --restore-request Days=1`

I'll work on getting the docs updated so the `restore-request` arg is better documented.

Oh, great. If they are already written, just not being generated, hopefully that's an easier fix. Thanks for looking into it.


Resolving patch:
diff --git a/awscli/argprocess.py b/awscli/argprocess.py
--- a/awscli/argprocess.py
+++ b/awscli/argprocess.py
@@ -303,7 +303,7 @@ def _docs_special_key_value_parse(self, param):
             # should be skipped for this arg.
             return None
         else:
-            self._docs_key_value_parse(param)
+            return self._docs_key_value_parse(param)
 
     def _docs_key_value_parse(self, param):
         s = '%s ' % param.cli_name


Test patch:
diff --git a/tests/unit/docs/test_help_output.py b/tests/unit/docs/test_help_output.py
--- a/tests/unit/docs/test_help_output.py
+++ b/tests/unit/docs/test_help_output.py
@@ -230,6 +230,15 @@ def test_no_examples_for_structure_single_scalar(self):
         self.assert_not_contains('"Value": "string"')
         self.assert_not_contains('Value=string')
 
+    def test_example_for_single_structure_not_named_value(self):
+        # Verify that if a structure does match our special case
+        # (single element named "Value"), then we still document
+        # the example syntax.
+        self.driver.main(['s3api', 'restore-object', 'help'])
+        self.assert_contains('Days=value')
+        # Also should see the JSON syntax in the help output.
+        self.assert_contains('"Days": integer')
+
 
 class TestJSONListScalarDocs(BaseAWSHelpOutputTest):
     def test_space_separated_list_docs(self):
Repository: aws/aws-cli

Issue:
ec2 create-tags doesn't allow tags via the "file" prefix
Since ec2 create-tags allows for tags to be supplied as JSON, I'd expect that I'd be able to supply that JSON via the "file" or "http" conventions - but it looks like the JSON has to actually supplied in-line on the command line. Not sure if this this behavior is expected or not - my assumption was that anywhere that JSON was used it could be as a file, but that may be incorrect. 

```
aws ec2 create-tags  --tags file://tags.json  --resources i-12345678
Error parsing parameter --tags, should be: Key value pairs, with multiple values separated by a space.  
--tags Key=string,Value=string
```



Discussion:
This should work. What's the contents of your json file?  It should look similar to the [reference docs](http://docs.aws.amazon.com/cli/latest/reference/ec2/create-tags.html)/man page:

```
JSON Syntax:

[
  {
    "Key": "string",
    "Value": "string"
  }
  ...
]
```

Did some more digging - it was failing because the JSON file had a leading space on the first line.  I don't believe leading whitespace should matter as the JSON is syntactically valid and passes jsonlint just fine, but sorry about the misleading initial issue. 

Problematic  JSON:

```
 [
    {
        "Key": "test", 
        "Value": "test"
    }
]
```

I agree, the spacing should not matter.  Looking into this.


Resolving patch:
diff --git a/awscli/argprocess.py b/awscli/argprocess.py
--- a/awscli/argprocess.py
+++ b/awscli/argprocess.py
@@ -139,7 +139,7 @@ def get_parse_method_for_param(self, param, value=None):
         if isinstance(value, list):
             check_val = value[0]
         else:
-            check_val = value
+            check_val = value.strip()
         if isinstance(check_val, str) and check_val.startswith(('[', '{')):
             LOG.debug("Param %s looks like JSON, not considered for "
                       "param shorthand.", param.py_name)


Test patch:
diff --git a/tests/unit/elb/test_register_instances_with_load_balancer.py b/tests/unit/elb/test_register_instances_with_load_balancer.py
--- a/tests/unit/elb/test_register_instances_with_load_balancer.py
+++ b/tests/unit/elb/test_register_instances_with_load_balancer.py
@@ -64,6 +64,14 @@ def test_two_instance_from_file(self):
         cmdline += ' --instances file://%s' % data_path
         self.assert_params_for_cmd(cmdline, TWO_INSTANCE_EXPECTED)
 
+    def test_json_file_with_spaces(self):
+        data_path = os.path.join(os.path.dirname(__file__),
+                                 'test_with_spaces.json')
+        cmdline = self.prefix
+        cmdline += ' --load-balancer-name my-lb'
+        cmdline += ' --instances file://%s' % data_path
+        self.assert_params_for_cmd(cmdline, TWO_INSTANCE_EXPECTED)
+
     def test_two_instance_shorthand(self):
         cmdline = self.prefix
         cmdline += ' --load-balancer-name my-lb'
diff --git a/tests/unit/elb/test_with_spaces.json b/tests/unit/elb/test_with_spaces.json
new file mode 100644
--- /dev/null
+++ b/tests/unit/elb/test_with_spaces.json
@@ -0,0 +1,2 @@
+  [{"InstanceId": "i-12345678"},
+ {"InstanceId": "i-87654321"}]
Repository: aws/aws-cli

Issue:
aws s3 cp/sync and canned ACLs
Looking at the aws s3 cp/sync documentation I see the following limitation for ACLs:

```
       --acl Sets the ACl for the object when the command is  performed.  Only
       accepts values of private, public-read, or public-read-write.
```

I think it would really be helpful if full support of all the canned ACLs as listed [here](http://docs.aws.amazon.com/AmazonS3/latest/dev/ACLOverview.html#CannedACL) was added. 

In particular, I am having a hard time replicating the behavior of `bucket-owner-full-control` when accessing a bucket using credentials from a different AWS account than the bucket.



Discussion:

Resolving patch:
diff --git a/awscli/customizations/s3/description.py b/awscli/customizations/s3/description.py
--- a/awscli/customizations/s3/description.py
+++ b/awscli/customizations/s3/description.py
@@ -71,7 +71,9 @@ def add_param_descriptions(params_dict):
 
     params_dict['acl']['documents'] = "Sets the ACl for the object when the " \
         "command is performed.  Only accepts values of ``private``, \
-        ``public-read``, or ``public-read-write``."
+        ``public-read``, ``public-read-write``, ``authenticated-read``, \
+        ``bucket-owner-read``, ``bucket-owner-full-control`` and \
+        ``log-delivery-write``."
 
     params_dict['force']['documents'] = "Deletes all objects in the bucket " \
         "including the bucket itself."
diff --git a/awscli/customizations/s3/s3.py b/awscli/customizations/s3/s3.py
--- a/awscli/customizations/s3/s3.py
+++ b/awscli/customizations/s3/s3.py
@@ -854,7 +854,11 @@ def check_endpoint_url(self, parsed_globals):
                            'dest': 'filters'}},
                'acl': {'options': {'nargs': 1,
                                    'choices': ['private', 'public-read',
-                                               'public-read-write']}},
+                                               'public-read-write', 
+                                               'authenticated-read',
+                                               'bucket-owner-read',
+                                               'bucket-owner-full-control',
+                                               'log-delivery-write']}},
                'grants': {'options': {'nargs': '+'}},
                'sse': {'options': {'action': 'store_true'}},
                'storage-class': {'options': {'nargs': 1,


Test patch:
Repository: aws/aws-cli

Issue:
Multipart COPY operation fails when source filename contains + character
If you perform a bucket -> bucket cp or sync operation and one of the filenames in the source bucket contains a `+` character, you get this error:

```
ClientError: A client error (NoSuchKey) occurred when calling the UploadPartCopy operation: The specified key does not exist.
```

The problem is that the filename is not being quoted in the `x-amz-copy-source` header.  For example, if the source filename is `foo++bar.txt` the header is:

```
x-amz-copy-source:fiebazfoobar/foo++bar.txt
```

but S3 expects it to be:

```
x-amz-copy-source:fiebazfoobar/foo%2B%2Bbar.txt
```

This only affects multipart COPY operations.



Discussion:

Resolving patch:
diff --git a/awscli/customizations/s3/fileinfo.py b/awscli/customizations/s3/fileinfo.py
--- a/awscli/customizations/s3/fileinfo.py
+++ b/awscli/customizations/s3/fileinfo.py
@@ -235,7 +235,7 @@ def copy(self):
         """
         Copies a object in s3 to another location in s3.
         """
-        copy_source = quote(self.src.encode('utf-8'), safe='/~')
+        copy_source = self.src
         bucket, key = find_bucket_key(self.dest)
         params = {'endpoint': self.endpoint, 'bucket': bucket,
                   'copy_source': copy_source, 'key': key}


Test patch:
diff --git a/tests/unit/customizations/s3/fake_session.py b/tests/unit/customizations/s3/fake_session.py
--- a/tests/unit/customizations/s3/fake_session.py
+++ b/tests/unit/customizations/s3/fake_session.py
@@ -13,10 +13,10 @@
 import hashlib
 from operator import itemgetter
 from botocore.vendored import requests
+from six import text_type
 from six import StringIO
 from io import BytesIO
 
-from botocore.compat import unquote
 from mock import MagicMock, Mock
 
 from awscli.customizations.s3.filegenerator import find_bucket_key
@@ -224,8 +224,7 @@ def copy_object(self, kwargs):
         key = kwargs['key']
         copy_source = kwargs['copy_source']
         src_bucket, src_key = find_bucket_key(copy_source)
-        src_key = unquote(src_key)
-        if hasattr(src_key, 'decode'):
+        if not isinstance(src_key, text_type) and hasattr(src_key, 'decode'):
             src_key = src_key.decode('utf-8')
         response_data = {}
         etag = ''