internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: aws/aws-cli

Issue:
datapipeline query-object --query argument is shadowed
The top level `--query` option shadows the `--query` from datapipeline query-object.  This can be addressed in the argrename customization model.  We can also take this opportunity to remove the `cli_name` from the `.extra.json` files in botocore.



Discussion:

Resolving patch:
diff --git a/awscli/customizations/argrename.py b/awscli/customizations/argrename.py
--- a/awscli/customizations/argrename.py
+++ b/awscli/customizations/argrename.py
@@ -26,6 +26,11 @@
     'ec2.create-image.no-no-reboot': 'reboot',
     'ec2.*.no-egress': 'ingress',
     'ec2.*.no-disable-api-termination': 'enable-api-termination',
+    'opsworks.*.region': 'stack-region',
+    'elastictranscoder.*.output': 'job-output',
+    'swf.register-activity-type.version': 'activity-version',
+    'swf.register-workflow-type.version': 'workflow-version',
+    'datapipeline.*.query': 'objects-query',
 }
 
 


Test patch:
diff --git a/tests/unit/datapipeline/test_query_objects.py b/tests/unit/datapipeline/test_query_objects.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/datapipeline/test_query_objects.py
@@ -0,0 +1,36 @@
+# Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+#     http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+from tests.unit import BaseAWSCommandParamsTest
+
+
+class TestDataPipelineQueryObjects(BaseAWSCommandParamsTest):
+    maxDiff = None
+
+    prefix = 'datapipeline query-objects '
+
+    def test_renamed_object_query_arg(self):
+        # --query is renamed to --objects-query so we don't
+        # conflict with the global --query argument.
+        args = ('--pipeline-id foo '
+                '--sphere INSTANCE '
+                '--objects-query {"selectors":[{"fieldName":"@status",'
+                '"operator":{"type":"EQ","values":["RUNNING"]}}]}')
+        cmdline = self.prefix + args
+        expected = {
+            'pipelineId': 'foo',
+            'query': {'selectors': [{'fieldName': '@status',
+                                     'operator': {'type': 'EQ',
+                                                  'values': ['RUNNING']}}]},
+            'sphere': 'INSTANCE'
+        }
+        self.assert_params_for_cmd(cmdline, expected)
Repository: aws/aws-cli

Issue:
JSON Cache for AssumeRoleProvider not truncating files
When we open a file for writing, if we're reusing the same file (same cache key) we don't truncate the file before writing.  If the second JSON response is smaller it will result in extra data at the end of the JSON document.

This will trigger a json parsing error, which raises a KeyError, which causes the cred provider to retrieve a new set of temporary credentials because it thinks the file is not in the cache.



Discussion:

Resolving patch:
diff --git a/awscli/customizations/assumerole.py b/awscli/customizations/assumerole.py
--- a/awscli/customizations/assumerole.py
+++ b/awscli/customizations/assumerole.py
@@ -80,6 +80,7 @@ def __setitem__(self, cache_key, value):
             os.makedirs(self._working_dir)
         with os.fdopen(os.open(full_key,
                                os.O_WRONLY | os.O_CREAT, 0o600), 'w') as f:
+            f.truncate()
             f.write(file_content)
 
     def _convert_cache_key(self, cache_key):


Test patch:
diff --git a/tests/unit/customizations/test_assumerole.py b/tests/unit/customizations/test_assumerole.py
--- a/tests/unit/customizations/test_assumerole.py
+++ b/tests/unit/customizations/test_assumerole.py
@@ -107,6 +107,13 @@ def test_key_error_raised_when_cache_key_does_not_exist(self):
         with self.assertRaises(KeyError):
             self.cache['foo']
 
+    def test_file_is_truncated_before_writing(self):
+        self.cache['mykey'] = {
+            'really long key in the cache': 'really long value in cache'}
+        # Now overwrite it with a smaller value.
+        self.cache['mykey'] = {'a': 'b'}
+        self.assertEqual(self.cache['mykey'], {'a': 'b'})
+
     @skip_if_windows('File permissions tests not supported on Windows.')
     def test_permissions_for_file_restricted(self):
         self.cache['mykey'] = {'foo': 'bar'}
Repository: aws/aws-cli

Issue:
groff 1.22.3 on smartos won't parse help text
Due to the way the latest groff 1.22.3 is installed on smartos (through pkgsrc), the traditional

`groff -man` will not parse man format pages correctly. I think this may be related to groff 1.22.3 in general, not the particular installation on smartos (illumos), but I do not have latest groff on either my mac or linux machines to test. In any case, as a result, there is empty help text when using `aws help`.

The fix, which should not interfere with other platforms, is to run `groff -m man`; you can change the line setting up the groff command in `help.py` to:

`cmdline = ['groff', '-m', 'man', '-T', 'ascii']`

In case someone has groff 1.22.3 installed on another platform it would be worth testing to see if the help pages are rendered correctly.



Discussion:
Upon closer inspection, this may be a problem with _any_ platform using **pkgsrc** installation of groff 1.22.3, wherein installation of `an.tmac` is commented out.

It would be helpful if someone with NetBSD or another platform using pkgsrc installation of groff could test this out.

I've run `aws help` with groff 1.22.3 installed via homebrew and also pkgsrc. Both installations worked. It looks like `-man` and `-m man` are identical, so I don't know why that would make a difference. This definitely seems like a bug with groff on smartos.

That's a huge help in terms of narrowing down the problem. 

I recognize -man and -m man should be identical, but the legacy (i.e. `-man`) behavior is implemented using a macro package called `an` which should redirect to mandoc but it is not. I note that in both smartos and NetBSD's pkgsrc installers the an.tmac file is not installed.

On what platform did you test a pkgsrc installed version of groff?

OSX. SmartOS is not available as an instance type on EC2, so I wasn't able to test there. This definitely looks like a change we could pull in, but I'd rather see the root cause fixed. Have you tried raising an issue with the SmartOS maintainers?

I will as soon as I can narrow it down to SmartOS. 

pkgsrc is also used by NetBSD and I noticed in the NetBSD version of the package that `an.tmac` is also commented out, so I need to track the error to the appropriate upstream source.

Thanks for your patience; will report back.


Resolving patch:
diff --git a/awscli/help.py b/awscli/help.py
--- a/awscli/help.py
+++ b/awscli/help.py
@@ -106,7 +106,7 @@ def _convert_doc_content(self, contents):
         man_contents = publish_string(contents, writer=manpage.Writer())
         if not self._exists_on_path('groff'):
             raise ExecutableNotFoundError('groff')
-        cmdline = ['groff', '-man', '-T', 'ascii']
+        cmdline = ['groff', '-m', 'man', '-T', 'ascii']
         LOG.debug("Running command: %s", cmdline)
         p3 = self._popen(cmdline, stdin=PIPE, stdout=PIPE, stderr=PIPE)
         groff_output = p3.communicate(input=man_contents)[0]


Test patch:
Repository: aws/aws-cli

Issue:
aws cloudformation create-change-set with 'template-url' broken
```
$ aws --region eu-west-1 cloudformation create-change-set --change-set-name test --stack-name autobuild --template-url https://s3-eu-west-1.amazonaws.com/BUCKET/TEMPLATE.json --parameters ...  --capabilities CAPABILITY_IAM

Error parsing parameter '--template-url': Unable to retrieve https://s3-eu-west-1.amazonaws.com/BUCKET/TEMPLATE.json: received non 200 status code of 403
```

The bucket is not public, and access is controlled via IAM.



Discussion:

Resolving patch:
diff --git a/awscli/paramfile.py b/awscli/paramfile.py
--- a/awscli/paramfile.py
+++ b/awscli/paramfile.py
@@ -29,6 +29,7 @@
     'apigateway.put-integration.uri',
     'cloudformation.create-stack.template-url',
     'cloudformation.update-stack.template-url',
+    'cloudformation.create-change-set.template-url',
     'cloudformation.validate-template.template-url',
     'cloudformation.estimate-template-cost.template-url',
 


Test patch:
Repository: aws/aws-cli

Issue:
[aws deploy push] Destination bundle type does not match extension
When specifying a destination with a suffix of ".zip" or ".tar" or ".tar.gz" for use with the `aws deploy push` command the resulting bundle is always only zipped.

```
$ aws deploy push --application-name test --s3-location s3://$AWS_DEPLOY_BUCKET/test/latest.tar --no-ignore-hidden-files --source .
To deploy with this revision, run:
aws deploy create-deployment --application-name test --s3-location bucket=BUCKET_NAME,key=test/latest.tar,bundleType=zip,eTag=... --deployment-group-name <deployment-group-name> --deployment-config-name <deployment-config-name> --description <description>
```

I downloaded and check the file it come back as a zip file.

```
$ file latest.tar 
latest.tar: Zip archive data, at least v2.0 to extract
```



Discussion:

Resolving patch:
diff --git a/awscli/customizations/codedeploy/push.py b/awscli/customizations/codedeploy/push.py
--- a/awscli/customizations/codedeploy/push.py
+++ b/awscli/customizations/codedeploy/push.py
@@ -35,7 +35,7 @@ class Push(BasicCommand):
 
     DESCRIPTION = (
         'Bundles and uploads to Amazon Simple Storage Service (Amazon S3) an '
-        'application revision, which is an archive file that contains '
+        'application revision, which is a zip archive file that contains '
         'deployable content and an accompanying Application Specification '
         'file (AppSpec file). If the upload is successful, a message is '
         'returned that describes how to call the create-deployment command to '
@@ -61,8 +61,8 @@ class Push(BasicCommand):
                 'Required. Information about the location of the application '
                 'revision to be uploaded to Amazon S3. You must specify both '
                 'a bucket and a key that represent the Amazon S3 bucket name '
-                'and the object key name. Use the format '
-                's3://\<bucket\>/\<key\>'
+                'and the object key name. Content will be zipped before '
+                'uploading. Use the format s3://\<bucket\>/\<key\>'
             )
         },
         {
@@ -90,7 +90,7 @@ class Push(BasicCommand):
             'help_text': (
                 'Optional. The location of the deployable content and the '
                 'accompanying AppSpec file on the development machine to be '
-                'bundled and uploaded to Amazon S3. If not specified, the '
+                'zipped and uploaded to Amazon S3. If not specified, the '
                 'current directory is used.'
             )
         },


Test patch:
Repository: aws/aws-cli

Issue:
Fix typo in SSE_C/SSE_C_KEY help text
s/specfied/specified/g


Discussion:

Resolving patch:
diff --git a/awscli/customizations/s3/subcommands.py b/awscli/customizations/s3/subcommands.py
--- a/awscli/customizations/s3/subcommands.py
+++ b/awscli/customizations/s3/subcommands.py
@@ -190,7 +190,7 @@
         'of the the object in S3. ``AES256`` is the only valid value. '
         'If the parameter is specified but no value is provided, '
         '``AES256`` is used. If you provide this value, ``--sse-c-key`` '
-        'must be specfied as well.'
+        'must be specified as well.'
     )
 }
 
@@ -200,7 +200,7 @@
     'help_text': (
         'The customer-provided encryption key to use to server-side '
         'encrypt the object in S3. If you provide this value, '
-        '``--sse-c`` must be specfied as well. The key provided should '
+        '``--sse-c`` must be specified as well. The key provided should '
         '**not** be base64 encoded.'
     )
 }


Test patch:
Repository: aws/aws-cli

Issue:
Adding new profile should add newline before
Have multiple IAM users configured in `~/.aws/credentials`

Added IAM user as follows:
`aws configure --profile NEW_PROFILE_NAME`
Completed sucessfully as expected.

However, next step of running a command resulted in error: 
```
PS C:\WINDOWS\system32> aws --profile NEW_PROFILE_NAME s3 ls
Unable to locate credentials. You can configure credentials by running "aws configure".
```
Checking `~/.aws/credentials` file showed new creds were appended to file without newline as follows:
```
[OLD_PROFILE_NAME]
aws_access_key_id = AAAAAAAAAAAAAAAAAAAA
aws_secret_access_key = KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
region=XX-east-X
toolkit_artifact_guid=11111111-1111-1111-1111-111111111111[NEW_PROFILE_NAME]
aws_access_key_id = BBBBBBBBBBBBBBBBBBBB
aws_secret_access_key = JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
```

IMHO this is easily fixable by adding a newline echo before the new creds

CLI version: `aws-cli/1.12.2 Python/2.7.14 Windows/10 botocore/1.8.2`


Discussion:

Resolving patch:
diff --git a/awscli/customizations/configure/writer.py b/awscli/customizations/configure/writer.py
--- a/awscli/customizations/configure/writer.py
+++ b/awscli/customizations/configure/writer.py
@@ -76,8 +76,22 @@ def _create_file(self, config_filename):
                                os.O_WRONLY | os.O_CREAT, 0o600), 'w'):
             pass
 
+    def _check_file_needs_newline(self, filename):
+        # check if the last byte is a newline
+        with open(filename, 'rb') as f:
+            # check if the file is empty
+            f.seek(0, os.SEEK_END)
+            if not f.tell():
+                return False
+            f.seek(-1, os.SEEK_END)
+            last = f.read()
+            return last != b'\n'
+
     def _write_new_section(self, section_name, new_values, config_filename):
+        needs_newline = self._check_file_needs_newline(config_filename)
         with open(config_filename, 'a') as f:
+            if needs_newline:
+                f.write('\n')
             f.write('[%s]\n' % section_name)
             contents = []
             self._insert_new_values(line_number=0,


Test patch:
diff --git a/tests/unit/customizations/configure/test_writer.py b/tests/unit/customizations/configure/test_writer.py
--- a/tests/unit/customizations/configure/test_writer.py
+++ b/tests/unit/customizations/configure/test_writer.py
@@ -356,3 +356,16 @@ def test_can_handle_empty_section(self):
             '[preview]\n'
             'cloudfront = true\n'
         )
+
+    def test_appends_newline_on_new_section(self):
+        original = (
+            '[preview]\n'
+            'cloudfront = true'
+        )
+        self.assert_update_config(
+            original, {'region': 'us-west-2', '__section__': 'new-section'},
+            '[preview]\n'
+            'cloudfront = true\n'
+            '[new-section]\n'
+            'region = us-west-2\n'
+        )
Repository: aws/aws-cli

Issue:
ONEZONE_IA should be an allowed parameter to --storage-class
I'm using CLI v1.15.2 and according to the [docs](https://docs.aws.amazon.com/cli/latest/reference/s3api/copy-object.html), `ONEZONE_IA` is one of the allowed parameters to `--storage-class`, yet when I use it, I get the error:

```
aws: error: argument --storage-class: Invalid choice, valid choices are:

STANDARD                                 | REDUCED_REDUNDANCY
STANDARD_IA
```


Discussion:
I am also seeing this issue with v1.15.4
```
$ aws --version
aws-cli/1.15.4 Python/2.7.12 Linux/4.4.0-43-Microsoft botocore/1.10.4

$ aws s3 mv s3://s3-*** s3://s3-*** --storage-class ONEZONE_IA

aws: error: argument --storage-class: Invalid choice, valid choices are:

STANDARD                                 | REDUCED_REDUNDANCY
STANDARD_IA
```
Ah looks like a new storage class came out. I'll take a look and see if it works with all the `s3` subcommands.

Resolving patch:
diff --git a/awscli/customizations/s3/subcommands.py b/awscli/customizations/s3/subcommands.py
--- a/awscli/customizations/s3/subcommands.py
+++ b/awscli/customizations/s3/subcommands.py
@@ -248,11 +248,12 @@
 
 
 STORAGE_CLASS = {'name': 'storage-class',
-                 'choices': ['STANDARD', 'REDUCED_REDUNDANCY', 'STANDARD_IA'],
+                 'choices': ['STANDARD', 'REDUCED_REDUNDANCY', 'STANDARD_IA',
+                             'ONEZONE_IA'],
                  'help_text': (
                      "The type of storage to use for the object. "
                      "Valid choices are: STANDARD | REDUCED_REDUNDANCY "
-                     "| STANDARD_IA. "
+                     "| STANDARD_IA | ONEZONE_IA. "
                      "Defaults to 'STANDARD'")}
 
 


Test patch:
diff --git a/tests/functional/s3/test_cp_command.py b/tests/functional/s3/test_cp_command.py
--- a/tests/functional/s3/test_cp_command.py
+++ b/tests/functional/s3/test_cp_command.py
@@ -98,6 +98,21 @@ def test_upload_expires(self):
         self.assertEqual(self.operations_called[0][1]['Bucket'], 'bucket')
         self.assertEqual(self.operations_called[0][1]['Expires'], '90')
 
+    def test_upload_onezone_ia(self):
+        full_path = self.files.create_file('foo.txt', 'mycontent')
+        cmdline = ('%s %s s3://bucket/key.txt --storage-class ONEZONE_IA' %
+                   (self.prefix, full_path))
+        self.parsed_responses = \
+            [{'ETag': '"c8afdb36c52cf4727836669019e69222"'}]
+        self.run_cmd(cmdline, expected_rc=0)
+        self.assertEqual(len(self.operations_called), 1,
+                         self.operations_called)
+        self.assertEqual(self.operations_called[0][0].name, 'PutObject')
+        args = self.operations_called[0][1]
+        self.assertEqual(args['Key'], 'key.txt')
+        self.assertEqual(args['Bucket'], 'bucket')
+        self.assertEqual(args['StorageClass'], 'ONEZONE_IA')
+
     def test_operations_used_in_download_file(self):
         self.parsed_responses = [
             {"ContentLength": "100", "LastModified": "00:00:00Z"},
Repository: aws/aws-cli

Issue:
Passing AWS_PROFILE on update-kubeconfig cli
Add profile details if calling update passing update-kubeconfig with  profile details 

#aws eks update-kubeconfig --name k8s-test --kubeconfig k8s-test.config --profile myprofile 

Currently this generates as 
```
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - k8s-test
      command: aws-iam-authenticator
```

Its good to add the profile details on the cli when not using default profile. 

```
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      args:
        - "token"
        - "-i"
        - "k8s-test"

      env:
        - name: AWS_PROFILE
          value: "myprofile"
```


Discussion:
Marking this as a feature request.
Hi @dileep-p , thanks for this request. I'll look into it.
Thanks @abeer91 

Resolving patch:
diff --git a/awscli/customizations/eks/update_kubeconfig.py b/awscli/customizations/eks/update_kubeconfig.py
--- a/awscli/customizations/eks/update_kubeconfig.py
+++ b/awscli/customizations/eks/update_kubeconfig.py
@@ -333,4 +333,10 @@ def get_user_entry(self):
                 self._role_arn
             ])
 
+        if self._session.profile:
+            generated_user["user"]["exec"]["env"] = [OrderedDict([
+                ("name", "AWS_PROFILE"),
+                ("value", self._session.profile)
+            ])]
+
         return generated_user


Test patch:
diff --git a/tests/unit/customizations/eks/test_update_kubeconfig.py b/tests/unit/customizations/eks/test_update_kubeconfig.py
--- a/tests/unit/customizations/eks/test_update_kubeconfig.py
+++ b/tests/unit/customizations/eks/test_update_kubeconfig.py
@@ -195,6 +195,7 @@ def setUp(self):
 
         self._session = mock.Mock(spec=botocore.session.Session)
         self._session.create_client.return_value = self._mock_client
+        self._session.profile = None
 
         self._client = EKSClient(self._session, "ExampleCluster", None)
 
@@ -261,3 +262,18 @@ def test_cluster_deleting(self):
             name="ExampleCluster"
         )
         self._session.create_client.assert_called_once_with("eks")
+
+    def test_profile(self):
+        self._session.profile = "profile"
+        self._correct_user_entry["user"]["exec"]["env"] = [
+            OrderedDict([
+                ("name", "AWS_PROFILE"),
+                ("value", "profile")
+            ])
+        ]
+        self.assertEqual(self._client.get_user_entry(),
+                         self._correct_user_entry)
+        self._mock_client.describe_cluster.assert_called_once_with(
+            name="ExampleCluster"
+        )
+        self._session.create_client.assert_called_once_with("eks")
Repository: aws/aws-cli

Issue:
The aws-cli bundle package uses an insecure version of PyYAML
### awscli version:<br>
`aws-cli/1.16.52 Python/2.7.15 Linux/4.14.77-69.57.amzn1.x86_64 exec-env/AWS_ECS_EC2 botocore/1.12.42`

[NVD entry](https://nvd.nist.gov/vuln/detail/CVE-2017-18342)

This issue was found when vulnerability alerts started appearing in Twistlock in response to scans of Docker images that we are using in several applications. The generic error found in these outlines is as such:<br>

```
Impacted versions: <=3.13
In PyYAML before 4.1, the yaml.load() API could execute arbitrary code. In other words, yaml.safe_load is not used.
```

These images are not natively using PyYAML, so this led us to a Docker `RUN` line in a Dockerfile that executed a script that contains a line of code that executes the installation of the `aws-cli` bundle using the following URL:<br>

`https://s3.amazonaws.com/aws-cli/awscli-bundle.zip`

Unpacking this archive shows a list of package dependencies that includes the vulnerable version of PyYAML:<br>

`awscli-bundle/packages/PyYAML-3.13.tar.gz`

The latest (and actually secure) version of PyYAML appears to be 4.1 according to the developer via the [GitHub repo](https://github.com/yaml/pyyaml).

### Request

Is it possible to have the patched version of PyYAML added to this bundle to avoid this vulnerability?

Thank you!


Discussion:

Resolving patch:
diff --git a/awscli/customizations/ecs/filehelpers.py b/awscli/customizations/ecs/filehelpers.py
--- a/awscli/customizations/ecs/filehelpers.py
+++ b/awscli/customizations/ecs/filehelpers.py
@@ -78,4 +78,4 @@ def parse_appspec(appspec_str):
     try:
         return json.loads(appspec_str)
     except ValueError:
-        return yaml.load(appspec_str)
+        return yaml.safe_load(appspec_str)


Test patch: