Repository: aws/aws-cli

Issue:
`aws route53 --max-items <n>` throws error when `--start*` parameters used
Description:    Ubuntu 14.04.2 LTS
aws --version:
aws-cli/1.7.16 Python/2.7.6 Linux/3.13.0-48-generic

`aws route53 list-resource-record-sets --hosted-zone-id XYZABC --max-items 1`

Works correctly and returns 1 record.

`aws route53 list-resource-record-sets --hosted-zone-id XYZABC --max-items 1 --start-record-name hostname.example.com`

Returns:

```
Parameter validation failed:
Unknown parameter in input: "max_items", must be one of: HostedZoneId, StartRecordName, StartRecordType, StartRecordIdentifier, MaxItems
```

The same error happens with the `--start-record-type` added:

`aws route53 list-resource-record-sets --hosted-zone-id XYZABC --max-items 1 --start-record-name hostname.example.com --start-record-type CNAME`

Debugging output:

```
2015-03-26 06:10:27,908 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.7.16 Python/2.7.6 Linux/3.13.0-48-generic, botocore version: 0.97.0
2015-03-26 06:10:27,908 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['route53', 'list-resource-record-sets', '--hosted-zone-id', 'XYZABC', '--max-items', '1', '--start-record-name', 'hostname.example.com', '--start-record-type', 'CNAME', '--debug']
2015-03-26 06:10:27,908 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7f4cac5109b0>
2015-03-26 06:10:27,909 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider at 0x7f4cac7d0668>
2015-03-26 06:10:27,925 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.route53: calling handler <function register_retries_for_service at 0x7f4caccae050>
2015-03-26 06:10:27,928 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: route53
2015-03-26 06:10:27,929 - MainThread - botocore.hooks - DEBUG - Event building-command-table.route53: calling handler <function add_waiters at 0x7f4cac75d410>
2015-03-26 06:10:27,931 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'hosted-zone-id', <awscli.arguments.CLIArgument object at 0x7f4cac389650>), (u'start-record-name', <awscli.arguments.CLIArgument object at 0x7f4cac3896d0>), (u'start-record-type', <awscli.arguments.CLIArgument object at 0x7f4cac389710>), (u'start-record-identifier', <awscli.arguments.CLIArgument object at 0x7f4cac389410>), (u'max-items', <awscli.arguments.CLIArgument object at 0x7f4cac389750>)])
2015-03-26 06:10:27,931 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.route53.list-resource-record-sets: calling handler <function add_streaming_output_arg at 0x7f4cac8b2410>
2015-03-26 06:10:27,931 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.route53.list-resource-record-sets: calling handler <function add_cli_input_json at 0x7f4cac7cb8c0>
2015-03-26 06:10:27,931 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.route53.list-resource-record-sets: calling handler <function unify_paging_params at 0x7f4cac8b8320>
2015-03-26 06:10:27,932 - MainThread - awscli.customizations.paginate - DEBUG - Modifying paging parameters for operation: ListResourceRecordSets
2015-03-26 06:10:27,932 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.route53.list-resource-record-sets: calling handler <function add_generate_skeleton at 0x7f4cac7cbde8>
2015-03-26 06:10:27,932 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.route53.list-resource-record-sets: calling handler <bound method CliInputJSONArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7f4cac389790>>
2015-03-26 06:10:27,932 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.route53.list-resource-record-sets: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f4cac389bd0>>
2015-03-26 06:10:27,933 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.route53.list-resource-record-sets: calling handler <functools.partial object at 0x7f4cac3831b0>
2015-03-26 06:10:27,934 - MainThread - awscli.customizations.paginate - DEBUG - User has specified a manual pagination arg. Automatically setting --no-paginate.
2015-03-26 06:10:27,934 - MainThread - awscli.customizations.paginate - DEBUG - User has specified a manual pagination arg. Automatically setting --no-paginate.
2015-03-26 06:10:27,934 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.route53.list-resource-record-sets.hosted-zone-id: calling handler <function uri_param at 0x7f4cac8edc80>
2015-03-26 06:10:27,934 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.route53.list-resource-record-sets: calling handler <awscli.argprocess.ParamShorthand object at 0x7f4cac5193d0>
2015-03-26 06:10:27,934 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2015-03-26 06:10:27,934 - MainThread - awscli.arguments - DEBUG - Unpacked value of u'XYZABC' for parameter "hosted_zone_id": u'XYZABC'
2015-03-26 06:10:27,934 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.route53.list-resource-record-sets.start-record-name: calling handler <function uri_param at 0x7f4cac8edc80>
2015-03-26 06:10:27,934 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.route53.list-resource-record-sets: calling handler <awscli.argprocess.ParamShorthand object at 0x7f4cac5193d0>
2015-03-26 06:10:27,934 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2015-03-26 06:10:27,934 - MainThread - awscli.arguments - DEBUG - Unpacked value of u'hostname.example.com' for parameter "start_record_name": u'hostname.example.com'
2015-03-26 06:10:27,934 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.route53.list-resource-record-sets.start-record-type: calling handler <function uri_param at 0x7f4cac8edc80>
2015-03-26 06:10:27,935 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.route53.list-resource-record-sets: calling handler <awscli.argprocess.ParamShorthand object at 0x7f4cac5193d0>
2015-03-26 06:10:27,935 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2015-03-26 06:10:27,935 - MainThread - awscli.arguments - DEBUG - Unpacked value of u'CNAME' for parameter "start_record_type": u'CNAME'
2015-03-26 06:10:27,935 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.route53.list-resource-record-sets.start-record-identifier: calling handler <function uri_param at 0x7f4cac8edc80>
2015-03-26 06:10:27,935 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.route53.list-resource-record-sets.max-items: calling handler <function uri_param at 0x7f4cac8edc80>
2015-03-26 06:10:27,935 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.route53.list-resource-record-sets.cli-input-json: calling handler <function uri_param at 0x7f4cac8edc80>
2015-03-26 06:10:27,935 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.route53.list-resource-record-sets.starting-token: calling handler <function uri_param at 0x7f4cac8edc80>
2015-03-26 06:10:27,935 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.route53.list-resource-record-sets.page-size: calling handler <function uri_param at 0x7f4cac8edc80>
2015-03-26 06:10:27,935 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.route53.list-resource-record-sets.generate-cli-skeleton: calling handler <function uri_param at 0x7f4cac8edc80>
2015-03-26 06:10:27,935 - MainThread - botocore.hooks - DEBUG - Event calling-command.route53.list-resource-record-sets: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f4cac389bd0>>
2015-03-26 06:10:27,935 - MainThread - botocore.hooks - DEBUG - Event calling-command.route53.list-resource-record-sets: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7f4cac389790>>
2015-03-26 06:10:27,936 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2015-03-26 06:10:27,936 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2015-03-26 06:10:27,953 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: route53
2015-03-26 06:10:27,953 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: route53
2015-03-26 06:10:27,957 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.route53.ListResourceRecordSets: calling handler <function fix_route53_ids at 0x7f4caccae578>
2015-03-26 06:10:27,957 - MainThread - botocore.handlers - DEBUG - HostedZoneId XYZABC -> XYZABC
2015-03-26 06:10:27,957 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 183, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 367, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 534, in __call__
    call_parameters, parsed_globals)
  File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 657, in invoke
    **parameters)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 187, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 228, in _make_api_call
    api_params, operation_model)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 259, in _convert_to_request_dict
    api_params, operation_model)
  File "/usr/local/lib/python2.7/dist-packages/botocore/validate.py", line 271, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
ParamValidationError: Parameter validation failed:
Unknown parameter in input: "max_items", must be one of: HostedZoneId, StartRecordName, StartRecordType, StartRecordIdentifier, MaxItems
2015-03-26 06:10:27,958 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

Parameter validation failed:
Unknown parameter in input: "max_items", must be one of: HostedZoneId, StartRecordName, StartRecordType, StartRecordIdentifier, MaxItems
```

The same command without `--max-items 1` works correctly:

`aws route53 list-resource-record-sets --hosted-zone-id XYZABC --start-record-name hostname.example.com --start-record-type CNAME` 



Discussion:
Taking a look.

Confirmed bug, working on fix.


Resolving patch:
diff --git a/awscli/customizations/paginate.py b/awscli/customizations/paginate.py
--- a/awscli/customizations/paginate.py
+++ b/awscli/customizations/paginate.py
@@ -73,7 +73,6 @@ def get_paginator_config(session, service_name, operation_name):
 
 def unify_paging_params(argument_table, operation_model, event_name,
                         session, **kwargs):
-
     paginator_config = get_paginator_config(
         session, operation_model.service_model.service_name,
         operation_model.name)
@@ -85,13 +84,11 @@ def unify_paging_params(argument_table, operation_model, event_name,
     _remove_existing_paging_arguments(argument_table, paginator_config)
     parsed_args_event = event_name.replace('building-argument-table.',
                                            'operation-args-parsed.')
-    session.register(
-        parsed_args_event,
-        partial(check_should_enable_pagination,
-                list(_get_all_cli_input_tokens(paginator_config))))
-    argument_table['starting-token'] = PageArgument('starting-token',
-                                                    STARTING_TOKEN_HELP,
-                                                    parse_type='string')
+    shadowed_args = {}
+    add_paging_argument(argument_table, 'starting-token',
+                        PageArgument('starting-token', STARTING_TOKEN_HELP,
+                                     parse_type='string'),
+                        shadowed_args)
     input_members = operation_model.input_shape.members
     type_name = 'integer'
     if 'limit_key' in paginator_config:
@@ -103,15 +100,35 @@ def unify_paging_params(argument_table, operation_model, event_name,
                  ' and parameter {2}').format(
                     type_name, operation_model.name,
                     paginator_config['limit_key']))
-        argument_table['page-size'] = PageArgument(
-            'page-size', PAGE_SIZE_HELP, parse_type=type_name)
+        add_paging_argument(argument_table, 'page-size',
+                            PageArgument('page-size', PAGE_SIZE_HELP,
+                                         parse_type=type_name),
+                            shadowed_args)
+
+    add_paging_argument(argument_table, 'max-items',
+                        PageArgument('max-items', MAX_ITEMS_HELP,
+                                     parse_type=type_name),
+                        shadowed_args)
+    session.register(
+        parsed_args_event,
+        partial(check_should_enable_pagination,
+                list(_get_all_cli_input_tokens(paginator_config)),
+                shadowed_args, argument_table))
+
 
-    argument_table['max-items'] = PageArgument('max-items', MAX_ITEMS_HELP,
-                                               parse_type=type_name)
+def add_paging_argument(argument_table, arg_name, argument, shadowed_args):
+    if arg_name in argument_table:
+        # If there's already an entry in the arg table for this argument,
+        # this means we're shadowing an argument for this operation.  We
+        # need to store this later in case pagination is turned off because
+        # we put these arguments back.
+        # See the comment in check_should_enable_pagination() for more info.
+        shadowed_args[arg_name] = argument_table[arg_name]
+    argument_table[arg_name] = argument
 
 
-def check_should_enable_pagination(input_tokens, parsed_args, parsed_globals,
-                                   **kwargs):
+def check_should_enable_pagination(input_tokens, shadowed_args, argument_table,
+                                   parsed_args, parsed_globals, **kwargs):
     normalized_paging_args = ['start_token', 'max_items']
     for token in input_tokens:
         py_name = token.replace('-', '_')
@@ -122,6 +139,16 @@ def check_should_enable_pagination(input_tokens, parsed_args, parsed_globals,
             logger.debug("User has specified a manual pagination arg. "
                          "Automatically setting --no-paginate.")
             parsed_globals.paginate = False
+            # Because we've now disabled pagination, there's a chance that
+            # we were shadowing arguments.  For example, we inject a
+            # --max-items argument in unify_paging_params().  If the
+            # the operation also provides its own MaxItems (which we
+            # expose as --max-items) then our custom pagination arg
+            # was shadowing the customers arg.  When we turn pagination
+            # off we need to put back the original argument which is
+            # what we're doing here.
+            for key, value in shadowed_args.items():
+                argument_table[key] = value
 
 
 def _remove_existing_paging_arguments(argument_table, pagination_config):


Test patch:
diff --git a/tests/unit/customizations/test_paginate.py b/tests/unit/customizations/test_paginate.py
--- a/tests/unit/customizations/test_paginate.py
+++ b/tests/unit/customizations/test_paginate.py
@@ -166,7 +166,7 @@ def test_should_not_enable_pagination(self):
         self.parsed_args.foo = None
         self.parsed_args.bar = 10
         paginate.check_should_enable_pagination(
-            input_tokens, self.parsed_args, self.parsed_globals)
+            input_tokens, {}, {}, self.parsed_args, self.parsed_globals)
         # We should have turned paginate off because the
         # user specified --bar 10
         self.assertFalse(self.parsed_globals.paginate)
@@ -178,7 +178,7 @@ def test_should_enable_pagination_with_no_args(self):
         self.parsed_args.foo = None
         self.parsed_args.bar = None
         paginate.check_should_enable_pagination(
-            input_tokens, self.parsed_args, self.parsed_globals)
+            input_tokens, {}, {}, self.parsed_args, self.parsed_globals)
         # We should have turned paginate off because the
         # user specified --bar 10
         self.assertTrue(self.parsed_globals.paginate)
@@ -195,6 +195,22 @@ def test_default_to_pagination_on_when_ambiguous(self):
         self.parsed_args.foo = None
         self.parsed_args.max_items = 10
         paginate.check_should_enable_pagination(
-            input_tokens, self.parsed_args, self.parsed_globals)
+            input_tokens, {}, {}, self.parsed_args, self.parsed_globals)
         self.assertTrue(self.parsed_globals.paginate,
                         "Pagination was not enabled.")
+
+    def test_shadowed_args_are_replaced_when_pagination_off(self):
+        input_tokens = ['foo', 'bar']
+        self.parsed_globals.paginate = True
+        # Corresponds to --bar 10
+        self.parsed_args.foo = None
+        self.parsed_args.bar = 10
+        shadowed_args = {'foo': mock.sentinel.ORIGINAL_ARG}
+        arg_table = {'foo': mock.sentinel.PAGINATION_ARG}
+        paginate.check_should_enable_pagination(
+            input_tokens, shadowed_args, arg_table,
+            self.parsed_args, self.parsed_globals)
+        # We should have turned paginate off because the
+        # user specified --bar 10
+        self.assertFalse(self.parsed_globals.paginate)
+        self.assertEqual(arg_table['foo'], mock.sentinel.ORIGINAL_ARG)
diff --git a/tests/unit/route53/test_list_resource_record_sets.py b/tests/unit/route53/test_list_resource_record_sets.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/route53/test_list_resource_record_sets.py
@@ -0,0 +1,45 @@
+#!/usr/bin/env python
+# Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+#     http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+from awscli.testutils import BaseAWSCommandParamsTest
+
+
+class TestGetHostedZone(BaseAWSCommandParamsTest):
+
+    prefix = 'route53 list-resource-record-sets'
+
+    def test_no_pagination_args(self):
+        args = ' --hosted-zone-id /hostedzone/ZD3IYMVP1KDDM'
+        cmdline = self.prefix + args
+        self.assert_params_for_cmd(
+            cmdline, {'HostedZoneId': 'ZD3IYMVP1KDDM'}, expected_rc=0)
+
+    def test_with_max_items_pagination(self):
+        args = ' --hosted-zone-id /hostedzone/ZD3IYMVP1KDDM --max-items 1'
+        cmdline = self.prefix + args
+        # We don't map to the service's max-items
+        self.assert_params_for_cmd(
+            cmdline, {'HostedZoneId': 'ZD3IYMVP1KDDM'}, expected_rc=0)
+
+    def test_with_max_override_starting_args(self):
+        args = (
+            ' --hosted-zone-id /hostedzone/ZD3IYMVP1KDDM'
+            ' --max-items 1'
+            ' --start-record-name foo')
+        cmdline = self.prefix + args
+        # We don't map to the service's max-items
+        self.assert_params_for_cmd(
+            cmdline, {'HostedZoneId': 'ZD3IYMVP1KDDM',
+                      'StartRecordName': 'foo',
+                      'MaxItems': '1'},
+            expected_rc=0)