Repository: aws/aws-cli

Issue:
[BUG] [aws kinesis] Unexpected output values by table & text format on describe-streams
- aws --version aws-cli/1.7.35 Python/2.7.5 Darwin/13.4.0
# normality

It expects "ACTIVE".

command (no output option):

``` bash
$ aws kinesis describe-stream --stream-name ${STREAM_NAME} --query 'StreamDescription.StreamStatus'
```

result:

``` text
"ACTIVE"
```

command (output json):

``` bash
$ aws kinesis describe-stream --stream-name ${STREAM_NAME} --query 'StreamDescription.StreamStatus' --output json
```

result:

``` text
"ACTIVE"
```
# BUG

It expects "ACTIVE".

command (output table):

``` bash
$ aws kinesis describe-stream --stream-name ${STREAM_NAME} --query 'StreamDescription.StreamStatus' --output table
```

result:

``` text
----------------
|DescribeStream|
+--------------+
```

command (output text):

``` bash
$ aws kinesis describe-stream --stream-name ${STREAM_NAME} --query 'StreamDescription.StreamStatus' --output text
```

result:

``` text
None
```

It happened not only "StreamStatus" but also "StreamName".

Thank you.



Discussion:
The output for the table is as expected because the end value is just a single string. Thus you cannot represent it as a table. It needs to be a list or structure. So something like this:

```
$ aws kinesis describe-stream --stream-name foo --query 'StreamDescription.[StreamStatus]' --output table
----------------
|DescribeStream|
+--------------+
|  ACTIVE      |
+--------------+
```

The output for text I can confirm and looks like a bug. That should not output `None` when all it needs to do is remove the quotes around a string.

After taking a deeper look, the root cause of this issue is that when using `--output text` on a paginated response, the resulting data is streamed as pages of results are retrieved from the service. The keys that we extract and stream are based on which keys change for each page of results. When applying the `--query` parameter to a paginated response using text output, the `--query` is applied to the filtered paginated keys rather than the entire response body. What this means is that you need to modify your `--query` to be relative to the paginated keys of an operation or you need to disable pagination using the `--no-paginate` argument.

As of right now, it's not obvious when an operation is paginated or what keys are streamed when paginating through responses using `--output text`. I'll work on a PR that injects this type of information in help output.


Resolving patch:
diff --git a/awscli/customizations/paginate.py b/awscli/customizations/paginate.py
--- a/awscli/customizations/paginate.py
+++ b/awscli/customizations/paginate.py
@@ -56,8 +56,8 @@
 
 
 def register_pagination(event_handlers):
-    event_handlers.register('building-argument-table',
-                            unify_paging_params)
+    event_handlers.register('building-argument-table', unify_paging_params)
+    event_handlers.register_last('doc-description', add_paging_description)
 
 
 def get_paginator_config(session, service_name, operation_name):
@@ -73,6 +73,35 @@ def get_paginator_config(session, service_name, operation_name):
     return operation_paginator_config
 
 
+def add_paging_description(help_command, **kwargs):
+    # This customization is only applied to the description of
+    # Operations, so we must filter out all other events.
+    if not isinstance(help_command.obj, model.OperationModel):
+        return
+    service_name = help_command.obj.service_model.service_name
+    paginator_config = get_paginator_config(
+        help_command.session, service_name, help_command.obj.name)
+    if not paginator_config:
+        return
+    help_command.doc.style.new_paragraph()
+    help_command.doc.writeln(
+        ('``%s`` is a paginated operation. Multiple API calls may be issued '
+         'in order to retrieve the entire data set of results. You can '
+         'disable pagination by providing the ``--no-paginate`` argument.')
+        % help_command.name)
+    # Only include result key information if it is present.
+    if paginator_config.get('result_key'):
+        queries = paginator_config['result_key']
+        if type(queries) is not list:
+            queries = [queries]
+        queries = ", ".join([('``%s``' % s) for s in queries])
+        help_command.doc.writeln(
+            ('When using ``--output text`` and the ``--query`` argument on a '
+             'paginated response, the ``--query`` argument must extract data '
+             'from the results of the following query expressions: %s')
+            % queries)
+
+
 def unify_paging_params(argument_table, operation_model, event_name,
                         session, **kwargs):
     paginator_config = get_paginator_config(


Test patch:
diff --git a/tests/functional/docs/test_help_output.py b/tests/functional/docs/test_help_output.py
--- a/tests/functional/docs/test_help_output.py
+++ b/tests/functional/docs/test_help_output.py
@@ -261,6 +261,12 @@ def test_builtin_paging_params_removed(self):
         self.assert_not_contains('``--next-token``')
         self.assert_not_contains('``--max-keys``')
 
+    def test_paging_documentation_added(self):
+        self.driver.main(['s3api', 'list-objects', 'help'])
+        self.assert_contains('``list-objects`` is a paginated operation')
+        self.assert_contains('When using ``--output text`` and the')
+        self.assert_contains('following query expressions: ')
+
 
 class TestMergeBooleanGroupArgs(BaseAWSHelpOutputTest):
     def test_merge_bool_args(self):
diff --git a/tests/unit/customizations/test_paginate.py b/tests/unit/customizations/test_paginate.py
--- a/tests/unit/customizations/test_paginate.py
+++ b/tests/unit/customizations/test_paginate.py
@@ -13,7 +13,11 @@
 from awscli.testutils import unittest
 
 from botocore.exceptions import DataNotFoundError
+from botocore.model import OperationModel
+from awscli.help import OperationHelpCommand, OperationDocumentEventHandler
+
 import mock
+from mock import Mock, patch
 
 from awscli.customizations import paginate
 
@@ -97,6 +101,56 @@ def test_service_with_no_paginate(self):
         self.assertEqual(starting_table, argument_table)
 
 
+class TestHelpDocumentationModifications(TestPaginateBase):
+    def test_injects_pagination_help_text(self):
+        with patch('awscli.customizations.paginate.get_paginator_config',
+                   return_value={'result_key': 'abc'}):
+            help_command = OperationHelpCommand(
+                Mock(), Mock(), Mock(), 'foo', OperationDocumentEventHandler)
+            help_command.obj = Mock(OperationModel)
+            help_command.obj.name = 'foo'
+            paginate.add_paging_description(help_command)
+            self.assertIn('``foo`` is a paginated operation. Multiple API',
+                          help_command.doc.getvalue().decode())
+            self.assertIn('following query expressions: ``abc``',
+                          help_command.doc.getvalue().decode())
+
+    def test_shows_result_keys_when_array(self):
+        with patch('awscli.customizations.paginate.get_paginator_config',
+                   return_value={'result_key': ['abc', '123']}):
+            help_command = OperationHelpCommand(
+                Mock(), Mock(), Mock(), 'foo', OperationDocumentEventHandler)
+            help_command.obj = Mock(OperationModel)
+            help_command.obj.name = 'foo'
+            paginate.add_paging_description(help_command)
+            self.assertIn('following query expressions: ``abc``, ``123``',
+                          help_command.doc.getvalue().decode())
+
+    def test_does_not_show_result_key_if_not_present(self):
+        with patch('awscli.customizations.paginate.get_paginator_config',
+                   return_value={'limit_key': 'aaa'}):
+            help_command = OperationHelpCommand(
+                Mock(), Mock(), Mock(), 'foo', OperationDocumentEventHandler)
+            help_command.obj = Mock(OperationModel)
+            help_command.obj.name = 'foo'
+            paginate.add_paging_description(help_command)
+            self.assertIn('``foo`` is a paginated operation. Multiple API',
+                          help_command.doc.getvalue().decode())
+            self.assertNotIn('following query expressions',
+                             help_command.doc.getvalue().decode())
+
+    def test_does_not_inject_when_no_pagination(self):
+        with patch('awscli.customizations.paginate.get_paginator_config',
+                   return_value=None):
+            help_command = OperationHelpCommand(
+                Mock(), Mock(), Mock(), 'foo', OperationDocumentEventHandler)
+            help_command.obj = Mock(OperationModel)
+            help_command.obj.name = 'foo'
+            paginate.add_paging_description(help_command)
+            self.assertNotIn('``foo`` is a paginated operation',
+                             help_command.doc.getvalue().decode())
+
+
 class TestStringLimitKey(TestPaginateBase):
 
     def setUp(self):