Repository: aws/aws-cli

Issue:
encoding Ö (O-Umlaut) error in command 'aws ec2 describe-instances' when redirecting to a file
we have a tag value with an Ö (O-Umlaut)  in the string value. 

"aws ec2 describe-instances --output=json" works just fine
"aws ec2 describe-instances --output=text (or output=table" gives the following error:
'ascii' codec can't encode character u'\xf6' in position 60: ordinal not in range(128)

I'd expect it to either go consistently wrong in all output formats or consistently go without error in all output formats.



Discussion:
What version of the CLI are you running?  There's a similar bug filed a while back that was fixed in later releases.

Ah found it: https://github.com/aws/aws-cli/pull/721
Let me know if you're still seeing this issue in the latest version of the CLI.

1.3.6—
Sent from Mailbox for iPhone

On Fri, Apr 4, 2014 at 11:46 PM, James Saryerwinnie
notifications@github.com wrote:

> ## What version of the CLI are you running?  There's a similar bug filed a while back that was fixed in later releases.
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/aws/aws-cli/issues/742#issuecomment-39614977

Interesting, not seeing that issue:

```
$ aws --version
aws-cli/1.3.6 Python/2.7.5 Darwin/12.5.0

$ aws ec2 describe-instances --query 'Reservations[].Instances[].Tags' --region us-east-1 --output table
----------------------------
|     DescribeInstances    |
+-------+------------------+
|  Key  |      Value       |
+-------+------------------+
|  Ö    |  Ö               |
+-------+------------------+

 aws ec2 describe-instances --query 'Reservations[].Instances[].Tags' --region us-east-1 --output text
Ö  Ö
```

It might also be an issue with the terminal encoding.  For example, I have my terminal set to use utf-8.  We encode the output based on the encoding of sys.stdout.  You can check this with:

```
$ python -c "import sys; print(sys.stdout.encoding)"
UTF-8
```

error still there........I just reproduced it and the terminal encoding is UTF-8:

$ aws --version
aws-cli/1.3.6 Python/2.7.5 Darwin/13.1.0
$ python -c "import sys; print(sys.stdout.encoding)"
UTF-8
$ aws ec2 describe-instances --output=table >instances_table
'ascii' codec can't encode character u'\xf6' in position 60: ordinal not in range(128)
(AWSCLI)defssw2s0:.aws mvn$

Ahh, that's the issue, we don't handle the case when we redirect stdout to a file.  Reopening...


Resolving patch:
diff --git a/awscli/compat.py b/awscli/compat.py
new file mode 100644
--- /dev/null
+++ b/awscli/compat.py
@@ -0,0 +1,23 @@
+# Copyright 2012-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+
+#     http://aws.amazon.com/apache2.0/
+
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+import sys
+import six
+
+if six.PY3:
+    def get_stdout_text_writer():
+        return sys.stdout
+else:
+    import codecs
+    import locale
+    def get_stdout_text_writer():
+        return codecs.getwriter(locale.getpreferredencoding())(sys.stdout)
diff --git a/awscli/formatter.py b/awscli/formatter.py
--- a/awscli/formatter.py
+++ b/awscli/formatter.py
@@ -13,12 +13,12 @@
 import logging
 import sys
 import json
-import jmespath
 
 from botocore.utils import set_value_from_jmespath
 
 from awscli.table import MultiTable, Styler, ColorizedStyler
 from awscli import text
+from awscli import compat
 
 
 LOG = logging.getLogger(__name__)
@@ -40,6 +40,16 @@ def _remove_request_id(self, response_data):
                     LOG.debug('RequestId: %s', request_id)
                 del response_data['ResponseMetadata']
 
+    def _get_default_stream(self):
+        if getattr(sys.stdout, 'encoding', None) is None:
+            # In python3, sys.stdout.encoding is always set.
+            # In python2, if you redirect to stdout, then
+            # encoding is not None.  In this case we'll default
+            # to utf-8.
+            return compat.get_stdout_text_writer()
+        else:
+            return sys.stdout
+
 
 class FullyBufferedFormatter(Formatter):
     def __call__(self, operation, response, stream=None):
@@ -47,7 +57,7 @@ def __call__(self, operation, response, stream=None):
             # Retrieve stdout on invocation instead of at import time
             # so that if anything wraps stdout we'll pick up those changes
             # (specifically colorama on windows wraps stdout).
-            stream = sys.stdout
+            stream = self._get_default_stream()
         # I think the interfaces between non-paginated
         # and paginated responses can still be cleaned up.
         if operation.can_paginate and self._args.paginate:
@@ -207,7 +217,7 @@ class TextFormatter(Formatter):
 
     def __call__(self, operation, response, stream=None):
         if stream is None:
-            stream = sys.stdout
+            stream = self._get_default_stream()
         try:
             if operation.can_paginate and self._args.paginate:
                 result_keys = response.result_keys


Test patch:
diff --git a/tests/unit/output/test_text_output.py b/tests/unit/output/test_text_output.py
--- a/tests/unit/output/test_text_output.py
+++ b/tests/unit/output/test_text_output.py
@@ -12,14 +12,19 @@
 # ANY KIND, either express or implied. See the License for the specific
 # language governing permissions and limitations under the License.
 from tests.unit import BaseAWSCommandParamsTest
+from tests import unittest
 import json
 import os
 import sys
 import re
+import locale
 
 from six.moves import cStringIO
+import six
 import mock
 
+from awscli.formatter import Formatter
+
 
 class TestListUsers(BaseAWSCommandParamsTest):
 
@@ -73,3 +78,22 @@ def test_text_response(self):
         self.assertEqual(
             output,
             'ENGINEDEFAULTS\tNone\n')
+
+
+class CustomFormatter(Formatter):
+    def __call__(self, operation, response, stream=None):
+        self.stream = self._get_default_stream()
+
+
+class TestDefaultStream(BaseAWSCommandParamsTest):
+    @unittest.skipIf(six.PY3, "Text writer only vaild on py3.")
+    def test_default_stream_with_table_output(self):
+        formatter = CustomFormatter(None)
+        stream = cStringIO()
+        with mock.patch('sys.stdout', stream):
+            formatter(None, None)
+            formatter.stream.write(u'\u00e9')
+        # Ensure the unicode data is written as UTF-8 by default.
+        self.assertEqual(
+            formatter.stream.getvalue(),
+            u'\u00e9'.encode(locale.getpreferredencoding()))