Repository: aws/aws-cli

Issue:
Reopen #2518: Prevent AttributeError when invoking awscli from python3 daemon
This is to reopen #2518. Details are there and will be submitting a new PR with an otherwise harmless, one-line fix. 
Fixes #2990 and #2518. Avoid AttributeError when awscli is invoked fr…
…om python3 daemon.


Discussion:
#2991 fixes
I'm not 100% sure why we closed the previous issue but just basing off the discussion it seems like we strayed away from merging this as it doesn't fully resolve the problem. In general it might resolve the issue but will lead to errors later on when you actually try to use a feature that relies on this. To be fair, I'm not sure how common this will be in a daemon context where there is no stdin anyways. 

This will also fail if a different IO object that doesn't have a buffer, such as `io.StringIO`, is used.
This fully resolves the problem. Right now among the python3 daemon users who invoke aws-cli, 100% error on the line in question. We have at least two users (myself and another on the previous issue), who have demonstrated that when the line in question is fixed, the aws-cli works through to completion. In other words, we go from 100% of this user group failing to less than 100% failing.  And keep in mind that as a rule of thumb, for every one user who complains there are a hundred more who work around the problem without notifying you. That alone is enough to justify fixing this line of code. By inspecting it, you can see that it does no harm. 

I suspect that it will actually fix it for all users in this group, as the python3 authors chose for the daemon to act this way for a reason--it has no need of providing binary input.  As mentioned on the previous issue, the offending line of code violates the spec:

The python docs are clear that a library should not rely on `sys.stdin.buffer`: 

from https://docs.python.org/3/library/sys.html:
> Note To write or read binary data from/to the standard streams, use the underlying binary buffer object. For example, to write bytes to stdout, use sys.stdout.buffer.write(b'abc').
However, if you are writing a library (and do not control in which context its code will be executed), be aware that the standard streams may be replaced with file-like objects like io.StringIO which do not support the buffer attribute.

Following comment added 2 hours after initial post...
To make the logic more rigorous, we can split python3 daemon users into two groups:
1. The users whose code observes the specification and does not rely on `sys.stdin`.
2. A hypothetical group of users who mistakenly expect a daemon to have access to `sys.stdin`.

For user group (1), #2991 fully resolves the issue.
For the hypothetical group (2)--whose existence is doubtful--#2991 shifts the problem to where it belongs--their code. aws-cli will do what it is expected to do.

And for all other users of aws-cli, #2991 makes absolutely no difference whatsoever--their code will continue to work as before.
I don't think simply making it `None` would be the best way to handle this. Any time we try to access it we would end up rasing an error like `'NoneType' object has no attribute 'write'` which isn't very helpful. I would prefer we add in a function `get_binary_stdin` with raises an exception with a descriptive method if stdin isn't available. That would delay the exception while also providing a helpful error message.
Making the `binary_stdin` variable `None` is consistent with `sys.stdin` being `None` and is far better than triggering an error. You are welcome to implement your preferred `get_binary_stdin` function. In the meantime, please merge #2991 so that the aws-cli works for this use case.

Resolving patch:
diff --git a/awscli/compat.py b/awscli/compat.py
--- a/awscli/compat.py
+++ b/awscli/compat.py
@@ -56,6 +56,14 @@
     default_pager = 'less -R'
 
 
+class StdinMissingError(Exception):
+    def __init__(self):
+        message = (
+            'stdin is required for this operation, but is not available.'
+        )
+        super(StdinMissingError, self).__init__(message)
+
+
 class NonTranslatedStdout(object):
     """ This context manager sets the line-end translation mode for stdout.
 
@@ -76,6 +84,7 @@ def __exit__(self, type, value, traceback):
             import msvcrt
             msvcrt.setmode(sys.stdout.fileno(), self.previous_mode)
 
+
 def ensure_text_type(s):
     if isinstance(s, six.text_type):
         return s
@@ -83,6 +92,7 @@ def ensure_text_type(s):
         return s.decode('utf-8')
     raise ValueError("Expected str, unicode or bytes, received %s." % type(s))
 
+
 if six.PY3:
     import locale
     import urllib.parse as urlparse
@@ -91,7 +101,10 @@ def ensure_text_type(s):
 
     raw_input = input
 
-    binary_stdin = sys.stdin.buffer
+    def get_binary_stdin():
+        if sys.stdin is None:
+            raise StdinMissingError()
+        return sys.stdin.buffer
 
     def get_binary_stdout():
         return sys.stdout.buffer
@@ -138,7 +151,10 @@ def bytes_print(statement, stdout=None):
 
     raw_input = raw_input
 
-    binary_stdin = sys.stdin
+    def get_binary_stdin():
+        if sys.stdin is None:
+            raise StdinMissingError()
+        return sys.stdin
 
     def get_binary_stdout():
         return sys.stdout
diff --git a/awscli/customizations/s3/s3handler.py b/awscli/customizations/s3/s3handler.py
--- a/awscli/customizations/s3/s3handler.py
+++ b/awscli/customizations/s3/s3handler.py
@@ -46,7 +46,7 @@
 from awscli.customizations.s3.utils import DeleteSourceFileSubscriber
 from awscli.customizations.s3.utils import DeleteSourceObjectSubscriber
 from awscli.customizations.s3.utils import DeleteCopySourceObjectSubscriber
-from awscli.compat import binary_stdin
+from awscli.compat import get_binary_stdin
 
 
 LOGGER = logging.getLogger(__name__)
@@ -471,6 +471,7 @@ def _add_additional_subscribers(self, subscribers, fileinfo):
             subscribers.append(ProvideSizeSubscriber(int(expected_size)))
 
     def _get_filein(self, fileinfo):
+        binary_stdin = get_binary_stdin()
         return NonSeekableStream(binary_stdin)
 
     def _format_local_path(self, path):


Test patch:
diff --git a/tests/functional/s3/test_cp_command.py b/tests/functional/s3/test_cp_command.py
--- a/tests/functional/s3/test_cp_command.py
+++ b/tests/functional/s3/test_cp_command.py
@@ -18,6 +18,12 @@
 from awscli.compat import six
 
 
+class BufferedBytesIO(six.BytesIO):
+    @property
+    def buffer(self):
+        return self
+
+
 class TestCPCommand(BaseAWSCommandParamsTest):
 
     prefix = 's3 cp '
@@ -484,9 +490,8 @@ def test_streaming_upload(self):
             'ETag': '"c8afdb36c52cf4727836669019e69222"'
         }]
 
-        binary_stdin = six.BytesIO(b'foo\n')
-        location = "awscli.customizations.s3.s3handler.binary_stdin"
-        with mock.patch(location, binary_stdin):
+        binary_stdin = BufferedBytesIO(b'foo\n')
+        with mock.patch('sys.stdin', binary_stdin):
             self.run_cmd(command)
 
         self.assertEqual(len(self.operations_called), 1)
@@ -506,9 +511,8 @@ def test_streaming_upload_with_expected_size(self):
             'ETag': '"c8afdb36c52cf4727836669019e69222"'
         }]
 
-        binary_stdin = six.BytesIO(b'foo\n')
-        location = "awscli.customizations.s3.s3handler.binary_stdin"
-        with mock.patch(location, binary_stdin):
+        binary_stdin = BufferedBytesIO(b'foo\n')
+        with mock.patch('sys.stdin', binary_stdin):
             self.run_cmd(command)
 
         self.assertEqual(len(self.operations_called), 1)
@@ -533,9 +537,8 @@ def test_streaming_upload_error(self):
         }]
         self.http_response.status_code = 404
 
-        binary_stdin = six.BytesIO(b'foo\n')
-        location = "awscli.customizations.s3.s3handler.binary_stdin"
-        with mock.patch(location, binary_stdin):
+        binary_stdin = BufferedBytesIO(b'foo\n')
+        with mock.patch('sys.stdin', binary_stdin):
             _, stderr, _ = self.run_cmd(command, expected_rc=1)
 
         error_message = (
@@ -544,6 +547,20 @@ def test_streaming_upload_error(self):
         )
         self.assertIn(error_message, stderr)
 
+    def test_streaming_upload_when_stdin_unavailable(self):
+        command = "s3 cp - s3://bucket/streaming.txt"
+        self.parsed_responses = [{
+            'ETag': '"c8afdb36c52cf4727836669019e69222"'
+        }]
+
+        with mock.patch('sys.stdin', None):
+            _, stderr, _ = self.run_cmd(command, expected_rc=1)
+
+        expected_message = (
+            'stdin is required for this operation, but is not available'
+        )
+        self.assertIn(expected_message, stderr)
+
     def test_streaming_download(self):
         command = "s3 cp s3://bucket/streaming.txt -"
         self.parsed_responses = [