Repository: aws/aws-cli

Issue:
Support generation of signed URL's for S3 access



Discussion:
Yeah I think this would be a great feature.  I'm thinking this would be a new s3 subcommand?  I've been working on some internal changes to the s3 command code to make it easier to add subcommands.  I've been using this to start to refactor the `ls` command to fix several of the reported issues with `ls`.

Not entirely done yet, but here's what I have so far: 
https://github.com/jamesls/aws-cli/compare/s3-ls-permissions#diff-b88a66f4bd148577a9390cb980d7eeb9R321

Yes, this I was looking for. Please, allow to specify expiration not only by duration, but also by a specific datetime. Requiring UTC timezone, ISO format, full seconds, the string ending with "Z" seems to me well usable (e.g. "2014-05-30T00:00:00Z").

In case, you really need to generate tmpurl from command line, I could recommend my command line tool [s3tmpgen](https://pypi.python.org/pypi/ttr.aws.utils.s3#s3tmpgen)

Anyway, I would really invite such functionality in AWS CLI, which became valuable tool for my daily work with AWS S3.

+1

+1

I'm looking to use S3 to host code bundles to deployed to Heroku and this would be an awesome feature :)

https://blog.heroku.com/archives/2014/5/22/introducing_the_app_json_application_manifest

+1

+1 would be great to have this in AWS CLI

Just a note for anyone who ends up here, you can do this yourself trivially by reaching down into the `boto` library:  https://boto.readthedocs.org/en/latest/ref/s3.html#boto.s3.key.Key.generate_url

Thanks for the feedback.  I'll take a look at this, I agree that this would be a great feature to add to the AWS CLI.

Thanks, @johnboxall. Boto is certainly an option, though for folks deploying the cli binary without native Python deployments (think Windows users), going that route is just as much work as pulling down the PowerShell SDK and doing things via the .NET SDK. Would still like to see this native in the CLI. :grin:

Just for anyone looking to use boto directly for now until this is added to aws-cli I figured I would add a quick sample instruction. I've now went looking for this a couple of times and would rather have it along with this ticket and @johnboxall comment. This may be extremely obvious for some, but not for non-python devs

On a box which already has python installed

```
$ python --version
Python 2.7.6
$ sudo pip install boto
$ python
>>> import boto
>>> s3 = boto.connect_s3()
>>> bucket = s3.get_bucket('your-bucket-name')
>>> key = bucket.get_key('the-prefix/the-name-of-the-object.mp4')
>>> key.generate_url(3600)
'https://your-bucket-name.s3.amazonaws.com/the-prefix/the-name-of-the-object.mp4?Signature=CgDfFa45DBXFiMfASxSTpiSuHKM%3D&Expires=1415913273&AWSAccessKeyId=ABCDEDKSY344ACVDG'
```

Building on @isleshocky77's example... adding arguments and minor error checking:

## Script

```
$ cat boto-get-signed-url.py
#!/usr/bin/python
import boto
import argparse

parser = argparse.ArgumentParser(description='Generate an S3 signed URL')
parser.add_argument('-b', '--bucket', help='bucket name')
parser.add_argument('-k', '--key', help='prefix/key')
parser.add_argument('-s', '--seconds', type=int, help='time in seconds until the URL will expire')
args = parser.parse_args()

s3 = boto.connect_s3()
bucket = s3.get_bucket(args.bucket)
key = bucket.get_key(args.key)
if bucket.get_key(args.key):
  print key.generate_url(args.seconds)
else:
  print 's3://' + args.bucket + '/' + args.key + ' does not exist'
```

## Example usage and help

```
$ ./boto-get-signed-url.py -b superbucket -k "test" -s 60
https://superbucket.s3.amazonaws.com/test?Signature=n6cO8RH%2FbNwQhuZVNNazo3q04x0%3D&Expires=1416695021&AWSAccessKeyId=AKIEXAMPLEKEYNOTREAL

$ boto-get-signed-url.py --help
usage: boto-get-signed-url.py [-h] [-b BUCKET] [-k KEY] [-s SECONDS]

Generate an S3 signed URL

optional arguments:
  -h, --help            show this help message and exit
  -b BUCKET, --bucket BUCKET
                        bucket name
  -k KEY, --key KEY     prefix/key
  -s SECONDS, --seconds SECONDS
                        time in seconds until the URL will expire
```

Hi all
After another check about status of this issue I made an update to Python package ttr.aws.utils.s3 providing the tool [s3tmpgen](https://pypi.python.org/pypi/ttr.aws.utils.s3#s3tmpgen). The update allows to generate the url not only using https, but (with option an `-http`) also http.

Still looking forward to replace this with awscli solution.

:+1: 

+1 for s3 signed url to be part of cli

:+1:

+1

I think, I understand, why it takes so long time: things must happen in order and good thing is, it looks like being on the roadmap.

As AWS CLI is based on `botocore`, it shall be resolved there first. boto/botocore#291 is asking for it.

Pull request boto/botocore#504 was already merged into branch `clients-only` and roadmap (mentioned on README at https://github.com/boto/botocore ) claims, botocore is currently just one step before merging the branch `clients-only` into develop (then beta, GA and AWS CLI has "native" tool to provide give functionality for us).

@vlcinsky 

Yep once the clients-only branch gets merged into botocore, we will be able to pick up the generation of signed url's in the CLI. Then, the main amount of work that would have to be done on the CLI side is to create a good API that exposes the feature.

https://github.com/boto/botocore/issues/291 was already closed on 23 Jun: ["The feature is currently available in botocore."](https://github.com/boto/botocore/issues/291#issuecomment-114577321)

:+1: would be a handy feature indeed

+1

As @kyleknap noted, one of things to do is to design good API for this feature.

I see following use cases:
- "tmpGET": create tmpurl for GET request
- "tmpPOST": create tmpurl for POST request
- "tmp???": is there any more method to support? I am not aware of (and have never used other than
  GET and POST)

The "tmpGET" is really easy as the only output is url and the only input is bucket/key and expiration
date (expiration time).

The "tmpPOST" is much more complex as one has to define a policy for posting.

One question to resolve is, where to put these actions, there seem to be two alternatives:
- `aws s3`
- `aws s3api`

I will not deal today with the more complex "tmpPOST" and will focus on simpler "tmpGET"

# Quick solution: add `aws s3 tmpurl`

Such a solution would be rather easy to implement, would help in 80% of real
use situations (publish an object to AWS S3 and provide temporary url to share
with someone).

The concept would build on existing `aws s3 ls` with following differences:
- add an option `--expires` to define date and time at which the url expires or `--expires-in` to
  specify number of seconds till the expiration
- would output only tmpurls, one line per listed object

I am afraid, there is currently no easy way to create tmp url for exact moment in time, so initial
version would offer only the `--expires-in` option with default value of 3600 seconds.

## Pros and cons

### Pros

It is very easy to create bunch of tmp urls for number of existing objects on AWS S3. It saves the
hurdle with getting exact bucket/key name values.

### Cons

Disadvantage of this `aws s3 ls` based approach is, one cannot create tmp url for an object, which
does not exist yet. 

Anyway, this could be resolved later on by `aws s3api` solution.

Very nicely put there.  Whereas a post is something that would almost certainly be built around some server side handling of the posted data, the get method use cases are usually quite a lot easier.  When you say that would cover most use cases, I would strongly agree.

I meant to add though, I don't think the temp URL should have anything to do with locating the files on s3.  If somebody wants to generate entire sets of signed URLs, that's logic the cli doesn't need to do more to wrap.  The tmpurl primitive is the minimum viable product in my mind .  

I've implemented(not fully) pre-sgined URL for s3 objects in my local branch.

  https://github.com/quiver/aws-cli/tree/s3-presigned-url

This is a thin-wrapper for `botocore.generate_presigned_url`, and it's not production ready yet. As @vlcinsky  noted, we need a good API design as `AWSCLI`. Once that's fixed I can implement that.

One thing I note is that `generate_presigned_url` requires `client_method` parameter, which specifies S3 API(e.g. `get_object`) to sign for. In my current implementation, it's exposed to the user, but it would be better to be user-friendly, like providing different sub-commands(`aws s3 geturl`, `aws s3 uploadurl`) or switch options(`--type upload`)

# Usage

## upload objects to S3

```
$ echo hello world > test.txt

$ aws s3 url s3://BUCKET/test.txt --client-method put_object --expires-in 180
https://BUCKET.s3.amazonaws.com/test.txt?AWSAccessKeyId=AKIAIXXXXXXXXXXXXXXX&Expires=1451449621&Signature=KgwO9lBx942fFvln0JW0NX7mKS0%3D

$ URL=`aws s3 url s3://BUCKET/test.txt --client-method put_object --expires-in 180`

$ curl -D - -X PUT --upload-file test.txt $URL
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
x-amz-id-2: /90B1axPysBg3P8kv8BlR8RoqdO1JfajCN5BM5/TxIT3VjGphKmyGX8EgCQEtCXYhuNkVne5+GM=
x-amz-request-id: 685F03CA6C84FAC0
Date: Wed, 30 Dec 2015 05:18:38 GMT
ETag: "6f5902ac237024bdd0c176cb93063dc4"
Content-Length: 0
Server: AmazonS3

$ aws s3 cp s3://BUCKET/test.txt -
hello world
```

## get objects from S3

```
$ URL=`aws s3 url s3://BUCKET/test.txt --client-method get_object --expires-in 180`

$ curl -D - -X GET $URL
HTTP/1.1 200 OK
x-amz-id-2: WuRokcBm9wnDMaRkD8kNeGijuKEzVp3eagi7JbpPXmmchEljsiP4wZX5w1TaeuK94n2526FGKMI=
x-amz-request-id: 1EBCAA7A691A577D
Date: Wed, 30 Dec 2015 05:20:14 GMT
Last-Modified: Wed, 30 Dec 2015 05:19:15 GMT
ETag: "6f5902ac237024bdd0c176cb93063dc4"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 12
Server: AmazonS3

hello world
```

The above python generates a url like: 'https://<bucket>.s3.amazonaws.com/dir/dir/file

When I attempt to curl the file, I get SSL certificate problem: Invalid certificate chain
If I curl with option -k, I can get past this, but I also know that if the url looks like: 'https://s3-<region>.amazonaws.com/<bucket>/dir/dir/file'
the cert is valid, is there a way to change the url, or some other fix?
Thanks,
Garry

eg: 'https://s3-us-west-1.amazonaws.com/bucket/dir/dir/file'

Update, using boto3 was able to generate the correctly signed url.

+1

Impressive - in 3 years official CLI tool did not get support for signed url implemented, allthough it exists in boto and [s3cmd](http://s3tools.org/usage)

1+

+1

+1

Any update on this? Surprised this isn't available yet.

+1

I wrote one up as a workaround and works as expected: https://github.com/gdbtek/aws-tools 

Hi everyone, thanks for the feedback.  This something that is on our backlog.  I don't have exact dates yet, but I'll link to this issue once we have a pull request up.


Resolving patch:
diff --git a/awscli/customizations/s3/s3.py b/awscli/customizations/s3/s3.py
--- a/awscli/customizations/s3/s3.py
+++ b/awscli/customizations/s3/s3.py
@@ -13,7 +13,8 @@
 from awscli.customizations import utils
 from awscli.customizations.commands import BasicCommand
 from awscli.customizations.s3.subcommands import ListCommand, WebsiteCommand, \
-    CpCommand, MvCommand, RmCommand, SyncCommand, MbCommand, RbCommand
+    CpCommand, MvCommand, RmCommand, SyncCommand, MbCommand, RbCommand, \
+    PresignCommand
 from awscli.customizations.s3.syncstrategy.register import \
     register_sync_strategies
 
@@ -58,7 +59,8 @@ class S3(BasicCommand):
         {'name': 'rm', 'command_class': RmCommand},
         {'name': 'sync', 'command_class': SyncCommand},
         {'name': 'mb', 'command_class': MbCommand},
-        {'name': 'rb', 'command_class': RbCommand}
+        {'name': 'rb', 'command_class': RbCommand},
+        {'name': 'presign', 'command_class': PresignCommand},
     ]
 
     def _run_main(self, parsed_args, parsed_globals):
diff --git a/awscli/customizations/s3/subcommands.py b/awscli/customizations/s3/subcommands.py
--- a/awscli/customizations/s3/subcommands.py
+++ b/awscli/customizations/s3/subcommands.py
@@ -605,6 +605,36 @@ def _get_bucket_name(self, path):
         return path
 
 
+class PresignCommand(S3Command):
+    NAME = 'presign'
+    DESCRIPTION = ("Generate a pre-signed URL for an Amazon S3 object. "
+                   "This allows anyone who receives the pre-signed URL "
+                   "to retrieve the S3 object with an HTTP GET request.")
+    USAGE = "<S3Uri>"
+    ARG_TABLE = [{'name': 'path',
+                  'positional_arg': True, 'synopsis': USAGE},
+                 {'name': 'expires-in', 'default': 3600,
+                  'cli_type_name': 'integer',
+                  'help_text': (
+                      'Number of seconds until the pre-signed '
+                      'URL expires.  Default is 3600 seconds.')}]
+
+    def _run_main(self, parsed_args, parsed_globals):
+        super(PresignCommand, self)._run_main(parsed_args, parsed_globals)
+        path = parsed_args.path
+        if path.startswith('s3://'):
+            path = path[5:]
+        bucket, key = find_bucket_key(path)
+        url = self.client.generate_presigned_url(
+            'get_object',
+            {'Bucket': bucket, 'Key': key},
+            ExpiresIn=parsed_args.expires_in
+        )
+        uni_print(url)
+        uni_print('\n')
+        return 0
+
+
 class S3TransferCommand(S3Command):
     def _run_main(self, parsed_args, parsed_globals):
         super(S3TransferCommand, self)._run_main(parsed_args, parsed_globals)
diff --git a/awscli/testutils.py b/awscli/testutils.py
--- a/awscli/testutils.py
+++ b/awscli/testutils.py
@@ -307,6 +307,7 @@ class BaseAWSCommandParamsTest(unittest.TestCase):
 
     def setUp(self):
         self.last_params = {}
+        self.last_kwargs = None
         # awscli/__init__.py injects AWS_DATA_PATH at import time
         # so that we can find cli.json.  This might be fixed in the
         # future, but for now we just grab that value out of the real
@@ -317,6 +318,8 @@ def setUp(self):
             'AWS_DEFAULT_REGION': 'us-east-1',
             'AWS_ACCESS_KEY_ID': 'access_key',
             'AWS_SECRET_ACCESS_KEY': 'secret_key',
+            'AWS_CONFIG_FILE': '',
+            'AWS_SHARED_CREDENTIALS_FILE': '',
         }
         self.environ_patch = mock.patch('os.environ', self.environ)
         self.environ_patch.start()


Test patch:
diff --git a/tests/functional/s3/test_presign_command.py b/tests/functional/s3/test_presign_command.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/s3/test_presign_command.py
@@ -0,0 +1,187 @@
+# Copyright 2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+#     http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+import datetime
+
+from botocore.compat import urlsplit
+from awscli.testutils import BaseAWSCommandParamsTest, mock, temporary_file
+from awscli.testutils import create_clidriver
+
+
+# Values used to fix time.time() and datetime.datetime.utcnow()
+# so we know the exact values of the signatures generated.
+FROZEN_TIMESTAMP = 1471305652
+DEFAULT_EXPIRES = 3600
+FROZEN_TIME = mock.Mock(return_value=FROZEN_TIMESTAMP)
+FROZEN_DATETIME = mock.Mock(
+    return_value=datetime.datetime(2016, 8, 18, 14, 33, 3, 0))
+
+
+class TestPresignCommand(BaseAWSCommandParamsTest):
+
+    prefix = 's3 presign '
+
+    def enable_addressing_mode_in_config(self, fileobj, mode):
+        fileobj.write(
+            "[default]\n"
+            "s3 =\n"
+            "    addressing_style = %s\n" % mode
+        )
+        fileobj.flush()
+        self.environ['AWS_CONFIG_FILE'] = fileobj.name
+        self.driver = create_clidriver()
+
+    def enable_sigv4_from_config_file(self, fileobj):
+        fileobj.write(
+            "[default]\n"
+            "s3 =\n"
+            "    signature_version = s3v4\n"
+        )
+        fileobj.flush()
+        self.environ['AWS_CONFIG_FILE'] = fileobj.name
+        self.driver = create_clidriver()
+
+    def assert_presigned_url_matches(self, actual_url, expected_match):
+        """Verify generated presigned URL matches expected dict.
+
+        This method compares an actual URL against a dict of expected
+        values.  The reason that the "expected_match" is a dict instead
+        of the expected presigned URL is because the query params
+        are unordered so we can't guarantee an expected query param
+        ordering.
+
+        """
+        parts = urlsplit(actual_url)
+        self.assertEqual(parts.netloc, expected_match['hostname'])
+        self.assertEqual(parts.path, expected_match['path'])
+        query_params = self.parse_query_string(parts.query)
+        self.assertEqual(query_params, expected_match['query_params'])
+
+    def parse_query_string(self, query_string):
+        pairs = []
+        for part in query_string.split('&'):
+            pairs.append(part.split('=', 1))
+        return dict(pairs)
+
+    def get_presigned_url_for_cmd(self, cmdline):
+        with mock.patch('time.time', FROZEN_TIME):
+            with mock.patch('datetime.datetime') as d:
+                d.utcnow = FROZEN_DATETIME
+                stdout = self.assert_params_for_cmd(cmdline, None)[0].strip()
+                return stdout
+
+    def test_generates_a_url(self):
+        stdout = self.get_presigned_url_for_cmd(
+            self.prefix + 's3://bucket/key')
+
+        self.assert_presigned_url_matches(
+            stdout, {
+                'hostname': 'bucket.s3.amazonaws.com',
+                'path': '/key',
+                'query_params': {
+                    'AWSAccessKeyId': 'access_key',
+                    'Expires': str(FROZEN_TIMESTAMP + DEFAULT_EXPIRES),
+                    'Signature': '2m9M0eLB%2BqI0nUpkyTskKmHd0Ig%3D',
+                }
+            }
+        )
+
+    def test_handles_non_dns_compatible_buckets(self):
+        stdout = self.get_presigned_url_for_cmd(
+            self.prefix + 's3://bucket.dots/key')
+
+        self.assert_presigned_url_matches(
+            stdout, {
+                'hostname': 's3.amazonaws.com',
+                'path': '/bucket.dots/key',
+                'query_params': {
+                    'AWSAccessKeyId': 'access_key',
+                    'Expires': str(FROZEN_TIMESTAMP + DEFAULT_EXPIRES),
+                    'Signature': '0IiC2vxub438EVcKfEFEMHuoHRw%3D',
+                }
+            }
+        )
+
+    def test_handles_expires_in(self):
+        expires_in = 1000
+        stdout = self.get_presigned_url_for_cmd(
+            self.prefix + 's3://bucket/key --expires-in %s' % expires_in)
+
+        self.assert_presigned_url_matches(
+            stdout, {
+                'hostname': 'bucket.s3.amazonaws.com',
+                'path': '/key',
+                'query_params': {
+                    'AWSAccessKeyId': 'access_key',
+                    'Expires': str(FROZEN_TIMESTAMP + expires_in),
+                    'Signature': 'WZEMcfBNlzfTZBq3bOvYef1cfoU%3D',
+                }
+            }
+        )
+
+    def test_handles_sigv4(self):
+        with temporary_file('w') as f:
+            self.enable_sigv4_from_config_file(f)
+            stdout = self.get_presigned_url_for_cmd(
+                self.prefix + 's3://bucket/key')
+
+        expected = {
+            'hostname': 's3.amazonaws.com',
+            'path': '/bucket/key',
+            'query_params': {
+                'X-Amz-Algorithm': 'AWS4-HMAC-SHA256',
+                'X-Amz-Credential': (
+                    'access_key%2F20160818%2Fus-east-1'
+                    '%2Fs3%2Faws4_request'),
+                'X-Amz-Date': '20160818T143303Z',
+                'X-Amz-Expires': '3600',
+                'X-Amz-Signature': (
+                    'd20178280d7521b384730c678549f6344401ae040bec559ad0602'
+                    '0854c6c718f'),
+                'X-Amz-SignedHeaders': 'host'
+            }
+        }
+        self.assert_presigned_url_matches(stdout, expected)
+
+    def test_s3_prefix_not_needed(self):
+        # Consistent with the 'ls' command.
+        stdout = self.get_presigned_url_for_cmd(
+            self.prefix + 'bucket/key')
+
+        self.assert_presigned_url_matches(
+            stdout, {
+                'hostname': 'bucket.s3.amazonaws.com',
+                'path': '/key',
+                'query_params': {
+                    'AWSAccessKeyId': 'access_key',
+                    'Expires': str(FROZEN_TIMESTAMP + DEFAULT_EXPIRES),
+                    'Signature': '2m9M0eLB%2BqI0nUpkyTskKmHd0Ig%3D',
+                }
+            }
+        )
+
+    def test_can_support_addressing_mode_config(self):
+        with temporary_file('w') as f:
+            self.enable_addressing_mode_in_config(f, 'path')
+            stdout = self.get_presigned_url_for_cmd(
+                self.prefix + 's3://bucket/key')
+        self.assert_presigned_url_matches(
+            stdout, {
+                'hostname': 's3.amazonaws.com',
+                'path': '/bucket/key',
+                'query_params': {
+                    'AWSAccessKeyId': 'access_key',
+                    'Expires': str(FROZEN_TIMESTAMP + DEFAULT_EXPIRES),
+                    'Signature': '2m9M0eLB%2BqI0nUpkyTskKmHd0Ig%3D',
+                }
+            }
+        )
diff --git a/tests/integration/customizations/s3/test_plugin.py b/tests/integration/customizations/s3/test_plugin.py
--- a/tests/integration/customizations/s3/test_plugin.py
+++ b/tests/integration/customizations/s3/test_plugin.py
@@ -29,7 +29,7 @@
 import copy
 import logging
 
-from awscli.compat import six
+from awscli.compat import six, urlopen
 from nose.plugins.attrib import attr
 import botocore.session
 
@@ -2089,3 +2089,16 @@ def test_smoke_sync_sse_c(self):
         self.assertTrue(os.path.isfile(file_name))
         with open(file_name, 'r') as f:
             self.assertEqual(f.read(), contents)
+
+
+class TestPresignCommand(BaseS3IntegrationTest):
+
+    def test_can_retrieve_presigned_url(self):
+        bucket_name = _SHARED_BUCKET
+        original_contents = b'this is foo.txt'
+        self.put_object(bucket_name, 'foo.txt', original_contents)
+        p = aws('s3 presign s3://%s/foo.txt' % (bucket_name,))
+        self.assert_no_errors(p)
+        url = p.stdout.strip()
+        contents = urlopen(url).read()
+        self.assertEqual(contents, original_contents)