Repository: localstack/localstack

Issue:
S3 objects larger than 6MB cannot be accessed via the S3 URL (403 forbidden)
Normally S3 objects in localstack can be accessed via `http://localhost:4572/bucket/key`.

However, if the object is 7 MBs or larger, attempting to access it via this URL will result in a 403 error. Objects any smaller are retrieved as expected.

(Possibly related: Lambda response size is limited to 6MB. Perhaps that is being used by accident?)


Discussion:
Thanks for reporting @Jaystified . Can you please provide more details - haven't been able to reproduce this.

Did you add `public-read` ACLs on the object or bucket level, e.g., by running `awslocal s3api put-bucket-acl --bucket buck1 --acl public-read`? If not, then 403 would in fact be the expected response. If we're not returning 403 for smaller objects, that that might be a bug.

Would be great if you could share a few shell commands (using `awslocal`) to reproduce this issue. Thanks!
Commands I'm using to reproduce:
```bash
sudo dd bs=1024 count=6144 skip="$RANDOM" if=/dev/sda of=6mb.bin
sudo dd bs=1024 count=61440 skip="$RANDOM" if=/dev/sda of=60mb.bin
awslocal s3 mb s3://test
awslocal s3api put-bucket-acl --bucket test --acl public-read
awslocal s3 cp 6mb.bin s3://test/6mb.bin
awslocal s3 cp 60mb.bin s3://test/60mb.bin
curl -I localhost:4572/test/6mb.bin # 200
curl -I localhost:4572/test/60mb.bin # 403
```
Here's both the commands and what I'm seeing:
```bash
$ sudo dd bs=1024 count=6144 skip="$RANDOM" if=/dev/sda of=6mb.bin
6144+0 records in
6144+0 records out
6291456 bytes (6.3 MB, 6.0 MiB) copied
$ sudo dd bs=1024 count=61440 skip="$RANDOM" if=/dev/sda of=60mb.bin
61440+0 records in
61440+0 records out
62914560 bytes (63 MB, 60 MiB) copied
$ awslocal s3 mb s3://test
make_bucket: test
$ awslocal s3api put-bucket-acl --bucket test --acl public-read
$ awslocal s3 cp 6mb.bin s3://test/6mb.bin
upload: ./6mb.bin to s3://test/6mb.bin  
$ awslocal s3 cp 60mb.bin s3://test/60mb.bin
upload: ./60mb.bin to s3://test/60mb.bin
$ curl -I localhost:4572/test/6mb.bin
HTTP/1.1 200 OK
Server: BaseHTTP/0.6 Python/3.6.9
Date: Sun, 10 Nov 2019 07:30:07 GMT
Content-Type: application/octet-stream
Content-Md5: 2moNCX4wesUu2bStVRgB/A==
ETag: "da6a0d097e307ac52ed9b4ad551801fc"
last-modified: Sun, 10 Nov 2019 07:29:36 GMT
Content-Length: 6291456
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH
Access-Control-Allow-Headers: authorization,content-type,content-md5,cache-control,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent,x-amz-target,x-amz-acl,x-amz-version-id,x-localstack-target,x-amz-tagging
Access-Control-Expose-Headers: x-amz-version-id
$ curl -I localhost:4572/test/60mb.bin
HTTP/1.1 403 Forbidden
Server: BaseHTTP/0.6 Python/3.6.9
Date: Sun, 10 Nov 2019 07:30:08 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH
Access-Control-Allow-Headers: authorization,content-type,content-md5,cache-control,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent,x-amz-target,x-amz-acl,x-amz-version-id,x-localstack-target,x-amz-tagging
Access-Control-Expose-Headers: x-amz-version-id
```

Resolving patch:
diff --git a/localstack/services/dynamodb/dynamodb_listener.py b/localstack/services/dynamodb/dynamodb_listener.py
--- a/localstack/services/dynamodb/dynamodb_listener.py
+++ b/localstack/services/dynamodb/dynamodb_listener.py
@@ -183,6 +183,11 @@ def return_response(self, method, path, data, headers, response):
                 keys = dynamodb_extract_keys(item=data['Item'], table_name=data['TableName'])
                 if isinstance(keys, Response):
                     return keys
+                # fix response
+                if response._content == '{}':
+                    response._content = json.dumps({'Attributes': data['Item']})
+                    fix_headers_for_updated_response(response)
+                # prepare record keys
                 record['dynamodb']['Keys'] = keys
                 record['dynamodb']['NewImage'] = data['Item']
                 record['dynamodb']['SizeBytes'] = len(json.dumps(data['Item']))
diff --git a/localstack/services/s3/s3_starter.py b/localstack/services/s3/s3_starter.py
--- a/localstack/services/s3/s3_starter.py
+++ b/localstack/services/s3/s3_starter.py
@@ -1,7 +1,9 @@
 import sys
+import types
 import logging
 import traceback
 from moto.s3 import models as s3_models
+from moto.s3 import responses as s3_responses
 from moto.server import main as moto_main
 from localstack import config
 from localstack.constants import DEFAULT_PORT_S3_BACKEND
@@ -11,7 +13,7 @@
     get_service_protocol, start_proxy_for_service, do_run)
 from localstack.utils.bootstrap import setup_logging
 
-LOGGER = logging.getLogger(__name__)
+LOG = logging.getLogger(__name__)
 
 # max file size for S3 objects (in MB)
 S3_MAX_FILE_SIZE_MB = 2048
@@ -26,7 +28,7 @@ def check_s3(expect_shutdown=False, print_error=False):
         out = aws_stack.connect_to_service(service_name='s3').list_buckets()
     except Exception as e:
         if print_error:
-            LOGGER.error('S3 health check failed: %s %s' % (e, traceback.format_exc()))
+            LOG.error('S3 health check failed: %s %s' % (e, traceback.format_exc()))
     if expect_shutdown:
         assert out is None
     else:
@@ -54,6 +56,19 @@ def init(self, name, value, storage='STANDARD', etag=None,
     original_init = s3_models.FakeKey.__init__
     s3_models.FakeKey.__init__ = init
 
+    def s3_key_response_post(self, request, body, bucket_name, query, key_name, *args, **kwargs):
+        result = s3_key_response_post_orig(request, body, bucket_name, query, key_name, *args, **kwargs)
+        if query.get('uploadId'):
+            # fix for https://github.com/localstack/localstack/issues/1733
+            key = self.backend.get_key(bucket_name, key_name)
+            acl = self._acl_from_headers(request.headers) or self.backend.get_bucket(bucket_name).acl
+            key.set_acl(acl)
+        return result
+
+    s3_key_response_post_orig = s3_responses.S3ResponseInstance._key_response_post
+    s3_responses.S3ResponseInstance._key_response_post = types.MethodType(
+        s3_key_response_post, s3_responses.S3ResponseInstance)
+
 
 def main():
     setup_logging()
diff --git a/localstack/utils/aws/aws_stack.py b/localstack/utils/aws/aws_stack.py
--- a/localstack/utils/aws/aws_stack.py
+++ b/localstack/utils/aws/aws_stack.py
@@ -234,15 +234,19 @@ def base64Decode(self, s):
             s = json.dumps(s)
         return base64.b64decode(s)
 
+    def toJson(self, obj):
+        return obj and json.dumps(obj)
 
-def render_velocity_template(template, context, as_json=False):
+
+def render_velocity_template(template, context, variables={}, as_json=False):
     import airspeed
     t = airspeed.Template(template)
-    variables = {
+    var_map = {
         'input': VelocityInput(context),
         'util': VelocityUtil()
     }
-    replaced = t.merge(variables)
+    var_map.update(variables or {})
+    replaced = t.merge(var_map)
     if as_json:
         replaced = json.loads(replaced)
     return replaced


Test patch: