Repository: serverless/serverless (JavaScript)
Date: 2017-05-30 08:13:56+00:00
Task type: Bug Fix

Issue:
S3 implementation fails sometimes due a CloudFormation race condition
# This is a Bug Report

## Description

In our integration tests we sometimes see errors. One of them is caused by the S3 integration test suite examples. This problem didn't occur earlier, but that only might have been by luck, because everything was running in serial and that means less load.

```
.......Serverless: Deployment failed!
  Serverless Error ---------------------------------------
 
     An error occurred while provisioning your stack: S3BucketTest1423445441836401
     - Unable to validate the following destination configurations.
 
  Get Support --------------------------------------------
``` 

Related read:
- https://forums.aws.amazon.com/thread.jspa?threadID=167470
- https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-destination-s3/

For feature proposals:
* What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.
* If there is additional config how would it look

I'm only seeing it for these two tests:
- multiple-events-multiple-functions-multiple-buckets
- multiple-events-multiple-functions-single-bucket

## Additional Data

* ***Serverless Framework Version you're using***: 1.4.0
* ***Operating System***: OSX

## Suggestion

We might be able to fix the race-condition using a `dependsOn` as suggested in the forum.



Issue discussion:
I am running into this issue I believe, trying serverless out with 2 functions, 2 buckets, 3 events (2 on one bucket, 1 on another).  Any update on fixing this?
We run into the same situation from time to time too. I believe it is because we have an S3 event on one function but resulting CF script creates the bucket without dependence on that function and I believe CloudFormation sometimes tries to create the bucket earlier then the function itself which results in this error. Adding of `dependsOn` to the bucket resource should fix it.

![cloudformation-template-update-stack json 2017-02-24 19-23-54](https://cloud.githubusercontent.com/assets/215660/23315534/e228aeba-fac6-11e6-9409-883f3c961dee.png)

That sounds reasonable. Thanks for the explanation @JakubMatejka 👍 

/cc @nikgraf @eahefnawy 
Today I faced the same issue on my local machine...

Here's the output I got:

```
Serverless: Stack removal finished...
(node:388) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: channel closed
(node:388) DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
START RequestId: d11871ce-1b90-11e7-b671-fd7c0aca094f Version: $LATEST
END RequestId: d11871ce-1b90-11e7-b671-fd7c0aca094f
REPORT RequestId: d11871ce-1b90-11e7-b671-fd7c0aca094f	Duration: 5.68 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 15 MB	

aws:snshello worldServerless: Getting all objects in S3 bucket...
Serverless: Removing objects in S3 bucket...
Serverless: Removing Stack...
Serverless: Checking Stack removal progress...
START RequestId: d5ba340c-1b90-11e7-8e16-516a8f203711 Version: $LATEST
END RequestId: d5ba340c-1b90-11e7-8e16-516a8f203711
REPORT RequestId: d5ba340c-1b90-11e7-8e16-516a8f203711	Duration: 8.35 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 15 MB	

START RequestId: d5b1a898-1b90-11e7-8108-a97aa445a5f5 Version: $LATEST
END RequestId: d5b1a898-1b90-11e7-8108-a97aa445a5f5
REPORT RequestId: d5b1a898-1b90-11e7-8108-a97aa445a5f5	Duration: 3.11 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 15 MB	

{"version":"0","id":"fe921f6c-cc35-48d5-83c5-ad9b793077fe","detail-type":"serverlessDetailType","source":"serverless.testapp2","account":"853749767834","time":"2017-04-07T12:50:52Z","region":"us-east-1","resources":[],"detail":{"key1":"value1"}}{"version":"0","id":"d121f159-b42b-493e-9236-2fe60749be30","detail-type":"serverlessDetailType","source":"serverless.testapp1","account":"853749767834","time":"2017-04-07T12:50:52Z","region":"us-east-1","resources":[],"detail":{"key1":"value1"}}....START RequestId: d5b181ed-1b90-11e7-810b-a1acca5185a4 Version: $LATEST
END RequestId: d5b181ed-1b90-11e7-810b-a1acca5185a4
REPORT RequestId: d5b181ed-1b90-11e7-810b-a1acca5185a4	Duration: 3.12 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 7 MB	

START RequestId: d5ba3429-1b90-11e7-963f-611e7a261854 Version: $LATEST
END RequestId: d5ba3429-1b90-11e7-963f-611e7a261854
REPORT RequestId: d5ba3429-1b90-11e7-963f-611e7a261854	Duration: 14.37 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 16 MB	

{"version":"0","id":"d121f159-b42b-493e-9236-2fe60749be30","detail-type":"serverlessDetailType","source":"serverless.testapp1","account":"853749767834","time":"2017-04-07T12:50:52Z","region":"us-east-1","resources":[],"detail":{"key1":"value1"}}{"version":"0","id":"fe921f6c-cc35-48d5-83c5-ad9b793077fe","detail-type":"serverlessDetailType","source":"serverless.testapp2","account":"853749767834","time":"2017-04-07T12:50:52Z","region":"us-east-1","resources":[],"detail":{"key1":"value1"}}.Serverless: Getting all objects in S3 bucket...
Serverless: Removing objects in S3 bucket...
Serverless: Removing Stack...
Serverless: Checking Stack removal progress...
........................
Serverless: Stack removal finished...
(node:1693) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: channel closed
(node:1693) DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
...............
Serverless: Stack removal finished...
(node:401) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: channel closed
(node:401) DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

```
I have now hit this too...
Any way to get around this?
@mic159 unfortunately not yet. How does your stack trace look like?
The workaround for me was to just run the deploy 4 times, and on the 4th time it worked.

I'm not sure how to get a stack trace, but the error message I get is the same as in the original report.
> The workaround for me was to just run the deploy 4 times, and on the 4th time it worked.

Yes, that's also something we've faced. Pretty undeterministic 🤔 

You can export the `SLS_DEBUG=*` environment variable to see the Serverless stack trace.
I've seen this when creating a bucket and SNS topics within resources after removing the policy and depensOn. I was able to reproduce it consistently, then added the policy back and the error went away.

I found this issue and tried the `multiple-events-multiple-functions-single-bucket` test and got the same error, sadly I forgot the `--verbose` flag and removed the stack afterwards, so no events logs for me. But I was able to reproduce it by adding `depensOn = ["FooBucket"]` to one of the permissions, so it's highly likely the cause is the missing `depensOn = ["FooLambdaPermission"]` on the bucket. My 2c.
Great! Thank you very much for looking into this @nitely! 👍 

That's really helpful and sounds reasonable. We'll look into it again and try to fix it based on your findings. 💯 
@nitely I tried to reproduce this, but unfortunately (of fortunately) I couldn't do so.

Could you provide a `serverless.yml` and maybe quick step-by-step guide on how to do so? That would be really helpful! 👍 
Well it is a RC, if you try to reproduce by just deploying it should occur sooner or later. This is how I reproduce it (by forcing the permission to be created before the bucket):

* `$ cd ./tests/integration/aws/s3/multiple-events-multiple-functions-single-bucket/service/`
* `$ export BUCKET_1="foobar.pmuens"`
* `$ sls package`
* Edit `.serverless/cloudformation-template-update-stack.json` and add `"DependsOn": ["S3BucketFoobarpmuens"],` into `CreateLambdaPermissionFoobarpmuensS3`. Like this:

      "CreateLambdaPermissionFoobarpmuensS3": {
      "DependsOn": [
        "S3BucketFoobarpmuens"
      ],
      "Type": "AWS::Lambda::Permission",
      "Properties": {
        "FunctionName": {
          "Fn::GetAtt": [
            "CreateLambdaFunction",
            "Arn"
          ]
        },
        "Action": "lambda:InvokeFunction",
        "Principal": "s3.amazonaws.com",
        "SourceArn": {
          "Fn::Join": [
            "",
            [
              "arn:aws:s3:::foobar.pmuens"
            ]
          ]
        }
      }
      },
* Upload the CF template manually (through aws console or cli)
* Check the Events log, you should find the S3 error mentioned by the OP
@nitely gotcha! Thanks for the steps to reproduce it!

I've translated it into this `serverless.yml` file which reproduces the bug correctly:

```yml
service: service

provider:
  name: aws
  runtime: nodejs6.10

functions:
  hello:
    handler: handler.hello
    events:
      - s3: foobar.pmuens

 resources:
   Resources:
     HelloLambdaPermissionFoobarpmuensS3:
       DependsOn:
         - S3BucketFoobarpmuens

     # having this (bottom) should resolve this problem
     # S3BucketFoobarpmuens:
     #   DependsOn:
     #     - HelloLambdaPermissionFoobarpmuensS3
```

I'm currently working on a fix and will PR smth. soon...

Solution patch:
diff --git a/lib/plugins/aws/package/compile/events/s3/index.js b/lib/plugins/aws/package/compile/events/s3/index.js
index 6ad7b7c3619..707d84aaeb9 100644
--- a/lib/plugins/aws/package/compile/events/s3/index.js
+++ b/lib/plugins/aws/package/compile/events/s3/index.js
@@ -137,8 +137,21 @@ class AwsCompileS3Events {
             LambdaConfigurations: bucketLambdaConfiguration,
           },
         },
+        DependsOn: [],
       };
 
+      // create the DependsOn properties for the buckets permissions (which are created later on)
+      const dependsOnToCreate = s3EnabledFunctions
+        .filter(func => func.bucketName === bucketName);
+
+      _.forEach(dependsOnToCreate, (item) => {
+        const lambdaPermissionLogicalId = this.provider.naming
+          .getLambdaS3PermissionLogicalId(item.functionName,
+            item.bucketName);
+
+        bucketTemplate.DependsOn.push(lambdaPermissionLogicalId);
+      });
+
       const bucketLogicalId = this.provider.naming
         .getBucketLogicalId(bucketName);
       const bucketCFResource = {


Test patch:
diff --git a/lib/plugins/aws/package/compile/events/s3/index.test.js b/lib/plugins/aws/package/compile/events/s3/index.test.js
index ebb8568f402..00c085d83b1 100644
--- a/lib/plugins/aws/package/compile/events/s3/index.test.js
+++ b/lib/plugins/aws/package/compile/events/s3/index.test.js
@@ -163,6 +163,44 @@ describe('AwsCompileS3Events', () => {
       ).to.equal('AWS::Lambda::Permission');
     });
 
+    it('should add the permission resource logical id to the buckets DependsOn array', () => {
+      awsCompileS3Events.serverless.service.functions = {
+        first: {
+          events: [
+            {
+              s3: 'first-function-bucket-one',
+            },
+            {
+              s3: {
+                bucket: 'first-function-bucket-two',
+              },
+            },
+          ],
+        },
+      };
+
+      awsCompileS3Events.compileS3Events();
+
+      expect(awsCompileS3Events.serverless.service.provider.compiledCloudFormationTemplate
+        .Resources.S3BucketFirstfunctionbucketone.Type
+      ).to.equal('AWS::S3::Bucket');
+      expect(awsCompileS3Events.serverless.service.provider.compiledCloudFormationTemplate
+        .Resources.S3BucketFirstfunctionbuckettwo.Type
+      ).to.equal('AWS::S3::Bucket');
+      expect(awsCompileS3Events.serverless.service.provider.compiledCloudFormationTemplate
+        .Resources.FirstLambdaPermissionFirstfunctionbucketoneS3.Type
+      ).to.equal('AWS::Lambda::Permission');
+      expect(awsCompileS3Events.serverless.service.provider.compiledCloudFormationTemplate
+        .Resources.FirstLambdaPermissionFirstfunctionbuckettwoS3.Type
+      ).to.equal('AWS::Lambda::Permission');
+      expect(awsCompileS3Events.serverless.service.provider.compiledCloudFormationTemplate
+        .Resources.S3BucketFirstfunctionbucketone.DependsOn
+      ).to.deep.equal(['FirstLambdaPermissionFirstfunctionbucketoneS3']);
+      expect(awsCompileS3Events.serverless.service.provider.compiledCloudFormationTemplate
+        .Resources.S3BucketFirstfunctionbuckettwo.DependsOn
+      ).to.deep.equal(['FirstLambdaPermissionFirstfunctionbuckettwoS3']);
+    });
+
     it('should not create corresponding resources when S3 events are not given', () => {
       awsCompileS3Events.serverless.service.functions = {
         first: {