Repository: serverless/serverless (JavaScript)
Date: 2017-01-17 16:26:57+00:00
Task type: Bug Fix

Issue:
New kinesis syntax not working
<!--
1. If you have a question and not a bug/feature request please ask it at http://forum.serverless.com
2. Please check if an issue already exists so there are no duplicates
3. Check out and follow our Guidelines: https://github.com/serverless/serverless/blob/master/CONTRIBUTING.md
4. Fill out the whole template so we have a good overview on the issue
5. Do not remove any section of the template. If something is not applicable leave it empty but leave it in the Issue
6. Please follow the template, otherwise we'll have to ask you to update it
-->
# This is a (Bug Report / Feature Proposal)
## Description

For bug reports:
- What went wrong?
  When I tried to convert my kinesis event mappings to the new syntax, like this:

`````` yaml

  extract:
    handler: extract/index.handler
    events:
      - stream:
          arn:
            Fn::GetAtt:
              - BookmarksStream
              - Arn

I got the following error "EventSourceArn.split is not a function"

Looking at the source code, I get the impression that this feature simply has not been implemented yet. However, it is already referenced in the docs here:

https://serverless.com/framework/docs/providers/aws/events/streams/

* What did you expect should have happened?

The cloudformation GetAtt should have resolved into a valid arn

* What was the config you used?

```yaml

  extract:
    handler: extract/index.handler
    events:
      - stream:
          arn:
            Fn::GetAtt:
              - BookmarksStream
              - Arn

* What stacktrace or error message from your provider did you see?

For feature proposals:
* What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.
* If there is additional config how would it look

Similar or dependent issues:
* #12345

## Additional Data

* ***Serverless Framework Version you're using***:
1.0.2 

* ***Operating System***:

linux (kubuntu)
* ***Stack Trace***:

TypeError: EventSourceArn.split is not a function
    at functionObj.events.forEach.event (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/lib/plugins/aws/deploy/compile/events/stream/index.js:82:47)
    at Array.forEach (native)
    at serverless.service.getAllFunctions.forEach (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/lib/plugins/aws/deploy/compile/events/stream/index.js:20:28)
    at Array.forEach (native)
    at AwsCompileStreamEvents.compileStreamEvents (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/lib/plugins/aws/deploy/compile/events/stream/index.js:16:47)
    at BbPromise.reduce (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/lib/classes/PluginManager.js:151:50)
    at tryCatcher (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/node_modules/bluebird/js/release/util.js:16:23)
    at Object.gotValue (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/node_modules/bluebird/js/release/reduce.js:157:18)
    at Object.gotAccum (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/node_modules/bluebird/js/release/reduce.js:144:25)
    at Object.tryCatcher (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:510:31)
    at Promise._settlePromise (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:567:18)
    at Promise._settlePromise0 (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:612:10)
    at Promise._settlePromises (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:691:18)
    at Async._drainQueue (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:138:16)
    at Async._drainQueues (/home/yeehaa/.nvm/versions/node/v6.8.0/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:148:10)

* ***Provider Error messages***:

EventSourceArn.split is not a function
``````



Issue discussion:
Hey @yeehaa123 thanks for reaching out!
This is definitely a bug.

The solution right now is to only use ARNs instead of using `Fn::GetAtt`.

We'll look into this!

Hi Philipp,

Thanks for the quick reply.

yes, I already went back to that. Let me know, if you need help with solving this. I could attempt to make the PR if you give me a short hint on how to implement it (or give me a link to another place in the code where you resolve the FN:GetAtt...

Thanks for all the awesome work. This project is amazing!

JH

Hi I got this issue as well - just to clarify, when you say 'The solution right now is to only use ARNs' can you elaborate please with example?

thanks!

@kurtmaile Here's the link on how to setup streams with ARNs:
https://serverless.com/framework/docs/providers/aws/events/streams/

@yeehaa123 Thanks for offering help here! Let me shed some more light into this bug:

The bug is a problem with the cause that we treat DynamoDB and Kinesis streams equally. For the CloudFormation we need the type (`dynamodb` or `kinesis` and the name of the stream). That's where it breaks as we're currently extracting that information from the ARN (and cannot access it from the `Fn::GetAtt`).

@pmuens okay, got it. But I'm not sure how to proceed here. As far as I can see, there are two possible solutions:
1. The stream type needs to be passed in explicitly like this:

``` yaml
- stream:
    type: dynamodb
    arn: ...
```

or this

``` yaml
- db-stream:
    arn:...
```
1. The ARN gets somehow resolved earlier. I'm not quite sure if that's even possible, because cloudformation then needs to be called twice somehow.

Obviously, 2 seems the better option here, but I am unsure how to approach this.

The fact that db-streams need the timestamp at the end does not make things easier either....

@yeehaa123 thanks for getting back. Yes, those are basically two valid options. It all comes down to the naming of the `ResourceLogicalId`. We don't need the type or the name of the string for the stream itself. Only for the resource name.

@eahefnawy will tackle this bug and can help here so he might have an idea how we can proceed here. Personally I'd like to keep it as it is and let Serverless auto detect the stream. Otherwise we might have a problem with the string syntax where we only provide the ARN and don't have a way to pass the type.

@pmuens Thanks!

@eahefnawy let me know if you need any feedback on this

The docs at https://serverless.com/framework/docs/providers/aws/events/streams/ continue to declare that the object notation can be used:

```
functions:
  compute:
    handler: handler.compute
    events:
      - stream: arn:aws:dynamodb:region:XXXXXX:table/foo/stream/1901T00:00:00.000
      - stream:
          arn:
            Fn::GetAtt:
              - MyKinesisStream
              - Arn
```
Thanks @erikerikson.

Yes, this is still something which needs to be fixed (docs and code) ASAP!
Ah ha!  This one was a bit painful ha.  Updating the documentation would make me feel like I'm not going insane :)

My (super hoodrat) workaround was to create a custom variable with the first part of my kinesis shard ARN, then prepend it with the stage. (and make sure the streams are also named that way)

Works, but going to take a stab at doing a PR :)


Just PRd a tiny doc update which removes the `Fn::GetAtt` usage --> https://github.com/serverless/serverless/pull/2962
Thanks @pmuens.  I still pine for this (e.g. `Fn::ImportValue`).  😄 
Also hitting this. The logical name could be, instead of `function+type+streamName` could be `function+type+table/stream` where table/stream is either:

1. The stream name (extracted with regexes as-is now)
2. The logical table or stream ID grabbed from the `Fn::GetAtt` statement
3. In the case of `Fn::ImportValue` the name of the variable in the ImportValue statement

That way, all the ways of specifying a stream in CFN (hardcoded, GetAtt, and ImportValue) can be properly named. Right now, the naive naming is [here](https://github.com/serverless/serverless/blob/master/lib/plugins/aws/lib/naming.js#L206-L212). 

Downside note: That may mean we have to, for GetAtt/ImportValue use cases, require a `type: ` that would tell us whether the ARN points to a Dynamo or Kinesis stream.
Here's a sample patch:

```{diff}
diff --git a/lib/plugins/aws/deploy/compile/events/stream/index.js b/lib/plugins/aws/deploy/compile/events/stream/index.js
index d18b53f..f0969b7 100644
--- a/lib/plugins/aws/deploy/compile/events/stream/index.js
+++ b/lib/plugins/aws/deploy/compile/events/stream/index.js
@@ -78,8 +78,16 @@ class AwsCompileStreamEvents {
                 .Error(errorMessage);
             }
 
-            const streamType = EventSourceArn.split(':')[2];
-            const streamName = EventSourceArn.split('/')[1];
+            const streamType = event.stream.type || EventSourceArn.split(':')[2];
+            if (typeof EventSourceArn === 'string') {
+                const streamName = EventSourceArn.split('/')[1];
+                // we need to add quotes to the string if it's not a GetAtt or other expression
+                EventSourceArn = `"${EventSourceArn}"`;
+            } else if (EventSourceArn['Fn::GetAtt']) {
+                const streamName = EventSourceArn['Fn::GetAtt'][0];
+            } else if (EventSourceArn['Fn::ImportValue']) {
+                const streamName = EventSourceArn['Fn::ImportValue'];
+            }
 
             const lambdaLogicalId = this.provider.naming
               .getLambdaLogicalId(functionName);
@@ -99,7 +107,7 @@ class AwsCompileStreamEvents {
                 "DependsOn": ${dependsOn},
                 "Properties": {
                   "BatchSize": ${BatchSize},
-                  "EventSourceArn": "${EventSourceArn}",
+                  "EventSourceArn": ${EventSourceArn},
                   "FunctionName": {
                     "Fn::GetAtt": [
                       "${lambdaLogicalId}",
```

Solution patch:
diff --git a/docs/providers/aws/events/streams.md b/docs/providers/aws/events/streams.md
index 9e506e3de45..eaf9013cd01 100644
--- a/docs/providers/aws/events/streams.md
+++ b/docs/providers/aws/events/streams.md
@@ -12,7 +12,11 @@ layout: Doc
 
 # DynamoDB / Kinesis Streams
 
-This setup specifies that the `compute` function should be triggered whenever the corresponding DynamoDB table is modified (e.g. a new entry is added).
+This setup specifies that the `compute` function should be triggered whenever:
+  1. the corresponding DynamoDB table is modified (e.g. a new entry is added).
+  2. the Lambda checkpoint has not reached the end of the Kinesis stream (e.g. a new record is added).
+
+The ARN for the stream can be specified as a string, the reference to the ARN of a resource by logical ID, or the import of an ARN that was exported by a different service or CloudFormation stack.
 
 **Note:** The `stream` event will hook up your existing streams to a Lambda function. Serverless won't create a new stream for you.
 
@@ -22,6 +26,26 @@ functions:
     handler: handler.compute
     events:
       - stream: arn:aws:dynamodb:region:XXXXXX:table/foo/stream/1970-01-01T00:00:00.000
+      - stream:
+          type: dynamodb
+          arn:
+            Fn::GetAtt:
+              - MyDynamoDbTable
+              - StreamArn
+      - stream:
+          type: dynamodb
+          arn:
+            Fn::ImportValue: MyExportedDynamoDbStreamArnId
+      - stream:
+          type: kinesis
+          arn:
+            Fn::GetAtt:
+              - MyKinesisStream
+              - Arn
+      - stream:
+          type: kinesis
+          arn:
+            Fn::ImportValue: MyExportedKinesisStreamArnId
 ```
 
 ## Setting the BatchSize and StartingPosition
diff --git a/lib/plugins/aws/deploy/compile/events/stream/index.js b/lib/plugins/aws/deploy/compile/events/stream/index.js
index ec14f4ed1e9..1b6d44e3328 100644
--- a/lib/plugins/aws/deploy/compile/events/stream/index.js
+++ b/lib/plugins/aws/deploy/compile/events/stream/index.js
@@ -57,6 +57,31 @@ class AwsCompileStreamEvents {
                 throw new this.serverless.classes
                   .Error(errorMessage);
               }
+              if (typeof event.stream.arn !== 'string') {
+                // for dynamic arns (GetAtt/ImportValue)
+                if (!event.stream.type) {
+                  const errorMessage = [
+                    `Missing "type" property for stream event in function "${functionName}"`,
+                    ' If the "arn" property on a stream is a complex type (such as Fn::GetAtt)',
+                    ' then a "type" must be provided for the stream, either "kinesis" or,',
+                    ' "dynamodb". Please check the docs for more info.',
+                  ].join('');
+                  throw new this.serverless.classes
+                    .Error(errorMessage);
+                }
+                if (Object.keys(event.stream.arn).length !== 1
+                    || !(_.has(event.stream.arn, 'Fn::ImportValue')
+                          || _.has(event.stream.arn, 'Fn::GetAtt'))) {
+                  const errorMessage = [
+                    `Bad dynamic ARN property on stream event in function "${functionName}"`,
+                    ' If you use a dynamic "arn" (such as with Fn::GetAtt or Fn::ImportValue)',
+                    ' there must only be one key (either Fn::GetAtt or Fn::ImportValue) in the arn',
+                    ' object. Please check the docs for more info.',
+                  ].join('');
+                  throw new this.serverless.classes
+                    .Error(errorMessage);
+                }
+              }
               EventSourceArn = event.stream.arn;
               BatchSize = event.stream.batchSize
                 || BatchSize;
@@ -78,8 +103,15 @@ class AwsCompileStreamEvents {
                 .Error(errorMessage);
             }
 
-            const streamType = EventSourceArn.split(':')[2];
-            const streamName = EventSourceArn.split('/')[1];
+            const streamType = event.stream.type || EventSourceArn.split(':')[2];
+            const streamName = (function () {
+              if (EventSourceArn['Fn::GetAtt']) {
+                return EventSourceArn['Fn::GetAtt'][0];
+              } else if (EventSourceArn['Fn::ImportValue']) {
+                return EventSourceArn['Fn::ImportValue'];
+              }
+              return EventSourceArn.split('/')[1];
+            }());
 
             const lambdaLogicalId = this.provider.naming
               .getLambdaLogicalId(functionName);
@@ -112,7 +144,7 @@ class AwsCompileStreamEvents {
                 "DependsOn": ${dependsOn},
                 "Properties": {
                   "BatchSize": ${BatchSize},
-                  "EventSourceArn": "${EventSourceArn}",
+                  "EventSourceArn": ${JSON.stringify(EventSourceArn)},
                   "FunctionName": {
                     "Fn::GetAtt": [
                       "${lambdaLogicalId}",


Test patch:
diff --git a/lib/plugins/aws/deploy/compile/events/stream/index.test.js b/lib/plugins/aws/deploy/compile/events/stream/index.test.js
index d6a6f38ef91..0ebc32de75e 100644
--- a/lib/plugins/aws/deploy/compile/events/stream/index.test.js
+++ b/lib/plugins/aws/deploy/compile/events/stream/index.test.js
@@ -340,6 +340,102 @@ describe('AwsCompileStreamEvents', () => {
         ).to.equal('True');
       });
 
+      it('should allow specifying DynamoDB and Kinesis streams as CFN reference types', () => {
+        awsCompileStreamEvents.serverless.service.functions = {
+          first: {
+            events: [
+              {
+                stream: {
+                  arn: { 'Fn::GetAtt': ['SomeDdbTable', 'StreamArn'] },
+                  type: 'dynamodb',
+                },
+              },
+              {
+                stream: {
+                  arn: { 'Fn::ImportValue': 'ForeignKinesis' },
+                  type: 'kinesis',
+                },
+              },
+            ],
+          },
+        };
+
+        awsCompileStreamEvents.compileStreamEvents();
+
+        // dynamodb version
+        expect(awsCompileStreamEvents.serverless.service
+          .provider.compiledCloudFormationTemplate.Resources
+          .FirstEventSourceMappingDynamodbSomeDdbTable.Properties.EventSourceArn
+        ).to.deep.equal(
+          { 'Fn::GetAtt': ['SomeDdbTable', 'StreamArn'] }
+        );
+        expect(awsCompileStreamEvents.serverless.service
+          .provider.compiledCloudFormationTemplate.Resources.IamRoleLambdaExecution
+          .Properties.Policies[0].PolicyDocument.Statement[0]
+        ).to.deep.equal(
+          {
+            Action: [
+              'dynamodb:GetRecords',
+              'dynamodb:GetShardIterator',
+              'dynamodb:DescribeStream',
+              'dynamodb:ListStreams',
+            ],
+            Effect: 'Allow',
+            Resource: [
+              {
+                'Fn::GetAtt': [
+                  'SomeDdbTable',
+                  'StreamArn',
+                ],
+              },
+            ],
+          }
+        );
+        // and now kinesis
+        expect(awsCompileStreamEvents.serverless.service
+          .provider.compiledCloudFormationTemplate.Resources
+          .FirstEventSourceMappingKinesisForeignKinesis.Properties.EventSourceArn
+        ).to.deep.equal(
+          { 'Fn::ImportValue': 'ForeignKinesis' }
+        );
+      });
+
+      it('fails if Fn::GetAtt/dynamic stream ARN is used without a type', () => {
+        awsCompileStreamEvents.serverless.service.functions = {
+          first: {
+            events: [
+              {
+                stream: {
+                  arn: { 'Fn::GetAtt': ['SomeDdbTable', 'StreamArn'] },
+                },
+              },
+            ],
+          },
+        };
+
+        expect(() => awsCompileStreamEvents.compileStreamEvents()).to.throw(Error);
+      });
+
+      it('fails if keys other than Fn::GetAtt/ImportValue are used for dynamic stream ARN', () => {
+        awsCompileStreamEvents.serverless.service.functions = {
+          first: {
+            events: [
+              {
+                stream: {
+                  type: 'dynamodb',
+                  arn: {
+                    'Fn::GetAtt': ['SomeDdbTable', 'StreamArn'],
+                    batchSize: 1,
+                  },
+                },
+              },
+            ],
+          },
+        };
+
+        expect(() => awsCompileStreamEvents.compileStreamEvents()).to.throw(Error);
+      });
+
       it('should add the necessary IAM role statements', () => {
         awsCompileStreamEvents.serverless.service.functions = {
           first: {