Repository: serverless/serverless (JavaScript)
Date: 2019-04-29 17:04:06+00:00
Task type: Feature

Issue:
Allow Fn::Join in stream event source
<!--
1. If you have a question and not a feature request please ask first at http://forum.serverless.com
2. Please check if an issue already exists. This feature may have already been requested
3. Check out and follow our Guidelines: https://github.com/serverless/serverless/blob/master/CONTRIBUTING.md
4. Fill out the whole template so we have a good overview on the issue
5. Do not remove any section of the template. If something is not applicable leave it empty but leave it in the Issue
6. Please follow the template, otherwise we'll have to ask you to update it
-->

# This is a Feature Proposal

## Description

I have a use case where I want to configure a lambda to read from a Kinesis stream with a different ARN based on account ID. This lambda will be deployed across different staging and production environments, and using the built in references for account ID simplifies config greatly.

Example config wanted:
```yml
events:
  - stream
      type: kinesis
      arn:
        Fn::Join:
          - ":"
          - - arn
            - aws
            - kinesis
            - Ref: AWS::Region
            - Ref: AWS::AccountId
            - MyOtherKinesisStream
```

This is currently not possible because of a filter only allowing the `Fn::GetAttr` and `Fn::ImportValue` methods to be used in stream event definitions.

Similar or dependent issues:

This is almost identical to an issue raised last year for SQS event sources.
* https://github.com/serverless/serverless/issues/5345

Following the PR (https://github.com/serverless/serverless/pull/5351) closing the above issue, I have a fork with the necessary changes ready to create a PR off of if there is a want for this feature: https://github.com/serverless/serverless/compare/master...Tybot204:master


Issue discussion:

Solution patch:
diff --git a/docs/providers/aws/events/streams.md b/docs/providers/aws/events/streams.md
index 530eefb77d1..2fc9e0b4c63 100644
--- a/docs/providers/aws/events/streams.md
+++ b/docs/providers/aws/events/streams.md
@@ -44,6 +44,17 @@ functions:
           type: kinesis
           arn:
             Fn::ImportValue: MyExportedKinesisStreamArnId
+      - stream
+          type: kinesis
+          arn:
+            Fn::Join:
+              - ":"
+              - - arn
+                - aws
+                - kinesis
+                - Ref: AWS::Region
+                - Ref: AWS::AccountId
+                - stream/MyOtherKinesisStream
 ```
 
 ## Setting the BatchSize and StartingPosition
diff --git a/lib/plugins/aws/package/compile/events/stream/index.js b/lib/plugins/aws/package/compile/events/stream/index.js
index 61820a09445..bf550a7ed92 100644
--- a/lib/plugins/aws/package/compile/events/stream/index.js
+++ b/lib/plugins/aws/package/compile/events/stream/index.js
@@ -70,13 +70,14 @@ class AwsCompileStreamEvents {
                     .Error(errorMessage);
                 }
                 if (Object.keys(event.stream.arn).length !== 1
-                    || !(_.has(event.stream.arn, 'Fn::ImportValue')
-                          || _.has(event.stream.arn, 'Fn::GetAtt'))) {
+                  || !(_.has(event.stream.arn, 'Fn::ImportValue')
+                    || _.has(event.stream.arn, 'Fn::GetAtt')
+                    || _.has(event.stream.arn, 'Fn::Join'))) {
                   const errorMessage = [
                     `Bad dynamic ARN property on stream event in function "${functionName}"`,
-                    ' If you use a dynamic "arn" (such as with Fn::GetAtt or Fn::ImportValue)',
-                    ' there must only be one key (either Fn::GetAtt or Fn::ImportValue) in the arn',
-                    ' object. Please check the docs for more info.',
+                    ' If you use a dynamic "arn" (such as with Fn::GetAtt, Fn::Join',
+                    ' or Fn::ImportValue) there must only be one key (either Fn::GetAtt, Fn::Join',
+                    ' or Fn::ImportValue) in the arn object. Please check the docs for more info.',
                   ].join('');
                   throw new this.serverless.classes
                     .Error(errorMessage);
@@ -109,6 +110,13 @@ class AwsCompileStreamEvents {
                 return EventSourceArn['Fn::GetAtt'][0];
               } else if (EventSourceArn['Fn::ImportValue']) {
                 return EventSourceArn['Fn::ImportValue'];
+              } else if (EventSourceArn['Fn::Join']) {
+                // [0] is the used delimiter, [1] is the array with values
+                const name = EventSourceArn['Fn::Join'][1].slice(-1).pop();
+                if (name.split('/').length) {
+                  return name.split('/').pop();
+                }
+                return name;
               }
               return EventSourceArn.split('/')[1];
             }());


Test patch:
diff --git a/lib/plugins/aws/package/compile/events/stream/index.test.js b/lib/plugins/aws/package/compile/events/stream/index.test.js
index 1571bab4ed8..cd15293da76 100644
--- a/lib/plugins/aws/package/compile/events/stream/index.test.js
+++ b/lib/plugins/aws/package/compile/events/stream/index.test.js
@@ -337,21 +337,21 @@ describe('AwsCompileStreamEvents', () => {
           .Properties.EventSourceArn
         ).to.equal(
           awsCompileStreamEvents.serverless.service.functions.first.events[0]
-          .stream.arn
+            .stream.arn
         );
         expect(awsCompileStreamEvents.serverless.service
           .provider.compiledCloudFormationTemplate.Resources.FirstEventSourceMappingDynamodbFoo
           .Properties.BatchSize
         ).to.equal(
           awsCompileStreamEvents.serverless.service.functions.first.events[0]
-          .stream.batchSize
+            .stream.batchSize
         );
         expect(awsCompileStreamEvents.serverless.service
           .provider.compiledCloudFormationTemplate.Resources.FirstEventSourceMappingDynamodbFoo
           .Properties.StartingPosition
         ).to.equal(
           awsCompileStreamEvents.serverless.service.functions.first.events[0]
-          .stream.startingPosition
+            .stream.startingPosition
         );
         expect(awsCompileStreamEvents.serverless.service
           .provider.compiledCloudFormationTemplate.Resources.FirstEventSourceMappingDynamodbFoo
@@ -372,7 +372,7 @@ describe('AwsCompileStreamEvents', () => {
           .Properties.EventSourceArn
         ).to.equal(
           awsCompileStreamEvents.serverless.service.functions.first.events[1]
-          .stream.arn
+            .stream.arn
         );
         expect(awsCompileStreamEvents.serverless.service
           .provider.compiledCloudFormationTemplate.Resources.FirstEventSourceMappingDynamodbBar
@@ -401,7 +401,7 @@ describe('AwsCompileStreamEvents', () => {
           .Properties.EventSourceArn
         ).to.equal(
           awsCompileStreamEvents.serverless.service.functions.first.events[2]
-          .stream
+            .stream
         );
         expect(awsCompileStreamEvents.serverless.service
           .provider.compiledCloudFormationTemplate.Resources.FirstEventSourceMappingDynamodbBaz
@@ -433,13 +433,30 @@ describe('AwsCompileStreamEvents', () => {
                   type: 'kinesis',
                 },
               },
+              {
+                stream: {
+                  arn: {
+                    'Fn::Join': [
+                      ':', [
+                        'arn', 'aws', 'kinesis', {
+                          Ref: 'AWS::Region',
+                        }, {
+                          Ref: 'AWS::AccountId',
+                        },
+                        'stream/MyStream',
+                      ],
+                    ],
+                  },
+                  type: 'kinesis',
+                },
+              },
             ],
           },
         };
 
         awsCompileStreamEvents.compileStreamEvents();
 
-        // dynamodb version
+        // dynamodb with Fn::GetAtt
         expect(awsCompileStreamEvents.serverless.service
           .provider.compiledCloudFormationTemplate.Resources
           .FirstEventSourceMappingDynamodbSomeDdbTable.Properties.EventSourceArn
@@ -468,13 +485,36 @@ describe('AwsCompileStreamEvents', () => {
             ],
           }
         );
-        // and now kinesis
+
+        // kinesis with Fn::ImportValue
         expect(awsCompileStreamEvents.serverless.service
           .provider.compiledCloudFormationTemplate.Resources
           .FirstEventSourceMappingKinesisForeignKinesis.Properties.EventSourceArn
         ).to.deep.equal(
           { 'Fn::ImportValue': 'ForeignKinesis' }
         );
+
+        // kinesis with Fn::Join
+        expect(awsCompileStreamEvents.serverless.service
+          .provider.compiledCloudFormationTemplate.Resources
+          .FirstEventSourceMappingKinesisMyStream.Properties.EventSourceArn
+        ).to.deep.equal(
+          {
+            'Fn::Join': [
+              ':', [
+                'arn',
+                'aws',
+                'kinesis',
+                {
+                  Ref: 'AWS::Region',
+                }, {
+                  Ref: 'AWS::AccountId',
+                },
+                'stream/MyStream',
+              ],
+            ],
+          }
+        );
       });
 
       it('fails if Fn::GetAtt/dynamic stream ARN is used without a type', () => {
@@ -493,25 +533,26 @@ describe('AwsCompileStreamEvents', () => {
         expect(() => awsCompileStreamEvents.compileStreamEvents()).to.throw(Error);
       });
 
-      it('fails if keys other than Fn::GetAtt/ImportValue are used for dynamic stream ARN', () => {
-        awsCompileStreamEvents.serverless.service.functions = {
-          first: {
-            events: [
-              {
-                stream: {
-                  type: 'dynamodb',
-                  arn: {
-                    'Fn::GetAtt': ['SomeDdbTable', 'StreamArn'],
-                    batchSize: 1,
+      it('fails if keys other than Fn::GetAtt/ImportValue/Join are used for dynamic stream ARN',
+        () => {
+          awsCompileStreamEvents.serverless.service.functions = {
+            first: {
+              events: [
+                {
+                  stream: {
+                    type: 'dynamodb',
+                    arn: {
+                      'Fn::GetAtt': ['SomeDdbTable', 'StreamArn'],
+                      batchSize: 1,
+                    },
                   },
                 },
-              },
-            ],
-          },
-        };
+              ],
+            },
+          };
 
-        expect(() => awsCompileStreamEvents.compileStreamEvents()).to.throw(Error);
-      });
+          expect(() => awsCompileStreamEvents.compileStreamEvents()).to.throw(Error);
+        });
 
       it('should add the necessary IAM role statements', () => {
         awsCompileStreamEvents.serverless.service.functions = {
@@ -594,21 +635,21 @@ describe('AwsCompileStreamEvents', () => {
           .Properties.EventSourceArn
         ).to.equal(
           awsCompileStreamEvents.serverless.service.functions.first.events[0]
-          .stream.arn
+            .stream.arn
         );
         expect(awsCompileStreamEvents.serverless.service
           .provider.compiledCloudFormationTemplate.Resources.FirstEventSourceMappingKinesisFoo
           .Properties.BatchSize
         ).to.equal(
           awsCompileStreamEvents.serverless.service.functions.first.events[0]
-          .stream.batchSize
+            .stream.batchSize
         );
         expect(awsCompileStreamEvents.serverless.service
           .provider.compiledCloudFormationTemplate.Resources.FirstEventSourceMappingKinesisFoo
           .Properties.StartingPosition
         ).to.equal(
           awsCompileStreamEvents.serverless.service.functions.first.events[0]
-          .stream.startingPosition
+            .stream.startingPosition
         );
         expect(awsCompileStreamEvents.serverless.service
           .provider.compiledCloudFormationTemplate.Resources.FirstEventSourceMappingKinesisFoo
@@ -629,7 +670,7 @@ describe('AwsCompileStreamEvents', () => {
           .Properties.EventSourceArn
         ).to.equal(
           awsCompileStreamEvents.serverless.service.functions.first.events[1]
-          .stream.arn
+            .stream.arn
         );
         expect(awsCompileStreamEvents.serverless.service
           .provider.compiledCloudFormationTemplate.Resources.FirstEventSourceMappingKinesisBar
@@ -658,7 +699,7 @@ describe('AwsCompileStreamEvents', () => {
           .Properties.EventSourceArn
         ).to.equal(
           awsCompileStreamEvents.serverless.service.functions.first.events[2]
-          .stream
+            .stream
         );
         expect(awsCompileStreamEvents.serverless.service
           .provider.compiledCloudFormationTemplate.Resources.FirstEventSourceMappingKinesisBaz