Repository: serverless/serverless (JavaScript)
Date: 2018-11-28 13:09:05+00:00
Task type: Bug Fix

Issue:
The final policy size is bigger than the limit (20480)
<!--
1. If you have a question and not a bug/feature request please ask it at http://forum.serverless.com
2. Please check if an issue already exists so there are no duplicates
3. Check out and follow our Guidelines: https://github.com/serverless/serverless/blob/master/CONTRIBUTING.md
4. Fill out the whole template so we have a good overview on the issue
5. Do not remove any section of the template. If something is not applicable leave it empty but leave it in the Issue
6. Please follow the template, otherwise we'll have to ask you to update it
-->

# This is a Bug Report

## Description

* What went wrong? **Stack update failed. Removing and redeploying did not help.**
* What did you expect should have happened? **Stack update should complete successfully.**
* What was the config you used? **serverless.yml contains a single function with dozens of event sources as subscription filters to CloudWatch Logs.**
* What stacktrace or error message from your provider did you see? `The final policy size (size here differs between deployments) is bigger than the limit (20480)`

Similar or dependent issues:
* None?

## Additional Data

* ***Serverless Framework Version you're using***: 1.32.0
* ***Operating System***: Ubuntu 18.04
* ***Stack Trace***:
* ***Provider Error messages***: 
```
  Serverless Error ---------------------------------------
 
  An error occurred: SubscriptionFilterLambdaLambdaPermissionLogsSubscriptionFilterCloudWatchLog28 - The final policy size (20623) is bigger than the limit (20480). (Service: AWSLambda; Status Code: 400; Error Code: PolicyLengthExceededException; Request ID: a87d245a-c85c-11e8-a18a-af6f9fead152).
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           8.9.1
     Serverless Version:     1.32.0
```
The issue seems to have been reported on the [forums](https://forum.serverless.com/t/aws-lambda-permission-policy-size/3952) however the linked solution does not work.


Issue discussion:
I think being able to replace the auto-generated Lambda::Permission objects with a custom one that uses a wildcard would solve this issue.

Solution patch:
diff --git a/lib/plugins/aws/lib/naming.js b/lib/plugins/aws/lib/naming.js
index a343ca56637..e9ad3294bbe 100644
--- a/lib/plugins/aws/lib/naming.js
+++ b/lib/plugins/aws/lib/naming.js
@@ -321,9 +321,9 @@ module.exports = {
     return `${this.getNormalizedFunctionName(functionName)}LambdaPermissionAlexaSmartHome${
       alexaSmartHomeIndex}`;
   },
-  getLambdaCloudWatchLogPermissionLogicalId(functionName, logsIndex) {
+  getLambdaCloudWatchLogPermissionLogicalId(functionName) {
     return `${this.getNormalizedFunctionName(functionName)
-    }LambdaPermissionLogsSubscriptionFilterCloudWatchLog${logsIndex}`;
+    }LambdaPermissionLogsSubscriptionFilterCloudWatchLog`;
   },
   getLambdaCognitoUserPoolPermissionLogicalId(functionName, poolId, triggerSource) {
     return `${this
diff --git a/lib/plugins/aws/package/compile/events/cloudWatchLog/index.js b/lib/plugins/aws/package/compile/events/cloudWatchLog/index.js
index f356d0bb779..b5808ef47c9 100644
--- a/lib/plugins/aws/package/compile/events/cloudWatchLog/index.js
+++ b/lib/plugins/aws/package/compile/events/cloudWatchLog/index.js
@@ -20,6 +20,8 @@ class AwsCompileCloudWatchLogEvents {
       let cloudWatchLogNumberInFunction = 0;
 
       if (functionObj.events) {
+        const logGroupNamesThisFunction = [];
+
         functionObj.events.forEach(event => {
           if (event.cloudwatchLog) {
             cloudWatchLogNumberInFunction++;
@@ -69,14 +71,14 @@ class AwsCompileCloudWatchLogEvents {
                 .Error(errorMessage);
             }
             logGroupNames.push(LogGroupName);
+            logGroupNamesThisFunction.push(LogGroupName);
 
             const lambdaLogicalId = this.provider.naming
               .getLambdaLogicalId(functionName);
             const cloudWatchLogLogicalId = this.provider.naming
               .getCloudWatchLogLogicalId(functionName, cloudWatchLogNumberInFunction);
             const lambdaPermissionLogicalId = this.provider.naming
-              .getLambdaCloudWatchLogPermissionLogicalId(functionName,
-              cloudWatchLogNumberInFunction);
+              .getLambdaCloudWatchLogPermissionLogicalId(functionName);
 
             // unescape quotes once when the first quote is detected escaped
             const idxFirstSlash = FilterPattern.indexOf('\\');
@@ -97,6 +99,8 @@ class AwsCompileCloudWatchLogEvents {
               }
             `;
 
+            const commonSuffixOfLogGroupName = this.longestCommonSuffix(logGroupNamesThisFunction);
+
             const permissionTemplate = `
             {
               "Type": "AWS::Lambda::Permission",
@@ -121,7 +125,7 @@ class AwsCompileCloudWatchLogEvents {
                   ":",
                   { "Ref": "AWS::AccountId" },
                   ":log-group:",
-                  "${LogGroupName}",
+                  "${commonSuffixOfLogGroupName}",
                   ":*"
                   ] ]
                 }
@@ -144,6 +148,19 @@ class AwsCompileCloudWatchLogEvents {
       }
     });
   }
+
+  longestCommonSuffix(logGroupNames) {
+    const first = logGroupNames[0];
+    const longestCommon = logGroupNames.reduce((last, current) => {
+      for (let i = 0; i < last.length; i++) {
+        if (last[i] !== current[i]) {
+          return last.substring(0, i);
+        }
+      }
+      return last;
+    }, first);
+    return longestCommon + ((longestCommon === first) ? '' : '*');
+  }
 }
 
 module.exports = AwsCompileCloudWatchLogEvents;


Test patch:
diff --git a/lib/plugins/aws/lib/naming.test.js b/lib/plugins/aws/lib/naming.test.js
index 5c76568b3bd..3a67858c575 100644
--- a/lib/plugins/aws/lib/naming.test.js
+++ b/lib/plugins/aws/lib/naming.test.js
@@ -508,8 +508,8 @@ describe('#naming()', () => {
   describe('#getLambdaCloudWatchLogPermissionLogicalId()', () => {
     it('should normalize the function name and add the standard suffix including event index',
       () => {
-        expect(sdk.naming.getLambdaCloudWatchLogPermissionLogicalId('functionName', 0))
-          .to.equal('FunctionNameLambdaPermissionLogsSubscriptionFilterCloudWatchLog0');
+        expect(sdk.naming.getLambdaCloudWatchLogPermissionLogicalId('functionName'))
+          .to.equal('FunctionNameLambdaPermissionLogsSubscriptionFilterCloudWatchLog');
       });
   });
 
diff --git a/lib/plugins/aws/package/compile/events/cloudWatchLog/index.test.js b/lib/plugins/aws/package/compile/events/cloudWatchLog/index.test.js
index bfc3263cd35..ddab8a51e45 100644
--- a/lib/plugins/aws/package/compile/events/cloudWatchLog/index.test.js
+++ b/lib/plugins/aws/package/compile/events/cloudWatchLog/index.test.js
@@ -108,11 +108,7 @@ describe('AwsCompileCloudWatchLogEvents', () => {
       ).to.equal('');
       expect(awsCompileCloudWatchLogEvents.serverless.service
         .provider.compiledCloudFormationTemplate.Resources
-        .FirstLambdaPermissionLogsSubscriptionFilterCloudWatchLog1.Type
-      ).to.equal('AWS::Lambda::Permission');
-      expect(awsCompileCloudWatchLogEvents.serverless.service
-        .provider.compiledCloudFormationTemplate.Resources
-        .FirstLambdaPermissionLogsSubscriptionFilterCloudWatchLog2.Type
+        .FirstLambdaPermissionLogsSubscriptionFilterCloudWatchLog.Type
       ).to.equal('AWS::Lambda::Permission');
     });
 
@@ -278,14 +274,37 @@ describe('AwsCompileCloudWatchLogEvents', () => {
       ).to.equal('');
       expect(awsCompileCloudWatchLogEvents.serverless.service
         .provider.compiledCloudFormationTemplate.Resources
-        .FirstLambdaPermissionLogsSubscriptionFilterCloudWatchLog1.Type
-      ).to.equal('AWS::Lambda::Permission');
-      expect(awsCompileCloudWatchLogEvents.serverless.service
-        .provider.compiledCloudFormationTemplate.Resources
-        .FirstLambdaPermissionLogsSubscriptionFilterCloudWatchLog2.Type
+        .FirstLambdaPermissionLogsSubscriptionFilterCloudWatchLog.Type
       ).to.equal('AWS::Lambda::Permission');
     });
 
+    it('should create a longest-common suffix of logGroup to minimize scope', () => {
+      expect(awsCompileCloudWatchLogEvents
+        .longestCommonSuffix(['/aws/lambda/hello1']))
+        .to.equal('/aws/lambda/hello1');
+      expect(awsCompileCloudWatchLogEvents
+        .longestCommonSuffix(['/aws/lambda/hello1', '/aws/lambda/hello2']))
+        .to.equal('/aws/lambda/hello*');
+      expect(awsCompileCloudWatchLogEvents
+        .longestCommonSuffix(['/aws/lambda/hello1', '/aws/lambda/hot']))
+        .to.equal('/aws/lambda/h*');
+      expect(awsCompileCloudWatchLogEvents
+        .longestCommonSuffix(['/aws/lambda/hello1', '/aws/lambda/tweet']))
+        .to.equal('/aws/lambda/*');
+      expect(awsCompileCloudWatchLogEvents
+        .longestCommonSuffix(['/aws/lambda/hello1', '/aws/lex/log1', '/aws/lightsail/log1']))
+        .to.equal('/aws/l*');
+      expect(awsCompileCloudWatchLogEvents
+        .longestCommonSuffix(['/aws/lambda/hello1', '/aws/batch/log1']))
+        .to.equal('/aws/*');
+      expect(awsCompileCloudWatchLogEvents
+        .longestCommonSuffix(['/aws/*', '/aws/lambda/hello']))
+        .to.equal('/aws/*');
+      expect(awsCompileCloudWatchLogEvents
+        .longestCommonSuffix(['/aws/lambda/*', '/aws/lambda/hello']))
+        .to.equal('/aws/lambda/*');
+    });
+
     it('should throw an error if "logGroup" is duplicated in one CloudFormation stack', () => {
       awsCompileCloudWatchLogEvents.serverless.service.functions = {
         first: {