Repository: serverless/serverless (JavaScript)
Date: 2019-07-29 17:29:08+00:00
Task type: Bug Fix

Issue:
Use function name instead of counter for multiple cloudwatchLogs for avoiding SubscriptionFilter ResourceExceeded error
# This is a Bug Report

## Description

While adding a cloudwatchLog trigger event (subscription event) to a function that has many of them, we're wrongly relying on a counter to create the CloudFormation's Logical IDs, and if we change the order of by just swapping a couple, deleting, or adding new (that's not in the end), we hit a ResourceExceeded error.

Similar or dependent issues:
* #5700, #5817

## Additional Data
Serverless adds a count at the end of the Logical IDs that start with 1. See: https://github.com/serverless/serverless/blob/master/lib/plugins/aws/package/compile/events/cloudWatchLog/index.js#L79

Let's say we have this:
```
functions:
  performUserExperiments:
    handler: dist/handler.performUserExperiments
```

Which generates (simplified):
```
{
   ...
   SumologicLoggingLogsSubscriptionFilterCloudWatchLog1: {
      Type: SubscriptionFilter
      Log: performUserExperimentsLogGroup
   }
}
```

We add a new fn at the beginning:
```
functions:
  getUserExperiments:
    handler: dist/handler.getUserExperiments
  performUserExperiments:
    handler: dist/handler.performUserExperiments
```

Which generates:
```
{
   ...
   SumologicLoggingLogsSubscriptionFilterCloudWatchLog1: {
      Type: SubscriptionFilter
      Log: getUserExperimentsLogGroup
   },
   SumologicLoggingLogsSubscriptionFilterCloudWatchLog2: {
      Type: SubscriptionFilter
      Log: performUserExperimentsLogGroup
   }
}
```

Now even though the performUserExperiments fn has not changed at all since its LogicalID changed, CloudFormation will try to create a new resource, in parallel while it applies the update at SumologicLoggingLogsSubscriptionFilterCloudWatchLog1, which before was pointing to performUserExperiments, but now it's pointing to getUserExperiments. This is an update that requires a new resource from scratch, by the way.

## Proposed Fix
Get rid of this count completely, and use the name of the log group as part of the Logical ID. 

For example, assuming we have:
```
/aws/lambda/performUserExperiment (log group)
sumologic-logging-dev (function that will be called by the log group)
```

We can name the Logical ID as:
```
SumologicLoggingLogsSubscriptionFilterPerformUserExperiment
```

Instead of:
```
SumologicLoggingLogsSubscriptionFilterX (X = this serial number we have today)
```

I'm happy to send a PR if you believe that this approach makes sense. @medikoo @pmuens 

* ***Serverless Framework Version you're using***: v1.45.1
* ***Operating System***: MacOS
* ***Provider Error messages***: An error occurred: SumologicLoggingLogsSubscriptionFilterCloudWatchLog2 - Resource limit exceeded. (Service: AWSLogs; Status Code: 400; Error Code: LimitExceededException; Request ID: ***).



Issue discussion:

Solution patch:
diff --git a/lib/plugins/aws/deploy/lib/checkForChanges.js b/lib/plugins/aws/deploy/lib/checkForChanges.js
index 570503e381b..ad71b3c4b15 100644
--- a/lib/plugins/aws/deploy/lib/checkForChanges.js
+++ b/lib/plugins/aws/deploy/lib/checkForChanges.js
@@ -160,6 +160,7 @@ module.exports = {
             return BbPromise.resolve();
           }
 
+          let logSubscriptionSerialNumber = 0;
           const promises = functionObj.events.map(event => {
             if (!event.cloudwatchLog) {
               return BbPromise.resolve();
@@ -180,6 +181,8 @@ module.exports = {
             const accountId = account.accountId;
             const partition = account.partition;
 
+            logSubscriptionSerialNumber++;
+
             /*
               return a new promise that will check the resource limit exceeded and will fix it if
               the option is enabled
@@ -188,9 +191,11 @@ module.exports = {
               cloudWatchLogsSdk,
               accountId,
               logGroupName,
+              functionName,
               functionObj,
               region,
               partition,
+              logSubscriptionSerialNumber,
             });
           });
 
@@ -204,9 +209,11 @@ module.exports = {
     const cloudWatchLogsSdk = params.cloudWatchLogsSdk;
     const accountId = params.accountId;
     const logGroupName = params.logGroupName;
+    const functionName = params.functionName;
     const functionObj = params.functionObj;
     const region = params.region;
     const partition = params.partition;
+    const logSubscriptionSerialNumber = params.logSubscriptionSerialNumber;
 
     return (
       cloudWatchLogsSdk
@@ -220,19 +227,26 @@ module.exports = {
             return false;
           }
 
-          const oldDestinationArn = subscriptionFilter.destinationArn;
           const filterName = subscriptionFilter.filterName;
+
+          const oldDestinationArn = subscriptionFilter.destinationArn;
           const newDestinationArn = `arn:${partition}:lambda:${region}:${accountId}:function:${functionObj.name}`;
 
+          const oldLogicalId = this.getLogicalIdFromFilterName(filterName);
+          const newLogicalId = this.provider.naming.getCloudWatchLogLogicalId(
+            functionName,
+            logSubscriptionSerialNumber
+          );
+
           // everything is fine, just return
-          if (oldDestinationArn === newDestinationArn) {
+          if (oldDestinationArn === newDestinationArn && oldLogicalId === newLogicalId) {
             return false;
           }
 
           /*
-          If the destinations functions' ARNs doesn't match, we need to delete the current
-          subscription filter to prevent the resource limit exceeded error to happen
-        */
+            If the destinations functions' ARNs doesn't match, we need to delete the current
+            subscription filter to prevent the resource limit exceeded error to happen
+          */
           return cloudWatchLogsSdk.deleteSubscriptionFilter({ logGroupName, filterName }).promise();
         })
         /*
@@ -242,4 +256,12 @@ module.exports = {
         .catch(() => undefined)
     );
   },
+
+  getLogicalIdFromFilterName(filterName) {
+    // Filter name format:
+    // {stack name}-{logical id}-{random alphanumeric characters}
+    // Note that the stack name can include hyphens
+    const split = filterName.split('-');
+    return split[split.length - 2];
+  },
 };


Test patch:
diff --git a/lib/plugins/aws/deploy/lib/checkForChanges.test.js b/lib/plugins/aws/deploy/lib/checkForChanges.test.js
index cf4519d56b4..915f072497e 100644
--- a/lib/plugins/aws/deploy/lib/checkForChanges.test.js
+++ b/lib/plugins/aws/deploy/lib/checkForChanges.test.js
@@ -593,7 +593,7 @@ describe('checkForChanges', () => {
           .then(() => expect(deleteSubscriptionFilterStub).to.not.have.been.called);
       });
 
-      it('should not call delete if there is a subFilter and the ARNs are the same', () => {
+      it('should not call delete if there is a subFilter and the ARNs/logical IDs are the same', () => {
         awsDeploy.serverless.service.functions = {
           first: {
             events: [{ cloudwatchLog: '/aws/lambda/hello1' }],
@@ -606,7 +606,7 @@ describe('checkForChanges', () => {
           subscriptionFilters: [
             {
               destinationArn: `arn:aws:lambda:${region}:${accountId}:function:${serviceName}-dev-first`,
-              filterName: 'dummy-filter',
+              filterName: 'stack-name-FirstLogsSubscriptionFilterCloudWatchLog1-1KAK9SAG7Y9YN',
             },
           ],
         };
@@ -629,7 +629,7 @@ describe('checkForChanges', () => {
           subscriptionFilters: [
             {
               destinationArn: `arn:aws:lambda:${region}:${accountId}:function:${serviceName}-dev-not-first`,
-              filterName: 'dummy-filter',
+              filterName: 'stack-name-FirstLogsSubscriptionFilterCloudWatchLog1-1KAK9SAG7Y9YN',
             },
           ],
         };
@@ -639,7 +639,30 @@ describe('checkForChanges', () => {
           .then(() => expect(deleteSubscriptionFilterStub).to.have.been.called);
       });
 
-      it('should not call delete if there is a subFilter and the ARNs are the same with custom function name', () => {
+      it('should call delete if there is a subFilter but the logical IDs are not the same', () => {
+        awsDeploy.serverless.service.functions = {
+          first: {
+            events: [{ cloudwatchLog: '/aws/lambda/hello1' }],
+          },
+        };
+
+        awsDeploy.serverless.service.setFunctionNames();
+
+        describeSubscriptionFiltersResponse = {
+          subscriptionFilters: [
+            {
+              destinationArn: `arn:aws:lambda:${region}:${accountId}:function:${serviceName}-dev-first`,
+              filterName: 'stack-name-FirstLogsSubscriptionFilterCloudWatchLog2-1KAK9SAG7Y9YN',
+            },
+          ],
+        };
+
+        return awsDeploy
+          .checkForChanges()
+          .then(() => expect(deleteSubscriptionFilterStub).to.have.been.called);
+      });
+
+      it('should not call delete if there is a subFilter and the ARNs/logical IDs are the same with custom function name', () => {
         awsDeploy.serverless.service.functions = {
           first: {
             name: 'my-test-function',
@@ -653,7 +676,7 @@ describe('checkForChanges', () => {
           subscriptionFilters: [
             {
               destinationArn: `arn:aws:lambda:${region}:${accountId}:function:my-test-function`,
-              filterName: 'dummy-filter',
+              filterName: 'stack-name-FirstLogsSubscriptionFilterCloudWatchLog1-1KAK9SAG7Y9YN',
             },
           ],
         };
@@ -663,7 +686,7 @@ describe('checkForChanges', () => {
           .then(() => expect(deleteSubscriptionFilterStub).to.not.have.been.called);
       });
 
-      it('should not call delete when ARN is the same accounting for non-standard partitions', () => {
+      it('should not call delete when ARN/logical IDs are the same accounting for non-standard partitions', () => {
         provider.getAccountInfo.restore();
         sandbox.stub(provider, 'getAccountInfo').returns(
           BbPromise.resolve({
@@ -684,7 +707,7 @@ describe('checkForChanges', () => {
           subscriptionFilters: [
             {
               destinationArn: `arn:aws-us-gov:lambda:${region}:${accountId}:function:my-test-function`,
-              filterName: 'dummy-filter',
+              filterName: 'stack-name-FirstLogsSubscriptionFilterCloudWatchLog1-1KAK9SAG7Y9YN',
             },
           ],
         };
@@ -708,7 +731,7 @@ describe('checkForChanges', () => {
           subscriptionFilters: [
             {
               destinationArn: `arn:aws:lambda:${region}:${accountId}:function:my-other-test-function`,
-              filterName: 'dummy-filter',
+              filterName: 'stack-name-FirstLogsSubscriptionFilterCloudWatchLog1-1KAK9SAG7Y9YN',
             },
           ],
         };