Repository: serverless/serverless (JavaScript)
Date: 2019-07-18 15:48:14+00:00
Task type: Bug Fix

Issue:
CloudWatch Log Event Subscriptions Deleted
# This is a Bug Report

## Description

We have a Serverless Lambda function that forwards logs from CloudWatch to Loggly similar to https://github.com/loggly/cloudwatch2loggly
After the update to 1.36.0 CloudWatch Logs triggers are deleted on deploy

## To reproduce
Deploy a function with 3 CloudWatch Log event triggers
```
functions:
  app:
    name: cloudwatch2loggly
    handler: src/index.handler
    events:
      - cloudwatchLog: '/aws/lambda/${self:provider.stage}-function1'
      - cloudwatchLog: '/aws/lambda/${self:provider.stage}-function2'
      - cloudwatchLog: '/aws/lambda/${self:provider.stage}-function3'
```
Run serverless deploy
`serverless deploy`
* Triggers are visible in the console and function is triggered as expected
![image](https://user-images.githubusercontent.com/797831/51149157-f5e33980-18b4-11e9-9833-22f1ac288ee7.png)

Run serverless deploy again (specify force as there are no code changes)
`serverless deploy --force`
* Triggers are deleted
![image](https://user-images.githubusercontent.com/797831/51149306-a5201080-18b5-11e9-937b-6254b4439481.png)

On each deployment I can see three calls to `DescribeSubscriptionFilters` in CloudTrail e.g.
```
    "requestParameters": {
        "logGroupName": "/aws/lambda/stage-function1"
    },
```
And on the deployment where the triggers are deleted we see three `DeleteSubscriptionFilter` in CloudTrail e.g.
```
"requestParameters": {
        "filterName": "cloudwatch2loggly-stage-AppLogsSubscriptionFilterCloudWatchLog1-XXXXXXXX",
        "logGroupName": "/aws/lambda/stage-function1"
    },
```

Drift detection also shows the subscriptions are deleted e.g. `AppLogsSubscriptionFilterCloudWatchLog1`
![image](https://user-images.githubusercontent.com/797831/51149885-1fea2b00-18b8-11e9-9370-1d6255303498.png)

Similar or dependent issues:
* Appears to be a result of the changes made in https://github.com/serverless/serverless/pull/5554

## Additional Data

* ***Serverless Framework Version you're using***: 1.36.1
* ***Operating System***: Windows 10
* ***Stack Trace***:
* ***Provider Error messages***:



Issue discussion:
Hey, I'm sorry my changes caused that. Going to try to reproduce the problem later this week.
Hi, please let me know the work around for this? Revert version back to 1.35.X?
@NishantRanjan Yes, in our case we reverted to 1.35.1 and the problem was fixed.
Any progress @rdsedmundo?
> Triggers are deleted

Perhaps, this is related to #5833: CW Log Subscriptions do not appear in Lambda settings Triggers.
Sorry, I was really busy those last weeks. I'll take a look at this at the end of this week/weekend.
@rdsedmundo Wondering if you ever get a chance to investigate? We're currently unable to upgrade due to this bug so missing out on new features and bug fixes.
Hey @rdsedmundo thanks for the contributions to date. Any chance you could tell us if you are going to have a look at fixing this. This currently blocks people from moving to NodeJs10.x in later versions of serverless. If you could just confirm you will or wont then maybe someone else would be more willing to engage in the fix, thank you.
I took the time to investigate this today. I wasn't able to reproduce the issue following steps that OP mentioned, i.e to just add some triggers and then re-deploy with the same ones, without changing anything. It works here without errors.

But re #5817 reported by @exoego, i.e re-ordering under the same function, it's definitely something that I wasn't accounting for on my original PR. The problem is on how Serverless is naming the Logical IDs of the resources on the CloudFormation template. Example, let's say we have:

```
  hello:
    handler: handler.hello
    events:
      - cloudwatchLog: '/aws/lambda/subscription-test-1'
      - cloudwatchLog: '/aws/lambda/subscription-test-2'
      - cloudwatchLog: '/aws/lambda/subscription-test-3'
```

We'll generate the following resources Logical IDs:

HelloLogsSubscriptionFilterCloudWatchLog1 (/aws/lambda/subscription-test-1)
HelloLogsSubscriptionFilterCloudWatchLog2 (/aws/lambda/subscription-test-2)
HelloLogsSubscriptionFilterCloudWatchLog3 (/aws/lambda/subscription-test-3)

If you then go and change the order on the yml file, for example to:

```
  hello:
    handler: handler.hello
    events:
      - cloudwatchLog: '/aws/lambda/subscription-test-2'
      - cloudwatchLog: '/aws/lambda/subscription-test-1'
      - cloudwatchLog: '/aws/lambda/subscription-test-3'
```

We'll generate this template for update:

HelloLogsSubscriptionFilterCloudWatchLog1 (/aws/lambda/subscription-test-2)
HelloLogsSubscriptionFilterCloudWatchLog2 (/aws/lambda/subscription-test-1)
HelloLogsSubscriptionFilterCloudWatchLog3 (/aws/lambda/subscription-test-3)

Pay attention to the collision that will happen, as now the HelloLogs...1 is pointing to `subscription-test-2` and not `subscription-test-1` as before, so CloudFormation will interpret this as the same resource as before and will first try to update HelloLogs...1 to point to `subscription-test-2`, before deleting the original subscription we already had on it (from HelloLogs..2 on the original template), leading to the ResourceLimitExceeded error. The original purpose of my PR was to just catch when you were transferring a subscription filter from one function to another.

We probably will need to restructure how the event is added, perhaps adding a property called `uniqueName` where the user provides some identifier for the subscription filter and we append it at the end of the Logical ID. Or maybe just using the name of the log group by itself? @pmuens @dschep 
I was investigating this issue (the same one as the OP, where subscription filters would just disappear without being swapped around) today as it was causing me some issues. First I tried implementing the solution suggested above, where you use the name of the log group to tag each subscription filter rather than a serial number. While this did fix the issue in #5817, it didn't do anything about the one encountered here.

I did some more work, adding a few print statements, and noticed that in the following few lines:
~~~~
const oldDestinationArn = subscriptionFilter.destinationArn;
const filterName = subscriptionFilter.filterName;
const newDestinationArn = `arn:aws:lambda:${region}:${accountId}:function:${serviceName}-${stage}-${functionName}`;
~~~~
 the values of `oldDestinationArn` and `newDestinationArn` did not match. While the ending (after `function:`) of `oldDestinationArn` matched the actual ARN of the subscription log, and the `name` field in my `serverless.yml` file, the ending of `newDestinationArn` instead matched the name of the YAML object corresponding to my function.

In other words, the code which deleted the old log subscriptions only worked if the `name` field was not set in `serverless.yml` and the default name was used. This was easily fixed by passing in `functionObj` to the `fixLogGroupSubscriptionFilters` method and changing the last line of the snippet above instead to:
~~~
const newDestinationArn = `arn:aws:lambda:${region}:${accountId}:function:${functionObj.name}`;
~~~

This seemed to resolve the issue. If you'd like I'm happy to make a PR with my changes (or if you'd prefer, two, since I also have the changes corresponding to the other issue as described in my first paragraph).

Solution patch:
diff --git a/lib/plugins/aws/deploy/lib/checkForChanges.js b/lib/plugins/aws/deploy/lib/checkForChanges.js
index ec4419c2599..1ef50534453 100644
--- a/lib/plugins/aws/deploy/lib/checkForChanges.js
+++ b/lib/plugins/aws/deploy/lib/checkForChanges.js
@@ -149,8 +149,6 @@ module.exports = {
    */
   checkLogGroupSubscriptionFilterResourceLimitExceeded() {
     const region = this.provider.getRegion();
-    const serviceName = this.serverless.service.getServiceName();
-    const stage = this.provider.getStage();
     const cloudWatchLogsSdk = new this.provider.sdk.CloudWatchLogs({ region });
 
     return this.provider.getAccountId().then(accountId =>
@@ -187,10 +185,8 @@ module.exports = {
               cloudWatchLogsSdk,
               accountId,
               logGroupName,
-              functionName,
+              functionObj,
               region,
-              serviceName,
-              stage,
             });
           });
 
@@ -204,10 +200,8 @@ module.exports = {
     const cloudWatchLogsSdk = params.cloudWatchLogsSdk;
     const accountId = params.accountId;
     const logGroupName = params.logGroupName;
-    const functionName = params.functionName;
+    const functionObj = params.functionObj;
     const region = params.region;
-    const serviceName = params.serviceName;
-    const stage = params.stage;
 
     return (
       cloudWatchLogsSdk
@@ -223,7 +217,7 @@ module.exports = {
 
           const oldDestinationArn = subscriptionFilter.destinationArn;
           const filterName = subscriptionFilter.filterName;
-          const newDestinationArn = `arn:aws:lambda:${region}:${accountId}:function:${serviceName}-${stage}-${functionName}`;
+          const newDestinationArn = `arn:aws:lambda:${region}:${accountId}:function:${functionObj.name}`;
 
           // everything is fine, just return
           if (oldDestinationArn === newDestinationArn) {


Test patch:
diff --git a/lib/plugins/aws/deploy/lib/checkForChanges.test.js b/lib/plugins/aws/deploy/lib/checkForChanges.test.js
index 9cc808d38aa..313c4c92479 100644
--- a/lib/plugins/aws/deploy/lib/checkForChanges.test.js
+++ b/lib/plugins/aws/deploy/lib/checkForChanges.test.js
@@ -577,6 +577,8 @@ describe('checkForChanges', () => {
           },
         };
 
+        awsDeploy.serverless.service.setFunctionNames();
+
         describeSubscriptionFiltersResponse = {
           subscriptionFilters: [],
         };
@@ -593,6 +595,8 @@ describe('checkForChanges', () => {
           },
         };
 
+        awsDeploy.serverless.service.setFunctionNames();
+
         describeSubscriptionFiltersResponse = {
           subscriptionFilters: [
             {
@@ -614,6 +618,8 @@ describe('checkForChanges', () => {
           },
         };
 
+        awsDeploy.serverless.service.setFunctionNames();
+
         describeSubscriptionFiltersResponse = {
           subscriptionFilters: [
             {
@@ -627,6 +633,54 @@ describe('checkForChanges', () => {
           .checkForChanges()
           .then(() => expect(deleteSubscriptionFilterStub).to.have.been.called);
       });
+
+      it('should not call delete if there is a subFilter and the ARNs are the same with custom function name', () => {
+        awsDeploy.serverless.service.functions = {
+          first: {
+            name: 'my-test-function',
+            events: [{ cloudwatchLog: '/aws/lambda/hello1' }],
+          },
+        };
+
+        awsDeploy.serverless.service.setFunctionNames();
+
+        describeSubscriptionFiltersResponse = {
+          subscriptionFilters: [
+            {
+              destinationArn: `arn:aws:lambda:${region}:${accountId}:function:my-test-function`,
+              filterName: 'dummy-filter',
+            },
+          ],
+        };
+
+        return awsDeploy
+          .checkForChanges()
+          .then(() => expect(deleteSubscriptionFilterStub).to.not.have.been.called);
+      });
+
+      it('should call delete if there is a subFilter but the ARNs are not the same with custom function name', () => {
+        awsDeploy.serverless.service.functions = {
+          first: {
+            name: 'my-test-function',
+            events: [{ cloudwatchLog: '/aws/lambda/hello1' }],
+          },
+        };
+
+        awsDeploy.serverless.service.setFunctionNames();
+
+        describeSubscriptionFiltersResponse = {
+          subscriptionFilters: [
+            {
+              destinationArn: `arn:aws:lambda:${region}:${accountId}:function:my-other-test-function`,
+              filterName: 'dummy-filter',
+            },
+          ],
+        };
+
+        return awsDeploy
+          .checkForChanges()
+          .then(() => expect(deleteSubscriptionFilterStub).to.have.been.called);
+      });
     });
   });
 });