Repository: ACloudGuru/serverless-plugin-aws-alerts
Pull request #125 (2020-06-05 19:35:29)

Support for AnomalyDetection Alarms and disabling alarms per-function
## What did you implement:

Two features proposed in this PR:

1. Allow disabling an alarm configured as a default on a per-function basis. It is very valuable to define alarms to be applied to each function in my CF stack, but on some occasions some functions should not have the default alarm(s) applied.  This new property allows a default alarm to be disabled for any given function(s).
1. Add support for AnomalyDetection type alarms.  Alarms of this type don't have a static threshold to trigger the alarm.  AWS generates a model based on past data, then a threshold is set for how many standard deviations outside the predicted model a data point must be to trigger the alarm.  This is very useful for functions with invocation patterns that aren't flat, for example.


## How did you implement it:

Trivial implementation changes to add two new properties to alarm definitions:
1. enabled
    1. If enabled is false no CloudFormation resources will be created for this alarm
1. type
    1. 'static' - default value when not specified, maintains existing behavior
    1. 'anomalyDetection' - creates CloudFormation resources to create an "anomaly detection" type alarm

## How can we verify it:

Not sure if anything needs to be provided here?  Examples in the readme demonstrate use, the result will be visible in the CloudWatch console as an alarm using the "anomaly detection" feature.


## Todos:

- [ ] Write tests
- [ ] Write documentation
- [ ] Fix linting errors
- [ ] Provide verification config/commands/resources




Patch:
diff --git a/README.md b/README.md
index 1105a08..d1c44e6 100644
--- a/README.md
+++ b/README.md
@@ -72,6 +72,41 @@ functions:
         comparisonOperator: GreaterThanOrEqualToThreshold
 ```
 
+## Anomaly Detection Alarms
+
+You can create alarms using [CloudWatch AnomalyDetection](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Anomaly_Detection.html) to alarm when data is outside a number of standard deviations of normal, rather than at a static threshold.
+When using anomaly detection alarms the threshold property specifies the "Anomaly Detection Threshold" seen in the AWS console.
+
+Default alarms can also be updated to be anomaly detection alarms by adding the `type: anomalyDetection` property.
+
+```yaml
+functions:
+  foo:
+    handler: foo.handler
+    alarms:
+      - name: fooAlarm
+        type: anomalyDetection
+        namespace: 'AWS/Lamabda'
+        metric: Invocations
+        threshold: 2
+        statistic: Sum
+        period: 60
+        evaluationPeriods: 1
+        datapointsToAlarm: 1
+        comparisonOperator: LessThanLowerOrGreaterThanUpperThreshold
+  bar:
+    handler: bar.handler
+    alarms:
+      - name: functionErrors
+        threshold: 2
+        type: anomalyDetection
+        comparisonOperator: LessThanLowerOrGreaterThanUpperThreshold
+      - name: functionInvocations
+        threshold: 2
+        type: anomalyDetection
+        comparisonOperator: LessThanLowerOrGreaterThanUpperThreshold
+```
+
 ## Multiple topic definitions
 
 You can define several topics for alarms. For example you want to have topics for critical alarms
@@ -284,6 +319,29 @@ definitions:
     comparisonOperator: GreaterThanOrEqualToThreshold
     treatMissingData: missing
 ```
+
+## Disabling default alarms for specific functions
+
+Default alarms can be disabled on a per-function basis:
+
+```yaml
+custom:
+  alerts:
+    alarms:
+      - functionThrottles
+      - functionErrors
+      - functionInvocations
+      - functionDuration
+
+functions:
+  bar:
+    handler: bar.handler
+    alarms:
+      - name: functionInvocations
+        enabled: false
+
+```
+
 ## Additional dimensions
 
 The plugin allows users to provide custom dimensions for the alarm. Dimensions are provided in a list of key/value pairs {Name: foo, Value:bar}
diff --git a/src/defaults/definitions.js b/src/defaults/definitions.js
index 06c4e77..9d393b0 100644
--- a/src/defaults/definitions.js
+++ b/src/defaults/definitions.js
@@ -5,6 +5,8 @@ const lambdaNamespace = 'AWS/Lambda';
 module.exports = {
   functionInvocations: {
     namespace: lambdaNamespace,
+    enabled: true,
+    type: 'static',
     metric: 'Invocations',
     threshold: 100,
     statistic: 'Sum',
@@ -15,6 +17,8 @@ module.exports = {
   },
   functionErrors: {
     namespace: lambdaNamespace,
+    enabled: true,
+    type: 'static',
     metric: 'Errors',
     threshold: 1,
     statistic: 'Sum',
@@ -25,6 +29,8 @@ module.exports = {
   },
   functionDuration: {
     namespace: lambdaNamespace,
+    enabled: true,
+    type: 'static',
     metric: 'Duration',
     threshold: 500,
     statistic: 'Average',
@@ -35,6 +41,8 @@ module.exports = {
   },
   functionThrottles: {
     namespace: lambdaNamespace,
+    enabled: true,
+    type: 'static',
     metric: 'Throttles',
     threshold: 1,
     statistic: 'Sum',
diff --git a/src/index.js b/src/index.js
index f0f8f9c..28d077a 100644
--- a/src/index.js
+++ b/src/index.js
@@ -41,11 +41,25 @@ class AlertsPlugin {
           throw new Error(`Alarm definition ${alarm} does not exist!`);
         }
 
-        result.push(Object.assign({}, definition, {
-          name: alarm
-        }));
+        result.push(Object.assign(
+          {
+            enabled: true,
+            type: 'static'
+          },
+          definition,
+          {
+            name: alarm
+          })
+        );
       } else if (_.isObject(alarm)) {
-        result.push(_.merge({}, definitions[alarm.name], alarm));
+        result.push(_.merge(
+          {
+            enabled: true,
+            type: 'static'
+          },
+          definitions[alarm.name],
+          alarm)
+        );
       }
 
       return result;
@@ -117,24 +131,72 @@ class AlertsPlugin {
 
     const treatMissingData = definition.treatMissingData ? definition.treatMissingData : 'missing';
 
-    const alarm = {
-      Type: 'AWS::CloudWatch::Alarm',
-      Properties: {
-        Namespace: namespace,
-        MetricName: metricId,
-        AlarmDescription: definition.description,
-        Threshold: definition.threshold,
-        Period: definition.period,
-        EvaluationPeriods: definition.evaluationPeriods,
-        DatapointsToAlarm: definition.datapointsToAlarm,
-        ComparisonOperator: definition.comparisonOperator,
-        OKActions: okActions,
-        AlarmActions: alarmActions,
-        InsufficientDataActions: insufficientDataActions,
-        Dimensions: dimensions,
-        TreatMissingData: treatMissingData,
+    const statisticValues = ['SampleCount', 'Average', 'Sum', 'Minimum', 'Maximum'];
+    let alarm;
+    if (definition.type === 'static') {
+      alarm = {
+        Type: 'AWS::CloudWatch::Alarm',
+        Properties: {
+          Namespace: namespace,
+          MetricName: metricId,
+          AlarmDescription: definition.description,
+          Threshold: definition.threshold,
+          Period: definition.period,
+          EvaluationPeriods: definition.evaluationPeriods,
+          DatapointsToAlarm: definition.datapointsToAlarm,
+          ComparisonOperator: definition.comparisonOperator,
+          OKActions: okActions,
+          AlarmActions: alarmActions,
+          InsufficientDataActions: insufficientDataActions,
+          Dimensions: dimensions,
+          TreatMissingData: treatMissingData,
+        }
+      };
+
+      if (_.includes(statisticValues, definition.statistic)) {
+        alarm.Properties.Statistic = definition.statistic
+      } else {
+        alarm.Properties.ExtendedStatistic = definition.statistic
       }
-    };
+    } else if (definition.type === 'anomalyDetection') {
+      alarm = {
+        Type: 'AWS::CloudWatch::Alarm',
+        Properties: {
+          AlarmDescription: definition.description,
+          EvaluationPeriods: definition.evaluationPeriods,
+          DatapointsToAlarm: definition.datapointsToAlarm,
+          ComparisonOperator: definition.comparisonOperator,
+          TreatMissingData: treatMissingData,
+          OKActions: okActions,
+          AlarmActions: alarmActions,
+          InsufficientDataActions: insufficientDataActions,
+          Metrics: [
+            {
+              Id: 'm1',
+              ReturnData: true,
+              MetricStat: {
+                Metric: {
+                  Namespace: namespace,
+                  MetricName: metricId,
+                  Dimensions: dimensions
+                },
+                Period: definition.period,
+                Stat: definition.statistic
+              }
+            },
+            {
+              Id: 'ad1',
+              Expression: `ANOMALY_DETECTION_BAND(m1, ${definition.threshold})`,
+              Label: `${metricId} (expected)`,
+              ReturnData: true
+            }
+          ],
+          ThresholdMetricId: 'ad1'
+        }
+      }
+    } else {
+      throw new Error(`Missing type for alarm ${alarm.name} on function ${functionName}, must be one of 'static' or 'anomalyDetection'`);
+    }
 
     if (definition.nameTemplate) {
       alarm.Properties.AlarmName = this.naming.getAlarmName({
@@ -147,13 +209,6 @@ class AlertsPlugin {
         stackName
       });
     }
-
-    const statisticValues = ['SampleCount', 'Average', 'Sum', 'Minimum', 'Maximum'];
-    if (_.includes(statisticValues, definition.statistic)) {
-      alarm.Properties.Statistic = definition.statistic
-    } else {
-      alarm.Properties.ExtendedStatistic = definition.statistic
-    }
     return alarm;
   }
 
@@ -278,12 +333,16 @@ class AlertsPlugin {
 
       const alarmStatements = alarms.reduce((statements, alarm) => {
         const key = this.naming.getAlarmCloudFormationRef(alarm.name, functionName);
-        const cf = this.getAlarmCloudFormation(alertTopics, alarm, functionName, normalizedFunctionName);
+        if (alarm.enabled) {
+          const cf = this.getAlarmCloudFormation(alertTopics, alarm, functionName, normalizedFunctionName);
 
-        statements[key] = cf;
+          statements[key] = cf;
 
-        const logMetricCF = this.getLogMetricCloudFormation(alarm, functionName, normalizedFunctionName, functionObj);
-        _.merge(statements, logMetricCF);
+          const logMetricCF = this.getLogMetricCloudFormation(alarm, functionName, normalizedFunctionName, functionObj);
+          _.merge(statements, logMetricCF);
+        } else {
+          delete statements[key];
+        }
 
         return statements;
       }, {});


Test patch:
diff --git a/src/index.test.js b/src/index.test.js
index 24177d5..62b7c3d 100644
--- a/src/index.test.js
+++ b/src/index.test.js
@@ -86,11 +86,16 @@ describe('#index', function () {
       const alarmsConfig = plugin.getAlarms(alarms, definitions);
       expect(alarmsConfig).toEqual([{
         name: 'test',
+        enabled: true,
+        type: 'static'
       }]);
     });
 
     it('should get alarms config by object', () => {
-      const testAlarm = {};
+      const testAlarm = {
+        enabled: true,
+        type: 'static'
+      };
       const alarms = [testAlarm];
       const definitions = {};
 
@@ -123,13 +128,19 @@ describe('#index', function () {
       const alarmsConfig = plugin.getAlarms(alarms, definitions);
       expect(alarmsConfig).toEqual([{
         name: 'testAlarm',
+        enabled: true,
+        type: 'static',
         threshold: 100,
         statistic: 'Sum'
       }]);
     });
 
     it('should import alarms from CloudFormation', () => {
-      const testAlarm = { 'Fn::ImportValue': "ServiceMonitoring:monitoring-${opt:stage, 'dev'}" };
+      const testAlarm = {
+        'Fn::ImportValue': "ServiceMonitoring:monitoring-${opt:stage, 'dev'}",
+        enabled: true,
+        type: 'static'
+      };
       const alarms = [testAlarm];
       const definitions = {};
 
@@ -183,6 +194,7 @@ describe('#index', function () {
       const config = {
         definitions: {
           functionErrors: {
+            type: 'static',
             metric: 'Errors',
             threshold: 1,
             statistic: 'Maximum',
@@ -192,6 +204,8 @@ describe('#index', function () {
             comparisonOperator: 'GreaterThanOrEqualToThreshold',
           },
           customDefinition: {
+            type: 'static',
+            enabled: true,
             namespace: 'AWS/Lambda',
             metric: 'Invocations',
             threshold: 5,
@@ -210,6 +224,8 @@ describe('#index', function () {
       expect(actual).toEqual({
         functionInvocations: {
           namespace: 'AWS/Lambda',
+          type: 'static',
+          enabled: true,
           metric: 'Invocations',
           threshold: 100,
           statistic: 'Sum',
@@ -220,6 +236,8 @@ describe('#index', function () {
         },
         functionErrors: {
           namespace: 'AWS/Lambda',
+          type: 'static',
+          enabled: true,
           metric: 'Errors',
           threshold: 1,
           statistic: 'Maximum',
@@ -230,6 +248,8 @@ describe('#index', function () {
         },
         functionDuration: {
           namespace: 'AWS/Lambda',
+          type: 'static',
+          enabled: true,
           metric: 'Duration',
           threshold: 500,
           statistic: 'Average',
@@ -240,6 +260,8 @@ describe('#index', function () {
         },
         functionThrottles: {
           namespace: 'AWS/Lambda',
+          type: 'static',
+          enabled: true,
           metric: 'Throttles',
           threshold: 1,
           statistic: 'Sum',
@@ -250,6 +272,8 @@ describe('#index', function () {
         },
         customDefinition: {
           namespace: 'AWS/Lambda',
+          type: 'static',
+          enabled: true,
           metric: 'Invocations',
           threshold: 5,
           statistic: 'Minimum',
@@ -311,6 +335,8 @@ describe('#index', function () {
 
       expect(actual).toEqual([{
         name: 'customAlarm',
+        enabled: true,
+        type: 'static',
         namespace: 'AWS/Lambda',
         metric: 'Invocations',
         threshold: 5,
@@ -341,6 +367,8 @@ describe('#index', function () {
 
       expect(actual).toEqual([{
         name: 'fooAlarm',
+        enabled: true,
+        type: 'static',
         namespace: 'AWS/Lambda',
         metric: 'Invocations',
         threshold: 5,
@@ -470,12 +498,12 @@ describe('#index', function () {
           ok: {
             Ref: `AwsAlertsCriticalOk`
           },
-      alert: {
-        Ref: `AwsAlertsCriticalAlert`
-      },
-      insufficientData: {
-        Ref: `AwsAlertsCriticalInsufficientData`
-      }
+          alert: {
+            Ref: `AwsAlertsCriticalAlert`
+          },
+          insufficientData: {
+            Ref: `AwsAlertsCriticalInsufficientData`
+          }
         },
         nonCritical: {
           alarm: {
@@ -630,7 +658,7 @@ describe('#index', function () {
       });
     });
 
-    it('should use globally defined nameTemplate when it`s not provided in definitions', function() {
+    it('should use globally defined nameTemplate when it`s not provided in definitions', function () {
       let config = {
         nameTemplate: '$[functionName]-global',
         function: ['functionErrors']
@@ -671,7 +699,7 @@ describe('#index', function () {
       });
     });
 
-    it('should use globally defined prefixTemplate when it`s not provided in definitions', function() {
+    it('should use globally defined prefixTemplate when it`s not provided in definitions', function () {
       let config = {
         nameTemplate: '$[functionName]-global',
         prefixTemplate: 'notTheStackName',
@@ -713,7 +741,7 @@ describe('#index', function () {
       });
     });
 
-    it('should overwrite globally defined nameTemplate using definitions', function() {
+    it('should overwrite globally defined nameTemplate using definitions', function () {
       let config = {
         nameTemplate: '$[functionName]-global',
         definitions: {
@@ -759,7 +787,7 @@ describe('#index', function () {
       });
     });
 
-    it('should overwrite globally defined prefixTemplate using definitions', function() {
+    it('should overwrite globally defined prefixTemplate using definitions', function () {
       let config = {
         nameTemplate: '$[functionName]-global',
         prefixTemplate: 'notTheStackName',
@@ -805,7 +833,56 @@ describe('#index', function () {
           }
         }
       });
-    });
+	});
+	
+	it('should skip alarms that are marked disabled', () => {
+		let config = {
+			definitions: {
+				functionErrors: {
+					enabled: false
+				}
+			},
+			function: [
+				'functionErrors',
+				'functionInvocations'
+			]
+		};
+
+		const plugin = pluginFactory(config);
+
+		config = plugin.getConfig();
+		const definitions = plugin.getDefinitions(config);
+		const alertTopics = plugin.compileAlertTopics(config);
+
+		plugin.compileAlarms(config, definitions, alertTopics);
+		expect(plugin.serverless.service.provider.compiledCloudFormationTemplate.Resources).toEqual({
+			FooFunctionInvocationsAlarm: {
+				Type: 'AWS::CloudWatch::Alarm',
+				Properties: {
+					Namespace: 'AWS/Lambda',
+					MetricName: 'Invocations',
+					Threshold: 100,
+					Statistic: 'Sum',
+					Period: 60,
+					EvaluationPeriods: 1,
+					DatapointsToAlarm: 1,
+					ComparisonOperator: 'GreaterThanOrEqualToThreshold',
+					Dimensions: [
+						{
+							Name: 'FunctionName',
+							Value: {
+								Ref: 'FooLambdaFunction'
+							}
+						}
+					],
+					AlarmActions: [],
+					OKActions: [],
+					InsufficientDataActions: [],
+					TreatMissingData: 'missing'
+				}
+			}
+		});
+	});
   });
 
   describe('#compileCloudWatchAlarms', () => {
@@ -913,6 +990,8 @@ describe('#index', function () {
 
       const definition = {
         description: 'An error alarm',
+        type: 'static',
+        name: 'test-alarm',
         namespace: 'AWS/Lambda',
         metric: 'Errors',
         threshold: 1,
@@ -970,6 +1049,8 @@ describe('#index', function () {
 
       const definition = {
         description: 'An error alarm',
+        type: 'static',
+        name: 'test-alarm',
         namespace: 'AWS/Lambda',
         metric: 'Errors',
         threshold: 1,
@@ -1013,7 +1094,6 @@ describe('#index', function () {
       });
     });
 
-
     it('should use the CloudFormation value ExtendedStatistic for p values', () => {
       const alertTopics = {
         ok: 'ok-topic',
@@ -1023,6 +1103,8 @@ describe('#index', function () {
 
       const definition = {
         description: 'An error alarm',
+        type: 'static',
+        name: 'test-alarm',
         namespace: 'AWS/Lambda',
         metric: 'Errors',
         threshold: 1,
@@ -1062,6 +1144,7 @@ describe('#index', function () {
         }
       });
     });
+
     it('should allow user to provide custom dimensions', () => {
       const alertTopics = {
         ok: 'ok-topic',
@@ -1071,6 +1154,8 @@ describe('#index', function () {
 
       const definition = {
         description: 'An error alarm',
+        type: 'static',
+        name: 'test-alarm',
         namespace: 'AWS/Lambda',
         metric: 'Errors',
         threshold: 1,
@@ -1079,7 +1164,7 @@ describe('#index', function () {
         evaluationPeriods: 1,
         comparisonOperator: 'GreaterThanThreshold',
         treatMissingData: 'breaching',
-        dimensions: [{'Name':'Cow', 'Value':'MOO'}, {'Name':'Duck', 'Value':'QUACK'}]
+        dimensions: [{ 'Name': 'Cow', 'Value': 'MOO' }, { 'Name': 'Duck', 'Value': 'QUACK' }]
       };
 
       const functionName = 'func-name';
@@ -1104,10 +1189,10 @@ describe('#index', function () {
           Dimensions: [{
             Name: "Cow",
             Value: "MOO"
-            },{
+          }, {
             Name: "Duck",
             Value: "QUACK"
-            },{
+          }, {
             Name: 'FunctionName',
             Value: {
               Ref: functionRef,
@@ -1117,6 +1202,7 @@ describe('#index', function () {
         }
       });
     });
+
     it('should add AlarmName property when nameTemplate is defined', () => {
       const alertTopics = {
         ok: 'ok-topic',
@@ -1124,9 +1210,11 @@ describe('#index', function () {
         insufficientData: 'insufficientData-topic',
       };
 
-       const definition = {
+      const definition = {
         nameTemplate: '$[functionName]-$[functionId]-$[metricName]-$[metricId]',
         description: 'An error alarm',
+        type: 'static',
+        name: 'test-alarm',
         namespace: 'AWS/Lambda',
         metric: 'Errors',
         threshold: 1,
@@ -1136,12 +1224,12 @@ describe('#index', function () {
         treatMissingData: 'breaching',
       };
 
-       const functionName = 'func-name';
-       const functionRef = 'func-ref';
+      const functionName = 'func-name';
+      const functionRef = 'func-ref';
 
-       const cf = plugin.getAlarmCloudFormation(alertTopics, definition, functionName, functionRef);
+      const cf = plugin.getAlarmCloudFormation(alertTopics, definition, functionName, functionRef);
 
-       expect(cf).toEqual({
+      expect(cf).toEqual({
         Type: 'AWS::CloudWatch::Alarm',
         Properties: {
           AlarmName: `fooservice-dev-${functionName}-${functionRef}-${definition.metric}-${definition.metric}`,
@@ -1165,5 +1253,74 @@ describe('#index', function () {
         }
       });
     });
-  })
+
+    it('should generate an AnomalyDetection alarm when type is anomalyDetection', () => {
+      const alertTopics = {
+        ok: 'ok-topic',
+        alarm: 'alarm-topic',
+        insufficientData: 'insufficientData-topic',
+      };
+
+      const definition = {
+        description: 'An error alarm',
+        type: 'anomalyDetection',
+        name: 'test-alarm',
+        namespace: 'AWS/Lambda',
+        metric: 'Errors',
+        threshold: 1,
+        period: 300,
+        statistic: 'Sum',
+        datapointsToAlarm: 1,
+        evaluationPeriods: 1,
+        comparisonOperator: 'GreaterThanThreshold',
+        treatMissingData: 'breaching'
+      };
+
+      const functionName = 'func-name';
+      const functionRef = 'func-ref';
+
+      const cf = plugin.getAlarmCloudFormation(alertTopics, definition, functionName, functionRef);
+
+      expect(cf).toEqual({
+        Type: 'AWS::CloudWatch::Alarm',
+        Properties: {
+          AlarmDescription: definition.description,
+          EvaluationPeriods: definition.evaluationPeriods,
+          DatapointsToAlarm: definition.datapointsToAlarm,
+          ComparisonOperator: definition.comparisonOperator,
+          TreatMissingData: 'breaching',
+          OKActions: ['ok-topic'],
+          AlarmActions: ['alarm-topic'],
+          InsufficientDataActions: ['insufficientData-topic'],
+          Metrics: [
+            {
+              Id: 'm1',
+              ReturnData: true,
+              MetricStat: {
+                Metric: {
+                  Namespace: definition.namespace,
+                  MetricName: definition.metric,
+                  Dimensions: [{
+                    Name: 'FunctionName',
+                    Value: {
+                      Ref: functionRef,
+                    }
+                  }]
+                },
+                Period: definition.period,
+                Stat: definition.statistic
+              }
+            },
+            {
+              Id: 'ad1',
+              Expression: `ANOMALY_DETECTION_BAND(m1, ${definition.threshold})`,
+              Label: `${definition.metric} (expected)`,
+              ReturnData: true
+            }
+          ],
+          ThresholdMetricId: 'ad1'
+        }
+      });
+    });
+  });
 });