Repository: serverless/serverless (JavaScript)
Date: 2019-04-26 13:16:25+00:00
Task type: Feature

Issue:
API Gateway REST API Logs
# This is a Feature Proposal

## Description

So, I started today looking about how I can enable the logs on API Gateway. After some googling, I concluded that there isnt a way currently to do it with Serverless. I found a [year old discussion here](https://github.com/serverless/serverless/issues/1918) and that from an [external plugin](https://github.com/HyperBrain/serverless-aws-alias/issues/57).

Both of these require external plugins to be added and that's something I don't want to do and I would be much happier if that was something that this framework provided outside the box.

I would like to start a conversation on how we can provide this feature and also learn more about why haven't provided this feature. It may be something quite difficult and I am missing it.

### What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.

[Enable API Gateway logs](https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-cloudwatch-logs/) that I can later view from Cloudwatch.

### If there is additional config how would it look

Per service or per function configuration. Just a boolean would be nice.

```yaml
provider:
  enableAPIGLogs: true
```

### Similar or dependent issues:
* #1918
* External: https://github.com/HyperBrain/serverless-aws-alias/issues/57

## Additional Data

* ***Serverless Framework Version you're using***:
* ***Operating System***:
* ***Stack Trace***:
* ***Provider Error messages***:



Issue discussion:
Hi @kbariotis ,

I'd propose a similar solution as in my plugin 😄 . There it is possible to configure all APIG stage settings (not only the logs, most notably the trace setting and caching) and that on all possible levels (per service, per function, per endpoint). The reason why I think a possible native SLS solution should be of the same functionality is, that in practice you'd need this fine-grained sttings especially for development stages. Additionally it is not really good to have only a per service setting, as a service with many endpoints/methods suddenly generates quite big costs...
Here's where I commented the current configurability in the plugin: https://github.com/HyperBrain/serverless-aws-alias/issues/57#issuecomment-310364055
@HyperBrain absolutely! I am not arguing about the implementation. I am stating that this is something that SLS should provide imo. 🙂
I think there is no reason for not supporting the functionality.
We can add that to SLS Core if CloudFormation supports for API Gateway logs.
@horike37 This needs some deeper planning and evaluation, because the APIG stage configuration can only be set if there is a dedicated `AWS::Stage` resource available in the template.
Currently the stage is implicitly created by setting the literal string stage name at the `Deployment` resource. This has to be changed (which is a *breaking change*, because users will have to manually delete their stages first - you'll receive a "stage already exists" error). There is no workaround to make that non-breaking.
I'll try to look up the issues where the separate stage was already discussed.
@HyperBrain I see the issue now. Given that the main goal here is to separate the creation of the `Stage` resource before the `Deployment` resource, in order to be able to add properties(such as APIG Logs) we need to keep backward compatibility for the deployments that have already created a `Stage` implicitly.

In a scenario where there is a fresh deployment, everything will work fine.

In a scenario with an existing deployment, we have to find a way to update the previously implicitly created `Stage` resource.

If we add the `Stage` resource to the CF template pointing to the `Deployment` (through [the `DeploymetId` property](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-stage.html#cfn-apigateway-stage-deploymentid)) would that make CF assume that we want to update the resource and not re-create it?
> If we add the Stage resource to the CF template pointing to the Deployment (through the DeploymetId property) would that make CF assume that we want to update the resource and not re-create it?

No. I faced exactly that problem in my alias plugin. The "old" stage resource created by Serverless is completely separated, and there is no way to "reassign" or update it. A new managed `AWS::Stage` resource can only be created if the old one is deleted. The stage created with the string name only, seems to be treated as "unmanaged" from CF side.
Thanks @HyperBrain to point out the problem `stage already exists`, I deal with this issue the whole weekend, until I saw your comment.

https://forum.serverless.com/t/how-to-enable-cloud-watch-logs-for-api-gateway-using-serverless/2067/4?u=bill

Very happy to hear Serverless team plan to support logs as core service. 

As you suggested, I manually delete the old `dev` stage, then I got new error:

> An error occurred: ApiGatewayStage - CloudWatch Logs role ARN must be set in account settings to enable logging.

Do you know how to fix it?
@ozbillwang you will find more information about this [here](https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-cloudwatch-logs/). The Role that manages the API Gateway needs an extra policy.

Solution patch:
diff --git a/docs/providers/aws/events/apigateway.md b/docs/providers/aws/events/apigateway.md
index 7ee56678461..79e83da85fd 100644
--- a/docs/providers/aws/events/apigateway.md
+++ b/docs/providers/aws/events/apigateway.md
@@ -47,7 +47,10 @@ layout: Doc
   - [Share Authorizer](#share-authorizer)
   - [Resource Policy](#resource-policy)
   - [Compression](#compression)
-  - [AWS X-Ray Tracing](#aws-x-ray-tracing)
+  - [Stage specific setups](#stage-specific-setups)
+    - [AWS X-Ray Tracing](#aws-x-ray-tracing)
+    - [Tags / Stack Tags](#tags--stack-tags)
+    - [Logs](#logs)
 
 _Are you looking for tutorials on using API Gateway? Check out the following resources:_
 
@@ -1421,8 +1424,7 @@ Disabling settings might result in unexpected behavior. We recommend to remove a
 
 ### AWS X-Ray Tracing
 
-API Gateway supports a form of out of the box distributed tracing via [AWS X-Ray](https://aws.amazon.com/xray/) though enabling [active tracing](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-xray.html). To enable this feature for your serverless
-application's API Gateway add the following to your `serverless.yml`
+API Gateway supports a form of out of the box distributed tracing via [AWS X-Ray](https://aws.amazon.com/xray/) though enabling [active tracing](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-xray.html). To enable this feature for your serverless application's API Gateway add the following to your `serverless.yml`
 
 ```yml
 # serverless.yml
@@ -1447,3 +1449,17 @@ provider:
   tags:
     tagKey: tagValue
 ```
+
+### Logs
+
+Use the following configuration to enable API Gateway logs:
+
+```yml
+# serverless.yml
+provider:
+  name: aws
+  apiGateway:
+    logs: true
+```
+
+The log streams will be generated in a dedicated log group which follows the naming schema `/aws/api-gateway/{service}-{stage}`.
diff --git a/docs/providers/aws/guide/serverless.yml.md b/docs/providers/aws/guide/serverless.yml.md
index cea75c42fef..b38b243bdfb 100644
--- a/docs/providers/aws/guide/serverless.yml.md
+++ b/docs/providers/aws/guide/serverless.yml.md
@@ -62,6 +62,7 @@ provider:
     apiKeySourceType: HEADER # Source of API key for usage plan. HEADER or AUTHORIZER.
     minimumCompressionSize: 1024 # Compress response when larger than specified size in bytes (must be between 0 and 10485760)
     description: Some Description # optional description for the API Gateway stage deployment
+    logs: true # Optional configuration which specifies if API Gateway logs are used
   usagePlan: # Optional usage plan configuration
     quota:
       limit: 5000
diff --git a/lib/plugins/aws/lib/naming.js b/lib/plugins/aws/lib/naming.js
index 67f2de68ed5..7ad93bc53b5 100644
--- a/lib/plugins/aws/lib/naming.js
+++ b/lib/plugins/aws/lib/naming.js
@@ -273,6 +273,15 @@ module.exports = {
   getStageLogicalId() {
     return 'ApiGatewayStage';
   },
+  getApiGatewayLogGroupLogicalId() {
+    return 'ApiGatewayLogGroup';
+  },
+  getApiGatewayLogsRoleLogicalId() {
+    return 'IamRoleApiGatewayLogs';
+  },
+  getApiGatewayAccountLogicalId() {
+    return 'ApiGatewayAccount';
+  },
 
   // S3
   getDeploymentBucketLogicalId() {
diff --git a/lib/plugins/aws/package/compile/events/apiGateway/lib/stage.js b/lib/plugins/aws/package/compile/events/apiGateway/lib/stage.js
index 79a2228bacf..83f7ce26430 100644
--- a/lib/plugins/aws/package/compile/events/apiGateway/lib/stage.js
+++ b/lib/plugins/aws/package/compile/events/apiGateway/lib/stage.js
@@ -1,3 +1,5 @@
+/* eslint-disable no-use-before-define */
+
 'use strict';
 
 const _ = require('lodash');
@@ -5,7 +7,13 @@ const BbPromise = require('bluebird');
 
 module.exports = {
   compileStage() {
+    const service = this.serverless.service.service;
+    const stage = this.options.stage;
     const provider = this.serverless.service.provider;
+    const cfTemplate = this.serverless.service.provider.compiledCloudFormationTemplate;
+
+    // logs
+    const logs = provider.apiGateway && provider.apiGateway.logs;
 
     // TracingEnabled
     const tracing = provider.tracing;
@@ -25,37 +33,136 @@ module.exports = {
 
     // NOTE: the DeploymentId is random, therefore we rely on prior usage here
     const deploymentId = this.apiGatewayDeploymentLogicalId;
+    const logGrouLogicalId = this.provider.naming
+      .getApiGatewayLogGroupLogicalId();
+    const logsRoleLogicalId = this.provider.naming
+      .getApiGatewayLogsRoleLogicalId();
+    const accountLogicalid = this.provider.naming
+      .getApiGatewayAccountLogicalId();
+
     this.apiGatewayStageLogicalId = this.provider.naming
       .getStageLogicalId();
 
     // NOTE: right now we're only using a dedicated Stage resource
     // - if AWS X-Ray tracing is enabled
     // - if Tags are provided
+    // - if logs are enabled
     // We'll change this in the future so that users can
     // opt-in for other features as well
-    if (TracingEnabled || Tags.length > 0) {
-      _.merge(this.serverless.service.provider.compiledCloudFormationTemplate.Resources, {
-        [this.apiGatewayStageLogicalId]: {
-          Type: 'AWS::ApiGateway::Stage',
-          Properties: {
-            DeploymentId: {
-              Ref: deploymentId,
-            },
-            RestApiId: this.provider.getApiGatewayRestApiId(),
-            StageName: this.provider.getStage(),
-            TracingEnabled,
-            Tags,
+    if (logs || TracingEnabled || Tags.length > 0) {
+      const stageResource = {
+        Type: 'AWS::ApiGateway::Stage',
+        Properties: {
+          DeploymentId: {
+            Ref: deploymentId,
           },
+          RestApiId: this.provider.getApiGatewayRestApiId(),
+          StageName: this.provider.getStage(),
+          TracingEnabled,
+          Tags,
         },
-      });
+      };
+
+      // create log-specific resources
+      if (logs) {
+        _.merge(stageResource.Properties, {
+          AccessLogSetting: {
+            DestinationArn: {
+              'Fn::GetAtt': [
+                'ApiGatewayLogGroup',
+                'Arn',
+              ],
+            },
+            // eslint-disable-next-line
+            Format: 'requestId: $context.requestId, ip: $context.identity.sourceIp, caller: $context.identity.caller, user: $context.identity.user, requestTime: $context.requestTime, httpMethod: $context.httpMethod, resourcePath: $context.resourcePath, status: $context.status, protocol: $context.protocol, responseLength: $context.responseLength',
+          },
+          MethodSettings: [
+            {
+              DataTraceEnabled: true,
+              HttpMethod: '*',
+              ResourcePath: '/*',
+              LoggingLevel: 'INFO',
+            },
+          ],
+        });
+
+        _.merge(cfTemplate.Resources, {
+          [logGrouLogicalId]: getLogGroupResource(service, stage),
+          [logsRoleLogicalId]: getIamRoleResource(service, stage),
+          [accountLogicalid]: getAccountResource(logsRoleLogicalId),
+        });
+      }
+
+      _.merge(cfTemplate.Resources, { [this.apiGatewayStageLogicalId]: stageResource });
 
       // we need to remove the stage name from the Deployment resource
-      delete this.serverless.service.provider.compiledCloudFormationTemplate
-        .Resources[deploymentId]
-        .Properties
-        .StageName;
+      delete cfTemplate.Resources[deploymentId].Properties.StageName;
     }
 
     return BbPromise.resolve();
   },
 };
+
+function getLogGroupResource(service, stage) {
+  return ({
+    Type: 'AWS::Logs::LogGroup',
+    Properties: {
+      LogGroupName: `/aws/api-gateway/${service}-${stage}`,
+    },
+  });
+}
+
+function getIamRoleResource(service, stage) {
+  return ({
+    Type: 'AWS::IAM::Role',
+    Properties: {
+      AssumeRolePolicyDocument: {
+        Version: '2012-10-17',
+        Statement: [
+          {
+            Effect: 'Allow',
+            Principal: {
+              Service: [
+                'apigateway.amazonaws.com',
+              ],
+            },
+            Action: [
+              'sts:AssumeRole',
+            ],
+          },
+        ],
+      },
+      ManagedPolicyArns: [
+        'arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs',
+      ],
+      Path: '/',
+      RoleName: {
+        'Fn::Join': [
+          '-',
+          [
+            service,
+            stage,
+            {
+              Ref: 'AWS::Region',
+            },
+            'apiGatewayLogsRole',
+          ],
+        ],
+      },
+    },
+  });
+}
+
+function getAccountResource(logsRoleLogicalId) {
+  return ({
+    Type: 'AWS::ApiGateway::Account',
+    Properties: {
+      CloudWatchRoleArn: {
+        'Fn::GetAtt': [
+          logsRoleLogicalId,
+          'Arn',
+        ],
+      },
+    },
+  });
+}


Test patch:
diff --git a/lib/plugins/aws/lib/naming.test.js b/lib/plugins/aws/lib/naming.test.js
index c8420721de3..78f9a6b2377 100644
--- a/lib/plugins/aws/lib/naming.test.js
+++ b/lib/plugins/aws/lib/naming.test.js
@@ -462,6 +462,24 @@ describe('#naming()', () => {
     });
   });
 
+  describe('#getApiGatewayLogGroupLogicalId()', () => {
+    it('should return the API Gateway log group logical id', () => {
+      expect(sdk.naming.getApiGatewayLogGroupLogicalId()).to.equal('ApiGatewayLogGroup');
+    });
+  });
+
+  describe('#getApiGatewayLogsRoleLogicalId()', () => {
+    it('should return the API Gateway logs IAM role logical id', () => {
+      expect(sdk.naming.getApiGatewayLogsRoleLogicalId()).to.equal('IamRoleApiGatewayLogs');
+    });
+  });
+
+  describe('#getApiGatewayAccountLogicalId()', () => {
+    it('should return the API Gateway account logical id', () => {
+      expect(sdk.naming.getApiGatewayAccountLogicalId()).to.equal('ApiGatewayAccount');
+    });
+  });
+
   describe('#getDeploymentBucketLogicalId()', () => {
     it('should return "ServerlessDeploymentBucket"', () => {
       expect(sdk.naming.getDeploymentBucketLogicalId()).to.equal('ServerlessDeploymentBucket');
diff --git a/lib/plugins/aws/package/compile/events/apiGateway/lib/stage.test.js b/lib/plugins/aws/package/compile/events/apiGateway/lib/stage.test.js
index d2e33d1df45..290c65b9ad5 100644
--- a/lib/plugins/aws/package/compile/events/apiGateway/lib/stage.test.js
+++ b/lib/plugins/aws/package/compile/events/apiGateway/lib/stage.test.js
@@ -11,6 +11,9 @@ describe('#compileStage()', () => {
   let awsCompileApigEvents;
   let stage;
   let stageLogicalId;
+  let accountLogicalid;
+  let logsRoleLogicalId;
+  let logGroupLogicalId;
 
   beforeEach(() => {
     const options = {
@@ -20,6 +23,7 @@ describe('#compileStage()', () => {
     serverless = new Serverless();
     provider = new AwsProvider(serverless, options);
     serverless.setProvider('aws', provider);
+    serverless.service.service = 'my-service';
     serverless.service.provider.compiledCloudFormationTemplate = {
       Resources: {},
       Outputs: {},
@@ -31,6 +35,12 @@ describe('#compileStage()', () => {
     stage = awsCompileApigEvents.provider.getStage();
     stageLogicalId = awsCompileApigEvents.provider.naming
       .getStageLogicalId();
+    accountLogicalid = awsCompileApigEvents.provider.naming
+      .getApiGatewayAccountLogicalId();
+    logsRoleLogicalId = awsCompileApigEvents.provider.naming
+      .getApiGatewayLogsRoleLogicalId();
+    logGroupLogicalId = awsCompileApigEvents.provider.naming
+      .getApiGatewayLogGroupLogicalId();
     // mocking the result of a Deployment resource since we remove the stage name
     // when using the Stage resource
     awsCompileApigEvents.serverless.service.provider.compiledCloudFormationTemplate
@@ -192,4 +202,132 @@ describe('#compileStage()', () => {
       });
     });
   });
+
+  describe('logs', () => {
+    beforeEach(() => {
+      // setting up API Gateway logs
+      awsCompileApigEvents.serverless.service.provider.apiGateway = {
+        logs: true,
+      };
+    });
+
+    it('should create a dedicated stage resource if logs are configured', () =>
+      awsCompileApigEvents.compileStage().then(() => {
+        const resources = awsCompileApigEvents.serverless.service.provider
+          .compiledCloudFormationTemplate.Resources;
+
+        expect(resources[stageLogicalId]).to.deep.equal({
+          Type: 'AWS::ApiGateway::Stage',
+          Properties: {
+            RestApiId: {
+              Ref: awsCompileApigEvents.apiGatewayRestApiLogicalId,
+            },
+            DeploymentId: {
+              Ref: awsCompileApigEvents.apiGatewayDeploymentLogicalId,
+            },
+            StageName: 'dev',
+            Tags: [],
+            TracingEnabled: false,
+            MethodSettings: [
+              {
+                DataTraceEnabled: true,
+                HttpMethod: '*',
+                LoggingLevel: 'INFO',
+                ResourcePath: '/*',
+              },
+            ],
+            AccessLogSetting: {
+              DestinationArn: {
+                'Fn::GetAtt': [
+                  logGroupLogicalId,
+                  'Arn',
+                ],
+              },
+              // eslint-disable-next-line
+              Format: 'requestId: $context.requestId, ip: $context.identity.sourceIp, caller: $context.identity.caller, user: $context.identity.user, requestTime: $context.requestTime, httpMethod: $context.httpMethod, resourcePath: $context.resourcePath, status: $context.status, protocol: $context.protocol, responseLength: $context.responseLength',
+            },
+          },
+        });
+
+        expect(resources[awsCompileApigEvents.apiGatewayDeploymentLogicalId]).to.deep.equal({
+          Properties: {},
+        });
+      }));
+
+    it('should create a Log Group resource', () =>
+      awsCompileApigEvents.compileStage().then(() => {
+        const resources = awsCompileApigEvents.serverless.service.provider
+          .compiledCloudFormationTemplate.Resources;
+
+        expect(resources[logGroupLogicalId]).to.deep.equal({
+          Type: 'AWS::Logs::LogGroup',
+          Properties: {
+            LogGroupName: '/aws/api-gateway/my-service-dev',
+          },
+        });
+      }));
+
+    it('should create a IAM Role resource', () =>
+      awsCompileApigEvents.compileStage().then(() => {
+        const resources = awsCompileApigEvents.serverless.service.provider
+          .compiledCloudFormationTemplate.Resources;
+
+        expect(resources[logsRoleLogicalId]).to.deep.equal({
+          Type: 'AWS::IAM::Role',
+          Properties: {
+            AssumeRolePolicyDocument: {
+              Statement: [
+                {
+                  Action: [
+                    'sts:AssumeRole',
+                  ],
+                  Effect: 'Allow',
+                  Principal: {
+                    Service: [
+                      'apigateway.amazonaws.com',
+                    ],
+                  },
+                },
+              ],
+              Version: '2012-10-17',
+            },
+            ManagedPolicyArns: [
+              'arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs',
+            ],
+            Path: '/',
+            RoleName: {
+              'Fn::Join': [
+                '-',
+                [
+                  'my-service',
+                  'dev',
+                  {
+                    Ref: 'AWS::Region',
+                  },
+                  'apiGatewayLogsRole',
+                ],
+              ],
+            },
+          },
+        });
+      }));
+
+    it('should create an Account resource', () =>
+      awsCompileApigEvents.compileStage().then(() => {
+        const resources = awsCompileApigEvents.serverless.service.provider
+          .compiledCloudFormationTemplate.Resources;
+
+        expect(resources[accountLogicalid]).to.deep.equal({
+          Type: 'AWS::ApiGateway::Account',
+          Properties: {
+            CloudWatchRoleArn: {
+              'Fn::GetAtt': [
+                logsRoleLogicalId,
+                'Arn',
+              ],
+            },
+          },
+        });
+      }));
+  });
 });