Repository: serverless/serverless (JavaScript) — Bug Fix

Issue:
Enabling API Gateway Logs in serverless.yml tries to create new IAM Role and a new Lambda
# Bug Report

## Description

1. What did you do?
 ```yml
logs:
    restApi: # Optional configuration which specifies if API Gateway logs are used. This can either be set to true to use defaults, or configured via subproperties.
      executionLogging: true # Optional configuration which enables or disables execution logging. Defaults to true.
      level: INFO # Optional configuration which specifies the log level to use for execution logging. May be set to either INFO or ERROR.
      fullExecutionData: true # Optional configuration which specifies whether or not to log full requests/responses for execution logging. Defaults to true.
      role: <role that allows pushing to logs to cloudwatch>#
```
I have added this configuration in my serverless.yml file to enable cloudwatch logging for API gateway.

2.  What happened?
 I got the following error while I tried to deploy
```
  An error occurred: IamRoleCustomResourcesLambdaExecution - API: iam:CreateRole User: arn:aws:sts::<some role name> is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::<some resource name>with an explicit deny
```
I have restricted access to this account and i dont have access to create IAM roles.I have observed that it tries to create a role (`IamRoleCustomResourcesLambdaExecution`) and new Lambda `-custom-resource-apigw-cw-role`

3. What should've happened?
 It should have enabled cloudwatch logs for API gateway

4. What's the content of your `serverless.yml` file?
```yml
# along with other resources
logs:
    restApi: # Optional configuration which specifies if API Gateway logs are used. This can either be set to true to use defaults, or configured via subproperties.
      executionLogging: true # Optional configuration which enables or disables execution logging. Defaults to true.
      level: INFO # Optional configuration which specifies the log level to use for execution logging. May be set to either INFO or ERROR.
      fullExecutionData: true # Optional configuration which specifies whether or not to log full requests/responses for execution logging. Defaults to true.
      role: <role to write to cloudwatch> #
```

Similar or dependent issues:

- #6134 
This issue was closed with suggesting a resolution.Hence creating another issue.



Discussion:
I'm having the same issue.
The error seems to be here.

https://github.com/serverless/serverless/blob/d33ee52d28b3d19c4df9bfdd63ac86b63ee6995d/lib/plugins/aws/package/compile/events/lib/ensureApiGatewayCloudWatchRole.js#L16

Even though `roleArn` is configured via `logs.restApi.role`, it still tries to create a custom resource `apiGatewayCloudWatchRole`.
In that case specify externally created role that should cover custom resource lambda invocations.
Role's ARN should be passed to `provider.cfnRole` setting.

Still, note this role will cover both CloudFormation stack deployments, and cover Lambda invocations, so both services need to be listed as trusted principal services (if not, deployments will fail as here: https://github.com/serverless/serverless/issues/6876)
@medikoo 

- I just need serverless to enable logs while creating/Adding resources to API gateway and I have provided the role in restApi.role.
- I do not want serverless to create any custom lambda resources just to enable logs in API gateway.I just want API gateway logs to flow to cloudwatch.

Can you help me understand why serverless is trying to create custom resources in the first place ?

> Can you help me understand why serverless is trying to create custom resources in the first place ?

Custom resources solve configuration which cannot be done efficiently via CloudFormation (it's either not supported, or supported in limited form).

In case of API Gateway logs, we need to ensure that there's IAM role with appriopriate access rights assigned to region wide APIGW CloudWatch logs role setting.

While in theory that can be accomplished via CloudFormation, in that form it requires creation of new IAM role with every new stack. We solve it via custom resources, so we reuse same role resource across different stacks.


In my case I have specific role created by my cloud admin team for API gateway to push logs to cloudwatch.
How can I use the same role  in CF stack to push API gateway logs.?
> How can I use the same role in CF stack to push API gateway logs.?

Configure its arn at `provider.logs.restApi.role` in `serverless.yml`
@medikoo 

So even if i set the above configuration, it still tries to create a new role.
That is the actual problem I am facing.
Is it possible to add some parameter "patchAPIGatewayAccountRole" with a default value of "true"? So if false, it wouldn't create a custom lambda and add CloudWatch put logs role to API Gateway account. 
The intention is to have a possibility to manage API Gateway account role by [CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-account.html) at some earlier step for account and not from serverless framework.

Apparently https://github.com/serverless/serverless/pull/6747 introduced the functionality to wrap the role into a custom CloudFormation resources which seems to be the main issue here.

@sunilhari so the idea is to simply not use custom CloudFormation resources at all when using the `provider.restApi.logs` property?

I'm not too familiar with the initial implementation of using custom CloudFormation resources for REST API roles. @medikoo any objections to change the behavior of passing everything through custom CloudFormation resources? Are there any downsides to this approach?
@pmuens to my understanding, the issue is, that what was implemented here: https://github.com/serverless/serverless/pull/6747 doesn't seem to work (I contacted you, as you were finalizing that PR :)

I've explained [here](https://github.com/serverless/serverless/issues/6973#issuecomment-557483607) why we handle that via custom resource. Technically via CF we will have to create with each new stack new role, without ability to remove it once stack is removed (hence in case where a lot of stacks are processed, user will end with many not needed roles created in his AWS console)
> @pmuens to my understanding, the issue is, that what was implemented here: #6747 doesn't seem to work (I contacted you, as you were finalizing that PR :)

I just tested it and it works just fine on my machine. The `role` arn I specify is set as the CloudWatch log role ARN for API Gateway.

> I've explained here why we handle that via custom resource. Technically via CF we will have to create with each new stack new role, without ability to remove it once stack is removed (hence in case where a lot of stacks are processed, user will end with many not needed roles created in his AWS console)

I see. So it's an either / or here. Either we go via custom CloudFormation resources to not deal with dozens of roles or we don't and users will face the aforementioned issue of having a bunch of unused roles in their account when they remove their stacks.

Given that, what is the solution here?
> Given that, what is the solution here?

I think we should coin, whether there is a bug or not.

@sunilhari and few others state, that even with `provider.logs.restApi.role` setting being set, the _apiGatewayCloudWatchRole_ custom resource attempts to create a new IAM role.

@pmuens you state that you've just tested it, and that's not the case, correct?
> @pmuens you state that you've just tested it, and that's not the case, correct?

@medikoo Yes, it's working in my case. It's still creating a role for the Lambda to execute the custom resource in which we wrap the role creation (see implementation in https://github.com/serverless/serverless/pull/6747).

According to the issue description, creating a new role is not allowed, correct @sunilhari? So apparently we can't wrap the role creation in a custom resource in that case since the custom resource which is backed by a Lambda needs an IAM Role to execute the custom resource Lambda code. That seems to be the main issue here.
> @medikoo Yes, it's working in my case. It's still creating a role for the Lambda to execute the custom resource in which we wrap the role creation (see implementation in #6747).

ok but that we allow to override with `provider.cfnRole`. If it's set, new role should not be created.

So the solution is to have both configured `provider.cfnRole` and `provider.logs.restApi.role` that should ensure there's not attempt to create a new role by the framework.
@sunilhari can you confirm then, if you still face the issue, having both `provider.cfnRole` and `provider.logs.restApi.role` configured ?
Correct me if am wrong.
So i will have to configure IAM role of the user account who is executing the CF stack.
In another words,I should specify my access role here
Is my understanding correct?
@sunilhari via `cfnRole` setting you may provision a dedicated role which should be used for stack deployment. For more info see `RoleARN` explanation at https://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_CreateStack.html 

The same role then will be used for custom resource lambda invocations. This the way to ensure that Framework will not create its own dedicated role for that.
Hi, What's the suggested solution? Could someone explain? Thanks
@ywang9009gmail The solution is mentioned a few comments above:

> So the solution is to have both configured provider.cfnRole and provider.logs.restApi.role that should ensure there's not attempt to create a new role by the framework.

The effect is that serverless will then use the `cfnRole` to run that custom resource lambda that checks and assigns to API Gateway the CloudWatch role you specified by `restApi.role`. If you already specify a `cfnRole` for limited-access deployments, that role must have a number of permissions assigned which were mentioned across various other issues. This is what policies I had to add to get it working (statements below extracted from role definition of MyCfnRole, I specified only stuff I had to add to be able to deploy once I enabled the logs):

```
Resources:
  MyCfnRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: MyCfnRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
[...]
      Policies:
        - PolicyName: CloudWatch
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - iam:PassRole
                Resource:
                  - !Sub arn:#{AWS::Partition}:iam::#{AWS::AccountId}:role/MyCfnRole
                  - !Sub arn:#{AWS::Partition}:iam::#{AWS::AccountId}:role/MyApiLogRole
              - Effect: Allow
                Action:
                  - apigateway:GET
                  - apigateway:PATCH
                Resource:
                  - arn:#{AWS::Partition}:apigateway:*::/account
[...]

      ManagedPolicyArns:
        # Required for deploying custom resources (which serverless uses to activate API Gateway logs)
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
```
With this above config, I can confirm it works using the latest serverless v1.63, so I mentioned here in case somebody else hits it.

@medikoo @pmuens  I played with this and I have to say I do not like the custom resource solution. At a minimum, it **needs** a dedicated documentation page to explain the effects of each setting, I had to wade through multiple github issues to figure out how it works and basically wasted a day. Admittedly, part of it discovering the missing permissions, given that CloudFormation is not always helpful. I have also ran into #6643, which in my case was likely due to lack of permissions, but the error was hidden away by the 10-retry logic in the custom resource which possibly got me the "too many requests errors".

I understand that the CloudWatch log ARN is a global setting and you can't add/remove it at will. What I would like to see then is for serverless **not** to try to set it at all. I am perfectly fine setting up a role at account level and assign it to API Gateway directly through CloudFormation resources (I do this in a separate "account setup" deployment). And I do not want the API deployment to create a custom resource just to double-check my ARN there, it looks like too much hassle to me and it forces me to grant additional permissions which are better left to the "account setup" part.

One simple way is to add an extra boolean setting to control this behavior. When false, assume it's been assigned externally and do not create the custom resource (but: more settings = more confusion). 

Another way is to just make this the default behavior and assume that when a role is given, it's been also configured in API Gateway, so no need to use the custom resource at all. Document this (see above) and it'll be fine. Figuring out all those permissions certainly takes more time than setting up the role via `resources` section.
I was also facing the issue that even though I specified an existing role, a new one was created.

I'm not sure if this is a reliable workaround but what seems to work for me is simply creating an `AWS::ApiGateway::Account` manually.

```
ApiGatewayAccount:
        Type: 'AWS::ApiGateway::Account'
        DependsOn: ApiGatewayRestApi
        Properties:
          CloudWatchRoleArn: 'arn:aws:iam::#{AWS::AccountId}:role/cloudwatch-logs-role-name'
```

After this, the correct role (`'arn:aws:iam::#{AWS::AccountId}:role/cloudwatch-logs-role-name'`) was referenced in the API Gateway settings.
@coyoteecd Thanks for all this input, it's very valuable

> I have also ran into #6643, which in my case was likely due to lack of permissions, but the error was hidden away by the 10-retry logic

It'll be good to have that fixed. Can you create a separate issue and explain exactly what error occured that should not be retried, when it was.

> And I do not want the API deployment to create a custom resource just to double-check my ARN there, it looks like too much hassle to me and it forces me to grant additional permissions which are better left to the "account setup" part.

We're definitely open for PR that provides that and improves the documentation so it's more transparent
> I was also facing the issue that even though I specified an existing role, a new one was created.

@larsbe are you using maybe Serverless Dashboard? Maybe it was the log access role that's created for you (that could be overridden by `custom.enterprise.logAccessIamRole` setting).

Otherwise if you feel that role remains created, when it shouldn't  please create a reproducible minimal test case.  Currently we're not aware of any bugs on that level.

Resolving patch:
diff --git a/docs/providers/aws/events/apigateway.md b/docs/providers/aws/events/apigateway.md
index cef6ea35b12..1e6f068a88c 100644
--- a/docs/providers/aws/events/apigateway.md
+++ b/docs/providers/aws/events/apigateway.md
@@ -1575,6 +1575,30 @@ provider:
 
 The log streams will be generated in a dedicated log group which follows the naming schema `/aws/api-gateway/{service}-{stage}`.
 
+To be able to write logs, API Gateway [needs a CloudWatch role configured](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html). This setting is per region, shared by all the APIs. There are three approaches for handling it:
+
+- Let Serverless create and assign an IAM role for you (default behavior). Note that since this is a shared setting, this role is not removed when you remove the deployment.
+- Let Serverless assign an existing IAM role that you created before the deployment, if not already assigned:
+
+  ```yml
+  # serverless.yml
+  provider:
+    logs:
+      restApi:
+        role: arn:aws:iam::123456:role
+  ```
+
+- Do not let Serverless manage the CloudWatch role configuration. In this case, you would create and assign the IAM role yourself, e.g. in a separate "account setup" deployment:
+
+  ```yml
+  provider:
+    logs:
+      restApi:
+        roleManagedExternally: true # disables automatic role creation/checks done by Serverless
+  ```
+
+**Note:** Serverless configures the API Gateway CloudWatch role setting using a custom resource lambda function. If you're using `cfnRole` to specify a limited-access IAM role for your serverless deployment, the custom resource lambda will assume this role during execution.
+
 By default, API Gateway access logs will use the following format:
 
 ```
diff --git a/docs/providers/aws/guide/serverless.yml.md b/docs/providers/aws/guide/serverless.yml.md
index c275ab0f9f7..907003d8397 100644
--- a/docs/providers/aws/guide/serverless.yml.md
+++ b/docs/providers/aws/guide/serverless.yml.md
@@ -152,7 +152,8 @@ provider:
       executionLogging: true # Optional configuration which enables or disables execution logging. Defaults to true.
       level: INFO # Optional configuration which specifies the log level to use for execution logging. May be set to either INFO or ERROR.
       fullExecutionData: true # Optional configuration which specifies whether or not to log full requests/responses for execution logging. Defaults to true.
-      role: arn:aws:iam::123456:role # Optional IAM role for ApiGateway to use when managing CloudWatch Logs
+      role: arn:aws:iam::123456:role # Existing IAM role for ApiGateway to use when managing CloudWatch Logs. If 'role' is not configured, a new role is automatically created.
+      roleManagedExternally: false # Specifies whether the ApiGateway CloudWatch Logs role setting is not managed by Serverless. Defaults to false.
     websocket: # Optional configuration which specifies if Websocket logs are used. This can either be set to `true` to use defaults, or configured via subproperties.
       level: INFO # Optional configuration which specifies the log level to use for execution logging. May be set to either INFO or ERROR.
     frameworkLambda: true # Optional, whether to write CloudWatch logs for custom resource lambdas as added by the framework
diff --git a/lib/plugins/aws/package/compile/events/lib/ensureApiGatewayCloudWatchRole.js b/lib/plugins/aws/package/compile/events/lib/ensureApiGatewayCloudWatchRole.js
index 04b005be988..503cbcedcee 100644
--- a/lib/plugins/aws/package/compile/events/lib/ensureApiGatewayCloudWatchRole.js
+++ b/lib/plugins/aws/package/compile/events/lib/ensureApiGatewayCloudWatchRole.js
@@ -6,15 +6,21 @@ const { addCustomResourceToService } = require('../../../../customResources');
 
 module.exports = memoize(provider =>
   BbPromise.try(() => {
-    const cfTemplate = provider.serverless.service.provider.compiledCloudFormationTemplate;
-    const customResourceLogicalId = provider.naming.getCustomResourceApiGatewayAccountCloudWatchRoleResourceLogicalId();
-    const customResourceFunctionLogicalId = provider.naming.getCustomResourceApiGatewayAccountCloudWatchRoleHandlerFunctionLogicalId();
-
     // There may be a specific role ARN provided in the configuration
     const config = provider.serverless.service.provider;
     const restApi = config.logs && config.logs.restApi;
     const configuredRoleArn = restApi && restApi.role;
 
+    // Check for CloudWatch role may be disabled, if so, skip it
+    const configuredRoleManagedExternally = restApi && restApi.roleManagedExternally;
+    if (configuredRoleManagedExternally) {
+      return null;
+    }
+
+    const cfTemplate = provider.serverless.service.provider.compiledCloudFormationTemplate;
+    const customResourceLogicalId = provider.naming.getCustomResourceApiGatewayAccountCloudWatchRoleResourceLogicalId();
+    const customResourceFunctionLogicalId = provider.naming.getCustomResourceApiGatewayAccountCloudWatchRoleHandlerFunctionLogicalId();
+
     cfTemplate.Resources[customResourceLogicalId] = {
       Type: 'Custom::ApiGatewayAccountRole',
       Version: 1.0,


Test patch:
diff --git a/lib/plugins/aws/package/compile/events/apiGateway/lib/stage/index.test.js b/lib/plugins/aws/package/compile/events/apiGateway/lib/stage/index.test.js
index 932fee7dde3..2f9d64df6f0 100644
--- a/lib/plugins/aws/package/compile/events/apiGateway/lib/stage/index.test.js
+++ b/lib/plugins/aws/package/compile/events/apiGateway/lib/stage/index.test.js
@@ -286,7 +286,7 @@ describe('#compileStage()', () => {
       });
     });
 
-    it('should ensure ClousWatch role custom resource', () => {
+    it('should ensure CloudWatch role custom resource', () => {
       return awsCompileApigEvents.compileStage().then(() => {
         const resources =
           awsCompileApigEvents.serverless.service.provider.compiledCloudFormationTemplate.Resources;
@@ -300,5 +300,24 @@ describe('#compileStage()', () => {
         ).to.equal(true);
       });
     });
+
+    it('should skip CloudWatch role custom resource when restApi.roleManagedExternally is set', () => {
+      awsCompileApigEvents.serverless.service.provider.logs.restApi = {
+        roleManagedExternally: true,
+      };
+
+      return awsCompileApigEvents.compileStage().then(() => {
+        const resources =
+          awsCompileApigEvents.serverless.service.provider.compiledCloudFormationTemplate.Resources;
+
+        expect(
+          _.isObject(
+            resources[
+              awsCompileApigEvents.provider.naming.getCustomResourceApiGatewayAccountCloudWatchRoleResourceLogicalId()
+            ]
+          )
+        ).to.equal(false);
+      });
+    });
   });
 });
diff --git a/lib/plugins/aws/package/compile/events/lib/ensureApiGatewayCloudWatchRole.test.js b/lib/plugins/aws/package/compile/events/lib/ensureApiGatewayCloudWatchRole.test.js
index b38a9943fc2..ba24fe5d142 100644
--- a/lib/plugins/aws/package/compile/events/lib/ensureApiGatewayCloudWatchRole.test.js
+++ b/lib/plugins/aws/package/compile/events/lib/ensureApiGatewayCloudWatchRole.test.js
@@ -66,6 +66,23 @@ describe('ensureApiGatewayCloudWatchRole', () => {
     });
   });
 
+  describe('when role assignment is managed externally', () => {
+    it('should not add any custom resources', () => {
+      provider.serverless.service.provider.logs = {
+        restApi: {
+          role: 'arn:aws:iam::XXXXX:role/api-gateway-role',
+          roleManagedExternally: true,
+        },
+      };
+
+      return expect(ensureApiGatewayCloudWatchRole(provider)).to.eventually.be.fulfilled.then(
+        () => {
+          expect(resources[customResourceLogicalId]).to.be.undefined;
+        }
+      );
+    });
+  });
+
   describe('when leveraging custom resources', () => {
     it('Should memoize custom resource generator', () => {
       return expect(