Repository: serverless/serverless (JavaScript)
Date: 2019-06-05 20:52:40+00:00
Task type: Feature

Issue:
Feature Proposal: join logs permissions to allow for scaling
# This is a Feature Proposal

## Description
We have a very large serverless application that we're upgrading from sls 0.5 to sls 1. We've been hitting a lot of scaling issues; most recently an error `Maximum policy size of 10240 bytes exceeded for role`. Looking at the policy generated in `.serverless`, I realised we have a separate definition of permission for creating and putting to logs; that causes an issue with size when you have 50+ functions.

## Proposed solution
I tinkered a bit with serverless code and came up with this. 

1) Joining statement to allow creating log streams and putting log events for all functions
```
        "Policies": [
            {
                "PolicyName": "[TO BE REPLACED]",
                "PolicyDocument": {
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Action": ["logs:CreateLogStream", "logs:PutLogEvents"],
                            "Resource": []
                        }
                    ]
                }
            }
        ]
```

2) Simplify the piece of code that adds functions (resources) to this statement in `mergeIamTemplates.js`:

```
      this.serverless.service.provider.compiledCloudFormationTemplate
        .Resources[this.provider.naming.getRoleLogicalId()]
        .Properties
        .Policies[0]
        .PolicyDocument
        .Statement[0]
        .Resource
        .push({ 'Fn::Sub': 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}' +
          `:log-group:${this.provider.naming.getLogGroupName(functionObject.name)}:*` });

      // here the same code (with .Statement[1]) was deleted
```

## Concerns
1) I assume there has been a reason for these two statements to be separated?
2) I know that technically serverless framework isn't made for this scale and we'll keep hitting scaling issue. Long term, we'd like to separate our application into several stacks. But as we're only trying to upgrade serverless and not to change everything at once, this would be very useful for us.

## Similar or dependent issues:
* #2508 - Maximum policy size of 10240 bytes exceeded for role; this issue has been closed but there are still people experiencing this issue.
* https://github.com/dougmoscrop/serverless-plugin-split-stacks/issues/25 - closer description of the original issue that I originally believed to come from the split stacks plugin



Issue discussion:
This plugin may help: https://github.com/AntonBazhal/serverless-plugin-custom-roles
@dougmoscrop I'm afraid that won't work for us since about 3/4 of all permissions are easily coming from the logs policies.
Well, that plugin will put those permissions in *separate* policies attached to *separate* roles so you will not have one big 10KB policy.
I tried the plugin out but since it splits everything into separate resources, it re-introduces the issue with number of resources (`The CloudFormation template is invalid: Template format error: Number of resources, 223, is greater than maximum allowed, 200`). I think in our case, we'll need something that will _connect_ policies, not split them.

I understand ours is an edge-case and that the ideal would be to split our service into smaller one. Joining the two logs policies seems like a move however that would harm none and would make it easier for everyone who needs to develop serverless apps on a large scale?
Yeah, you use the split-stacks plugin _with_ the custom-roles plugin together to work around this.

There's no perfect solution, because serverless trying to programatically generate things, often on a per-function basis, is always going to find a way to hit some limit. The log statements don't help, but you will also hit this with just having a bunch of iamRoleStatements that allow you access to other resources within your sls app.

The statements you cited above aren't identical:

```
...
        .Statement[0]
        .Resource
        .push({ 'Fn::Sub': 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}' +
          `:log-group:${this.provider.naming.getLogGroupName(functionObject.name)}:*` });

...
        .Statement[1]
        .Resource
        .push({ 'Fn::Sub': 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}' +
          `:log-group:${this.provider.naming.getLogGroupName(functionObject.name)}:*:*` });
```

the diff is `:*` vs `:*:*` - I don't know if IAM requires the Resource part of Statement[1] to be as precise as it is or if they could be collapsed like you're suggesting. It would help, but if you're in a pinch the immediate-term thing to try would be combining those two plugins together.
I did use them together, even tried different order in `serverless.yml` to see if one coming before the other would make a difference. 

```yml
plugins:
  - serverless-plugin-split-stacks
  - serverless-plugin-custom-roles
  - "@connected-home/serverless-plugin-kms"
  - "@connected-home/serverless-plugin-stringify-variables"
  - serverless-plugin-tracing
  - serverless-webpack
```

I see what you mean by different precision for resources though. It does work using just `:*` but it would be good to know if there would be any security concerns?
Can you share your serverless.yml (in a gist or something? -- you can also bring this back to the split-stacks repo if you want to troubleshoot a workaround) when you used the custom roles plugin?
Here it is: https://gist.github.com/lithin/9010845ae97ebb0304a30bc89c44f516

I have tried this plugin order as well 👍 
```yml
plugins:
  - serverless-plugin-custom-roles
  - serverless-plugin-split-stacks
```

Both result in too many resources.

I'd prefer to keep the conversation here since this issue is likely to cause trouble to everyone who is upgrading sls0.5 to sls1 with a larger service. Just seems like it would be useful to keep the conversation close to the core :)
Hmm! So, the plugin (custom-roles) relies on serverless behavior to *remove* the "shared" iamRole/Statements:(https://github.com/serverless/serverless/blame/890c042fbd84b4a1091174e0b4d039be1a88f98d/lib/plugins/aws/package/lib/mergeIamTemplates.js#L53)

If you do an `sls package` - do you still see the policy there like you mentioned in your initial report? It should not be there..
I can see the policies being split. But that's where the problem with too many resources comes from.

```
Serverless: [serverless-plugin-split-stacks]: Summary: 137 resources migrated in to 3 nested stacks
Serverless: [serverless-plugin-split-stacks]:    Resources per stack:
Serverless: [serverless-plugin-split-stacks]:    - (root): 223
Serverless: [serverless-plugin-split-stacks]:    - APINestedStack: 48
Serverless: [serverless-plugin-split-stacks]:    - PermissionsNestedStack: 43
Serverless: [serverless-plugin-split-stacks]:    - VersionsNestedStack: 46
```

Root is at 223 which is 23 too many :/
Yes, sorry, I was not thinking properly about this before when you said too many resources.

The reason is that split-stacks does not migrate every resource, it has a whitelist, which by default is fairly conservative.

If you look at your root CloudFormation template, I suspect it's now over limit due to IAM roles. You can [modify the stack mapping](https://github.com/dougmoscrop/serverless-plugin-split-stacks#stack-mappings) to include those!
Wooo! 🎉 That did it :)

I added this to `stacks-map.js` in root folder:

```
const stacksMap = require('serverless-plugin-split-stacks').stacksMap;

stacksMap['AWS::IAM::Role'] = { destination: 'Roles' };
```

Thanks so much for your help 🙏 
I'm using CodeStar and I get the same error

> AWS CodeStar encountered an error while handling your request: One or more policies in IAM could not be created or updated. The number of resources in your project has exceeded the IAM policy size limit.. Try again with different parameters, or contact AWS Support.

I just have ~19 functions. there is no big project which can hold only that low amount of functions.
What fixed it for me is to override the default IAM role that Serverless gives for Lambda and replacing it with a different one using the `provider.role` option in serverless.yml. 

In my case, I gave `Allow` permissions for the log operations on the following resource: `arn:aws:logs:region:accountId:log-group:/aws/lambda/apiName-stageName-*:*:*`. That got rid of the one-log-permission-statement-per-function issue.
> Wooo! 🎉 That did it :)
> 
> I added this to `stacks-map.js` in root folder:
> 
> ```
> const stacksMap = require('serverless-plugin-split-stacks').stacksMap;
> 
> stacksMap['AWS::IAM::Role'] = { destination: 'Roles' };
> ```
> Thanks so much for your help 🙏

can you show me the full of stack-map.js file ?

Why was this closed? This is a real issue for growing applications. I added two new functions and now I can't deploy my application without serious effort in re-jigging these things. It seems reasonable to collapse the log stream permissions with a wildcard or provide that behavior behind a flag at least. I think the worst part about this issue is that it's not the developers fault for hitting this limit.
I would be good to have an option to make just:
```
    - Effect: Allow
      Action:
       - logs:CreateLogStream
      Resource: "arn:aws:logs:${self:provider.region}:*:log-group:*:*"
    - Effect: Allow
      Action:
       - logs:PutLogEvents
      Resource: "arn:aws:logs:${self:provider.region}:*:log-group:*:*:*"
```
instead of current resource lists.

Solution patch:
diff --git a/lib/plugins/aws/package/lib/mergeIamTemplates.js b/lib/plugins/aws/package/lib/mergeIamTemplates.js
index cb74e820d85..2698be83fc6 100644
--- a/lib/plugins/aws/package/lib/mergeIamTemplates.js
+++ b/lib/plugins/aws/package/lib/mergeIamTemplates.js
@@ -83,33 +83,32 @@ module.exports = {
       }
     );
 
-    this.serverless.service.getAllFunctions().forEach((functionName) => {
-      const functionObject = this.serverless.service.getFunction(functionName);
-
-      this.serverless.service.provider.compiledCloudFormationTemplate
-        .Resources[this.provider.naming.getRoleLogicalId()]
-        .Properties
-        .Policies[0]
-        .PolicyDocument
-        .Statement[0]
-        .Resource
-        .push({
-          'Fn::Sub': 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}' +
-            `:log-group:${this.provider.naming.getLogGroupName(functionObject.name)}:*`,
-        });
+    const logGroupsPrefix = this.provider.naming
+      .getLogGroupName(`${this.provider.serverless.service.service}-${this.provider.getStage()}`);
+
+    this.serverless.service.provider.compiledCloudFormationTemplate
+      .Resources[this.provider.naming.getRoleLogicalId()]
+      .Properties
+      .Policies[0]
+      .PolicyDocument
+      .Statement[0]
+      .Resource
+      .push({
+        'Fn::Sub': 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}' +
+          `:log-group:${logGroupsPrefix}*:*`,
+      });
 
-      this.serverless.service.provider.compiledCloudFormationTemplate
-        .Resources[this.provider.naming.getRoleLogicalId()]
-        .Properties
-        .Policies[0]
-        .PolicyDocument
-        .Statement[1]
-        .Resource
-        .push({
-          'Fn::Sub': 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}' +
-            `:log-group:${this.provider.naming.getLogGroupName(functionObject.name)}:*:*`,
-        });
-    });
+    this.serverless.service.provider.compiledCloudFormationTemplate
+      .Resources[this.provider.naming.getRoleLogicalId()]
+      .Properties
+      .Policies[0]
+      .PolicyDocument
+      .Statement[1]
+      .Resource
+      .push({
+        'Fn::Sub': 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}' +
+          `:log-group:${logGroupsPrefix}*:*:*`,
+      });
 
     if (this.serverless.service.provider.iamRoleStatements) {
       // add custom iam role statements


Test patch:
diff --git a/lib/plugins/aws/package/lib/mergeIamTemplates.test.js b/lib/plugins/aws/package/lib/mergeIamTemplates.test.js
index 43df16c776f..59a41473cba 100644
--- a/lib/plugins/aws/package/lib/mergeIamTemplates.test.js
+++ b/lib/plugins/aws/package/lib/mergeIamTemplates.test.js
@@ -50,7 +50,9 @@ describe('#mergeIamTemplates()', () => {
   it('should merge the IamRoleLambdaExecution template into the CloudFormation template',
     () => awsPackage.mergeIamTemplates()
       .then(() => {
-        const qualifiedFunction = awsPackage.serverless.service.getFunction(functionName).name;
+        const canonicalFunctionsPrefix =
+          `${awsPackage.serverless.service.service}-${awsPackage.provider.getStage()}`;
+
         expect(awsPackage.serverless.service.provider.compiledCloudFormationTemplate
           .Resources[awsPackage.provider.naming.getRoleLogicalId()]
         ).to.deep.equal({
@@ -96,7 +98,7 @@ describe('#mergeIamTemplates()', () => {
                       Resource: [
                         {
                           'Fn::Sub': 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:'
-                            + `log-group:/aws/lambda/${qualifiedFunction}:*`,
+                            + `log-group:/aws/lambda/${canonicalFunctionsPrefix}*:*`,
                         },
                       ],
                     },
@@ -108,7 +110,7 @@ describe('#mergeIamTemplates()', () => {
                       Resource: [
                         {
                           'Fn::Sub': 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:'
-                            + `log-group:/aws/lambda/${qualifiedFunction}:*:*`,
+                            + `log-group:/aws/lambda/${canonicalFunctionsPrefix}*:*:*`,
                         },
                       ],
                     },
@@ -374,91 +376,6 @@ describe('#mergeIamTemplates()', () => {
     });
   });
 
-  it('should update IamRoleLambdaExecution with a logging resource for the function', () => {
-    const qualifiedFunction = awsPackage.serverless.service.getFunction(functionName).name;
-    return awsPackage.mergeIamTemplates().then(() => {
-      expect(awsPackage.serverless.service.provider.compiledCloudFormationTemplate
-        .Resources[awsPackage.provider.naming.getRoleLogicalId()]
-        .Properties
-        .Policies[0]
-        .PolicyDocument
-        .Statement[0]
-        .Resource
-      ).to.deep.equal([
-        {
-          'Fn::Sub': 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:'
-            + `log-group:/aws/lambda/${qualifiedFunction}:*`,
-        },
-      ]);
-      expect(awsPackage.serverless.service.provider.compiledCloudFormationTemplate
-        .Resources[awsPackage.provider.naming.getRoleLogicalId()]
-        .Properties
-        .Policies[0]
-        .PolicyDocument
-        .Statement[1]
-        .Resource
-      ).to.deep.equal([
-        {
-          'Fn::Sub': 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:'
-            + `log-group:/aws/lambda/${qualifiedFunction}:*:*`,
-        },
-      ]);
-    });
-  });
-
-  it('should update IamRoleLambdaExecution with each function\'s logging resources', () => {
-    awsPackage.serverless.service.functions = {
-      func0: {
-        handler: 'func.function.handler',
-        name: 'func0',
-      },
-      func1: {
-        handler: 'func.function.handler',
-        name: 'func1',
-      },
-    };
-    return awsPackage.mergeIamTemplates().then(() => {
-      expect(awsPackage.serverless.service.provider.compiledCloudFormationTemplate
-        .Resources[awsPackage.provider.naming.getRoleLogicalId()]
-        .Properties
-        .Policies[0]
-        .PolicyDocument
-        .Statement[0]
-        .Resource
-      ).to.deep.equal(
-        [
-          {
-            'Fn::Sub': 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:'
-              + 'log-group:/aws/lambda/func0:*',
-          },
-          {
-            'Fn::Sub': 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:'
-              + 'log-group:/aws/lambda/func1:*',
-          },
-        ]
-        );
-      expect(awsPackage.serverless.service.provider.compiledCloudFormationTemplate
-        .Resources[awsPackage.provider.naming.getRoleLogicalId()]
-        .Properties
-        .Policies[0]
-        .PolicyDocument
-        .Statement[1]
-        .Resource
-      ).to.deep.equal(
-        [
-          {
-            'Fn::Sub': 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:'
-              + 'log-group:/aws/lambda/func0:*:*',
-          },
-          {
-            'Fn::Sub': 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:'
-              + 'log-group:/aws/lambda/func1:*:*',
-          },
-        ]
-        );
-    });
-  });
-
   it('should add default role if one of the functions has an ARN role', () => {
     awsPackage.serverless.service.functions = {
       func0: {