Repository: cloud-custodian/cloud-custodian

Issue:
0.8.45.0 and 0.8.45.1 Releases Break Lambda
We're using c7n-org and c7n, and upon updating to 0.8.45.0 the lambdas deployed start to throw an error about a missing CloudWatch prefix. The same error is thrown in 0.8.45.1. Is there a new parameter we need to configure on the policies? For what it's worth, manually pushing a lambda with just c7n and not c7n-org does not throw this error. 

Stack trace:

```
Invalid length for parameter logGroupNamePrefix, value: 0, valid range: 1-inf
Traceback (most recent call last):
  File "/var/task/custodian_policy.py", line 4, in run
    return handler.dispatch_event(event, context)
  File "/var/task/c7n/handler.py", line 168, in dispatch_event
    p.push(event, context)
  File "/var/task/c7n/policy.py", line 990, in push
    return mode.run(event, lambda_ctx)
  File "/var/task/c7n/policy.py", line 590, in run
    return PullMode.run(self)
  File "/var/task/c7n/policy.py", line 278, in run
    with self.policy.ctx:
  File "/var/task/c7n/ctx.py", line 89, in __enter__
    self.logs.__enter__()
  File "/var/task/c7n/output.py", line 340, in __enter__
    self.join_log()
  File "/var/task/c7n/output.py", line 349, in join_log
    self.handler = self.get_handler()
  File "/var/task/c7n/resources/aws.py", line 273, in get_handler
    session_factory=lambda x=None: self.ctx.session_factory(region=self.region))
  File "/var/task/c7n/log.py", line 91, in __init__
    logGroupNamePrefix=self.log_group)['logGroups']
  File "/var/task/c7n/utils.py", line 390, in _retry
    return func(*args, **kw)
  File "/var/runtime/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 634, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/var/runtime/botocore/client.py", line 682, in _convert_to_request_dict
    api_params, operation_model)
  File "/var/runtime/botocore/validate.py", line 297, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
```


Discussion:
whats your command line for invoking? are you specifying a cloud watch log group?
Our command line for the apply is something like 
`c7n-org run --config accounts.yml $REGIONS --tags nonprod --policytags nonprod --output-dir ./output --use $MERGED_POLICIES` where `REGIONS` is just a list of regions applicable and `MERGED_POLICIES` is various files collapsed into one for deployment's sake. We haven't explicitly specified a cloud watch log group anywhere, but version 0.8.44.3 worked just fine. The lambdas are also logging out to a cloudwatch log group the same as before, so I'm unsure of what this error is specifying when saying it's missing the key. 
Do you have a sample of one of these policies that’s failing? The trace back looks like a mode: periodic.
The policy below is pretty typical for our implementation. The redacted fields are our Slack webhook and the ARN of the IAM role provisioned for the Lambdas.

```yaml
policies:
  - name: s3-bucket-secure-transport
    description: &s3_secure_transport S3 buckets should deny all traffic that doesn't go over HTTPS.
    tags: [nonprod, prod]
    region: us-east-1
    resource: s3
    filters:
      - type: missing-statement
        statement_ids:
          - C7nSecureTransport
    actions:
      - type: set-statements
        statements:
          - Sid: "C7nSecureTransport"
            Effect: "Deny"
            Action: ["s3:GetObject","s3:PutObject"]
            Principal:
              AWS: "*"
            Resource: ["arn:aws:s3:::{bucket_name}/*","arn:aws:s3:::{bucket_name}"]
            Condition:
              Bool:
                "aws:SecureTransport": false
      - type: notify
        violation_desc: *s3_secure_transport
        action_desc: Adding secure transport statement to S3 bucket policy
        to:
          - redacted
        transport:
          type: sqs
          # Quotes + no spaces is required for var substitution.
          queue: "{notify_queue_arn}"
    mode:
      type: periodic
      schedule: "rate(1 day)"
      role: arn:aws:iam::{account_id}:role/redacted
```
i suspect it maybe related to this pr with orgs, though i haven't confirmed the issue yet. 
https://github.com/cloud-custodian/cloud-custodian/pull/4809

is there something customized about your installation? i haven't seen any other reports of issues.

if you downloaded one of these lambdas it would be helpful to see what's in the policy.json file for the execution-options key.
We haven't customized anything about Custodian itself, no. Below is a redacted copy of the `config.json` file that was in a lambda throwing the same error as above.

```json
{
  "execution-options": {
    "region": "us-west-2",
    "regions": [],
    "cache": "/root/.cache/c7n-org/redacted-us-west-2.cache",
    "profile": null,
    "account_id": "redacted",
    "assume_role": "arn:aws:iam::redacted:role/pet/pet-cloud-custodian-pipeline-popular-cicada",
    "external_id": null,
    "log_group": null,
    "tracer": "default",
    "metrics_enabled": false,
    "metrics": null,
    "output_dir": "./output/redacted/us-west-2",
    "cache_period": 15,
    "dryrun": false,
    "authorization_file": null
  },
  "policies": [
    {
      "actions": [
        {
          "key": "alias/aws/ebs",
          "state": true,
          "type": "set-ebs-encryption"
        },
        {
          "action_desc": "Turning on default EBS encryption using default encryption key `alias/aws/ebs`.",
          "slack_template": "slack_default",
          "to": [
            "contact",
            "redacted"
          ],
          "transport": {
            "queue": "https://sqs.us-east-1.amazonaws.com/redacted/c7n_mailer20190522205029608700000001",
            "type": "sqs"
          },
          "type": "notify",
          "violation_desc": "Accounts should have the default encryption setting for EBS volumes turned on."
        }
      ],
      "description": "Accounts should have the default encryption setting for EBS volumes turned on.",
      "filters": [
        {
          "state": false,
          "type": "default-ebs-encryption"
        }
      ],
      "mode": {
        "execution-options": {
          "log_group": "/cloud-custodian/"
        },
        "packages": [
          "certifi"
        ],
        "role": "arn:aws:iam::redacted:role/pet/pet-cloud-custodian-lambda-popular-cicada",
        "schedule": "rate(1 day)",
        "type": "periodic",
        "tags": {
          "custodian-info": "mode=periodic:version=0.8.45.0"
        }
      },
      "name": "default-ebs-encryption",
      "resource": "aws.account",
      "tags": [
        "nonprod"
      ]
    }
  ]
}

```
I had this issue too, adding cloudwatch-logs log_group to execution-options fixed it for me
> I had this issue too, adding cloudwatch-logs log_group to execution-options fixed it for me

I did notice that there are two places in there where the log_group is defined; I'll try this when I get a chance and report back. 
I found a way to download our policies after they've been touched by a regulating script, and it explicitly defines a log group in the mode. Is there another place to define a log_group?  Policy after it has been updated looks more like this:

```yaml
policies:
  - actions:
      - days: 35
        enforce: exact
        type: retention
      - action_desc: Set retention period to 35 days for any Aurora clusters that are
          set to less than 35 days
        slack_template: slack_default
        to:
          - contact
          - redacted
        transport:
          queue: '{notify_queue_arn}'
          type: sqs
        type: notify
        violation_desc: Set retention period to 35 days for any Aurora clusters that are
          set to less than 35 days
    description: Set retention period to 35 days for any Aurora clusters that are set
      to less than 35 days
    filters:
      - and:
          - key: BackupRetentionPeriod
            op: lt
            type: value
            value: 35
          - key: Engine
            op: contains
            type: value
            value: aurora
    mode:
      execution-options:
        log_group: /cloud-custodian/
      role: arn:aws:iam::{account_id}:role/pet/pet-cloud-custodian-lambda-popular-cicada
      schedule: cron(* */4 * * ? *)
      type: periodic
    name: aurora-max-retention
    resource: rds-cluster
    tags:
      - nonprod
```
Having this issue too. But somehow it doesn't happen on all lambdas. I have one policy that deletes unused SG's, works fine. Another policy should terminate untagged ec2's which fails with above issue.

Version 0.8.45.1

Invoking command is very simple:

```bash
custodian run --cache-period=0 --policies a -policies b -policies c
```

I'm not really having a clue how the changes from #4809 can be used. But I wonder, if [`self.ctx.options.log_group`](https://github.com/cloud-custodian/cloud-custodian/pull/4809/files#diff-c445195d6cd47ce9428dd65c00b65fdfL234) isn't missing somewhere? I mean, if I do not use the this _netloc_ thing, I think it should use the previous log_group? But then again, some lambdas do work, so it probably isn't that. 

I also tried to change line [237](https://github.com/cloud-custodian/cloud-custodian/pull/4809/files#diff-c445195d6cd47ce9428dd65c00b65fdfR237) to 

```py
self.log_group = self.config.get('netloc', self.ctx.options.log_group)
```

But having the same result...


I'm able to reproduce if I manually hack the uploaded function, values for log_group of null and "" seem to work fine, where as a value of "/" reproduces (we strip the leading "/" as its not allowed as a log group prefix).
Just checked, on the failing policy I got this defined:

```
      execution-options:
        log_group: /aws/lambda/custodian-auto-terminator
```

Indeed, the working policies have no `log_group` defined. 
^^, seeing the same, rolling back to 8.44 all is good again, and validation errors seem to be better (e.g. invalid characters in log_group throws an descriptive boto3 error vs. 8.45 has a crypic schema invalid error).
confirmed via bisect that the issue was introduced in #4809 

Resolving patch:
diff --git a/c7n/log.py b/c7n/log.py
--- a/c7n/log.py
+++ b/c7n/log.py
@@ -82,7 +82,7 @@ def __init__(self, log_group=__name__, log_stream=None,
         # Logging module internally is tracking all handlers, for final
         # cleanup atexit, custodian is a bit more explicitly scoping shutdown to
         # each policy, so use a sentinel value to avoid deadlocks.
-        self.shutdown = False
+        self.shutdown = True
         retry = get_retry(('ThrottlingException',))
         try:
             client = self.session_factory().client('logs')
@@ -113,6 +113,7 @@ def emit(self, message):
 
         msg = self.format_message(message)
         if not self.transport:
+            self.shutdown = False
             self.start_transports()
         self.buf.append(msg)
         self.flush_buffers(
diff --git a/c7n/resources/aws.py b/c7n/resources/aws.py
--- a/c7n/resources/aws.py
+++ b/c7n/resources/aws.py
@@ -231,11 +231,13 @@ class CloudWatchLogOutput(LogOutput):
 
     def __init__(self, ctx, config=None):
         super(CloudWatchLogOutput, self).__init__(ctx, config)
-        if self.config.get('netloc') == 'master':
-            self.log_group = self.config.get('path').strip("/")
+        if self.config['netloc'] == 'master' or not self.config['netloc']:
+            self.log_group = self.config['path'].strip('/')
         else:
-            self.log_group = self.config.get('netloc')
-        self.region = self.config.get('region')
+            # join netloc to path for casual usages of aws://log/group/name
+            self.log_group = ("%s/%s" % (
+                self.config['netloc'], self.config['path'].strip('/'))).strip('/')
+        self.region = self.config.get('region', ctx.options.region)
         self.destination = (
             self.config.scheme == 'aws' and
             self.config.get('netloc') == 'master') and 'master' or None
@@ -251,27 +253,18 @@ def construct_stream_name(self):
             log_stream = self.config.get('stream').format(
                 region=self.ctx.options.region,
                 account=self.ctx.options.account_id,
-                policy=self.ctx.policy.name
-            )
+                policy=self.ctx.policy.name,
+                now=datetime.datetime.utcnow())
         return log_stream
 
     def get_handler(self):
         log_stream = self.construct_stream_name()
-        if self.destination == 'master':
-            handler = CloudWatchLogHandler(
-                log_group=self.log_group,
-                log_stream=log_stream,
-                session_factory=lambda x=None: self.ctx.session_factory(
-                    assume=False,
-                    region=self.region
-                )
-            )
-        else:
-            handler = CloudWatchLogHandler(
-                log_group=self.log_group,
-                log_stream=log_stream,
-                session_factory=lambda x=None: self.ctx.session_factory(region=self.region))
-        return handler
+        params = dict(
+            log_group=self.log_group, log_stream=log_stream,
+            session_factory=(
+                lambda x=None: self.ctx.session_factory(
+                    region=self.region, assume=self.destination != 'master')))
+        return CloudWatchLogHandler(**params)
 
     def __repr__(self):
         return "<%s to group:%s stream:%s>" % (


Test patch:
diff --git a/tests/test_aws.py b/tests/test_aws.py
--- a/tests/test_aws.py
+++ b/tests/test_aws.py
@@ -198,37 +198,52 @@ def test_metrics(self):
 
 
 class OutputLogsTest(BaseTest):
+    # cloud watch logging
 
-    def test_log_handler(self):
-        session_factory = self.replay_flight_data(
-            'test_log_handler')
-        conf = Bag({
-            'region': 'us-east-2',
-            'scheme': 'aws',
-            'netloc': 'master',
-            'path': 'custodian'
-        })
+    def test_default_log_group(self):
+        ctx = Bag(session_factory=None,
+                  options=Bag(account_id='001100', region='us-east-1'),
+                  policy=Bag(name='test', resource_type='ec2'))
+
+        log_output = output.log_outputs.select('custodian/xyz', ctx)
+        self.assertEqual(log_output.log_group, 'custodian/xyz')
+        self.assertEqual(log_output.construct_stream_name(), 'test')
+
+        log_output = output.log_outputs.select('/custodian/xyz/', ctx)
+        self.assertEqual(log_output.log_group, 'custodian/xyz')
+
+        log_output = output.log_outputs.select('aws://somewhere/out/there', ctx)
+        self.assertEqual(log_output.log_group, 'somewhere/out/there')
+
+        log_output = output.log_outputs.select('aws:///somewhere/out', ctx)
+        self.assertEqual(log_output.log_group, 'somewhere/out')
+
+        log_output = output.log_outputs.select('aws://somewhere', ctx)
+        self.assertEqual(log_output.log_group, 'somewhere')
+
+        log_output = output.log_outputs.select(
+            "aws:///somewhere/out?stream={region}/{policy}", ctx)
+        self.assertEqual(log_output.log_group, 'somewhere/out')
+        self.assertEqual(log_output.construct_stream_name(), 'us-east-1/test')
+
+    def test_master_log_handler(self):
+        session_factory = self.replay_flight_data('test_log_handler')
         ctx = Bag(session_factory=session_factory,
-            options=Bag(account_id='001100', region='us-east-1'),
-            policy=Bag(name='test', resource_type='ec2'))
-        output = aws.CloudWatchLogOutput(ctx, conf)
-        stream = output.get_handler()
+                  options=Bag(account_id='001100', region='us-east-1'),
+                  policy=Bag(name='test', resource_type='ec2'))
+        log_output = output.log_outputs.select(
+            'aws://master/custodian?region=us-east-2', ctx)
+        stream = log_output.get_handler()
         self.assertTrue(stream.log_group == 'custodian')
         self.assertTrue(stream.log_stream == '001100/us-east-1/test')
 
     def test_stream_override(self):
         session_factory = self.replay_flight_data(
             'test_log_stream_override')
-        conf = Bag({
-            'region': 'us-east-2',
-            'scheme': 'aws',
-            'netloc': 'master',
-            'path': 'custodian',
-            'stream': "testing"
-        })
         ctx = Bag(session_factory=session_factory,
             options=Bag(account_id='001100', region='us-east-1'),
             policy=Bag(name='test', resource_type='ec2'))
-        output = aws.CloudWatchLogOutput(ctx, conf)
-        stream = output.get_handler()
+        log_output = output.log_outputs.select(
+            'aws://master/custodian?region=us-east-2&stream=testing', ctx)
+        stream = log_output.get_handler()
         self.assertTrue(stream.log_stream == 'testing')