Repository: cloud-custodian/cloud-custodian

Issue:
logs not being generated from c7n-org 
I have the following policy saved as s3.yaml:

```
policies:
  - name: cloud-custodian-s3-encryption-policy
    # ensure s3 buckets are encrypted
    resource: s3
    actions:
      - set-bucket-encryption

  - name: cloud-custodian-s3-delete-global-grants-policy
    # global grants are not allowed
    resource: s3
    filters:
      - type: global-grants
    actions:
      - delete-global-grants
```


Executing with the following command:
`c7n-org run -c accounts.yml -s s3://my-bucketname -u global/s3.yaml --region all`

The log file (custodian-run.log.gz) that are generated are empty.
[custodian-run.log.gz](https://github.com/capitalone/cloud-custodian/files/2233536/custodian-run.log.gz)




Discussion:
so does c7n-org itself post to stdout? can you post a sanitized version of those logs?
Here's an example of what c7n-org is generating for stdout

```
2018-07-26 18:30:37,352: c7n_org:INFO Ran account:my-account region:ca-central-1 policy:cloud-custodian-s3-encryption-policy matched:28 time:64.76
2018-07-26 18:30:38,145: c7n_org:INFO Ran account:my-account region:ap-northeast-2 policy:cloud-custodian-s3-encryption-policy matched:28 time:65.59
2018-07-26 18:30:38,504: c7n_org:INFO Ran account:my-account region:eu-west-1 policy:cloud-custodian-s3-encryption-policy matched:28 time:65.90
2018-07-26 18:30:40,218: c7n_org:INFO Ran account:my-account region:eu-central-1 policy:cloud-custodian-s3-encryption-policy matched:28 time:67.63
2018-07-26 18:30:40,331: c7n_org:INFO Ran account:my-account region:ap-northeast-1 policy:cloud-custodian-s3-encryption-policy matched:28 time:67.77
2018-07-26 18:30:43,101: c7n_org:INFO Ran account:my-account region:ap-south-1 policy:cloud-custodian-s3-encryption-policy matched:28 time:70.52
2018-07-26 18:30:43,173: c7n_org:INFO Ran account:my-account region:ap-southeast-1 policy:cloud-custodian-s3-encryption-policy matched:28 time:70.60
2018-07-26 18:30:45,112: c7n_org:INFO Ran account:my-account region:ap-southeast-2 policy:cloud-custodian-s3-encryption-policy matched:29 time:72.53
2018-07-26 18:32:35,273: c7n_org:INFO Ran account:my-account region:eu-west-2 policy:cloud-custodian-s3-encryption-policy matched:28 time:66.85
2018-07-26 18:32:38,797: c7n_org:INFO Ran account:my-account region:eu-west-3 policy:cloud-custodian-s3-encryption-policy matched:28 time:66.03
2018-07-26 18:32:41,973: c7n_org:INFO Ran account:my-account region:us-east-2 policy:cloud-custodian-s3-encryption-policy matched:28 time:61.39
2018-07-26 18:32:43,742: c7n_org:INFO Ran account:my-account region:sa-east-1 policy:cloud-custodian-s3-encryption-policy matched:29 time:66.76
2018-07-26 18:32:44,269: c7n_org:INFO Ran account:my-account region:us-east-1 policy:cloud-custodian-s3-encryption-policy matched:28 time:64.79
2018-07-26 18:32:47,820: c7n_org:INFO Ran account:my-account region:us-west-2 policy:cloud-custodian-s3-encryption-policy matched:28 time:61.97
2018-07-26 18:32:49,186: c7n_org:INFO Ran account:my-account region:us-west-1 policy:cloud-custodian-s3-encryption-policy matched:28 time:63.40
2018-07-26 18:33:34,613: c7n_org:INFO Policy resource counts Counter({'cloud-custodian-s3-encryption-policy': 422, 'cloud-custodian-s3-delete-global-grants-policy': 0})
```

Any updates?  Is there any additional information that is required?
@pwatt thanks for the bug report, i've confirmed that this isn't working on trunk, I'll try to find some time to look into this
Hello - looking for an update on this as well.  

Resolving patch:
diff --git a/c7n/output.py b/c7n/output.py
--- a/c7n/output.py
+++ b/c7n/output.py
@@ -413,6 +413,7 @@ def get_output_path(self, output_url):
     def get_output_vars(self):
         data = {
             'account_id': self.ctx.options.account_id,
+            'region': self.ctx.options.region,
             'policy': self.ctx.policy.name,
             'now': datetime.utcnow(),
             'uuid': str(uuid.uuid4())}
diff --git a/tools/c7n_kube/c7n_kube/entry.py b/tools/c7n_kube/c7n_kube/entry.py
--- a/tools/c7n_kube/c7n_kube/entry.py
+++ b/tools/c7n_kube/c7n_kube/entry.py
@@ -21,4 +21,5 @@
 
 
 def initialize_kube():
-    log.info('initialize_kube')
+    """kubernetes entry point
+    """
diff --git a/tools/c7n_org/c7n_org/cli.py b/tools/c7n_org/c7n_org/cli.py
--- a/tools/c7n_org/c7n_org/cli.py
+++ b/tools/c7n_org/c7n_org/cli.py
@@ -131,6 +131,27 @@ def cli():
     """custodian organization multi-account runner."""
 
 
+class LogFilter(object):
+    """We want to keep the main c7n-org cli output to be readable.
+
+    We previously did so via squelching custodian's log output via
+    level filter on the logger, however doing that meant that log
+    outputs stored to output locations were also squelched.
+
+    We effectively want differential handling at the top level logger
+    stream handler, ie. we want `custodian` log messages to propagate
+    to the root logger based on level, but we also want them to go the
+    custodian logger's directly attached handlers on debug level.
+    """
+
+    def filter(self, r):
+        if not r.name.startswith('custodian'):
+            return 1
+        elif r.levelno >= logging.WARNING:
+            return 1
+        return 0
+
+
 def init(config, use, debug, verbose, accounts, tags, policies, resource=None, policy_tags=()):
     level = verbose and logging.DEBUG or logging.INFO
     logging.basicConfig(
@@ -138,10 +159,16 @@ def init(config, use, debug, verbose, accounts, tags, policies, resource=None, p
         format="%(asctime)s: %(name)s:%(levelname)s %(message)s")
 
     logging.getLogger('botocore').setLevel(logging.ERROR)
-    logging.getLogger('custodian').setLevel(logging.WARNING)
+    logging.getLogger('s3transfer').setLevel(logging.WARNING)
     logging.getLogger('custodian.s3').setLevel(logging.ERROR)
     logging.getLogger('urllib3').setLevel(logging.WARNING)
 
+    # Filter out custodian log messages on console output if not
+    # at warning level or higher, see LogFilter docs and #2674
+    for h in logging.getLogger().handlers:
+        if isinstance(h, logging.StreamHandler):
+            h.addFilter(LogFilter())
+
     with open(config) as fh:
         accounts_config = yaml.safe_load(fh.read())
         jsonschema.validate(accounts_config, CONFIG_SCHEMA)
@@ -220,9 +247,10 @@ def filter_policies(policies_config, tags, policies, resource, not_policies=None
     policies_config['policies'] = filtered_policies
 
 
-def report_account(account, region, policies_config, output_path, debug):
-    cache_path = os.path.join(output_path, "c7n.cache")
+def report_account(account, region, policies_config, output_path, cache_path, debug):
     output_path = os.path.join(output_path, account['name'], region)
+    cache_path = os.path.join(cache_path, "%s-%s.cache" % (account['name'], region))
+
     config = Config.empty(
         region=region,
         output_dir=output_path,
@@ -273,9 +301,10 @@ def report_account(account, region, policies_config, output_path, debug):
               multiple=True, default=None, help="Policy tag filter")
 @click.option('--format', default='csv', type=click.Choice(['csv', 'json']))
 @click.option('--resource', default=None)
+@click.option('--cache-path', required=False, type=click.Path(), default="~/.cache/c7n-org")
 def report(config, output, use, output_dir, accounts,
            field, no_default_fields, tags, region, debug, verbose,
-           policy, policy_tags, format, resource):
+           policy, policy_tags, format, resource, cache_path):
     """report on a cross account policy execution."""
     accounts_config, custodian_config, executor = init(
         config, use, debug, verbose, accounts, tags, policy,
@@ -299,6 +328,7 @@ def report(config, output, use, output_dir, accounts,
                     a, r,
                     custodian_config,
                     output_dir,
+                    cache_path,
                     debug)] = (a, r)
 
         for f in as_completed(futures):
@@ -440,23 +470,25 @@ def accounts_iterator(config):
 
 
 def run_account(account, region, policies_config, output_path,
-                cache_period, metrics, dryrun, debug):
+                cache_period, cache_path, metrics, dryrun, debug):
     """Execute a set of policies on an account.
     """
     logging.getLogger('custodian.output').setLevel(logging.ERROR + 1)
     CONN_CACHE.session = None
     CONN_CACHE.time = None
-    output_path = os.path.join(output_path, account['name'], region)
-    if not os.path.exists(output_path):
-        os.makedirs(output_path)
 
-    cache_path = os.path.join(output_path, "c7n.cache")
+    # allow users to specify interpolated output paths
+    if '{' not in output_path:
+        output_path = os.path.join(output_path, account['name'], region)
+
+    cache_path = os.path.join(cache_path, "%s-%s.cache" % (account['account_id'], region))
 
     config = Config.empty(
-        region=region,
+        region=region, cache=cache_path,
         cache_period=cache_period, dryrun=dryrun, output_dir=output_path,
         account_id=account['account_id'], metrics_enabled=metrics,
-        cache=cache_path, log_group=None, profile=None, external_id=None)
+        log_group=None, profile=None, external_id=None)
+
     if account.get('role'):
         config['assume_role'] = account['role']
         config['external_id'] = account.get('external_id')
@@ -523,17 +555,29 @@ def run_account(account, region, policies_config, output_path,
 @click.option('-l', '--policytags', 'policy_tags',
               multiple=True, default=None, help="Policy tag filter")
 @click.option('--cache-period', default=15, type=int)
+@click.option('--cache-path', required=False,
+              type=click.Path(
+                  writable=True, readable=True, exists=True,
+                  resolve_path=True, allow_dash=False,
+                  file_okay=False, dir_okay=True),
+              default=None)
 @click.option("--metrics", default=False, is_flag=True)
 @click.option("--dryrun", default=False, is_flag=True)
 @click.option('--debug', default=False, is_flag=True)
 @click.option('-v', '--verbose', default=False, help="Verbose", is_flag=True)
-def run(config, use, output_dir, accounts, tags,
-        region, policy, policy_tags, cache_period, metrics, dryrun, debug, verbose):
+def run(config, use, output_dir, accounts, tags, region,
+        policy, policy_tags, cache_period, cache_path, metrics, dryrun, debug, verbose):
     """run a custodian policy across accounts"""
     accounts_config, custodian_config, executor = init(
         config, use, debug, verbose, accounts, tags, policy, policy_tags=policy_tags)
     policy_counts = Counter()
     success = True
+
+    if not cache_path:
+        cache_path = os.path.expanduser("~/.cache/c7n-org")
+        if not os.path.exists(cache_path):
+            os.makedirs(cache_path)
+
     with executor(max_workers=WORKER_COUNT) as w:
         futures = {}
         for a in accounts_config['accounts']:
@@ -544,6 +588,7 @@ def run(config, use, output_dir, accounts, tags,
                     custodian_config,
                     output_dir,
                     cache_period,
+                    cache_path,
                     metrics,
                     dryrun,
                     debug)] = (a, r)


Test patch:
diff --git a/tools/c7n_org/tests/test_org.py b/tools/c7n_org/tests/test_org.py
new file mode 100644
--- /dev/null
+++ b/tools/c7n_org/tests/test_org.py
@@ -0,0 +1,153 @@
+import copy
+import mock
+import os
+import yaml
+
+from c7n.testing import TestUtils
+from click.testing import CliRunner
+
+from c7n_org import cli as org
+
+
+ACCOUNTS_AWS_DEFAULT = yaml.safe_dump({
+    'accounts': [
+        {'name': 'dev',
+         'account_id': 'foobar',
+         'tags': ['red', 'black'],
+         'role': 'arn:aws:iam:{account_id}::/role/foobar'},
+        {'name': 'qa',
+         'account_id': 'barfoo',
+         'tags': ['red', 'green'],
+         'role': 'arn:aws:iam:{account_id}::/role/foobar'},
+    ],
+}, default_flow_style=False)
+
+POLICIES_AWS_DEFAULT = yaml.safe_dump({
+    'policies': [
+        {'name': 'compute',
+         'resource': 'aws.ec2',
+         'tags': ['red', 'green']},
+        {'name': 'serverless',
+         'resource': 'aws.lambda',
+         'tags': ['red', 'black']},
+
+    ],
+}, default_flow_style=False)
+
+
+class OrgTest(TestUtils):
+
+    def setup_run_dir(self, accounts=None, policies=None):
+        root = self.get_temp_dir()
+
+        if accounts:
+            accounts = yaml.safe_dump(accounts, default_flow_style=False)
+        else:
+            accounts = ACCOUNTS_AWS_DEFAULT
+
+        with open(os.path.join(root, 'accounts.yml'), 'w') as fh:
+            fh.write(accounts)
+
+        if policies:
+            policies = yaml.safe_dump(policies, default_flow_style=False)
+        else:
+            policies = POLICIES_AWS_DEFAULT
+
+        with open(os.path.join(root, 'policies.yml'), 'w') as fh:
+            fh.write(policies)
+
+        cache_path = os.path.join(root, 'cache')
+        os.makedirs(cache_path)
+        return root
+
+    def test_cli_run_aws(self):
+        run_dir = self.setup_run_dir()
+        logger = mock.MagicMock()
+        run_account = mock.MagicMock()
+        run_account.return_value = (
+            {'compute': 24, 'serverless': 12}, True)
+        self.patch(org, 'logging', logger)
+        self.patch(org, 'run_account', run_account)
+        self.change_cwd(run_dir)
+        log_output = self.capture_logging('c7n_org')
+        runner = CliRunner()
+        result = runner.invoke(
+            org.cli,
+            ['run', '-c', 'accounts.yml', '-u', 'policies.yml',
+             '--debug', '-s', 'output', '--cache-path', 'cache'],
+            catch_exceptions=False)
+
+        self.assertEqual(result.exit_code, 0)
+        self.assertEqual(
+            log_output.getvalue().strip(),
+            "Policy resource counts Counter({'compute': 96, 'serverless': 48})")
+
+    def test_filter_policies(self):
+        d = {'policies': [
+            {'name': 'find-ml',
+             'tags': ['bar:xyz', 'red', 'black'],
+             'resource': 'gcp.instance'},
+            {'name': 'find-serverless',
+             'resource': 'aws.lambda',
+             'tags': ['blue', 'red']}]}
+
+        t1 = copy.deepcopy(d)
+        org.filter_policies(t1, [], [], [], [])
+        self.assertEqual(
+            [n['name'] for n in t1['policies']],
+            ['find-ml', 'find-serverless'])
+
+        t2 = copy.deepcopy(d)
+        org.filter_policies(t2, ['blue', 'red'], [], [], [])
+        self.assertEqual(
+            [n['name'] for n in t2['policies']], ['find-serverless'])
+
+        t3 = copy.deepcopy(d)
+        org.filter_policies(t3, [], ['find-ml'], [], [])
+        self.assertEqual(
+            [n['name'] for n in t3['policies']], ['find-ml'])
+
+        t4 = copy.deepcopy(d)
+        org.filter_policies(t4, [], [], 'gcp.instance', [])
+        self.assertEqual(
+            [n['name'] for n in t4['policies']], ['find-ml'])
+
+    def test_resolve_regions(self):
+        self.assertEqual(
+            org.resolve_regions(['us-west-2']),
+            ['us-west-2'])
+        self.assertEqual(
+            org.resolve_regions([]),
+            ('us-east-1', 'us-west-2'))
+
+    def test_filter_accounts(self):
+
+        d = {'accounts': [
+            {'name': 'dev',
+             'tags': ['blue', 'red']},
+            {'name': 'prod',
+             'tags': ['green', 'red']}]}
+
+        t1 = copy.deepcopy(d)
+        org.filter_accounts(t1, [], [], [])
+        self.assertEqual(
+            [a['name'] for a in t1['accounts']],
+            ['dev', 'prod'])
+
+        t2 = copy.deepcopy(d)
+        org.filter_accounts(t2, [], [], ['prod'])
+        self.assertEqual(
+            [a['name'] for a in t2['accounts']],
+            ['dev'])
+
+        t3 = copy.deepcopy(d)
+        org.filter_accounts(t3, [], ['dev'], [])
+        self.assertEqual(
+            [a['name'] for a in t3['accounts']],
+            ['dev'])
+
+        t4 = copy.deepcopy(d)
+        org.filter_accounts(t4, ['red', 'blue'], [], [])
+        self.assertEqual(
+            [a['name'] for a in t4['accounts']],
+            ['dev'])