Repository: cloud-custodian/cloud-custodian

Issue:
c7n-org run returns zero return code with failed run
# c7n-org run returns zero return code with failed run

When executing a policy run against multiple accounts, failure to apply to accounts with access denied is reported as a zero return code.

## How to replicate this problem
1. pip install `c7n_org` (and if required `pip install --upgrade botocore`)
2. Generate the `accounts.yml` with steps from [here](https://github.com/capitalone/cloud-custodian/tree/master/tools/c7n_org#config-file-generation)
3. Run the command: `c7n-org run -c accounts.yml -s output -u test.yml --dryrun`
4. If you have the wrong policy role, some of the accounts will return `AccessDenied` similar to:
    ```bash
    $ c7n-org run -c ~/accounts.yml -s output -u test.yml --dryrun -t path:/Sandbox --region us-east-1
    2018-07-19 12:18:19,005: c7n_org:INFO Ran account:ate-io-product11 region:us-east-1 policy:s3-enable-versioning matched:1 time:0.01
    2018-07-19 12:18:19,011: c7n_org:INFO Ran account:ate-io-product7 region:us-east-1 policy:s3-enable-versioning matched:1 time:0.01
    2018-07-19 12:18:19,010: c7n_org:INFO Ran account:ate-io-product35 region:us-east-1 policy:s3-enable-versioning matched:2 time:0.00
    2018-07-19 12:18:19,959: c7n_org:WARNING Access denied account:ate-io-product3 region:us-east-1
    2018-07-19 12:18:19,961: c7n_org:WARNING Access denied account:ate-io-product2 region:us-east-1
    2018-07-19 12:18:19,970: c7n_org:WARNING Access denied account:ate-io-product6 region:us-east-1
    2018-07-19 12:18:19,972: c7n_org:WARNING Access denied account:ate-io-product1 region:us-east-1
    2018-07-19 12:18:19,996: c7n_org:WARNING Access denied account:ate-io-product5 region:us-east-1
    2018-07-19 12:18:20,016: c7n_org:WARNING Access denied account:ate-io-product12 region:us-east-1
    2018-07-19 12:18:20,865: c7n_org:WARNING Access denied account:ate-io-product4 region:us-east-1
    2018-07-19 12:18:20,938: c7n_org:WARNING Access denied account:ate-io-product8 region:us-east-1
    2018-07-19 12:18:21,239: c7n_org:WARNING Access denied account:ate-io-product10 region:us-east-1
    2018-07-19 12:18:25,407: c7n_org:WARNING Access denied account:ate-io-product9 region:us-east-1
    2018-07-19 12:18:25,432: c7n_org:INFO Policy resource counts Counter({'s3-enable-versioning': 4})
    $
    ```
5. Run the command: `echo $?` and it will return `0`
    ```bash
    $ echo $?
    $ 0
    ```

## Problem statement

The c7n-org command should return a non-zero return code when one of its child policy runs fails.
As it stands _Access denied_ messages are written to `STDERR`, all non-error messages to `STDOUT`, I can potentially read data from `STDERR` in a wrapper as a short-term work around, but it would be more reliable to exit non-zero on error.

Other relevant info:
- c7n-org version `0.3.1`
- Python version `2.7.15`
- OSX version (uname -v) `Darwin Kernel Version 17.6.0: Tue May  8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASE_X86_64`
- Linux version `Linux 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 Linux`

## Proposed solution

Make c7n-org return non-zero if any account policy runs fail.


Discussion:
This also holds true for an expired access token.
In the short term, i'm using a wrapper script similar to:

```bash
RUNAS=${RUNAS:-default}
ACCOUNTS=${ACCOUNTS:-${HOME}/accounts.yml}
POLICY=${POLICY:-${HOME}/enable-s3-versioning.yml}
BASE=$PWD;
OUTPUT=${OUTPUT:-$BASE/output}

echo -n "running...";
aws-runas ${RUNAS} c7n-org run -c ${ACCOUNTS} \
    -s ${OUTPUT} \
    -u ${POLICY} \
    -t path:/Sandbox \
    --region us-east-1 \
    --dryrun \
    2> ${BASE}/STDERR 1> ${BASE}/STDOUT;
echo "done";

LINES=$(wc -l ${BASE}/STDERR | awk '{print$1}')
echo "${BASE}/STDERR: ${LINES} line(s)"
if [ ${LINES} -gt 0 ]; then
  ERRORS=$(grep -cE 'ERROR|WARNING' ${BASE}/STDERR)
  if [ ${ERRORS} -gt 0 ]; then
    echo "failed to run, found ${ERRORS} error(s)";
    cat ${BASE}/STDERR ${BASE}/STDOUT;
    exit 1;
  else
    echo "good run";
    cat ${BASE}/STDERR ${BASE}/STDOUT;
    exit 0;
  fi
fi
```

Resolving patch:
diff --git a/tools/c7n_org/c7n_org/cli.py b/tools/c7n_org/c7n_org/cli.py
--- a/tools/c7n_org/c7n_org/cli.py
+++ b/tools/c7n_org/c7n_org/cli.py
@@ -465,7 +465,9 @@ def run_account(account, region, policies_config, output_path,
 
     policies = PolicyCollection.from_data(policies_config, config)
     policy_counts = {}
+    success = True
     st = time.time()
+
     with environ(**account_tags(account)):
         for p in policies:
 
@@ -486,15 +488,17 @@ def run_account(account, region, policies_config, output_path,
                     account['name'], region, p.name, len(resources),
                     time.time() - st)
             except ClientError as e:
+                success = False
                 if e.response['Error']['Code'] == 'AccessDenied':
                     log.warning('Access denied account:%s region:%s',
                                 account['name'], region)
-                    return policy_counts
+                    return policy_counts, success
                 log.error(
                     "Exception running policy:%s account:%s region:%s error:%s",
                     p.name, account['name'], region, e)
                 continue
             except Exception as e:
+                success = False
                 log.error(
                     "Exception running policy:%s account:%s region:%s error:%s",
                     p.name, account['name'], region, e)
@@ -505,7 +509,7 @@ def run_account(account, region, policies_config, output_path,
                 pdb.post_mortem(sys.exc_info()[-1])
                 raise
 
-    return policy_counts
+    return policy_counts, success
 
 
 @cli.command(name='run')
@@ -529,6 +533,7 @@ def run(config, use, output_dir, accounts, tags,
     accounts_config, custodian_config, executor = init(
         config, use, debug, verbose, accounts, tags, policy, policy_tags=policy_tags)
     policy_counts = Counter()
+    success = True
     with executor(max_workers=WORKER_COUNT) as w:
         futures = {}
         for a in accounts_config['accounts']:
@@ -552,7 +557,14 @@ def run(config, use, output_dir, accounts, tags,
                     "Error running policy in %s @ %s exception: %s",
                     a['name'], r, f.exception())
 
-            for p, count in f.result().items():
+            account_region_pcounts, account_region_success = f.result()
+            for p, count in account_region_pcounts:
                 policy_counts[p] += count
 
+            if not account_region_success:
+                success = False
+
     log.info("Policy resource counts %s" % policy_counts)
+
+    if not success:
+        sys.exit(1)


Test patch: