Repository: cloud-custodian/cloud-custodian

Issue:
c7n-org + mugc.py multi-account resource destruction issues
**Describe the bug**
I'm using c7n-org + mugc.py to destroy resources that were deployed using c7n-org. When running the following command: 
```AWS_PROFILE='account_A_profile' c7n-org run-script -s out/ -c accounts.json "/path/to/python /path/to/mugc.py -v --present -c policy.json"```
resources are only destroyed in account A, even though the following output is received:

> 2021-03-02 19:15:43,761: c7n_org:INFO ran script on account:account_A region:us-west-1 script: ...
2021-03-02 19:15:43,762: c7n_org:INFO ran script on account:account_B region:us-west-1 script: ...

`stderr` contains no information regarding why this may have failed on `account_B`.

After some digging, the bug appears to be the `assume_role` parameter used by `mugc.py`. The command shown above to run c7n-org run-script is not passing an [assume_role](https://github.com/cloud-custodian/cloud-custodian/blob/dd03cc4ae2241c75de46a05eb6c4714d50bc5583/tools/ops/mugc.py#L165) parameter to `mugc.py`, thus `mugc.py` is running garbage collection on resources in `account_A` from `account_B`, instead of the resources in `account_B`. This was determined by logging the functions discovered [here](https://github.com/cloud-custodian/cloud-custodian/blob/dd03cc4ae2241c75de46a05eb6c4714d50bc5583/tools/ops/mugc.py#L42), and noting that the function was for `account_A` and not `account_B`.

For reference, here are the accounts.json and policy.json files:
**accounts.json**
```
{
  "accounts": [
    {
      "account_id": "12345",
      "name": "account_A",
      "regions": [
        "us-west-1"
      ],
      "role": "arn:aws:iam::12345:role/account_A_admin_role"
    },
    {
      "account_id": "67890",
      "name": "account_B",
      "regions": [
        "us-west-1"
      ],
      "role": "arn:aws:iam::67890:role/account_B_admin_role"
    }
  ]
}
```
**policy.json**
```
{
  "policies": [
    {
      "name": "mandatory_tag_enforcer_s3",
      "mode": {
        "type": "config-rule"
      },
      "resource": "s3",
      "filters": [
        {
          "tag:Owner": "absent"
        }
      ]
    }
  ]
}
```
I attempted to add an `assume_role` field to the `accounts.json` file in the hopes that it would be passed to `mugc.py`, but this doesn't work, no parameter is passed to `mugc.py`. 

I can verify that if I modify my initial command to: 
```AWS_PROFILE='account_A_profile' c7n-org run-script -s out/ -c accounts.json "/path/to/python /path/to/mugc.py --assume arn:aws:iam::67890:role/account_B_admin_role -v --present -c policy.json"``` the correct role is assumed and resources are cleaned as necessary.

**To Reproduce**
Steps to reproduce the behavior:
1. User 2 AWS accounts (referenced as account_A and account_B)
2. Give the user permissions to assume roles in both account_A and account_B
3. Deploy using```AWS_PROFILE='account_A_profile' c7n-org run --verbose -c accounts.json -u policy.json -s out/``` with the accounts and policy files shown above. Where `account_A_profile` links to the credentials of the user created in step 1.
4. Run ```AWS_PROFILE='account_A_profile' c7n-org run-script -s out/ -c accounts.json "/path/to/python /path/to/mugc.py -v --present -c policy.json"```.
5. Note that only the resources deployed in `account_A` are destroyed.


**Expected behavior**
Perhaps I have a misunderstanding, but I had interpreted that my initial command would, for each account listed, assume the role specified and attempt to clean up the chosen policies in their respective accounts, using their respective roles as long as the credentials provided are valid (in my case, AWS_PROFILE='account_A_profile' has a trust relationship with roles in both `account_A` and `account_B`).

If this is not the case, thats OK, but how would I go about destroying resources across all accounts listed in `accounts.json` without having to change the `assume_role` parameter for each account in `mugc.py`?



Discussion:
can you run aws sts get-caller-identity as your script to confirm your theory?  generally c7n-org does the role assume and exports standard sdk environment variables before calling. its possible the aws profile  env var is throwing that off here, afaicr we only set that if thats defined in the accounts config file.
@kapilt the command:
```
AWS_PROFILE='account_A_profile' c7n-org run-script -s testout/ -c accounts.json "aws sts get-caller-identity"
```
yields the following in `stdout`:
```
{
    "UserId": "{userId1}:org-script",
    "Account": "12345",
    "Arn": "arn:aws:sts::12345:assumed-role/account_A_admin_role/org-script"
}
{
    "UserId": "{userId2}:org-script",
    "Account": "67890",
    "Arn": "arn:aws:sts::67890:assumed-role/account_B_admin_role/org-script"
}
```
So c7n-org looks to be acting as the correct role, so the issue would be in `mugc.py`.

Looking over to `mugc.py`, when creating the session factory, [here](https://github.com/cloud-custodian/cloud-custodian/blob/dd03cc4ae2241c75de46a05eb6c4714d50bc5583/tools/ops/mugc.py#L35), the same parameters are passed for both accounts `region=us-west-1`, `assume_role=None`, `profile=account_A_profile`, `external_id=None`, which would start a session against the account associated with `account_A_profile`, correct?
yeah.. there's something odd here depending on environment variables set when executing c7n-org on run-script.
unsetting AWS_DEFAULT_REGION / AWS_REGION in the executing shell sufficed as a work-around fwiw when using a AWS_PROFILE.. else the region seemed to be coming from the shell env when using profile for auth.
@kapilt Thanks for the reply. Are you saying that unsetting those variables fixed the `mugc.py` resource destruction issue, or just a region setting issue? I tested the workaround and still ran into the same issue as before (`account B`'s resources aren't removed).

For the time being, it seems like manually specifying which role `mugc.py` should assume (`--assume my-role-arn`) is the main workaround? Is there another known way to provide credentials that would perhaps circumvent the environment variable issue?
for the --assume-role paramter to mugc you would need to quote the command, but i'm not clear why you would be passing assume role to it, since c7n-org already does the role assume based on what's in the account config file for you (unless your saying your config file is profile based).
i thought there was an issue with c7n-org, but i'm not able to reproduce. i was trying out this script via run-script and it produces the correct result, but i could have sworn i saw an issue with profile and region env vars set at the same time. it does feel like we should drop profile from the env variables in this context though.
```
import os

x = os.environ['AWS_DEFAULT_REGION']
y = os.environ['AWS_PROFILE']

import boto3

z = boto3.client('ec2').meta.region_name
n = boto3.client('sts').get_caller_identity()

print(f"region: {x} profile: {y} client-region: {z} account: {n['Account']}")

```

Resolving patch:
diff --git a/tools/c7n_org/c7n_org/cli.py b/tools/c7n_org/c7n_org/cli.py
--- a/tools/c7n_org/c7n_org/cli.py
+++ b/tools/c7n_org/c7n_org/cli.py
@@ -410,15 +410,18 @@ def report(config, output, use, output_dir, accounts,
     writer.writerows(rows)
 
 
-def _get_env_creds(account, session, region):
-    env = {}
+def _get_env_creds(account, session, region, env=None):
+    env = env or {}
     if account["provider"] == 'aws':
         creds = session._session.get_credentials()
         env['AWS_ACCESS_KEY_ID'] = creds.access_key
         env['AWS_SECRET_ACCESS_KEY'] = creds.secret_key
         env['AWS_SESSION_TOKEN'] = creds.token
         env['AWS_DEFAULT_REGION'] = region
+        env['AWS_REGION'] = region
         env['AWS_ACCOUNT_ID'] = account["account_id"]
+        # we're explicitly setting credential and region configuratio
+        env.pop('AWS_PROFILE', None)
     elif account["provider"] == 'azure':
         env['AZURE_SUBSCRIPTION_ID'] = account["account_id"]
     elif account["provider"] == 'gcp':
@@ -434,9 +437,7 @@ def run_account_script(account, region, output_dir, debug, script_args):
     except ClientError:
         return 1
 
-    env = os.environ.copy()
-    env.update(_get_env_creds(account, session, region))
-
+    env = _get_env_creds(account, session, region, dict(os.environ))
     log.info("running script on account:%s region:%s script: `%s`",
              account['name'], region, " ".join(script_args))
 


Test patch: