Repository: cloud-custodian/cloud-custodian

Issue:
c7n-org azure - issue with duplicate subscription names
**Describe the bug**
As mentioned below, we have multiple subscriptions with same subscription name and it's causing issue when we run policies with c7n-org command at a time for these subscriptions. It's creating folder with same name for multiple subscriptions and keep overwriting until last instance of same subscription.

```
- name: MS Azure Sub
  subscription_id: bb2cf9ee-bc49-4e35-8ac6-3cf606d65653e

- name: MS Azure Sub
  subscription_id: 3cf606d6-e6d9-4a70-b8d2-134d5368be1c4

- name: MS Azure Sub
  subscription_id: 4a70f9ee-8ac6-3cf6-bc49-r53242343445e
```


**To Reproduce**
Create subscriptions in azure portal with same name under same tenant and generate YML file using azuresubs.py. Then run a simple policy against those subscriptions using c7n-org command

`c7n-org run -c subscriptions.yml -s /tmp/ -u sample_policy.yml`


**Expected behavior**
Expected to create folders with subscription id instead of subscription name


**Background (please complete the following information):**
 - OS: Ubuntu
 - Python Version:  3.6.9
 - Custodian Version: 0.9.10
 - Tool Version: NA
 - Cloud Provider: Azure
 - Policy: NA
 - Traceback: [if applicable, please exclude sensitive/account information]
 - `custodian version --debug` output

Custodian: 0.9.10
Python: 3.6.9 (default, Oct 8 2020, 12:12:24)
[GCC 8.4.0]
Platform: posix.uname_result(sysname='Linux', nodename='demo-vm02', release='5.4.0-1039-azure', version='#41~18.04.1-Ubuntu SMP Mon Jan 18 14:00:01 UTC 2021', machine='x86_64')
Using venv: True
Docker: False
Installed:
PyJWT==1.7.1
adal==1.2.5
applicationinsights==0.11.9
apscheduler==3.6.3
argcomplete==1.12.2
attrs==20.3.0
azure-cli-core==2.16.0
azure-cli-telemetry==1.0.6
azure-common==1.1.25
azure-core==1.8.2
azure-cosmos==3.2.0
azure-cosmosdb-nspkg==2.0.2
azure-cosmosdb-table==1.0.6
azure-functions==1.4.0
azure-graphrbac==0.61.1
azure-keyvault==1.1.0
azure-mgmt-apimanagement==0.1.0
azure-mgmt-applicationinsights==0.2.0
azure-mgmt-authorization==0.60.0
azure-mgmt-batch==7.0.0
azure-mgmt-cdn==4.0.0
azure-mgmt-cognitiveservices==5.0.0
azure-mgmt-compute==10.0.0
azure-mgmt-containerinstance==1.5.0
azure-mgmt-containerregistry==2.8.0
azure-mgmt-containerservice==8.3.0
azure-mgmt-core==1.2.1
azure-mgmt-cosmosdb==0.11.0
azure-mgmt-costmanagement==0.1.0
azure-mgmt-databricks==0.1.0
azure-mgmt-datafactory==0.8.0
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-dns==3.0.0
azure-mgmt-eventgrid==2.2.0
azure-mgmt-eventhub==3.1.0
azure-mgmt-hdinsight==1.7.0
azure-mgmt-iothub==0.10.0
azure-mgmt-keyvault==1.1.0
azure-mgmt-logic==3.0.0
azure-mgmt-managementgroups==0.2.0
azure-mgmt-monitor==0.7.0
azure-mgmt-msi==1.0.0
azure-mgmt-network==9.0.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-policyinsights==0.4.0
azure-mgmt-rdbms==1.9.0
azure-mgmt-redis==6.0.0
azure-mgmt-resource==6.0.0
azure-mgmt-resourcegraph==2.0.0
azure-mgmt-search==2.1.0
azure-mgmt-sql==0.16.0
azure-mgmt-storage==7.2.0
azure-mgmt-subscription==0.5.0
azure-mgmt-web==0.44.0
azure-nspkg==3.0.2
azure-storage-blob==2.1.0
azure-storage-common==2.1.0
azure-storage-file==2.1.0
azure-storage-queue==2.1.0
bcrypt==3.2.0
boto3==1.16.59
botocore==1.19.59
c7n==0.9.10
certifi==2020.6.20
cffi==1.14.3
chardet==3.0.4
click==7.1.2
colorama==0.4.4
cryptography==2.9.2
distlib==0.3.1
humanfriendly==8.2
idna==2.10
importlib-metadata==3.4.0
importlib-resources==3.3.0
isodate==0.6.0
jmespath==0.10.0
jsonpickle==1.3
jsonschema==3.2.0
knack==0.7.2
msal==1.0.0
msal-extensions==0.1.3
msrest==0.6.19
msrestazure==0.6.4
netaddr==0.7.20
oauthlib==3.1.0
paramiko==2.7.2
pkginfo==1.6.1
portalocker==1.7.1
pycparser==2.20
pygments==2.7.2
pynacl==1.4.0
pyopenssl==19.1.0
pyrsistent==0.17.3
python-dateutil==2.8.1
pytz==2019.1
pyyaml==5.4.1
requests==2.24.0
requests-oauthlib==1.3.0
s3transfer==0.3.4
setuptools==52.0.0
six==1.15.0
tabulate==0.8.7
typing-extensions==3.7.4.3
tzlocal==2.1
urllib3==1.26.2
zipp==3.4.0


**Additional context**
Add any other context about the problem here.



Discussion:
modify your accounts.yml to change to unique subscription names.
In our case, we are generating **accounts.yml** dynamically every day to get new subscriptions list. As of now, I added a custom python script to our automation to add a unique suffix to handle duplicate subscription names. But I suggest using **Azure subscription id** over **subscription name** would be best solution to create folders.
Yeah, lets either switch to ID or do `name + [last 4 of ID]`

Resolving patch:
diff --git a/tools/c7n_org/scripts/azuresubs.py b/tools/c7n_org/scripts/azuresubs.py
--- a/tools/c7n_org/scripts/azuresubs.py
+++ b/tools/c7n_org/scripts/azuresubs.py
@@ -6,6 +6,8 @@
 from c7n.utils import yaml_dump
 from azure.mgmt.resource.subscriptions import SubscriptionClient
 
+NAME_TEMPLATE = "{name}"
+
 
 @click.command()
 @click.option(
@@ -16,7 +18,11 @@
         ['Enabled', 'Warned', 'PastDue', 'Disabled', 'Deleted']),
     default=('Enabled',),
     help="File to store the generated config (default stdout)")
-def main(output, state):
+@click.option(
+    '--name',
+    default=NAME_TEMPLATE,
+    help="Name template for subscriptions in the config, defaults to %s" % NAME_TEMPLATE)
+def main(output, state, name):
     """
     Generate a c7n-org subscriptions config file
     """
@@ -31,6 +37,7 @@ def main(output, state):
             'subscription_id': sub['subscriptionId'],
             'name': sub['displayName']
         }
+        sub_info['name'] = name.format(**sub_info)
         results.append(sub_info)
 
     print(yaml_dump({'subscriptions': results}), file=output)


Test patch:
diff --git a/tools/c7n_azure/tests_azure/test_resource_graph_source.py b/tools/c7n_azure/tests_azure/test_resource_graph_source.py
--- a/tools/c7n_azure/tests_azure/test_resource_graph_source.py
+++ b/tools/c7n_azure/tests_azure/test_resource_graph_source.py
@@ -57,7 +57,9 @@ def test_resource_graph_and_arm_sources_storage_are_equivalent(self):
 
         resources_resource_graph = json.loads(json.dumps(p2.run()[0]))
 
-        self.assertTrue(resource_cmp(resources_arm, resources_resource_graph))
+        self.assertTrue(resource_cmp(resources_arm,
+                                     resources_resource_graph,
+                                     ignore_properties=['keyCreationTime']))
 
     @arm_template('vm.json')
     def test_resource_graph_and_arm_sources_vm_are_equivalent(self):
@@ -108,7 +110,8 @@ def resource_cmp(res1, res2, ignore_properties=[]):
     if isinstance(res1, dict):
         for prop in res1:
             if prop not in ignore_properties and \
-                    (prop not in res2 or not resource_cmp(res1[prop], res2[prop])):
+                    (prop not in res2 or not resource_cmp(
+                        res1[prop], res2[prop], ignore_properties)):
                 return False
 
     elif isinstance(res1, list):
@@ -116,7 +119,7 @@ def resource_cmp(res1, res2, ignore_properties=[]):
             return False
 
         for item1, item2 in zip(res1, res2):
-            if not resource_cmp(item1, item2):
+            if not resource_cmp(item1, item2, ignore_properties):
                 return False
 
     elif isinstance(res1, str):