Repository: cloud-custodian/cloud-custodian

Issue:
validation output improvements
I was attempting to validate a large policy file (working around #3695) and found that the default validate output is unhelpful:

```
2019-04-09 10:36:05,815: custodian.commands:ERROR {'tag': 'Warning', 'type': 'mark', 'value': 'Untagged Instance'} is not valid under any of the given schemas

Failed validating u'anyOf' in schema[90][u'allOf'][1][u'properties'][u'actions'][u'items']:
    {u'anyOf': [{u'$ref': u'#/definitions/actions/put-metric'},
                {u'$ref': u'#/definitions/actions/notify'},
                {u'$ref': u'#/definitions/actions/post-finding'},
                {u'$ref': u'#/definitions/actions/invoke-lambda'},
                {u'enum': ['put-metric',
                           u'notify',
                           'post-finding',
                           'invoke-lambda']}]}

On instance[u'actions'][0]:
    {'tag': 'Warning', 'type': 'mark', 'value': 'Untagged Instance'}
```

This would be helpful if it gave something like the resource type, line number in the source file, etc. to help you identify what the problem is short of doing a binary search.


Discussion:
@acdha ! long time.

yeah we do some customizations of the default jsonschema to try and make the output nicer than the default jsonschema  (which is sigificantly worse). generally it ends up going to the most specific dictionary, we can try walking up the policy instance though if we find ourselves on a leaf schema element.

do you have an example policy that triggered this message?
Yeah, it's been quite awhile…

Here's a simple test-case:

```yaml
policies:
    - name: tagging-compliance-ec2
      description: Identify resources which lack our accounting tags
      resource: aws.ec2
      filters:
          - 'tag:Environment': absent
          - 'tag:Service': absent
          - or:
                - 'tag:Owner': absent
                - 'tag:ResponsibleParty': absent
                - 'tag:Contact': absent
                - 'tag:Creator': absent
      actions:
          - type: tag
            key: TagPolicyCompliance
            value: This resource should have tags following policy
    - name: tagging-compliance-waf
      description: Identify resources which lack our accounting tags
      resource: aws.waf
      filters:
          - 'tag:Environment': absent
          - 'tag:Service': absent
          - or:
                - 'tag:Owner': absent
                - 'tag:ResponsibleParty': absent
                - 'tag:Contact': absent
                - 'tag:Creator': absent
      actions:
          - type: tag
            key: TagPolicyCompliance
            value: This resource should have tags following policy
```

```
2019-04-09 13:17:10,102: custodian.commands:ERROR Configuration invalid: test.yaml
2019-04-09 13:17:10,103: custodian.commands:ERROR {'type': 'tag', 'key': 'TagPolicyCompliance', 'value': 'This resource should have tags following policy'} is not valid under any of the given schemas

Failed validating u'anyOf' in schema[88][u'allOf'][1][u'properties'][u'actions'][u'items']:
    {u'anyOf': [{u'$ref': u'#/definitions/actions/put-metric'},
                {u'$ref': u'#/definitions/actions/notify'},
                {u'$ref': u'#/definitions/actions/post-finding'},
                {u'$ref': u'#/definitions/actions/invoke-lambda'},
                {u'enum': ['put-metric',
                           u'notify',
                           'post-finding',
                           'invoke-lambda']}]}

On instance[u'actions'][0]:
    {'key': 'TagPolicyCompliance',
     'type': 'tag',
     'value': 'This resource should have tags following policy'}
2019-04-09 13:17:10,103: custodian.commands:ERROR tagging-compliance
```

I should note that I originally found that list to be longer because I was using the `mark` action which seems to be common to most of the examples but far more resources support `tag`, so at first it looked like things like S3 didn't support tagging since I'd started with the list in https://gist.github.com/jtroberts83/56c1abe81e273f495c6754020409b480 while working on https://gist.github.com/acdha/d0b4d07e581d6c03b8b615346f98d882.
yeah.. mark is a deprecated alias for tag, which we stopped adding as redundant, but we haven't gone back through to remove extant due to compatibility concerns with extant policies, till we have a better deprecation framework in place.

two comments on the tag policy setup, i'd recommend either using yaml anchors and alias to avoid the repetition, or alternatively generating the policy set (ruamel supports anchor/ref writes incidentally), and `custodian schema can` output json
Oh, yes, in my non-toy example it's just a reference — I was happy to see that PyYAML added that automatically:

```yaml
policies:
- actions: &id001
  - key: TagPolicyCompliance
    type: tag
    value: This resource should have tags following policy
  description: Identify resources which lack our accounting tags
  filters: &id002
  - tag:Environment: absent
  - tag:Service: absent
  - or:
    - tag:Owner: absent
    - tag:ResponsibleParty: absent
    - tag:Contact: absent
    - tag:Creator: absent
  name: tagging-compliance-acm-certificate
  resource: aws.acm-certificate
- actions: *id001
  description: Identify resources which lack our accounting tags
  filters: *id002
  name: tagging-compliance-ami
  resource: aws.ami
- actions: *id001
  description: Identify resources which lack our accounting tags
  filters: *id002
  name: tagging-compliance-app-elb
  resource: aws.app-elb
…
```

The main TODO point I had was using the schema definition to output all resource types which support the tag action but I had to switch projects.

Resolving patch:
diff --git a/c7n/schema.py b/c7n/schema.py
--- a/c7n/schema.py
+++ b/c7n/schema.py
@@ -61,7 +61,7 @@ def validate(data, schema=None):
                     ", ".join(dupes))), dupes[0]]
         return []
     try:
-        resp = specific_error(errors[0])
+        resp = policy_error_scope(specific_error(errors[0]), data)
         name = isinstance(
             errors[0].instance,
             dict) and errors[0].instance.get(
@@ -78,6 +78,18 @@ def validate(data, schema=None):
     ]))
 
 
+def policy_error_scope(error, data):
+    """Scope a schema error to its policy name and resource."""
+    err_path = list(error.absolute_path)
+    if err_path[0] != 'policies':
+        return error
+    pdata = data['policies'][err_path[1]]
+    pdata.get('name', 'unknown')
+    error.message = "Error on policy:{} resource:{}\n".format(
+        pdata.get('name', 'unknown'), pdata.get('resource', 'unknown')) + error.message
+    return error
+
+
 def specific_error(error):
     """Try to find the best error for humans to resolve
 


Test patch:
diff --git a/tests/test_schema.py b/tests/test_schema.py
--- a/tests/test_schema.py
+++ b/tests/test_schema.py
@@ -18,7 +18,7 @@
 from jsonschema.exceptions import best_match
 
 from c7n.manager import resources
-from c7n.schema import Validator, validate, generate, specific_error
+from c7n.schema import Validator, validate, generate, specific_error, policy_error_scope
 from .common import BaseTest
 
 
@@ -104,6 +104,28 @@ def test_semantic_mode_error(self):
         self.assertTrue("'scheduled' was unexpected" in str(error))
         self.assertTrue(len(str(error)) < 2000)
 
+    def test_semantic_error_policy_scope(self):
+
+        data = {
+            'policies': [
+                {'actions': [{'key': 'TagPolicyCompliance',
+                              'type': 'tag',
+                              'value': 'This resource should have tags following policy'}],
+                 'description': 'Identify resources which lack our accounting tags',
+                 'filters': [{'tag:Environment': 'absent'},
+                             {'tag:Service': 'absent'},
+                             {'or': [{'tag:Owner': 'absent'},
+                                     {'tag:ResponsibleParty': 'absent'},
+                                     {'tag:Contact': 'absent'},
+                                     {'tag:Creator': 'absent'}]}],
+                 'name': 'tagging-compliance-waf',
+                 'resource': 'aws.waf'}]}
+
+        errors = list(self.validator.iter_errors(data))
+        self.assertEqual(len(errors), 1)
+        error = policy_error_scope(specific_error(errors[0]), data)
+        self.assertTrue("policy:tagging-compliance-waf" in error.message)
+
     def test_semantic_error(self):
         data = {
             "policies": [