Repository: aws-cloudformation/cfn-lint

Issue:
Custom resource types can't be validated with --override-spec
*cfn-lint version:* 0.8.3

*Description of issue:* When using custom resource types and supplying an override spec for the custom type, no property validations seem to be performed.

Example: specify a `Custom::Example` type with a required `ID` property and an optional Boolean `Enabled` property:

test.cf.yaml:
```yaml
Resources:
  GoodExample:
    Type: Custom::Example
    Properties:
      ServiceToken: example.arn
      ID: good
      Enabled: true

  BadExample:
    Type: Custom::Example
    Properties:
      ServiceToken: example.arn
      # Missing required prop `ID`
      Enabled: [invalid, 'must be a boolean']
      Wrong: 'this prop is not in the spec'
```

CustomExampleSpec.json:
```json
{
  "ResourceTypes": {
    "Custom::Example": {
      "Properties": {
        "ServiceToken": {"PrimitiveType": "String", "Required": true, "UpdateType": "Immutable"},
        "ID": {"PrimitiveType": "String", "Required": true, "UpdateType": "Immutable"},
        "Enabled": {"PrimitiveType": "Boolean", "Required": false, "UpdateType": "Mutable"}
      }
    }
  },
  "ResourceSpecificationVersion": "2.11.0"
}
```

Run: `cfn-lint --override-spec CustomExampleSpec.json test.cf.yaml`

Results: No lint errors or warnings

Expected:
```
E3003 Property ID missing at Resources/BadExample/Properties
test.cf.yaml:11:5

E3002 Property should be of type Boolean not List at Resources/BadExample/Properties/Enabled
test.cf.yaml:14:7

E3002 Invalid Property Resources/BadExample/Properties/Wrong
test.cf.yaml:15:7
```

This is occurring because #295 unconditionally forces any type starting with `Custom::` to `AWS::CloudFormation::CustomResource`, which deliberately allows all properties. For unknown custom types, that's incredibly useful. But if you have a known spec for the custom type, it would be good to use that instead.

I think this can be accomplished by changing `if resourcetype.startswith('Custom::')` to add `... and not resourcetype in self.resourcetypes` in both [Properties.py](https://github.com/awslabs/cfn-python-lint/blob/v0.8.3/src/cfnlint/rules/resources/properties/Properties.py#L270) and [Required.py](https://github.com/awslabs/cfn-python-lint/blob/v0.8.3/src/cfnlint/rules/resources/properties/Required.py#L132).



Discussion:
Oooh. I don’t think it ever occurred to us that people could add specs for custom resources, but that makes perfect sense.

Any chance you’d want to take a stab at a PR? If you’re uncomfortable testing against your custom spec, we could certainly gin up a fake one.

Resolving patch:
diff --git a/src/cfnlint/rules/resources/properties/Properties.py b/src/cfnlint/rules/resources/properties/Properties.py
--- a/src/cfnlint/rules/resources/properties/Properties.py
+++ b/src/cfnlint/rules/resources/properties/Properties.py
@@ -267,7 +267,7 @@ def match(self, cfn):
         for resourcename, resourcevalue in cfn.get_resources().items():
             if 'Properties' in resourcevalue and 'Type' in resourcevalue:
                 resourcetype = resourcevalue.get('Type', None)
-                if resourcetype.startswith('Custom::'):
+                if resourcetype.startswith('Custom::') and resourcetype not in self.resourcetypes:
                     resourcetype = 'AWS::CloudFormation::CustomResource'
                 if resourcetype in self.resourcetypes:
                     path = ['Resources', resourcename, 'Properties']
diff --git a/src/cfnlint/rules/resources/properties/Required.py b/src/cfnlint/rules/resources/properties/Required.py
--- a/src/cfnlint/rules/resources/properties/Required.py
+++ b/src/cfnlint/rules/resources/properties/Required.py
@@ -129,7 +129,7 @@ def match(self, cfn):
         for resourcename, resourcevalue in cfn.get_resources().items():
             if 'Properties' in resourcevalue and 'Type' in resourcevalue:
                 resourcetype = resourcevalue['Type']
-                if resourcetype.startswith('Custom::'):
+                if resourcetype.startswith('Custom::') and resourcetype not in self.resourcetypes:
                     resourcetype = 'AWS::CloudFormation::CustomResource'
                 if resourcetype in self.resourcetypes:
                     tree = ['Resources', resourcename, 'Properties']


Test patch:
diff --git a/test/fixtures/templates/bad/properties_required.yaml b/test/fixtures/templates/bad/properties_required.yaml
--- a/test/fixtures/templates/bad/properties_required.yaml
+++ b/test/fixtures/templates/bad/properties_required.yaml
@@ -733,3 +733,8 @@ Resources:
     Properties:
       # ServiceToken: arn
       StackName: StackName
+  CustomResource3:
+    Type: 'Custom::SpecifiedCustomResource'
+    Properties:
+      ServiceToken: arn
+      # RequiredString: present
diff --git a/test/fixtures/templates/bad/resources/properties/custom.yaml b/test/fixtures/templates/bad/resources/properties/custom.yaml
new file mode 100644
--- /dev/null
+++ b/test/fixtures/templates/bad/resources/properties/custom.yaml
@@ -0,0 +1,8 @@
+Resources:
+  CustomResource4:
+    Type: 'Custom::SpecifiedCustomResource'
+    Properties:
+      ServiceToken: arn
+      # RequiredString: present
+      OptionalBoolean: ['not-a-boolean']
+      NotAProperty: this-is-not-in-the-custom-spec
diff --git a/test/fixtures/templates/good/resource_properties.yaml b/test/fixtures/templates/good/resource_properties.yaml
--- a/test/fixtures/templates/good/resource_properties.yaml
+++ b/test/fixtures/templates/good/resource_properties.yaml
@@ -604,6 +604,12 @@ Resources:
     Properties:
       ServiceToken: arn
       StackName: StackName
+  CustomResource3:
+    Type: 'Custom::SpecifiedCustomResource'
+    Properties:
+      ServiceToken: arn
+      RequiredString: present
+      OptionalBoolean: true
   Distribution:
     Type: "AWS::CloudFront::Distribution"
     Properties:
diff --git a/test/fixtures/templates/good/resources/properties/custom.yaml b/test/fixtures/templates/good/resources/properties/custom.yaml
new file mode 100644
--- /dev/null
+++ b/test/fixtures/templates/good/resources/properties/custom.yaml
@@ -0,0 +1,7 @@
+Resources:
+  CustomResource4:
+    Type: 'Custom::SpecifiedCustomResource'
+    Properties:
+      ServiceToken: arn
+      RequiredString: present
+      OptionalBoolean: true
diff --git a/test/fixtures/templates/override_spec/custom.json b/test/fixtures/templates/override_spec/custom.json
new file mode 100644
--- /dev/null
+++ b/test/fixtures/templates/override_spec/custom.json
@@ -0,0 +1,24 @@
+{
+  "ResourceTypes": {
+    "Custom::SpecifiedCustomResource": {
+      "Properties": {
+        "ServiceToken": {
+          "PrimitiveType": "String",
+          "Required": true,
+          "UpdateType": "Immutable"
+        },
+        "RequiredString": {
+          "PrimitiveType": "String",
+          "Required": true,
+          "UpdateType": "Immutable"
+        },
+        "OptionalBoolean": {
+          "PrimitiveType": "Boolean",
+          "Required": false,
+          "UpdateType": "Mutable"
+        }
+      }
+    }
+  },
+  "ResourceSpecificationVersion": "2.11.0"
+}
diff --git a/test/rules/resources/properties/test_properties.py b/test/rules/resources/properties/test_properties.py
--- a/test/rules/resources/properties/test_properties.py
+++ b/test/rules/resources/properties/test_properties.py
@@ -14,6 +14,8 @@
   OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
   SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 """
+import cfnlint.helpers
+import json
 from cfnlint.rules.resources.properties.Properties import Properties  # pylint: disable=E0401
 from ... import BaseRuleTestCase
 
@@ -44,3 +46,22 @@ def test_file_negative_2(self):
     def test_file_negative_3(self):
         """Failure test"""
         self.helper_file_negative('fixtures/templates/bad/resource_properties.yaml', 4)
+
+
+class TestSpecifiedCustomResourceProperties(TestResourceProperties):
+    """Repeat Resource Properties tests with Custom Resource override spec provided"""
+    def setUp(self):
+        """Setup"""
+        super(TestSpecifiedCustomResourceProperties, self).setUp()
+        # Add a Spec override that specifies the Custom::SpecifiedCustomResource type
+        with open('fixtures/templates/override_spec/custom.json') as fp:
+            custom_spec = json.load(fp)
+        cfnlint.helpers.set_specs(custom_spec)
+        # Reset Spec override after test
+        self.addCleanup(cfnlint.helpers.initialize_specs)
+
+    # ... all TestResourceProperties test cases are re-run with override spec ...
+
+    def test_file_negative_custom(self):
+        """Additional failure test for specified Custom Resource validation"""
+        self.helper_file_negative('fixtures/templates/bad/resources/properties/custom.yaml', 2)
diff --git a/test/rules/resources/properties/test_required.py b/test/rules/resources/properties/test_required.py
--- a/test/rules/resources/properties/test_required.py
+++ b/test/rules/resources/properties/test_required.py
@@ -14,6 +14,8 @@
   OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
   SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 """
+import cfnlint.helpers
+import json
 from cfnlint.rules.resources.properties.Required import Required  # pylint: disable=E0401
 from ... import BaseRuleTestCase
 
@@ -36,3 +38,23 @@ def test_file_negative(self):
     def test_file_negative_generic(self):
         """Generic Test failure"""
         self.helper_file_negative('fixtures/templates/bad/generic.yaml', 7)
+
+
+class TestSpecifiedCustomResourceRequiredProperties(TestResourceConfiguration):
+    """Test Custom Resource Required Properties when override spec is provided"""
+    def setUp(self):
+        """Setup"""
+        super(TestSpecifiedCustomResourceRequiredProperties, self).setUp()
+        # Add a Spec override that specifies the Custom::SpecifiedCustomResource type
+        with open('fixtures/templates/override_spec/custom.json') as fp:
+            custom_spec = json.load(fp)
+        cfnlint.helpers.set_specs(custom_spec)
+        # Reset Spec override after test
+        self.addCleanup(cfnlint.helpers.initialize_specs)
+
+    # ... all TestResourceConfiguration test cases are re-run with override spec ...
+
+    def test_file_negative(self):
+        """Test failure"""
+        # Additional Custom::SpecifiedCustomResource failure detected with custom spec
+        self.helper_file_negative('fixtures/templates/bad/properties_required.yaml', 13)