Repository: aws-cloudformation/cfn-lint

Issue:
Running `cfn-lint file_that_does_not_exist.yml` has exit code 0 and no error message
### CloudFormation Lint Version

cfn-lint 1.19.0

### What operating system are you using?

Ubuntu

### Describe the bug

Running `cfn-lint file_that_does_not_exist.yml` should return exit_code > 0, and should ideally give a stderr message that the file doesn't exist. 

Currently, the program exists the exact same way as if nothing happened.

### Expected behavior

Running `cfn-lint file_that_does_not_exist.yml` should return exit_code > 0, and should ideally give a stderr message that the file doesn't exist. 

### Reproduction template

N/A


Discussion:
I think this is related to #3603. I agree on this.  With an individual file this should fail.  Looking into it.
Since glob returns an empty list if the path doesn't exist or the results are empty we aren't detecting if an individual file is valid.  I think the original issue falls into an issue of caveats with how shells work.  Certain shells (and configurations) send the glob pattern to as an argument if no files are found.  Some shells error before calling cfn-lint (my default shell of zsh does this)

I don't believe there is an easy option for determining if a path is a glob pattern or not (crazy regex patterns may have to be created).  The end result seems to be to bring back the functionality prior to #3603.  Additionally we can add a parameter to not error if there are no files to lint.  At least that customer group would have an option at that point.

This is additionally issue if I mistype the path.  For instance `dne\**\*.yaml` and the question should be is this an error as well?  It can be misleading to not know no files were scanned because I mistyped the pattern.  Which is why I think the default should be to error.

Understandably the original issue was for a valid path glob pattern just not having any matching files.  Which I would also agree shouldn't error but we may have to rely on a new parameter to make that determination since we can't easily determine if the path is valid or not using the glob functionality.

@thecodingsysadmin do you have any thoughts on this?  Would you be okay with a new parameter that would not error if the file list is empty?

Seems the Posix standard (or at least Ubuntu's chosen implementation of it) returns >0 when globs fail.

For example:

* `cat this_file_does_not_exist.txt` -> Return 1.
* `cat this_file*does_not_exist.txt` -> Return 1.

Consider this scenario: Say you're relying on this in a deployment pipeline. The file gets renamed by someone. The check will continue to pass, no matter what's in the file. It seems crazy to think anyone would want this behaviour.
Came to see if there were any open issues relating to this behavior change. In our use case, we can check for the existence of the file before linting. We had relied on a non-zero exit code for lint when a file didn't exist but this is not a huge change if the default functionality is intended to stay this way.

I don't know that a single solution will fit all cases because you could easily justify both an error and a graceful exit with 0 linted results. On one hand in my organization we never call lint in a place where we would expect it to return 0 linted files even with a wildcard, so we would expect a failure. You could easily have an org with the reverse though where they mix cloudformation templates with other code in various folders and only some of them might contain templates that need linting.

One thing PowerShell does that I'm not sure I see in linux commands is that there's a general option for cmdlets called ErrorAction that allows you to configure whether the cmdlet will fail silently or throw an exception if it fails. The linux alternative would be that the command always fails and you would pipe the errors to null or something, I suppose.

Glad the issue already existed, thanks for looking at it. I'll keep an eye on this issue before committing any changes to address this behavior change.
Regarding PowerShell's `ErrorAction`, bash has `set -e` which enables "fail on error". By default, bash scripts "steamroll" (continue on errors, just with a >0 return code). 
Any update on a course of action for this issue? We will likely need to modify our pipelines to check for the existence of the file explicitly regardless, but it would be good to put this to rest.



Patch:
diff --git a/src/cfnlint/config.py b/src/cfnlint/config.py
index 7e81b98203..03846af54d 100644
--- a/src/cfnlint/config.py
+++ b/src/cfnlint/config.py
@@ -796,10 +796,10 @@ def _glob_filenames(self, filenames: Sequence[str]) -> list[str]:
         for filename in filenames:
             add_filenames = glob.glob(filename, recursive=True)
 
-            if isinstance(add_filenames, list):
-                all_filenames.extend(add_filenames)
-            else:
-                LOGGER.error(f"{filename} could not be processed by glob.glob")
+            if not add_filenames and not self.ignore_bad_template:
+                raise ValueError(f"{filename} could not be processed by glob.glob")
+
+            all_filenames.extend(add_filenames)
 
         return sorted(list(map(str, map(Path, all_filenames))))
 
diff --git a/src/cfnlint/data/CfnLintCli/config/schema.json b/src/cfnlint/data/CfnLintCli/config/schema.json
index eb8974c8fb..f72fc6189f 100644
--- a/src/cfnlint/data/CfnLintCli/config/schema.json
+++ b/src/cfnlint/data/CfnLintCli/config/schema.json
@@ -58,6 +58,10 @@
    "description": "custom rule file to use",
    "type": "string"
   },
+  "ignore_bad_template": {
+   "description": "Ignore bad templates",
+   "type": "boolean"
+  },
   "ignore_checks": {
    "description": "List of checks to ignore",
    "items": {
diff --git a/src/cfnlint/rules/errors/__init__.py b/src/cfnlint/rules/errors/__init__.py
index 142af37032..52f3f257cc 100644
--- a/src/cfnlint/rules/errors/__init__.py
+++ b/src/cfnlint/rules/errors/__init__.py
@@ -3,6 +3,7 @@
 SPDX-License-Identifier: MIT-0
 """
 
+from cfnlint.rules.errors.config import ConfigError
 from cfnlint.rules.errors.parse import ParseError
 from cfnlint.rules.errors.rule import RuleError
 from cfnlint.rules.errors.transform import TransformError
diff --git a/src/cfnlint/rules/errors/config.py b/src/cfnlint/rules/errors/config.py
new file mode 100644
index 0000000000..1a7b533152
--- /dev/null
+++ b/src/cfnlint/rules/errors/config.py
@@ -0,0 +1,15 @@
+"""
+Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+SPDX-License-Identifier: MIT-0
+"""
+
+from cfnlint.rules._rule import CloudFormationLintRule
+
+
+class ConfigError(CloudFormationLintRule):
+
+    id = "E0003"
+    shortdesc = "Error with cfn-lint configuration"
+    description = "Error as a result of the cfn-lint configuration"
+    source_url = "https://github.com/aws-cloudformation/cfn-lint"
+    tags = ["base", "rule"]
diff --git a/src/cfnlint/runner.py b/src/cfnlint/runner.py
index e1954650c8..6b8acabd5c 100644
--- a/src/cfnlint/runner.py
+++ b/src/cfnlint/runner.py
@@ -17,7 +17,7 @@
 from cfnlint.decode.decode import decode
 from cfnlint.helpers import REGIONS
 from cfnlint.rules import Match, Rules
-from cfnlint.rules.errors import ParseError, TransformError
+from cfnlint.rules.errors import ConfigError, ParseError, TransformError
 from cfnlint.schema import PROVIDER_SCHEMA_MANAGER
 from cfnlint.template.template import Template
 
@@ -223,10 +223,13 @@ def __init__(self, config: ConfigMixIn) -> None:
                 settings for the template scan.
         """
         self.config = config
-        self.config.templates
         self.formatter = get_formatter(self.config)
         self.rules: Rules = Rules()
         self._get_rules()
+        try:
+            self.config.templates
+        except ValueError as e:
+            self._cli_output([Match(str(e), ConfigError(), None)])
         # load registry schemas before patching
         if self.config.registry_schemas:
             for path in self.config.registry_schemas:


Test patch:
diff --git a/test/unit/module/config/test_config_mixin.py b/test/unit/module/config/test_config_mixin.py
index f6b5abc594..9e584ead07 100644
--- a/test/unit/module/config/test_config_mixin.py
+++ b/test/unit/module/config/test_config_mixin.py
@@ -208,6 +208,20 @@ def test_config_expand_paths_nomatch(self, yaml_mock):
         ]
         config = cfnlint.config.ConfigMixIn([])
 
+        with self.assertRaises(ValueError):
+            self.assertEqual(len(config.templates), 1)
+
+    @patch("cfnlint.config.ConfigFileArgs._read_config", create=True)
+    def test_config_expand_paths_nomatch_ignore_bad_template(self, yaml_mock):
+        """Test precedence in"""
+
+        filename = "test/fixtures/templates/nonexistant/*.yaml"
+        yaml_mock.side_effect = [
+            {"templates": [filename], "ignore_bad_template": True},
+            {},
+        ]
+        config = cfnlint.config.ConfigMixIn([])
+
         # test defaults
         self.assertEqual(config.templates, [])
 
diff --git a/test/unit/module/maintenance/test_update_documentation.py b/test/unit/module/maintenance/test_update_documentation.py
index 6613e85a4f..aecd9ef138 100644
--- a/test/unit/module/maintenance/test_update_documentation.py
+++ b/test/unit/module/maintenance/test_update_documentation.py
@@ -77,6 +77,14 @@ def test_update_docs(self):
                     " [Source](https://github.com/aws-cloudformation/cfn-lint) |"
                     " `base`,`rule` |\n"
                 ),
+                call(
+                    '| [E0003<a name="E0003"></a>]'
+                    "(../src/cfnlint/rules/errors/config.py) | "
+                    "Error with cfn-lint configuration | "
+                    "Error as a result of the cfn-lint configuration |  | "
+                    "[Source](https://github.com/aws-cloudformation/cfn-lint) "
+                    "| `base`,`rule` |\n"
+                ),
                 call("\n\\* experimental rules\n"),
             ]
             mock_builtin_open.return_value.write.assert_has_calls(expected_calls)
diff --git a/test/unit/module/runner/test_runner.py b/test/unit/module/runner/test_runner.py
index 60290d37b8..185e9c161c 100644
--- a/test/unit/module/runner/test_runner.py
+++ b/test/unit/module/runner/test_runner.py
@@ -3,6 +3,7 @@
 SPDX-License-Identifier: MIT-0
 """
 
+from io import StringIO
 from unittest.mock import patch
 
 import pytest
@@ -93,3 +94,18 @@ def test_init_schemas(name, registry_path, patch_path, expected):
 
     PROVIDER_SCHEMA_MANAGER._registry_schemas = {}
     PROVIDER_SCHEMA_MANAGER.reset()
+
+
+def test_no_templates():
+    params = ["--template", "does-not-exist.yaml"]
+
+    config = ConfigMixIn(params)
+    with patch("sys.exit") as exit:
+        with patch("sys.stdout", new=StringIO()) as out:
+            exit.assert_not_called()
+            Runner(config)
+            assert out.getvalue().strip() == (
+                "E0003 does-not-exist.yaml could not "
+                "be processed by glob.glob\nNone:1:1"
+            )
+            exit.assert_called_once_with(2)