Repository: ansible/ansible

Issue:
parse_sources and parse_source between them mishandle INVENTORY_UNPARSED_IS_FAILED
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
Setting ANSIBLE_INVENTORY_UNPARSED_FAILED=True in the environment or inventory.unparsed_is_failed=True in the configuration is supposed to turn failures into fatal errors:

```
INVENTORY_UNPARSED_IS_FAILED:
  name: Unparsed Inventory failure
  default: False
  description: If 'true' unparsed inventory sources become fatal errors, they are warnings otherwise.
  env: [{name: ANSIBLE_INVENTORY_UNPARSED_FAILED}]
  ini:
  - {key: unparsed_is_failed, section: inventory}
  type: bool
```

However, this works as intended only if the inventory contains a single source. If you have an inventory with multiple sources, ``parse_source`` will fail and return a failure to ``parse_sources``, but the latter tests and acts on ``C.INVENTORY_UNPARSED_IS_FAILED`` only if **ALL** invocations of ``parse_source`` failed to parse correctly.

So for an inventory with a static ini file and ec2.py, if ec2.py fails, execution continues blithely onwards regardless of the configuration.

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
core

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
HEAD
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
``inventory.unparsed_is_failed=True``

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
N/A

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->
One needs and inventory directory that contains multiple sources (in my case, one static ini-format source file, and one script source: ec2.py), and cause the script to fail somehow (e.g., kill it while it's running, or firewall away AWS and wait for a timeout).

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Execution should stop after script failure.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
Execution continues after script failure.

<!--- Paste verbatim command output between quotes below -->



Discussion:
This is working as designed, it just doesn't cover this case, at this point i think we need 3 settings:

- fail if no inventory was parsed but source(s) was/were provided (current setting)
- fail if any of the sources provided failed to parse (this ticket)
- fail on any parse failure(#40512)

If you have an inventory comprising multiple sources, is it really
useful to continue blithely on with playbook execution if parts of the
inventory are unavailable, but have a special setting to fail only if
ALL of the inventory sources fail together?

If all of your inventory fails, your plays probably aren't going to run
because of having no matching hosts anyway, so why do you need to do
anything special? Or why have such a setting at all? Why not always
fail if your whole inventory is unparseable?

(With the current behaviour, if you EVER have a static ini format source
in your inventory directory, no combination of failures of the dynamic
scripts and the configuration settings will cause an overall failure.)

Is there a use-case in which this is actually useful and desired
behaviour?

Yes, which is why i added the feature. I understand it is not the case for everyone, that is why i think we need multiple toggles so it works correctly in many varied contexts.
> Yes, which is why i added the feature.

OK. I'll take your word for it, since the original PR/commit/changelog
does not include any rationale for the change.

In that case, the current description for the configuration setting is
extremely misleading:

    If 'true' unparsed inventory sources become fatal errors, they are
    warnings otherwise.

That should say something like:

    If 'true' it is a fatal error if every single potential inventory
    source fails to parse, otherwise this situation will only display a
    warning.

agreed, i had not considered the other cases i outlined above when writing it

the case was presented to me in irc, i should have documented it better
@bcoca does something like this look like what you had in mind?

```
diff --git a/lib/ansible/config/base.yml b/lib/ansible/config/base.yml
index 020ffeb0c7..1becb5330c 100644
--- a/lib/ansible/config/base.yml
+++ b/lib/ansible/config/base.yml
@@ -1375,6 +1375,18 @@ HOST_KEY_CHECKING:
   ini:
   - {key: host_key_checking, section: defaults}
   type: boolean
+INVENTORY_ANY_ERRORS_FATAL:
+  name: Controls whether any unparseable inventory source is a fatal error
+  default: False
+  description: >
+    If 'true', it is a fatal error when any given inventory source
+    cannot be successfully parsed by any available inventory plugin;
+    otherwise, this situation only attracts a warning.
+  type: boolean
+  env: [{name: ANSIBLE_INVENTORY_ERRORS_FATAL}]
+  ini:
+    - {key: any_errors_fatal, section: inventory}
+  version_added: "2.7"
 INVENTORY_ENABLED:
   name: Active Inventory plugins
   default: ['host_list', 'script', 'yaml', 'ini', 'auto']
@@ -1412,7 +1424,10 @@ INVENTORY_IGNORE_PATTERNS:
 INVENTORY_UNPARSED_IS_FAILED:
   name: Unparsed Inventory failure
   default: False
-  description: If 'true' unparsed inventory sources become fatal errors, they are warnings otherwise.
+  description: >
+    If 'true' it is a fatal error if every single potential inventory
+    source fails to parse, otherwise this situation will only attract a
+    warning.
   env: [{name: ANSIBLE_INVENTORY_UNPARSED_FAILED}]
   ini:
   - {key: unparsed_is_failed, section: inventory}
diff --git a/lib/ansible/inventory/manager.py b/lib/ansible/inventory/manager.py
index 0772cff636..a39bc620f1 100644
--- a/lib/ansible/inventory/manager.py
+++ b/lib/ansible/inventory/manager.py
@@ -285,6 +285,8 @@ class InventoryManager(object):
                         display.warning(u'\n* Failed to parse %s with %s plugin: %s' % (to_text(fail['src']), fail['plugin'], to_text(fail['exc'])))
                         if hasattr(fail['exc'], 'tb'):
                             display.vvv(to_text(fail['exc'].tb))
+                    if C.INVENTORY_ANY_ERRORS_FATAL:
+                        raise AnsibleError(u'Completely failed to parse inventory source %s' % (to_text(source)))
         if not parsed:
             display.warning("Unable to parse %s as an inventory source" % to_text(source))
```

I've tested it and it works in exactly the way I expect, and it still displays the individual warnings from each inventory plugin (i.e., it doesn't make debugging harder or anything like that).

Resolving patch:
diff --git a/lib/ansible/inventory/manager.py b/lib/ansible/inventory/manager.py
--- a/lib/ansible/inventory/manager.py
+++ b/lib/ansible/inventory/manager.py
@@ -285,6 +285,8 @@ def parse_source(self, source, cache=False):
                         display.warning(u'\n* Failed to parse %s with %s plugin: %s' % (to_text(fail['src']), fail['plugin'], to_text(fail['exc'])))
                         if hasattr(fail['exc'], 'tb'):
                             display.vvv(to_text(fail['exc'].tb))
+                    if C.INVENTORY_ANY_UNPARSED_IS_FAILED:
+                        raise AnsibleError(u'Completely failed to parse inventory source %s' % (to_text(source)))
         if not parsed:
             display.warning("Unable to parse %s as an inventory source" % to_text(source))
 


Test patch: