Repository: ansible/ansible
Pull request: Add an `undef` global Jinja function
##### SUMMARY
Factors in the feedback on the previous attempt #75407 ; I couldn't reopen the PR. This PR originally made a filter, but we felt that wasn't really appropriate given the semantics (it doesn't actually filter anything). I think explicitly calling it Undefined also clarifies its semantics; the fail filter would just return `Undefined` so it didn't explode things, but people might have expected it to behave almost like the `fail` module.

Fixes #69255

Enables the creation of `Undefined` values. This is useful for specifying variables that must be overridden. These variables don't require an intermediate filtering and local variable For example, in a role's defaults file:
```yaml
global_mysql_root_user: "sqladmin"
global_mysql_root_password: "{{ undef(hint='required, generate with pwgen -s 20') }}"
```
instead of the suggested workaround
```yaml
global_mysql_root_user: "sqladmin"
# global_mysql_root_password: 
_global_mysql_root_password: "{{ global_mysql_root_password | mandatory('required, generate with pwgen -s 20') }}"
```
This makes it easier to search a codebase for where a variable is used, as it doesn't require also looking for all the local variants of that variable.

###### Alternatives

One suggestion was to extend the "mandatory" filter to treat `null` values as undefined. But there are valid use cases for requiring a user to specify a value but where `null` is a valid value.

Another suggestion was to use `validate_argument_spec`. I like this suggestion, argspec looks cool. It currently lacks documentation. As well, there are things like vars_files where the `fail` filter is much simpler than an argspec.

##### ISSUE TYPE
- Feature Pull Request

##### COMPONENT NAME
ansible.template.Templar

##### ADDITIONAL INFORMATION
###### Basic Usage
If not overridden:
```bash
ansible localhost -m debug -a msg="{{ foo }}" -e foo="{{ undef('xfail') }}"
localhost | FAILED! => {
    "msg": "The task includes an option with an undefined variable. The error was: {{ undef('xfail') }}: xfail. {{ undef('xfail') }}: xfail"
}

ansible localhost -m debug -a msg="{{ foo }}" -e foo="{{ undef('xfail') }}" -e foo="hi"
localhost | SUCCESS => {
    "msg": "hi"
}
```
###### Other Usage:

```yaml
- name: demo
  hosts: localhost
  connection: local
  gather_facts: false

  tasks:
    - name: print mandatory
      debug:
        var: v0
      ignore_errors: true

    - name: "print fail"
      debug:
        var: v1

    - name: "print fail with default"
      debug:
        var: v2
      ignore_errors: true

    - name: "print fail in ternary"
      debug:
        var: v3
      loop: [True, False]
      ignore_errors: true

    - name: "print msg undef"
      debug:
        msg: "{{ v4 }}"
      ignore_errors: true
    - name: "inspect undef"
      debug:
        var: undefined
      ignore_errors: true

    - name: "undef is undefined"
      assert:
        that: undef() is undefined

  vars:
    v0: "{{ undef() | mandatory('xfail') }}"
    v1: "{{ undef(hint='xfail') }}"
    v2: "{{ undef('xfail') | default('xx') }}"
    v3: "{{ item | ternary(0, undef('xfail')) }}"
    v4: "{{ undef() }}"

```

yields:
```bash
ansible-playbook demo.yml -vvv

PLAYBOOK: demo.yml **************************************************************************************************************************
1 plays in demo.yml

PLAY [demo] *********************************************************************************************************************************
META: ran handlers

TASK [print mandatory] **********************************************************************************************************************
task path: /home/lilatomic/GIT/ansible/demo.yml:7
fatal: [localhost]: FAILED! => {
    "msg": "An unhandled exception occurred while templating '{{ undef() | mandatory('xfail') }}'. Error was a <class 'ansible.errors.AnsibleFilterError'>, original message: xfail"
}
...ignoring

TASK [print fail] ***************************************************************************************************************************
task path: /home/lilatomic/GIT/ansible/demo.yml:12
ok: [localhost] => {
    "v1": "VARIABLE IS NOT DEFINED!: {{ undef(hint='xfail') }}: xfail"
}

TASK [print fail with default] **************************************************************************************************************
task path: /home/lilatomic/GIT/ansible/demo.yml:16
ok: [localhost] => {
    "v2": "xx"
}

TASK [print fail in ternary] ****************************************************************************************************************
task path: /home/lilatomic/GIT/ansible/demo.yml:21
ok: [localhost] => (item=True) => {
    "ansible_loop_var": "item",
    "item": true,
    "v3": "0"
}
ok: [localhost] => (item=False) => {
    "ansible_loop_var": "item",
    "item": false,
    "v3": "VARIABLE IS NOT DEFINED!: {{ item | ternary(0, undef('xfail')) }}: xfail"
}

TASK [print msg undef] **********************************************************************************************************************
task path: /home/lilatomic/GIT/ansible/demo.yml:27
fatal: [localhost]: FAILED! => {
    "msg": "The task includes an option with an undefined variable. The error was: {{ undef() }}: Mandatory variable has not been overridden\n\nThe error appears to be in '/home/lilatomic/GIT/ansible/demo.yml': line 27, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n    - name: \"print msg undef\"\n      ^ here\n"
}
...ignoring

TASK [inspect undef] ************************************************************************************************************************
task path: /home/lilatomic/GIT/ansible/demo.yml:31
ok: [localhost] => {
    "undefined": "VARIABLE IS NOT DEFINED!: 'undefined' is undefined"
}

TASK [undef is undefined] *******************************************************************************************************************
task path: /home/lilatomic/GIT/ansible/demo.yml:36
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}
META: ran handlers
META: ran handlers

PLAY RECAP **********************************************************************************************************************************
localhost                  : ok=7    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2   

```

Problem statement:
add a "fail" filter
<!--- Verify first that your feature was not already discussed on GitHub -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY

Add a "fail" filter that always returns a FilterError with a specific message.

##### ISSUE TYPE
- Feature Idea

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->
core filters

##### ADDITIONAL INFORMATION
<!--- Describe how the feature would be used, why it is needed and what it would solve -->

I would like to list and document all variables. Role specific variables could be listed and documented in "roles/\*/defaults/\*" and global variables in "group_vars/all/global.yml". Unfortunately there are some variables which do not have a good default value. For example, setting a variable like "mysql_password" to an empty string might be fatal. Sure, I could assert at the start of every role that the variable is not empty, but the chances are high that I might forget to add the assert on one place or another. Therefore, something like the following would be nice:

group_vars/all/global.yml:
```yaml
# the password for the mysql root user.
# this password is used in several roles (like monitoring, application setup, etc.)
global_mysql_root_password: "{{ 'required, generate with pwgen -s 20' | fail }}"
```

If I then happen to forget to override the variable somewhere, an error like "failed to render global_mysql_root_password: required, generate with pwgen -s 20" is thrown.

The filter is easy to write (I can provide a PR if you want), is less specific than several other filters already present (e.g. "debug_type") and might be useful in a lot of places.

Hints from issue discussion:
Files identified in the description:
* [`lib/ansible/plugins/filter/core.py`](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/filter/core.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Is there a reason you couldn't just use the mandatory filter?

Don't define the var at all, but you can put a comment like:

```
# the password for the mysql root user.
# this password is used in several roles (like monitoring, application setup, etc.)
# required, generate with pwgen -s 20
# global_mysql_root_password: 
```

And then in your task use:

```
password: "{{ global_mysql_root_password | mandatory('required, generate with pwgen -s 20') }}"
```
Thanks for your reply. That's basically how we are doing it now, but I think this approach has some drawbacks:
1) commenting out the variable makes the file harder to parse visually (no separation, no different syntax color)
2) the variable is used at multiple places in multiple roles. I would have to repeat the mandatory filter multiple times.
3) when reading the variable file it is not clear that this variable is required and how it can be generated.
This would be the filter equivalent of the `fail` module: https://docs.ansible.com/ansible/latest/modules/fail_module.html
What about extending the `mandatory` filter to check not only for `Undefined` but also for the null/None value?
That would allow the following:
```yaml
# the password for the mysql root user.
# this password is used in several roles (like monitoring, application setup, etc.)
# required, generate with pwgen -s 20
global_mysql_root_password:
```
or any of the null YAML specifiers like `~`, `null`, `Null`, `NULL` like:
```yaml
global_mysql_root_password: ~
global_mysql_root_password: null
```
closing as per #75407

Commit message: rebuild as the builtin `Undefined`

Patch to review:
diff --git a/changelogs/fragments/75435-creating-Undefined.yml b/changelogs/fragments/75435-creating-Undefined.yml
new file mode 100644
index 00000000000000..b5ac19c6c433c0
--- /dev/null
+++ b/changelogs/fragments/75435-creating-Undefined.yml
@@ -0,0 +1,2 @@
+minor_changes:
+  - adds the ``Undef`` keyword to the templating environment. This allows for directly creating Undefined values in templates. It is most useful for providing a hint for variables which must be overridden.
diff --git a/docs/docsite/rst/user_guide/playbooks_filters.rst b/docs/docsite/rst/user_guide/playbooks_filters.rst
index f9c032d03785c7..9988041b086e6f 100644
--- a/docs/docsite/rst/user_guide/playbooks_filters.rst
+++ b/docs/docsite/rst/user_guide/playbooks_filters.rst
@@ -68,6 +68,11 @@ If you configure Ansible to ignore undefined variables, you may want to define s
 
 The variable value will be used as is, but the template evaluation will raise an error if it is undefined.
 
+A convenient way of requiring a variable to be overridden is to give it an undefined value using the ``Undef`` keyword. This can be useful in a role's defaults::
+
+    galaxy_url: "https://galaxy.ansible.com"
+    galaxy_api_key: {{ Undef(hint="You must specify your Galaxy API key") }}
+
 Defining different values for true/false/null (ternary)
 =======================================================
 
diff --git a/lib/ansible/plugins/filter/core.py b/lib/ansible/plugins/filter/core.py
index 8c1c55b337a839..7c25b30343b8cb 100644
--- a/lib/ansible/plugins/filter/core.py
+++ b/lib/ansible/plugins/filter/core.py
@@ -32,7 +32,7 @@
 from ansible.module_utils.common.yaml import yaml_load, yaml_load_all
 from ansible.parsing.ajson import AnsibleJSONEncoder
 from ansible.parsing.yaml.dumper import AnsibleDumper
-from ansible.template import recursive_check_defined
+from ansible.template import AnsibleUndefined, recursive_check_defined
 from ansible.utils.display import Display
 from ansible.utils.encrypt import passlib_or_crypt
 from ansible.utils.hashing import md5s, checksum_s
diff --git a/lib/ansible/template/__init__.py b/lib/ansible/template/__init__.py
index f5c0c9cc9c6e26..59830b1a678fac 100644
--- a/lib/ansible/template/__init__.py
+++ b/lib/ansible/template/__init__.py
@@ -693,6 +693,7 @@ def __init__(self, loader, shared_loader_obj=None, variables=None):
         self.environment.globals['query'] = self.environment.globals['q'] = self._query_lookup
         self.environment.globals['now'] = self._now_datetime
         self.environment.globals['finalize'] = self._finalize
+        self.environment.globals['Undef'] = self._make_undefined
 
         # the current rendering context under which the templar class is working
         self.cur_context = None
@@ -1068,6 +1069,13 @@ def _lookup(self, name, *args, **kwargs):
 
         return ran
 
+    def _make_undefined(self, msg=None):
+        from jinja2.runtime import Undefined
+
+        if msg is None or isinstance(msg, Undefined) or msg == '':
+            msg = "Mandatory variable has not been overridden"
+        return AnsibleUndefined(msg)
+
     def do_template(self, data, preserve_trailing_newlines=True, escape_backslashes=True, fail_on_undefined=None, overrides=None, disable_lookups=False):
         if self.jinja2_native and not isinstance(data, string_types):
             return data
diff --git a/test/integration/targets/filter_core/tasks/main.yml b/test/integration/targets/filter_core/tasks/main.yml
index c637fc1754cc42..0c35bb169b61cb 100644
--- a/test/integration/targets/filter_core/tasks/main.yml
+++ b/test/integration/targets/filter_core/tasks/main.yml
@@ -494,6 +494,39 @@
       - mandatory_2 is failed
       - "mandatory_2.msg == 'You did not give me a variable. I am a sad wolf.'"
 
+- name: Verify Undef throws if resolved
+  set_fact:
+    foo: '{{ fail_foo }}'
+  vars:
+    fail_foo: '{{ Undef("Expected failure") }}'
+  ignore_errors: yes
+  register: fail_1
+
+- name: Setup fail_foo for overriding in test
+  block:
+    - name: Verify Undef not executed if overridden
+      set_fact:
+        foo: '{{ fail_foo }}'
+      vars:
+        fail_foo: 'overridden value'
+      register: fail_2
+  vars:
+    fail_foo: '{{ Undef(hint="Expected failure") }}'
+
+- name: Verify Undef is inspectable
+  debug:
+    var: fail_foo
+  vars:
+    fail_foo: '{{ Undef("Expected failure") }}'
+  register: fail_3
+
+- name: Verify Undef
+  assert:
+    that:
+      - fail_1 is failed
+      - not (fail_2 is failed)
+      - not (fail_3 is failed)
+
 - name: Verify comment
   assert:
     that:
diff --git a/test/units/plugins/filter/test_core.py b/test/units/plugins/filter/test_core.py
index 8a626d9a75c5bb..df4e4725def313 100644
--- a/test/units/plugins/filter/test_core.py
+++ b/test/units/plugins/filter/test_core.py
@@ -3,6 +3,8 @@
 # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
 
 from __future__ import absolute_import, division, print_function
+from jinja2.runtime import Undefined
+from jinja2.exceptions import UndefinedError
 __metaclass__ = type
 
 import pytest


Code review comments:
diff_hunk: @@ -68,6 +68,11 @@ If you configure Ansible to ignore undefined variables, you may want to define s
 
 The variable value will be used as is, but the template evaluation will raise an error if it is undefined.
 
+A convenient way of requiring a variable to be overridden is to give it an undefined value using the ``Undef`` keyword. This can be useful in a role's defaults::
path: docs/docsite/rst/user_guide/playbooks_filters.rst
text: @user1:
Please use a code-block here instead of :: .  The translation of Ansible docs can ignore content within a code-block but has problems with ::.  See https://docs.ansible.com/ansible/latest/dev_guide/style_guide/index.html#syntax-highlighting-pygments for an example of a code-block.

@author:
I had copypasted from another section in that file. Is it a problem that many of the other segments don't use a code-block?

@user1:
It makes things harder for translation. If you feel like changing the rest, sure, but don't feel you have to.

diff_hunk: @@ -68,6 +68,11 @@ If you configure Ansible to ignore undefined variables, you may want to define s
 
 The variable value will be used as is, but the template evaluation will raise an error if it is undefined.
 
+A convenient way of requiring a variable to be overridden is to give it an undefined value using the ``Undef`` keyword. This can be useful in a role's defaults::
+
+    galaxy_url: "https://galaxy.ansible.com"
+    galaxy_api_key: {{ Undef(hint="You must specify your Galaxy API key") }}
path: docs/docsite/rst/user_guide/playbooks_filters.rst
text: @user1:
The docs/tests use `hint`, but the code uses `msg`.