Repository: ansible/ansible

Issue:
None becomes "" when assigned to variable
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
None becomes `""` when assigned to variable.

A comparison between a variable and what was assigned to it will return `false`.

Ran into this while using `lookup('file', '/nonexistent', errors='ignore')` - had to change a `when:` condition depending on whether comparing the lookup, or a variable with the lookup result.

##### ISSUE TYPE
Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
core

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.7.0.dev0
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/foo/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.5 (default, May 11 2018, 04:00:52) [GCC 8.1.0]
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
N/A

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- set_fact:
    foo: "{{ None }}"
- debug:
    msg: "{{ foo == None }}"
- debug:
    msg: "{{ foo == '' }}"
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
`true`, `false`

##### ACTUAL RESULTS
`false`, `true`

<!--- Paste verbatim command output between quotes below -->
```
TASK [debug] **********************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": false
}

TASK [debug] **********************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": true
}
```



Discussion:
This is specifically implemented functionality of our jinja2 implementation:

https://github.com/ansible/ansible/blob/a9e53cdb68cfb80a69297a8ab6266d0654d9b6a1/lib/ansible/template/__init__.py#L610-L614

At this point I'm not sure it would be feasible to change this implementation.
Well, that's pretty terrible ;]
I mean it's `foo = bar; foo != bar`. That can't be considered correct, right ?;]
It's possible, that we could bypass this if utilizing native jinja2 types: 

https://github.com/ansible/ansible/blob/edb1735ca07059a3b644867045f70f9f4ee63edd/lib/ansible/config/base.yml#L697-L706

@jctanner @mkrizek do you guys have an opinion on whether we should skip the `_finalize` functionality when using native types?
The other problem is that None means nothing in 'YAML'  so it would be translated into the string 'None', for yaml `~` is the Null/None token.
> The other problem is that None means nothing in 'YAML' so it would be translated into the string 'None', for yaml ~ is the Null/None token.

Yes, I think it would be best if we only allowed it with native types as a result of this.  Unsure if other changes would have to happen to permit this.

Resolving patch:
diff --git a/lib/ansible/template/__init__.py b/lib/ansible/template/__init__.py
--- a/lib/ansible/template/__init__.py
+++ b/lib/ansible/template/__init__.py
@@ -609,8 +609,13 @@ def _convert_bare_variable(self, variable, bare_deprecated):
 
     def _finalize(self, thing):
         '''
-        A custom finalize method for jinja2, which prevents None from being returned
+        A custom finalize method for jinja2, which prevents None from being returned. This
+        avoids a string of ``"None"`` as ``None`` has no importance in YAML.
+
+        If using ANSIBLE_JINJA2_NATIVE we bypass this and return the actual value always
         '''
+        if USE_JINJA2_NATIVE:
+            return thing
         return thing if thing is not None else ''
 
     def _fail_lookup(self, name, *args, **kwargs):


Test patch:
diff --git a/test/integration/targets/jinja2_native_types/runtests.yml b/test/integration/targets/jinja2_native_types/runtests.yml
--- a/test/integration/targets/jinja2_native_types/runtests.yml
+++ b/test/integration/targets/jinja2_native_types/runtests.yml
@@ -29,6 +29,7 @@
       b_false: False
       s_true: "True"
       s_false: "False"
+      yaml_none: ~
   tasks:
       - name: check jinja version
         shell: python -c 'import jinja2; print(jinja2.__version__)'
@@ -44,4 +45,5 @@
             - import_tasks: test_bool.yml
             - import_tasks: test_dunder.yml
             - import_tasks: test_types.yml
+            - import_tasks: test_none.yml
         when: is_native
diff --git a/test/integration/targets/jinja2_native_types/test_casting.yml b/test/integration/targets/jinja2_native_types/test_casting.yml
--- a/test/integration/targets/jinja2_native_types/test_casting.yml
+++ b/test/integration/targets/jinja2_native_types/test_casting.yml
@@ -11,11 +11,11 @@
 - assert:
     that:
         - 'int_to_str == "2"'
-        - 'int_to_str|type_debug in ["string", "unicode"]'
+        - 'int_to_str|type_debug in ["str", "unicode"]'
         - 'str_to_int == 2'
         - 'str_to_int|type_debug == "int"'
-        - 'dict_to_str|type_debug in ["string", "unicode"]'
-        - 'list_to_str|type_debug in ["string", "unicode"]'
+        - 'dict_to_str|type_debug in ["str", "unicode"]'
+        - 'list_to_str|type_debug in ["str", "unicode"]'
         - 'int_to_bool is sameas true'
         - 'int_to_bool|type_debug == "bool"'
         - 'str_true_to_bool is sameas true'
diff --git a/test/integration/targets/jinja2_native_types/test_concatentation.yml b/test/integration/targets/jinja2_native_types/test_concatentation.yml
--- a/test/integration/targets/jinja2_native_types/test_concatentation.yml
+++ b/test/integration/targets/jinja2_native_types/test_concatentation.yml
@@ -23,7 +23,7 @@
 - assert:
     that:
         - 'string_sum == "12"'
-        - 'string_sum|type_debug in ["string", "unicode"]'
+        - 'string_sum|type_debug in ["str", "unicode"]'
 
 - name: add two lists
   set_fact:
@@ -40,7 +40,7 @@
 
 - assert:
     that:
-        - 'list_sum_multi|type_debug in ["string", "unicode"]'
+        - 'list_sum_multi|type_debug in ["str", "unicode"]'
 
 - name: add two dicts
   set_fact:
@@ -58,7 +58,7 @@
 - assert:
     that:
         - 'list_for_strings == "onetwo"'
-        - 'list_for_strings|type_debug in ["string", "unicode"]'
+        - 'list_for_strings|type_debug in ["str", "unicode"]'
 
 - name: loop through list with int
   set_fact:
diff --git a/test/integration/targets/jinja2_native_types/test_dunder.yml b/test/integration/targets/jinja2_native_types/test_dunder.yml
--- a/test/integration/targets/jinja2_native_types/test_dunder.yml
+++ b/test/integration/targets/jinja2_native_types/test_dunder.yml
@@ -20,4 +20,4 @@
 
 - assert:
     that:
-        - 'const_dunder|type_debug in ["string", "unicode"]'
+        - 'const_dunder|type_debug in ["str", "unicode"]'
diff --git a/test/integration/targets/jinja2_native_types/test_none.yml b/test/integration/targets/jinja2_native_types/test_none.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/jinja2_native_types/test_none.yml
@@ -0,0 +1,11 @@
+- name: test none
+  set_fact:
+      none_var: "{{ yaml_none }}"
+      none_var_direct: "{{ None }}"
+
+- assert:
+    that:
+        - 'none_var is sameas none'
+        - 'none_var|type_debug == "NoneType"'
+        - 'none_var_direct is sameas none'
+        - 'none_var_direct|type_debug == "NoneType"'