Repository: ansible/ansible
Pull request: fix flatten handling of nulls/nones
fixed null break bug and added option to include

fixes #69012
fixes #69013 (adds clog, tests and options to orig fix)

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
- Bugfix Pull Request


##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below -->
flatten

Problem statement:
Flatten field yields unexpected results when it hits None

##### SUMMARY
Flatten field yields unexpected results when it hits None. Instead of skipping over none it silently discards the to-be-flatten list. See example below.



##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
flatten filter

##### ANSIBLE VERSION

```paste below
ansible 2.8.1
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.8 (default, Apr  8 2019, 18:17:52) [GCC 8.3.0]
```



##### OS / ENVIRONMENT

`FROM pad92/ansible-alpine:latest`


##### STEPS TO REPRODUCE


<!--- Paste example playbooks or commands between quotes below -->
```yaml
- hosts: localhost
  vars:
    nested_a:
      - "a"
      - "b"
      - ["c","d"]

    nested_b:
      - ["a", None, "b"]
      - "c"
      - "d"

  tasks:
    - name: Works 
      debug:
        msg: "Flatten list: {{ nested_a | flatten }}"

    - name: Where is 'b'
      debug:
        msg: "Flatten list: {{ nested_b | flatten }}"

```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->
ok: [localhost] => {
    "msg": "Flatten list: ['a', 'b', 'c', 'd']"
}

TASK [WTF is 'b'] ************************************************************************************************************
ok: [localhost] => {
    "msg": "Flatten list: ['a', 'b', 'c', 'd']'      <--------- 'b' is present
}

##### ACTUAL RESULTS
ok: [localhost] => {
    "msg": "Flatten list: ['a', 'b', 'c', 'd']"
}

TASK [where is 'b'] ************************************************************************************************************
ok: [localhost] => {
    "msg": "Flatten list: ['a', 'c', 'd']"     <--------- 'b' is missing
}

Hints from issue discussion:
It looks like the issue is with this part of the code:

https://github.com/ansible/ansible/blob/835ad75a0a95566e5d960daf7d962f122eca6d14/lib/ansible/plugins/filter/core.py#L468-L470

That `break` should be a `continue`.
@sivel yes so it looks. 
(I was using locally patched version for few days until my conscience nagging got too loud)

#69013 

ps - thanks for the quick triage :)
resolved_by_pr #69013
@borisovano, Could you please check PR #69013 and let us know if it works for you ? <br><br>Thanks. <br><br> needs_info

Commit message: c

Patch to review:
diff --git a/changelogs/fragments/fix_lp_flat.yml b/changelogs/fragments/fix_lp_flat.yml
new file mode 100644
index 00000000000000..84c5643c642ad0
--- /dev/null
+++ b/changelogs/fragments/fix_lp_flat.yml
@@ -0,0 +1,2 @@
+bugfixes:
+ - Continue execution when  'flatten' filter when it hits a None/null value as part of the list.
diff --git a/docs/docsite/rst/user_guide/playbooks_filters.rst b/docs/docsite/rst/user_guide/playbooks_filters.rst
index 90b010b026e42d..1bd7a5016e3cb2 100644
--- a/docs/docsite/rst/user_guide/playbooks_filters.rst
+++ b/docs/docsite/rst/user_guide/playbooks_filters.rst
@@ -876,6 +876,14 @@ Flatten only the first level of a list (akin to the `items` lookup)::
     {{ [3, [4, [2]] ] | flatten(levels=1) }}
 
 
+.. versionadded:: 2.10
+
+Preserve nulls in a list, by default flatten removes them. ::
+
+    {{ [3, None, [4, [2]] ] | flatten(levels=1, skip_nulls=False) }}
+
+
+
 .. _set_theory_filters:
 
 Set theory filters
diff --git a/lib/ansible/plugins/filter/core.py b/lib/ansible/plugins/filter/core.py
index 9eecd79b79d2a0..0ecc8fa7a3da8d 100644
--- a/lib/ansible/plugins/filter/core.py
+++ b/lib/ansible/plugins/filter/core.py
@@ -467,13 +467,13 @@ def b64decode(string, encoding='utf-8'):
     return to_text(base64.b64decode(to_bytes(string, errors='surrogate_or_strict')), encoding=encoding)
 
 
-def flatten(mylist, levels=None):
+def flatten(mylist, levels=None, skip_nulls=True):
 
     ret = []
     for element in mylist:
-        if element in (None, 'None', 'null'):
-            # ignore undefined items
-            break
+        if skip_nulls and element in (None, 'None', 'null'):
+            # ignore null items
+            continue
         elif is_sequence(element):
             if levels is None:
                 ret.extend(flatten(element))
diff --git a/test/integration/targets/filter_core/tasks/main.yml b/test/integration/targets/filter_core/tasks/main.yml
index e8ef2345eb09ec..54056c24bb3481 100644
--- a/test/integration/targets/filter_core/tasks/main.yml
+++ b/test/integration/targets/filter_core/tasks/main.yml
@@ -113,6 +113,7 @@
       - '"corned beef"|hash("haha, get it?") == None'
 
 - name: Flatten tests
+  tags: flatten
   block:
     - name: use flatten
       set_fact:
@@ -120,6 +121,12 @@
         flat_one: '{{orig_list|flatten(levels=1)}}'
         flat_two: '{{orig_list|flatten(levels=2)}}'
         flat_tuples: '{{ [1,3] | zip([2,4]) | list | flatten }}'
+        flat_full_null: '{{list_with_nulls|flatten(skip_nulls=False)}}'
+        flat_one_null: '{{list_with_nulls|flatten(levels=1, skip_nulls=False)}}'
+        flat_two_null: '{{list_with_nulls|flatten(levels=2, skip_nulls=False)}}'
+        flat_full_nonull: '{{list_with_nulls|flatten(skip_nulls=True)}}'
+        flat_one_nonull: '{{list_with_nulls|flatten(levels=1, skip_nulls=True)}}'
+        flat_two_nonull: '{{list_with_nulls|flatten(levels=2, skip_nulls=True)}}'
 
     - name: Verify flatten filter works as expected
       assert:
@@ -128,8 +135,15 @@
           - flat_one == [1, 2, 3, [4, [5]], 6, 7]
           - flat_two == [1, 2, 3, 4, [5], 6, 7]
           - flat_tuples == [1, 2, 3, 4]
+          - flat_full_null == [1, 'None', 3, 4, 5, 6, 7]
+          - flat_one_null == [1, 'None', 3, [4, [5]], 6, 7]
+          - flat_two_null == [1, 'None', 3, 4, [5], 6, 7]
+          - flat_full_nonull == [1, 3, 4, 5, 6, 7]
+          - flat_one_nonull == [1, 3, [4, [5]], 6, 7]
+          - flat_two_nonull == [1, 3, 4, [5], 6, 7]
   vars:
     orig_list: [1, 2, [3, [4, [5]], 6], 7]
+    list_with_nulls: [1, None, [3, [4, [5]], 6], 7]
 
 - name: Test base64 filter
   assert:


Code review comments:
diff_hunk: @@ -467,13 +467,13 @@ def b64decode(string, encoding='utf-8'):
     return to_text(base64.b64decode(to_bytes(string, errors='surrogate_or_strict')), encoding=encoding)
 
 
-def flatten(mylist, levels=None):
+def flatten(mylist, levels=None, skip_nulls=True):
 
     ret = []
     for element in mylist:
-        if element in (None, 'None', 'null'):
-            # ignore undefined items
-            break
+        if skip_nulls and element in (None, 'None', 'null'):
+            # ignore null items
+            continue
         elif is_sequence(element):
             if levels is None:
                 ret.extend(flatten(element))
path: lib/ansible/plugins/filter/core.py
text: @user1:
```suggestion
                ret.extend(flatten(element, skip_nulls=skip_nulls))
```