Repository: ansible/ansible

Issue:
set_fact and with_items broken
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
set_fact

##### ANSIBLE VERSION
```
ansible 2.6.0 (detached HEAD 3bec76fc85) last updated 2018/04/04 14:21:07 (GMT +200)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [' /xxx/src/ansible/library']
  ansible python module location =   #xxx/src/ansible/lib/ansible
  executable location = xxx/src/ansible/bin/ansible
  python version = 3.6.4 (default, Jan  5 2018, 02:35:40) [GCC 7.2.1 20171224]
```

##### OS / ENVIRONMENT
N/A

##### SUMMARY
Appending to a list in set_fact using with_items is broken.

##### STEPS TO REPRODUCE
Run this playbook.

```yaml
---
- name: test set_fact bug
  hosts: localhost
  connection: local
  gather_facts: no
  vars:
    - dig_list:
      - one

    - digits:
      - two
      - three
      - four
  tasks:

    - name: test bug
      set_fact:
        dig_list: "{{ dig_list + [ digit ] }}"
      with_items: "{{ digits }}"
      loop_control:
        loop_var: digit

    - debug: msg="{{ dig_list }}"

```
##### EXPECTED RESULTS
An Array `['one', ..., 'four']`

##### ACTUAL RESULTS
An Array `['one', 'four']`

See also #23624. This breaks one of my roles.
Git bisect shows me that #37538 breaks this.


Discussion:
Files identified in the description:
* [lib/ansible/modules/utilities/logic/set_fact.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/utilities/logic/set_fact.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @dagwieers
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/lib/ansible/executor/task_executor.py b/lib/ansible/executor/task_executor.py
--- a/lib/ansible/executor/task_executor.py
+++ b/lib/ansible/executor/task_executor.py
@@ -593,10 +593,13 @@ def _evaluate_failed_when_result(result):
                     failed_when_result = False
                 return failed_when_result
 
-            if 'ansible_facts' in result and self._task.action not in ('set_fact', 'include_vars'):
-                vars_copy.update(namespace_facts(result['ansible_facts']))
-                if C.INJECT_FACTS_AS_VARS:
-                    vars_copy.update(clean_facts(result['ansible_facts']))
+            if 'ansible_facts' in result:
+                if self._task.action in ('set_fact', 'include_vars'):
+                    vars_copy.update(result['ansible_facts'])
+                else:
+                    vars_copy.update(namespace_facts(result['ansible_facts']))
+                    if C.INJECT_FACTS_AS_VARS:
+                        vars_copy.update(clean_facts(result['ansible_facts']))
 
             # set the failed property if it was missing.
             if 'failed' not in result:
@@ -651,10 +654,13 @@ def _evaluate_failed_when_result(result):
         if self._task.register:
             variables[self._task.register] = wrap_var(result)
 
-        if 'ansible_facts' in result and self._task.action not in ('set_fact', 'include_vars'):
-            variables.update(namespace_facts(result['ansible_facts']))
-            if C.INJECT_FACTS_AS_VARS:
-                variables.update(clean_facts(result['ansible_facts']))
+        if 'ansible_facts' in result:
+            if self._task.action in ('set_fact', 'include_vars'):
+                variables.update(result['ansible_facts'])
+            else:
+                variables.update(namespace_facts(result['ansible_facts']))
+                if C.INJECT_FACTS_AS_VARS:
+                    variables.update(clean_facts(result['ansible_facts']))
 
         # save the notification target in the result, if it was specified, as
         # this task may be running in a loop in which case the notification


Test patch:
diff --git a/test/integration/targets/set_fact/incremental.yml b/test/integration/targets/set_fact/incremental.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/set_fact/incremental.yml
@@ -0,0 +1,19 @@
+- name: test set_fact incremental https://github.com/ansible/ansible/issues/38271
+  hosts: testhost
+  gather_facts: no
+  tasks:
+    - name: Generate inline loop for set_fact
+      set_fact:
+        dig_list: "{{ dig_list + [ item ] }}"
+      loop:
+        - two
+        - three
+        - four
+      vars:
+        dig_list:
+            - one
+
+    - name: verify cumulative set fact worked
+      assert:
+        that:
+           - dig_list == ['one', 'two', 'three', 'four']
diff --git a/test/integration/targets/set_fact/nowarn_clean_facts.yml b/test/integration/targets/set_fact/nowarn_clean_facts.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/set_fact/nowarn_clean_facts.yml
@@ -0,0 +1,7 @@
+- name: Test no warnings ref "http://github.com/ansible/ansible/issues/37535"
+  hosts: testhost
+  gather_facts: false
+  tasks:
+    - name: set ssh jump host args
+      set_fact:
+        ansible_ssh_common_args: "-o ProxyCommand='ssh -W %h:%p -q root@localhost'"
diff --git a/test/integration/targets/set_fact/runme.sh b/test/integration/targets/set_fact/runme.sh
--- a/test/integration/targets/set_fact/runme.sh
+++ b/test/integration/targets/set_fact/runme.sh
@@ -5,9 +5,15 @@ set -eux
 MYTMPDIR=$(mktemp -d 2>/dev/null || mktemp -d -t 'mytmpdir')
 trap 'rm -rf "${MYTMPDIR}"' EXIT
 
-ANSIBLE_CACHE_PLUGIN=jsonfile ANSIBLE_CACHE_PLUGIN_CONNECTION="${MYTMPDIR}" ansible-playbook -i ../../inventory "$@" set_fact_cached_1.yml
+# ensure we can incrementally set fact via loopi, injection or not
+ANSIBLE_INJECT_FACT_VARS=0 ansible-playbook -i ../../inventory incremental.yml
+ANSIBLE_INJECT_FACT_VARS=1 ansible-playbook -i ../../inventory incremental.yml
 
-ANSIBLE_CACHE_PLUGIN=jsonfile ANSIBLE_CACHE_PLUGIN_CONNECTION="${MYTMPDIR}" ansible-playbook -i ../../inventory "$@" set_fact_cached_2.yml
-
-ANSIBLE_CACHE_PLUGIN=jsonfile ANSIBLE_CACHE_PLUGIN_CONNECTION="${MYTMPDIR}" ansible-playbook -i ../../inventory --flush-cache "$@" set_fact_no_cache.yml
+# ensure we dont have spurious warnings do to clean_facts
+ansible-playbook -i ../../inventory nowarn_clean_facts.yml | grep '[WARNING]: Removed restricted key from module data: ansible_ssh_common_args' && exit 1
 
+# test cached feature
+export ANSIBLE_CACHE_PLUGIN=jsonfile ANSIBLE_CACHE_PLUGIN_CONNECTION="${MYTMPDIR}"
+ansible-playbook -i ../../inventory "$@" set_fact_cached_1.yml
+ansible-playbook -i ../../inventory "$@" set_fact_cached_2.yml
+ansible-playbook -i ../../inventory --flush-cache "$@" set_fact_no_cache.yml