Repository: ansible/ansible

Issue:
delegate_facts does not work with run_once
A regression has appeared between v2.0.0.2-1 (7de237c) and devel (or v2.0.2.0-0.3rc3)
when trying to delegate facts with the run_once flag. Same as documented at (http://docs.ansible.com/ansible/playbooks_delegation.html#delegated-facts) except with the addition of the `run_once: true` parameter to prevent ssh errors, which I presume is as a result of NxM vs M connections. 
##### ANSIBLE VERSION

```
ansible 2.1.0 (devel 54acdd7ead) last updated 2016/04/11 17:27:14 (GMT +1050)
  lib/ansible/modules/core: (detached HEAD c52f475c64) last updated 2016/04/11 17:57:12 (GMT +1050)
  lib/ansible/modules/extras: (detached HEAD 5abb914315) last updated 2016/04/11 17:57:13 (GMT +1050)
  config file = <redacted>/ansible-playbooks/ansible.cfg
  configured module search path = Default w/o overrides
```
##### CONFIGURATION

```
[defaults]
host_key_checking = False

[ssh_connection]
control_path = %(directory)s/%%h-%%r
```
##### OS / ENVIRONMENT

OS X El Capitan v10.11.3
##### STEPS TO REPRODUCE

Playbook:

```

---
- hosts: hostsA
  become: yes
  tasks:
    - name: Gather facts other hosts
      setup:
      delegate_to: "{{ item }}"
      delegate_facts: true
      with_items: "{{ groups.hostsB }}"
      run_once: true

    - name: Print other hosts ipv4 addresses
      debug:
        var: "hostvars[item].ansible_eth0.ipv4.address"
      with_items: "{{ groups.hostsB }}"
```
##### EXPECTED RESULTS

```
ok: [host-a1.example.net] => (item=host-b1.example.net) => {
    "hostvars[item].ansible_eth0.ipv4.address": "xxx.xxx.210.25",
    "item": "host-b1.example.net"
}
ok: [host-a1.example.net] => (item=host-b2.example.net) => {
    "hostvars[item].ansible_eth0.ipv4.address": "xxx.xxx.210.69",
    "item": "host-b2.example.net"
}
ok: [host-a2.example.net] => (item=host-b1.example.net) => {
    "hostvars[item].ansible_eth0.ipv4.address": "xxx.xxx.210.25",
    "item": "host-b1.example.net"
}
ok: [host-a2.example.net] => (item=host-b2.example.net) => {
    "hostvars[item].ansible_eth0.ipv4.address": "xxx.xxx.210.69",
    "item": "host-b2.example.net"
}
ok: [host-a3.example.net] => (item=host-b1.example.net) => {
    "hostvars[item].ansible_eth0.ipv4.address": "xxx.xxx.210.25",
    "item": "host-b1.example.net"
}
ok: [host-a3.example.net] => (item=host-b2.example.net) => {
    "hostvars[item].ansible_eth0.ipv4.address": "xxx.xxx.210.69",
    "item": "host-b2.example.net"
}
```
##### ACTUAL RESULTS

```
ok: [host-a1.example.net] => (item=host-b1.example.net) => {
    "hostvars[item].ansible_eth0.ipv4.address": "VARIABLE IS NOT DEFINED!",
    "item": "host-b1.example.net"
}
ok: [host-a2.example.net] => (item=host-b1.example.net) => {
    "hostvars[item].ansible_eth0.ipv4.address": "VARIABLE IS NOT DEFINED!",
    "item": "host-b1.example.net"
}
ok: [host-a3.example.net] => (item=host-b1.example.net) => {
    "hostvars[item].ansible_eth0.ipv4.address": "VARIABLE IS NOT DEFINED!",
    "item": "host-b1.example.net"
}
ok: [host-a1.example.net] => (item=host-b2.example.net) => {
    "hostvars[item].ansible_eth0.ipv4.address": "VARIABLE IS NOT DEFINED!",
    "item": "host-b2.example.net"
}
ok: [host-a2.example.net] => (item=host-b2.example.net) => {
    "hostvars[item].ansible_eth0.ipv4.address": "VARIABLE IS NOT DEFINED!",
    "item": "host-b2.example.net"
}
ok: [host-a3.example.net] => (item=host-b2.example.net) => {
    "hostvars[item].ansible_eth0.ipv4.address": "VARIABLE IS NOT DEFINED!",
    "item": "host-b2.example.net"
}
```
##### Related issues:

https://github.com/ansible/ansible/issues/14166
https://github.com/ansible/ansible/issues/13880



Discussion:
See also #14279 for another run_once / vars issue that may or may not be related (basically, consider both issues while fixing this)

I'm seeing this issue with 2.2.0 as well.
I am having this exact problem with 2.2.0 as well. delegate_facts does not work with run_once, which seems to be necessary in order to prevent ssh errors.

ssh errors can be avoided with serial: 1 in the play, which allows for the task to work as expected without run_once, albeit very slowly.
I use following workaround:
```
- hosts: "{{ play_hosts[0] }}"
  tasks:
    - setup:
      delegate_to: "{{ item }}"
      delegate_facts: True
      with_items: "{{ groups['all']}}"
```
So run_once also implies 'copy facts to all hosts in play', in this case you are giving contradictory instructions to Ansible. Leaving open as I'm not sure if we want to a)document better , b) issue a warning or c)make this a special case
I'm seeing a traceback attempting to reproduce this.

hosts
``` ini
hostA1 ansible_host=127.0.0.1
#hostB1 ansible_host=192.168.1.36
hostB1 ansible_host=127.0.0.1
me ansible_host=127.0.0.1

[hostsA]
hostA1

[hostsB]
hostB1
```

test.yml
``` yaml
---
- hosts: hostsA
  become: yes
  tasks:
    - name: Gather facts other hosts
      setup:
      delegate_to: "{{ item }}"
      delegate_facts: true
      with_items: "{{ groups.hostsB }}"
      run_once: true

    - name: Print other hosts ipv4 addresses
      debug:
        var: "hostvars[item].ansible_eth0.ipv4.address"
      with_items: "{{ groups.hostsB }}"
```

traceback:

```
ERROR! Unexpected Exception: global name 'task_vars' is not defined
the full traceback was:

Traceback (most recent call last):
  File "/home/adrian/src/ansible/bin/ansible-playbook", line 103, in <module>
    exit_code = cli.run()
  File "/home/adrian/src/ansible/lib/ansible/cli/playbook.py", line 154, in run
    results = pbex.run()
  File "/home/adrian/src/ansible/lib/ansible/executor/playbook_executor.py", line 153, in run
    result = self._tqm.run(play=play)
  File "/home/adrian/src/ansible/lib/ansible/executor/task_queue_manager.py", line 284, in run
    play_return = strategy.run(iterator, play_context)
  File "/home/adrian/src/ansible/lib/ansible/plugins/strategy/linear.py", line 278, in run
    results += self._wait_on_pending_results(iterator)
  File "/home/adrian/src/ansible/lib/ansible/plugins/strategy/__init__.py", line 570, in _wait_on_pending_results
    results = self._process_pending_results(iterator)
  File "/home/adrian/src/ansible/lib/ansible/plugins/strategy/__init__.py", line 483, in _process_pending_results
    task_vars[loop_var] = item
NameError: global name 'task_vars' is not defined
```

The traceback in https://github.com/ansible/ansible/issues/15365#issuecomment-282752186  looks related to 3d65482927e33daa30bb6bfc2153b4c64775ee15 so likely unrelated.
Going back to the commit before 3d65482, I can reproduce the observed behavior:


hosts
``` ini
hostA1 ansible_host=127.0.0.1
#hostB1 ansible_host=192.168.1.36
hostB1 ansible_host=127.0.0.1
me ansible_host=127.0.0.1

[hostsA]
hostA1

[hostsB]
hostB1
```

test.yml
``` yaml
---
- hosts: hostsA
  become: yes
  tasks:
    - name: Gather facts other hosts
      setup:
      delegate_to: "{{ item }}"
      delegate_facts: true
      with_items: "{{ groups.hostsB }}"
      run_once: true

    - name: Print other hosts ipv4 addresses
      debug:
        var: "hostvars[item].ansible_eth0.ipv4.address"
      with_items: "{{ groups.hostsB }}"
```

output:

```
[fedora-25:ansible ((40235d7...) %)]$ git log -1
commit 40235d7b995f4d03c58d62234a60f7b774485401
Author: jctanner <tanner.jc@gmail.com>
Date:   Tue Feb 21 14:46:10 2017 -0500

    Skip fact gathering if the entire play was included via conditional and False (#21734)
    
    Addresses #21528
[fedora-25:ansible ((40235d7...) %)]$ ansible-playbook -i repro/run_once_delegated_facts_15365/hosts -vv repro/run_once_delegated_facts_15365/test.yml 
Using /home/adrian/.ansible.cfg as config file

PLAYBOOK: test.yml ********************************************************************************************************************************************************************************
1 plays in repro/run_once_delegated_facts_15365/test.yml

PLAY [hostsA] *************************************************************************************************************************************************************************************

TASK [Gathering Facts] ****************************************************************************************************************************************************************************
ok: [hostA1]
META: ran handlers

TASK [Gather facts other hosts] *******************************************************************************************************************************************************************
task path: /home/adrian/src/ansible/repro/run_once_delegated_facts_15365/test.yml:5
ok: [hostA1 -> 127.0.0.1] => (item=hostB1)

TASK [Print other hosts ipv4 addresses] ***********************************************************************************************************************************************************
task path: /home/adrian/src/ansible/repro/run_once_delegated_facts_15365/test.yml:16
ok: [hostA1] => (item=hostB1) => {
    "hostvars[item].ansible_eth0.ipv4.address": "VARIABLE IS NOT DEFINED!", 
    "item": "hostB1"
}
META: ran handlers
META: ran handlers

PLAY RECAP ****************************************************************************************************************************************************************************************
hostA1                     : ok=3    changed=0    unreachable=0    failed=0   
```
traceback only happens when looping and is unrelated issue (previous change removed used task_vars)
After applying patch from https://gist.github.com/bcoca/66f5cc81716bace24499f1a437299816 to devel/, I seem to still see the behavior mentioned in original report.

```
[fedora-25:ansible (devel % u=)]$ cat repro/run_once_delegated_facts_15365/test.yml
---
- hosts: hostsA
  become: yes
  tasks:
    - name: Gather facts other hosts
      setup:
      delegate_to: "{{ item }}"
      delegate_facts: true
      with_items: "{{ groups.hostsB }}"
      run_once: true


    - name: Print other hosts ipv4 addresses
      debug:
        var: "hostvars[item].ansible_eth0.ipv4.address"
      with_items: "{{ groups.hostsB }}"

      #    - name: debug vars
      #  debug:
      #   var: groups
[fedora-25:ansible (devel % u=)]$ ansible-playbook -i repro/run_once_delegated_facts_15365/hosts -vv repro/run_once_delegated_facts_15365/test.yml 
Using /home/adrian/.ansible.cfg as config file

PLAYBOOK: test.yml ********************************************************************************************************************************************************************************
1 plays in repro/run_once_delegated_facts_15365/test.yml

PLAY [hostsA] *************************************************************************************************************************************************************************************

TASK [Gathering Facts] ****************************************************************************************************************************************************************************
ok: [hostA1]
META: ran handlers

TASK [Gather facts other hosts] *******************************************************************************************************************************************************************
task path: /home/adrian/src/ansible/repro/run_once_delegated_facts_15365/test.yml:5
ok: [hostA1 -> 127.0.0.1] => (item=hostB1)
ok: [hostA1 -> 127.0.0.1] => (item=me)

TASK [Print other hosts ipv4 addresses] ***********************************************************************************************************************************************************
task path: /home/adrian/src/ansible/repro/run_once_delegated_facts_15365/test.yml:13
ok: [hostA1] => (item=hostB1) => {
    "hostvars[item].ansible_eth0.ipv4.address": "VARIABLE IS NOT DEFINED!", 
    "item": "hostB1"
}
ok: [hostA1] => (item=me) => {
    "hostvars[item].ansible_eth0.ipv4.address": "VARIABLE IS NOT DEFINED!", 
    "item": "me"
}
META: ran handlers
META: ran handlers

PLAY RECAP ****************************************************************************************************************************************************************************************
hostA1                     : ok=3    changed=0    unreachable=0    failed=0   

```

Resolving patch:
diff --git a/lib/ansible/plugins/strategy/__init__.py b/lib/ansible/plugins/strategy/__init__.py
--- a/lib/ansible/plugins/strategy/__init__.py
+++ b/lib/ansible/plugins/strategy/__init__.py
@@ -236,6 +236,15 @@ def get_task_hosts(self, iterator, task_host, task):
             host_list = [task_host]
         return host_list
 
+    def get_delegated_hosts(self, result, loop_var, task):
+
+        host_name = task.delegate_to
+        actual_host = self._inventory.get_host(host_name)
+        if actual_host is None:
+            actual_host = Host(name=host_name)
+
+        return [actual_host]
+
     def _process_pending_results(self, iterator, one_pass=False, max_passes=None):
         '''
         Reads results off the final queue and takes appropriate action
@@ -483,27 +492,25 @@ def parent_handler_match(target_handler, handler_name):
 
                     if 'ansible_facts' in result_item:
 
-                        # if delegated fact and we are delegating facts, we need to change target host for them
-                        if original_task.delegate_to is not None and original_task.delegate_facts:
-                            item = result_item.get(loop_var, None)
-                            if item is not None:
-                                task_vars[loop_var] = item
-                            host_name = templar.template(original_task.delegate_to)
-                            actual_host = self._inventory.get_host(host_name)
-                            if actual_host is None:
-                                actual_host = Host(name=host_name)
-                        else:
-                            actual_host = original_host
-
-                        host_list = self.get_task_hosts(iterator, actual_host, original_task)
                         if original_task.action == 'include_vars':
 
+                            if original_task.delegate_to is not None:
+                                host_list = self.get_delegated_hosts(result_item, loop_var, original_task)
+                            else:
+                                host_list = self.get_task_hosts(iterator, original_host, original_task)
+
                             for (var_name, var_value) in iteritems(result_item['ansible_facts']):
                                 # find the host we're actually refering too here, which may
                                 # be a host that is not really in inventory at all
                                 for target_host in host_list:
                                     self._variable_manager.set_host_variable(target_host, var_name, var_value)
                         else:
+                            # if delegated fact and we are delegating facts, we need to change target host for them
+                            if original_task.delegate_to is not None and original_task.delegate_facts:
+                                host_list = self.get_delegated_hosts(result_item, loop_var, original_task)
+                            else:
+                                host_list = self.get_task_hosts(iterator, original_host, original_task)
+
                             for target_host in host_list:
                                 if original_task.action == 'set_fact':
                                     self._variable_manager.set_nonpersistent_facts(target_host, result_item['ansible_facts'].copy())


Test patch: