Repository: ansible/ansible

Issue:
Problem with delegate_to + with_items + random
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
- Ansible itself (?)

##### ANSIBLE VERSION
```
ansible-playbook 2.3.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
```

##### CONFIGURATION

- no configuration changes

##### OS / ENVIRONMENT

- Linux, Ubuntu 16.04, but I think the problem is platform-independent

##### SUMMARY

- Combining `delegate_to` with `with_items` and `random` because the connection is not made on an IP basis but rather by name and the name does not resolve.

##### STEPS TO REPRODUCE

The command to run

```
ansible-playbook -i inventory playbook.yml
```

The `playbook.yml`

```yaml
---
- hosts: nodes
  user: root
  tasks:
  - name: MWE for bug
    shell: hostname
    delegate_to: "{{ item }}"
    # with_items:
    ## Working
    # - "{{ groups['hosts'] }}"
    ## Also working
    # - "{{ groups['hosts']|first }}"
    ## Breaks occasionally
    # - "{{ groups['hosts']|random }}"
    ## Also breaks occasionally
    # - "{{ groups['hosts']|shuffle|first }}"
    ## Also fails randomly
    # with_random_choice:
    # - the_server_1
    # - the_server_2
    # - the_server_3
    # - the_server_4
    # - the_server_5
```

The `inventory` file

```ini
[nodes]
the_node ansible_host=172.16.128.8

[hosts]
the_server_1   ansible_host=172.16.96.10
the_server_2   ansible_host=172.16.96.11
the_server_3   ansible_host=172.16.96.12
the_server_4   ansible_host=172.16.96.13
the_server_5   ansible_host=172.16.96.9
```

##### EXPECTED RESULTS

When uncommenting the `with_random_choice:` section or the `with_items:` line with either entry, I would expect Ansible to connect to the delegate host and execute "hostname".

##### ACTUAL RESULTS

Occasionally, there are problems where the short name is not resolved to the IP which causes a connection problem. Below, the `- "{{ groups['hosts']|first }}"` line is uncommented which works

```
PLAY [nodes] *******************************************************************

TASK [Gathering Facts] *********************************************************
ok: [the_node]

TASK [MWE for bug] *************************************************************
changed: [the_node -> 172.16.96.10] => (item=the_server_1)

PLAY RECAP *********************************************************************
the_node                   : ok=2    changed=1    unreachable=0    failed=0   
```

With `- "{{ groups['hosts'] }}"` it also works:

```
PLAY [nodes] *******************************************************************

TASK [Gathering Facts] *********************************************************
ok: [the_node]

TASK [MWE for bug] *************************************************************
changed: [the_node -> 172.16.96.10] => (item=the_server_1)
changed: [the_node -> 172.16.96.11] => (item=the_server_2)
changed: [the_node -> 172.16.96.12] => (item=the_server_3)
changed: [the_node -> 172.16.96.13] => (item=the_server_4)
changed: [the_node -> 172.16.96.9] => (item=the_server_5)

PLAY RECAP *********************************************************************
the_node                   : ok=2    changed=1    unreachable=0    failed=0   
```

With `- "{{ groups['hosts']|random }}"` it fails:

```
PLAY [nodes] *******************************************************************

TASK [Gathering Facts] *********************************************************
ok: [the_node]

TASK [MWE for bug] *************************************************************
fatal: [the_node]: UNREACHABLE! => {"changed": false, "msg": "All items completed", "results": [{"_ansible_item_result": true, "item": "the_server_4", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname the_server_4: Name or service not known\r\n", "unreachable": true}]}
failed: [the_node] (item=the_server_4) => {"item": "the_server_4", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname the_server_4: Name or service not known\r\n", "unreachable": true}
	to retry, use: --limit @/home/mholtgre/tm/ansible/playbook.retry

PLAY RECAP *********************************************************************
the_node                   : ok=1    changed=0    unreachable=1    failed=0   
```

With `- "{{ groups['hosts']|shuffle|first }}"` it also fails

```
PLAY [nodes] *******************************************************************

TASK [Gathering Facts] *********************************************************
ok: [the_node]

TASK [MWE for bug] *************************************************************
fatal: [the_node]: UNREACHABLE! => {"changed": false, "msg": "All items completed", "results": [{"_ansible_item_result": true, "item": "the_server_3", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname the_server_3: Name or service not known\r\n", "unreachable": true}]}
failed: [the_node] (item=the_server_3) => {"item": "the_server_3", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname the_server_3: Name or service not known\r\n", "unreachable": true}
	to retry, use: --limit @/home/mholtgre/tm/ansible/playbook.retry

PLAY RECAP *********************************************************************
the_node                   : ok=1    changed=0    unreachable=1    failed=0   
```


Discussion:
Are you able to provide more details by running it with `-vvvv` for more verbose logs so we can try and see what is happening.
needs_info
Of course, please find the output below

```
Using /etc/ansible/ansible.cfg as config file
Loading callback plugin default of type stdout, v2.0 from /home/mholtgre/local/venv/lib/python3.5/site-packages/ansible/plugins/callback/__init__.py

PLAYBOOK: playbook.yml **********************************************************************************************************************************************************************************************************************
1 plays in playbook.yml

PLAY [nodes] ********************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************************************************************************************************
Using module file /home/mholtgre/local/venv/lib/python3.5/site-packages/ansible/modules/system/setup.py
<172.16.128.8> ESTABLISH SSH CONNECTION FOR USER: root
<172.16.128.8> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/mholtgre/.ansible/cp/a9e7f0c262 172.16.128.8 '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<172.16.128.8> (0, b'/root\n', b'OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016\r\ndebug1: Reading configuration data /home/mholtgre/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 15803\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<172.16.128.8> ESTABLISH SSH CONNECTION FOR USER: root
<172.16.128.8> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/mholtgre/.ansible/cp/a9e7f0c262 172.16.128.8 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1502963601.1522179-134475669624972 `" && echo ansible-tmp-1502963601.1522179-134475669624972="` echo /root/.ansible/tmp/ansible-tmp-1502963601.1522179-134475669624972 `" ) && sleep 0'"'"''
<172.16.128.8> (0, b'ansible-tmp-1502963601.1522179-134475669624972=/root/.ansible/tmp/ansible-tmp-1502963601.1522179-134475669624972\n', b'OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016\r\ndebug1: Reading configuration data /home/mholtgre/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 15803\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<172.16.128.8> PUT /tmp/tmp0zgjvf8s TO /root/.ansible/tmp/ansible-tmp-1502963601.1522179-134475669624972/setup.py
<172.16.128.8> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/mholtgre/.ansible/cp/a9e7f0c262 '[172.16.128.8]'
<172.16.128.8> (0, b'sftp> put /tmp/tmp0zgjvf8s /root/.ansible/tmp/ansible-tmp-1502963601.1522179-134475669624972/setup.py\n', b'OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016\r\ndebug1: Reading configuration data /home/mholtgre/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 15803\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug3: Sent message fd 3 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /root size 0\r\ndebug3: Looking up /tmp/tmp0zgjvf8s\r\ndebug3: Sent message fd 3 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/root/.ansible/tmp/ansible-tmp-1502963601.1522179-134475669624972/setup.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:5485\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 5485 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<172.16.128.8> ESTABLISH SSH CONNECTION FOR USER: root
<172.16.128.8> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/mholtgre/.ansible/cp/a9e7f0c262 172.16.128.8 '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1502963601.1522179-134475669624972/ /root/.ansible/tmp/ansible-tmp-1502963601.1522179-134475669624972/setup.py && sleep 0'"'"''
<172.16.128.8> (0, b'', b'OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016\r\ndebug1: Reading configuration data /home/mholtgre/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 15803\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<172.16.128.8> ESTABLISH SSH CONNECTION FOR USER: root
<172.16.128.8> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/mholtgre/.ansible/cp/a9e7f0c262 -tt 172.16.128.8 '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1502963601.1522179-134475669624972/setup.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1502963601.1522179-134475669624972/" > /dev/null 2>&1 && sleep 0'"'"''
<172.16.128.8> (0, b'\r\n{"invocation": {"module_args": {"filter": "*", "gather_subset": ["all"], "fact_path": "/etc/ansible/facts.d", "gather_timeout": 10}}, "changed": false, "ansible_facts": {"ansible_gather_subset": ["hardware", "network", "virtual"], "ansible_product_serial": "YLXR008900", "ansible_form_factor": "Tower", "ansible_product_version": "NA", "ansible_fips": false, "ansible_service_mgr": "systemd", "ansible_user_id": "root", "ansible_user_dir": "/root", "ansible_memtotal_mb": 64315, "ansible_architecture": "x86_64", "ansible_distribution_version": "16.04", "ansible_domain": "bihealth.org", "ansible_date_time": {"weekday_number": "4", "iso8601_basic_short": "20170817T115321", "tz": "CEST", "weeknumber": "33", "hour": "11", "year": "2017", "minute": "53", "tz_offset": "+0200", "month": "08", "epoch": "1502963601", "iso8601_micro": "2017-08-17T09:53:21.363452Z", "weekday": "Thursday", "time": "11:53:21", "date": "2017-08-17", "iso8601": "2017-08-17T09:53:21Z", "day": "17", "iso8601_basic": "20170817T115321363364", "second": "21"}, "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIF1EKvXIdgnBHU3ncvjsGbxe0qFwf2uTuXUY3Nm3bwyO", "ansible_processor_cores": 4, "ansible_virtualization_role": "host", "ansible_dns": {"nameservers": ["127.0.1.1"], "search": ["bihealth.org"]}, "ansible_processor_vcpus": 8, "ansible_docker0": {"macaddress": "02:42:34:f3:fd:b9", "features": {"tx_checksum_ipv4": "off [fixed]", "generic_receive_offload": "on", "tx_checksum_ipv6": "off [fixed]", "tx_scatter_gather_fraglist": "on", "rx_all": "off [fixed]", "highdma": "on", "rx_fcs": "off [fixed]", "tx_lockless": "on [fixed]", "tx_tcp_ecn_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_gso_robust": "on", "tx_ipip_segmentation": "on", "tx_checksumming": "on", "vlan_challenged": "off [fixed]", "loopback": "off [fixed]", "fcoe_mtu": "off [fixed]", "tx_checksum_sctp": "off [fixed]", "tx_vlan_stag_hw_insert": "on", "rx_vlan_stag_hw_parse": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "large_receive_offload": "off [fixed]", "tx_scatter_gather": "on", "rx_checksumming": "off [fixed]", "tx_tcp_segmentation": "on", "netns_local": "on [fixed]", "busy_poll": "off [fixed]", "generic_segmentation_offload": "on", "tx_udp_tnl_segmentation": "on", "tcp_segmentation_offload": "on", "l2_fwd_offload": "off [fixed]", "rx_vlan_offload": "off [fixed]", "ntuple_filters": "off [fixed]", "tx_vlan_offload": "on", "tx_nocache_copy": "off", "udp_fragmentation_offload": "on", "scatter_gather": "on", "tx_sit_segmentation": "on", "tx_checksum_fcoe_crc": "off [fixed]", "hw_tc_offload": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_fcoe_segmentation": "on", "rx_vlan_filter": "off [fixed]", "receive_hashing": "off [fixed]", "tx_gre_segmentation": "on"}, "interfaces": [], "mtu": 1500, "device": "docker0", "promisc": false, "stp": false, "ipv4": {"broadcast": "global", "netmask": "255.255.0.0", "network": "172.17.0.0", "address": "172.17.0.1"}, "active": false, "type": "bridge", "id": "8000.024234f3fdb9"}, "ansible_bios_version": "V5.0.0.9 R1.8.0 for D3348-A1x", "ansible_processor": ["GenuineIntel", "Intel(R) Xeon(R) CPU E5-1630 v3 @ 3.70GHz", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1630 v3 @ 3.70GHz", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1630 v3 @ 3.70GHz", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1630 v3 @ 3.70GHz", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1630 v3 @ 3.70GHz", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1630 v3 @ 3.70GHz", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1630 v3 @ 3.70GHz", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1630 v3 @ 3.70GHz"], "ansible_virtualization_type": "kvm", "ansible_lo": {"features": {"tx_checksum_ipv4": "off [fixed]", "generic_receive_offload": "on", "tx_checksum_ipv6": "off [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "rx_all": "off [fixed]", "highdma": "on [fixed]", "rx_fcs": "off [fixed]", "tx_lockless": "on [fixed]", "tx_tcp_ecn_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_checksumming": "on", "vlan_challenged": "on [fixed]", "loopback": "on [fixed]", "fcoe_mtu": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "large_receive_offload": "off [fixed]", "tx_scatter_gather": "on [fixed]", "rx_checksumming": "on [fixed]", "tx_tcp_segmentation": "on", "netns_local": "on [fixed]", "busy_poll": "off [fixed]", "generic_segmentation_offload": "on", "tx_udp_tnl_segmentation": "off [fixed]", "tcp_segmentation_offload": "on", "l2_fwd_offload": "off [fixed]", "rx_vlan_offload": "off [fixed]", "ntuple_filters": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_nocache_copy": "off [fixed]", "udp_fragmentation_offload": "on", "scatter_gather": "on", "tx_sit_segmentation": "off [fixed]", "tx_checksum_fcoe_crc": "off [fixed]", "hw_tc_offload": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_fcoe_segmentation": "off [fixed]", "rx_vlan_filter": "off [fixed]", "receive_hashing": "off [fixed]", "tx_gre_segmentation": "off [fixed]"}, "mtu": 65536, "device": "lo", "promisc": false, "ipv4": {"broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0", "address": "127.0.0.1"}, "ipv6": [{"scope": "host", "prefix": "128", "address": "::1"}], "active": true, "type": "loopback"}, "ansible_userspace_bits": "64", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGwmDnvLUUKELH97bwjTkklG1wy9v9tWtEQSCQeD+R1mS+Dwg+48gtrbobBVj7auAQeAGqBd1u3Jy9ZzqHWk3Bo=", "ansible_real_user_id": 0, "ansible_default_ipv4": {"macaddress": "90:1b:0e:51:1f:c5", "network": "172.16.128.0", "mtu": 1500, "broadcast": "172.16.131.255", "alias": "enp0s25", "netmask": "255.255.252.0", "address": "172.16.128.8", "interface": "enp0s25", "type": "ether", "gateway": "172.16.128.1"}, "ansible_swapfree_mb": 65351, "ansible_default_ipv6": {}, "ansible_distribution_release": "xenial", "ansible_system_vendor": "FUJITSU", "ansible_apparmor": {"status": "enabled"}, "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.4.0-91-generic.efi.signed", "quiet": true, "splash": true, "vt.handoff": "7", "ro": true, "root": "UUID=ccf2755a-d965-44ce-ad5d-9a1096133c91"}, "ansible_effective_user_id": 0, "ansible_mounts": [{"uuid": "ccf2755a-d965-44ce-ad5d-9a1096133c91", "size_total": 183857750016, "mount": "/", "size_available": 81865752576, "fstype": "ext4", "device": "/dev/sda2", "options": "rw,relatime,errors=remount-ro,data=ordered"}, {"uuid": "AC90-3AEE", "size_total": 535805952, "mount": "/boot/efi", "size_available": 532242432, "fstype": "vfat", "device": "/dev/sda1", "options": "rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro"}, {"uuid": "1aba97b7-7b8a-4da1-bb60-3754e7bae440", "size_total": 1968874332160, "mount": "/vol/local/data", "size_available": 799776894976, "fstype": "ext4", "device": "/dev/sdb1", "options": "rw,nodiratime,relatime,data=ordered"}, {"uuid": "N/A", "size_total": 536823726080, "mount": "/cubit", "size_available": 487093960704, "fstype": "nfs", "device": "cubihome.bihealth.org:/tank/cubit", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 905244049408, "mount": "/home", "size_available": 732450258944, "fstype": "nfs", "device": "cubihome.bihealth.org:/home", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 95760154624, "mount": "/home/eblanc", "size_available": 54476668928, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/eblanc", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 107374182400, "mount": "/home/bkehr", "size_available": 82516639744, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/bkehr", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 93463773184, "mount": "/home/mholtgre", "size_available": 21071134720, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/mholtgre", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 101589188608, "mount": "/home/ostolpe", "size_available": 58900611072, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/ostolpe", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 107374182400, "mount": "/home/mirko", "size_available": 107374182400, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/mirko", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 104360574976, "mount": "/home/ygloaguen", "size_available": 96895762432, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/ygloaguen", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 107374182400, "mount": "/home/twhite", "size_available": 99056877568, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/twhite", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 97081360384, "mount": "/home/cmessers", "size_available": 46955233280, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/cmessers", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 107374182400, "mount": "/home/mkircher", "size_available": 107374182400, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/mkircher", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 107374182400, "mount": "/home/tzito", "size_available": 104983429120, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/tzito", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 107374182400, "mount": "/home/kbentele", "size_available": 90371522560, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/kbentele", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 99045343232, "mount": "/home/dharnett", "size_available": 72922169344, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/dharnett", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 101767446528, "mount": "/home/sroskosch", "size_available": 84887470080, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/sroskosch", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 107374182400, "mount": "/home/jmuino", "size_available": 95780077568, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/jmuino", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 266026876928, "mount": "/home/aivanov", "size_available": 138995040256, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/aivanov", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 105332604928, "mount": "/home/tkrannich", "size_available": 102429097984, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/tkrannich", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 107374182400, "mount": "/home/lkuchenb", "size_available": 104786296832, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/lkuchenb", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "N/A", "size_total": 103245938688, "mount": "/home/mnieminen", "size_available": 60835233792, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/mnieminen", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}, {"uuid": "ccf2755a-d965-44ce-ad5d-9a1096133c91", "size_total": 183857750016, "mount": "/var/lib/docker/aufs", "size_available": 81865752576, "fstype": "ext4", "device": "/dev/sda2", "options": "rw,relatime,errors=remount-ro,data=ordered,bind"}, {"uuid": "N/A", "size_total": 0, "mount": "/vol/sshfs/mholtgre/bih_cluster", "size_available": 0, "fstype": "fuse.sshfs", "device": "bihcluster_mount:/", "options": "rw,nosuid,nodev,relatime,user_id=5005,group_id=5000"}, {"uuid": "N/A", "size_total": 107356356608, "mount": "/home/mcouteli", "size_available": 107062755328, "fstype": "nfs", "device": "cubihome.bihealth.org:/home/mcouteli", "options": "rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.128.6,mountvers=3,mountport=41175,mountproto=udp,local_lock=none,addr=172.16.128.6"}], "ansible_selinux": {"status": "disabled"}, "ansible_os_family": "Debian", "ansible_userspace_architecture": "x86_64", "ansible_product_uuid": "ACAA2FDD-DC91-E511-B126-FAB7C570D31B", "ansible_product_name": "CELSIUS M740", "ansible_pkg_mgr": "apt", "ansible_memfree_mb": 35272, "ansible_devices": {"sr0": {"scheduler_mode": "deadline", "rotational": "1", "vendor": "HL-DT-ST", "sectors": "2097151", "sas_device_handle": null, "sas_address": null, "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "sectorsize": "512", "removable": "1", "support_discard": "0", "model": "DVDRAM GHC0N", "partitions": {}, "holders": [], "size": "1024.00 MB"}, "sda": {"scheduler_mode": "deadline", "rotational": "0", "vendor": "ATA", "sectors": "500118192", "sas_device_handle": null, "sas_address": null, "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "sectorsize": "512", "removable": "0", "support_discard": "512", "model": "SAMSUNG MZ7LN256", "partitions": {"sda2": {"sectorsize": 512, "uuid": "ccf2755a-d965-44ce-ad5d-9a1096133c91", "sectors": "365088768", "start": "1050624", "holders": [], "size": "174.09 GB"}, "sda3": {"sectorsize": 512, "uuid": "e0162722-c06c-457f-8a46-0218347dddba", "sectors": "133978112", "start": "366139392", "holders": [], "size": "63.89 GB"}, "sda1": {"sectorsize": 512, "uuid": "AC90-3AEE", "sectors": "1048576", "start": "2048", "holders": [], "size": "512.00 MB"}}, "holders": [], "size": "238.47 GB"}, "sdb": {"scheduler_mode": "deadline", "rotational": "1", "vendor": "ATA", "sectors": "3907029168", "sas_device_handle": null, "sas_address": null, "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "sectorsize": "512", "removable": "0", "support_discard": "0", "model": "WDC WD2000FYYZ-5", "partitions": {"sdb1": {"sectorsize": 512, "uuid": "1aba97b7-7b8a-4da1-bb60-3754e7bae440", "sectors": "3907027120", "start": "2048", "holders": [], "size": "1.82 TB"}}, "holders": [], "size": "1.82 TB"}}, "ansible_user_uid": 0, "ansible_memory_mb": {"real": {"total": 64315, "used": 29043, "free": 35272}, "swap": {"cached": 7, "total": 65418, "free": 65351, "used": 67}, "nocache": {"used": 22488, "free": 41827}}, "ansible_distribution": "Ubuntu", "ansible_system_capabilities_enforced": "True", "ansible_env": {"LANG": "en_US.utf8", "TERM": "xterm-256color", "SHELL": "/bin/bash", "XDG_RUNTIME_DIR": "/run/user/0", "LANGUAGE": "en_US", "MAIL": "/var/mail/root", "SHLVL": "1", "SSH_TTY": "/dev/pts/31", "SSH_AUTH_SOCK": "/tmp/ssh-NF6fjLRxkv/agent.15798", "SSH_CLIENT": "172.16.128.8 47564 22", "PWD": "/root", "LOGNAME": "root", "USER": "root", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games", "HOME": "/root", "SSH_CONNECTION": "172.16.128.8 47564 172.16.128.8 22", "XDG_SESSION_ID": "1048", "LC_ALL": "en_US.UTF-8", "_": "/bin/sh"}, "ansible_distribution_major_version": "16", "module_setup": true, "ansible_processor_count": 1, "ansible_hostname": "cubi16", "ansible_effective_group_id": 0, "ansible_swaptotal_mb": 65418, "ansible_lsb": {"release": "16.04", "major_release": "16", "codename": "xenial", "id": "Ubuntu", "description": "Ubuntu 16.04.3 LTS"}, "ansible_real_group_id": 0, "ansible_bios_date": "04/02/2015", "ansible_all_ipv6_addresses": ["fe80::921b:eff:fe51:1fc5"], "ansible_interfaces": ["enp0s25", "docker0", "br-78f685c58a69", "lo"], "ansible_uptime_seconds": 517123, "ansible_machine_id": "bdafefce7e27885d9f722cd958aacb01", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQC/RswUyoPpy43icwO2JvAgEro5VIzoZ1P0w29ug8Zwb3PV1m5bUlX5Z2F2u3ILRQ65GYDa5o5hAxPjQPmByEFk+n+Z3bQeHyaq5HMLP34eym+3VPzy+1Psgnx69AnYBsuHIHRdt1XYrno6L9qgYdYGFyJ7PgaQyKHSfP+9FIn4Wn6NT/3pFMPf6OdP4Dop0Ct5en/+1zJUZOY8XIKGksa65cZVYtl3ExAGRCOe0vnXXgCAIb8TxLMtd64Wys/4OY+56Kqys67LGiDF2YIFLXcA2v/2Q5AQ9ZeVHLN1yFNm7HQuuB1wpMIE3MEFfvjia4sH9y25nyUBRUNNqCGrxvSN", "ansible_br_78f685c58a69": {"macaddress": "02:42:3f:66:97:01", "features": {"tx_checksum_ipv4": "off [fixed]", "generic_receive_offload": "on", "tx_checksum_ipv6": "off [fixed]", "tx_scatter_gather_fraglist": "on", "rx_all": "off [fixed]", "highdma": "on", "rx_fcs": "off [fixed]", "tx_lockless": "on [fixed]", "tx_tcp_ecn_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_gso_robust": "on", "tx_ipip_segmentation": "on", "tx_checksumming": "on", "vlan_challenged": "off [fixed]", "loopback": "off [fixed]", "fcoe_mtu": "off [fixed]", "tx_checksum_sctp": "off [fixed]", "tx_vlan_stag_hw_insert": "on", "rx_vlan_stag_hw_parse": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "large_receive_offload": "off [fixed]", "tx_scatter_gather": "on", "rx_checksumming": "off [fixed]", "tx_tcp_segmentation": "on", "netns_local": "on [fixed]", "busy_poll": "off [fixed]", "generic_segmentation_offload": "on", "tx_udp_tnl_segmentation": "on", "tcp_segmentation_offload": "on", "l2_fwd_offload": "off [fixed]", "rx_vlan_offload": "off [fixed]", "ntuple_filters": "off [fixed]", "tx_vlan_offload": "on", "tx_nocache_copy": "off", "udp_fragmentation_offload": "on", "scatter_gather": "on", "tx_sit_segmentation": "on", "tx_checksum_fcoe_crc": "off [fixed]", "hw_tc_offload": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_fcoe_segmentation": "on", "rx_vlan_filter": "off [fixed]", "receive_hashing": "off [fixed]", "tx_gre_segmentation": "on"}, "interfaces": [], "mtu": 1500, "device": "br-78f685c58a69", "promisc": false, "stp": false, "ipv4": {"broadcast": "global", "netmask": "255.255.0.0", "network": "172.18.0.0", "address": "172.18.0.1"}, "active": false, "type": "bridge", "id": "8000.02423f669701"}, "ansible_user_gecos": "root", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "cap_wake_alarm", "cap_block_suspend", "37+ep"], "ansible_python": {"executable": "/usr/bin/python", "version": {"micro": 12, "major": 2, "releaselevel": "final", "serial": 0, "minor": 7}, "type": "CPython", "has_sslcontext": true, "version_info": [2, 7, 12, "final", 0]}, "ansible_kernel": "4.4.0-91-generic", "ansible_processor_threads_per_core": 2, "ansible_fqdn": "cubi16.bihealth.org", "ansible_user_gid": 0, "ansible_nodename": "cubi16", "ansible_system": "Linux", "ansible_user_shell": "/bin/bash", "ansible_machine": "x86_64", "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBANh1VfuGr8qKemU38qBnD3E6pPG6LECVZ1WEhfPKZ2RjGKOJbLdMdPg2I796DzmkWyfktRgpSh5D1/75/14jd9TFDh0RO31lyx43Y4isJ6GGL0saNmRT1FuVn84qjF+vpzyCAdlMedcoAMcWkqIjpjyC/y5CYOWwR81FBisvikEHAAAAFQDNb26fMqeo/dmigFFm8NTfxUn1wQAAAIEAn4UAoEMPTD4UwmFqJlvoGESmdg8to9hMlZavEFG4Mp7NTAr9NHVHnpKLbR2N7LvG6uv5pn1UOrOvJWF3QwkaN9CkdSslzdrcEznM5KQJEdyaJN5Xx3XIPkqUD4ejFPu5Y1DTghj5HtRrnkz1YcvyNMvw0BZl5LbI2lXoTfIohP8AAACAIv8/sBWczxx5jfFxsYRFZZjgLMDlaQ6ctdngvGRkKBBHRMJHph0OcX+TE6LFBF+Q/A5m1/8jJPb6KXS1GkdGkI539jwWzRBtopsSHo0sIaYryEriK2OJWzrG1RSR9sXzjZlhkLa2Ze2nSRHllqhdpvCbGT2EmY6vfjMnfFwmY/c=", "ansible_all_ipv4_addresses": ["172.16.128.8", "172.18.0.1", "172.17.0.1"], "ansible_python_version": "2.7.12", "ansible_enp0s25": {"macaddress": "90:1b:0e:51:1f:c5", "features": {"tx_checksum_ipv4": "off [fixed]", "generic_receive_offload": "on", "tx_checksum_ipv6": "off [fixed]", "tx_scatter_gather_fraglist": "off [fixed]", "rx_all": "off", "highdma": "on [fixed]", "rx_fcs": "off", "tx_lockless": "off [fixed]", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_checksumming": "on", "vlan_challenged": "off [fixed]", "loopback": "off [fixed]", "fcoe_mtu": "off [fixed]", "tx_checksum_sctp": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "large_receive_offload": "off [fixed]", "tx_scatter_gather": "on", "rx_checksumming": "on", "tx_tcp_segmentation": "on", "netns_local": "off [fixed]", "busy_poll": "off [fixed]", "generic_segmentation_offload": "on", "tx_udp_tnl_segmentation": "off [fixed]", "tcp_segmentation_offload": "on", "l2_fwd_offload": "off [fixed]", "rx_vlan_offload": "on", "ntuple_filters": "off [fixed]", "tx_vlan_offload": "on", "tx_nocache_copy": "off", "udp_fragmentation_offload": "off [fixed]", "scatter_gather": "on", "tx_sit_segmentation": "off [fixed]", "tx_checksum_fcoe_crc": "off [fixed]", "hw_tc_offload": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_fcoe_segmentation": "off [fixed]", "rx_vlan_filter": "off [fixed]", "receive_hashing": "on", "tx_gre_segmentation": "off [fixed]"}, "type": "ether", "pciid": "0000:00:19.0", "module": "e1000e", "mtu": 1500, "device": "enp0s25", "promisc": false, "ipv4": {"broadcast": "172.16.131.255", "netmask": "255.255.252.0", "network": "172.16.128.0", "address": "172.16.128.8"}, "ipv6": [{"scope": "link", "prefix": "64", "address": "fe80::921b:eff:fe51:1fc5"}], "active": true, "speed": 1000}}}\r\n', b'OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016\r\ndebug1: Reading configuration data /home/mholtgre/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 15803\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to 172.16.128.8 closed.\r\n')
ok: [the_node]
META: ran handlers

TASK [MWE for bug] **************************************************************************************************************************************************************************************************************************
task path: /home/mholtgre/tm/ansible/playbook.yml:5
Using module file /home/mholtgre/local/venv/lib/python3.5/site-packages/ansible/modules/commands/command.py
<the_server_5> ESTABLISH SSH CONNECTION FOR USER: root
<the_server_5> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/mholtgre/.ansible/cp/3041dbd995 the_server_5 '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<the_server_5> (255, b'', b'OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016\r\ndebug1: Reading configuration data /home/mholtgre/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket "/home/mholtgre/.ansible/cp/3041dbd995" does not exist\r\ndebug2: resolving "the_server_5" port 22\r\nssh: Could not resolve hostname the_server_5: Name or service not known\r\n')
fatal: [the_node]: UNREACHABLE! => {
    "changed": false,
    "msg": "All items completed",
    "results": [
        {
            "_ansible_item_result": true,
            "item": "the_server_5",
            "msg": "Failed to connect to the host via ssh: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016\r\ndebug1: Reading configuration data /home/mholtgre/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/home/mholtgre/.ansible/cp/3041dbd995\" does not exist\r\ndebug2: resolving \"the_server_5\" port 22\r\nssh: Could not resolve hostname the_server_5: Name or service not known\r\n",
            "unreachable": true
        }
    ]
}
failed: [the_node] (item=the_server_5) => {
    "item": "the_server_5",
    "msg": "Failed to connect to the host via ssh: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016\r\ndebug1: Reading configuration data /home/mholtgre/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/home/mholtgre/.ansible/cp/3041dbd995\" does not exist\r\ndebug2: resolving \"the_server_5\" port 22\r\nssh: Could not resolve hostname the_server_5: Name or service not known\r\n",
    "unreachable": true
}
	to retry, use: --limit @/home/mholtgre/tm/ansible/playbook.retry

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
the_node                   : ok=1    changed=0    unreachable=1    failed=0  
```
I did a small amount of research on this, and have found it is due to the jinja2 in the loop being evaluated multiple times.

It is first evaluated in `VariableManager._get_delegated_vars ` and then again in `TaskExecutor._run_loop`.

As such, if the expression does not produce the same result each time, it may cause templating of both to produce different results.

Due to this, `_get_delegated_vars` will create `ansible_delegated_vars` for a different host than the one targeted by `delegate_to`, and results in no `ansible_host` being found, falling back to just the literal name of the host to connect to.

Resolving patch:
diff --git a/lib/ansible/vars/manager.py b/lib/ansible/vars/manager.py
--- a/lib/ansible/vars/manager.py
+++ b/lib/ansible/vars/manager.py
@@ -498,10 +498,19 @@ def _get_delegated_vars(self, play, task, existing_variables):
                     # This task will be skipped later due to this, so we just setup
                     # a dummy array for the later code so it doesn't fail
                     items = [None]
+                # Update task.loop with templated items, this ensures that delegate_to+loop
+                # doesn't produce different restuls than TaskExecutor which may reprocess the loop
+                # Set loop_with to None, so we don't do extra unexpected processing on the cached items later
+                # in TaskExecutor
+                task.loop_with = None
+                task.loop = items
             else:
                 raise AnsibleError("Failed to find the lookup named '%s' in the available lookup plugins" % task.loop_with)
         elif task.loop is not None:
             items = templar.template(task.loop)
+            # Update task.loop with templated items, this ensures that delegate_to+loop
+            # doesn't produce different restuls than TaskExecutor which may reprocess the loop
+            task.loop = items
         else:
             items = [None]
 


Test patch:
diff --git a/test/integration/targets/delegate_to/runme.sh b/test/integration/targets/delegate_to/runme.sh
--- a/test/integration/targets/delegate_to/runme.sh
+++ b/test/integration/targets/delegate_to/runme.sh
@@ -6,3 +6,5 @@ ANSIBLE_SSH_ARGS='-C -o ControlMaster=auto -o ControlPersist=60s -o UserKnownHos
     ANSIBLE_HOST_KEY_CHECKING=false ansible-playbook test_delegate_to.yml -i ../../inventory -v "$@"
 
 ansible-playbook test_loop_control.yml -v "$@"
+
+ansible-playbook test_delegate_to_loop_randomness.yml -v "$@"
diff --git a/test/integration/targets/delegate_to/test_delegate_to_loop_randomness.yml b/test/integration/targets/delegate_to/test_delegate_to_loop_randomness.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/delegate_to/test_delegate_to_loop_randomness.yml
@@ -0,0 +1,58 @@
+---
+- name: Integration tests for #28231
+  hosts: localhost
+  gather_facts: false
+  tasks:
+    - name: Add some test hosts
+      add_host:
+        name: "foo{{item}}"
+        groups: foo
+      loop: "{{ range(10)|list }}"
+
+    # We expect all of the next 3 runs to succeeed
+    # this is done multiple times to increase randomness
+    - assert:
+        that:
+          - item in ansible_delegated_vars
+      delegate_to: "{{ item }}"
+      loop:
+        - "{{ groups.foo|random }}"
+      ignore_errors: true
+      register: result1
+
+    - assert:
+        that:
+          - item in ansible_delegated_vars
+      delegate_to: "{{ item }}"
+      loop:
+        - "{{ groups.foo|random }}"
+      ignore_errors: true
+      register: result2
+
+    - assert:
+        that:
+          - item in ansible_delegated_vars
+      delegate_to: "{{ item }}"
+      loop:
+        - "{{ groups.foo|random }}"
+      ignore_errors: true
+      register: result3
+
+    - debug:
+        var: result1
+
+    - debug:
+        var: result2
+
+    - debug:
+        var: result3
+
+    - name: Ensure all of the 3 asserts were successful
+      assert:
+        that:
+          - results is all
+      vars:
+        results:
+          - "{{ (result1.results|first) is successful }}"
+          - "{{ (result2.results|first) is successful }}"
+          - "{{ (result3.results|first) is successful }}"