Repository: ansible/ansible

Issue:
Ansible 2.1.1.0 ignoring remote_user with delegated tasks
##### ISSUE TYPE
- Bug Report
##### ANSIBLE VERSION

<!--- Paste verbatim output from “ansible --version” between quotes below -->

```
ansible 2.1.1.0
  config file = /Users/metalrufflez/Workdesk/Work/git/phoenix/ansible/ansible.cfg
  configured module search path = ['library/', '../library/', '../../library/']
```
##### CONFIGURATION

<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

```
[defaults]
ansible_python_interpreter = /usr/bin/env python
host_key_checking = False
accept_hostkeys = yes
remote_user = admin
hash_behaviour = merge
pipelining = True
retry_files_enabled = False
roles_path = roles/:../roles/:../../roles/
library = library/:../library/:../../library/
display_skipped_hosts = True
```
##### OS / ENVIRONMENT

<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->

Happens on both Debian and OS X
##### SUMMARY

<!--- Explain the problem briefly -->

After updating from 2.1.0.0 to 2.1.1.0 tasks with `delegate_to` ignores remote_user config and use local user instead to ssh to delegated host 
##### STEPS TO REPRODUCE

<!---
For bugs, show exactly how to reproduce the problem.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->

```
- name: Gather EC2 fact from random instance in group phoenix
  hosts: 127.0.0.1
  connection: local
  gather_facts: False

  tasks:
    - name: Get random instance facts
      action: ec2_facts
      delegate_to: "{{ item }}"
      with_random_choice: "{{ groups[app_full_name] | intersect(groups.phoenix) }}"
```

<!--- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS

<!--- What did you expect to happen when running the steps above? -->

When running with verbose output in 2.1.0.0 I get this:

```
> ansible-playbook -i inventory/staging/ playbug.yml -e "app_full_name=desktop_frontend" -vvvv
Using /Users/metalrufflez/Workdesk/Work/git/phoenix/ansible/ansible.cfg as config file
Loaded callback default of type stdout, v2.0

PLAYBOOK: playbug.yml **********************************************************
1 plays in playbug.yml

PLAY [Gather EC2 fact from random instance in group phoenix] *******************

TASK [Get random instance facts] ***********************************************
task path: /Users/metalrufflez/Workdesk/Work/git/phoenix/ansible/playbug.yml:7
<172.32.88.161> ESTABLISH SSH CONNECTION FOR USER: admin
<172.32.88.161> SSH: EXEC ssh -C -vvv -F aux/local_ssh_config -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=admin -o ConnectTimeout=10 172.32.88.161 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1469746205.68-246164042651653 `" && echo ansible-tmp-1469746205.68-246164042651653="` echo $HOME/.ansible/tmp/ansible-tmp-1469746205.68-246164042651653 `" ) && sleep 0'"'"''
<172.32.88.161> PUT /var/folders/p7/b7z8dc7d1b58fjt7ksmyv6f40000gn/T/tmpwFJWet TO /home/admin/.ansible/tmp/ansible-tmp-1469746205.68-246164042651653/ec2_facts
<172.32.88.161> SSH: EXEC sftp -b - -C -vvv -F aux/local_ssh_config -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=admin -o ConnectTimeout=10 '[172.32.88.161]'
<172.32.88.161> ESTABLISH SSH CONNECTION FOR USER: admin
<172.32.88.161> SSH: EXEC ssh -C -vvv -F aux/local_ssh_config -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=admin -o ConnectTimeout=10 -tt 172.32.88.161 '/bin/sh -c '"'"'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/admin/.ansible/tmp/ansible-tmp-1469746205.68-246164042651653/ec2_facts; rm -rf "/home/admin/.ansible/tmp/ansible-tmp-1469746205.68-246164042651653/" > /dev/null 2>&1 && sleep 0'"'"''
ok: [127.0.0.1] => (item=172.32.88.161) => {"ansible_facts": (...)}

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=1    changed=0    unreachable=0    failed=0
```
##### ACTUAL RESULTS

<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

On 2.1.1.0 it ignored my remote_user setting in ansible.cfg and used my local user instead

<!--- Paste verbatim command output between quotes below -->

```
> ansible-playbook -i inventory/staging/ playbug.yml -e "app_full_name=desktop_frontend" -vvvv
Using /Users/metalrufflez/Workdesk/Work/git/phoenix/ansible/ansible.cfg as config file
Loaded callback default of type stdout, v2.0

PLAYBOOK: playbug.yml **********************************************************
1 plays in playbug.yml

PLAY [Gather EC2 fact from random instance in group phoenix] *******************

TASK [Get random instance facts] ***********************************************
task path: /Users/metalrufflez/Workdesk/Work/git/phoenix/ansible/playbug.yml:7
<172.32.88.161> ESTABLISH SSH CONNECTION FOR USER: metalrufflez
<172.32.88.161> SSH: EXEC ssh -C -vvv -F aux/local_ssh_config -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=metalrufflez -o ConnectTimeout=10 172.32.88.161 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1469746084.88-235788747680127 `" && echo ansible-tmp-1469746084.88-235788747680127="` echo $HOME/.ansible/tmp/ansible-tmp-1469746084.88-235788747680127 `" ) && sleep 0'"'"''
failed: [127.0.0.1] (item=172.32.88.161) => {"item": "172.32.88.161", "msg": "Failed to connect to the host via ssh.", "unreachable": true}
fatal: [127.0.0.1]: UNREACHABLE! => {"changed": false, "msg": "All items completed", "results": [{"_ansible_item_result": true, "item": "172.32.88.161", "msg": "Failed to connect to the host via ssh.", "unreachable": true}]}

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=0    changed=0    unreachable=1    failed=0
```

Ansible 2.1.1.0 ignoring remote_user with delegated tasks
##### ISSUE TYPE
- Bug Report
##### ANSIBLE VERSION

<!--- Paste verbatim output from “ansible --version” between quotes below -->

```
ansible 2.1.1.0
  config file = /Users/metalrufflez/Workdesk/Work/git/phoenix/ansible/ansible.cfg
  configured module search path = ['library/', '../library/', '../../library/']
```
##### CONFIGURATION

<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

```
[defaults]
ansible_python_interpreter = /usr/bin/env python
host_key_checking = False
accept_hostkeys = yes
remote_user = admin
hash_behaviour = merge
pipelining = True
retry_files_enabled = False
roles_path = roles/:../roles/:../../roles/
library = library/:../library/:../../library/
display_skipped_hosts = True
```
##### OS / ENVIRONMENT

<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->

Happens on both Debian and OS X
##### SUMMARY

<!--- Explain the problem briefly -->

After updating from 2.1.0.0 to 2.1.1.0 tasks with `delegate_to` ignores remote_user config and use local user instead to ssh to delegated host 
##### STEPS TO REPRODUCE

<!---
For bugs, show exactly how to reproduce the problem.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->

```
- name: Gather EC2 fact from random instance in group phoenix
  hosts: 127.0.0.1
  connection: local
  gather_facts: False

  tasks:
    - name: Get random instance facts
      action: ec2_facts
      delegate_to: "{{ item }}"
      with_random_choice: "{{ groups[app_full_name] | intersect(groups.phoenix) }}"
```

<!--- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS

<!--- What did you expect to happen when running the steps above? -->

When running with verbose output in 2.1.0.0 I get this:

```
> ansible-playbook -i inventory/staging/ playbug.yml -e "app_full_name=desktop_frontend" -vvvv
Using /Users/metalrufflez/Workdesk/Work/git/phoenix/ansible/ansible.cfg as config file
Loaded callback default of type stdout, v2.0

PLAYBOOK: playbug.yml **********************************************************
1 plays in playbug.yml

PLAY [Gather EC2 fact from random instance in group phoenix] *******************

TASK [Get random instance facts] ***********************************************
task path: /Users/metalrufflez/Workdesk/Work/git/phoenix/ansible/playbug.yml:7
<172.32.88.161> ESTABLISH SSH CONNECTION FOR USER: admin
<172.32.88.161> SSH: EXEC ssh -C -vvv -F aux/local_ssh_config -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=admin -o ConnectTimeout=10 172.32.88.161 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1469746205.68-246164042651653 `" && echo ansible-tmp-1469746205.68-246164042651653="` echo $HOME/.ansible/tmp/ansible-tmp-1469746205.68-246164042651653 `" ) && sleep 0'"'"''
<172.32.88.161> PUT /var/folders/p7/b7z8dc7d1b58fjt7ksmyv6f40000gn/T/tmpwFJWet TO /home/admin/.ansible/tmp/ansible-tmp-1469746205.68-246164042651653/ec2_facts
<172.32.88.161> SSH: EXEC sftp -b - -C -vvv -F aux/local_ssh_config -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=admin -o ConnectTimeout=10 '[172.32.88.161]'
<172.32.88.161> ESTABLISH SSH CONNECTION FOR USER: admin
<172.32.88.161> SSH: EXEC ssh -C -vvv -F aux/local_ssh_config -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=admin -o ConnectTimeout=10 -tt 172.32.88.161 '/bin/sh -c '"'"'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/admin/.ansible/tmp/ansible-tmp-1469746205.68-246164042651653/ec2_facts; rm -rf "/home/admin/.ansible/tmp/ansible-tmp-1469746205.68-246164042651653/" > /dev/null 2>&1 && sleep 0'"'"''
ok: [127.0.0.1] => (item=172.32.88.161) => {"ansible_facts": (...)}

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=1    changed=0    unreachable=0    failed=0
```
##### ACTUAL RESULTS

<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

On 2.1.1.0 it ignored my remote_user setting in ansible.cfg and used my local user instead

<!--- Paste verbatim command output between quotes below -->

```
> ansible-playbook -i inventory/staging/ playbug.yml -e "app_full_name=desktop_frontend" -vvvv
Using /Users/metalrufflez/Workdesk/Work/git/phoenix/ansible/ansible.cfg as config file
Loaded callback default of type stdout, v2.0

PLAYBOOK: playbug.yml **********************************************************
1 plays in playbug.yml

PLAY [Gather EC2 fact from random instance in group phoenix] *******************

TASK [Get random instance facts] ***********************************************
task path: /Users/metalrufflez/Workdesk/Work/git/phoenix/ansible/playbug.yml:7
<172.32.88.161> ESTABLISH SSH CONNECTION FOR USER: metalrufflez
<172.32.88.161> SSH: EXEC ssh -C -vvv -F aux/local_ssh_config -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=metalrufflez -o ConnectTimeout=10 172.32.88.161 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1469746084.88-235788747680127 `" && echo ansible-tmp-1469746084.88-235788747680127="` echo $HOME/.ansible/tmp/ansible-tmp-1469746084.88-235788747680127 `" ) && sleep 0'"'"''
failed: [127.0.0.1] (item=172.32.88.161) => {"item": "172.32.88.161", "msg": "Failed to connect to the host via ssh.", "unreachable": true}
fatal: [127.0.0.1]: UNREACHABLE! => {"changed": false, "msg": "All items completed", "results": [{"_ansible_item_result": true, "item": "172.32.88.161", "msg": "Failed to connect to the host via ssh.", "unreachable": true}]}

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=0    changed=0    unreachable=1    failed=0
```



Discussion:
Same happens if you use `connection: local` and with variables like `ansible_python_interpreter`. Completely ignored.

Confirmed on OS X and Ubuntu.

This breaks roles that depend on `ansible_python_interpreter` to talk to instances like CoreOS which don't come with python.

most likely 9248cde2398e3dd604d3d079e8fe0061d72c9a19 related

If folks can reproduce and test out a pr, the pr at https://github.com/ansible/ansible/pull/16900 is potentially related.

I might've caught @bcoca in the middle of a revert but this still fails using https://github.com/bcoca/ansible/commit/d03f3b2696b070bf79c604a3c659e2655bf1a1ed with error message:

```
TASK [defunctzombie.coreos-bootstrap : Check if bootstrap is needed] ***********
task path: /etc/ansible/roles/defunctzombie.coreos-bootstrap/tasks/main.yml:1
<c.c.c.c> ESTABLISH SSH CONNECTION FOR USER: None
<c.c.c.c> SSH: EXEC ssh -q -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="keyfilethatssecret"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/ansible-ssh-%h-%p-%r -tt c.c.c.c 'stat $HOME/.bootstrapped'
<b.b.b.b> ESTABLISH SSH CONNECTION FOR USER: None
<b.b.b.b> SSH: EXEC ssh -q -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="keyfilethatssecret"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/ansible-ssh-%h-%p-%r -tt b.b.b.b 'stat $HOME/.bootstrapped'
<a.a.a.a> ESTABLISH SSH CONNECTION FOR USER: None
<a.a.a.a> SSH: EXEC ssh -q -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="keyfilethatssecret"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/ansible-ssh-%h-%p-%r -tt a.a.a.a 'stat $HOME/.bootstrapped'
fatal: [c.c.c.c]: UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh.", 
    "unreachable": true
}
fatal: [a.a.a.a]: UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh.", 
    "unreachable": true
}
fatal: [b.b.b.b]: UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh.", 
    "unreachable": true
}
    to retry, use: --limit @playbooks/kubernetes.retry

PLAY RECAP *********************************************************************
a.a.a.a               : ok=0    changed=0    unreachable=1    failed=0   
b.b.b.b               : ok=0    changed=0    unreachable=1    failed=0   
c.c.c.c               : ok=0    changed=0    unreachable=1    failed=0   
```

Tested by building ansible using `make deb` using that commit.

After further troubleshooting, I was able to get this to work by passing in `--user core` to the `ansible-playbook` cli. Its interesting to note that both `ansible_ssh_user` and `ansible_user` are ignored.

FYI. I have to comment out https://github.com/ansible/ansible/blob/devel/lib/ansible/playbook/play_context.py#L417-L418 and it works for me. I don't know what this will affect but FYI for whoever come to fix this.

thanks for your testing @winggundamth! 

Any update on fixing this?

Same happens if you use `connection: local` and with variables like `ansible_python_interpreter`. Completely ignored.

Confirmed on OS X and Ubuntu.

This breaks roles that depend on `ansible_python_interpreter` to talk to instances like CoreOS which don't come with python.

most likely 9248cde2398e3dd604d3d079e8fe0061d72c9a19 related

If folks can reproduce and test out a pr, the pr at https://github.com/ansible/ansible/pull/16900 is potentially related.

I might've caught @bcoca in the middle of a revert but this still fails using https://github.com/bcoca/ansible/commit/d03f3b2696b070bf79c604a3c659e2655bf1a1ed with error message:

```
TASK [defunctzombie.coreos-bootstrap : Check if bootstrap is needed] ***********
task path: /etc/ansible/roles/defunctzombie.coreos-bootstrap/tasks/main.yml:1
<c.c.c.c> ESTABLISH SSH CONNECTION FOR USER: None
<c.c.c.c> SSH: EXEC ssh -q -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="keyfilethatssecret"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/ansible-ssh-%h-%p-%r -tt c.c.c.c 'stat $HOME/.bootstrapped'
<b.b.b.b> ESTABLISH SSH CONNECTION FOR USER: None
<b.b.b.b> SSH: EXEC ssh -q -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="keyfilethatssecret"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/ansible-ssh-%h-%p-%r -tt b.b.b.b 'stat $HOME/.bootstrapped'
<a.a.a.a> ESTABLISH SSH CONNECTION FOR USER: None
<a.a.a.a> SSH: EXEC ssh -q -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="keyfilethatssecret"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/ansible-ssh-%h-%p-%r -tt a.a.a.a 'stat $HOME/.bootstrapped'
fatal: [c.c.c.c]: UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh.", 
    "unreachable": true
}
fatal: [a.a.a.a]: UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh.", 
    "unreachable": true
}
fatal: [b.b.b.b]: UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh.", 
    "unreachable": true
}
    to retry, use: --limit @playbooks/kubernetes.retry

PLAY RECAP *********************************************************************
a.a.a.a               : ok=0    changed=0    unreachable=1    failed=0   
b.b.b.b               : ok=0    changed=0    unreachable=1    failed=0   
c.c.c.c               : ok=0    changed=0    unreachable=1    failed=0   
```

Tested by building ansible using `make deb` using that commit.

After further troubleshooting, I was able to get this to work by passing in `--user core` to the `ansible-playbook` cli. Its interesting to note that both `ansible_ssh_user` and `ansible_user` are ignored.

FYI. I have to comment out https://github.com/ansible/ansible/blob/devel/lib/ansible/playbook/play_context.py#L417-L418 and it works for me. I don't know what this will affect but FYI for whoever come to fix this.

thanks for your testing @winggundamth! 

Any update on fixing this?


Resolving patch:
diff --git a/lib/ansible/playbook/play_context.py b/lib/ansible/playbook/play_context.py
--- a/lib/ansible/playbook/play_context.py
+++ b/lib/ansible/playbook/play_context.py
@@ -415,12 +415,6 @@ def set_task_and_variable_override(self, task, variables, templar):
         if new_info.port is None and C.DEFAULT_REMOTE_PORT is not None:
             new_info.port = int(C.DEFAULT_REMOTE_PORT)
 
-        # if the final connection type is local, reset the remote_user value
-        # to that of the currently logged in user, to ensure any become settings
-        # are obeyed correctly
-        if new_info.connection == 'local':
-            new_info.remote_user = pwd.getpwuid(os.getuid()).pw_name
-
         # special overrides for the connection setting
         if len(delegated_vars) > 0:
             # in the event that we were using local before make sure to reset the
@@ -437,6 +431,14 @@ def set_task_and_variable_override(self, task, variables, templar):
                 elif getattr(new_info, 'connection', None) == 'local' and (not remote_addr_local or not inv_hostname_local):
                     setattr(new_info, 'connection', C.DEFAULT_TRANSPORT)
 
+        # if the final connection type is local, reset the remote_user value
+        # to that of the currently logged in user, to ensure any become settings
+        # are obeyed correctly
+        # additionally, we need to do this check after final connection has been
+        # correctly set above ...
+        if new_info.connection == 'local':
+            new_info.remote_user = pwd.getpwuid(os.getuid()).pw_name
+
         # set no_log to default if it was not previouslly set
         if new_info.no_log is None:
             new_info.no_log = C.DEFAULT_NO_LOG


Test patch: