Repository: ansible/ansible

Issue:
file transfer failure after upgrade from 2.2.0 to 2.2.1
##### ISSUE TYPE

 - Bug Report

##### COMPONENT NAME

Local connection plugin?

##### ANSIBLE VERSION

```
ansible 2.2.1.0
  config file =
  configured module search path = Default w/o overrides
```

##### CONFIGURATION

```
[ssh_connection]
# http://docs.ansible.com/intro_configuration.html#pipelining
pipelining=True
# http://docs.ansible.com/intro_configuration.html#ssh-args
ssh_args = -o ControlMaster=auto -o ControlPersist=30m
forks = 50
```

##### OS / ENVIRONMENT

`16.04.1 LTS (Xenial Xerus)`

##### SUMMARY

When `cloud-init` runs `ansible-playbook -i hosts -c local /tmp/ansible/playbooks/env_production.yml --limit=bastion`, I get the following error:
```
PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
fatal: [bastion]: FAILED! => {"failed": true, "msg": "failed to transfer file to ~/.ansible/tmp/ansible-tmp-1484618416.23-141753847363017/setup.py: [Errno 2] No such file or directory: '~/.ansible/tmp/ansible-tmp-1484618416.23-141753847363017/setup.py'"}
        to retry, use: --limit @/tmp/ansible/playbooks/env_production.retry

PLAY RECAP *********************************************************************
bastion : ok=0    changed=0    unreachable=0    failed=1
```

This doesn't happen Ansible 2.2.0. No other changes have been done.
This doesn't happen if I run the script manually from a shell.

The only way I have to reproduce the error is to terminate and spawn a new EC2 instance.
I'm trying to find another way to reproduce it.


Discussion:
With pipelining enabled, you should not have a copy of the script happening. I cannot reproduce with a simple setup, since this works for you on the command line I suspect some issues with the environment when using cloud-init
I've set the user script to run ansible with `-vvvv`.
I now get:
```
No config file found; using defaults
...
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: env_production.yml ***************************************************
11 plays in /tmp/ansible/playbooks/env_production.yml

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/core/system/setup.py
<54.238.202.20> ESTABLISH LOCAL CONNECTION FOR USER: root
<54.238.202.20> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1484621108.81-173128532792632 `" && echo ansible-tmp-1484621108.81-173128532792632="` echo ~/.ansible/tmp/ansible-tmp-1484621108.81-173128532792632 `" ) && sleep 0'
<54.238.202.20> PUT /tmp/tmpEMtZUj TO ~/.ansible/tmp/ansible-tmp-1484621108.81-173128532792632/setup.py
fatal: [bastion]: FAILED! => {
    "failed": true,
    "msg": "failed to transfer file to ~/.ansible/tmp/ansible-tmp-1484621108.81-173128532792632/setup.py: [Errno 2] No such file or directory: '~/.ansible/tmp/ansible-tmp-1484621108.81-173128532792632/setup.py'"
}
        to retry, use: --limit @/tmp/ansible/playbooks/env_production.retry

PLAY RECAP *********************************************************************
bastion : ok=0    changed=0    unreachable=0    failed=1
```

I noticed that my config file was not loaded (because cloud-init is not running ansible from the directory containing the config file), so I've added `export ANSIBLE_CONFIG=/tmp/ansible/ansible.cfg`.

The playbook start to run, but at some point, another error happens.
```
TASK [AerisCloud.consul : Configure consul] ************************************
task path: /tmp/ansible/playbooks/roles/AerisCloud.consul/tasks/main.yml:61
<54.249.4.95> ESTABLISH LOCAL CONNECTION FOR USER: root
<54.249.4.95> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1484622713.74-232938059428255 `" && echo ansible-tmp-1484622713.74-232938059428255="` echo ~/.ansible/tmp/ansible-tmp-1484622713.74-232938059428255 `" ) && sleep 0'
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/core/files/stat.py
<54.249.4.95> EXEC /bin/sh -c '/usr/bin/python && sleep 0'
<54.249.4.95> PUT /tmp/tmpl8_J4L TO ~/.ansible/tmp/ansible-tmp-1484622713.74-232938059428255/source
fatal: [bastion001]: FAILED! => {
    "failed": true,
    "msg": "failed to transfer file to ~/.ansible/tmp/ansible-tmp-1484622713.74-232938059428255/source: [Errno 2] No such file or directory: '~/.ansible/tmp/ansible-tmp-1484622713.74-232938059428255/source'"
}
```

The role being run is available here: https://github.com/AerisCloud/ansible-consul/blob/master/tasks/main.yml#L61-L69

My environment variables:
```
HOSTNAME=bastion001-0277
DEBCONF_NONINTERACTIVE_SEEN=true
PROJECT=management
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
PWD=/
STACK_NAME=management
LANG=en_US.UTF-8
VERSION=develop
ENVIRONMENT=staging
REGION=ap-northeast-1
SHLVL=2
ROLE=bastion
TTL=300
DEBIAN_FRONTEND=noninteractive
_=/usr/bin/env
```

If I run it manually, it works fine:
```
TASK [AerisCloud.consul : Configure consul] ************************************
task path: /tmp/ansible/playbooks/roles/AerisCloud.consul/tasks/main.yml:61
<54.249.4.95> ESTABLISH LOCAL CONNECTION FOR USER: root
<54.249.4.95> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372 `" && echo ansible-tmp-1484622998.73-100967000052372="` echo ~/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372 `" ) && sleep 0'
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/core/files/stat.py
<54.249.4.95> PUT /tmp/tmp22OgUA TO /home/ubuntu/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372/stat.py
<54.249.4.95> EXEC /bin/sh -c 'chmod u+x /home/ubuntu/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372/ /home/ubuntu/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372/stat.py && sleep 0'
<54.249.4.95> EXEC /bin/sh -c '/usr/bin/python /home/ubuntu/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372/stat.py && sleep 0'
<54.249.4.95> PUT /tmp/tmpbnheuy TO /home/ubuntu/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372/source
<54.249.4.95> EXEC /bin/sh -c 'chmod u+x /home/ubuntu/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372/ /home/ubuntu/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372/source && sleep 0'
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/core/files/copy.py
<54.249.4.95> PUT /tmp/tmpyd1aV1 TO /home/ubuntu/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372/copy.py
<54.249.4.95> EXEC /bin/sh -c 'chmod u+x /home/ubuntu/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372/ /home/ubuntu/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372/copy.py && sleep 0'
<54.249.4.95> EXEC /bin/sh -c '/usr/bin/python /home/ubuntu/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372/copy.py && sleep 0'
<54.249.4.95> EXEC /bin/sh -c 'rm -f -r /home/ubuntu/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372/ > /dev/null 2>&1 && sleep 0'
NOTIFIED HANDLER Restart consul
```

Running `/usr/bin/cloud-init -d modules --mode final` manually also seems to work.
I have the same issue with ansible-playbook run with AWS user data.  

```
PLAY [Do basic startup and configuration tasks] ********************************

TASK [setup] *******************************************************************
fatal: [localhost]: FAILED! => {"failed": true, "msg": "failed to transfer file to ~/.ansible/tmp/ansible-tmp-1484654747.04-32500983939738/setup.py: [Errno 2] No such file or directory: '~/.ansible/tmp/ansible-tmp-1484654747.04-32500983939738/setup.py'"}
	to retry, use: --limit @/tmp/ansible_startup/main.retry
```
Hi, same issue.

We also have a rescue to tell us which box had failed but that too fails as hostname_value is not set.
The regression happened between the RC1 and RC2, I'm trying to track down the faulty commit.
https://github.com/ansible/ansible/compare/v2.2.1.0-0.1.rc1...v2.2.1.0-0.2.rc2
The regression has been introduced with https://github.com/ansible/ansible/commit/74bb122598cfcb050c9090684320ec2d5253ae4c
(PR: https://github.com/ansible/ansible/pull/18551, issue: https://github.com/ansible/ansible/issues/16032)

[`ShellBase.mkdtemp`](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/shell/__init__.py#L94-L128) is failing with `~` instead of `$HOME` when run from `cloud-init`.
+1. just started to have the issue yesterday with no ansible changes.
side question, how you install old ansible on ubuntu?

I get the ansible version from the list : https://launchpad.net/~ansible/+archive/ubuntu/ansible/+packages

and do 
`sudo apt-add-repository ppa:ansible/ansible`
`sudo apt-get update`
`sudo apt-get -y install ansible=2.2.0.0-1ppa~wily`

it says package not found.

I have to download the deb package and install it with dpkg.
I'm using pip to install ansible on my servers (so that it's the same whatever is the OS).
http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip
https://pypi.python.org/pypi/ansible

You are probably looking for `pip install ansible==2.2.0.0`.
```
pip install ansible 2.2.0.0
Requirement already satisfied (use --upgrade to upgrade): ansible in /usr/lib/python2.7/dist-packages
Downloading/unpacking 2.2.0.0
  Could not find any downloads that satisfy the requirement 2.2.0.0
Cleaning up...
No distributions at all found for 2.2.0.0
```
You need the `==`
As a temporary workaround, I've added the following to my `ansible.cfg` file:
```
[defaults]
remote_tmp=$HOME/.ansible/tmp
```
Ran into this exact issue, @MiLk, running a configuration script from cloud-init on localhost, fails on copying of templates with:

`fatal: [127.0.0.1]: FAILED! => {"changed": true, "failed": true, "msg": "Source /root/.ansible/tmp/ansible-tmp-1484769451.75-93960335696440/source not found"}`

Works perfectly with 2.2.0, reverting until a fix is in place.
@mike10010100 Could you add the `-vvv` flag? It doesn't seem to be the same issue.
At this point I'm not sure if this is an Ansible bug or a cloud-init one, `~/` should be a more reliable way of expressing `$HOME/`.
Please note, it also affects 2.3.

It might also be an issue with `systemd`.

I don't have much more time to spend on that issue, but I think ansible should work out of the box with the default config with cloud-init.
@MiLk the change was made to make it work 'out of the box'  with other systems, `$HOME` can be undefined (looking at variables provided above, it is also missing in cloud-init), `~/` should never be undefined. 

Before 're breaking' other systems I would like to get to the bottom of the issue, I've added more info to the error message to current devel, if you could try that and see where exactly cloud-init is choking? 

The message might be it cannot find the local file and i would like to rule that out, if so, why would it have issues with `~/` which should 'resolve' on the target, not on the cloud-init execution environment.
I agree that we should take the necessary time to fix the issue and get the greater amount of systems working out of the box. I was not suggesting that the fix should be reverted or to 're break' other systems.

I will port the changes you did in https://github.com/ansible/ansible/commit/d8cefee68c0690f98ba15294452471a222a4fcdc to the local connection plugin. Is there any other changes that would help finding the root cause of the issue?
We could add some debugging, for example checking that the in_data exists before the transfer, I suspect the problem is the 'local' resolution to find the file inside the cloud-init execution and not the remote.
I've run `2.2.1` once again with `ANSIBLE_DEBUG=1`, pipelining disabled, and improved error message in [`put_file`](https://github.com/ansible/ansible/blob/v2.2.1.0-1/lib/ansible/plugins/connection/local.py#L133)

```
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/core/system/setup.py
  3542 1484804799.05440: _low_level_execute_command(): starting
  3542 1484804799.05444: _low_level_execute_command(): executing: /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1484804799.05-86244258266200 `" && echo ansible-tmp-1484804799.05-862442
58266200="` echo ~/.ansible/tmp/ansible-tmp-1484804799.05-86244258266200 `" ) && sleep 0'
<10.12.11.114> ESTABLISH LOCAL CONNECTION FOR USER: root
  3542 1484804799.05458: in local.exec_command()
<10.12.11.114> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1484804799.05-86244258266200 `" && echo ansible-tmp-1484804799.05-86244258266200="` echo ~/.ansible/tmp/ansible-tmp-14
84804799.05-86244258266200 `" ) && sleep 0'
  3542 1484804799.05462: opening command with Popen()
  3542 1484804799.05790: done running command with Popen()
  3542 1484804799.05800: getting output with communicate()
  3542 1484804799.06103: done communicating
  3542 1484804799.06109: done with local.exec_command()
  3542 1484804799.06135: _low_level_execute_command() done: rc=0, stdout=ansible-tmp-1484804799.05-86244258266200=~/.ansible/tmp/ansible-tmp-1484804799.05-86244258266200
, stderr=
  3542 1484804799.06145: transferring module to remote ~/.ansible/tmp/ansible-tmp-1484804799.05-86244258266200/setup.py
<10.12.11.114> PUT /tmp/tmpUCffav TO ~/.ansible/tmp/ansible-tmp-1484804799.05-86244258266200/setup.py
  3542 1484804799.06746: done running TaskExecutor() for staging001..../TASK: setup
  3542 1484804799.06755: sending task result
  3542 1484804799.06796: done sending task result
  3542 1484804799.06799: WORKER PROCESS EXITING
  3429 1484804799.06894: in VariableManager get_vars()
  3429 1484804799.06997: done with get_vars()
  3429 1484804799.07010: marking staging001.... as failed
  3429 1484804799.07018: marking host staging001.... failed, current state: HOST STATE: block=0, task=0, rescue=0, always=0, role=None, run_state=ITERATING_SETUP, fail_state=FAILED_NONE, pending_setup=True, tasks child state? (None), rescue child state? (None), always child state? (None), did rescue? False, did start at task? False
  3429 1484804799.07023: ^ failed state is now: HOST STATE: block=0, task=0, rescue=0, always=0, role=None, run_state=ITERATING_COMPLETE, fail_state=FAILED_SETUP, pending_setup=True, tasks child state? (None), rescue child state? (None), always child state? (None), did rescue? False, did start at task? False
  3429 1484804799.07027: getting the next task for host staging001....
  3429 1484804799.07031: host staging001.... is done iterating, returning
fatal: [staging001....]: FAILED! => {
    "failed": true,
    "msg": "failed to transfer file /tmp/tmpUCffav to ~/.ansible/tmp/ansible-tmp-1484804799.05-86244258266200/setup.py: [Errno 2] No such file or directory: '~/.ansible/tmp/ansible-tmp-1484804799.05-86244258266200/setup.py'"
}
  3429 1484804799.07062: no more pending results, returning what we have
```
I've written the following script to test the behaviour more easily:
```python
#!/usr/bin/env python

import subprocess

cmd = """( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372 `" && echo ansible-tmp-1484622998.73-100967000052372="` echo ~/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372 `" ) && sleep 0 && echo "Check:" && (cd ~ && pwd) && sleep 0"""

p = subprocess.Popen(
            cmd,
            shell=True,
            executable="/bin/sh",
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
        )
stdout, stderr = p.communicate(None)
print(stdout)
```

And the following systemd unit file (`/etc/systemd/system/ansible.service`):
```
[Unit]
Description=Execute cloud user/final scripts
After=network-online.target cloud-config.service rc-local.service
Wants=network-online.target cloud-config.service

[Service]
Type=oneshot
ExecStart=/home/ubuntu/test.py
RemainAfterExit=yes
TimeoutSec=0
KillMode=process

# Output needs to appear in instance console output
StandardOutput=journal+console

[Install]
WantedBy=cloud-init.target
```

The result is the following:
```
Jan 19 06:02:40 staging001 test.py[5190]: ansible-tmp-1484622998.73-100967000052372=~/.ansible/tmp/ansible-tmp-1484622998.73-100967000052372
Jan 19 06:02:40 staging001 test.py[5190]: Check:
Jan 19 06:02:40 staging001 test.py[5190]: /~
```

That shows that `~` is not replaced in the shell started by `systemd`.
If I use:
```python
basetmp = "~/.ansible/tmp"

cmd = """( umask 77 && mkdir -p "` echo {0}/ansible-tmp-1484622998.73-100967000052372 `" && echo ansible-tmp-1484622998.73-100967000052372="` echo {0}/ansible-tmp-1484622998.73-100967000052372 `" ) && sleep 0""".format(os.path.expanduser(basetmp))
```

It's now working.
Fix available in https://github.com/ansible/ansible/pull/20441
the fix won't work unless both systems have synchronized user information

So this looks to me like a systemd bug, to compensate we will probably have to create specific shell plugin and add note that it is needed for cloud-init.

@MiLk have you reported it to systemd yet?
I didn't report it to systemd as I don't know if it's a bug or a design choice.
I will spend more time on it later today or next week.
please confirm that the PR above fixes the issue (I don't have access to cloud-init to test)
Thank you! I will try it.
bcoca It doesn't seem to work.
I get:
```
fatal: [bastion001-099f....]: FAILED! => {"failed": true, "msg": "failed to transfer file to /root/.ansible/tmp/ansible-tmp-1484874295.03-36722174966291/source: [Errno 2] No such file or directory: '/root/.ansible/tmp/ansible-tmp-1484874295.03-36722174966291/source'"}
```
Which is very strange.
With full info:
```
 33759 1484877394.66683: _low_level_execute_command() done: rc=0, stdout=
{"invocation": {"module_args": {"checksum_algorithm": "sha1", "mime": false, "get_checksum": true, "path": "/etc/consul/config.json", "checksum_algo": "sha1", "follow": false, "get_md5": false}}, "stat": {"ui
d": 999, "exists": true, "woth": false, "mtime": 1477616922.7014303, "inode": 399141, "isgid": false, "size": 344, "wgrp": false, "executable": false, "isuid": false, "readable": true, "isreg": true, "pw_name
": "consul", "gid": 1001, "ischr": false, "wusr": true, "writeable": true, "xoth": false, "islnk": false, "nlink": 1, "issock": false, "rgrp": true, "gr_name": "consul", "path": "/etc/consul/config.json", "xu
sr": false, "atime": 1484877149.7006958, "isdir": false, "ctime": 1477616922.8374305, "isblk": false, "xgrp": false, "dev": 2049, "roth": true, "isfifo": false, "mode": "0644", "checksum": "309c492e6f89cba5d7
18e72cde4bf5fca7ce614d", "rusr": true}, "changed": false}
, stderr=
 33759 1484877394.66707: done with _execute_module (stat, {'_ansible_version': '2.2.1.0', '_ansible_selinux_special_fs': ['fuse', 'nfs', 'vboxsf', 'ramfs'], '_ansible_no_log': False, '_ansible_module_name': '
stat', '_ansible_debug': True, '_ansible_verbosity': 3, '_ansible_syslog_facility': 'LOG_USER', '_ansible_diff': False, 'get_checksum': True, '_ansible_check_mode': False, 'checksum_algo': 'sha1', 'follow': F
alse, 'get_md5': False, 'path': u'/etc/consul/config.json'})
 33759 1484877394.66719: _low_level_execute_command(): starting
 33759 1484877394.66724: _low_level_execute_command(): executing: /bin/sh -c 'echo ~root && sleep 0'
 33759 1484877394.66808: in local.exec_command()
<54.250.143.210> EXEC /bin/sh -c 'echo ~root && sleep 0'
 33759 1484877394.66815: opening command with Popen()
 33759 1484877394.67523: done running command with Popen()
 33759 1484877394.67530: getting output with communicate()
 33759 1484877394.67785: done communicating
 33759 1484877394.67789: done with local.exec_command()
 33759 1484877394.67801: _low_level_execute_command() done: rc=0, stdout=/root
, stderr=
<54.250.143.210> PUT /tmp/tmpoERhEw TO /root/.ansible/tmp/ansible-tmp-1484877394.42-156541555057019/source
 33759 1484877394.67951: done running TaskExecutor() for bastion001-013d..../TASK: AerisCloud.consul : Configure consul
 33759 1484877394.67958: sending task result
 33759 1484877394.68327: done sending task result
 33759 1484877394.68331: WORKER PROCESS EXITING
 11336 1484877394.68449: in VariableManager get_vars()
 11336 1484877394.68725: done with get_vars()
 11336 1484877394.68740: marking bastion001-013d.... as failed
 11336 1484877394.68749: marking host bastion001-013d....failed, current state: HOST STATE: block=4, task=8, rescue=0, always=0, role=AerisCloud.consul, run_state=ITERATING_TASKS
, fail_state=FAILED_NONE, pending_setup=False, tasks child state? (None), rescue child state? (None), always child state? (None), did rescue? False, did start at task? False
 11336 1484877394.68757: ^ failed state is now: HOST STATE: block=4, task=8, rescue=0, always=0, role=AerisCloud.consul, run_state=ITERATING_COMPLETE, fail_state=FAILED_TASKS, pending_setup=False, tasks child
 state? (None), rescue child state? (None), always child state? (None), did rescue? False, did start at task? False
 11336 1484877394.68762: getting the next task for host bastion001-013d....
 11336 1484877394.69159: host bastion001-013d.... is done iterating, returning
fatal: [bastion001-013d....]: FAILED! => {
    "failed": true,
    "msg": "failed to transfer file to /root/.ansible/tmp/ansible-tmp-1484877394.42-156541555057019/source: [Errno 2] No such file or directory: '/root/.ansible/tmp/ansible-tmp-1484877394.42-156541555057019/source'"
}
 11336 1484877394.69184: no more pending results, returning what we have
```

With pipelining disabled (to make trigger the error faster):
```
<54.250.143.210> ESTABLISH LOCAL CONNECTION FOR USER: root
  2222 1484878877.28186: in local.exec_command()
<54.250.143.210> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1484878877.28-221544317678806 `" && echo ansible-tmp-1484878877.28-221544317678806="` echo ~/.ansible/tmp/ansible-tmp-1484878877.28-221544317678806 `" ) && sleep 0'
  2222 1484878877.28206: opening command with Popen()
  2222 1484878877.28455: done running command with Popen()
  2222 1484878877.28586: getting output with communicate()
  2222 1484878877.29772: done communicating
  2222 1484878877.29776: done with local.exec_command()
  2222 1484878877.29784: _low_level_execute_command() done: rc=0, stdout=ansible-tmp-1484878877.28-221544317678806=~/.ansible/tmp/ansible-tmp-1484878877.28-221544317678806
, stderr=
  2222 1484878877.29794: transferring module to remote ~/.ansible/tmp/ansible-tmp-1484878877.28-221544317678806/setup.py
  2222 1484878877.29800: _low_level_execute_command(): starting
  2222 1484878877.29806: _low_level_execute_command(): executing: /bin/sh -c 'echo ~root && sleep 0'
  2222 1484878877.29816: in local.exec_command()
<54.250.143.210> EXEC /bin/sh -c 'echo ~root && sleep 0'
  2222 1484878877.29821: opening command with Popen()
  2222 1484878877.30102: done running command with Popen()
  2222 1484878877.30110: getting output with communicate()
  2222 1484878877.30860: done communicating
  2222 1484878877.30865: done with local.exec_command()
  2222 1484878877.30871: _low_level_execute_command() done: rc=0, stdout=/root
, stderr=
<54.250.143.210> PUT /tmp/tmph_WC5b TO /root/.ansible/tmp/ansible-tmp-1484878877.28-221544317678806/setup.py
  2222 1484878877.31101: done running TaskExecutor() for bastion001-013d..../TASK: setup
  2222 1484878877.31132: sending task result
  2222 1484878877.31510: done sending task result
  2222 1484878877.31514: WORKER PROCESS EXITING
  2210 1484878877.31664: in VariableManager get_vars()
  2210 1484878877.31749: done with get_vars()
  2210 1484878877.31760: marking bastion001-013d.... as failed
  2210 1484878877.31766: marking host bastion001-013d.... failed, current state: HOST STATE: block=0, task=0, rescue=0, always=0, role=None, run_state=ITERATING_SETUP, fail_state=FAILED_NONE, pending_setup=True, tasks child state? (None), rescue child state? (None), always child state? (None), did rescue? False, did start at task? False
  2210 1484878877.31769: ^ failed state is now: HOST STATE: block=0, task=0, rescue=0, always=0, role=None, run_state=ITERATING_COMPLETE, fail_state=FAILED_SETUP, pending_setup=True, tasks child state? (None), rescue child state? (None), always child state? (None), did rescue? False, did start at task? False
  2210 1484878877.31773: getting the next task for host bastion001-013d....
  2210 1484878877.31776: host bastion001-013d.... is done iterating, returning
fatal: [bastion001-013d....]: FAILED! => {
    "failed": true,
    "msg": "failed to transfer file to /root/.ansible/tmp/ansible-tmp-1484878877.28-221544317678806/setup.py: [Errno 2] No such file or directory: '/root/.ansible/tmp/ansible-tmp-1484878877.28-221544317678806/setup.py'"
}
  2210 1484878877.31794: no more pending results, returning what we have
```

The shell command is still containing `~` and it's not expanded.
well .. we can rule out the 'non translation of ~/' 
can you check that the directory exists? are we failing to create it? 
/root/.ansible/tmp/ansible-tmp-1484877394.42-156541555057019/
The `/root/.ansible/tmp/` folder contains only one cache file generated by my inventory script.
There is no `ansible-tmp-*` directory.
on the remote machine? try enabling keep remote files
I've added an `os.listdir()` just after https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/action/__init__.py#L222

I now get:
```
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/core/system/setup.py
  3502 1484880378.07451: _low_level_execute_command(): starting
  3502 1484880378.07466: _low_level_execute_command(): executing: /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1484880378.07-84315295215642 `" && echo ansible-tmp-1484880378.07-843152
95215642="` echo ~/.ansible/tmp/ansible-tmp-1484880378.07-84315295215642 `" ) && sleep 0'
<54.250.143.210> ESTABLISH LOCAL CONNECTION FOR USER: root
  3502 1484880378.07484: in local.exec_command()
<54.250.143.210> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1484880378.07-84315295215642 `" && echo ansible-tmp-1484880378.07-84315295215642="` echo ~/.ansible/tmp/ansible-tmp-
1484880378.07-84315295215642 `" ) && sleep 0'
  3502 1484880378.07492: opening command with Popen()
  3502 1484880378.08161: done running command with Popen()
  3502 1484880378.08171: getting output with communicate()
  3502 1484880378.08852: done communicating
  3502 1484880378.08856: done with local.exec_command()
  3502 1484880378.08865: _low_level_execute_command() done: rc=0, stdout=ansible-tmp-1484880378.07-84315295215642=~/.ansible/tmp/ansible-tmp-1484880378.07-84315295215642
, stderr=

DEBUG make_tmp_path ['profile_iam', 'ansible-local-3460iUeKpz']

  3502 1484880378.08880: transferring module to remote ~/.ansible/tmp/ansible-tmp-1484880378.07-84315295215642/setup.py
  3502 1484880378.08885: _low_level_execute_command(): starting
  3502 1484880378.08890: _low_level_execute_command(): executing: /bin/sh -c 'echo ~root && sleep 0'
  3502 1484880378.08898: in local.exec_command()
<54.250.143.210> EXEC /bin/sh -c 'echo ~root && sleep 0'
  3502 1484880378.08903: opening command with Popen()
  3502 1484880378.09375: done running command with Popen()
  3502 1484880378.09383: getting output with communicate()
  3502 1484880378.09704: done communicating
  3502 1484880378.09708: done with local.exec_command()
  3502 1484880378.09715: _low_level_execute_command() done: rc=0, stdout=/root
, stderr=
<54.250.143.210> PUT /tmp/tmpqSh15U TO /root/.ansible/tmp/ansible-tmp-1484880378.07-84315295215642/setup.py
  3502 1484880378.10034: done running TaskExecutor() for bastion001-013d..../TASK: setup
  3502 1484880378.10042: sending task result
  3502 1484880378.10067: done sending task result
  3502 1484880378.10070: WORKER PROCESS EXITING
  3460 1484880378.10305: in VariableManager get_vars()
  3460 1484880378.10823: done with get_vars()
  3460 1484880378.10837: marking bastion001-013d.... as failed
```

The `DEBUG make_tmp_path` line show that only `ansible-local-3460iUeKpz` is created.

The machine is provisioning itself using the local connection plugin.
If I run a sh script from systemd:
```sh
#!/bin/sh

( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1484878877.28-221544317678806 `" && echo ansible-tmp-1484878877.28-221544317678806="` echo ~/.ansible/tmp/ansible-tmp-1484878877.28-221544317678806 `" ) && sleep 0

echo "/"
ls -la /
echo "/~"
ls -la /~
echo "~"
ls -la ~
```

I get the following:
```
Jan 20 02:52:45 bastion001-013d test.sh[3794]: ansible-tmp-1484878877.28-221544317678806=~/.ansible/tmp/ansible-tmp-1484878877.28-221544317678806
Jan 20 02:52:45 bastion001-013d test.sh[3794]: /
Jan 20 02:52:45 bastion001-013d test.sh[3794]: total 96
Jan 20 02:52:45 bastion001-013d test.sh[3794]: drwx------   3 root root  4096 Jan 20 02:52 ~
Jan 20 02:52:45 bastion001-013d test.sh[3794]: drwxr-xr-x  23 root root  4096 Jan 20 02:52 .
Jan 20 02:52:45 bastion001-013d test.sh[3794]: drwxr-xr-x  23 root root  4096 Jan 20 02:52 ..
...
Jan 20 02:52:45 bastion001-013d test.sh[3794]: /~
Jan 20 02:52:45 bastion001-013d test.sh[3794]: total 12
Jan 20 02:52:45 bastion001-013d test.sh[3794]: drwx------  3 root root 4096 Jan 20 02:52 .
Jan 20 02:52:45 bastion001-013d test.sh[3794]: drwxr-xr-x 23 root root 4096 Jan 20 02:52 ..
Jan 20 02:52:45 bastion001-013d test.sh[3794]: drwx------  3 root root 4096 Jan 20 02:52 .ansible
Jan 20 02:52:45 bastion001-013d test.sh[3794]: ~
Jan 20 02:52:45 bastion001-013d test.sh[3794]: total 12
Jan 20 02:52:45 bastion001-013d test.sh[3794]: drwx------  3 root root 4096 Jan 20 02:52 .
Jan 20 02:52:45 bastion001-013d test.sh[3794]: drwxr-xr-x 23 root root 4096 Jan 20 02:52 ..
Jan 20 02:52:45 bastion001-013d test.sh[3794]: drwx------  3 root root 4096 Jan 20 02:52 .ansible
```

Which shows that the issue is not coming from python, but is related to systemd/dash.
https://github.com/koalaman/shellcheck/issues/195
> When GNU make tries to find executables, it fails to find them, because it likes to fork out shells with dash instead of bash, and dash does not translate tildes.

That might be the issue, as dash is the default shell on ubuntu.
that is wrong, dash DOES interpret ~ .. its part of the POSIX standard, no clue what make has to do with this
If I run 2.2.1 with the following config, it works:
```
[defaults]
executable = /bin/bash
[ssh_connection]
pipelining=True
```
```
/bin/dash
$ cd /
$ pwd
/
$ cd ~/
$pwd
/home/bcoca
```
You are right.
I'll try to confirm it's working with bash. And try to check why dash is not doing substitution when ran by systemd.

Resolving patch:
diff --git a/lib/ansible/plugins/action/__init__.py b/lib/ansible/plugins/action/__init__.py
--- a/lib/ansible/plugins/action/__init__.py
+++ b/lib/ansible/plugins/action/__init__.py
@@ -218,7 +218,12 @@ def _make_tmp_path(self, remote_user):
 
         tmp_mode = 0o700
 
-        cmd = self._connection._shell.mkdtemp(basefile, use_system_tmp, tmp_mode)
+        if use_system_tmp:
+            tmpdir = None
+        else:
+            tmpdir =  self._remote_expand_user(C.DEFAULT_REMOTE_TMP)
+
+        cmd = self._connection._shell.mkdtemp(basefile, use_system_tmp, tmp_mode, tmpdir)
         result = self._low_level_execute_command(cmd, sudoable=False)
 
         # error handling on this seems a little aggressive?
diff --git a/lib/ansible/plugins/shell/__init__.py b/lib/ansible/plugins/shell/__init__.py
--- a/lib/ansible/plugins/shell/__init__.py
+++ b/lib/ansible/plugins/shell/__init__.py
@@ -91,7 +91,7 @@ def exists(self, path):
         cmd = ['test', '-e', shlex_quote(path)]
         return ' '.join(cmd)
 
-    def mkdtemp(self, basefile=None, system=False, mode=None):
+    def mkdtemp(self, basefile=None, system=False, mode=None, tmpdir=None):
         if not basefile:
             basefile = 'ansible-tmp-%s-%s' % (time.time(), random.randint(0, 2**48))
 
@@ -107,13 +107,17 @@ def mkdtemp(self, basefile=None, system=False, mode=None):
         # to somewhere in or below /var/tmp and if so use /var/tmp.  If
         # anything else we use /tmp (because /tmp is specified by POSIX nad
         # /var/tmp is not).
+
         if system:
             if C.DEFAULT_REMOTE_TMP.startswith('/var/tmp'):
                 basetmpdir = '/var/tmp'
             else:
                 basetmpdir = '/tmp'
-        else:
+        elif tmpdir is None:
             basetmpdir = C.DEFAULT_REMOTE_TMP
+        else:
+            basetmpdir = tmpdir
+
         basetmp = self.join_path(basetmpdir, basefile)
 
         cmd = 'mkdir -p %s echo %s %s' % (self._SHELL_SUB_LEFT, basetmp, self._SHELL_SUB_RIGHT)


Test patch: