Repository: ansible/ansible

Issue:
ERROR! failed to transfer file to       ^/setup:
I just installed Ansible devel branch like so:  `pip2 install git+https://github.com/ansible/ansible.git@devel --upgrade`. When I use any module besides `raw`, I get an error when it tries to PUT the file to the host. It seems the file name is messed up.

Ansible 1.9 works fine. I'm running OpenBSD 5.8-current with Python 2.7.10 and Fish Shell.

```
~ (master|✚29…) ❯❯❯ ansible -i ~/src/config/hosts -m setup pi.ggr.com -vvvv
No config file found; using defaults
Loaded callback minimal of type stdout, v2.0
<pi.ggr.com> ESTABLISH SSH CONNECTION FOR USER: None
<pi.ggr.com> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/scott/.ansible/cp/ansible-ssh-%h-%p-%r -tt pi.ggr.com (umask 22 && mkdir -p "`echo $HOME/.ansible/tmp/ansible-tmp-1447809899.71-249179464121878`" && echo "`echo $HOME/.ansible/tmp/ansible-tmp-1447809899.71-249179464121878`")
<pi.ggr.com> PUT /tmp/tmp1IkHgE TO       ^/setup
<pi.ggr.com> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/scott/.ansible/cp/ansible-ssh-%h-%p-%r [pi.ggr.com]
pi.ggr.com | FAILED! => {
    "failed": true, 
    "msg": "ERROR! failed to transfer file to       ^/setup:\nsftp> put /tmp/tmp1IkHgE '      ^/setup'\n\nOpenSSH_7.1, LibreSSL 2.3.2\r\ndebug1: Reading configuration data /home/scott/.ssh/config\r\ndebug1: /home/scott/.ssh/config line 3: Skipping Host block because of negated match for pi.ggr.com\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 1967\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension \"posix-rename@openssh.com\" revision 1\r\ndebug2: Server supports extension \"statvfs@openssh.com\" revision 2\r\ndebug2: Server supports extension \"fstatvfs@openssh.com\" revision 2\r\ndebug2: Server supports extension \"hardlink@openssh.com\" revision 1\r\ndebug2: Server supports extension \"fsync@openssh.com\" revision 1\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/scott size 0\r\ndebug3: Looking up /tmp/tmp1IkHgE\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn't stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/scott/      ^/setup\r\nremote open(\"/home/scott/      ^/setup\"): No such file or directory\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n"
}
```



Discussion:
What does you ansible.cfg look like?

I don't have a config file, here's the first line of the output:

~ (master|✚29…) ❯❯❯ ansible -i ~/src/config/hosts -m setup pi.ggr.com -vvvv
No config file found; using defaults

@bonds seems like your destination system doesn't like sftp. You can change to scp in your ansible.cfg to try and work around this, otherwise you may need to debug the sftp error manually as not much info is showing up in the output above.

http://docs.ansible.com/ansible/intro_configuration.html#scp-if-ssh

Correction, doesn't like sftp under 2.0.

cc @amenonsen 

I'm thinking it might be elsewhere, but I don't have too much proof yet.  We see the following in the output:

```
PUT /tmp/tmp1IkHgE TO       ^/setup
```

That spacing and the `^` are what concern me.  It looks like we are somehow building a path that starts with 6 spaces and a `^`.

Eventually leading to SSH trying to `open` the following path:

```
/home/scott/      ^/setup
```

And then it fails.

@sivel ahh, I missed that. Guessing this has to do with some difference in fish then. I have an OpenBSD system at home I can install that on and test with. 

@bonds are you specifying `ansible_shell_type=fish` in your inventory? Not sure if that'll help in this situation, but it's worth testing. Looking at the fish shell plugin, it mainly only changes the way environment variables are specified.

@jimi-c  tried scp, same error:

```
~ (master|✚29…) ❯❯❯ mv .ansible.cfg.old .ansible.cfg
~ (master|✚29…) ❯❯❯ cat .ansible.cfg                Tue Nov 24 18:26:13 PST 2015
[ssh_connection]
scp_if_ssh=True
```

then

```
~/s/config (master|✚23…) ❯❯❯ ansible -i ~/src/config/hosts -m setup pi.ggr.com
pi.ggr.com | FAILED! => {
    "failed": true, 
    "msg": "ERROR! failed to transfer file to       ^/setup:\n\nscp:       ^/setup: No such file or directory\n"
}
```

tried `ansible_shell_type=fish`

```
~/s/config (master|✚23…) ❯❯❯ cat hosts 
[openbsd]
pi.ggr.com

[openbsd:vars]
ansible_shell_type=fish
```

unfortunately, it didn't help, got the same error

I switched my shell to Korn Shell and reran the command and I do NOT get the error, it works fine. So, something to do with Fish Shell 2.2 and Ansible 2.0.

```
~ (master|✚29…) ❯❯❯ fish --version                  Tue Nov 24 18:26:18 PST 2015
fish, version 2.2.0-414-g3b53abe
```

Not to beat a dead horse but I get the same issue when my default shell is fish on remote hosts but not when I configure my localhost with fish as my default shell. I also tried using configuring my remote hosts with `ansible_shell_type=fish` which didn't work.

The reason we are getting the error: 

```
"msg": "ERROR! failed to transfer file to       ^/setup:\n\nscp:       ^/setup: No such file or directory\n"
```

seems to be because ansible is trying to create the `setup` directory using non fish compatible commands during the connection via the initial ssh exec:

```
(umask 22 && mkdir -p "`echo $HOME/.ansible/tmp/ansible-tmp-1448742456.96-92512334359907`" && echo "`echo $HOME/.ansible/tmp/ansible-tmp-1448742456.96-92512334359907`")
```

in fish, all the `&&`'s would have to be `; and`'s...

so setup is a file, which should be transferred to `$HOME/.ansible/tmp/ansible-tmp-1448742456.96-92512334359907`, but the command seems to 'not fail' yet does not execute the 
`echo`, which gives back the actual directory to use. Since the return is 'empty' you get `/setup` instead of `$HOME/.ansible/tmp/ansible-tmp-1448742456.96-92512334359907/setup`.

This used to work for fish users in 1.x, I'm not sure what we changed to break it, but I'll look into it on Monday.


Resolving patch:
diff --git a/lib/ansible/plugins/shell/fish.py b/lib/ansible/plugins/shell/fish.py
--- a/lib/ansible/plugins/shell/fish.py
+++ b/lib/ansible/plugins/shell/fish.py
@@ -21,5 +21,12 @@
 
 class ShellModule(ShModule):
 
+    _SHELL_AND = '; and'
+    _SHELL_OR = '; or'
+    _SHELL_SUB_LEFT = '('
+    _SHELL_SUB_RIGHT = ')'
+    _SHELL_GROUP_LEFT = ''
+    _SHELL_GROUP_RIGHT = ''
+
     def env_prefix(self, **kwargs):
         return 'env %s' % super(ShellModule, self).env_prefix(**kwargs)
diff --git a/lib/ansible/plugins/shell/sh.py b/lib/ansible/plugins/shell/sh.py
--- a/lib/ansible/plugins/shell/sh.py
+++ b/lib/ansible/plugins/shell/sh.py
@@ -33,6 +33,12 @@ class ShellModule(object):
     # How to end lines in a python script one-liner
     _SHELL_EMBEDDED_PY_EOL = '\n'
     _SHELL_REDIRECT_ALLNULL = '> /dev/null 2>&1'
+    _SHELL_AND = '&&'
+    _SHELL_OR = '||'
+    _SHELL_SUB_LEFT = '"$('
+    _SHELL_SUB_RIGHT = ')"'
+    _SHELL_GROUP_LEFT = '('
+    _SHELL_GROUP_RIGHT = ')'
 
     def env_prefix(self, **kwargs):
         '''Build command prefix with environment variables.'''
@@ -71,14 +77,14 @@ def mkdtemp(self, basefile=None, system=False, mode=None):
         basetmp = self.join_path(C.DEFAULT_REMOTE_TMP, basefile)
         if system and (basetmp.startswith('$HOME') or basetmp.startswith('~/')):
             basetmp = self.join_path('/tmp', basefile)
-        cmd = 'mkdir -p "`echo %s`"' % basetmp
-        cmd += ' && echo "`echo %s`"' % basetmp
+        cmd = 'mkdir -p %s echo %s %s' % (self._SHELL_SUB_LEFT, basetmp, self._SHELL_SUB_RIGHT)
+        cmd += ' %s echo %s echo %s %s' % (self._SHELL_AND, self._SHELL_SUB_LEFT, basetmp, self._SHELL_SUB_RIGHT)
 
         # change the umask in a subshell to achieve the desired mode
         # also for directories created with `mkdir -p`
         if mode:
             tmp_umask = 0o777 & ~mode
-            cmd = '(umask %o && %s)' % (tmp_umask, cmd)
+            cmd = '%s umask %o %s %s %s' % (self._SHELL_GROUP_LEFT, tmp_umask, self._SHELL_AND, cmd, self._SHELL_GROUP_RIGHT)
 
         return cmd
 
@@ -128,14 +134,14 @@ def checksum(self, path, python_interp):
         # used by a variety of shells on the remote host to invoke a python
         # "one-liner".
         shell_escaped_path = pipes.quote(path)
-        test = "rc=flag; [ -r %(p)s ] || rc=2; [ -f %(p)s ] || rc=1; [ -d %(p)s ] && rc=3; %(i)s -V 2>/dev/null || rc=4; [ x\"$rc\" != \"xflag\" ] && echo \"${rc}  \"%(p)s && exit 0" % dict(p=shell_escaped_path, i=python_interp)
+        test = "rc=flag; [ -r %(p)s ] %(shell_or)s rc=2; [ -f %(p)s ] %(shell_or)s rc=1; [ -d %(p)s ] %(shell_and)s rc=3; %(i)s -V 2>/dev/null %(shell_or)s rc=4; [ x\"$rc\" != \"xflag\" ] %(shell_and)s echo \"${rc}  \"%(p)s %(shell_and)s exit 0" % dict(p=shell_escaped_path, i=python_interp, shell_and=self._SHELL_AND, shell_or=self._SHELL_OR)
         csums = [
             "({0} -c 'import hashlib; BLOCKSIZE = 65536; hasher = hashlib.sha1();{2}afile = open(\"'{1}'\", \"rb\"){2}buf = afile.read(BLOCKSIZE){2}while len(buf) > 0:{2}\thasher.update(buf){2}\tbuf = afile.read(BLOCKSIZE){2}afile.close(){2}print(hasher.hexdigest())' 2>/dev/null)".format(python_interp, shell_escaped_path, self._SHELL_EMBEDDED_PY_EOL),      # Python > 2.4 (including python3)
             "({0} -c 'import sha; BLOCKSIZE = 65536; hasher = sha.sha();{2}afile = open(\"'{1}'\", \"rb\"){2}buf = afile.read(BLOCKSIZE){2}while len(buf) > 0:{2}\thasher.update(buf){2}\tbuf = afile.read(BLOCKSIZE){2}afile.close(){2}print(hasher.hexdigest())' 2>/dev/null)".format(python_interp, shell_escaped_path, self._SHELL_EMBEDDED_PY_EOL),      # Python == 2.4
         ]
 
-        cmd = " || ".join(csums)
-        cmd = "%s; %s || (echo \'0  \'%s)" % (test, cmd, shell_escaped_path)
+        cmd = (" %s " % self._SHELL_OR).join(csums)
+        cmd = "%s; %s %s (echo \'0  \'%s)" % (test, cmd, self._SHELL_OR, shell_escaped_path)
         return cmd
 
     def build_module_command(self, env_string, shebang, cmd, arg_path=None, rm_tmp=None):


Test patch: