Repository: ansible/ansible

Issue:
unarchive fails on tar files containing filenames resembling environment variables.
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
unarchive module
##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
(tried with version from git)
```
ansible 2.3.0
  config file =
  configured module search path = Default w/o overrides
```
(tried also with --- also not working)
```
ansible 2.2.1.0
  config file =
  configured module search path = Default w/o overrides
```
(tried with --- it works)
```
ansible 2.0.0.1
  config file =
  configured module search path = Default w/o overrides
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
Run all the tests on localhost with a virtualenv.

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
Tests executed on Ubuntu 16.04 on localhost.

##### SUMMARY
<!--- Explain the problem briefly -->
unarchive fails on tar files containing filenames resembling environment variables

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
1. Create a sample tar file:
```shell
mkdir -p x/\$USER/y
touch x/\$USER/y/z
tar -cf x.tar x
````
(note how the $ is escaped)
2. Run the ansible command:
```shell
ansible -m unarchive -a 'src=x.tar dest=/tmp/' localhost
```
<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
This should work and the tar file untarred to /tmp:
```shell
$ find /tmp/x
/tmp/x
/tmp/x/$USER
/tmp/x/$USER/y
/tmp/x/$USER/y/z
```
The output of the ansible 2.0.0.1 is
```
localhost | SUCCESS => {
    "changed": true,
    "check_results": {
        "cmd": "/bin/tar -C \"/tmp/\" --diff -zf \"/home/jsantand/.ansible/tmp/ansible-tmp-1487785544.15-256800309243600/source\"",
        "err": "/bin/tar: x: Warning: Cannot stat: No such file or directory\n/bin/tar: x/$USER: Warning: Cannot stat: No such file or directory\n/bin/tar: x/$USER/y: Warning: Cannot stat: No such file or directory\n/bin/tar: x/$USER/y/z: Warning: Cannot stat: No such file or directory\n",
        "out": "",
        "rc": 1,
        "unarchived": false
    },
    "dest": "/tmp/",
    "extract_results": {
        "cmd": "/bin/tar -xzf \"/home/jsantand/.ansible/tmp/ansible-tmp-1487785544.15-256800309243600/source\"",
        "err": "",
        "out": "",
        "rc": 0
    },
    "gid": 0,
    "group": "root",
    "handler": "TgzArchive",
    "mode": "01777",
    "owner": "root",
    "size": 4096,
    "src": "/home/jsantand/.ansible/tmp/ansible-tmp-1487785544.15-256800309243600/source",
    "state": "directory",
    "uid": 0
}
```
... although there're some warnings, the result is SUCCESS.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
The command fails on the current git version and on 2.2.1,
The command complains about:
```
"Unexpected error when accessing exploded file: [Errno 2] No such file or directory: '/tmp/x/jsantand/'",
```
Note that $USER got replaced somehow by the current username.
The tar file got successfully untarred.
```
/tmp/x
/tmp/x/$USER
/tmp/x/$USER/y
/tmp/x/$USER/y/z

```
<!--- Paste verbatim command output between quotes below -->
The output of the 2.3.0 with -vvvv
```
(t3) jsantand@ESJSANTAND-C2-VM:~$ ansible -vvvv -m unarchive -a 'src=x.tar dest=/tmp/' localhost
No config file found; using defaults
 [WARNING]: Host file not found: /etc/ansible/hosts

 [WARNING]: provided hosts list is empty, only localhost is available

Loading callback plugin minimal of type stdout, v2.0 from /home/jsantand/t3/local/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc
META: ran handlers
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: jsantand
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.29-184135596704384 `" && echo ansible-tmp-1487785925.29-184135596704384="` echo /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.29-184135596704384 `" ) && sleep 0'
Using module file /home/jsantand/t3/local/lib/python2.7/site-packages/ansible/modules/files/stat.py
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.47-103603087530334 `" && echo ansible-tmp-1487785925.47-103603087530334="` echo /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.47-103603087530334 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmpD0Jptv TO /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.47-103603087530334/stat.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.47-103603087530334/ /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.47-103603087530334/stat.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/home/jsantand/t3/bin/python /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.47-103603087530334/stat.py; rm -rf "/home/jsantand/.ansible/tmp/ansible-tmp-1487785925.47-103603087530334/" > /dev/null 2>&1 && sleep 0'
<127.0.0.1> PUT /home/jsantand/x.tar TO /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.29-184135596704384/source
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.29-184135596704384/ /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.29-184135596704384/source && sleep 0'
Using module file /home/jsantand/t3/local/lib/python2.7/site-packages/ansible/modules/files/unarchive.py
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.94-47412414857617 `" && echo ansible-tmp-1487785925.94-47412414857617="` echo /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.94-47412414857617 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmppSdhQC TO /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.94-47412414857617/unarchive.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.94-47412414857617/ /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.94-47412414857617/unarchive.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/home/jsantand/t3/bin/python /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.94-47412414857617/unarchive.py; rm -rf "/home/jsantand/.ansible/tmp/ansible-tmp-1487785925.94-47412414857617/" > /dev/null 2>&1 && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/jsantand/.ansible/tmp/ansible-tmp-1487785925.29-184135596704384/ > /dev/null 2>&1 && sleep 0'
localhost | FAILED! => {
    "changed": true,
    "dest": "/tmp/",
    "extract_results": {
        "cmd": [
            "/bin/tar",
            "--extract",
            "-C",
            "/tmp/",
            "-z",
            "-f",
            "/home/jsantand/.ansible/tmp/ansible-tmp-1487785925.29-184135596704384/source"
        ],
        "err": "",
        "out": "",
        "rc": 0
    },
    "failed": true,
    "gid": 0,
    "group": "root",
    "handler": "TgzArchive",
    "invocation": {
        "module_args": {
            "attributes": null,
            "backup": null,
            "content": null,
            "copy": true,
            "creates": null,
            "delimiter": null,
            "dest": "/tmp/",
            "directory_mode": null,
            "exclude": [],
            "extra_opts": [],
            "follow": false,
            "force": null,
            "group": null,
            "keep_newer": false,
            "list_files": false,
            "mode": null,
            "original_basename": "x.tar",
            "owner": null,
            "regexp": null,
            "remote_src": false,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "src": "/home/jsantand/.ansible/tmp/ansible-tmp-1487785925.29-184135596704384/source",
            "unsafe_writes": null,
            "validate_certs": true
        }
    },
    "mode": "01777",
    "msg": "Unexpected error when accessing exploded file: [Errno 2] No such file or directory: '/tmp/x/jsantand/'",
    "owner": "root",
    "size": 4096,
    "src": "/home/jsantand/.ansible/tmp/ansible-tmp-1487785925.29-184135596704384/source",
    "state": "directory",
    "uid": 0
}
```



Discussion:
cc @dagwieers @pileofrogs
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Some additional information, I've further debug this to the call of:
```python
            try:
                res_args['changed'] = module.set_fs_attributes_if_different(file_args, res_args['changed'])
            except (IOError, OSError):
                e = get_exception()
                module.fail_json(msg="Unexpected error when accessing exploded file: %s" % str(e), **res_args)
```
On `set_fs_attributes_if_different()` it fails on the `set_mode_if_different` 

that goes:
```python
    def set_mode_if_different(self, path, mode, changed, diff=None):
        b_path = to_bytes(path, errors='surrogate_or_strict')
        b_path = os.path.expanduser(os.path.expandvars(b_path))
        path_stat = os.lstat(b_path)
```
It is the `os.path.expandvars()` the one transforming the `/tmp/x/$USER/` into `/tmp/x/jsantand/`

I'm a bit of at a loss here on how to proceed (otherwise I would have submitted a pull request by now :)). 
On one hand,it strikes me as odd to do this expansion (both of variables and user) when it is not done for any of the other checks in `set_fs_attributes_if_different()`, so my initial approach would be to assume that the path is already expanded. However I cannot calibrate what other parts of the code are using this function.

A second approach would be to add an additional optional parameter to avoid expansion, defaulting to *expanding* and making the call from `set_fs_attributes_if_different()` to request no expansion.

@j-santander Ah, that's great. I had been looking into this, but did not understand what was going on (because none of my code would be actively expanding variables in filenames). But now it makes sense.

The behaviour of set_mode_if_different() indeed seems to be wrong (or at least optional).
PS The intention always has been to make a pure-python unarchive implementation, which would not be using set_mode_if_different() but simply would perform it inline as part of the extraction.
@dagwieers ... so, what's next?
with some guidance I wouldn't mind contributing with a fix.
@j-santander I would change the default behaviour and check if any integration test is complaining. If not, I would make it optional but disable expanding by default.

Resolving patch:
diff --git a/lib/ansible/module_utils/basic.py b/lib/ansible/module_utils/basic.py
--- a/lib/ansible/module_utils/basic.py
+++ b/lib/ansible/module_utils/basic.py
@@ -902,10 +902,11 @@ def selinux_context(self, path):
         context = ret[1].split(':', 3)
         return context
 
-    def user_and_group(self, filename):
-        filename = os.path.expanduser(os.path.expandvars(filename))
-        b_filename = to_bytes(filename, errors='surrogate_or_strict')
-        st = os.lstat(b_filename)
+    def user_and_group(self, path, expand=True):
+        b_path = to_bytes(path, errors='surrogate_then_strict')
+        if expand:
+            b_path = os.path.expanduser(os.path.expandvars(b_path))
+        st = os.lstat(b_path)
         uid = st.st_uid
         gid = st.st_gid
         return (uid, gid)
@@ -987,12 +988,14 @@ def set_context_if_different(self, path, context, changed, diff=None):
             changed = True
         return changed
 
-    def set_owner_if_different(self, path, owner, changed, diff=None):
-        path = os.path.expanduser(os.path.expandvars(path))
-        b_path = to_bytes(path, errors='surrogate_or_strict')
+    def set_owner_if_different(self, path, owner, changed, diff=None, expand=True):
+        b_path = to_bytes(path, errors='surrogate_then_strict')
+        if expand:
+            b_path = os.path.expanduser(os.path.expandvars(b_path))
+        path = to_text(b_path, errors='surrogate_then_strict')
         if owner is None:
             return changed
-        orig_uid, orig_gid = self.user_and_group(path)
+        orig_uid, orig_gid = self.user_and_group(path, expand)
         try:
             uid = int(owner)
         except ValueError:
@@ -1019,12 +1022,14 @@ def set_owner_if_different(self, path, owner, changed, diff=None):
             changed = True
         return changed
 
-    def set_group_if_different(self, path, group, changed, diff=None):
-        path = os.path.expanduser(os.path.expandvars(path))
-        b_path = to_bytes(path, errors='surrogate_or_strict')
+    def set_group_if_different(self, path, group, changed, diff=None, expand=True):
+        b_path = to_bytes(path, errors='surrogate_then_strict')
+        if expand:
+            b_path = os.path.expanduser(os.path.expandvars(b_path))
+        path = to_text(b_path, errors='surrogate_then_strict')
         if group is None:
             return changed
-        orig_uid, orig_gid = self.user_and_group(b_path)
+        orig_uid, orig_gid = self.user_and_group(b_path, expand)
         try:
             gid = int(group)
         except ValueError:
@@ -1051,9 +1056,11 @@ def set_group_if_different(self, path, group, changed, diff=None):
             changed = True
         return changed
 
-    def set_mode_if_different(self, path, mode, changed, diff=None):
-        b_path = to_bytes(path, errors='surrogate_or_strict')
-        b_path = os.path.expanduser(os.path.expandvars(b_path))
+    def set_mode_if_different(self, path, mode, changed, diff=None, expand=True):
+        b_path = to_bytes(path, errors='surrogate_then_strict')
+        if expand:
+            b_path = os.path.expanduser(os.path.expandvars(b_path))
+        path = to_text(b_path, errors='surrogate_then_strict')
         path_stat = os.lstat(b_path)
 
         if mode is None:
@@ -1125,14 +1132,15 @@ def set_mode_if_different(self, path, mode, changed, diff=None):
                 changed = True
         return changed
 
-    def set_attributes_if_different(self, path, attributes, changed, diff=None):
+    def set_attributes_if_different(self, path, attributes, changed, diff=None, expand=True):
 
         if attributes is None:
             return changed
 
-        b_path = to_bytes(path, errors='surrogate_or_strict')
-        b_path = os.path.expanduser(os.path.expandvars(b_path))
-        existing = self.get_file_attributes(b_path)
+        b_path = to_bytes(path, errors='surrogate_then_strict')
+        if expand:
+            b_path = os.path.expanduser(os.path.expandvars(b_path))
+        path = to_text(b_path, errors='surrogate_then_strict')
 
         if existing.get('attr_flags','') != attributes:
             attrcmd = self.get_bin_path('chattr')
@@ -1273,30 +1281,30 @@ def _get_octal_mode_from_symbolic_perms(self, path_stat, user, perms):
         or_reduce = lambda mode, perm: mode | user_perms_to_modes[user][perm]
         return reduce(or_reduce, perms, 0)
 
-    def set_fs_attributes_if_different(self, file_args, changed, diff=None):
+    def set_fs_attributes_if_different(self, file_args, changed, diff=None, expand=True):
         # set modes owners and context as needed
         changed = self.set_context_if_different(
             file_args['path'], file_args['secontext'], changed, diff
         )
         changed = self.set_owner_if_different(
-            file_args['path'], file_args['owner'], changed, diff
+            file_args['path'], file_args['owner'], changed, diff, expand
         )
         changed = self.set_group_if_different(
-            file_args['path'], file_args['group'], changed, diff
+            file_args['path'], file_args['group'], changed, diff, expand
         )
         changed = self.set_mode_if_different(
-            file_args['path'], file_args['mode'], changed, diff
+            file_args['path'], file_args['mode'], changed, diff, expand
         )
         changed = self.set_attributes_if_different(
-            file_args['path'], file_args['attributes'], changed, diff
+            file_args['path'], file_args['attributes'], changed, diff, expand
         )
         return changed
 
-    def set_directory_attributes_if_different(self, file_args, changed, diff=None):
-        return self.set_fs_attributes_if_different(file_args, changed, diff)
+    def set_directory_attributes_if_different(self, file_args, changed, diff=None, expand=True):
+        return self.set_fs_attributes_if_different(file_args, changed, diff, expand)
 
-    def set_file_attributes_if_different(self, file_args, changed, diff=None):
-        return self.set_fs_attributes_if_different(file_args, changed, diff)
+    def set_file_attributes_if_different(self, file_args, changed, diff=None, expand=True):
+        return self.set_fs_attributes_if_different(file_args, changed, diff, expand)
 
     def add_path_info(self, kwargs):
         '''
diff --git a/lib/ansible/modules/files/unarchive.py b/lib/ansible/modules/files/unarchive.py
--- a/lib/ansible/modules/files/unarchive.py
+++ b/lib/ansible/modules/files/unarchive.py
@@ -865,10 +865,10 @@ def main():
         for filename in handler.files_in_archive:
             file_args['path'] = os.path.join(dest, filename)
             try:
-                res_args['changed'] = module.set_fs_attributes_if_different(file_args, res_args['changed'])
+                res_args['changed'] = module.set_fs_attributes_if_different(file_args, res_args['changed'], expand=False)
             except (IOError, OSError):
                 e = get_exception()
-                module.fail_json(msg="Unexpected error when accessing exploded file: %s" % str(e), **res_args)
+                module.fail_json(msg="Unexpected error when accessing exploded file: %s" % e, **res_args)
 
     if module.params['list_files']:
         res_args['files'] = handler.files_in_archive


Test patch: