Repository: ansible/ansible

Issue:
incompatible change in copy resultset in 2.4
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
copy

##### ANSIBLE VERSION
```
ansible 2.4.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/phil/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.13 (default, Sep  5 2017, 08:53:59) [GCC 7.1.1 20170622 (Red Hat 7.1.1-3)]
```

##### CONFIGURATION
n/a

##### OS / ENVIRONMENT
n/a

##### SUMMARY
Between ansible 2.3.2 to 2.4.0 the `results` array of the `copy` module has changed in an incompatible way regarding `path` and `dest`.

##### STEPS TO REPRODUCE
1. `ansible-playbook copy.yml`

```yaml
---
- hosts: localhost
  gather_facts: no
  tasks:
    - name: "cleanup"
      command: "rm -rf /tmp/ansibleresults"
      args:
        warn: no

    - name: "create directory structure"
      file: path={{item}} state=directory
      with_items:
        - /tmp/ansibleresults
        - /tmp/ansibleresults/src
        - /tmp/ansibleresults/dest

    - name: "create some dummy files"
      copy: content="I am a dummy file {{item}}" dest=/tmp/ansibleresults/src/myfile{{item}}
      with_items: [ 1,2,3,4 ]

    - name: "pre-copy a file"
      copy: src=/tmp/ansibleresults/src/myfile3 dest=/tmp/ansibleresults/dest/

    - name: "copy all source files"
      copy: src={{item}} dest=/tmp/ansibleresults/dest/
      register: t_copyresults
      with_fileglob:
        - /tmp/ansibleresults/src/*

    - name: "show attribute 'dest'"
      debug: var=item
      with_items:
        - '{{t_copyresults.results|map(attribute="dest")|list}}'
```


##### EXPECTED RESULTS
```
TASK [show attribute 'dest'] ***
ok: [localhost] => (item=/tmp/ansibleresults/dest/myfile4) => {
    "item": "/tmp/ansibleresults/dest/myfile4"
}
ok: [localhost] => (item=/tmp/ansibleresults/dest/myfile3) => {
    "item": "/tmp/ansibleresults/dest/myfile3"
}
ok: [localhost] => (item=/tmp/ansibleresults/dest/myfile2) => {
    "item": "/tmp/ansibleresults/dest/myfile2"
}
ok: [localhost] => (item=/tmp/ansibleresults/dest/myfile1) => {
    "item": "/tmp/ansibleresults/dest/myfile1"
}
```

##### ACTUAL RESULTS
```
TASK [show attribute 'dest'] ***
ok: [localhost] => (item=[u'/tmp/ansibleresults/dest/myfile4', Undefined, u'/tmp/ansibleresults/dest/myfile2', u'/tmp/ansibleresults/dest/myfile1']) => {
    "item": "[u'/tmp/ansibleresults/dest/myfile4', Undefined, u'/tmp/ansibleresults/dest/myfile2', u'/tmp/ansibleresults/dest/myfile1']"
}
```

incompatible change in copy resultset in 2.4
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
copy

##### ANSIBLE VERSION
```
ansible 2.4.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/phil/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.13 (default, Sep  5 2017, 08:53:59) [GCC 7.1.1 20170622 (Red Hat 7.1.1-3)]
```

##### CONFIGURATION
n/a

##### OS / ENVIRONMENT
n/a

##### SUMMARY
Between ansible 2.3.2 to 2.4.0 the `results` array of the `copy` module has changed in an incompatible way regarding `path` and `dest`.

##### STEPS TO REPRODUCE
1. `ansible-playbook copy.yml`

```yaml
---
- hosts: localhost
  gather_facts: no
  tasks:
    - name: "cleanup"
      command: "rm -rf /tmp/ansibleresults"
      args:
        warn: no

    - name: "create directory structure"
      file: path={{item}} state=directory
      with_items:
        - /tmp/ansibleresults
        - /tmp/ansibleresults/src
        - /tmp/ansibleresults/dest

    - name: "create some dummy files"
      copy: content="I am a dummy file {{item}}" dest=/tmp/ansibleresults/src/myfile{{item}}
      with_items: [ 1,2,3,4 ]

    - name: "pre-copy a file"
      copy: src=/tmp/ansibleresults/src/myfile3 dest=/tmp/ansibleresults/dest/

    - name: "copy all source files"
      copy: src={{item}} dest=/tmp/ansibleresults/dest/
      register: t_copyresults
      with_fileglob:
        - /tmp/ansibleresults/src/*

    - name: "show attribute 'dest'"
      debug: var=item
      with_items:
        - '{{t_copyresults.results|map(attribute="dest")|list}}'
```


##### EXPECTED RESULTS
```
TASK [show attribute 'dest'] ***
ok: [localhost] => (item=/tmp/ansibleresults/dest/myfile4) => {
    "item": "/tmp/ansibleresults/dest/myfile4"
}
ok: [localhost] => (item=/tmp/ansibleresults/dest/myfile3) => {
    "item": "/tmp/ansibleresults/dest/myfile3"
}
ok: [localhost] => (item=/tmp/ansibleresults/dest/myfile2) => {
    "item": "/tmp/ansibleresults/dest/myfile2"
}
ok: [localhost] => (item=/tmp/ansibleresults/dest/myfile1) => {
    "item": "/tmp/ansibleresults/dest/myfile1"
}
```

##### ACTUAL RESULTS
```
TASK [show attribute 'dest'] ***
ok: [localhost] => (item=[u'/tmp/ansibleresults/dest/myfile4', Undefined, u'/tmp/ansibleresults/dest/myfile2', u'/tmp/ansibleresults/dest/myfile1']) => {
    "item": "[u'/tmp/ansibleresults/dest/myfile4', Undefined, u'/tmp/ansibleresults/dest/myfile2', u'/tmp/ansibleresults/dest/myfile1']"
}
```



Discussion:
possible workaround:
```yaml
    - name: "show attribute 'dest'"
      debug: var=item
      with_items:
        - '{{(t_copyresults.results|selectattr("path", "defined")|map(attribute="path")|list+t_copyresults.results|selectattr("dest", "defined")|map(attribute="dest")|list)|list|unique}}'
```
but... really?
Bisected this to f86ce0975d8496b61ed6e2206cab917d0779cec9, seems to work on stable-2.3 (contrary to #25431)
This is pretty much making Ansible 2.4 unusable for us. Confirmed that the resultset contains `dest` when copy task is changed, but contains `path` when the copy task is unchanged (ie: file already exists).

Example when file was copied:
```
ok: [192.168.11.10] => {
    "rpm_files": {
        "changed": true,
        "msg": "All items completed",
        "results": [
            {
                "_ansible_ignore_errors": null,
                "_ansible_item_result": true,
                "_ansible_no_log": false,
                "_ansible_parsed": true,
                "changed": true,
                "checksum": "e54a942de3eef088b5db7374b236c4ae8a61f04b",

                "dest": "/tmp/my.rpm",   <- Note the dest attribute, which is what's in ansible 2.2
...
```

Example when running the same task a 2nd time, this time with the file:

```
ok: [192.168.11.10] => {
    "rpm_files": {
        "changed": false,
        "msg": "All items completed",
        "results": [
            {
                "_ansible_ignore_errors": null,
                "_ansible_item_result": true,
                "_ansible_no_log": false,
                "_ansible_parsed": true,
                "changed": false,
                "checksum": "e54a942de3eef088b5db7374b236c4ae8a61f04b",

                 <--dest is missing here!!!

                "diff": {
                    "after": {
                        "path": "/tmp/my.rpm"
                    },
                    "before": {
                        "path": "/tmp/my.rpm"
                    }
                },
                "failed": false,
                "gid": 0,
                "group": "root",
                "invocation": {
                   ...
                    }
                },
                "item": "/src/path/to/my.rpm",
                "mode": "0644",
                "owner": "root",

                "path": "my.rpm",           <- dest was replaced by path?!

                "secontext": "unconfined_u:object_r:admin_home_t:s0",
                "size": 41525950,
                "state": "file",
                "uid": 0
...
```
Looks like when the file needs to be copied, the `copy` action is the one that is fired.

However when the file checksum is the same, the `file` action is the one that is fired (to do any permission changes that are needed).

The `file` module seems to accept/return only `path` (which it says is an alias to `dest`), whereas the `copy` module seems to accept/return `dest` only.

I still don't see exactly where in commit https://github.com/ansible/ansible/commit/f86ce0975d8496b61ed6e2206cab917d0779cec9 this got changed, however the copy action was pretty heavily refactored and the diff is hard to read, so seems plausible the change was introduced there.
possible workaround:
```yaml
    - name: "show attribute 'dest'"
      debug: var=item
      with_items:
        - '{{(t_copyresults.results|selectattr("path", "defined")|map(attribute="path")|list+t_copyresults.results|selectattr("dest", "defined")|map(attribute="dest")|list)|list|unique}}'
```
but... really?
Bisected this to f86ce0975d8496b61ed6e2206cab917d0779cec9, seems to work on stable-2.3 (contrary to #25431)
This is pretty much making Ansible 2.4 unusable for us. Confirmed that the resultset contains `dest` when copy task is changed, but contains `path` when the copy task is unchanged (ie: file already exists).

Example when file was copied:
```
ok: [192.168.11.10] => {
    "rpm_files": {
        "changed": true,
        "msg": "All items completed",
        "results": [
            {
                "_ansible_ignore_errors": null,
                "_ansible_item_result": true,
                "_ansible_no_log": false,
                "_ansible_parsed": true,
                "changed": true,
                "checksum": "e54a942de3eef088b5db7374b236c4ae8a61f04b",

                "dest": "/tmp/my.rpm",   <- Note the dest attribute, which is what's in ansible 2.2
...
```

Example when running the same task a 2nd time, this time with the file:

```
ok: [192.168.11.10] => {
    "rpm_files": {
        "changed": false,
        "msg": "All items completed",
        "results": [
            {
                "_ansible_ignore_errors": null,
                "_ansible_item_result": true,
                "_ansible_no_log": false,
                "_ansible_parsed": true,
                "changed": false,
                "checksum": "e54a942de3eef088b5db7374b236c4ae8a61f04b",

                 <--dest is missing here!!!

                "diff": {
                    "after": {
                        "path": "/tmp/my.rpm"
                    },
                    "before": {
                        "path": "/tmp/my.rpm"
                    }
                },
                "failed": false,
                "gid": 0,
                "group": "root",
                "invocation": {
                   ...
                    }
                },
                "item": "/src/path/to/my.rpm",
                "mode": "0644",
                "owner": "root",

                "path": "my.rpm",           <- dest was replaced by path?!

                "secontext": "unconfined_u:object_r:admin_home_t:s0",
                "size": 41525950,
                "state": "file",
                "uid": 0
...
```
Looks like when the file needs to be copied, the `copy` action is the one that is fired.

However when the file checksum is the same, the `file` action is the one that is fired (to do any permission changes that are needed).

The `file` module seems to accept/return only `path` (which it says is an alias to `dest`), whereas the `copy` module seems to accept/return `dest` only.

I still don't see exactly where in commit https://github.com/ansible/ansible/commit/f86ce0975d8496b61ed6e2206cab917d0779cec9 this got changed, however the copy action was pretty heavily refactored and the diff is hard to read, so seems plausible the change was introduced there.

Resolving patch:
diff --git a/lib/ansible/plugins/action/copy.py b/lib/ansible/plugins/action/copy.py
--- a/lib/ansible/plugins/action/copy.py
+++ b/lib/ansible/plugins/action/copy.py
@@ -539,13 +539,13 @@ def run(self, tmp=None, task_vars=None):
 
             changed = changed or module_return.get('changed', False)
 
-            # the file module returns the file path as 'path', but
-            # the copy module uses 'dest', so add it if it's not there
-            if 'path' in module_return and 'dest' not in module_return:
-                module_return['dest'] = module_return['path']
-
         if module_executed and len(source_files['files']) == 1:
             result.update(module_return)
+
+            # the file module returns the file path as 'path', but
+            # the copy module uses 'dest', so add it if it's not there
+            if 'path' in result and 'dest' not in result:
+                result['dest'] = result['path']
         else:
             result.update(dict(dest=dest, src=source, changed=changed))
 


Test patch:
diff --git a/test/integration/targets/copy/tasks/tests.yml b/test/integration/targets/copy/tasks/tests.yml
--- a/test/integration/targets/copy/tasks/tests.yml
+++ b/test/integration/targets/copy/tasks/tests.yml
@@ -37,6 +37,7 @@
 
 - set_fact:
     remote_dir_expanded: '{{ echo.stdout }}'
+    remote_file_expanded: '{{ echo.stdout }}/foo.txt'
 
 - debug:
     var: copy_result
@@ -46,7 +47,7 @@
   assert:
     that:
       - "'changed' in copy_result"
-      - "'dest' in copy_result"
+      - copy_result.dest == remote_file_expanded
       - "'group' in copy_result"
       - "'gid' in copy_result"
       - "'md5sum' in copy_result"
@@ -111,6 +112,19 @@
     that:
       - "copy_result2 is not changed"
 
+- name: Assert basic copy worked
+  assert:
+    that:
+      - "'changed' in copy_result2"
+      - copy_result2.dest == remote_file_expanded
+      - "'group' in copy_result2"
+      - "'gid' in copy_result2"
+      - "'checksum' in copy_result2"
+      - "'owner' in copy_result2"
+      - "'size' in copy_result2"
+      - "'state' in copy_result2"
+      - "'uid' in copy_result2"
+
 - name: Overwrite the file using the content system
   copy:
     content: "modified"