Repository: ansible/ansible

Issue:
assemble module cannot assemble files with unicode characters in their name
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
ansible_module_assemble

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.2.0.0
  config file = /home/.../ansible.cfg
  configured module search path = Default w/o overrides
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
<!----->
Ubuntu 16.04.1 LTS
Python 2.7.12

##### SUMMARY
When using assemble on files with unicode characters in their file name, the assemble module fails.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

```mkdir /tmp/unicode-fragments
touch /tmp/unicode-fragments/äöü
```

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- assemble:
    src: /tmp/unicode-fragments
    dest: /tmp/dest

```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS

Empty file assembled.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_FZJteE/ansible_module_assemble.py", line 258, in <module>
    main()
  File "/tmp/ansible_FZJteE/ansible_module_assemble.py", line 220, in main
    path = assemble_from_fragments(src, delimiter, compiled_regexp, ignore_hidden)
  File "/tmp/ansible_FZJteE/ansible_module_assemble.py", line 131, in assemble_from_fragments
    fragment = u"%s/%s" % (src_path, f)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128)
```

#### Analysis

With https://github.com/ansible/ansible-modules-core/commit/e4c5a13a7ad6355f779a88456780bdcd86baee31#diff-204c9a1c75ed1b68284601e5efe69ac0R131, the fragment path was changed to unicode. However, `os.listdir()`, from which the fragment path is constructed, does not return unicode unless its argument is also unicode (which does not seem to be the case here).


Discussion:

Resolving patch:
diff --git a/lib/ansible/modules/files/assemble.py b/lib/ansible/modules/files/assemble.py
--- a/lib/ansible/modules/files/assemble.py
+++ b/lib/ansible/modules/files/assemble.py
@@ -142,7 +142,7 @@ def assemble_from_fragments(src_path, delimiter=None, compiled_regexp=None, igno
     for f in sorted(os.listdir(src_path)):
         if compiled_regexp and not compiled_regexp.search(f):
             continue
-        fragment = u"%s/%s" % (src_path, f)
+        fragment = os.path.join(src_path, f)
         if not os.path.isfile(fragment) or (ignore_hidden and os.path.basename(fragment).startswith('.')):
             continue
         fragment_content = open(fragment, 'rb').read()


Test patch:
diff --git a/test/integration/targets/assemble/tasks/main.yml b/test/integration/targets/assemble/tasks/main.yml
--- a/test/integration/targets/assemble/tasks/main.yml
+++ b/test/integration/targets/assemble/tasks/main.yml
@@ -29,6 +29,15 @@
   copy: src="./" dest="{{output_dir}}/src"
   register: result
 
+- name: create unicode file for test
+  shell: echo "π" > {{ output_dir }}/src/ßΩ.txt
+  register: result
+
+- name: assert that the new file was created
+  assert:
+    that:
+    - "result.changed == true"
+
 - name: test assemble with all fragments
   assemble: src="{{output_dir}}/src" dest="{{output_dir}}/assembled1"
   register: result
@@ -38,7 +47,7 @@
     that:
     - "result.state == 'file'"
     - "result.changed == True"
-    - "result.checksum == '048a1bd1951aa5ccc427eeb4ca19aee45e9c68b3'"
+    - "result.checksum == '74152e9224f774191bc0bedf460d35de86ad90e6'"
 
 - name: test assemble with all fragments
   assemble: src="{{output_dir}}/src" dest="{{output_dir}}/assembled1"
@@ -49,7 +58,7 @@
     that:
     - "result.state == 'file'"
     - "result.changed == False"
-    - "result.checksum == '048a1bd1951aa5ccc427eeb4ca19aee45e9c68b3'"
+    - "result.checksum == '74152e9224f774191bc0bedf460d35de86ad90e6'"
 
 - name: test assemble with fragments matching a regex
   assemble: src="{{output_dir}}/src" dest="{{output_dir}}/assembled2" regexp="^fragment[1-3]$"
@@ -69,7 +78,7 @@
   assert:
     that:
     - "result.state == 'file'"
-    - "result.checksum == '505359f48c65b3904127cf62b912991d4da7ed6d'"
+    - "result.checksum == 'd986cefb82e34e4cf14d33a3cda132ff45aa2980'"
 
 - name: test assemble with remote_src=False
   assemble: src="./" dest="{{output_dir}}/assembled4" remote_src=no