Repository: ansible/ansible

Issue:
b64encode filter can't handle non-ascii UTF-8 strings
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
plugins/filter: b64encode

##### ANSIBLE VERSION
```
ansible 2.4.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/emiikar/my_modules/']
  python version = 2.7.12+ (default, Sep 17 2016, 12:08:02) [GCC 6.2.0 20160914]
```

##### OS / ENVIRONMENT
Ubuntu 16.10

##### SUMMARY
b64encode filters fail with non-ascii string variable.

##### STEPS TO REPRODUCE

Run the following playbook:

```yaml
---
- hosts: all
  connection: local
  vars:
    data: öö
  tasks:
    - debug:
        msg: "{{ data | b64encode }}"
```


##### EXPECTED RESULTS

The  base64 encoded string generated by Python's base64.b64encode function:
```pycon
$ python2.7
Python 2.7.12+ (default, Sep 17 2016, 12:08:02) 
[GCC 6.2.0 20160914] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import base64
>>> print(base64.b64encode("öö"))
w7bDtg==
```

##### ACTUAL RESULTS
```
$ ./ansible-playbook -vvvv -i ~/inv ~/playbook.yml
Using /etc/ansible/ansible.cfg as config file
Loading callback plugin default of type stdout, v2.0 from /home/emiikar/ansible/ansible-devel/local/lib/python2.7/site-packages/ansible-2.4.0-py2.7.egg/ansible/plugins/callback/__init__.pyc

PLAYBOOK: playbook.yml *********************************************************************************************************************************
1 plays in /home/emiikar/playbook.yml

PLAY [all] *********************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************
Using module file /home/emiikar/ansible/ansible-devel/local/lib/python2.7/site-packages/ansible-2.4.0-py2.7.egg/ansible/modules/system/setup.py
<192.168.0.1> ESTABLISH LOCAL CONNECTION FOR USER: emiikar
<192.168.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<192.168.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/emiikar/.ansible/tmp/ansible-tmp-1490185332.21-111905409243237 `" && echo ansible-tmp-1490185332.21-111905409243237="` echo /home/emiikar/.ansible/tmp/ansible-tmp-1490185332.21-111905409243237 `" ) && sleep 0'
<192.168.0.1> PUT /tmp/tmpsggB6X TO /home/emiikar/.ansible/tmp/ansible-tmp-1490185332.21-111905409243237/setup.py
<192.168.0.1> EXEC /bin/sh -c 'chmod u+x /home/emiikar/.ansible/tmp/ansible-tmp-1490185332.21-111905409243237/ /home/emiikar/.ansible/tmp/ansible-tmp-1490185332.21-111905409243237/setup.py && sleep 0'
<192.168.0.1> EXEC /bin/sh -c '/usr/bin/python /home/emiikar/.ansible/tmp/ansible-tmp-1490185332.21-111905409243237/setup.py; rm -rf "/home/emiikar/.ansible/tmp/ansible-tmp-1490185332.21-111905409243237/" > /dev/null 2>&1 && sleep 0'
ok: [192.168.0.1]
META: ran handlers

TASK [debug] *******************************************************************************************************************************************
task path: /home/emiikar/playbook.yml:7
fatal: [192.168.0.1]: FAILED! => {
    "failed": true, 
    "msg": "the field 'args' has an invalid value ([]), and could not be converted to an dict. Error was: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)\n\nThe error appears to have been in '/home/emiikar/playbook.yml': line 7, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n    - debug:\n      ^ here\n"
}
	to retry, use: --limit @/home/emiikar/playbook.retry

PLAY RECAP *********************************************************************************************************************************************
192.168.0.1              : ok=1    changed=0    unreachable=0    failed=1
```

With Ansible 2.2.0, the similar error was happening with hash filter too, but it seems to be fixed at least in the development branch of Ansible.


Discussion:

Resolving patch:
diff --git a/lib/ansible/plugins/filter/core.py b/lib/ansible/plugins/filter/core.py
--- a/lib/ansible/plugins/filter/core.py
+++ b/lib/ansible/plugins/filter/core.py
@@ -459,6 +459,14 @@ def do_groupby(environment, value, attribute):
     return [tuple(t) for t in _do_groupby(environment, value, attribute)]
 
 
+def b64encode(string):
+    return to_text(base64.b64encode(to_bytes(string, errors='surrogate_then_strict')))
+
+
+def b64decode(string):
+    return to_text(base64.b64decode(to_bytes(string, errors='surrogate_then_strict')))
+
+
 class FilterModule(object):
     ''' Ansible core jinja2 filters '''
 
@@ -468,8 +476,8 @@ def filters(self):
             'groupby': do_groupby,
 
             # base 64
-            'b64decode': partial(unicode_wrap, base64.b64decode),
-            'b64encode': partial(unicode_wrap, base64.b64encode),
+            'b64decode': b64decode,
+            'b64encode': b64encode,
 
             # uuid
             'to_uuid': to_uuid,


Test patch: