Repository: ansible/ansible

Issue:
Avoid crash when module diff has non utf8 translatable text
Try to call to_text against result. If we get an exception, we return the text as bytes.

##### ISSUE TYPE
 - Bugfix Pull Request

##### COMPONENT NAME

Crash when using diff option with iso8859-15 files.

##### ANSIBLE VERSION

```
ansible 2.3.0 (diff-iso8859-15 2d2d12f0cc) last updated 2017/02/22 23:45:17 (GMT +200)
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
```

##### SUMMARY

Fixes #21803. Try to call to_text against result string. If we get an exception, we return text as bytes.

```
ansible-playbook crash-diff.yml --diff
PLAY [Crash when using --diff option] ***********************************************************************************************************

TASK [Copy a dummy iso8859-15 file] *************************************************************************************************************
--- before: /tmp/test.txt
+++ after: /home/drayan/dev/yannig-ansible-playbooks/diff-iso8859-15/iso8859-15.txt
@@ -1,3 +1,3 @@
-Un super fichier avec é en iso8859-15
+Un super fichier avec � en iso8859-15
 
-A super file with é in iso8859-15
+A super file with � in iso8859-15

changed: [localhost]
```



Discussion:
I've pinged jimi-c on slack as well.  This will need revision afterwards, though.

needs_revision
I would add warning to the exception 'We were unable to decode all characters, skipped some in effort to return as much as possible'

@bcoca done.
@jimi-c, @Yannig does this satisfy both of your concerns? https://gist.github.com/abadger/7098ef7e89ef2b7d00d7193f31d83ea6

Update:  jimi-c gave his approval of the gist's strategy in slack.
@Yannig Cool, I'll PR that solution then.

Resolving patch:
diff --git a/lib/ansible/executor/task_executor.py b/lib/ansible/executor/task_executor.py
--- a/lib/ansible/executor/task_executor.py
+++ b/lib/ansible/executor/task_executor.py
@@ -130,17 +130,26 @@ def run(self):
             if 'changed' not in res:
                 res['changed'] = False
 
-            def _clean_res(res):
+            def _clean_res(res, errors='surrogate_or_strict'):
                 if isinstance(res, UnsafeProxy):
                     return res._obj
                 elif isinstance(res, binary_type):
-                    return to_text(res, errors='surrogate_or_strict')
+                    return to_text(res, errors=errors)
                 elif isinstance(res, dict):
                     for k in res:
-                        res[k] = _clean_res(res[k])
+                        try:
+                            res[k] = _clean_res(res[k], errors=errors)
+                        except UnicodeError:
+                            if k == 'diff':
+                                # If this is a diff, substitute a replacement character if the value
+                                # is undecodable as utf8.  (Fix #21804)
+                                display.warning("We were unable to decode all characters, replaced some in an effort to return as much as possible")
+                                res[k] = _clean_res(res[k], errors='surrogate_then_replace')
+                            else:
+                                raise
                 elif isinstance(res, list):
                     for idx,item in enumerate(res):
-                        res[idx] = _clean_res(item)
+                        res[idx] = _clean_res(item, errors=errors)
                 return res
 
             display.debug("dumping result to json")


Test patch: