Repository: ansible-collections/community.general

Issue:
ldap_search: callback utf-8 error "surrogates not allowed" on ansible-core 2.14
### Summary

Searching for computer objects from an Active Directory with the `ldap_search` module fails to print to output search results in `-vv` mode of `ansible-playbook` and instead a `WARNING` is printed about a utf-8 error `surrogates not allowed`.

My guess would be that some of the binary data included in the objects returned by the query is not compatible with the callback plugin's expected UTF-8 encoding.

The same query works on ansible-core 2.11 and prints to output the search results, some attributes in the objects looks like binary data.

### Issue Type

Bug Report

### Component Name

ldap_search

### Ansible Version

```console (paste below)
$ ansible --version
ansible [core 2.14.1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /ansible/lib64/python3.9/site-packages/ansible
  ansible collection location = /home/ansible/.ansible/collections:/usr/share/ansible/collections
  executable location = /ansible/bin/ansible
  python version = 3.9.14 (main, Nov  7 2022, 00:00:00) [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)] (/ansible/bin/python3)
  jinja version = 3.1.2
  libyaml = True
```


### Community.general Version

```console (paste below)
$ ansible-galaxy collection list community.general
# /usr/share/ansible/collections/ansible_collections
Collection        Version
----------------- -------
community.general 6.1.0
```


### Configuration

```console (paste below)
$ ansible-config dump --only-changed
DEFAULT_STDOUT_CALLBACK(/etc/ansible/ansible.cfg) = community.general.yaml
```


### OS / Environment

EL9.1, Python 3.9

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
Query computer objects from Active Directory and run `ansible-playbook -vv` to make the task print the search results to screen.

On ansible-core 2.11 the output includes binary-looking data such as:
```
    objectGUID: |-
      þ-ÎDBÎâªØÓ¥
    objectSid: "[...]E¹Sîà¹'3Ô[...]"
```

### Expected Results

The task should print to output the search results in the same way it does on ansible-core 2.11.

### Actual Results

```console (paste below)
[WARNING]: Failure using method (v2_runner_on_ok) in callback plugin (<ansible_collections.community.general.plugins.callback.yaml.CallbackModule object at 0x7f4071cb7b20>): 'utf-8' codec can't encode characters in position 6-9: surrogates not allowed
```

Instead of printing search results, only the above warning is printed. Otherwise the task seems to work correctly, results are registered.

### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/modules/ldap_search.py`](https://github.com/ansible-collections/community.general/blob/main/plugins/modules/ldap_search.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @eryx12o45 @jtyr
[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I get the same error, when I try to output the results
In general Ansible does not handle binary output well; modules should not return binary output. This is a bit tricky since the ldap_search module doesn't know what it returns, and Base64 encoding everything makes the module harder to work with.

One could add an option to the module which allows to specify a list of fields that should be Base64 encoded, then you could make sure that `objectGUID` and `objectSid` are Base64 encoded (from the example in the issue itself), but other fields (like imaginary fields `objectName` and `objectDescription`, which contain text and no binary).

Alternatively (or at the same time), the yaml callback could be fixed to behave better with binary data. @stefanDeveloper are you also using the yaml callback, or are you using another one?
!component +plugins/callback/yaml.py
Files identified in the description:
* [`plugins/callback/yaml.py`](https://github.com/ansible-collections/community.general/blob/main/plugins/callback/yaml.py)
* [`plugins/modules/ldap_search.py`](https://github.com/ansible-collections/community.general/blob/main/plugins/modules/ldap_search.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
This used to work in 2.11, don't know what has changed.

I do not care much about the binary data but *some* output would be necessary. I do not know if/how this could work in practice but IMO the easiest solution would be to just check if the data is binary and then just print `(binary data)` or something similar.  
Edit: this would move the problem from core/callback plugin to the module which might in practice be easier to get released workflow-wise.
Did anyone check whether this also happens with the default callback? The only examples in this issue where anything more concrete is mentioned are with the community.general.yaml callback. That would help to decide where this error actually comes from (ansible-core or the yaml callback).
The problem seems to arise with all callbacks (that use `Dispay`) and can easily be reproduced by running `ansible localhost -m command -a 'dd if=/dev/urandom of=/dev/stdout bs=1 count=20'`. With that it's easy to test that this did work with ansible-core 2.13, but it does no longer work with ansible-core 2.14. I guess it is related to all the encoding / locale changes mentioned in https://github.com/ansible/ansible/blob/stable-2.14/changelogs/CHANGELOG-v2.14.rst.
Hello,
I have the same problem here.
In a playbook i launch the latex binary and i have and error/warning on stdout : 
```
[WARNING]: Failure using method (v2_runner_on_failed) in callback plugin (<ansible_collections.community.general.plugins.callback.yaml.CallbackModule object
at 0x7f57d7166cd0>): 'utf-8' codec can't encode character '\udce9' in position 9543: surrogates not allowed
```

I think it's time to create an issue in ansible-core for this, since this is an ansible-core problem, not really a community.general problem.
I've created https://github.com/ansible/ansible/issues/80258 for this. Please add more examples that are not related to ldap_search there and not here :)
According to https://github.com/ansible/ansible/issues/80258#issuecomment-1487023872 the module must not return binary data as text, so we need to adjust the module's output. I would probably use a similar code as in https://github.com/ansible/ansible/issues/80258#issuecomment-1477038952:
```
            data = to_bytes(data, encoding='utf-8')
            data = to_text(data, 'utf-8', errors='replace')
```
Probably we should have some better way to output data, but that can happen later (resp. when someone has a good idea how to actually do it).

Resolving patch:
diff --git a/plugins/modules/ldap_search.py b/plugins/modules/ldap_search.py
--- a/plugins/modules/ldap_search.py
+++ b/plugins/modules/ldap_search.py
@@ -61,6 +61,17 @@
     description:
       - Set to C(true) to return the full attribute schema of entries, not
         their attribute values. Overrides I(attrs) when provided.
+  base64_attributes:
+    description:
+      - If provided, all attribute values returned that are listed in this option
+        will be Base64 encoded.
+      - If the special value C(*) appears in this list, all attributes will be
+        Base64 encoded.
+      - All other attribute values will be converted to UTF-8 strings. If they
+        contain binary data, please note that invalid UTF-8 bytes will be omitted.
+    type: list
+    elements: str
+    version_added: 7.0.0
 extends_documentation_fragment:
   - community.general.ldap.documentation
   - community.general.attributes
@@ -81,10 +92,27 @@
   register: ldap_group_gids
 """
 
+RESULTS = """
+results:
+  description:
+    - For every entry found, one dictionary will be returned.
+    - Every dictionary contains a key C(dn) with the entry's DN as a value.
+    - Every attribute of the entry found is added to the dictionary. If the key
+      has precisely one value, that value is taken directly, otherwise the key's
+      value is a list.
+    - Note that all values (for single-element lists) and list elements (for multi-valued
+      lists) will be UTF-8 strings. Some might contain Base64-encoded binary data; which
+      ones is determined by the I(base64_attributes) option.
+  type: list
+  elements: dict
+"""
+
+import base64
 import traceback
 
 from ansible.module_utils.basic import AnsibleModule, missing_required_lib
-from ansible.module_utils.common.text.converters import to_native
+from ansible.module_utils.common.text.converters import to_bytes, to_native, to_text
+from ansible.module_utils.six import string_types, text_type
 from ansible_collections.community.general.plugins.module_utils.ldap import LdapGeneric, gen_specs
 
 LDAP_IMP_ERR = None
@@ -105,6 +133,7 @@ def main():
             filter=dict(type='str', default='(objectClass=*)'),
             attrs=dict(type='list', elements='str'),
             schema=dict(type='bool', default=False),
+            base64_attributes=dict(type='list', elements='str'),
         ),
         supports_check_mode=True,
     )
@@ -118,16 +147,30 @@ def main():
     except Exception as exception:
         module.fail_json(msg="Attribute action failed.", details=to_native(exception))
 
-    module.exit_json(changed=False)
+
+def _normalize_string(val, convert_to_base64):
+    if isinstance(val, string_types):
+        if isinstance(val, text_type):
+            val = to_bytes(val, encoding='utf-8')
+        if convert_to_base64:
+            val = to_text(base64.b64encode(val))
+        else:
+            # See https://github.com/ansible/ansible/issues/80258#issuecomment-1477038952 for details.
+            # We want to make sure that all strings are properly UTF-8 encoded, even if they were not,
+            # or happened to be byte strings.
+            val = to_text(val, 'utf-8', errors='replace')
+            # See also https://github.com/ansible-collections/community.general/issues/5704.
+    return val
 
 
-def _extract_entry(dn, attrs):
+def _extract_entry(dn, attrs, base64_attributes):
     extracted = {'dn': dn}
     for attr, val in list(attrs.items()):
+        convert_to_base64 = '*' in base64_attributes or attr in base64_attributes
         if len(val) == 1:
-            extracted[attr] = val[0]
+            extracted[attr] = _normalize_string(val[0], convert_to_base64)
         else:
-            extracted[attr] = val
+            extracted[attr] = [_normalize_string(v, convert_to_base64) for v in val]
     return extracted
 
 
@@ -140,9 +183,10 @@ def __init__(self, module):
         self._load_scope()
         self._load_attrs()
         self._load_schema()
+        self._base64_attributes = set(self.module.params['base64_attributes'] or [])
 
     def _load_schema(self):
-        self.schema = self.module.boolean(self.module.params['schema'])
+        self.schema = self.module.params['schema']
         if self.schema:
             self.attrsonly = 1
         else:
@@ -179,7 +223,7 @@ def perform_search(self):
                     if self.schema:
                         ldap_entries.append(dict(dn=result[0], attrs=list(result[1].keys())))
                     else:
-                        ldap_entries.append(_extract_entry(result[0], result[1]))
+                        ldap_entries.append(_extract_entry(result[0], result[1], self._base64_attributes))
             return ldap_entries
         except ldap.NO_SUCH_OBJECT:
             self.module.fail_json(msg="Base not found: {0}".format(self.dn))


Test patch: