Repository: ansible/ansible
Pull request: wait_for - Read file and perform comparisons using bytes to avoid decode errors
##### SUMMARY
wait_for - Read file and perform comparisons using bytes to avoid decode errors. Fixes #78214

##### ISSUE TYPE
- Bugfix Pull Request

##### COMPONENT NAME
lib/ansible/modules/wait_for.py

##### ADDITIONAL INFORMATION
With the test added, and without this change:

```
TASK [wait_for : test non-ascii file] ******************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 5: invalid continuation byte
fatal: [testhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 121, in <module>\n  File \"<stdin>\", line 113, in _ansiballz_main\n  File \"<stdin>\", line 61, in invoke_module\n  File \"/usr/lib64/python3.10/runpy.py\", line 209, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.10/runpy.py\", line 96, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib64/python3.10/runpy.py\", line 86, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_wait_for_payload_jmij59ub/ansible_wait_for_payload.zip/ansible/modules/wait_for.py\", line 681, in <module>\n  File \"/tmp/ansible_wait_for_payload_jmij59ub/ansible_wait_for_payload.zip/ansible/modules/wait_for.py\", line 580, in main\n  File \"/usr/lib64/python3.10/codecs.py\", line 322, in decode\n    (result, consumed) = self._buffer_decode(data, self.errors, final)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 5: invalid continuation byte\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
```

Problem statement:
wait_for with search_regex : UnicodeDecodeError
### Summary

`wait_for` with arguments `path` and `search_regex` throws UnicodeDecodeError when the source file (given in `path`) is not UTF-8. It happens a lot with log files, which is our use case.

It works fine with Python 2.7, fails as soon as I add ` -e ansible_python_interpreter="/usr/bin/python3"`

The regex itself has no impact (I tried with a simple string).

### Issue Type

Bug Report

### Component Name

ansible.builtin.wait_for

### Ansible Version

```console
$ ansible --version
ansible [core 2.12.6]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['xxxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = xxxx/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.13 (main, May 28 2022, 14:03:04) [GCC 10.2.1 20210110]
  jinja version = 3.1.2
  libyaml = True
```


### Configuration

```console
# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed
ANSIBLE_FORCE_COLOR(/etc/ansible/ansible.cfg) = True
DEFAULT_TIMEOUT(/etc/ansible/ansible.cfg) = 180
DEFAULT_VAULT_PASSWORD_FILE(/etc/ansible/ansible.cfg) = .../.vault_pass.txt
GALAXY_SERVER(/etc/ansible/ansible.cfg) = .../nexus/repository/galaxy.ansible.com/
```


### OS / Environment

RHEL 7, Python 3.6

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```yaml (paste below)
- name: Match "Server startup" appearing in new lines of tomcat logs
  wait_for:
    path: "{{ tomcat_log_path }}"
    search_regex: "{{ startup_regex }}"
  timeout: 420
```


### Expected Results

No error

### Actual Results

```console
TASK  : Match "Server startup" appearing in new lines of tomcat logs ***************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 324385004: invalid continuation byte
fatal: [xxxx]: FAILED! => {"changed": false, "module_stderr": "Shared connection to xxxx closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/var/tmp/ansible-tmp-1657113434.9852784-3039-230603050398226/AnsiballZ_wait_for.py\", line 107, in <module>\r\n    _ansiballz_main()\r\n  File \"/var/tmp/ansible-tmp-1657113434.9852784-3039-230603050398226/AnsiballZ_wait_for.py\", line 99, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/var/tmp/ansible-tmp-1657113434.9852784-3039-230603050398226/AnsiballZ_wait_for.py\", line 48, in invoke_module\r\n    run_name='__main__', alter_sys=True)\r\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\r\n    return _run_module_code(code, init_globals, run_name, mod_spec)\r\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\r\n    mod_name, mod_spec, pkg_name, script_name)\r\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\r\n    exec(code, run_globals)\r\n  File \"/tmp/ansible_wait_for_payload_bkth5jhn/ansible_wait_for_payload.zip/ansible/modules/wait_for.py\", line 682, in <module>\r\n  File \"/tmp/ansible_wait_for_payload_bkth5jhn/ansible_wait_for_payload.zip/ansible/modules/wait_for.py\", line 581, in main\r\n  File \"/usr/lib64/python3.6/codecs.py\", line 321, in decode\r\n    (result, consumed) = self._buffer_decode(data, self.errors, final)\r\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 324385004: invalid continuation byte\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct

Hints from issue discussion:
Files identified in the description:
* [`lib/ansible/modules/wait_for.py`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/wait_for.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Yep, looks like some vestigial stuff that never got fully updated to work with Python3

Commit message: wait_for - Read file and perform comparisons using bytes to avoid decode errors. Fixes #78214

Patch to review:
diff --git a/changelogs/fragments/78214-wait-for-compare-bytes.yml b/changelogs/fragments/78214-wait-for-compare-bytes.yml
new file mode 100644
index 00000000000000..252d7ca4a99f49
--- /dev/null
+++ b/changelogs/fragments/78214-wait-for-compare-bytes.yml
@@ -0,0 +1,3 @@
+bugfixes:
+- wait_for - Read file and perform comparisons using bytes to avoid decode errors
+  (https://github.com/ansible/ansible/issues/78214)
diff --git a/lib/ansible/modules/wait_for.py b/lib/ansible/modules/wait_for.py
index 2244f357bc10d7..3cc0766366f60f 100644
--- a/lib/ansible/modules/wait_for.py
+++ b/lib/ansible/modules/wait_for.py
@@ -227,6 +227,7 @@
 import datetime
 import errno
 import math
+import mmap
 import os
 import re
 import select
@@ -236,7 +237,7 @@
 
 from ansible.module_utils.basic import AnsibleModule, missing_required_lib
 from ansible.module_utils.common.sys_info import get_platform_subclass
-from ansible.module_utils._text import to_native
+from ansible.module_utils._text import to_bytes
 
 
 HAS_PSUTIL = False
@@ -496,14 +497,22 @@ def main():
     delay = module.params['delay']
     port = module.params['port']
     state = module.params['state']
+
     path = module.params['path']
+    b_path = to_bytes(path, errors='surrogate_or_strict', nonstring='passthru')
+
     search_regex = module.params['search_regex']
+    b_search_regex = to_bytes(search_regex, errors='surrogate_or_strict', nonstring='passthru')
+
     msg = module.params['msg']
 
     if search_regex is not None:
-        compiled_search_re = re.compile(search_regex, re.MULTILINE)
+        try:
+            b_compiled_search_re = re.compile(b_search_regex, re.MULTILINE)
+        except re.error as e:
+            module.fail_json(msg="Invalid regular expression: %s" % e)
     else:
-        compiled_search_re = None
+        b_compiled_search_re = None
 
     match_groupdict = {}
     match_groups = ()
@@ -536,7 +545,7 @@ def main():
         while datetime.datetime.utcnow() < end:
             if path:
                 try:
-                    if not os.access(path, os.F_OK):
+                    if not os.access(b_path, os.F_OK):
                         break
                 except IOError:
                     break
@@ -562,7 +571,7 @@ def main():
         while datetime.datetime.utcnow() < end:
             if path:
                 try:
-                    os.stat(path)
+                    os.stat(b_path)
                 except OSError as e:
                     # If anything except file not present, throw an error
                     if e.errno != 2:
@@ -571,22 +580,20 @@ def main():
                     # file doesn't exist yet, so continue
                 else:
                     # File exists.  Are there additional things to check?
-                    if not compiled_search_re:
+                    if not b_compiled_search_re:
                         # nope, succeed!
                         break
                     try:
-                        f = open(path)
-                        try:
-                            search = re.search(compiled_search_re, f.read())
-                            if search:
-                                if search.groupdict():
-                                    match_groupdict = search.groupdict()
-                                if search.groups():
-                                    match_groups = search.groups()
-
-                                break
-                        finally:
-                            f.close()
+                        with open(b_path, 'rb') as f:
+                            with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
+                                search = b_compiled_search_re.search(mm)
+                                if search:
+                                    if search.groupdict():
+                                        match_groupdict = search.groupdict()
+                                    if search.groups():
+                                        match_groups = search.groups()
+
+                                    break
                     except IOError:
                         pass
             elif port:
@@ -598,8 +605,8 @@ def main():
                     pass
                 else:
                     # Connected -- are there additional conditions?
-                    if compiled_search_re:
-                        data = ''
+                    if b_compiled_search_re:
+                        b_data = b''
                         matched = False
                         while datetime.datetime.utcnow() < end:
                             max_timeout = math.ceil(_timedelta_total_seconds(end - datetime.datetime.utcnow()))
@@ -612,8 +619,8 @@ def main():
                             if not response:
                                 # Server shutdown
                                 break
-                            data += to_native(response, errors='surrogate_or_strict')
-                            if re.search(compiled_search_re, data):
+                            b_data += response
+                            if b_compiled_search_re.search(b_data):
                                 matched = True
                                 break
 
diff --git a/test/integration/targets/wait_for/files/latin1.txt b/test/integration/targets/wait_for/files/latin1.txt
new file mode 100644
index 00000000000000..4394e91fe7545a
--- /dev/null
+++ b/test/integration/targets/wait_for/files/latin1.txt
@@ -0,0 +1,4 @@
+premi�re is first
+premie?re is slightly different
+????????? is Cyrillic
+? am Deseret
diff --git a/test/integration/targets/wait_for/tasks/main.yml b/test/integration/targets/wait_for/tasks/main.yml
index c524f990bce80c..bcb129a8cea689 100644
--- a/test/integration/targets/wait_for/tasks/main.yml
+++ b/test/integration/targets/wait_for/tasks/main.yml
@@ -104,6 +104,16 @@
     - waitfor['match_groupdict']['foo'] == 'data'
     - waitfor['match_groups'] == ['data', '123']
 
+- name: copy non-ascii file
+  copy:
+    src: latin1.txt
+    dest: "{{remote_tmp_dir}}/latin1.txt"
+
+- name: test non-ascii file
+  wait_for:
+    path: "{{remote_tmp_dir}}/latin1.txt"
+    search_regex: slightly
+
 - name: test wait for port timeout
   wait_for:
     port: 12121


Code review comments:
diff_hunk: @@ -571,22 +580,20 @@ def main():
                     # file doesn't exist yet, so continue
                 else:
                     # File exists.  Are there additional things to check?
-                    if not compiled_search_re:
+                    if not b_compiled_search_re:
                         # nope, succeed!
                         break
                     try:
-                        f = open(path)
-                        try:
-                            search = re.search(compiled_search_re, f.read())
-                            if search:
-                                if search.groupdict():
-                                    match_groupdict = search.groupdict()
-                                if search.groups():
-                                    match_groups = search.groups()
-
-                                break
-                        finally:
-                            f.close()
+                        with open(b_path, 'rb') as f:
+                            with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
path: lib/ansible/modules/wait_for.py
text: @user1:
Python 2.7 doesn't support using this as a context manager.

diff_hunk: @@ -0,0 +1,4 @@
+premi�re is first
path: test/integration/targets/wait_for/files/latin1.txt
text: @user1:
Since this file is not UTF-8 encoded it should be generated when the test runs. Perhaps using a Python script with escape sequences, or base 64 encoded.

@author:
I can do that, it was in a script in expect, and I just wrote it out statically.