Repository: ansible/ansible
Pull request: Allow non-ascii (UTF-8) chars in cron file paths and jobs
##### SUMMARY

It may fix #69492

##### ISSUE TYPE
- Bugfix Pull Request

##### COMPONENT NAME

`lib/ansible/modules/cron.py`


Problem statement:
UnicodeDecodeError using cron module with Python3

##### SUMMARY
I just updated my ansible host & I just fixed some trouble about Python3 on remote host.
I only use cron module once & it was previously working fine.
Now it is failing, returning an exception __main__.CronTabError: ('Unexpected error:', <class 'UnicodeDecodeError'>)

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
cron

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```
ansible 2.9.6
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.2 (default, Apr 27 2020, 15:53:34) [GCC 9.3.0]
```
I just updated the system but I was also getting this issue with python 2.8.7 if it could help.

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```
ANSIBLE_SSH_ARGS(/path/to/project/scripts/ansible.cfg) = -o ForwardAgent=yes -o ControlMas
DEFAULT_REMOTE_USER(/path/to/project/scripts/ansible.cfg) = root
DISPLAY_SKIPPED_HOSTS(/path/to/project/scripts/ansible.cfg) = False
HOST_KEY_CHECKING(/path/to/project/scripts/ansible.cfg) = False
INTERPRETER_PYTHON(/path/to/project/scripts/ansible.cfg) = auto
```

##### OS / ENVIRONMENT
Ansible Host : Ubuntu 20.04 LTS
Remote: Ubuntu 18.04.4 LTS

##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
-   name: "Cron tasks of {{ project.label }} ({{ 'enabled' if enable_cron else 'disabled' }})"
    cron:
        name: "{{ project.label }} - {{ cron_task.name }}"
        cron_file: "{{ project_user }}"
        disabled: "{{ not enable_cron }}"
        minute: "*/10"
        user: "{{ web_user }}"
        job: "{{ project.path }}{{ cron_task.path }}"
    loop: "{{ project.cron | dict2items(key_name='name', value_name='path') }}"
    loop_control:
        loop_var: cron_task
```
```
ok: [cco-web-uat] => {
    "project.cron": {
        "run-all-tasks": "/scripts/run-all-tasks.php"
    }
}
```
project.label is "Zero Frais de Courtage"
web_user is "www-data"
project_user is "cco"
project.path is a local path like "/home/cco/project"

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->
Running cron task already executed before with success

##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->

```
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_cron_payload_0y8i2pgj/ansible_cron_payload.zip/ansible/modules/system/cron.py", line 257, in read
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 11: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1589383434.7764497-217077099734091/AnsiballZ_cron.py", line 102, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1589383434.7764497-217077099734091/AnsiballZ_cron.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1589383434.7764497-217077099734091/AnsiballZ_cron.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible.modules.system.cron', init_globals=None, run_name='__main__', alter_sys=True)
  File "/usr/lib/python3.6/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_cron_payload_0y8i2pgj/ansible_cron_payload.zip/ansible/modules/system/cron.py", line 770, in <module>
  File "/tmp/ansible_cron_payload_0y8i2pgj/ansible_cron_payload.zip/ansible/modules/system/cron.py", line 617, in main
  File "/tmp/ansible_cron_payload_0y8i2pgj/ansible_cron_payload.zip/ansible/modules/system/cron.py", line 248, in __init__
  File "/tmp/ansible_cron_payload_0y8i2pgj/ansible_cron_payload.zip/ansible/modules/system/cron.py", line 264, in read
__main__.CronTabError: ('Unexpected error:', <class 'UnicodeDecodeError'>)
failed: [cco-web-uat] (item={'name': 'run-all-tasks', 'path': '/scripts/run-all-tasks.php'}) => {
    "ansible_loop_var": "cron_task",
    "changed": false,
    "cron_task": {
        "name": "run-all-tasks",
        "path": "/scripts/run-all-tasks.php"
    },
    "module_stderr": "Shared connection to server.fr closed.\r\n",
    "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_cron_payload_0y8i2pgj/ansible_cron_payload.zip/ansible/modules/system/cron.py\", line 257, in read\r\n  File \"/usr/lib/python3.6/encodings/ascii.py\", line 26, in decode\r\n    return codecs.ascii_decode(input, self.errors)[0]\r\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 11: ordinal not in range(128)\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/root/.ansible/tmp/ansible-tmp-1589383434.7764497-217077099734091/AnsiballZ_cron.py\", line 102, in <module>\r\n    _ansiballz_main()\r\n  File \"/root/.ansible/tmp/ansible-tmp-1589383434.7764497-217077099734091/AnsiballZ_cron.py\", line 94, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/root/.ansible/tmp/ansible-tmp-1589383434.7764497-217077099734091/AnsiballZ_cron.py\", line 40, in invoke_module\r\n    runpy.run_module(mod_name='ansible.modules.system.cron', init_globals=None, run_name='__main__', alter_sys=True)\r\n  File \"/usr/lib/python3.6/runpy.py\", line 205, in run_module\r\n    return _run_module_code(code, init_globals, run_name, mod_spec)\r\n  File \"/usr/lib/python3.6/runpy.py\", line 96, in _run_module_code\r\n    mod_name, mod_spec, pkg_name, script_name)\r\n  File \"/usr/lib/python3.6/runpy.py\", line 85, in _run_code\r\n    exec(code, run_globals)\r\n  File \"/tmp/ansible_cron_payload_0y8i2pgj/ansible_cron_payload.zip/ansible/modules/system/cron.py\", line 770, in <module>\r\n  File \"/tmp/ansible_cron_payload_0y8i2pgj/ansible_cron_payload.zip/ansible/modules/system/cron.py\", line 617, in main\r\n  File \"/tmp/ansible_cron_payload_0y8i2pgj/ansible_cron_payload.zip/ansible/modules/system/cron.py\", line 248, in __init__\r\n  File \"/tmp/ansible_cron_payload_0y8i2pgj/ansible_cron_payload.zip/ansible/modules/system/cron.py\", line 264, in read\r\n__main__.CronTabError: ('Unexpected error:', <class 'UnicodeDecodeError'>)\r\n",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}
```

Hints from issue discussion:
Files identified in the description:
* [`lib/ansible/modules/system/cron.py`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/system/cron.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Files identified in the description:
* [`lib/ansible/modules/cron.py`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cron.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
I also faced this problem in our private project!

I used under these settings:

Server:

 - Python 3.6.9
 - Ubuntu 18.04 LTS

Client:
 - Python 3.8.2
 - ansible 2.8.7
 - Ubuntu 20.04

It looks `encoding` param is not specified when opening cron file:

https://github.com/ansible/ansible/blob/4c4406b2df3eb23ae0015067fa8c026777cd961e/lib/ansible/modules/cron.py#L255-L264

When `encoding` param is not specified, `locale.getpreferredencoding()` will be used, but it will be `ANSI_X3.4-1968`, which is one of alisases of `ascii`.

```
 % ansible -m shell -i hosts.ini <hostname> -a "python3 -c 'import locale; print(locale.getpreferredencoding())'"
<hostname> | CHANGED | rc=0 >>
ANSI_X3.4-1968
```

I also observed that locale will always be `C` even though I set it to `en_US.UTF-8` by hand. Is it expected?

```
% ansible -m shell -i hosts.ini <hostname> -a "locale"
<hostname> | CHANGED | rc=0 >>
LANG=C
LANGUAGE=
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C
```
This issue is still alive. Is there any news about a fix please ?
I have to disable task about cron tasks to have Ansible working.
@Cartman34 I found a workaround. It occurs when the server does not support your locale.
For example, I set my locale as `en_CA.UTF-8`, but the server does not support that locale.
After I set `LC_ALL` to `C.UTF-8` (by `export LC_ALL=C.UTF-8`), I could execute cron module correctly.

Please tell me your result!

Commit message: Also use non-ascii chars in replaced text and ensure not to break cron syntax.

Patch to review:
diff --git a/changelogs/fragments/70426-allow-non-ascii-chars-in-cron.yml b/changelogs/fragments/70426-allow-non-ascii-chars-in-cron.yml
new file mode 100644
index 00000000000000..65275d82dc0203
--- /dev/null
+++ b/changelogs/fragments/70426-allow-non-ascii-chars-in-cron.yml
@@ -0,0 +1,2 @@
+bugfixes:
+  - cron - encode and decode crontab files in UTF-8 explicitly to allow non-ascii chars in cron filepath and job (https://github.com/ansible/ansible/issues/69492)
diff --git a/lib/ansible/modules/cron.py b/lib/ansible/modules/cron.py
index 781a75facfb47c..f8af0e56a64820 100644
--- a/lib/ansible/modules/cron.py
+++ b/lib/ansible/modules/cron.py
@@ -210,6 +210,7 @@
 import tempfile
 
 from ansible.module_utils.basic import AnsibleModule
+from ansible.module_utils.common.text.converters import to_bytes, to_text
 from ansible.module_utils.six.moves import shlex_quote
 
 
@@ -236,9 +237,9 @@ def __init__(self, module, user=None, cron_file=None):
 
         if cron_file:
             if os.path.isabs(cron_file):
-                self.cron_file = cron_file
+                self.cron_file = to_text(cron_file, errors='surrogate_or_strict')
             else:
-                self.cron_file = os.path.join('/etc/cron.d', cron_file)
+                self.cron_file = os.path.join(u'/etc/cron.d', to_text(cron_file, errors='surrogate_or_strict'))
         else:
             self.cron_file = None
 
@@ -250,8 +251,8 @@ def read(self):
         if self.cron_file:
             # read the cronfile
             try:
-                f = open(self.cron_file, 'r')
-                self.existing = f.read()
+                f = open(self.cron_file, 'rb')
+                self.existing = to_text(f.read(), errors='surrogate_or_strict')
                 self.lines = self.existing.splitlines()
                 f.close()
             except IOError:
@@ -291,15 +292,15 @@ def write(self, backup_file=None):
         Write the crontab to the system. Saves all information.
         """
         if backup_file:
-            fileh = open(backup_file, 'w')
+            fileh = open(backup_file, 'wb')
         elif self.cron_file:
-            fileh = open(self.cron_file, 'w')
+            fileh = open(self.cron_file, 'wb')
         else:
             filed, path = tempfile.mkstemp(prefix='crontab')
             os.chmod(path, int('0644', 8))
-            fileh = os.fdopen(filed, 'w')
+            fileh = os.fdopen(filed, 'wb')
 
-        fileh.write(self.render())
+        fileh.write(to_bytes(self.render()))
         fileh.close()
 
         # return if making a backup
diff --git a/test/integration/targets/cron/tasks/main.yml b/test/integration/targets/cron/tasks/main.yml
index 155fc2d5809046..950d52dc4fcac7 100644
--- a/test/integration/targets/cron/tasks/main.yml
+++ b/test/integration/targets/cron/tasks/main.yml
@@ -122,3 +122,65 @@
 
   - assert:
       that: not cron_file_stats.stat.exists
+
+- name: Allow non-ascii chars in job (#69492)
+  block:
+  - name: Cron file creation
+    cron:
+      cron_file: cron_filename
+      name: "cron job that contain non-ascii chars in job (これは日本語です; This is Japanese)"
+      job: 'echo "うどんは好きだがお化け👻は苦手である。"'
+      user: root
+
+  - name: "Ensure cron_file contains job string"
+    replace:
+      path: /etc/cron.d/cron_filename
+      regexp: "うどんは好きだがお化け👻は苦手である。"
+      replace: "それは機密情報🔓です。"
+    register: find_chars
+    failed_when: (find_chars is not changed) or (find_chars is failed)
+
+  - name: Cron file deletion
+    cron:
+      cron_file: cron_filename
+      name: "cron job that contain non-ascii chars in job (これは日本語です; This is Japanese)"
+      state: absent
+
+  - name: Check file succesfull deletion
+    stat:
+      path: /etc/cron.d/cron_filename
+    register: cron_file_stats
+
+  - assert:
+      that: not cron_file_stats.stat.exists
+
+- name: Allow non-ascii chars in cron_file (#69492)
+  block:
+  - name: Cron file creation with non-ascii filename (これは日本語です; This is Japanese)
+    cron:
+      cron_file: 'なせば大抵なんとかなる👊'
+      name: "integration test cron"
+      job: 'echo "Hello, ansible!"'
+      user: root
+
+  - name: Check file exists
+    stat:
+      path: "/etc/cron.d/なせば大抵なんとかなる👊"
+    register: cron_file_stats
+
+  - assert:
+      that: cron_file_stats.stat.exists
+
+  - name: Cron file deletion
+    cron:
+      cron_file: 'なせば大抵なんとかなる👊'
+      name: "integration test cron"
+      state: absent
+
+  - name: Check file succesfull deletion
+    stat:
+      path: "/etc/cron.d/なせば大抵なんとかなる👊"
+    register: cron_file_stats
+
+  - assert:
+      that: not cron_file_stats.stat.exists


Code review comments:
diff_hunk: @@ -291,15 +292,15 @@ def write(self, backup_file=None):
         Write the crontab to the system. Saves all information.
         """
         if backup_file:
-            fileh = open(backup_file, 'w')
+            fileh = open(backup_file, 'wb')
         elif self.cron_file:
-            fileh = open(self.cron_file, 'w')
+            fileh = open(self.cron_file, 'wb')
path: lib/ansible/modules/cron.py
text: @user1:
```suggestion
            fileh = open(self.b_cron_file, 'wb')
```

diff_hunk: @@ -236,9 +237,9 @@ def __init__(self, module, user=None, cron_file=None):
 
         if cron_file:
             if os.path.isabs(cron_file):
-                self.cron_file = cron_file
+                self.cron_file = to_text(cron_file, errors='surrogate_or_strict')
             else:
-                self.cron_file = os.path.join('/etc/cron.d', cron_file)
+                self.cron_file = os.path.join(u'/etc/cron.d', to_text(cron_file, errors='surrogate_or_strict'))
path: lib/ansible/modules/cron.py
text: @user1:
Same here.
```suggestion
                self.cron_file = os.path.join('/etc/cron.d', cron_file)
                self.b_cron_file = os.path.join(b'/etc/cron.d', to_bytes(cron_file, errors='surrogate_or_strict'))
```

diff_hunk: @@ -236,9 +237,9 @@ def __init__(self, module, user=None, cron_file=None):
 
         if cron_file:
             if os.path.isabs(cron_file):
-                self.cron_file = cron_file
+                self.cron_file = to_text(cron_file, errors='surrogate_or_strict')
path: lib/ansible/modules/cron.py
text: @user1:
We should use bytes with file paths since the text type will vary between Python 2 (`unicode`) and Python 3 (`str`). We also prefix byte strings with `b_` as a convention.
```suggestion
                self.cron_file = cron_file
                self.b_cron_file = to_bytes(cron_file, errors='surrogate_or_strict')
```

This likely means there are other places that should use the bytes version of the path that need to be updated.

diff_hunk: @@ -250,8 +251,8 @@ def read(self):
         if self.cron_file:
             # read the cronfile
             try:
-                f = open(self.cron_file, 'r')
-                self.existing = f.read()
+                f = open(self.cron_file, 'rb')
+                self.existing = to_text(f.read(), errors='surrogate_or_strict')
                 self.lines = self.existing.splitlines()
path: lib/ansible/modules/cron.py
text: @user1:
Convert this to the native type, not text.
```suggestion
                f = open(self.b_cron_file, 'rb')
                self.b_existing = to_native(f.read(), errors='surrogate_or_strict')
```

Ideally, this would be `bytes` and we would work with the data in bytes throughout the module. But there are many places using `self.lines` that need the native text type, so the least invasive change is to convert at the borders: read in the lines as bytes then immediately convert to native text type. Lastly, write the file out as bytes.

Also, we like to prefix native strings with `n_` but that would result in similar ripple effect as using bytes, having to change the variable name throughout the module. I suppose we can omit the name change for now.

@author:
@user1 

> Also, we like to prefix native strings with n_ but that would result in similar ripple effect as using bytes, having to change the variable name throughout the module. I suppose we can omit the name change for now.

So... `self.existing` should be `self.n_existing`, instead of `self.b_existing`, shouldn't be?

I renamed to `self.n_existing`. Please check.


@user1:
Oops, that is correct.