Repository: ansible/ansible
Pull request: Preflight encodings check
##### SUMMARY
Preflight encodings check. Fixes #77658

##### ISSUE TYPE
- Feature Pull Request

##### COMPONENT NAME
lib/ansible/cli/__init__.py

##### ADDITIONAL INFORMATION
<!--- Include additional information to help people understand the change here -->
<!--- A step-by-step reproduction of the problem is helpful if there is no related issue -->

<!--- Paste verbatim command output below, e.g. before and after your change -->
```paste below

```


Problem statement:
Implement pre-flight checks for locale and fsencoding
### Summary

Add preflight checks to `lib/ansible/cli/__init__.py` for both the local locale and the fsencoding to ensure they are `UTF-8`, and abort when they are not.

We implicitly require this now, but have never strictly enforced it. Due to using `pathlib` in more places, to avoid undefined behavior in our codebase, due to the utf-8 expectation, check this early and fail fast.

### Issue Type

Feature Idea

### Component Name
```
lib/ansible/cli/__init__.py
```
### Additional Information

We may also need to evaluate the `C` locale.  I know some users are running ansible from devices with only the `C` locale.

### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct

Hints from issue discussion:
Files identified in the description:
* [`lib/ansible/cli/__init__.py`](https://github.com/ansible/ansible/blob/devel/lib/ansible/cli/__init__.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->

Commit message: Remove inspection of LOCALE_INITIALIZED in get_text_width

Patch to review:
diff --git a/changelogs/fragments/ansible-require-utf8.yml b/changelogs/fragments/ansible-require-utf8.yml
new file mode 100644
index 00000000000000..ee5a4198359650
--- /dev/null
+++ b/changelogs/fragments/ansible-require-utf8.yml
@@ -0,0 +1,6 @@
+major_changes:
+  - ansible - At startup the filesystem encoding and locale are checked to verify they are UTF-8.
+              If not, the process exits with an error reporting the errant encoding.
+breaking_changes:
+  - ansible - At startup the filesystem encoding and locale are checked to verify they are UTF-8.
+              If not, the process exits with an error reporting the errant encoding.
diff --git a/docs/docsite/rst/porting_guides/porting_guide_core_2.14.rst b/docs/docsite/rst/porting_guides/porting_guide_core_2.14.rst
index 6e0cf9bca0d3d8..3f3e644d03dabe 100644
--- a/docs/docsite/rst/porting_guides/porting_guide_core_2.14.rst
+++ b/docs/docsite/rst/porting_guides/porting_guide_core_2.14.rst
@@ -37,7 +37,7 @@ Playbook
 Command Line
 ============
 
-No notable changes
+* At startup the filesystem encoding and locale are checked to verify they are UTF-8. If not, the process exits with an error reporting the errant encoding. If you were previously using the ``C`` or ``POSIX`` locale, you may be able to use ``C.UTF-8``. If you were previously using a locale such as ``en_US.ISO-8859-1``, you may be able to use ``en_US.UTF-8``. For simplicity it may be easiest to export the appropriate locale using the ``LC_ALL`` environment variable.
 
 
 Deprecated
diff --git a/lib/ansible/cli/__init__.py b/lib/ansible/cli/__init__.py
index 731033fcf288cc..0bd0dcb48a78be 100644
--- a/lib/ansible/cli/__init__.py
+++ b/lib/ansible/cli/__init__.py
@@ -40,6 +40,28 @@ def check_blocking_io():
 
 check_blocking_io()
 
+
+def check_encoding():
+    """Check file system encoding and locale to ensure UTF-8."""
+    import ansible.utils.locale as locale_utils
+    fs_enc = sys.getfilesystemencoding()
+    if fs_enc.lower() != 'utf-8':
+        raise SystemExit('ERROR: Ansible requires the system encoding to be UTF-8; Detected %s.' % fs_enc)
+
+    locale, encoding = locale_utils.initialize_locale()
+
+    if not locale_utils.LOCALE_INITIALIZED:
+        raise SystemExit(
+            'ERROR: Ansible could not initialize the preferred locale: %s' % locale_utils.LOCALE_INITIALIZATION_ERR
+        )
+
+    if not encoding or encoding.lower() not in ('utf-8', 'utf8'):
+        raise SystemExit('ERROR: Ansible requires the locale encoding to be UTF-8; Detected %s.' % encoding)
+
+
+check_encoding()
+
+
 from importlib.metadata import version
 from ansible.module_utils.compat.version import LooseVersion
 
@@ -61,8 +83,7 @@ def check_blocking_io():
 
 try:
     from ansible import constants as C
-    from ansible.utils.display import Display, initialize_locale
-    initialize_locale()
+    from ansible.utils.display import Display
     display = Display()
 except Exception as e:
     print('ERROR: %s' % e, file=sys.stderr)
diff --git a/lib/ansible/utils/__init__.py b/lib/ansible/utils/__init__.py
index ae8ccff5952585..45500ec0fab42b 100644
--- a/lib/ansible/utils/__init__.py
+++ b/lib/ansible/utils/__init__.py
@@ -15,6 +15,9 @@
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
+# NOTE: This file must remain empty, to avoid side effects when importing
+#       ``ansible.utils.locale``
+
 # Make coding more python3-ish
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
diff --git a/lib/ansible/utils/display.py b/lib/ansible/utils/display.py
index b11998fe584fc1..a26cab27021d1e 100644
--- a/lib/ansible/utils/display.py
+++ b/lib/ansible/utils/display.py
@@ -53,24 +53,6 @@
 # Max for c_int
 _MAX_INT = 2 ** (ctypes.sizeof(ctypes.c_int) * 8 - 1) - 1
 
-_LOCALE_INITIALIZED = False
-_LOCALE_INITIALIZATION_ERR = None
-
-
-def initialize_locale():
-    """Set the locale to the users default setting
-    and set ``_LOCALE_INITIALIZED`` to indicate whether
-    ``get_text_width`` may run into trouble
-    """
-    global _LOCALE_INITIALIZED, _LOCALE_INITIALIZATION_ERR
-    if _LOCALE_INITIALIZED is False:
-        try:
-            locale.setlocale(locale.LC_ALL, '')
-        except locale.Error as e:
-            _LOCALE_INITIALIZATION_ERR = e
-        else:
-            _LOCALE_INITIALIZED = True
-
 
 def get_text_width(text):
     """Function that utilizes ``wcswidth`` or ``wcwidth`` to determine the
@@ -86,19 +68,6 @@ def get_text_width(text):
     if not isinstance(text, text_type):
         raise TypeError('get_text_width requires text, not %s' % type(text))
 
-    if _LOCALE_INITIALIZATION_ERR:
-        Display().warning(
-            'An error occurred while calling ansible.utils.display.initialize_locale '
-            '(%s). This may result in incorrectly calculated text widths that can '
-            'cause Display to print incorrect line lengths' % _LOCALE_INITIALIZATION_ERR
-        )
-    elif not _LOCALE_INITIALIZED:
-        Display().warning(
-            'ansible.utils.display.initialize_locale has not been called, '
-            'this may result in incorrectly calculated text widths that can '
-            'cause Display to print incorrect line lengths'
-        )
-
     try:
         width = _LIBC.wcswidth(text, _MAX_INT)
     except ctypes.ArgumentError:
@@ -130,10 +99,9 @@ def get_text_width(text):
             w = 0
         width += w
 
-    if width == 0 and counter and not _LOCALE_INITIALIZED:
+    if width == 0 and counter:
         raise EnvironmentError(
-            'ansible.utils.display.initialize_locale has not been called, '
-            'and get_text_width could not calculate text width of %r' % text
+            'get_text_width could not calculate text width of %r' % text
         )
 
     # It doesn't make sense to have a negative printable width
diff --git a/lib/ansible/utils/locale.py b/lib/ansible/utils/locale.py
new file mode 100644
index 00000000000000..13670320e185dd
--- /dev/null
+++ b/lib/ansible/utils/locale.py
@@ -0,0 +1,32 @@
+# Copyright: Contributors to the Ansible project
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+# Make coding more python3-ish
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+
+import locale
+
+
+LOCALE_INITIALIZED = False
+LOCALE_INITIALIZATION_ERR = None
+LOCALE = (None, None)
+
+
+def initialize_locale():
+    """Set the locale to the users default setting
+    and set ``LOCALE_INITIALIZED`` to indicate whether
+    ``get_text_width`` may run into trouble
+    """
+    global LOCALE, LOCALE_INITIALIZED, LOCALE_INITIALIZATION_ERR
+    if LOCALE_INITIALIZED is False:
+        try:
+            locale.setlocale(locale.LC_ALL, '')
+            LOCALE = locale.getlocale()
+        except locale.Error as e:
+            LOCALE_INITIALIZATION_ERR = e
+        else:
+            LOCALE_INITIALIZED = True
+            return LOCALE
+    return LOCALE
diff --git a/test/integration/targets/connection/test.sh b/test/integration/targets/connection/test.sh
index ad672e23c64625..6e16a87ea7bcd2 100755
--- a/test/integration/targets/connection/test.sh
+++ b/test/integration/targets/connection/test.sh
@@ -4,10 +4,7 @@ set -eux
 
 [ -f "${INVENTORY}" ]
 
-# Run connection tests with both the default and C locale.
-
-                ansible-playbook test_connection.yml -i "${INVENTORY}" "$@"
-LC_ALL=C LANG=C ansible-playbook test_connection.yml -i "${INVENTORY}" "$@"
+ansible-playbook test_connection.yml -i "${INVENTORY}" "$@"
 
 # Check that connection vars do not appear in the output
 # https://github.com/ansible/ansible/pull/70853
diff --git a/test/units/utils/test_display.py b/test/units/utils/test_display.py
index f0a6b6eefbb081..39aabd5fe6ca03 100644
--- a/test/units/utils/test_display.py
+++ b/test/units/utils/test_display.py
@@ -10,7 +10,8 @@
 import pytest
 
 from ansible.module_utils.six import PY3
-from ansible.utils.display import Display, get_text_width, initialize_locale
+from ansible.utils.display import Display, get_text_width
+from ansible.utils.locale import initialize_locale
 from ansible.utils.multiprocessing import context as multiprocessing_context
 
 


Code review comments:
diff_hunk: @@ -40,6 +40,28 @@ def check_blocking_io():
 
 check_blocking_io()
 
+
+def check_encoding():
+    """Check file system encoding and locale to ensure UTF-8."""
+    import ansible.utils.locale as locale_utils
+    fs_enc = sys.getfilesystemencoding()
+    if fs_enc.lower() != 'utf-8':
+        raise SystemExit('ERROR: Ansible requires the system encoding to be UTF-8; Detected %s.' % fs_enc)
path: lib/ansible/cli/__init__.py
text: @user1:
I missed this one in my last review:

```suggestion
        raise SystemExit('ERROR: Ansible requires the filesystem encoding to be UTF-8; Detected %s.' % fs_enc)
```