Repository: ansible/ansible

Issue:
Ansible python centos 7.2 encoding error utf-8
##### ISSUE TYPE
- Bug Report
##### COMPONENT NAME
- Ansible
##### ANSIBLE VERSION

ansible 2.1.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
##### CONFIGURATION

Default configuration
ansible 2.1.1.0
ansible 2.1.1.0
##### OS / ENVIRONMENT

cat /etc/*release
CentOS Linux release 7.2.1511 (Core)
##### SUMMARY

We have multiples playbook, since the last update of python, we are not able to run any of our playbooks anymore (was working before the update of Sep.08), the following error occurs: 

```
ERROR! 'utf8' codec can't decode byte 0xe9 in position 618: invalid continuation byte
'utf8' codec can't decode byte 0xe9 in position 618: invalid continuation byte
```
##### STEPS TO REPRODUCE
- Centos 7.2
  Last updates of python:
- python-libs-2.7.5-38.el7_2.x86_64
- python-2.7.5-38.el7_2.x86_64
- python-perf-3.10.0-327.28.3.el7.x86_64
- ansible 2.1.1
- try to execute any type of playbook

```
# ansible-playbook nagios.yml
ERROR! 'utf8' codec can't decode byte 0xe9 in position 618: invalid continuation byte
'utf8' codec can't decode byte 0xe9 in position 618: invalid continuation byte

```



Discussion:
With the new update of today from python and python lib the issue is still present :
python 2.7.5-39.el7_2 
python-libs 2.7.5-39.el7_2
Python 2.7.5 (default, Sep 15 2016, 22:37:39)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2

Can you share one of the playbook ?

Please see below: 

https://www.dropbox.com/s/vykt0qw4ulodyow/nagios.zip?dl=0

Did you check that your current directory doesn't contain any accent or special character?

Edit: by current directory, I mean more exactly the entire path of your current directory

Hello the entire path is: 

/etc/ansible/playbooks

I don't think this is the issue, but might let us know. ;)

@dbayle we need more information in order to reproduce your problem.  Are you able to give us a simple playbook that reproduces your issue as @mscherer asked for?

needs_info

Greetings, the playbook my colleague sent you is one of none working playbook.
https://www.dropbox.com/s/vykt0qw4ulodyow/nagios.zip?dl=0

BTW all playbooks are not working as said before.
I m pretty sure it s an issue related to latest Python updates on CentOS 7.2

Any feedback on this ?

I am unable to reproduce this with your playboook.  UnicodeError is usually the result of mixing non-ascii strings improperly in python. But that means there needs to be a source of the non-ascii characters.  I do not see any non-ascii in your playbooks.  Are there some coming from somewhere else?  Perhaps the directory that you're running ansible inside of?  I have recently fixed a bug in devel (will be 2.2) where running ansible with playbooks in non-ascii directories could have raised an error.  Is that the scenario that you are in?

Hmm... I see that you already said the playbook path was /etc/ansible/playbooks so that isn't it... Can you figure out where non-ascii characters would be coming from, though?

I'm in the same position as you are, I don't have any clue where those unicodeerrors are coming from.
The only thing we can let you know is the fact that all our playbooks stopped working from one day to another and the only change were the python updates on CentOS 7.2.

I'm still investigating on a way to solve this, I will try to copy our playbooks elsewhere but I'm not convinced about the result.

Could be from anything in the environment, could you share it (ie, the output of 'env') ?

Waht do you mean, what are you thinking of ?

[root@**************\* ~]# env
XDG_SESSION_ID=3275
HOSTNAME=*****************
TERM=xterm
SHELL=/bin/bash
HISTSIZE=1000
SSH_CLIENT=*****************
SSH_TTY=/dev/pts/0
USER=root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:_.tar=01;31:_.tgz=01;31:_.arc=01;31:_.arj=01;31:_.taz=01;31:_.lha=01;31:_.lz4=01;31:_.lzh=01;31:_.lzma=01;31:_.tlz=01;31:_.txz=01;31:_.tzo=01;31:_.t7z=01;31:_.zip=01;31:_.z=01;31:_.Z=01;31:_.dz=01;31:_.gz=01;31:_.lrz=01;31:_.lz=01;31:_.lzo=01;31:_.xz=01;31:_.bz2=01;31:_.bz=01;31:_.tbz=01;31:_.tbz2=01;31:_.tz=01;31:_.deb=01;31:_.rpm=01;31:_.jar=01;31:_.war=01;31:_.ear=01;31:_.sar=01;31:_.rar=01;31:_.alz=01;31:_.ace=01;31:_.zoo=01;31:_.cpio=01;31:_.7z=01;31:_.rz=01;31:_.cab=01;31:_.jpg=01;35:_.jpeg=01;35:_.gif=01;35:_.bmp=01;35:_.pbm=01;35:_.pgm=01;35:_.ppm=01;35:_.tga=01;35:_.xbm=01;35:_.xpm=01;35:_.tif=01;35:_.tiff=01;35:_.png=01;35:_.svg=01;35:_.svgz=01;35:_.mng=01;35:_.pcx=01;35:_.mov=01;35:_.mpg=01;35:_.mpeg=01;35:_.m2v=01;35:_.mkv=01;35:_.webm=01;35:_.ogm=01;35:_.mp4=01;35:_.m4v=01;35:_.mp4v=01;35:_.vob=01;35:_.qt=01;35:_.nuv=01;35:_.wmv=01;35:_.asf=01;35:_.rm=01;35:_.rmvb=01;35:_.flc=01;35:_.avi=01;35:_.fli=01;35:_.flv=01;35:_.gl=01;35:_.dl=01;35:_.xcf=01;35:_.xwd=01;35:_.yuv=01;35:_.cgm=01;35:_.emf=01;35:_.axv=01;35:_.anx=01;35:_.ogv=01;35:_.ogx=01;35:_.aac=01;36:_.au=01;36:_.flac=01;36:_.mid=01;36:_.midi=01;36:_.mka=01;36:_.mp3=01;36:_.mpc=01;36:_.ogg=01;36:_.ra=01;36:_.wav=01;36:_.axa=01;36:_.oga=01;36:_.spx=01;36:_.xspf=01;36:
SSH_AUTH_SOCK=/tmp/ssh-ZlNGa7S6uQ/_*********
MAIL=/var/spool/mail/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
PWD=/root
LANG=en_US.UTF-8
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
LOGNAME=root
SSH_CONNECTION=*********************************
LESSOPEN=||/usr/bin/lesspipe.sh %s
XDG_RUNTIME_DIR=/run/user/0
HISTTIMEFORMAT=%F %T
_=/usr/bin/env

Greetings, we found the issue on the Ansible server, someone in our team left a comment with special characters in /etc/ansible/hosts file. Once this comment was removed, all playbook and ansible worked fine.

so in /etc/ansible/hosts:

# Comment with special char░ in there

Even as a comment it was producing all these errors.
Thank you for your help, hope this feedback can help.

Regards.

Thanks @dbayle.  i'll take a look and see if there's something we can do to mitigate that.


Resolving patch:
diff --git a/lib/ansible/inventory/dir.py b/lib/ansible/inventory/dir.py
--- a/lib/ansible/inventory/dir.py
+++ b/lib/ansible/inventory/dir.py
@@ -45,11 +45,10 @@ def get_file_parser(hostsfile, groups, loader):
     parser = None
 
     try:
-        inv_file = open(hostsfile)
-        first_line = inv_file.readlines()[0]
-        inv_file.close()
-        if first_line.startswith('#!'):
-            shebang_present = True
+        with open(hostsfile, 'rb') as inv_file:
+            initial_chars = inv_file.read(2)
+            if initial_chars.startswith(b'#!'):
+                shebang_present = True
     except:
         pass
 
diff --git a/lib/ansible/inventory/ini.py b/lib/ansible/inventory/ini.py
--- a/lib/ansible/inventory/ini.py
+++ b/lib/ansible/inventory/ini.py
@@ -40,7 +40,6 @@ class InventoryParser(object):
     """
 
     def __init__(self, loader, groups, filename=C.DEFAULT_HOST_LIST):
-        self._loader = loader
         self.filename = filename
 
         # Start with an empty host list and whatever groups we're passed in
@@ -53,12 +52,17 @@ def __init__(self, loader, groups, filename=C.DEFAULT_HOST_LIST):
         # Read in the hosts, groups, and variables defined in the
         # inventory file.
 
-        if loader:
-            (data, private) = loader._get_file_contents(filename)
-        else:
-            with open(filename) as fh:
-                data = to_text(fh.read())
-        data = data.split('\n')
+        with open(filename, 'rb') as fh:
+            data = fh.read()
+            try:
+                # Faster to do to_text once on a long string than many
+                # times on smaller strings
+                data = to_text(data, errors='surrogate_or_strict')
+                data = [line for line in data.splitlines() if not (line.startswith(u';') or line.startswith(u'#'))]
+            except UnicodeError:
+                # Skip comment lines here to avoid potential undecodable
+                # errors in comments: https://github.com/ansible/ansible/issues/17593
+                data = [to_text(line, errors='surrogate_or_strict') for line in data.splitlines() if not (line.startswith(b';') or line.startswith(b'#'))]
 
         self._parse(data)
 
@@ -88,8 +92,8 @@ def _parse(self, lines):
 
             line = line.strip()
 
-            # Skip empty lines and comments
-            if line == '' or line.startswith(";") or line.startswith("#"):
+            # Skip empty lines
+            if not line:
                 continue
 
             # Is this a [section] header? That tells us what group we're parsing


Test patch:
diff --git a/test/units/vars/test_variable_manager.py b/test/units/vars/test_variable_manager.py
--- a/test/units/vars/test_variable_manager.py
+++ b/test/units/vars/test_variable_manager.py
@@ -20,17 +20,20 @@
 __metaclass__ = type
 
 from collections import defaultdict
-from six import iteritems
 
+from ansible.compat.six import iteritems
+from ansible.compat.six.moves import builtins
 from ansible.compat.tests import unittest
-from ansible.compat.tests.mock import patch, MagicMock
+from ansible.compat.tests.mock import MagicMock, mock_open, patch
 from ansible.inventory import Inventory
 from ansible.playbook.play import Play
-from ansible.vars import VariableManager
 
 from units.mock.loader import DictDataLoader
 from units.mock.path import mock_unfrackpath_noop
 
+from ansible.vars import VariableManager
+
+
 class TestVariableManager(unittest.TestCase):
 
     def setUp(self):
@@ -192,9 +195,7 @@ def test_variable_manager_precedence(self, mock_basedir):
         v = VariableManager()
         v._fact_cache = defaultdict(dict)
 
-        fake_loader = DictDataLoader({
-            # inventory1
-            '/etc/ansible/inventory1': """
+        inventory1_filedata = """
             [group2:children]
             group1
 
@@ -206,8 +207,11 @@ def test_variable_manager_precedence(self, mock_basedir):
 
             [group2:vars]
             group_var = group_var_from_inventory_group2
-            """,
+            """
 
+        fake_loader = DictDataLoader({
+            # inventory1
+            '/etc/ansible/inventory1': inventory1_filedata,
             # role defaults_only1
             '/etc/ansible/roles/defaults_only1/defaults/main.yml': """
             default_var: "default_var_from_defaults_only1"
@@ -231,7 +235,8 @@ def test_variable_manager_precedence(self, mock_basedir):
         })
 
         mock_basedir.return_value = './'
-        inv1 = Inventory(loader=fake_loader, variable_manager=v, host_list='/etc/ansible/inventory1')
+        with patch.object(builtins, 'open', mock_open(read_data=inventory1_filedata)):
+            inv1 = Inventory(loader=fake_loader, variable_manager=v, host_list='/etc/ansible/inventory1')
         inv1.set_playbook_basedir('./')
 
         play1 = Play.load(dict(