Repository: ansible/ansible

Issue:
apt module is not idempotent when locale is not English
##### Issue Type:

Bug Report
##### Ansible Version:

```
ansible 2.0.0 (devel 9f1cd0f42a) last updated 2015/05/22 20:12:34 (GMT +200)
  lib/ansible/modules/core: (detached HEAD 150b71f11a) last updated 2015/05/22 20:13:13 (GMT +200)
  lib/ansible/modules/extras: (detached HEAD 5187c7fcd7) last updated 2015/05/22 20:13:13 (GMT +200)
  v2/ansible/modules/core: (detached HEAD 85c8a892c8) last updated 2015/05/22 20:13:13 (GMT +200)
  v2/ansible/modules/extras: (detached HEAD 70ea058563) last updated 2015/05/22 20:13:13 (GMT +200)
  configured module search path = None
```
##### Environment:

What OS are you running Ansible from: Debian GNU/Linux testing (stretch)
What OS are you managing: Debian GNU/Linux 7.8 (wheezy)
##### Summary:

The `apt` module fails to detect idempotency when locale is not set to English.
##### Steps To Reproduce:

Playbook `test.yml` :

``` yaml
    - name: Upgrade packages
      apt: upgrade=yes update_cache=yes cache_valid_time=86400
      when: ansible_os_family == 'Debian'
```

When locale is set to English (I tested with both `C` and `en_US.UTF-8`), the idempotency is well detected (ie. the status is **ok**) :

```
LC_ALL=C ansible-playbook test.yml --limit server01
[...]
TASK: [Upgrade packages] ****************************************************** 
ok: [server01]
[...]
```

However, when locale is **not** English (confirmed with `fr_FR.UTF-8`), the idempotency fails, and the status is always **changed** :

```
LC_ALL=fr_FR.UTF-8 ansible-playbook test.yml --limit server01
[...]
TASK: [Upgrade packages] ****************************************************** 
changed: [server01]
[...]
```

Note that it's the same when using `ansible` (ie. not through a playbook):

```
LC_ALL=en_US.UTF-8 ansible 'server01' -m apt -a 'upgrade=yes update_cache=yes cache_valid_time=86400'
server01 | success >> {
    "changed": false,
    "msg": "Reading package lists... [...]

LC_ALL=fr_FR.UTF-8 ansible 'server01' -m apt -a 'upgrade=yes update_cache=yes cache_valid_time=86400'
server01 | success >> {
    "changed": true,
    "msg": "Lecture des listes de paquets... [...]
```

Isn't the parsing done in English only? If so, then shouldn't the locale be forced inside Ansible?

apt module is not idempotent when locale is not English
##### Issue Type:

Bug Report
##### Ansible Version:

```
ansible 2.0.0 (devel 9f1cd0f42a) last updated 2015/05/22 20:12:34 (GMT +200)
  lib/ansible/modules/core: (detached HEAD 150b71f11a) last updated 2015/05/22 20:13:13 (GMT +200)
  lib/ansible/modules/extras: (detached HEAD 5187c7fcd7) last updated 2015/05/22 20:13:13 (GMT +200)
  v2/ansible/modules/core: (detached HEAD 85c8a892c8) last updated 2015/05/22 20:13:13 (GMT +200)
  v2/ansible/modules/extras: (detached HEAD 70ea058563) last updated 2015/05/22 20:13:13 (GMT +200)
  configured module search path = None
```
##### Environment:

What OS are you running Ansible from: Debian GNU/Linux testing (stretch)
What OS are you managing: Debian GNU/Linux 7.8 (wheezy)
##### Summary:

The `apt` module fails to detect idempotency when locale is not set to English.
##### Steps To Reproduce:

Playbook `test.yml` :

``` yaml
    - name: Upgrade packages
      apt: upgrade=yes update_cache=yes cache_valid_time=86400
      when: ansible_os_family == 'Debian'
```

When locale is set to English (I tested with both `C` and `en_US.UTF-8`), the idempotency is well detected (ie. the status is **ok**) :

```
LC_ALL=C ansible-playbook test.yml --limit server01
[...]
TASK: [Upgrade packages] ****************************************************** 
ok: [server01]
[...]
```

However, when locale is **not** English (confirmed with `fr_FR.UTF-8`), the idempotency fails, and the status is always **changed** :

```
LC_ALL=fr_FR.UTF-8 ansible-playbook test.yml --limit server01
[...]
TASK: [Upgrade packages] ****************************************************** 
changed: [server01]
[...]
```

Note that it's the same when using `ansible` (ie. not through a playbook):

```
LC_ALL=en_US.UTF-8 ansible 'server01' -m apt -a 'upgrade=yes update_cache=yes cache_valid_time=86400'
server01 | success >> {
    "changed": false,
    "msg": "Reading package lists... [...]

LC_ALL=fr_FR.UTF-8 ansible 'server01' -m apt -a 'upgrade=yes update_cache=yes cache_valid_time=86400'
server01 | success >> {
    "changed": true,
    "msg": "Lecture des listes de paquets... [...]
```

Isn't the parsing done in English only? If so, then shouldn't the locale be forced inside Ansible?



Discussion:
I just merged #9636, which should help with this issue, please confirm

It may work with `LC_MESSAGES` for **yum**, but AFAIK, for **apt-get**, it requires to set the `LC_ALL` environment variable. I tried all `LC_*` variables, but only the `LC_ALL` one did changed apt-get language.

it might be worth adding LC_ALL on top of the changes in the PR above

You are right @bcoca. I do confirmed the below patch with `apt`. As far as I know, `LC_ALL` does apply to all other `LC_*`, but someone should test it against `yum` to confirm it still work:

``` diff
diff --git a/lib/ansible/module_utils/basic.py b/lib/ansible/module_utils/basic.py
index 99971d1..74ebf42 100644
--- a/lib/ansible/module_utils/basic.py
+++ b/lib/ansible/module_utils/basic.py
@@ -866,9 +866,8 @@ class AnsibleModule(object):
             # issues but is preferable to simply failing because
             # of an unknown locale
             locale.setlocale(locale.LC_ALL, 'C')
-            os.environ['LANG']     = 'C'
-            os.environ['LC_CTYPE'] = 'C'
-            os.environ['LC_MESSAGES'] = 'C'
+            os.environ['LANG']   = 'C'
+            os.environ['LC_ALL'] = 'C'
         except Exception, e:
             self.fail_json(msg="An unknown error was encountered while attempting to validate the locale: %s" % e)

diff --git a/lib/ansible/runner/shell_plugins/sh.py b/lib/ansible/runner/shell_plugins/sh.py
index 81810bc..1b0407e 100644
--- a/lib/ansible/runner/shell_plugins/sh.py
+++ b/lib/ansible/runner/shell_plugins/sh.py
@@ -30,9 +30,8 @@ class ShellModule(object):
     def env_prefix(self, **kwargs):
         '''Build command prefix with environment variables.'''
         env = dict(
-            LANG     = C.DEFAULT_MODULE_LANG,
-            LC_CTYPE = C.DEFAULT_MODULE_LANG,
-            LC_MESSAGES = C.DEFAULT_MODULE_LANG,
+            LANG   = C.DEFAULT_MODULE_LANG,
+            LC_ALL = C.DEFAULT_MODULE_LANG,
         )
         env.update(kwargs)
         return ' '.join(['%s=%s' % (k, pipes.quote(unicode(v))) for k,v in env.items()])
```

I did test it with CentOS and Red Hat machines, and I do confirm the above patch (ie. replacing all `LC_*` by `LC_ALL = C`) make Ansible works well from a localized host (`fr_FR.UTF-8` on mine).

Is it going to be merged?

submit a PR, right now there is nothing to merge

I just merged #9636, which should help with this issue, please confirm

It may work with `LC_MESSAGES` for **yum**, but AFAIK, for **apt-get**, it requires to set the `LC_ALL` environment variable. I tried all `LC_*` variables, but only the `LC_ALL` one did changed apt-get language.

it might be worth adding LC_ALL on top of the changes in the PR above

You are right @bcoca. I do confirmed the below patch with `apt`. As far as I know, `LC_ALL` does apply to all other `LC_*`, but someone should test it against `yum` to confirm it still work:

``` diff
diff --git a/lib/ansible/module_utils/basic.py b/lib/ansible/module_utils/basic.py
index 99971d1..74ebf42 100644
--- a/lib/ansible/module_utils/basic.py
+++ b/lib/ansible/module_utils/basic.py
@@ -866,9 +866,8 @@ class AnsibleModule(object):
             # issues but is preferable to simply failing because
             # of an unknown locale
             locale.setlocale(locale.LC_ALL, 'C')
-            os.environ['LANG']     = 'C'
-            os.environ['LC_CTYPE'] = 'C'
-            os.environ['LC_MESSAGES'] = 'C'
+            os.environ['LANG']   = 'C'
+            os.environ['LC_ALL'] = 'C'
         except Exception, e:
             self.fail_json(msg="An unknown error was encountered while attempting to validate the locale: %s" % e)

diff --git a/lib/ansible/runner/shell_plugins/sh.py b/lib/ansible/runner/shell_plugins/sh.py
index 81810bc..1b0407e 100644
--- a/lib/ansible/runner/shell_plugins/sh.py
+++ b/lib/ansible/runner/shell_plugins/sh.py
@@ -30,9 +30,8 @@ class ShellModule(object):
     def env_prefix(self, **kwargs):
         '''Build command prefix with environment variables.'''
         env = dict(
-            LANG     = C.DEFAULT_MODULE_LANG,
-            LC_CTYPE = C.DEFAULT_MODULE_LANG,
-            LC_MESSAGES = C.DEFAULT_MODULE_LANG,
+            LANG   = C.DEFAULT_MODULE_LANG,
+            LC_ALL = C.DEFAULT_MODULE_LANG,
         )
         env.update(kwargs)
         return ' '.join(['%s=%s' % (k, pipes.quote(unicode(v))) for k,v in env.items()])
```

I did test it with CentOS and Red Hat machines, and I do confirm the above patch (ie. replacing all `LC_*` by `LC_ALL = C`) make Ansible works well from a localized host (`fr_FR.UTF-8` on mine).

Is it going to be merged?

submit a PR, right now there is nothing to merge


Resolving patch:
diff --git a/lib/ansible/module_utils/basic.py b/lib/ansible/module_utils/basic.py
--- a/lib/ansible/module_utils/basic.py
+++ b/lib/ansible/module_utils/basic.py
@@ -735,8 +735,8 @@ def _check_locale(self):
             # issues but is preferable to simply failing because
             # of an unknown locale
             locale.setlocale(locale.LC_ALL, 'C')
-            os.environ['LANG']     = 'C'
-            os.environ['LC_CTYPE'] = 'C'
+            os.environ['LANG']   = 'C'
+            os.environ['LC_ALL'] = 'C'
         except Exception, e:
             self.fail_json(msg="An unknown error was encountered while attempting to validate the locale: %s" % e)
 
diff --git a/lib/ansible/runner/shell_plugins/sh.py b/lib/ansible/runner/shell_plugins/sh.py
--- a/lib/ansible/runner/shell_plugins/sh.py
+++ b/lib/ansible/runner/shell_plugins/sh.py
@@ -24,8 +24,8 @@ class ShellModule(object):
     def env_prefix(self, **kwargs):
         '''Build command prefix with environment variables.'''
         env = dict(
-            LANG     = C.DEFAULT_MODULE_LANG,
-            LC_CTYPE = C.DEFAULT_MODULE_LANG,
+            LANG   = C.DEFAULT_MODULE_LANG,
+            LC_ALL = C.DEFAULT_MODULE_LANG,
         )
         env.update(kwargs)
         return ' '.join(['%s=%s' % (k, pipes.quote(unicode(v))) for k,v in env.items()])


Test patch: