Repository: ansible/ansible

Issue:
ansible(-playbook) not support non ASCII chars as ansible-vault encrypted string
##### ISSUE TYPE
Bug Report

##### COMPONENT NAME
ansible-vault, ansible vault decryption

##### ANSIBLE VERSION
```
ansible 2.4.3.0
  config file = /opt/it-infra/ansible.cfg
  configured module search path = [u'/opt/it-infra/library']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
```

##### CONFIGURATION
ANSIBLE_NOCOWS(/opt/it-infra/ansible.cfg) = False
DEFAULT_CALLBACK_WHITELIST(/opt/it-infra/ansible.cfg) = ['profile_tasks', 'timer
DEFAULT_FILTER_PLUGIN_PATH(/opt/it-infra/ansible.cfg) = [u'/opt/it-infra/filter_
DEFAULT_MODULE_PATH(/opt/it-infra/ansible.cfg) = [u'/opt/it-infra/library']
DEFAULT_ROLES_PATH(/opt/it-infra/ansible.cfg) = [u'/opt/it-infra/roles']
DEFAULT_UNDEFINED_VAR_BEHAVIOR(/opt/it-infra/ansible.cfg) = True
DEFAULT_VAULT_PASSWORD_FILE(/opt/it-infra/ansible.cfg) = /opt/it-infra/vpass.py
DISPLAY_SKIPPED_HOSTS(/opt/it-infra/ansible.cfg) = False
HOST_KEY_CHECKING(/opt/it-infra/ansible.cfg) = False
RETRY_FILES_ENABLED(/opt/it-infra/ansible.cfg) = False


##### OS / ENVIRONMENT
Ubuntu 16.04 Managing multiple Windows Host

##### SUMMARY
I want to use a Password like follow  **aöffü** in an Hostvar file as var

When i encrypt the password with ansible-vault encrypt to get a hash for a variable
it is working ok, buth when ansible-playbook or ansible tryes to decrypt it on execution i get the Python error

**Looks like ansible-vault only supports ascii chars as source**

would be epic if we could use utf8 here, but prbly not this would be a breaking change for all

##### STEPS TO REPRODUCE

## generate a encrypted hash with ansible-vault

```bash
ansible-vault encrypt
````
Enter follow Keystrokes **aöffü**

Copy Encrypted String

<!--- Paste example playbooks or commands between quotes below -->

myhost.yml
```yaml
myvar: !vault | 
  $ANSIBLE_VAULT;1.1;AES256
  <Long AES256 Encoded Stuff>
```

playbook.yml
```yaml
- name: "DoSomeStuff"
  hosts: myhost
  tasks:
    - debug:
         var: myvar
```
##### EXPECTED RESULTS
No Python Exception

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
The full traceback is:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 130, in run
    res = self._execute()
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 528, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/action/debug.py", line 57, in run
    results = self._templar.template(self._task.args['var'], convert_bare=True, fail_on_undefined=True, bare_deprecated=False)
  File "/usr/local/lib/python2.7/dist-packages/ansible/template/__init__.py", line 450, in template
    disable_lookups=disable_lookups,
  File "/usr/local/lib/python2.7/dist-packages/ansible/template/__init__.py", line 672, in do_template
    res = j2_concat(rf)
  File "<template>", line 11, in root
  File "/usr/local/lib/python2.7/dist-packages/ansible/parsing/yaml/objects.py", line 114, in __repr__
    return repr(self.data)
  File "/usr/local/lib/python2.7/dist-packages/ansible/parsing/yaml/objects.py", line 107, in data
    return self.vault.decrypt(self._ciphertext).decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)
```



Discussion:
Files identified in the description:
* [lib/ansible/parsing/vault](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/vault)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
I can reproduce this with repro at https://github.com/alikins/ansible-bug-repro/tree/master/vault_unicode_37258 

It fails with python 2.7 with ansible 2.4/2.5/devel.
It works with python3.6 with ansible 2.4/2.5/devel.

playbook.yml
``` yaml
- name: "DoSomeStuff"
  gather_facts: false
  hosts: myhost
  tasks:
    - debug:
         var: myvar
```

host_vars/myhost.yml
``` yaml
myvar: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          31356335363836383937363933366135623233343830326234633633623734336636343630396464
          3234343638313166663237343536646336323862613739380a346266316336356230643838663031
          34623034383639323062373235356564393337346666393665313237313231306131356637346537
          3966393238666430310a363462326639323033653237373036643936613234623063643761663033
          3832
```


python 2.7 and ansible 2.4

```
[newswoop:F27:vault_unicode_37258 (master % u=)]$ ansible-playbook --vault-id password -vvvvv -i hosts playbook.yml 
ansible-playbook 2.4.4 (stable-2.4 02e591746c) last updated 2018/03/16 15:13:01 (GMT -400)
  config file = None
  configured module search path = [u'/home/adrian/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/adrian/src/ansible/lib/ansible
  executable location = /home/adrian/src/ansible/bin/ansible-playbook
  python version = 2.7.14 (default, Feb 27 2018, 20:43:24) [GCC 7.3.1 20180130 (Red Hat 7.3.1-2)]
No config file found; using defaults
Reading vault password file: password
setting up inventory plugins
Set default localhost to localhost
Parsed /home/adrian/src/ansible-bug-repro/vault_unicode_37258/hosts inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /home/adrian/src/ansible/lib/ansible/plugins/callback/default.pyc

PLAYBOOK: playbook.yml **************************************************************************
1 plays in playbook.yml

PLAY [DoSomeStuff] ******************************************************************************
META: ran handlers

TASK [debug] ************************************************************************************
task path: /home/adrian/src/ansible-bug-repro/vault_unicode_37258/playbook.yml:5
Found a vault_id (default) in the vaulttext
We have a secret associated with vault id (default), will try to use to decrypt None
Trying to use vault secret=(FileVaultSecret(filename='/home/adrian/src/ansible-bug-repro/vault_unicode_37258/password')) id=default to decrypt None
Trying secret FileVaultSecret(filename='/home/adrian/src/ansible-bug-repro/vault_unicode_37258/password') for vault_id=default
decrypt succesful with secret=FileVaultSecret(filename='/home/adrian/src/ansible-bug-repro/vault_unicode_37258/password') and vault_id=default
The full traceback is:
Traceback (most recent call last):
  File "/home/adrian/src/ansible/lib/ansible/executor/task_executor.py", line 148, in run
    res = self._execute()
  File "/home/adrian/src/ansible/lib/ansible/executor/task_executor.py", line 551, in _execute
    result = self._handler.run(task_vars=variables)
  File "/home/adrian/src/ansible/lib/ansible/plugins/action/debug.py", line 57, in run
    results = self._templar.template(self._task.args['var'], convert_bare=True, fail_on_undefined=True, bare_deprecated=False)
  File "/home/adrian/src/ansible/lib/ansible/template/__init__.py", line 450, in template
    disable_lookups=disable_lookups,
  File "/home/adrian/src/ansible/lib/ansible/template/__init__.py", line 672, in do_template
    res = j2_concat(rf)
  File "<template>", line 11, in root
  File "/home/adrian/src/ansible/lib/ansible/parsing/yaml/objects.py", line 134, in __unicode__
    return to_text(self.data, errors='surrogate_or_strict')
  File "/home/adrian/src/ansible/lib/ansible/parsing/yaml/objects.py", line 107, in data
    return self.vault.decrypt(self._ciphertext).decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

fatal: [myhost]: FAILED! => {
    "msg": "Unexpected failure during module execution.", 
    "stdout": ""
}
	to retry, use: --limit @/home/adrian/src/ansible-bug-repro/vault_unicode_37258/playbook.retry

PLAY RECAP **************************************************************************************
myhost                     : ok=0    changed=0    unreachable=0    failed=1   
```

python 3.6 and ansible 2.4

```
[newswoop:F27:vault_unicode_37258 (master % u=)]$ python3.6 ~/src/ansible/bin/ansible-playbook --vault-id password -vvvvv -i hosts playbook.yml 
ansible-playbook 2.4.4 (stable-2.4 02e591746c) last updated 2018/03/16 15:13:01 (GMT -400)
  config file = None
  configured module search path = ['/home/adrian/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/adrian/src/ansible/lib/ansible
  executable location = /home/adrian/src/ansible/bin/ansible-playbook
  python version = 3.6.4 (default, Feb  8 2018, 14:42:51) [GCC 7.3.1 20180130 (Red Hat 7.3.1-2)]
No config file found; using defaults
Reading vault password file: password
setting up inventory plugins
Set default localhost to localhost
Parsed /home/adrian/src/ansible-bug-repro/vault_unicode_37258/hosts inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /home/adrian/src/ansible/lib/ansible/plugins/callback/default.py

PLAYBOOK: playbook.yml **************************************************************************
1 plays in playbook.yml

PLAY [DoSomeStuff] ******************************************************************************
META: ran handlers

TASK [debug] ************************************************************************************
task path: /home/adrian/src/ansible-bug-repro/vault_unicode_37258/playbook.yml:5
Found a vault_id (default) in the vaulttext
We have a secret associated with vault id (default), will try to use to decrypt None
Trying to use vault secret=(FileVaultSecret(filename='/home/adrian/src/ansible-bug-repro/vault_unicode_37258/password')) id=default to decrypt None
Trying secret FileVaultSecret(filename='/home/adrian/src/ansible-bug-repro/vault_unicode_37258/password') for vault_id=default
decrypt succesful with secret=FileVaultSecret(filename='/home/adrian/src/ansible-bug-repro/vault_unicode_37258/password') and vault_id=default
ok: [myhost] => {
    "myvar": "aöffü"
}
META: ran handlers
META: ran handlers

PLAY RECAP **************************************************************************************
myhost                     : ok=1    changed=0    unreachable=0    failed=0   
```

Resolving patch:
diff --git a/lib/ansible/parsing/yaml/objects.py b/lib/ansible/parsing/yaml/objects.py
--- a/lib/ansible/parsing/yaml/objects.py
+++ b/lib/ansible/parsing/yaml/objects.py
@@ -104,7 +104,7 @@ def data(self):
         if not self.vault:
             # FIXME: raise exception?
             return self._ciphertext
-        return self.vault.decrypt(self._ciphertext).decode()
+        return to_text(self.vault.decrypt(self._ciphertext))
 
     @data.setter
     def data(self, value):


Test patch:
diff --git a/test/integration/targets/vault/host_vars/myhost.yml b/test/integration/targets/vault/host_vars/myhost.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/vault/host_vars/myhost.yml
@@ -0,0 +1,7 @@
+myvar: !vault |
+          $ANSIBLE_VAULT;1.1;AES256
+          31356335363836383937363933366135623233343830326234633633623734336636343630396464
+          3234343638313166663237343536646336323862613739380a346266316336356230643838663031
+          34623034383639323062373235356564393337346666393665313237313231306131356637346537
+          3966393238666430310a363462326639323033653237373036643936613234623063643761663033
+          3832
diff --git a/test/integration/targets/vault/host_vars/testhost.yml b/test/integration/targets/vault/host_vars/testhost.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/vault/host_vars/testhost.yml
@@ -0,0 +1,7 @@
+vaulted_utf8_value: !vault |
+          $ANSIBLE_VAULT;1.1;AES256
+          39313961356631343234656136636231663539363963386364653436346133366366633031366364
+          3332376636333837333036633662316135383365343335380a393331663434663238666537343163
+          62363561336431623666633735313766613663333736653064373632666131356434336537383336
+          3333343436613232330a643461363831633166333237653530353131316361643465353132616362
+          3461
diff --git a/test/integration/targets/vault/runme.sh b/test/integration/targets/vault/runme.sh
--- a/test/integration/targets/vault/runme.sh
+++ b/test/integration/targets/vault/runme.sh
@@ -359,7 +359,7 @@ echo "rc was $WRONG_RC (5 is expected)"
 
 # encrypt with a password from a vault encrypted password file and multiple vault-ids
 # but this time specify with --encrypt-vault-id, but specifying vault-id names (instead of default)
-# ansible-vault encrypt "$@" --vault-id from_vault_password@vault-password --vault-id from_encrypted_vault_password@encrypted-vault-password --encrypt-vault-id from_encrypted_vault_password "${TEST_FILE_ENC_PASSWORD}"
+# ansible-vault encrypt "$@" --vault-id from_vault_password@vault-password --vault-id from_encrypted_vault_password@encrypted-vault-password --encrypt-vault-id from_encrypted_vault_password "${TEST_FILE(_ENC_PASSWORD}"
 
 # try to view the file encrypted with the vault-password we didnt specify
 # to verify we didnt choose the wrong vault-id
@@ -391,6 +391,9 @@ ansible-playbook test_vault_embedded.yml -i ../../inventory -v "$@" --vault-pass
 ansible-playbook test_vaulted_inventory.yml -i vaulted.inventory -v "$@" --vault-password-file vault-password
 ansible-playbook test_vaulted_template.yml -i ../../inventory -v "$@" --vault-password-file vault-password
 
+# test a playbook with a host_var whose value is non-ascii utf8 (see https://github.com/ansible/ansible/issues/37258)
+ansible-playbook -i ../../inventory -v "$@" --vault-id vault-password test_vaulted_utf8_value.yml
+
 # test with password from password script
 ansible-playbook test_vault.yml          -i ../../inventory -v "$@" --vault-password-file password-script.py
 ansible-playbook test_vault_embedded.yml -i ../../inventory -v "$@" --vault-password-file password-script.py
diff --git a/test/integration/targets/vault/test_vaulted_utf8_value.yml b/test/integration/targets/vault/test_vaulted_utf8_value.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/vault/test_vaulted_utf8_value.yml
@@ -0,0 +1,15 @@
+- name: "test that the vaulted_utf8_value decrypts correctly"
+  gather_facts: false
+  hosts: testhost
+  vars:
+    expected: "aöffü"
+  tasks:
+    - name: decrypt vaulted_utf8_value and show it in debug
+      debug:
+         var: vaulted_utf8_value
+
+    - name: assert decrypted vaulted_utf8_value matches expected
+      assert:
+        that:
+          - "vaulted_utf8_value == expected"
+          - "vaulted_utf8_value == 'aöffü'"
diff --git a/test/units/parsing/yaml/test_objects.py b/test/units/parsing/yaml/test_objects.py
--- a/test/units/parsing/yaml/test_objects.py
+++ b/test/units/parsing/yaml/test_objects.py
@@ -1,4 +1,5 @@
 # This file is part of Ansible
+# -*- coding: utf-8 -*-
 #
 # Ansible is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -149,3 +150,8 @@ def compare(avu, seq):
             return avu == seq
 
         self.assertRaises(AnsibleError, compare, avu, seq)
+
+    def test_vaulted_utf8_value_37258(self):
+        seq = u"aöffü"
+        avu = self._from_plaintext(seq)
+        self.assert_values(avu, seq)