Repository: ansible/ansible

Issue:
ansible-vault edit and encrypt output different payload format version
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
ansible-vault

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.0.0
```

##### CONFIGURATION
<!---
-->

##### OS / ENVIRONMENT
<!---
OSX
-->

##### SUMMARY
<!--- Explain the problem briefly -->
When I encrypt a file with ```ansible-vault encrypt vault_name --vault-id staging@vault_pass``` it creates a file with the header
```$ANSIBLE_VAULT;1.2;AES256;staging```
When editing a vault file with ```ansible-vault edit vault_name --vault-id staging@vault_pass``` the vault is outputting with a header of
```$ANSIBLE_VAULT;1.1;AES256```
If I have made any changes and try to edit the file again, the following error occurs.
```ERROR! Unexpected Exception, this is probably a bug: list index out of range```
There is no error if I make no changes.

If I run decrypt it is successful. If I then run
```ansible-vault encrypt vault_name --vault_id staging@vault_pass```
the vault is outputting with a header of ```$ANSIBLE_VAULT;1.2;AES256;staging``` and I can run ```ansible-deploy edit``` again successfully.

##### STEPS TO REPRODUCE
```bash
ansible-vault edit vault_name --vault-id staging@vault_pass
ansible-vault edit vault_name --vault-id staging@vault_pass
ansible-vault decrypt vault_name --vault_id staging@vault_pass
ansible-vault encrypt vault_name --vault_id staging@vault_pass
```
##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```bash
$ head -n1 vault
$ANSIBLE_VAULT;1.2;AES256;staging
$ ansible-vault edit vault_name --vault_id staging@vault_pass
$ head -n1 group_vars/staging/vault
$ANSIBLE_VAULT;1.1;AES256
$ ansible-vault edit vault_name --vault_id staging@vault_pass
$ ERROR! Unexpected Exception, this is probably a bug: list index out of range
ansible-vault decrypt vault_name --vault_id staging@vault_pass
ansible-vault encrypt vault_name --vault_id staging@vault_pass
$ head -n1 vault
$ANSIBLE_VAULT;1.2;AES256;staging
```



Discussion:
Is there an 'vault_password_file' or 'vault_identity_list' setup in configuration?

Running 'ansible-vault' with a high verbosity level will show some details about which vault-ids are being used.
I can reproduce this:

```
[newswoop:F25:ansible (facts_arch_linux_30600 %)]$ echo '30575' > vault_30575.yml

[newswoop:F25:ansible (facts_arch_linux_30600 %)]$ ansible-vault encrypt -vvv --vault-id staging@password vault_30575.yml 
ansible-vault 2.5.0 (facts_arch_linux_30600 08cca42fd4) last updated 2017/09/22 10:22:26 (GMT -400)
  config file = /home/adrian/src/ansible/ansible.cfg
  configured module search path = [u'/home/adrian/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/adrian/src/ansible/lib/ansible
  executable location = /home/adrian/src/ansible/bin/ansible-vault
  python version = 2.7.13 (default, May 10 2017, 20:04:28) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]
Using /home/adrian/src/ansible/ansible.cfg as config file
Encryption successful

[newswoop:F25:ansible (facts_arch_linux_30600 %)]$ cat vault_30575.yml 
$ANSIBLE_VAULT;1.2;AES256;staging
<..>
```

vault edit without any changes

```
[newswoop:F25:ansible (facts_arch_linux_30600 %)]$ ansible-vault edit -vvvvvv --vault-id staging@password vault_30575.yml 
< version info snipped >
Using /home/adrian/src/ansible/ansible.cfg as config file
Reading vault password file: password
Found a vault_id (staging) in the vaulttext
We have a secret associated with vault id (staging), will try to use to decrypt None
Trying to use vault secret=(FileVaultSecret(filename='/home/adrian/src/ansible/password')) id=staging to decrypt None
Trying secret FileVaultSecret(filename='/home/adrian/src/ansible/password') for vault_id=staging
decrypt succesful with secret=FileVaultSecret(filename='/home/adrian/src/ansible/password') and vault_id=staging

[newswoop:F25:ansible (facts_arch_linux_30600 %)]$ cat vault_30575.yml
$ANSIBLE_VAULT;1.2;AES256;staging
<..>
```

vault edit, making and saving changes:

```
[newswoop:F25:ansible (facts_arch_linux_30600 %)]$ ansible-vault edit -vvvvvv --vault-id staging@password vault_30575.yml 
< version info snipped >

Using /home/adrian/src/ansible/ansible.cfg as config file
Reading vault password file: password
Found a vault_id (staging) in the vaulttext
We have a secret associated with vault id (staging), will try to use to decrypt None
Trying to use vault secret=(FileVaultSecret(filename='/home/adrian/src/ansible/password')) id=staging to decrypt None
Trying secret FileVaultSecret(filename='/home/adrian/src/ansible/password') for vault_id=staging
decrypt succesful with secret=FileVaultSecret(filename='/home/adrian/src/ansible/password') and vault_id=staging
Encrypting with vault secret FileVaultSecret(filename='/home/adrian/src/ansible/password')

[newswoop:F25:ansible (facts_arch_linux_30600 %)]$ cat vault_30575.yml
$ANSIBLE_VAULT;1.1;AES256
<..>
```

vault edit again: 

```
[newswoop:F25:ansible (facts_arch_linux_30600 %)]$ ansible-vault edit -vvvvvv --vault-id staging@password vault_30575.yml 
< version info snipped >
Using /home/adrian/src/ansible/ansible.cfg as config file
Reading vault password file: password
Found a vault_id (default) in the vaulttext
Found a vault_id (default) in the vault text, but we do not have a associated secret (--vault-id)
Trying to use vault secret=(FileVaultSecret(filename='/home/adrian/src/ansible/password')) id=staging to decrypt None
Trying secret FileVaultSecret(filename='/home/adrian/src/ansible/password') for vault_id=staging
decrypt succesful with secret=FileVaultSecret(filename='/home/adrian/src/ansible/password') and vault_id=staging
ERROR! Unexpected Exception, this is probably a bug: list index out of range
the full traceback was:

Traceback (most recent call last):
  File "/home/adrian/src/ansible/bin/ansible-vault", line 109, in <module>
    exit_code = cli.run()
  File "/home/adrian/src/ansible/lib/ansible/cli/vault.py", line 228, in run
    self.execute()
  File "/home/adrian/src/ansible/lib/ansible/cli/__init__.py", line 154, in execute
    fn()
  File "/home/adrian/src/ansible/lib/ansible/cli/vault.py", line 419, in execute_edit
    self.editor.edit_file(f)
  File "/home/adrian/src/ansible/lib/ansible/parsing/vault/__init__.py", line 768, in edit_file
    secret = secrets[0][1]
IndexError: list index out of range

```

@colinjc Thanks for the great bug report btw! Looking into a fix now.
Looks like the vault_id is getting dropped on the encrypt after edit with change. (vault_id isnt passed to encrypt correctly there so got a fix for that).

The traceback is from an assumption in edit that if we can decrypt, then we have a matching vault-id in the secrets. Because of above bug, on the second edit, the vault id from the edited file is unset (or 'default') but the only vault-id in secrets is 'staging'. Then the vault-id match returns a empty secrets list, and the the secrets[0][1] reference throws the IndexError.

Resolving patch:
diff --git a/lib/ansible/parsing/vault/__init__.py b/lib/ansible/parsing/vault/__init__.py
--- a/lib/ansible/parsing/vault/__init__.py
+++ b/lib/ansible/parsing/vault/__init__.py
@@ -491,6 +491,20 @@ def encrypt(self, plaintext, secret=None, vault_id=None):
         return b_vaulttext
 
     def decrypt(self, vaulttext, filename=None):
+        '''Decrypt a piece of vault encrypted data.
+
+        :arg vaulttext: a string to decrypt.  Since vault encrypted data is an
+            ascii text format this can be either a byte str or unicode string.
+        :kwarg filename: a filename that the data came from.  This is only
+            used to make better error messages in case the data cannot be
+            decrypted.
+        :returns: a byte string containing the decrypted data and the vault-id that was used
+
+        '''
+        plaintext, vault_id = self.decrypt_and_get_vault_id(vaulttext, filename=filename)
+        return plaintext
+
+    def decrypt_and_get_vault_id(self, vaulttext, filename=None):
         """Decrypt a piece of vault encrypted data.
 
         :arg vaulttext: a string to decrypt.  Since vault encrypted data is an
@@ -498,7 +512,8 @@ def decrypt(self, vaulttext, filename=None):
         :kwarg filename: a filename that the data came from.  This is only
             used to make better error messages in case the data cannot be
             decrypted.
-        :returns: a byte string containing the decrypted data
+        :returns: a byte string containing the decrypted data and the vault-id that was used
+
         """
         b_vaulttext = to_bytes(vaulttext, errors='strict', encoding='utf-8')
 
@@ -536,6 +551,7 @@ def decrypt(self, vaulttext, filename=None):
         # we check it first.
 
         vault_id_matchers = []
+        vault_id_used = None
 
         if vault_id:
             display.vvvvv('Found a vault_id (%s) in the vaulttext' % (vault_id))
@@ -563,6 +579,7 @@ def decrypt(self, vaulttext, filename=None):
                 display.vvvv('Trying secret %s for vault_id=%s' % (vault_secret, vault_secret_id))
                 b_plaintext = this_cipher.decrypt(b_vaulttext, vault_secret)
                 if b_plaintext is not None:
+                    vault_id_used = vault_secret_id
                     display.vvvvv('decrypt succesful with secret=%s and vault_id=%s' % (vault_secret, vault_secret_id))
                     break
             except AnsibleError as e:
@@ -581,7 +598,7 @@ def decrypt(self, vaulttext, filename=None):
                 msg += " on %s" % filename
             raise AnsibleError(msg)
 
-        return b_plaintext
+        return b_plaintext, vault_id_used
 
 
 class VaultEditor:
@@ -692,6 +709,7 @@ def _edit_file_helper(self, filename, secret,
 
         # shuffle tmp file into place
         self.shuffle_files(tmp_path, filename)
+        display.vvvvv('Saved edited file "%s" encrypted using %s and  vault id "%s"' % (filename, secret, vault_id))
 
     def _real_path(self, filename):
         # '-' is special to VaultEditor, dont expand it.
@@ -754,7 +772,8 @@ def edit_file(self, filename):
 
         try:
             # vaulttext gets converted back to bytes, but alas
-            plaintext = self.vault.decrypt(vaulttext)
+            # TODO: return the vault_id that worked?
+            plaintext, vault_id_used = self.vault.decrypt_and_get_vault_id(vaulttext)
         except AnsibleError as e:
             raise AnsibleError("%s for %s" % (to_bytes(e), to_bytes(filename)))
 
@@ -762,15 +781,25 @@ def edit_file(self, filename):
         # (duplicates parts of decrypt, but alas...)
         dummy, dummy, cipher_name, vault_id = parse_vaulttext_envelope(b_vaulttext)
 
-        # if we could decrypt, the vault_id should be in secrets
+        # vault id here may not be the vault id actually used for decrypting
+        # as when the edited file has no vault-id but is decrypted by non-default id in secrets
+        # (vault_id=default, while a different vault-id decrypted)
+
+        # if we could decrypt, the vault_id should be in secrets or we use vault_id_used
         # though we could have multiple secrets for a given vault_id, pick the first one
-        secrets = match_secrets(self.vault.secrets, [vault_id])
+        secrets = match_secrets(self.vault.secrets, [vault_id_used, vault_id])
+
+        if not secrets:
+            raise AnsibleVaultError('Attempting to encrypt "%s" but no vault secrets were found for vault ids "%s" or "%s"' %
+                                    (filename, vault_id, vault_id_used))
+
         secret = secrets[0][1]
+
         if cipher_name not in CIPHER_WRITE_WHITELIST:
             # we want to get rid of files encrypted with the AES cipher
-            self._edit_file_helper(filename, secret, existing_data=plaintext, force_save=True)
+            self._edit_file_helper(filename, secret, existing_data=plaintext, force_save=True, vault_id=vault_id)
         else:
-            self._edit_file_helper(filename, secret, existing_data=plaintext, force_save=False)
+            self._edit_file_helper(filename, secret, existing_data=plaintext, force_save=False, vault_id=vault_id)
 
     def plaintext(self, filename):
 


Test patch:
diff --git a/test/integration/targets/vault/faux-editor.py b/test/integration/targets/vault/faux-editor.py
new file mode 100755
--- /dev/null
+++ b/test/integration/targets/vault/faux-editor.py
@@ -0,0 +1,44 @@
+#!/usr/bin/env python
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+# ansible-vault is a script that encrypts/decrypts YAML files. See
+# http://docs.ansible.com/playbooks_vault.html for more details.
+
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+import sys
+import time
+import os
+
+
+def main(args):
+    path = os.path.abspath(args[1])
+
+    fo = open(path, 'r+')
+
+    content = fo.readlines()
+
+    content.append('faux editor added at %s\n' % time.time())
+
+    fo.seek(0)
+    fo.write(''.join(content))
+    fo.close()
+
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv[:]))
diff --git a/test/integration/targets/vault/runme.sh b/test/integration/targets/vault/runme.sh
--- a/test/integration/targets/vault/runme.sh
+++ b/test/integration/targets/vault/runme.sh
@@ -14,7 +14,14 @@ echo "This is a test file for format 1.2" > "${TEST_FILE_1_2}"
 
 TEST_FILE_OUTPUT="${MYTMPDIR}/test_file_output"
 
+TEST_FILE_EDIT="${MYTMPDIR}/test_file_edit"
+echo "This is a test file for edit" > "${TEST_FILE_EDIT}"
 
+TEST_FILE_EDIT2="${MYTMPDIR}/test_file_edit2"
+echo "This is a test file for edit2" > "${TEST_FILE_EDIT2}"
+
+FORMAT_1_1_HEADER="\$ANSIBLE_VAULT;1.1;AES256"
+FORMAT_1_2_HEADER="\$ANSIBLE_VAULT;1.2;AES256"
 
 # old format
 ansible-vault view "$@" --vault-password-file vault-password-ansible format_1_0_AES.yml
@@ -234,6 +241,27 @@ ansible-vault encrypt_string "$@" --vault-password-file "${NEW_VAULT_PASSWORD}"
 # write to file
 ansible-vault encrypt_string "$@" --vault-password-file "${NEW_VAULT_PASSWORD}" --name "blippy" "a test string names blippy" --output "${MYTMPDIR}/enc_string_test_file"
 
+# test ansible-vault edit with a faux editor
+ansible-vault encrypt "$@" --vault-password-file vault-password "${TEST_FILE_EDIT}"
+
+# edit a 1.1 format with no vault-id, should stay 1.1
+EDITOR=./faux-editor.py ansible-vault edit "$@" --vault-password-file vault-password "${TEST_FILE_EDIT}"
+head -1 "${TEST_FILE_EDIT}" | grep "${FORMAT_1_1_HEADER}"
+
+# edit a 1.1 format with vault-id, should stay 1.1
+EDITOR=./faux-editor.py ansible-vault edit "$@" --vault-id vault_password@vault-password "${TEST_FILE_EDIT}"
+head -1 "${TEST_FILE_EDIT}" | grep "${FORMAT_1_1_HEADER}"
+
+ansible-vault encrypt "$@" --vault-id vault_password@vault-password "${TEST_FILE_EDIT2}"
+
+# edit a 1.2 format with vault id, should keep vault id and 1.2 format
+EDITOR=./faux-editor.py ansible-vault edit "$@" --vault-id vault_password@vault-password "${TEST_FILE_EDIT2}"
+head -1 "${TEST_FILE_EDIT2}" | grep "${FORMAT_1_2_HEADER};vault_password"
+
+# edit a 1.2 file with no vault-id, should keep vault id and 1.2 format
+EDITOR=./faux-editor.py ansible-vault edit "$@" --vault-password-file vault-password "${TEST_FILE_EDIT2}"
+head -1 "${TEST_FILE_EDIT2}" | grep "${FORMAT_1_2_HEADER};vault_password"
+
 
 # test playbooks using vaulted files
 ansible-playbook test_vault.yml          -i ../../inventory -v "$@" --vault-password-file vault-password --list-tasks
diff --git a/test/units/parsing/vault/test_vault_editor.py b/test/units/parsing/vault/test_vault_editor.py
--- a/test/units/parsing/vault/test_vault_editor.py
+++ b/test/units/parsing/vault/test_vault_editor.py
@@ -309,7 +309,7 @@ def test_encrypt_file_symlink(self):
         self._assert_file_is_link(src_file_link_path, src_file_path)
 
     @patch('ansible.parsing.vault.subprocess.call')
-    def test_edit_file(self, mock_sp_call):
+    def test_edit_file_no_vault_id(self, mock_sp_call):
         self._test_dir = self._create_test_dir()
         src_contents = to_bytes("some info in a file\nyup.")
 
@@ -330,6 +330,36 @@ def faux_editor(editor_args):
         new_src_file = open(src_file_path, 'rb')
         new_src_file_contents = new_src_file.read()
 
+        self.assertTrue(b'$ANSIBLE_VAULT;1.1;AES256' in new_src_file_contents)
+
+        src_file_plaintext = ve.vault.decrypt(new_src_file_contents)
+        self.assertEqual(src_file_plaintext, new_src_contents)
+
+    @patch('ansible.parsing.vault.subprocess.call')
+    def test_edit_file_with_vault_id(self, mock_sp_call):
+        self._test_dir = self._create_test_dir()
+        src_contents = to_bytes("some info in a file\nyup.")
+
+        src_file_path = self._create_file(self._test_dir, 'src_file', content=src_contents)
+
+        new_src_contents = to_bytes("The info is different now.")
+
+        def faux_editor(editor_args):
+            self._faux_editor(editor_args, new_src_contents)
+
+        mock_sp_call.side_effect = faux_editor
+
+        ve = self._vault_editor()
+
+        ve.encrypt_file(src_file_path, self.vault_secret,
+                        vault_id='vault_secrets')
+        ve.edit_file(src_file_path)
+
+        new_src_file = open(src_file_path, 'rb')
+        new_src_file_contents = new_src_file.read()
+
+        self.assertTrue(b'$ANSIBLE_VAULT;1.2;AES256;vault_secrets' in new_src_file_contents)
+
         src_file_plaintext = ve.vault.decrypt(new_src_file_contents)
         self.assertEqual(src_file_plaintext, new_src_contents)