Repository: ansible/ansible

Issue:
ERROR! Unexpected Exception, this is probably a bug: u'prompt_ask_vault_pass'
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
ansible-vault

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0 (devel e8410c210d) last updated 2017/08/08 12:37:14 (GMT +100)
  config file = None
  configured module search path = [u'/home/jamie/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jamie/ansible/lib/ansible
  executable location = /home/jamie/ansible/bin/ansible
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
```

##### CONFIGURATION
<!---
-->

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
WSL on Windows 10 x64

##### SUMMARY
<!--- Explain the problem briefly -->
ansible-vault edit and encrypt no longer working

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
```
$ ansible-vault edit host_vars/project/vault.yml -vvv
ansible-vault 2.4.0 (devel e8410c210d) last updated 2017/08/08 12:37:14 (GMT +100)
  config file = None
  configured module search path = [u'/home/jamie/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jamie/ansible/lib/ansible
  executable location = /home/jamie/ansible/bin/ansible-vault
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
No config file found; using defaults
Usage: ansible-vault edit [options] file_name

encryption/decryption utility for Ansible data files

Options:
  --ask-vault-pass      ask for vault password
  -h, --help            show this help message and exit
  --new-vault-id=NEW_VAULT_ID
                        the new vault identity to use for rekey
  --new-vault-password-file=NEW_VAULT_PASSWORD_FILES
                        new vault password file for rekey
  --output=OUTPUT_FILE  output file name for encrypt or decrypt; use - for
                        stdout
  --vault-id=VAULT_IDS  the vault identity to use
  --vault-password-file=VAULT_PASSWORD_FILES
                        vault password file
  -v, --verbose         verbose mode (-vvv for more, -vvvv to enable
                        connection debugging)
  --version             show program's version number and exit

 See 'ansible-vault <command> --help' for more information on a specific
command.
ERROR! A vault password is required to use Ansible's Vault
```

```
$ ansible-vault edit --ask-vault-pass host_vars/project/vault.yml -vvv
ansible-vault 2.4.0 (devel e8410c210d) last updated 2017/08/08 12:37:14 (GMT +100)
  config file = None
  configured module search path = [u'/home/jamie/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jamie/ansible/lib/ansible
  executable location = /home/jamie/ansible/bin/ansible-vault
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
No config file found; using defaults
ERROR! Unexpected Exception, this is probably a bug: u'prompt_ask_vault_pass'
the full traceback was:

Traceback (most recent call last):
  File "/home/jamie/ansible/bin/ansible-vault", line 106, in <module>
    exit_code = cli.run()
  File "/home/jamie/ansible/lib/ansible/cli/vault.py", line 180, in run
    create_new_password=True)
  File "/home/jamie/ansible/lib/ansible/cli/__init__.py", line 237, in setup_vault_secrets
    prompted_vault_secret = PromptVaultSecret(prompt_formats=prompt_formats[vault_id_value],
KeyError: u'prompt_ask_vault_pass'
```

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Was expecting to be prompted automatically, expected `--ask-vault-pass` to work when added

##### ACTUAL RESULTS
Exception

##### NOTES
`$ ansible-vault decrypt ./host_vars/project/vault.yml --ask-vault-pass` worked as expected

However

`$ ansible-vault encrypt ./host_vars/project/vault.yml`

```
Usage: ansible-vault encrypt [options] file_name

encryption/decryption utility for Ansible data files

Options:
  --ask-vault-pass      ask for vault password
  -h, --help            show this help message and exit
  --new-vault-id=NEW_VAULT_ID
                        the new vault identity to use for rekey
  --new-vault-password-file=NEW_VAULT_PASSWORD_FILES
                        new vault password file for rekey
  --output=OUTPUT_FILE  output file name for encrypt or decrypt; use - for
                        stdout
  --vault-id=VAULT_IDS  the vault identity to use
  --vault-password-file=VAULT_PASSWORD_FILES
                        vault password file
  -v, --verbose         verbose mode (-vvv for more, -vvvv to enable
                        connection debugging)
  --version             show program's version number and exit

 See 'ansible-vault <command> --help' for more information on a specific
command.
ERROR! A vault password is required to use Ansible's Vault
```

`$ ansible-vault encrypt ./host_vars/project/vault.yml --ask-vault-pass`

```
ERROR! Unexpected Exception, this is probably a bug: u'prompt_ask_vault_pass'
to see the full traceback, use -vvv
```

Rolling back to tag v2.3.2.0-1 seems to fix these issues.


Discussion:
I confirm this issue on ansible 2.4.0 (devel babec35faa)
I can use *ansible-vault view* but not *ansible-vault edit* : 

```bash
ansible-vault edit host_vars/myhost/myhost_vault.yml                                           
Usage: ansible-vault edit [options] file_name

encryption/decryption utility for Ansible data files

Options:
  --ask-vault-pass      ask for vault password
  -h, --help            show this help message and exit
  --new-vault-id=NEW_VAULT_ID
                        the new vault identity to use for rekey
  --new-vault-password-file=NEW_VAULT_PASSWORD_FILES
                        new vault password file for rekey
  --output=OUTPUT_FILE  output file name for encrypt or decrypt; use - for
                        stdout
  --vault-id=VAULT_IDS  the vault identity to use
  --vault-password-file=VAULT_PASSWORD_FILES
                        vault password file
  -v, --verbose         verbose mode (-vvv for more, -vvvv to enable
                        connection debugging)
  --version             show program's version number and exit

 See 'ansible-vault <command> --help' for more information on a specific
command.
ERROR! A vault password is required to use Ansible's Vault
```

If --ask-vault-pass option is provided, I get following error : 

```bash
ansible-vault edit --ask-vault-pass host_vars/myhost/myhost_vault.yml
ERROR! Unexpected Exception, this is probably a bug: u'prompt_ask_vault_pass'
```
I think it is related to 934b64519191b0df70495deecbb349532af48597
I can edit vault using *--vault-id=prompt* but I get weird prompts : 

```bash
ansible-vault edit --vault-id=prompt host_vars/myhost/myhost_vault.yml
New vault password (default):
Confirm vew vault password (default):
```
By answering with my vault password to these two prompts, editor opens and vault can be edited

Resolving patch:
diff --git a/lib/ansible/cli/__init__.py b/lib/ansible/cli/__init__.py
--- a/lib/ansible/cli/__init__.py
+++ b/lib/ansible/cli/__init__.py
@@ -218,6 +218,9 @@ def setup_vault_secrets(loader, vault_ids, vault_password_files=None,
         if create_new_password:
             prompt_formats['prompt'] = ['New vault password (%(vault_id)s): ',
                                         'Confirm vew vault password (%(vault_id)s): ']
+            # 2.3 format prompts for --ask-vault-pass
+            prompt_formats['prompt_ask_vault_pass'] = ['New Vault password: ',
+                                                       'Confirm New Vault password: ']
         else:
             prompt_formats['prompt'] = ['Vault password (%(vault_id)s): ']
             # The format when we use just --ask-vault-pass needs to match 'Vault password:\s*?$'


Test patch:
diff --git a/test/units/cli/test_cli.py b/test/units/cli/test_cli.py
--- a/test/units/cli/test_cli.py
+++ b/test/units/cli/test_cli.py
@@ -46,6 +46,9 @@ def test_version_info_gitinfo(self):
 
 
 class TestCliSetupVaultSecrets(unittest.TestCase):
+    def setUp(self):
+        self.fake_loader = DictDataLoader({})
+
     def test(self):
         res = cli.CLI.setup_vault_secrets(None, None)
         self.assertIsInstance(res, list)
@@ -56,8 +59,7 @@ def test_password_file(self, mock_file_secret):
         mock_file_secret.return_value = MagicMock(bytes=b'file1_password',
                                                   vault_id='file1',
                                                   filename=filename)
-        fake_loader = DictDataLoader({})
-        res = cli.CLI.setup_vault_secrets(loader=fake_loader,
+        res = cli.CLI.setup_vault_secrets(loader=self.fake_loader,
                                           vault_ids=['secret1@%s' % filename, 'secret2'],
                                           vault_password_files=[filename])
         self.assertIsInstance(res, list)
@@ -71,15 +73,145 @@ def test_prompt(self, mock_prompt_secret):
         mock_prompt_secret.return_value = MagicMock(bytes=b'prompt1_password',
                                                     vault_id='prompt1')
 
-        fake_loader = DictDataLoader({})
-        res = cli.CLI.setup_vault_secrets(loader=fake_loader,
+        res = cli.CLI.setup_vault_secrets(loader=self.fake_loader,
                                           vault_ids=['prompt1@prompt'],
                                           ask_vault_pass=True)
 
         self.assertIsInstance(res, list)
         matches = vault.match_secrets(res, ['prompt1'])
         self.assertIn('prompt1', [x[0] for x in matches])
-        # self.assertIn('prompt1', res)
-        # self.assertIn('secret1', res)
         match = matches[0][1]
         self.assertEqual(match.bytes, b'prompt1_password')
+
+    def _assert_ids(self, vault_id_names, res, password=b'prompt1_password'):
+        self.assertIsInstance(res, list)
+        len_ids = len(vault_id_names)
+        matches = vault.match_secrets(res, vault_id_names)
+        self.assertEqual(len(res), len_ids)
+        self.assertEqual(len(matches), len_ids)
+        for index, prompt in enumerate(vault_id_names):
+            self.assertIn(prompt, [x[0] for x in matches])
+            # simple mock, same password/prompt for each mock_prompt_secret
+            self.assertEqual(matches[index][1].bytes, password)
+
+    @patch('ansible.cli.PromptVaultSecret')
+    def test_multiple_prompts(self, mock_prompt_secret):
+        mock_prompt_secret.return_value = MagicMock(bytes=b'prompt1_password',
+                                                    vault_id='prompt1')
+
+        res = cli.CLI.setup_vault_secrets(loader=self.fake_loader,
+                                          vault_ids=['prompt1@prompt',
+                                                     'prompt2@prompt'],
+                                          ask_vault_pass=False)
+
+        vault_id_names = ['prompt1', 'prompt2']
+        self._assert_ids(vault_id_names, res)
+
+    @patch('ansible.cli.PromptVaultSecret')
+    def test_multiple_prompts_and_ask_vault_pass(self, mock_prompt_secret):
+        mock_prompt_secret.return_value = MagicMock(bytes=b'prompt1_password',
+                                                    vault_id='prompt1')
+
+        res = cli.CLI.setup_vault_secrets(loader=self.fake_loader,
+                                          vault_ids=['prompt1@prompt',
+                                                     'prompt2@prompt',
+                                                     'prompt3@prompt_ask_vault_pass'],
+                                          ask_vault_pass=True)
+
+        vault_id_names = ['prompt1', 'prompt2', 'prompt3', 'default']
+        self._assert_ids(vault_id_names, res)
+
+    @patch('ansible.cli.C', name='MockConfig')
+    @patch('ansible.cli.get_file_vault_secret')
+    @patch('ansible.cli.PromptVaultSecret')
+    def test_default_file_vault(self, mock_prompt_secret,
+                                mock_file_secret,
+                                mock_config):
+        mock_prompt_secret.return_value = MagicMock(bytes=b'prompt1_password',
+                                                    vault_id='default')
+        mock_file_secret.return_value = MagicMock(bytes=b'file1_password',
+                                                  vault_id='default')
+        mock_config.DEFAULT_VAULT_PASSWORD_FILE = '/dev/null/faux/vault_password_file'
+        mock_config.DEFAULT_VAULT_IDENTITY = 'default'
+
+        res = cli.CLI.setup_vault_secrets(loader=self.fake_loader,
+                                          vault_ids=[],
+                                          create_new_password=False,
+                                          ask_vault_pass=True)
+
+        self.assertIsInstance(res, list)
+        matches = vault.match_secrets(res, ['default'])
+        # --vault-password-file/DEFAULT_VAULT_PASSWORD_FILE is higher precendce than prompts
+        # if the same vault-id ('default') regardless of cli order since it didn't matter in 2.3
+        self.assertEqual(matches[0][1].bytes, b'file1_password')
+        self.assertEqual(matches[1][1].bytes, b'prompt1_password')
+
+    @patch('ansible.cli.PromptVaultSecret')
+    def test_prompt_just_ask_vault_pass(self, mock_prompt_secret):
+        mock_prompt_secret.return_value = MagicMock(bytes=b'prompt1_password',
+                                                    vault_id='default')
+
+        res = cli.CLI.setup_vault_secrets(loader=self.fake_loader,
+                                          vault_ids=[],
+                                          create_new_password=False,
+                                          ask_vault_pass=True)
+
+        self.assertIsInstance(res, list)
+        match = vault.match_secrets(res, ['default'])[0][1]
+        self.assertEqual(match.bytes, b'prompt1_password')
+
+    @patch('ansible.cli.PromptVaultSecret')
+    def test_prompt_new_password_ask_vault_pass(self, mock_prompt_secret):
+        mock_prompt_secret.return_value = MagicMock(bytes=b'prompt1_password',
+                                                    vault_id='default')
+
+        res = cli.CLI.setup_vault_secrets(loader=self.fake_loader,
+                                          vault_ids=[],
+                                          create_new_password=True,
+                                          ask_vault_pass=True)
+
+        self.assertIsInstance(res, list)
+        match = vault.match_secrets(res, ['default'])[0][1]
+        self.assertEqual(match.bytes, b'prompt1_password')
+
+    @patch('ansible.cli.PromptVaultSecret')
+    def test_prompt_new_password_vault_id_prompt(self, mock_prompt_secret):
+        mock_prompt_secret.return_value = MagicMock(bytes=b'prompt1_password',
+                                                    vault_id='some_vault_id')
+
+        res = cli.CLI.setup_vault_secrets(loader=self.fake_loader,
+                                          vault_ids=['some_vault_id@prompt'],
+                                          create_new_password=True,
+                                          ask_vault_pass=False)
+
+        self.assertIsInstance(res, list)
+        match = vault.match_secrets(res, ['some_vault_id'])[0][1]
+        self.assertEqual(match.bytes, b'prompt1_password')
+
+    @patch('ansible.cli.PromptVaultSecret')
+    def test_prompt_new_password_vault_id_prompt_ask_vault_pass(self, mock_prompt_secret):
+        mock_prompt_secret.return_value = MagicMock(bytes=b'prompt1_password',
+                                                    vault_id='default')
+
+        res = cli.CLI.setup_vault_secrets(loader=self.fake_loader,
+                                          vault_ids=['some_vault_id@prompt_ask_vault_pass'],
+                                          create_new_password=True,
+                                          ask_vault_pass=False)
+
+        self.assertIsInstance(res, list)
+        match = vault.match_secrets(res, ['some_vault_id'])[0][1]
+        self.assertEqual(match.bytes, b'prompt1_password')
+
+    @patch('ansible.cli.PromptVaultSecret')
+    def test_prompt_new_password_vault_id_prompt_ask_vault_pass_ask_vault_pass(self, mock_prompt_secret):
+        mock_prompt_secret.return_value = MagicMock(bytes=b'prompt1_password',
+                                                    vault_id='default')
+
+        res = cli.CLI.setup_vault_secrets(loader=self.fake_loader,
+                                          vault_ids=['some_vault_id@prompt_ask_vault_pass'],
+                                          create_new_password=True,
+                                          ask_vault_pass=True)
+
+        self.assertIsInstance(res, list)
+        match = vault.match_secrets(res, ['some_vault_id'])[0][1]
+        self.assertEqual(match.bytes, b'prompt1_password')