Repository: ansible-collections/community.general

Issue:
passwordstore does not handle multiline yaml
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
<!--- Explain the problem briefly below -->

When using the `community.general.passwordstore` lookup with a yaml formatted file, it does not actually handle multiline YAML.

Instead it only returns the string character starting the multiline string.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->

community.general.passwordstore

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
ansible 2.9.16
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/florian/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.9/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.9.1 (default, Dec  8 2020, 00:00:00) [GCC 10.2.1 20201125 (Red Hat 10.2.1-9)]
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```paste below

```

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->

```
cat /etc/os-release 
NAME=Fedora
VERSION="33 (Workstation Edition)"
ID=fedora
VERSION_ID=33
VERSION_CODENAME=""
PLATFORM_ID="platform:f33"
PRETTY_NAME="Fedora 33 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:33"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f33/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=33
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=33
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation
```

##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->

Assuming there is a password file like this:

```sh
# cat ~/.password-store/mytestpassword
mypassword
mykey: |
  this value is multiple
  lines long
```

And using a playbook to attempt to retrieve the `mykey` subkey:

```yaml
- hosts: localhost
  tasks:
    - debug:
        msg: "{{ lookup('community.general.passwordstore', 'mytestpassword subkey=mykey') }}"
```

Running the playbook will only return the "|" character that should denote a multiline string.

I guess the problem is that the code doesn't actually ever try to treat the rest of the password file as YAML, instead just parses it line by line: https://github.com/ansible-collections/community.general/blob/main/plugins/lookup/passwordstore.py#L211-L215

Changing it now to just assume everything after the first line is YAML would break existing usecases though, so not sure if this ever was inteded to work how I assumed, based on the documentation (`It also retrieves YAML style keys stored as multilines in the passwordfile.`).

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->

The msg should contain the whole string including newlines.

##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->

Running the playbook will only return the "|" character that should denote a multiline string.

<!--- Paste verbatim command output between quotes -->
```paste below
PLAY [Debug playbook] **********************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************
ok: [localhost]

TASK [debug] *******************************************************************************************************************************
Please unlock the card
<stripped the card info>
ok: [localhost] => {
    "msg": "|"
}

PLAY RECAP *********************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
```



Discussion:
Files identified in the description:
None

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
!component =plugins/lookup/passwordstore.py
Files identified in the description:
* [`plugins/lookup/passwordstore.py`](https://github.com/ansible-collections/community.general/blob/master/plugins/lookup/passwordstore.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
The documentation was added here: https://github.com/ansible/ansible/commit/24d4787b2d3bf531c048fe3cdce0534180193475#diff-3a076e9ec5639f631eb3113e5a405ec4929b023d3d32acedc6d01d26e1fe3092R16

@bcoca since you added the documentation (including the multiline YAML claim): did you take the docs from somewhere else, or did you just claimed that? Because the code back then didn't support it either.

Resolving patch:
diff --git a/plugins/lookup/passwordstore.py b/plugins/lookup/passwordstore.py
--- a/plugins/lookup/passwordstore.py
+++ b/plugins/lookup/passwordstore.py
@@ -105,6 +105,8 @@
 import os
 import subprocess
 import time
+import yaml
+
 
 from distutils import util
 from ansible.errors import AnsibleError, AnsibleAssertionError
@@ -209,10 +211,15 @@ def check_pass(self):
             ).splitlines()
             self.password = self.passoutput[0]
             self.passdict = {}
-            for line in self.passoutput[1:]:
-                if ':' in line:
-                    name, value = line.split(':', 1)
-                    self.passdict[name.strip()] = value.strip()
+            try:
+                values = yaml.safe_load('\n'.join(self.passoutput[1:]))
+                for key, item in values.items():
+                    self.passdict[key] = item
+            except (yaml.YAMLError, AttributeError):
+                for line in self.passoutput[1:]:
+                    if ':' in line:
+                        name, value = line.split(':', 1)
+                        self.passdict[name.strip()] = value.strip()
         except (subprocess.CalledProcessError) as e:
             if e.returncode != 0 and 'not in the password store' in e.output:
                 # if pass returns 1 and return string contains 'is not in the password store.'


Test patch:
diff --git a/tests/integration/targets/lookup_passwordstore/tasks/tests.yml b/tests/integration/targets/lookup_passwordstore/tasks/tests.yml
--- a/tests/integration/targets/lookup_passwordstore/tasks/tests.yml
+++ b/tests/integration/targets/lookup_passwordstore/tasks/tests.yml
@@ -60,3 +60,63 @@
   assert:
     that:
       - readpass == newpass
+
+# As inserting multiline passwords on the commandline would require something
+# like expect, simply create it by using default gpg on a file with the correct
+# structure.
+- name: Create the YAML password content
+  copy:
+    dest: "~/.password-store/test-yaml-pass"
+    content: |
+      testpassword
+      key: |
+        multi
+        line
+
+- name: Read .gpg-id from .password-store
+  set_fact:
+    gpgid: "{{ lookup('file', '~/.password-store/.gpg-id') }}"
+
+- name: Encrypt the file using the gpg key
+  command: "{{ gpg2_bin }} --batch --encrypt -r {{ gpgid }} ~/.password-store/test-yaml-pass"
+
+- name: Fetch a password with YAML subkey
+  set_fact:
+    readyamlpass: "{{ lookup('community.general.passwordstore', 'test-yaml-pass subkey=key') }}"
+
+- name: Read a yaml subkey
+  assert:
+    that:
+      - readyamlpass == 'multi\nline'
+
+- name: Create a non-YAML multiline file
+  copy:
+    dest: "~/.password-store/test-multiline-pass"
+    content: |
+      testpassword
+      random additional line
+
+- name: Read .gpg-id from .password-store
+  set_fact:
+    gpgid: "{{ lookup('file', '~/.password-store/.gpg-id') }}"
+
+- name: Encrypt the file using the gpg key
+  command: "{{ gpg2_bin }} --batch --encrypt -r {{ gpgid }} ~/.password-store/test-multiline-pass"
+
+- name: Fetch password from multiline file
+  set_fact:
+    readyamlpass: "{{ lookup('community.general.passwordstore', 'test-multiline-pass') }}"
+
+- name: Multiline pass only returns first line
+  assert:
+    that:
+      - readyamlpass == 'testpassword'
+
+- name: Fetch all from multiline file
+  set_fact:
+    readyamlpass: "{{ lookup('community.general.passwordstore', 'test-multiline-pass returnall=yes') }}"
+
+- name: Multiline pass returnall returns everything in the file
+  assert:
+    that:
+      - readyamlpass == 'testpassword\nrandom additional line'