Repository: ansible-collections/community.general

Issue:
passwordstore lookup plugin gopass compatibility
### Summary

The passwordstore lookup plugin uses the binary `pass`. The plugin used to work fine with the [gopass](https://www.gopass.pw) drop-in replacement using a small [wrapper script](https://github.com/takelwerk/takelage-dev/blob/main/ansible/roles/takel_gopass/templates/pass.j2). But sadly, that does not work anymore. 

The reason is that the passwordstore plugin checks the internal pass configuration. It checks for [the `~/.password-store` directory](https://github.com/ansible-collections/community.general/blob/0be68bf04bd9b8dcf99583ebf5ad6deac4736283/plugins/lookup/passwordstore.py#L274) and [the `.gpg` file](https://github.com/ansible-collections/community.general/blob/0be68bf04bd9b8dcf99583ebf5ad6deac4736283/plugins/lookup/passwordstore.py#L305) at a location where `pass` would store it. If either fails the plugin throws an error.

For me, the main advantage of `gopass` over `pass` is the mounts feature which allows multiple password stores at arbitrary paths. They happen to be stored somewhere completely different than `pass` would store them but the interface to retrieve them is the same. I think that it's not a good idea if the passwordstore lookup plugin verifies the backend's configuration. It should rely on the `pass` interface instead.

I've tried to understand why the passwordstore plugin checks for the directory and the gpg file and I have two questions:

Why is there an [`else`](https://github.com/ansible-collections/community.general/blob/0be68bf04bd9b8dcf99583ebf5ad6deac4736283/plugins/lookup/passwordstore.py#L276) which kicks in if the passwordstore directory does not exist? Isn't this handled by `pass`?

Then there is [a comment](https://github.com/ansible-collections/community.general/blob/0be68bf04bd9b8dcf99583ebf5ad6deac4736283/plugins/lookup/passwordstore.py#L306) I do not understand:
```
# Only accept password as found, if there a .gpg file for it (might be a tree node otherwise)
```
What kind of tree node is this? This looks suspiciously like a workaround for some `pass` bug which should be fixed upstream. 

For the time being, I've implemented a workaround for our (virtual) development environment by applying [a simple patch](https://github.com/takelwerk/takelage-dev/blob/main/ansible/roles/takel_takelage/files/patch/passwordstore.py.patch) which removes both checks. Of course, this is hardly a general solution, as those checks probably exist for a reason. Could someone please tell me for which reason?

### Issue Type

Bug Report

### Component Name

passwordstore lookup plugin

### Ansible Version

```console (paste below)
$ ansible --version
ansible [core 2.12.6]
  config file = None
  configured module search path = ['/Users/tovenaar/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/dist-packages/ansible
  ansible collection location = /Users/tovenaar/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]
  jinja version = 3.1.2
  libyaml = True
```


### Community.general Version

```console (paste below)
$ ansible-galaxy collection list community.general
Collection        Version
----------------- -------
community.general 4.8.1
```


### Configuration

```console (paste below)
$ ansible-config dump --only-changed
```


### OS / Environment

Debian GNU/Linux 11 (bullseye), cf. [takelage-dev](https://github.com/takelwerk/takelage-dev)

### Steps to Reproduce

Use `gopass` as backend instead of `pass`.

### Expected Results

`gopass` should work as a passwordstore lookup plugin backend as its interface is the same as the `pass` interface.

### Actual Results

The passwordstore lookup plugin complains about the missing `pass` configuration directory and missing gpg files.

### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* `lib/ansible/plugins/lookup`

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
This was introduced in #4192, for the rationale see that PR. Maybe @grembo, the PR's author, can also comment on this.
> This was introduced in #4192, for the rationale see that PR. Maybe @grembo, the PR's author, can also comment on this.

@stephan-devop  Please see #4185 for a detailed description of the problem this fixes, short version: `pass` doesn't distinguish between a password stored or a tree node of the same name. It has been like that forever. This isn't a problem for interactive use (which is what `pass` was designed to do), as a human would of course notice the problem.

One is of course free to open an issue there and request a change in behavior, but this will take a while to trickle down into the field reliably (I would assume for it to take years), also because it should somehow be backwards compatible, so probably adding a command line switch, which could then be added to the plugin combined with a version check.

Now, nowhere does the plugin state that it supports anything besides "real" passwordstore, that's why I optimized the behavior for it to stop horrible things to happen by accident (e.g., using the tree node name as a cryptographic key is terribly insecure) . Therefore this wasn't foreseen, but having a compatibility shim is not something outlandish either, so I can feel your pain.

That said, I see multiple practical solutions:
1. Add a gopass plugin to ansible, so it becomes a first class citizen (some work and forces gopass on all users - so if you happen to also have users using pass, probably not a good solution).
2. Alter the passwordstore plugin to have a way to disable the check automatically (e.g. based on the output of `pass --version`)
3. Alter the passwordstore plugin to add a conditional variable that tells it to skip these checks (something like `pass_shim=true/false`). I would assume gopass doesn't need those checks.

The last option is probably the most realistic short term solution to your specific problem.

There have been fixes in the past (I quickly found #1493, #1589) to make the lookup compatible with gopass, but we do not explicitly declare that it is compatible (for that we should have tests for gopass as well). 2 and 3 sound like good solutions to me, but someone would have to implement them, preferably with tests.
> @stephan-devop Please see #4185 for a detailed description of the problem this fixes, short version: `pass` doesn't distinguish between a password stored or a tree node of the same name. 

Why don't we check if the `password` is equal to the `passname` and if so then reject it?
Because

1. This is not what is returned in the first place
2. What is returned exactly can also differ based on the version of `tree` installed (and potentially the character set installed)
3. It is not precise - what we do now is exactly the same check that pass does, so it’s a really very high chance it’s doing the right thing vs a hack like comparing the results of the operation
4. It prevents storing ‘passname’ as a password (well, that’s not a terrible limitation to have, but still, kind of unpredicted behavior, imagine storing “test” in “test” and then the plug-in telling you “password not found”)

Adding what I proposed above is a few lines of code changed plus another five lines of documentation (which then should also mention gopass and the wrapper script).

Is your gopass wrapper available somewhere (if yes, it should be linked to from documentation so it’s actually a thing).
 
I agree with your reasoning and it's fine for me. I've opted for 2 as I think it is more user-friendly. This is my first pull request for this project so please point out any code style violations I might have committed.

I've created a [draft pull request](https://github.com/ansible-collections/community.general/pull/4779) introducing a variable `self.backend` which is set to the string `pass` by default. Only if the output of `pass --version` contains the string `gopass` then `self.backend` is set to `gopass`. Both exceptions for pass are applied unless a different backend has been detected.

> Adding what I proposed above is a few lines of code changed plus another five lines of documentation (which then should also mention gopass and the wrapper script).

I've linked the [wrapper script](https://github.com/takelwerk/takelage-dev/blob/main/ansible/roles/takel_gopass/templates/pass.j2) above. But the logical next step would be to incorporate the script into the lookup plugin by adding "exceptions" for gopass, right?

Then I had a look at the integration tests. How do I run them locally? Is there some kind of docker environment available?
The [wrapper script](https://github.com/takelwerk/takelage-dev/blob/main/ansible/roles/takel_gopass/templates/pass.j2) does two things which I've added to the plugin:
1. It calls `gopass version` to get rid of the "You haven't checked for updates in a while." message which confuses the parser because it expects a password. This part is already handled by the `setup_backend()` method which calls `pass --version` (which is nearly the same and should be ok, too, I guess).
2. It adds `--password` to `gopass show`. Someone in the gopass team apparently thought that it's a smart idea to make the verbose output the default. So we have to add the flag `--password` to fall back to good old pass behaviour: show shows the password and nothing but the password.
Hi @stephan-devop,

I also created a patch (should have probably coordinated with you).

I also went for option 2, but in a lighter way:
1. Only check the version of the pass utility if there is a problem (therefore it's zero cost for `pass` users)
2. Don't be gopass specific
3. Add integration tests (this was the nasty bit)

Now, I don't want to take away your steam/spirit, but I think this is an easier implementation (also to review). It lacks the extra "--password" flag of course, this would still have to be done in the wrapper.

I'll open a draft review, so we can compare.
 

Resolving patch:
diff --git a/plugins/lookup/passwordstore.py b/plugins/lookup/passwordstore.py
--- a/plugins/lookup/passwordstore.py
+++ b/plugins/lookup/passwordstore.py
@@ -106,6 +106,22 @@
         type: str
         default: 15m
         version_added: 4.5.0
+      backend:
+        description:
+          - Specify which backend to use.
+          - Defaults to C(pass), passwordstore.org's original pass utility.
+          - C(gopass) support is incomplete.
+        ini:
+          - section: passwordstore_lookup
+            key: backend
+        vars:
+          - name: passwordstore_backend
+        type: str
+        default: pass
+        choices:
+          - pass
+          - gopass
+        version_added: 5.2.0
 '''
 EXAMPLES = """
 ansible.cfg: |
@@ -231,6 +247,24 @@ def check_output2(*popenargs, **kwargs):
 
 
 class LookupModule(LookupBase):
+    def __init__(self, loader=None, templar=None, **kwargs):
+
+        super(LookupModule, self).__init__(loader, templar, **kwargs)
+        self.realpass = None
+
+    def is_real_pass(self):
+        if self.realpass is None:
+            try:
+                self.passoutput = to_text(
+                    check_output2([self.pass_cmd, "--version"], env=self.env),
+                    errors='surrogate_or_strict'
+                )
+                self.realpass = 'pass: the standard unix password manager' in self.passoutput
+            except (subprocess.CalledProcessError) as e:
+                raise AnsibleError(e)
+
+        return self.realpass
+
     def parse_params(self, term):
         # I went with the "traditional" param followed with space separated KV pairs.
         # Waiting for final implementation of lookup parameter parsing.
@@ -270,10 +304,12 @@ def parse_params(self, term):
             self.env = os.environ.copy()
             self.env['LANGUAGE'] = 'C'  # make sure to get errors in English as required by check_output2
 
-            # Set PASSWORD_STORE_DIR
-            if os.path.isdir(self.paramvals['directory']):
+            if self.backend == 'gopass':
+                self.env['GOPASS_NO_REMINDER'] = "YES"
+            elif os.path.isdir(self.paramvals['directory']):
+                # Set PASSWORD_STORE_DIR
                 self.env['PASSWORD_STORE_DIR'] = self.paramvals['directory']
-            else:
+            elif self.is_real_pass():
                 raise AnsibleError('Passwordstore directory \'{0}\' does not exist'.format(self.paramvals['directory']))
 
             # Set PASSWORD_STORE_UMASK if umask is set
@@ -288,7 +324,9 @@ def parse_params(self, term):
     def check_pass(self):
         try:
             self.passoutput = to_text(
-                check_output2(["pass", "show", self.passname], env=self.env),
+                check_output2([self.pass_cmd, 'show'] +
+                              (['--password'] if self.backend == 'gopass' else []) +
+                              [self.passname], env=self.env),
                 errors='surrogate_or_strict'
             ).splitlines()
             self.password = self.passoutput[0]
@@ -302,8 +340,10 @@ def check_pass(self):
                     if ':' in line:
                         name, value = line.split(':', 1)
                         self.passdict[name.strip()] = value.strip()
-            if os.path.isfile(os.path.join(self.paramvals['directory'], self.passname + ".gpg")):
-                # Only accept password as found, if there a .gpg file for it (might be a tree node otherwise)
+            if (self.backend == 'gopass' or
+                    os.path.isfile(os.path.join(self.paramvals['directory'], self.passname + ".gpg"))
+                    or not self.is_real_pass()):
+                # When using real pass, only accept password as found if there is a .gpg file for it (might be a tree node otherwise)
                 return True
         except (subprocess.CalledProcessError) as e:
             # 'not in password store' is the expected error if a password wasn't found
@@ -339,7 +379,7 @@ def update_password(self):
         if self.paramvals['backup']:
             msg += "lookup_pass: old password was {0} (Updated on {1})\n".format(self.password, datetime)
         try:
-            check_output2(['pass', 'insert', '-f', '-m', self.passname], input=msg, env=self.env)
+            check_output2([self.pass_cmd, 'insert', '-f', '-m', self.passname], input=msg, env=self.env)
         except (subprocess.CalledProcessError) as e:
             raise AnsibleError(e)
         return newpass
@@ -351,7 +391,7 @@ def generate_password(self):
         datetime = time.strftime("%d/%m/%Y %H:%M:%S")
         msg = newpass + '\n' + "lookup_pass: First generated by ansible on {0}\n".format(datetime)
         try:
-            check_output2(['pass', 'insert', '-f', '-m', self.passname], input=msg, env=self.env)
+            check_output2([self.pass_cmd, 'insert', '-f', '-m', self.passname], input=msg, env=self.env)
         except (subprocess.CalledProcessError) as e:
             raise AnsibleError(e)
         return newpass
@@ -380,6 +420,8 @@ def opt_lock(self, type):
             yield
 
     def setup(self, variables):
+        self.backend = self.get_option('backend')
+        self.pass_cmd = self.backend  # pass and gopass are commands as well
         self.locked = None
         timeout = self.get_option('locktimeout')
         if not re.match('^[0-9]+[smh]$', timeout):
@@ -402,6 +444,7 @@ def setup(self, variables):
         }
 
     def run(self, terms, variables, **kwargs):
+        self.set_options(var_options=variables, direct=kwargs)
         self.setup(variables)
         result = []
 


Test patch:
diff --git a/tests/integration/targets/lookup_passwordstore/tasks/tests.yml b/tests/integration/targets/lookup_passwordstore/tasks/tests.yml
--- a/tests/integration/targets/lookup_passwordstore/tasks/tests.yml
+++ b/tests/integration/targets/lookup_passwordstore/tasks/tests.yml
@@ -19,6 +19,44 @@
     - "~/.gnupg"
     - "~/.password-store"
 
+- name: Get path of pass executable
+  command: which pass
+  register: result
+
+- name: Store path of pass executable
+  set_fact:
+     passpath: "{{ result.stdout }}"
+
+- name: Move original pass into place if there was a leftover
+  command:
+    argv:
+      - mv
+      - "{{ passpath }}.testorig"
+      - "{{ passpath }}"
+  args:
+    removes: "{{ passpath }}.testorig"
+
+# having gopass is not required for this test, but we store
+# its path in case it is installed, so we can restore it
+- name: Try to find gopass in path
+  command: which gopass
+  register: result
+  ignore_errors: yes
+
+- name: Store path of gopass executable
+  set_fact:
+     gopasspath: "{{ (result.rc == 0) |
+       ternary(result.stdout, (passpath | dirname, 'gopass') | path_join) }}"
+
+- name: Move original gopass into place if there was a leftover
+  command:
+    argv:
+      - mv
+      - "{{ gopasspath }}.testorig"
+      - "{{ gopasspath }}"
+  args:
+    removes: "{{ gopasspath }}.testorig"
+
 # How to generate a new GPG key:
 #   gpg2 --batch --gen-key input  # See templates/input
 #   gpg2 --list-secret-keys --keyid-format LONG
@@ -151,3 +189,163 @@
   assert:
     that:
       - readyamlpass == 'testpassword\nrandom additional line'
+
+- name: Create a password in a folder
+  set_fact:
+    newpass: "{{ lookup('community.general.passwordstore', 'folder/test-pass length=8 create=yes') }}"
+
+- name: Fetch password from folder
+  set_fact:
+    readpass: "{{ lookup('community.general.passwordstore', 'folder/test-pass') }}"
+
+- name: Verify password from folder
+  assert:
+    that:
+      - readpass == newpass
+
+- name: Try to read folder as passname
+  set_fact:
+    newpass: "{{ lookup('community.general.passwordstore', 'folder') }}"
+  ignore_errors: true
+  register: eval_error
+
+- name: Make sure reading folder as passname failed
+  assert:
+    that:
+      - eval_error is failed
+      - '"passname folder not found" in eval_error.msg'
+
+- name: Change passwordstore location explicitly
+  set_fact:
+    passwordstore: "{{ lookup('env','HOME') }}/.password-store"
+
+- name: Make sure password store still works with explicit location set
+  set_fact:
+    newpass: "{{ lookup('community.general.passwordstore', 'test-pass') }}"
+
+- name: Change passwordstore location to a non-existent place
+  set_fact:
+    passwordstore: "somenonexistentplace"
+
+- name: Try reading from non-existent passwordstore location
+  set_fact:
+    newpass: "{{ lookup('community.general.passwordstore', 'test-pass') }}"
+  ignore_errors: true
+  register: eval_error
+
+- name: Make sure reading from non-existent passwordstore location failed
+  assert:
+    that:
+      - eval_error is failed
+      - >-
+        "Passwordstore directory 'somenonexistentplace' does not exist" in eval_error.msg
+
+- name: Test pass compatibility shim detection
+  block:
+  - name: Move original pass out of the way
+    command:
+      argv:
+        - mv
+        - "{{ passpath }}"
+        - "{{ passpath }}.testorig"
+    args:
+      creates: "{{ passpath }}.testorig"
+
+  - name: Create dummy pass script
+    ansible.builtin.copy:
+      content: |
+        #!/bin/sh
+        echo "shim_ok"
+      dest: "{{ passpath }}"
+      mode: '0755'
+
+  - name: Try reading from non-existent passwordstore location with different pass utility
+    set_fact:
+      newpass: "{{ lookup('community.general.passwordstore', 'test-pass') }}"
+    environment:
+      PATH: "/tmp"
+
+  - name: Verify password received from shim
+    assert:
+      that:
+        - newpass == "shim_ok"
+
+  - name: Try to read folder as passname with a different pass utility
+    set_fact:
+      newpass: "{{ lookup('community.general.passwordstore', 'folder') }}"
+
+  - name: Verify password received from shim
+    assert:
+      that:
+        - newpass == "shim_ok"
+
+  always:
+  - name: Move original pass back into place
+    command:
+      argv:
+        - mv
+        - "{{ passpath }}.testorig"
+        - "{{ passpath }}"
+    args:
+      removes: "{{ passpath }}.testorig"
+
+- name: Very basic gopass compatibility test
+  vars:
+    passwordstore_backend: "gopass"
+  block:
+  - name: check if gopass executable exists
+    stat:
+      path: "{{ gopasspath }}"
+    register: gopass_check
+
+  - name: Move original gopass out of the way
+    command:
+      argv:
+        - mv
+        - "{{ gopasspath }}"
+        - "{{ gopasspath }}.testorig"
+    args:
+      creates: "{{ gopasspath }}.testorig"
+    when: gopass_check.stat.exists == true
+
+  - name: Create mocked gopass script
+    ansible.builtin.copy:
+      content: |
+        #!/bin/sh
+        if [ "$GOPASS_NO_REMINDER" != "YES" ]; then
+          exit 1
+        fi
+        if [ "$1" = "--version" ]; then
+          exit 2
+        fi
+        if [ "$1" = "show" ] && [ "$2" != "--password" ]; then
+          exit 3
+        fi
+        echo "gopass_ok"
+      dest: "{{ gopasspath }}"
+      mode: '0755'
+
+  - name: Try to read folder as passname using gopass
+    set_fact:
+      newpass: "{{ lookup('community.general.passwordstore', 'folder') }}"
+
+  - name: Verify password received from gopass
+    assert:
+      that:
+        - newpass == "gopass_ok"
+
+  always:
+  - name: Remove mocked gopass
+    ansible.builtin.file:
+      path: "{{ gopasspath }}"
+      state: absent
+
+  - name: Move original gopass back into place
+    command:
+      argv:
+        - mv
+        - "{{ gopasspath }}.testorig"
+        - "{{ gopasspath }}"
+    args:
+      removes: "{{ gopasspath }}.testorig"
+    when: gopass_check.stat.exists == true