Repository: spack/spack

Issue:
Spack reports multiple keys for gpg v2.2 even though only one was created
During the hackathon multiple people ran into an issue with gpg 2.2 because of the difference in output with `gpg --list-secret-keys --with-colons --fingerprint`. @opadron  was able to figure out that this was due to a difference in output from the different versions of gpg. 


### Steps to reproduce the issue

```shell
# use gpg v2.2.X
$ gpg --list-secret-keys --with-colons --fingerprint
sec:-:2048:1:XXXXXXXXXX:XXXXXXXX:::-:::escaESCA:::+:::23::0:
fpr:::::::::XXXXXXXXXXXXXX:
grp:::::::::XXXXXXXXXXXXXX:
uid:-::::XXXXXXXXX::XXXXXXXXX::Spack Build Pipeline (Demo Key) <key@spack.demo>::::::::::0:
ssb:-:2048:1:XXXXXXXXX::::::esa:::+:::23:
fpr:::::::::XXXXXXXXXXX:
grp:::::::::XXXXXXXXXXX:

```
```shell
# with gpg version 2.0.4
gpg --list-secret-keys --with-colons --fingerprint
sec::2048:1:XXXXXXXXXXX::::::::::
fpr:::::::::XXXXXXXXXXXXXX:
uid:::::::XXXXXXXXXXXXXXXX::Spack Build Pipeline (Demo Key) <key@spack.demo>:
ssb::2048:1:XXXXXXXXXXXXXXXX::::::::::
```

As you can see in gpg version 2.2 both public and private keys are included in the output. I think the way spack currently handles this is by finding the first occurrence of `fpr` but since there are two `fpr`s in later versions Spack thinks there are multiple keys.

### Error Message
```console
==> [2019-08-06-17:27:51.463007] creating binary cache file for package diffutils@3.7%gcc@8.2.0 arch=cray-cnl7-haswell 
Traceback (most recent call last):
  File "/global/project/projectdirs/swowner/JrAqyGHx/14/NERSC/spack-downstream-repo/lib/spack/spack/main.py", line 697, in main
    return _invoke_command(command, parser, args, unknown)
  File "/global/project/projectdirs/swowner/JrAqyGHx/14/NERSC/spack-downstream-repo/lib/spack/spack/main.py", line 447, in _invoke_command
    return_val = command(parser, args)
  File "/global/project/projectdirs/swowner/JrAqyGHx/14/NERSC/spack-downstream-repo/lib/spack/spack/cmd/buildcache.py", line 676, in buildcache
    args.func(args)
  File "/global/project/projectdirs/swowner/JrAqyGHx/14/NERSC/spack-downstream-repo/lib/spack/spack/cmd/buildcache.py", line 370, in createtarball
    not args.no_rebuild_index)
  File "/global/project/projectdirs/swowner/JrAqyGHx/14/NERSC/spack-downstream-repo/lib/spack/spack/binary_distribution.py", line 408, in build_tarball
    sign_tarball(key, force, specfile_path)
  File "/global/project/projectdirs/swowner/JrAqyGHx/14/NERSC/spack-downstream-repo/lib/spack/spack/binary_distribution.py", line 247, in sign_tarball
    raise PickKeyException(str(keys))
spack.binary_distribution.PickKeyException: Multi keys available for signing
[u'XXXXXXXXXXXX', u'XXXXXXXXXXXX']
Use spack buildcache create -k <key hash> to pick a key.
...
```

Omar did a quick fix in `gpg.py` but we're not sure if this is the right way to approach this issue since we've not considered every case of output.




Discussion:
I'm pretty confident that [this](https://github.com/spack/spack/blob/develop/lib/spack/spack/util/gpg.py#L47) is where the fix needs to go, and here's my interim fix:

```patch
commit fd93ae9a76610b130174c5dec8c0c10f6f4bfec4
Author: Omar Padron <omar.padron@kitware.com>
Date:   Wed Aug 7 16:40:36 2019 -0700

    (TEMPORARY) fix for elusive gpg-handling bug

diff --git a/lib/spack/spack/util/gpg.py b/lib/spack/spack/util/gpg.py
index f6fd7b7a2..b879d1055 100644
--- a/lib/spack/spack/util/gpg.py
+++ b/lib/spack/spack/util/gpg.py
@@ -48,9 +48,14 @@ class Gpg(object):
         keys = []
         output = cls.gpg()('--list-secret-keys', '--with-colons',
                            '--fingerprint', output=str)
+        found_sec = False
         for line in output.split('\n'):
-            if line.startswith('fpr'):
-                keys.append(line.split(':')[9])
+            if found_sec:
+                if line.startswith('fpr'):
+                    keys.append(line.split(':')[9])
+                    found_sec = False
+            elif line.startswith('sec'):
+                found_sec = True
         return keys
 
     @classmethod
```

I just want to do a little more homework on the problem, and try to see if there is possibly a more reliable way to parse out this information from gnupg (and prepare a unit test or two).

This would also be a good first issue if someone wanted to run with what I've put together so far instead of waiting for me to get around to it.
I have `gpg` version 2.2.4 installed on my machine, where the man page includes:

```
    --with-colons
              Print  key  listings  delimited  by  colons.  Note that the output will be encoded in UTF-8 regardless of any --display-charset setting. This format is useful when GnuPG is called from scripts and other programs as it is easily machine parsed. The details of this format are documented in the file 'doc/DETAILS’, which is included in the GnuPG source distribution.
```

That indicates to me that we're relatively safe intending to parse the output of listed secret keys, as long as we use the `--with-colons` option.  The file referenced there, for a recent master, can be viewed online [here](https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob_plain;f=doc/DETAILS;hb=refs/heads/master).  In that document, it is made clear that first field is the record type, and that `sec` is a `secret key` record, while `ssb` is a `secret subkey` record.  However, while that file has a pretty good description of the fields (what's in between the colons), it doesn't describe much about how the ordered lines of output are to be interpreted.

More reading through the `gpg` man page helps to clear that up though, especially the parts about the `--fingerprint` option.  Here's what I found to be relevant:

```
--fingerprint
              List all keys (or the specified ones) along with their fingerprints. This is the same output as --list-keys but with the additional output of a line with the fingerprint. May also be combined with --check-signatures.   If  this command is given twice, the fingerprints of all secondary keys are listed too.  This command also forces pretty printing of fingerprints if the keyid format has been set to "none".
```

and, re-iterating the part about giving the `--fingerprint` command twice:

```
--with-subkey-fingerprint
              If  a fingerprint is printed for the primary key, this option forces printing of the fingerprint for all subkeys.  This could also be achieved by using the --with-fingerprint twice but by using this option along with keyid-format "none" a compact fingerprint is printed.
```

Using an older version of `gpg` (2.0.22), I can verify that those option descriptions are true, i.e., a single `--fingerprint` option prints the fingerprints for keys (`sec` in this case) , while adding a second one additionally prints the fingerprints for subkeys (`sec` and `ssb` in this case):

```
$ gpg --version
gpg (GnuPG) 2.0.22
...
```

```
$ gpg --list-secret-keys --with-colons
sec::2048:1:FB578971ABEAEB08:1568062380:1568667180:::::::::
uid:::::::797908A322756EC462E6E075C2D2D1DF316AC6B7::Joe Smith (Testing ID) <joe.smith@smith.com>:
ssb::2048:1:59E1097CFDBDE496:1568062380::::::::::
sec::2048:1:4532173FD8EC833B:1568062426:1568667226:::::::::
uid:::::::6BC8025A00F4EFD53AE6CCB2B2271989A6EABF4C::John Smith (Testing ID) <john.smith@smith.com>:
ssb::2048:1:B2E44594CD89EB16:1568062426::::::::::
```

Above, secret and subkeys were listed, but with no fingerprints.

```
$ gpg --list-secret-keys --with-colons --fingerprint
sec::2048:1:FB578971ABEAEB08:1568062380:1568667180:::::::::
fpr:::::::::738FFFD8056D41BC06A9D586FB578971ABEAEB08:
uid:::::::797908A322756EC462E6E075C2D2D1DF316AC6B7::Joe Smith (Testing ID) <joe.smith@smith.com>:
ssb::2048:1:59E1097CFDBDE496:1568062380::::::::::
sec::2048:1:4532173FD8EC833B:1568062426:1568667226:::::::::
fpr:::::::::ACA823CFAB8D51CD57ED49E54532173FD8EC833B:
uid:::::::6BC8025A00F4EFD53AE6CCB2B2271989A6EABF4C::John Smith (Testing ID) <john.smith@smith.com>:
ssb::2048:1:B2E44594CD89EB16:1568062426::::::::::
```

Above, secret keys and subkeys were listed, with fingerprints only given for primary keys.  This is what spack currently asks for when it parses the output of `gpg --list-secret-keys ...`.

```
$ gpg --list-secret-keys --with-colons --fingerprint --fingerprint
sec::2048:1:FB578971ABEAEB08:1568062380:1568667180:::::::::
fpr:::::::::738FFFD8056D41BC06A9D586FB578971ABEAEB08:
uid:::::::797908A322756EC462E6E075C2D2D1DF316AC6B7::Joe Smith (Testing ID) <joe.smith@smith.com>:
ssb::2048:1:59E1097CFDBDE496:1568062380::::::::::
fpr:::::::::E5ECA4FFBF36146F1D4AFC2E59E1097CFDBDE496:
sec::2048:1:4532173FD8EC833B:1568062426:1568667226:::::::::
fpr:::::::::ACA823CFAB8D51CD57ED49E54532173FD8EC833B:
uid:::::::6BC8025A00F4EFD53AE6CCB2B2271989A6EABF4C::John Smith (Testing ID) <john.smith@smith.com>:
ssb::2048:1:B2E44594CD89EB16:1568062426::::::::::
fpr:::::::::111C19C9646D01FD53D694E8B2E44594CD89EB16:
```

Above, secret keys and subkeys were listed, with fingerprints given for both.  As we'll see below, that most closely matches the output we're getting from more recent `gpg` versions.

It seems that in recent versions of `gpg` (tested here with 2.2.17), the `--fingerprint` option is ignored (either one or both times it is used), and instead the output always appears as if two `--fingerprint` options were supplied:

```
$ gpg --version
gpg (GnuPG) 2.2.17
...
```

```
$ gpg --list-secret-keys --with-colons
sec:u:3072:1:8A011610CB031D35:1568067111:1568671911::u:::scESC:::+:::23::0:
fpr:::::::::959F791B5745DA2E7E9E41138A011610CB031D35:
grp:::::::::1CCA4BE1058FED45D7FBBFD660762EA9644F1860:
uid:u::::1568067111::FF2D35D021F75920284C9FD51D4511AD4F9CBD79::Joe Smith (Testing Id for Joe) <joe.smith@smith.com>::::::::::0:
ssb:u:3072:1:E94AE2763EEDC134:1568067111:1568671911:::::e:::+:::23:
fpr:::::::::8603C9F6266480E2664F8008E94AE2763EEDC134:
grp:::::::::63B41DB5B380293505447CB19AFB83A1F7EDBEAC:
sec:u:3072:1:1C183EBF9CF3479D:1568067152:1568671952::u:::scESC:::+:::23::0:
fpr:::::::::552CBC180F4F1C9C8549A8D31C183EBF9CF3479D:
grp:::::::::2ED02FCA1195360CC9D12C6E3DEB089178ADF9CC:
uid:u::::1568067152::AC06C0E60063AE91DD22CC2F6A0039DA9C4D1C49::John Smith (Testing ID for John) <john.smith@smith.com>::::::::::0:
ssb:u:3072:1:9A59706AFCD2FB61:1568067152:1568671952:::::e:::+:::23:
fpr:::::::::431E4A9ADA1CFF1A6D8E422A9A59706AFCD2FB61:
grp:::::::::C140A5D3E81EB4E5F4D6D10DFB4934FCFE0EE827:
```

The output below shows that adding one or two `--fingerprint` options makes no difference at all:

```
$ gpg --list-secret-keys --with-colons  > none_fp.txt
$ gpg --list-secret-keys --with-colons --fingerprint > one_fp.txt
$ gpg --list-secret-keys --with-colons --fingerprint --fingerprint > two_fp.txt
$ diff none_fp.txt one_fp.txt
$ diff one_fp.txt two_fp.txt
$
```

To summarize, it seems that newer `gpg` versions always behave as if fingerprints are requested, and are requested for each subkey as well.  Since that's the documented behavior when using `--fingerprint --fingerprint` (in old and new versions), and since old versions do, in fact, behave that way, it seems like the right solution to this issue in spack is first to supply two `--fingerprint` arguments to the `gpg --list-secret-keys ...` command (making older versions report fingerprints for subkeys), and second, do something similar to what @opadron has done already.  That is, we should only report the fingerprints for primary keys (i.e. fingerprints that come after a `sec` record).

Unless I hear objections, I'll put together a PR that does this (as well a provides some tests of the behavior using the generated output above).

Pinging @mathstuf since your name is on the code that parses `gpg` output in spack, and in case you have insight into the issue.


Resolving patch:
diff --git a/lib/spack/spack/util/gpg.py b/lib/spack/spack/util/gpg.py
--- a/lib/spack/spack/util/gpg.py
+++ b/lib/spack/spack/util/gpg.py
@@ -12,6 +12,21 @@
 GNUPGHOME = spack.paths.gpg_path
 
 
+def parse_keys_output(output):
+    keys = []
+    found_sec = False
+    for line in output.split('\n'):
+        if found_sec:
+            if line.startswith('fpr'):
+                keys.append(line.split(':')[9])
+                found_sec = False
+            elif line.startswith('ssb'):
+                found_sec = False
+        elif line.startswith('sec'):
+            found_sec = True
+    return keys
+
+
 class Gpg(object):
     @staticmethod
     def gpg():
@@ -45,13 +60,9 @@ def create(cls, **kwargs):
 
     @classmethod
     def signing_keys(cls):
-        keys = []
         output = cls.gpg()('--list-secret-keys', '--with-colons',
-                           '--fingerprint', output=str)
-        for line in output.split('\n'):
-            if line.startswith('fpr'):
-                keys.append(line.split(':')[9])
-        return keys
+                           '--fingerprint', '--fingerprint', output=str)
+        return parse_keys_output(output)
 
     @classmethod
     def export_keys(cls, location, *keys):


Test patch:
diff --git a/lib/spack/spack/test/util/util_gpg.py b/lib/spack/spack/test/util/util_gpg.py
new file mode 100644
--- /dev/null
+++ b/lib/spack/spack/test/util/util_gpg.py
@@ -0,0 +1,60 @@
+# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+import spack.util.gpg as gpg
+
+
+def test_parse_gpg_output_case_one():
+    # Two keys, fingerprint for primary keys, but not subkeys
+    output = """sec::2048:1:AAAAAAAAAAAAAAAA:AAAAAAAAAA:AAAAAAAAAA:::::::::
+fpr:::::::::XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:
+uid:::::::AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA::Joe (Test) <j.s@s.com>:
+ssb::2048:1:AAAAAAAAAAAAAAAA:AAAAAAAAAA::::::::::
+sec::2048:1:AAAAAAAAAAAAAAAA:AAAAAAAAAA:AAAAAAAAAA:::::::::
+fpr:::::::::YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY:
+uid:::::::AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA::Joe (Test) <j.s@s.com>:
+ssb::2048:1:AAAAAAAAAAAAAAAA:AAAAAAAAAA::::::::::
+"""
+    keys = gpg.parse_keys_output(output)
+
+    assert len(keys) == 2
+    assert keys[0] == 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
+    assert keys[1] == 'YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY'
+
+
+def test_parse_gpg_output_case_two():
+    # One key, fingerprint for primary key as well as subkey
+    output = """sec:-:2048:1:AAAAAAAAAA:AAAAAAAA:::-:::escaESCA:::+:::23::0:
+fpr:::::::::XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:
+grp:::::::::AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA:
+uid:-::::AAAAAAAAA::AAAAAAAAA::Joe (Test) <j.s@s.com>::::::::::0:
+ssb:-:2048:1:AAAAAAAAA::::::esa:::+:::23:
+fpr:::::::::YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY:
+grp:::::::::AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA:
+"""
+    keys = gpg.parse_keys_output(output)
+
+    assert len(keys) == 1
+    assert keys[0] == 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
+
+
+def test_parse_gpg_output_case_three():
+    # Two keys, fingerprint for primary keys as well as subkeys
+    output = """sec::2048:1:AAAAAAAAAAAAAAAA:AAAAAAAAAA:AAAAAAAAAA:::::::::
+fpr:::::::::WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW:
+uid:::::::AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA::Joe (Test) <j.s@s.com>:
+ssb::2048:1:AAAAAAAAAAAAAAAA:AAAAAAAAAA::::::::::
+fpr:::::::::XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:
+sec::2048:1:AAAAAAAAAAAAAAAA:AAAAAAAAAA:AAAAAAAAAA:::::::::
+fpr:::::::::YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY:
+uid:::::::AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA::Joe (Test) <j.s@s.com>:
+ssb::2048:1:AAAAAAAAAAAAAAAA:AAAAAAAAAA::::::::::
+fpr:::::::::ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ:"""
+
+    keys = gpg.parse_keys_output(output)
+
+    assert len(keys) == 2
+    assert keys[0] == 'WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW'
+    assert keys[1] == 'YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY'