Repository: freedomofpress/securedrop

Issue:
[xenial] Journalist replies are not displayed in source interface
## Steps to Reproduce

* Spin up a Xenial staging environment (`make build-debs-xenial` and `molecule converge -s libvirt-staging-xenial`)
* Submit a document or message from the source interface
* Reply to said document or message
* Observe empty reply on the source interface

## Expected Behavior

Source should be able to view journalist reply
## Actual Behavior

Journalist reply is blank in source interface

## Comments

Several changes were made in gpg 2.1 which is being used by Xenial [0], specifically the removal of the secret keyring

[0] : https://www.gnupg.org/faq/whats-new-in-2.1.html



Discussion:
I took a look at this today, here's where I'm at so far: I can reproduce this bug in a Xenial-based SecureDrop dev container. 

The secret keys for decrypting the replies are available, e.g.:

```
>>> self.gpg.list_keys(secret=True)
gnupg._parsers.ListKeys([{'dummy': u'', 'keyid': u'53ABF186E68948B9', 'expires': u'', 'rev': {}, 'sigs': {u'Autogenerated Key <JO2JDJSLGIPNTKEDZUECNFMUG2QYXVAS6VAMNXIMIAAIORSFJSG  : []}, 'subkeys': [], 'length': u'4096', 'ownertrust': u'u', 'algo': u'1', 'fingerprint': u'DE410BBA310AA8745E3BE7E653ABF186E68948B9', 'date': u'1368489600', 'trust': u'u', 'type': u'sec', 'uids': [u'Autogenerated Key <JO2JDJSLGIPNTKEDZUECNFMUG2QYXVAS6VAMNXIMIAAIORSFJSG  ]  }, {'dummy': u'', 'keyid': u'5761A79A11779651', 'expires': u'', 'rev': {}, 'sigs': {u'Autogenerated Key <3OQPA6YBNFBM3BEQH7KHYWPNLDBHKHP76HP2SCW6324K45QDHVD  : []}, 'subkeys': [], 'length': u'4096', 'ownertrust': u'u', 'algo': u'1', 'fingerprint': u'D90BD4822154FF63A54B95A55761A79A11779651', 'date': u'1368489600', 'trust': u'u', 'type': u'sec', 'uids': [u'Autogenerated Key <3OQPA6YBNFBM3BEQH7KHYWPNLDBHKHP76HP2SCW6324K45QDHVD  ]  }, {'dummy': u'', 'keyid': u'B4863A093C809921', 'expires': u'', 'rev': {}, 'sigs': {u'Autogenerated Key <ZOO3FPHBLKQQREIQ7ECPMGTPVAFXUSLEDUY2ZENQSWRWABALMPS  : []}, 'subkeys': [], 'length': u'4096', 'ownertrust': u'u', 'algo': u'1', 'fingerprint': u'2E945059A3A56EAE4F15DD7DB4863A093C809921', 'date': u'1368489600', 'trust': u'u', 'type': u'sec', 'uids': [u'Autogenerated Key <ZOO3FPHBLKQQREIQ7ECPMGTPVAFXUSLEDUY2ZENQSWRWABALMPS  ]  }])
```

But indeed, when I try decrypting a test file, the decrypted content is empty (one sees `self.gpg` below because I put a breakpoint in `crypto_util.py`):

```
>>> input_data = self.gpg.gen_key_input(name_email='jen+test@freedom.press', passphrase='test')
>>> key = self.gpg.gen_key(input_data)
>>> key.fingerprint
u'E27203CAB6B15F1B4A19260276A8A236B3C334CA'
>>> encrypted = self.gpg.encrypt('test data', key.fingerprint)
>>> encrypted.data
'-----BEGIN PGP MESSAGE-----\n\nhQEMA8k3DSXIftBUAQf/YZg9yiIRoWv8NEtwLFR2j  
>>> encrypted.ok
True
>>> decrypted = self.gpg.decrypt(str(encrypted), passphrase='test')
>>> decrypted.data
''
>>> decrypted.ok
False
>>> decrypted.stderr
u'[GNUPG:] ENC_TO C9370D25C87ED054 1 0\n[GNUPG:] PINENTRY_LAUNCHED 984\ngpg: encrypted with 2048-bit RSA key, ID C87ED054, created 2019-01-11\n      "Autogenerated Key <jen+test@freedom.press>"\ngpg: public key decryption failed: Inappropriate ioctl for device\n[GNUPG:] ERROR pkdecrypt_failed 83918950\n[GNUPG:] BEGIN_DECRYPTION\n[GNUPG:] DECRYPTION_FAILED\ngpg: decryption failed: No secret key\n[GNUPG:] END_DECRYPTION\n'  
>>> 
```

This looks to be the `PINENTRY_LAUNCHED` issue described in  https://github.com/isislovecruft/python-gnupg/issues/98,  https://github.com/isislovecruft/python-gnupg/issues/122, and https://github.com/isislovecruft/python-gnupg/issues/137. The version of python-gnupg we are using includes the patches that correspond to those issues (released in 2.2.0, we're on 2.3.1). In those issues modifying `gpg-agent.conf` to allow the loopback option resolved the issue. I haven't tried yet, but will try this next. If anyone beats me to it, please comment on this issue. 
Running the full application test suite on Xenial indicates that there are a [number of other test failures](https://circleci.com/gh/freedomofpress/securedrop/20624) which need to be investigated and resolved (will scope as part of this issue and file followups where appropriate). 
Adding the loopback pinentry configuration did not help in my system. It still failed to decrypt.

Resolving patch:
diff --git a/securedrop/crypto_util.py b/securedrop/crypto_util.py
--- a/securedrop/crypto_util.py
+++ b/securedrop/crypto_util.py
@@ -1,7 +1,8 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
-import gnupg
+from distutils.version import StrictVersion
+import pretty_bad_protocol as gnupg
 import os
 import io
 import scrypt
@@ -11,7 +12,7 @@
 from base64 import b32encode
 from datetime import date
 from flask import current_app
-from gnupg._util import _is_stream, _make_binary_stream
+from pretty_bad_protocol._util import _is_stream, _make_binary_stream
 
 import typing
 # https://www.python.org/dev/peps/pep-0484/#runtime-or-type-checking
@@ -79,7 +80,16 @@ def __init__(self,
 
         self.do_runtime_tests()
 
-        self.gpg = gnupg.GPG(binary='gpg2', homedir=gpg_key_dir)
+        # --pinentry-mode, required for SecureDrop on gpg 2.1.x+, was
+        # added in gpg 2.1.
+        self.gpg_key_dir = gpg_key_dir
+        gpg_binary = gnupg.GPG(binary='gpg2', homedir=self.gpg_key_dir)
+        if StrictVersion(gpg_binary.binary_version) >= StrictVersion('2.1'):
+            self.gpg = gnupg.GPG(binary='gpg2',
+                                 homedir=gpg_key_dir,
+                                 options=['--pinentry-mode loopback'])
+        else:
+            self.gpg = gpg_binary
 
         # map code for a given language to a localized wordlist
         self.__language2words = {}  # type: Dict[Text, List[str]]
@@ -176,7 +186,7 @@ def genkeypair(self, name, secret):
         """
         name = clean(name)
         secret = self.hash_codename(secret, salt=self.scrypt_gpg_pepper)
-        return self.gpg.gen_key(self.gpg.gen_key_input(
+        genkey_obj = self.gpg.gen_key(self.gpg.gen_key_input(
             key_type=self.GPG_KEY_TYPE,
             key_length=self.__gpg_key_length,
             passphrase=secret,
@@ -184,6 +194,7 @@ def genkeypair(self, name, secret):
             creation_date=self.DEFAULT_KEY_CREATION_DATE.isoformat(),
             expire_date=self.DEFAULT_KEY_EXPIRATION_DATE
         ))
+        return genkey_obj
 
     def delete_reply_keypair(self, source_filesystem_id):
         key = self.getkey(source_filesystem_id)
@@ -191,10 +202,12 @@ def delete_reply_keypair(self, source_filesystem_id):
         # keypair
         if not key:
             return
-        # The private key needs to be deleted before the public key can be
-        # deleted. http://pythonhosted.org/python-gnupg/#deleting-keys
-        self.gpg.delete_keys(key, True)  # private key
-        self.gpg.delete_keys(key)  # public key
+
+        # Always delete keys without invoking pinentry-mode = loopback
+        # see: https://lists.gnupg.org/pipermail/gnupg-users/2016-May/055965.html
+        temp_gpg = gnupg.GPG(binary='gpg2', homedir=self.gpg_key_dir)
+        # The subkeys keyword argument deletes both secret and public keys.
+        temp_gpg.delete_keys(key, secret=True, subkeys=True)
 
     def getkey(self, name):
         for key in self.gpg.list_keys():
diff --git a/securedrop/secure_tempfile.py b/securedrop/secure_tempfile.py
--- a/securedrop/secure_tempfile.py
+++ b/securedrop/secure_tempfile.py
@@ -4,7 +4,7 @@
 import io
 from tempfile import _TemporaryFileWrapper
 
-from gnupg._util import _STREAMLIKE_TYPES
+from pretty_bad_protocol._util import _STREAMLIKE_TYPES
 from cryptography.exceptions import AlreadyFinalized
 from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives.ciphers.algorithms import AES
diff --git a/securedrop/source_app/main.py b/securedrop/source_app/main.py
--- a/securedrop/source_app/main.py
+++ b/securedrop/source_app/main.py
@@ -86,9 +86,8 @@ def lookup():
             try:
                 with io.open(reply_path, "rb") as f:
                     contents = f.read()
-                reply.decrypted = current_app.crypto_util.decrypt(
-                    g.codename,
-                    contents).decode('utf-8')
+                reply_obj = current_app.crypto_util.decrypt(g.codename, contents)
+                reply.decrypted = reply_obj.decode('utf-8')
             except UnicodeDecodeError:
                 current_app.logger.error("Could not decode reply %s" %
                                          reply.filename)


Test patch:
diff --git a/securedrop/tests/conftest.py b/securedrop/tests/conftest.py
--- a/securedrop/tests/conftest.py
+++ b/securedrop/tests/conftest.py
@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 
-import gnupg
+import pretty_bad_protocol as gnupg
 import logging
 import os
 import io
@@ -89,6 +89,11 @@ def config(tmpdir):
     tmp = data.mkdir('tmp')
     sqlite = data.join('db.sqlite')
 
+    # gpg 2.1+ requires gpg-agent, see #4013
+    gpg_agent_config = str(keys.join('gpg-agent.conf'))
+    with open(gpg_agent_config, 'w+') as f:
+        f.write('allow-loopback-pinentry')
+
     gpg = gnupg.GPG('gpg2', homedir=str(keys))
     for ext in ['sec', 'pub']:
         with io.open(path.join(path.dirname(__file__),
diff --git a/securedrop/tests/test_integration.py b/securedrop/tests/test_integration.py
--- a/securedrop/tests/test_integration.py
+++ b/securedrop/tests/test_integration.py
@@ -1,12 +1,10 @@
 # -*- coding: utf-8 -*-
 
-import gnupg
+from distutils.version import StrictVersion
 import gzip
 import os
 import random
 import re
-import shutil
-import tempfile
 import zipfile
 
 from base64 import b32encode
@@ -326,11 +324,10 @@ def assertion():
 
     zf = zipfile.ZipFile(StringIO(resp.data), 'r')
     data = zf.read(zf.namelist()[0])
-    _can_decrypt_with_key(journalist_app, data, config.JOURNALIST_KEY)
+    _can_decrypt_with_key(journalist_app, data)
     _can_decrypt_with_key(
         journalist_app,
         data,
-        current_app.crypto_util.getkey(filesystem_id),
         codename)
 
     # Test deleting reply on the journalist interface
@@ -400,39 +397,28 @@ def assertion():
     utils.async.wait_for_assertion(assertion)
 
 
-def _can_decrypt_with_key(journalist_app, msg, key_fpr, passphrase=None):
+def _can_decrypt_with_key(journalist_app, msg, passphrase=None):
     """
-    Test that the given GPG message can be decrypted with the given key
-    (identified by its fingerprint).
+    Test that the given GPG message can be decrypted.
     """
-    # GPG does not provide a way to specify which key to use to decrypt a
-    # message. Since the default keyring that we use has both the
-    # `config.JOURNALIST_KEY` and all of the reply keypairs, there's no way
-    # to use it to test whether a message is decryptable with a specific
-    # key.
-    gpg_tmp_dir = tempfile.mkdtemp()
-    gpg = gnupg.GPG('gpg2', homedir=gpg_tmp_dir)
-
-    # Export the key of interest from the application's keyring
-    pubkey = journalist_app.crypto_util.gpg.export_keys(key_fpr)
-    seckey = journalist_app.crypto_util.gpg.export_keys(key_fpr, secret=True)
-    # Import it into our isolated temporary GPG directory
-    for key in (pubkey, seckey):
-        gpg.import_keys(key)
-
-    # Attempt decryption with the given key
+
+    # For gpg 2.1+, a non null passphrase _must_ be passed to decrypt()
+    using_gpg_2_1 = StrictVersion(
+        journalist_app.crypto_util.gpg.binary_version) >= StrictVersion('2.1')
+
     if passphrase:
         passphrase = journalist_app.crypto_util.hash_codename(
             passphrase,
             salt=journalist_app.crypto_util.scrypt_gpg_pepper)
-    decrypted_data = gpg.decrypt(msg, passphrase=passphrase)
+    elif using_gpg_2_1:
+        passphrase = 'dummy passphrase'
+
+    decrypted_data = journalist_app.crypto_util.gpg.decrypt(
+        msg, passphrase=passphrase)
     assert decrypted_data.ok, \
         "Could not decrypt msg with key, gpg says: {}" \
         .format(decrypted_data.stderr)
 
-    # We have to clean up the temporary GPG dir
-    shutil.rmtree(gpg_tmp_dir)
-
 
 def test_reply_normal(journalist_app,
                       source_app,
diff --git a/securedrop/tests/test_journalist.py b/securedrop/tests/test_journalist.py
--- a/securedrop/tests/test_journalist.py
+++ b/securedrop/tests/test_journalist.py
@@ -1586,13 +1586,16 @@ def test_render_locales(config, journalist_app, test_journo, test_source):
     assert url_end + '?l=en_US' in text, text
 
 
-def test_render_xenial_positive(config, journalist_app, test_journo):
+def test_render_xenial_positive(config, journalist_app, test_journo, mocker):
     yesterday = datetime.datetime.now() - datetime.timedelta(days=1)
     journalist_app.config.update(
         XENIAL_WARNING_DATE=yesterday,
         XENIAL_VER='16.04'
     )
 
+    mocked_error_platform = mocker.patch('platform.linux_distribution')
+    mocked_error_platform.return_value = ('Ubuntu', '14.04', 'trusty')
+
     with journalist_app.test_client() as app:
         _login_user(app, test_journo['username'], test_journo['password'],
                     test_journo['otp_secret'])
@@ -1603,13 +1606,16 @@ def test_render_xenial_positive(config, journalist_app, test_journo):
     assert "critical-skull" in text, text
 
 
-def test_render_xenial_negative_version(config, journalist_app, test_journo):
+def test_render_xenial_negative_version(config, journalist_app, test_journo, mocker):
     yesterday = datetime.datetime.now() - datetime.timedelta(days=1)
     journalist_app.config.update(
         XENIAL_WARNING_DATE=yesterday,
-        XENIAL_VER='14.04'
+        XENIAL_VER='16.04'
     )
 
+    mocked_error_platform = mocker.patch('platform.linux_distribution')
+    mocked_error_platform.return_value = ('Ubuntu', '16.04', 'xenial')
+
     with journalist_app.test_client() as app:
         _login_user(app, test_journo['username'], test_journo['password'],
                     test_journo['otp_secret'])
diff --git a/securedrop/tests/test_secure_tempfile.py b/securedrop/tests/test_secure_tempfile.py
--- a/securedrop/tests/test_secure_tempfile.py
+++ b/securedrop/tests/test_secure_tempfile.py
@@ -3,7 +3,7 @@
 import os
 import pytest
 
-from gnupg._util import _is_stream
+from pretty_bad_protocol._util import _is_stream
 
 os.environ['SECUREDROP_ENV'] = 'test'  # noqa
 from secure_tempfile import SecureTemporaryFile
diff --git a/securedrop/tests/utils/env.py b/securedrop/tests/utils/env.py
--- a/securedrop/tests/utils/env.py
+++ b/securedrop/tests/utils/env.py
@@ -1,7 +1,8 @@
 # -*- coding: utf-8 -*-
 """Testing utilities related to setup and teardown of test environment.
 """
-import gnupg
+from distutils.version import StrictVersion
+import pretty_bad_protocol as gnupg
 import os
 import io
 import shutil
@@ -37,7 +38,20 @@ def init_gpg():
     """Initialize the GPG keyring and import the journalist key for
     testing.
     """
-    gpg = gnupg.GPG('gpg2', homedir=config.GPG_KEY_DIR)
+
+    # gpg 2.1+ requires gpg-agent, see #4013
+    gpg_agent_config = os.path.join(config.GPG_KEY_DIR, 'gpg-agent.conf')
+    with open(gpg_agent_config, 'w+') as f:
+        f.write('allow-loopback-pinentry')
+
+    gpg_binary = gnupg.GPG(binary='gpg2', homedir=config.GPG_KEY_DIR)
+    if StrictVersion(gpg_binary.binary_version) >= StrictVersion('2.1'):
+        gpg = gnupg.GPG(binary='gpg2',
+                        homedir=config.GPG_KEY_DIR,
+                        options=['--pinentry-mode loopback'])
+    else:
+        gpg = gpg_binary
+
     # Faster to import a pre-generated key than to gen a new one every time.
     for keyfile in (join(FILES_DIR, "test_journalist_key.pub"),
                     join(FILES_DIR, "test_journalist_key.sec")):