Repository: freedomofpress/securedrop

Issue:
Unicode in messages throws internal server error
Writing unicode as as a message throws an internal server error. SecureDrop should support unicode in messages for those of us whose instances are used by non-English speakers. Not only does this happen, we get the generic "internal server error" message that doesn't explain what went wrong or how. This is very bad for sources who upon reaching a SD instance will be unable to submit, not see why, and presumably assume SD isn't a service worth using.

Moving to Python 3 (#997) is probably the easiest way to do this since we get better unicode handling for minimal overhead.



Discussion:
Duplicate from #1343 ?

@KwadroNaut Somewhat. Both the source and journalist app (SD 0.3.8) break if you pass in ö, ü, ä, or ß (that's all I tested, because that's the most pressing for me). That issue only mentions journalist replies. This specifies that this is a usability problem since non English speakers are likely to encounter "unresolvable" errors if they attempt to submit.

On a related note, I have mostly ported everything to Python 3 already. When I get a free evening, I can get the last error with hashing figured out.

https://github.com/heartsucker/securedrop/tree/python3

Hey guys. Since it seems like no one else is patching this, can I get some input to help me patch it faster?
@conorsch @garrettr @fowlslegs 

Currently the issue is that Isis's `python-gnupg` package in Python 3 attempts to read files and strings with the buffer interface. Since we wrap all the files with the `SecureTemporaryFile`, this interface isn't exposed. I haven't found a way to get this class to work with the GPG package.

In [this comment](https://github.com/freedomofpress/securedrop/issues/1283#issuecomment-218835538), it's said that the physical security of the servers is outside the threat model. Does this mean it is acceptable to remove that class entirely and just use the normal Python file class?

@heartsucker Can you provide test input to trigger failure? As @mdrose notes in #1343, the error doesn't seem to occur under 0.3.8 or develop. On both the Source Interface (as a text submission) and Document Interface (as a text reply), the following strings submit successfully with no errors:
- Ich möchte dir was sagen
- Hallo! ö, ü, ä, or ß
- 我不好說中文

So I don't think the problem is as simple as we first thought.

The string "Hallo! ö, ü, ä, or ß" breaks my instance. You can test the one
linked at https://berlinleaks.org (on mobile, short reply).

Am 10.08.2016 21:25 schrieb "Conor" notifications@github.com:

> @heartsucker https://github.com/heartsucker Can you provide test input
> to trigger failure? As @mdrose https://github.com/mdrose notes in #1343
> https://github.com/freedomofpress/securedrop/issues/1343, the error
> doesn't seem to occur under 0.3.8 or develop. On both the Source Interface
> (as a text submission) and Document Interface (as a text reply), the
> following strings submit successfully with no errors:
> - Ich möchte dir was sagen
> - Hallo! ö, ü, ä, or ß
> - 我不好說中文
> 
> So I don't think the problem is as simple as we first thought.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> https://github.com/freedomofpress/securedrop/issues/1360#issuecomment-238976216,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AD0DAEByUSffcwa279FTFD4_ot-GeuJDks5qeiWegaJpZM4JOUO4
> .

Update: I can trigger the error with all three of those strings when submitting from Tails to the test VM. Since it failed under Tails, but not under a dev environment using Tor, that may indicate we're looking at a client-side locale issue. It seems to be specifically Tor Browser that triggers the failure; I've not been able to reproduce the failure with other browsers, even when routing their traffic over Tor.

@conorsch I was also able to reproduce the error; all three strings break my instance in Tor Browser on Windows 10.

> Currently the issue is that Isis's `python-gnupg` package in Python 3 attempts to read files and strings with the buffer interface. Since we wrap all the files with the `SecureTemporaryFile`, this interface isn't exposed. I haven't found a way to get this class to work with the GPG package.

I don't think this is correct. `python-gnupg` calls the `SecureTemporaryFile.read()` [1] to read from the file or message submitted. The `SecureTemporaryFile.read()` method just calls the `read()` method on the underlying open file object [2]. This should not cause any problems with unicode characters because during writing to this file data is encoded as UTF-8 [3] and the `encryptor.encrypt()`/`decryptor.decrypt()` methods operate on raw bytes, so should have no effect on what's returned.

[1] https://github.com/isislovecruft/python-gnupg/blob/2.0.2/gnupg/_util.py#L238
[2] https://github.com/freedomofpress/securedrop/blob/release/0.3.8/securedrop/secure_tempfile.py#L75
[3] https://github.com/freedomofpress/securedrop/blob/release/0.3.8/securedrop/secure_tempfile.py#L61

@heartsucker Can you try to reproduce w/ the Deutsch locale version of TB: https://www.torproject.org/dist/torbrowser/6.0.3/tor-browser-linux64-6.0.3_de.tar.xz?

Or are you already using that?

@heartsucker Regarding your question:

> In this comment, it's said that the physical security of the servers is outside the threat model. Does this mean it is acceptable to remove [the `SecureTemporaryFile`] class entirely and just use the normal Python file class?

No, we don't want to drop use of that class lightly. I hear you on the physical-access-equals-root-access angle, but using `SecureTemporaryFile` grants us significant protections against forensic discovery, by preventing plaintext submissions from ever being written to disk, even temporarily. 

I can test the German version next time I have time. I'm using English
locale TBB and English locale Tails. Both have the same problem.

As for the error I'm getting, it specifically says that SecureTempFile
doesn't support the buffer interface when I call gpg.encrypt(stf). I can
get a traceback when I have time as well.

Am 11.08.2016 00:11 schrieb "Conor" notifications@github.com:

> @heartsucker https://github.com/heartsucker Regarding your question:
> 
> In this comment, it's said that the physical security of the servers is
> outside the threat model. Does this mean it is acceptable to remove [the
> SecureTemporaryFile] class entirely and just use the normal Python file
> class?
> 
> No, we don't want to drop use of that class lightly. I hear you on the
> physical-access-equals-root-access angle, but using SecureTemporaryFile
> grants us significant protections against forensic discovery, by preventing
> plaintext submissions from ever being written to disk, even temporarily.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> https://github.com/freedomofpress/securedrop/issues/1360#issuecomment-239020680,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AD0DAJrZBOfaWv46MdMP6rk4Usl1WLoSks5qekx7gaJpZM4JOUO4
> .

I haven't had time to install the German TBB, but this is the traceback I'm getting with my attempted Python 3 implementation. Hopefully I can get to that this weekend.

```
tests/test_unit_source.py:246: in test_submit_sanitizes_filename
    ), follow_redirects=True)
/usr/local/lib/python3.4/dist-packages/werkzeug/test.py:788: in post
    return self.open(*args, **kw)
/usr/local/lib/python3.4/dist-packages/flask/testing.py:113: in open
    follow_redirects=follow_redirects)
/usr/local/lib/python3.4/dist-packages/werkzeug/test.py:751: in open
    response = self.run_wsgi_app(environ, buffered=buffered)
/usr/local/lib/python3.4/dist-packages/werkzeug/test.py:668: in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
/usr/local/lib/python3.4/dist-packages/werkzeug/test.py:871: in run_wsgi_app
    app_rv = app(environ, start_response)
/usr/local/lib/python3.4/dist-packages/flask/app.py:2000: in __call__
    return self.wsgi_app(environ, start_response)
/usr/local/lib/python3.4/dist-packages/flask/app.py:1991: in wsgi_app
    response = self.make_response(self.handle_exception(e))
/usr/local/lib/python3.4/dist-packages/flask/app.py:1567: in handle_exception
    reraise(exc_type, exc_value, tb)
/usr/local/lib/python3.4/dist-packages/flask/_compat.py:33: in reraise
    raise value
/usr/local/lib/python3.4/dist-packages/flask/app.py:1988: in wsgi_app
    response = self.full_dispatch_request()
/usr/local/lib/python3.4/dist-packages/flask/app.py:1641: in full_dispatch_request
    rv = self.handle_user_exception(e)
/usr/local/lib/python3.4/dist-packages/flask/app.py:1544: in handle_user_exception
    reraise(exc_type, exc_value, tb)
/usr/local/lib/python3.4/dist-packages/flask/_compat.py:33: in reraise
    raise value
/usr/local/lib/python3.4/dist-packages/flask/app.py:1639: in full_dispatch_request
    rv = self.dispatch_request()
/usr/local/lib/python3.4/dist-packages/flask/app.py:1625: in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
source.py:72: in decorated_function
    return f(*args, **kwargs)
source.py:301: in submit
    fh.stream))
store.py:115: in save_file_submission
    crypto_util.encrypt(stf, config.JOURNALIST_KEY, encrypted_file_path)
crypto_util.py:158: in encrypt
    armor=False)
/usr/local/lib/python3.4/dist-packages/gnupg/gnupg.py:972: in encrypt
    stream = _make_binary_stream(data, self._encoding)

thing = <secure_tempfile.SecureTemporaryFile object at 0x7f383018a400>, encoding = 'utf_8', armor = True

    def _make_binary_stream(thing, encoding=None, armor=True):
        """Encode **thing**, then make it stream/file-like.

        :param thing: The thing to turn into a encoded stream.
        :rtype: ``io.BytesIO`` or ``io.StringIO``.
        :returns: The encoded **thing**, wrapped in an ``io.BytesIO`` (if
            available), otherwise wrapped in a ``io.StringIO``.
        """
        if _py3k:
            if isinstance(thing, str):
                thing = thing.encode(encoding)
        else:
            if type(thing) is not str:
                thing = thing.encode(encoding)

        try:
>           rv = BytesIO(thing)
E           TypeError: 'SecureTemporaryFile' does not support the buffer interface
```

Hey @heartsucker, thanks for reporting this, and sorry for the delay in patching it. We spent some time digging into this issue this afternoon; as a result, we have identified the cause, and have a proposal for a solution.

## Cause

First, to clarify: this issue is server-side, not client-side. I reproduced on a staging environment with either Chrome or Firefox as my client browser. This is not, as some earlier comments suggested, an issue related to the use of Tor Browser.

This issue occurs whenever we pass user-submitted text from the web application to the python-gnupg library. There are two primary situations where this occurs:
- When a source submits a message
  - The message is encrypted to the application GPG public key
- When a journalist submits a reply
  - The reply is encrypted to the corresponding source's public key (as well as the application GPG public key).

The web application consistently uses UTF-8 encoding, and the underlying Werkzeug library used by Flask helpfully converts values from the web application (such as user-submitted form fields in a POST request) to Python 2.x Unicode strings. **The bug is that we are passing these unicode strings directly to python-gnupg, when we should be encoding them first.**

`gnupg` gamely attempts to encode the strings for us (see [`_make_binary_stream`](https://github.com/isislovecruft/python-gnupg/blob/2beac24161b07e40642d93f5d85faeb9740f78db/gnupg/_util.py#L534)). However, for SecureDrop this fails with a `UnicodeEncodeError` when a message containing non-ASCII characters is submitted (which is the underlying cause of the internal server errors reported by OP).

This was a little surprising to us at first, because when we tried to reproduce the error in a Python interpreter there was no issue. python-gnupg [picked up the system's default encoding (UTF-8) as the preferred encoding and used that](https://github.com/isislovecruft/python-gnupg/blob/2beac24161b07e40642d93f5d85faeb9740f78db/gnupg/_meta.py#L194-L197), and everything was sunshine and moonbeams. So why wasn't it working in the production web application?

At this point, a useful clue was that this issue was only reproducible in the production environment, and not in the development environment. This environments use the same code for the web application, but differ significantly otherwise. One of the most significant differences between the environments is that production/staging uses Apache and runs the Flask web application with mod_wsgi, while development just uses the Flask-provided debugging server.

The discrepancy is caused by the fact that when Apache runs an application for WSGI, it starts it in an Apache-controlled environment. This environment can be configured via `/etc/apache2/envvars`, but SecureDrop does not modify it from the default. If you examine the default `envvars` file, you will see that it sets `LANG=C`, which in turn leads Python's `locale.getpreferredencoding` to return the non-Unicode-compatible `ANSI_X3.4-1968` instead of `UTF-8`. python-gnupg then tries to encode a Unicode string with `ANSI_X3.4-1968`, which is naturally doomed to fail, raising a UnicodeEncodeError which causes the observed internal server error by the user (source or journalist).

## Fix

One possible fix is to change the encoding used by mod_wsgi so it's `UTF-8` instead of `C`. Django [recommends](https://docs.djangoproject.com/en/1.10/howto/deployment/wsgi/modwsgi/) doing this  so your application can handle file uploads where the filename contains Unicode characters. We should test file uploads with Unicode characters in the filename while we're addressing Unicode issues 😅 .

In addition, we should encode all of our strings as UTF-8 before passing them to python-gnupg, since it's better for the application to set the encoding explicitly rather than relying on the python-gnupg library's best guess of preferred output encoding. @fowlslegs is already working on a patch for this.

I did some perusing of the logs and found that this issue is a regression that was introduced in 0.3.7 as part of https://github.com/freedomofpress/securedrop/pull/1306/commits/f0e3fc9e1aa42442123c7edc556bf5a79471cab1, which was merged into release/0.3.7 in 37ba11c7b94f1a547aa5e8ed342e3da85a663362.

On the upside, this mistake exposed that our tests for this issue (`test_unit_integration::test_reply_unicode`) were not working correctly, so this provides an opportunity to fix them! Yay better tests!


Resolving patch:
diff --git a/securedrop/crypto_util.py b/securedrop/crypto_util.py
--- a/securedrop/crypto_util.py
+++ b/securedrop/crypto_util.py
@@ -5,6 +5,7 @@
 
 from Crypto.Random import random
 import gnupg
+from gnupg._util import _is_stream, _make_binary_stream
 import scrypt
 
 import config
@@ -154,6 +155,9 @@ def encrypt(plaintext, fingerprints, output=None):
         fingerprints = [fingerprints, ]
     fingerprints = [fpr.replace(' ', '') for fpr in fingerprints]
 
+    if not _is_stream(plaintext):
+        plaintext = _make_binary_stream(plaintext, "utf_8")
+
     out = gpg.encrypt(plaintext,
                       *fingerprints,
                       output=output,
@@ -165,7 +169,7 @@ def encrypt(plaintext, fingerprints, output=None):
         raise CryptoException(out.stderr)
 
 
-def decrypt(secret, plain_text):
+def decrypt(secret, ciphertext):
     """
     >>> key = genkeypair('randomid', 'randomid')
     >>> decrypt('randomid', 'randomid',
@@ -174,8 +178,7 @@ def decrypt(secret, plain_text):
     'Goodbye, cruel world!'
     """
     hashed_codename = hash_codename(secret, salt=SCRYPT_GPG_PEPPER)
-    return gpg.decrypt(plain_text, passphrase=hashed_codename).data
-
+    return gpg.decrypt(ciphertext, passphrase=hashed_codename).data
 
 if __name__ == "__main__":
     import doctest


Test patch:
diff --git a/securedrop/tests/test_unit_integration.py b/securedrop/tests/test_unit_integration.py
--- a/securedrop/tests/test_unit_integration.py
+++ b/securedrop/tests/test_unit_integration.py
@@ -257,8 +257,21 @@ def test_submit_file(self):
     def test_reply_normal(self):
         self.helper_test_reply("This is a test reply.", True)
 
-    def test_reply_unicode(self):
-        self.helper_test_reply("Teşekkürler", True)
+    def test_unicode_reply_with_ansi_env(self):
+        # This makes python-gnupg handle encoding equivalent to if we were
+        # running SD in an environment where os.getenv("LANG") == "C".
+        # Unfortunately, with the way our test suite is set up simply setting
+        # that env var here will not have the desired effect. Instead we
+        # monkey-patch the GPG object that is called crypto_util to imitate the
+        # _encoding attribute it would have had it been initialized in a "C"
+        # environment. See
+        # https://github.com/freedomofpress/securedrop/issues/1360 for context.
+        old_encoding = crypto_util.gpg._encoding
+        crypto_util.gpg._encoding = "ansi_x3.4_1968"
+        try:
+            self.helper_test_reply("ᚠᛇᚻ᛫ᛒᛦᚦ᛫ᚠᚱᚩᚠᚢᚱ᛫ᚠᛁᚱᚪ᛫ᚷᛖᚻᚹᛦᛚᚳᚢᛗ", True)
+        finally:
+            crypto_util.gpg._encoding = old_encoding
 
     def _can_decrypt_with_key(self, msg, key_fpr, passphrase=None):
         """