Repository: freedomofpress/securedrop

Issue:
Source cannot log in with codename if reply files are missing
## Description

If a source has received a reply, and the reply file is removed from (or never written to) `/var/lib/securedrop/store`, then the source cannot successfully log in and reach the `/lookup` page. Instead a 500 error is generated, and the only option available is to log in via  another codename or log out. On the JI, the source page is listed as normal, but the reply file cannot be downloaded.

## Steps to Reproduce

- create a new source on the SI, submit a message, and note their codename
- reply to the new source in the JI
- verify that the reply is visible on the SI for the new source
- on the app server, identify and delete the corresponding `reply.gpg`
- log out from the SI and attempt to log in using the new source's codename

## Expected Behavior
Source can log in and submit documents or read other replies

## Actual Behavior
Source cannot log in, and a stack trace similar to the following is recorded in the source error logs (if enabled):
```
[Wed Jul 22 22:10:08.442888 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108] [2020-07-22 22:10:08,442] ERROR in app: Exception on /lookup [GET]
[Wed Jul 22 22:10:08.442937 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108] Traceback (most recent call last):
[Wed Jul 22 22:10:08.442946 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]   File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
[Wed Jul 22 22:10:08.442953 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]     response = self.full_dispatch_request()
[Wed Jul 22 22:10:08.442959 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]   File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request
[Wed Jul 22 22:10:08.442966 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]     rv = self.handle_user_exception(e)
[Wed Jul 22 22:10:08.442972 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]   File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception
[Wed Jul 22 22:10:08.442979 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]     reraise(exc_type, exc_value, tb)
[Wed Jul 22 22:10:08.442985 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]   File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
[Wed Jul 22 22:10:08.442992 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]     raise value
[Wed Jul 22 22:10:08.442998 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]   File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
[Wed Jul 22 22:10:08.443005 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]     rv = self.dispatch_request()
[Wed Jul 22 22:10:08.443011 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]   File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
[Wed Jul 22 22:10:08.443017 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]     return self.view_functions[rule.endpoint](**req.view_args)
[Wed Jul 22 22:10:08.443023 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]   File "/var/www/securedrop/source_app/decorators.py", line 12, in decorated_function
[Wed Jul 22 22:10:08.443029 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]     return f(*args, **kwargs)
[Wed Jul 22 22:10:08.443060 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]   File "/var/www/securedrop/source_app/main.py", line 115, in lookup
[Wed Jul 22 22:10:08.443067 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108]     with io.open(reply_path, "rb") as f:
[Wed Jul 22 22:10:08.443076 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108] FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/securedrop/store/YJSI6RQBP5JTHYYMZ4L4MUTQWAI7MNTK5ZUU4K2OXSD34PALIFT7DK6LTAYT43VLGAEOPWBBNV6JWDTHQPFITD6UFPLNYN25RHMJJOY=/2-damp_burner-reply.gpg'
[Wed Jul 22 22:10:08.443111 2020] [wsgi:error] [pid 11154:tid 115647289255680] [remote 127.0.0.1:40108] 
```

## Comments

This is a pernicious bug, as it prevents a source from logging in with an existing codename but doesn't present any obvious problems on the JI side (unless the journalist tries to download one of their old replies). It's offset by the recent changes to add nightly checks for orphaned db entries and submission/reply files. However, if an admin isn't paying attention to OSSEC alerts, it may seem to a journalist that a source just stopped responding, while the source may want to keep the conversation going but is blocked from doing so. 

Fix is probably simple enough, if the try/catch loop in source_app/main.py caught ` FileNotFoundError`s that might be enough for a basic effort. A more complex fix could also flag the fact that the file was missing in the JI more obviously, possibly prompting journalists to get their admin to run the cleanup scripts.


Discussion:
Thanks for the detailed report!

> on the app server, identify and delete the corresponding reply.gpg
> log out from the SI and attempt to log in using the new source's codename

I can reproduce this also by simply reloading the Source Interface after deleting the reply file on disk (tested in the Docker dev env). However, I cannot reproduce by deleting the reply through the JI.

If this only arises in cases of DB entries with references to files that no longer exist, it seems like a bit of an edge case that's worth fixing, but not urgent enough to try to rush it into 1.5.0 or a point release. Does that seem reasonable? 
It definitely doesn't need to be a point release, as I'm not aware of any affected instances - the problem is tho, that we wouldn't really have any way of knowing. I'm going to give a little time now to a potential fix and if it's straightforward we can maybe talk about adding it in to 1.5.0 at daily standup.
Per discussion at sprint planning, targeting 1.6.0 milestone for now, if we're in good shape on all other release commitments and logistics (e.g., docs protocol), we may still be able to pull it into 1.5.0, but that's very much a stretch goal at this point.
> [...] I'm not aware of any affected instances - the problem is tho, that we wouldn't really have any way of knowing.

Could this [forum thread](https://forum.securedrop.org/t/codename-not-accepted/1344) be related? I get that there isn't any way of knowing, but I thought it could be useful to put it on your radar @zenmonkeykstop?

Resolving patch:
diff --git a/securedrop/source_app/main.py b/securedrop/source_app/main.py
--- a/securedrop/source_app/main.py
+++ b/securedrop/source_app/main.py
@@ -129,6 +129,9 @@ def lookup():
             except UnicodeDecodeError:
                 current_app.logger.error("Could not decode reply %s" %
                                          reply.filename)
+            except FileNotFoundError:
+                current_app.logger.error("Reply file missing: %s" %
+                                         reply.filename)
             else:
                 reply.date = datetime.utcfromtimestamp(
                     os.stat(reply_path).st_mtime)


Test patch:
diff --git a/securedrop/tests/test_source.py b/securedrop/tests/test_source.py
--- a/securedrop/tests/test_source.py
+++ b/securedrop/tests/test_source.py
@@ -269,7 +269,8 @@ def test_user_must_log_in_for_protected_views(source_app):
 
 def test_login_with_whitespace(source_app):
     """
-    Test that codenames with leading or trailing whitespace still work"""
+    Test that codenames with leading or trailing whitespace still work
+    """
 
     def login_test(app, codename):
         resp = app.get(url_for('main.login'))
@@ -299,6 +300,32 @@ def login_test(app, codename):
             login_test(app, codename_)
 
 
+def test_login_with_missing_reply_files(source_app):
+    """
+    Test that source can log in when replies are present in database but missing
+    from storage.
+    """
+    source, codename = utils.db_helper.init_source()
+    journalist, _ = utils.db_helper.init_journalist()
+    replies = utils.db_helper.reply(journalist, source, 1)
+    assert len(replies) > 0
+    with source_app.test_client() as app:
+        with patch("io.open") as ioMock:
+            ioMock.side_effect = FileNotFoundError
+            resp = app.get(url_for('main.login'))
+            assert resp.status_code == 200
+            text = resp.data.decode('utf-8')
+            assert "Enter Codename" in text
+
+            resp = app.post(url_for('main.login'),
+                            data=dict(codename=codename),
+                            follow_redirects=True)
+            assert resp.status_code == 200
+            text = resp.data.decode('utf-8')
+            assert "Submit Files" in text
+            assert session['logged_in'] is True
+
+
 def _dummy_submission(app):
     """
     Helper to make a submission (content unimportant), mostly useful in