Repository: freedomofpress/securedrop

Issue:
unhandled FileNotFoundError when files have been manually deleted from disk
## Description

In 1.0.0, we introduced logic for an admin to check if there are files that have been deleted on disk but not had their corresponding db rows deleted. Right now in the JI, an unhandled exception will occur in the JI if a user tries to download all submissions and there are any missing files on disk. 

## Steps to Reproduce

1. Submit a file as a source
2. Delete that file on the server at the filesystem level
3. Now download the file from the journalist interface (e.g. Select All -> Download)

## Expected Behavior

All files on the server are downloaded. 

## Actual Behavior

500 occurs and `FileNotFoundError` is unhandled

## Comments

I don't think this the highest priority since this is an edge case but it might be nice to add exception handling to flash a message instructing the user to get their admin to run the cleanup tool (it's easy for an admin to miss a notification in an OSSEC alert). 


Discussion:
There was a case with this on the SI as well (fix in for 1.6.0) - would be cool to look for other potential cases as well.
Looks like this issue is not completely addressed: While #5549 addresses the issue for deletions, it does not address this issue for downloads. Steps to reproduce are as follows:

1. Go to source interface, submit two messages
2. SSH into app server, go to `/var/lib/securedrop/store` and delete one of the messages
3. Go to Journalist Interface and try to download the deleted message
4. Observe error 500

We may need to wrap the `send_file` call with try/except in https://github.com/freedomofpress/securedrop/blob/bee63b1b44a0d27f9fdf7aad14f9ef32207502c3/securedrop/journalist_app/utils.py#L211
This issue is flagged with "Help Wanted". I'd be interested in working on it if no one else is.

From a UX perspective, when the journalist deletes a message, they don't have to know that the corresponding file was missing. However if the file is missing when they try to download it, they will expect some sort of feedback to tell them the message cannot be downloaded.

I see different possible scenarios:

#### Missing files cases:
A. One of multiple message files is missing
B. All of the message files from a source are missing (individual files or the whole source folder)

#### Journalist actions:
1. From `/`, clicks "Download" unread against the source or selects the source only and clicks Download or Download Unread (zip)
1. From `/` selects more than one source and clicks Download or Download Unread (zip)
1. From `/col/<filesystem_id>`, clicks on the message with missing file (gpg)
1. From `/col/<filesystem_id>`, selects the message with missing file and clicks Download Selected (zip)
1. From `/col/<filesystem_id>`, selects multiple/all messages and clicks Download Selected (zip)

In cases A1, A2, B2 and A5, the zip file can be produced with the remaining available message files.
In cases A3 and B3 the gpg file cannot be downloaded.
In cases A4, B1, B4 and B5, the zip file cannot be produced/downloaded.

Opinions on the way to handle these scenarios from a UX prespective would be helpful.

Thx.

Hey @DrGFreeman thanks for the analysis and offer of help! Would be very happy to have your involvement on this.

For the deletion case  addressed in 1.6.0, it originally summarized any non-fatal errors in an additional flash message, after deletions were complete. That flash message was dropped to avoid reopening translations late in the release cycle, but will probably be added again for next release. A similar mechanism would probably work OK for submissions IMO but it would be cool to get @ninavizz' or others' opinions. (In A3 and B3, there would just be the error flash message.)

One other thing to bear in mind is that these errors would occur in the context of an inconsistent file store on the server, so there should be:
- logging on the JI to reflect this, which will trigger OSSEC alerts for the admins
- error messaging that directs journalists to get their admins involved (in case they miss the OSSEC alerts)
Thanks from my end as well @DrGFreeman for the awesome breakdown :)

It's important to note that this issue is not expected to arise on production systems, and we have no report of journalists or admins seeing these 500 errors. If it does arise, it indicates a serious issue with the file store, and possible tampering. It really does require admin involvement.

We discussed this a bit at today's tech meeting and agreed that we can start with the simple possible improvement for now upon any cases where users currently see a 500 error, and that would be a catch-all error message. It would be great to get Nina's input on the wording here. @ninavizz, let's discuss a bit, I can get you up to speed. For now, DrG, you can just put in a placeholder error.

What is the current behavior in collections where only some files in a collection are available? Do some of those cases you listed currently work, or do they all throw a 500?
> What is the current behavior in collections where only some files in a collection are available? Do some of those cases you listed currently work, or do they all throw a 500?

All cases above throw a 500.

I will start working on a solution with a generic flash message for now + logging of the errors.
That sounds great, thank you @DrGFreeman! :) 

Resolving patch:
diff --git a/securedrop/journalist_app/col.py b/securedrop/journalist_app/col.py
--- a/securedrop/journalist_app/col.py
+++ b/securedrop/journalist_app/col.py
@@ -1,5 +1,7 @@
 # -*- coding: utf-8 -*-
 
+from pathlib import Path
+
 from flask import (
     Blueprint,
     abort,
@@ -93,6 +95,18 @@ def download_single_file(filesystem_id: str, fn: str) -> werkzeug.Response:
         if '..' in fn or fn.startswith('/'):
             abort(404)
 
+        file = current_app.storage.path(filesystem_id, fn)
+        if not Path(file).is_file():
+            flash(
+                gettext(
+                    "Your download failed because a file could not be found. An admin can find "
+                    + "more information in the system and monitoring logs."
+                ),
+                "error"
+            )
+            current_app.logger.error("File {} not found".format(file))
+            return redirect(url_for("col.col", filesystem_id=filesystem_id))
+
         # mark as seen by the current user
         try:
             journalist = g.get("user")
diff --git a/securedrop/journalist_app/utils.py b/securedrop/journalist_app/utils.py
--- a/securedrop/journalist_app/utils.py
+++ b/securedrop/journalist_app/utils.py
@@ -3,6 +3,7 @@
 import datetime
 import os
 from typing import Optional, List, Union, Any
+from urllib.parse import urlparse
 
 import flask
 import werkzeug
@@ -190,7 +191,7 @@ def mark_seen(targets: List[Union[Submission, Reply]], user: Journalist) -> None
             raise
 
 
-def download(zip_basename: str, submissions: List[Union[Submission, Reply]]) -> flask.Response:
+def download(zip_basename: str, submissions: List[Union[Submission, Reply]]) -> werkzeug.Response:
     """Send client contents of ZIP-file *zip_basename*-<timestamp>.zip
     containing *submissions*. The ZIP-file, being a
     :class:`tempfile.NamedTemporaryFile`, is stored on disk only
@@ -201,7 +202,19 @@ def download(zip_basename: str, submissions: List[Union[Submission, Reply]]) ->
     :param list submissions: A list of :class:`models.Submission`s to
                              include in the ZIP-file.
     """
-    zf = current_app.storage.get_bulk_archive(submissions, zip_directory=zip_basename)
+    try:
+        zf = current_app.storage.get_bulk_archive(submissions, zip_directory=zip_basename)
+    except FileNotFoundError:
+        flash(
+            gettext(
+                "Your download failed because a file could not be found. An admin can find "
+                + "more information in the system and monitoring logs."
+            ),
+            "error"
+        )
+        referrer = urlparse(str(request.referrer)).path
+        return redirect(referrer)
+
     attachment_filename = "{}--{}.zip".format(
         zip_basename, datetime.datetime.utcnow().strftime("%Y-%m-%d--%H-%M-%S")
     )
diff --git a/securedrop/store.py b/securedrop/store.py
--- a/securedrop/store.py
+++ b/securedrop/store.py
@@ -201,6 +201,8 @@ def get_bulk_archive(self,
                        for i in selected_submissions])
         # The below nested for-loops are there to create a more usable
         # folder structure per #383
+        missing_files = False
+
         with zipfile.ZipFile(zip_file, 'w') as zip:
             for source in sources:
                 fname = ""
@@ -208,18 +210,27 @@ def get_bulk_archive(self,
                                if s.source.journalist_designation == source]
                 for submission in submissions:
                     filename = self.path(submission.source.filesystem_id, submission.filename)
-                    document_number = submission.filename.split('-')[0]
-                    if zip_directory == submission.source.journalist_filename:
-                        fname = zip_directory
+
+                    if os.path.exists(filename):
+                        document_number = submission.filename.split('-')[0]
+                        if zip_directory == submission.source.journalist_filename:
+                            fname = zip_directory
+                        else:
+                            fname = os.path.join(zip_directory, source)
+                        zip.write(filename, arcname=os.path.join(
+                            fname,
+                            "%s_%s" % (document_number,
+                                       submission.source.last_updated.date()),
+                            os.path.basename(filename)
+                        ))
                     else:
-                        fname = os.path.join(zip_directory, source)
-                    zip.write(filename, arcname=os.path.join(
-                        fname,
-                        "%s_%s" % (document_number,
-                                   submission.source.last_updated.date()),
-                        os.path.basename(filename)
-                    ))
-        return zip_file
+                        missing_files = True
+                        current_app.logger.error("File {} not found".format(filename))
+
+        if missing_files:
+            raise FileNotFoundError
+        else:
+            return zip_file
 
     def move_to_shredder(self, path: str) -> None:
         """


Test patch:
diff --git a/securedrop/tests/functional/journalist_navigation_steps.py b/securedrop/tests/functional/journalist_navigation_steps.py
--- a/securedrop/tests/functional/journalist_navigation_steps.py
+++ b/securedrop/tests/functional/journalist_navigation_steps.py
@@ -753,6 +753,17 @@ def cookie_string_from_selenium_cookies(cookies):
 
         assert self.secret_message == submission
 
+    def _journalist_downloads_message_missing_file(self):
+        self._journalist_selects_the_first_source()
+
+        self.wait_for(lambda: self.driver.find_element_by_css_selector("ul#submissions"))
+
+        submissions = self.driver.find_elements_by_css_selector("#submissions a")
+        assert 1 == len(submissions)
+
+        file_link = submissions[0]
+        file_link.click()
+
     def _journalist_composes_reply(self):
         reply_text = (
             "Thanks for the documents. Can you submit more " "information about the main program?"
@@ -1097,3 +1108,42 @@ def _journalist_uses_js_buttons_to_download_unread(self):
         for checkbox in checkboxes:
             classes = checkbox.get_attribute("class")
             assert "unread-cb" in classes
+
+    def _journalist_sees_missing_file_error_message(self):
+        notification = self.driver.find_element_by_css_selector(".error")
+
+        if self.accept_languages is None:
+            expected_text = (
+                "Your download failed because a file could not be found. An admin can find "
+                + "more information in the system and monitoring logs."
+            )
+            assert expected_text == notification.text
+
+    def _journalist_is_on_collection_page(self):
+        return self.wait_for(
+            lambda: self.driver.find_element_by_css_selector("div.journalist-view-single")
+        )
+
+    def _journalist_clicks_source_unread(self):
+        self.driver.find_element_by_css_selector("span.unread a").click()
+
+    def _journalist_selects_first_source_then_download_all(self):
+        checkboxes = self.driver.find_elements_by_name("cols_selected")
+        assert len(checkboxes) == 1
+        checkboxes[0].click()
+
+        self.driver.find_element_by_xpath("//button[@value='download-all']").click()
+
+    def _journalist_selects_first_source_then_download_unread(self):
+        checkboxes = self.driver.find_elements_by_name("cols_selected")
+        assert len(checkboxes) == 1
+        checkboxes[0].click()
+
+        self.driver.find_element_by_xpath("//button[@value='download-unread']").click()
+
+    def _journalist_selects_message_then_download_selected(self):
+        checkboxes = self.driver.find_elements_by_name("doc_names_selected")
+        assert len(checkboxes) == 1
+        checkboxes[0].click()
+
+        self.driver.find_element_by_xpath("//button[@value='download']").click()
diff --git a/securedrop/tests/functional/test_journalist.py b/securedrop/tests/functional/test_journalist.py
--- a/securedrop/tests/functional/test_journalist.py
+++ b/securedrop/tests/functional/test_journalist.py
@@ -15,6 +15,9 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
+from pathlib import Path
+
+import pytest
 
 from . import functional_test as ft
 from . import journalist_navigation_steps
@@ -76,3 +79,76 @@ def test_journalist_interface_ui_with_modal(self):
         self._journalist_selects_the_first_source()
         self._journalist_uses_js_buttons_to_download_unread()
         self._journalist_delete_all_confirmation()
+
+
+class TestJournalistMissingFile(
+    ft.FunctionalTest,
+    source_navigation_steps.SourceNavigationStepsMixin,
+    journalist_navigation_steps.JournalistNavigationStepsMixin
+):
+    """Test error handling when a message file has been deleted from disk but remains in the
+    database. Ref #4787."""
+
+    @pytest.fixture(scope="function")
+    def missing_msg_file(self):
+        """Fixture to setup the message with missing file used in the following tests."""
+        # Submit a message
+        self._source_visits_source_homepage()
+        self._source_chooses_to_submit_documents()
+        self._source_continues_to_submit_page()
+        self._source_submits_a_message()
+        self._source_logs_out()
+
+        # Remove the message file from the store
+        storage_path = Path(self.journalist_app.storage.storage_path)
+        msg_files = [p for p in storage_path.rglob("*") if p.is_file()]
+        assert len(msg_files) == 1
+        msg_files[0].unlink()
+
+        self.switch_to_firefox_driver()
+
+        yield
+
+        self.switch_to_torbrowser_driver()
+
+    def test_download_source_unread(self, missing_msg_file):
+        """Test behavior when the journalist clicks on the source's "n unread" button from the
+        journalist home page."""
+        self._journalist_logs_in()
+        self._journalist_clicks_source_unread()
+        self._journalist_sees_missing_file_error_message()
+        self._is_on_journalist_homepage()
+
+    def test_select_source_and_download_all(self, missing_msg_file):
+        """Test behavior when the journalist selects the source then clicks the "Download" button
+        from the journalist home page."""
+        self._journalist_logs_in()
+        self._journalist_selects_first_source_then_download_all()
+        self._journalist_sees_missing_file_error_message()
+        self._is_on_journalist_homepage()
+
+    def test_select_source_and_download_unread(self, missing_msg_file):
+        """Test behavior when the journalist selects the source then clicks the "Download Unread"
+        button from the journalist home page."""
+        self._journalist_logs_in()
+        self._journalist_selects_first_source_then_download_unread()
+        self._journalist_sees_missing_file_error_message()
+        self._is_on_journalist_homepage()
+
+    def test_download_message(self, missing_msg_file):
+        """Test behavior when the journalist clicks on the individual message from the source page.
+        """
+        self._journalist_logs_in()
+        self._journalist_checks_messages()
+        self._journalist_downloads_message_missing_file()
+        self._journalist_sees_missing_file_error_message()
+        self._journalist_is_on_collection_page()
+
+    def test_select_message_and_download_selected(self, missing_msg_file):
+        """Test behavior when the journalist selects the individual message from the source page
+        then clicks "Download Selected"."""
+        self._journalist_logs_in()
+        self._journalist_selects_the_first_source()
+        self._journalist_selects_message_then_download_selected()
+        self._journalist_sees_missing_file_error_message()
+        self._journalist_is_on_collection_page()
diff --git a/securedrop/tests/test_journalist.py b/securedrop/tests/test_journalist.py
--- a/securedrop/tests/test_journalist.py
+++ b/securedrop/tests/test_journalist.py
@@ -2,7 +2,9 @@
 import base64
 import binascii
 import io
+from mock import call
 import os
+from pathlib import Path
 import pytest
 import random
 import zipfile
@@ -2286,6 +2288,97 @@ def test_download_selected_submissions_previously_downloaded(
             )
 
 
+@pytest.fixture(scope="function")
+def selected_missing_files(journalist_app, test_source):
+    """Fixture for the download tests with missing files in storage."""
+    source = Source.query.get(test_source["id"])
+    submissions = utils.db_helper.submit(source, 2)
+    selected = sorted([s.filename for s in submissions])
+
+    storage_path = Path(journalist_app.storage.storage_path)
+    msg_files = sorted([p for p in storage_path.rglob("*") if p.is_file()])
+    assert len(msg_files) == 2
+    for file in msg_files:
+        file.unlink()
+
+    yield selected
+
+
+def test_download_selected_submissions_missing_files(
+    journalist_app, test_journo, test_source, mocker, selected_missing_files
+):
+    """Tests download of selected submissions with missing files in storage."""
+    mocked_error_logger = mocker.patch('journalist.app.logger.error')
+    journo = Journalist.query.get(test_journo["id"])
+
+    with journalist_app.test_client() as app:
+        _login_user(
+            app,
+            journo.username,
+            test_journo["password"],
+            test_journo["otp_secret"]
+        )
+        resp = app.post(
+            url_for("main.bulk"),
+            data=dict(
+                action="download",
+                filesystem_id=test_source["filesystem_id"],
+                doc_names_selected=selected_missing_files,
+            )
+        )
+
+    assert resp.status_code == 302
+
+    expected_calls = []
+    for file in selected_missing_files:
+        missing_file = (
+            Path(journalist_app.storage.storage_path)
+            .joinpath(test_source["filesystem_id"])
+            .joinpath(file)
+            .as_posix()
+        )
+        expected_calls.append(call("File {} not found".format(missing_file)))
+
+    mocked_error_logger.assert_has_calls(expected_calls)
+
+
+def test_download_single_submission_missing_file(
+    journalist_app, test_journo, test_source, mocker, selected_missing_files
+):
+    """Tests download of single submissions with missing files in storage."""
+    mocked_error_logger = mocker.patch('journalist.app.logger.error')
+    journo = Journalist.query.get(test_journo["id"])
+    missing_file = selected_missing_files[0]
+
+    with journalist_app.test_client() as app:
+        _login_user(
+            app,
+            journo.username,
+            test_journo["password"],
+            test_journo["otp_secret"]
+        )
+        resp = app.get(
+            url_for(
+                "col.download_single_file",
+                filesystem_id=test_source["filesystem_id"],
+                fn=missing_file,
+            )
+        )
+
+    assert resp.status_code == 302
+
+    missing_file = (
+        Path(journalist_app.storage.storage_path)
+        .joinpath(test_source["filesystem_id"])
+        .joinpath(missing_file)
+        .as_posix()
+    )
+
+    mocked_error_logger.assert_called_once_with(
+        "File {} not found".format(missing_file)
+    )
+
+
 def test_download_unread_all_sources(journalist_app, test_journo):
     """
     Test that downloading all unread creates a zip that contains all unread submissions from the