Repository: freedomofpress/securedrop

Issue:
manage: clean-tmp races against get_bulk_archive
# Bug

## Description

**./manage clean-tmp** races against get_bulk_archive

## Steps to Reproduce

* In the app-staging VM (the development server of werkzeug does not support X-Sendfile and won't reproduce the race)
* modify the journalist.py download function like so:
<pre>
def download(zip_basename, submissions):
    """Send client contents of zipfile *zip_basename*-<timestamp>.zip
    containing *submissions*. The zipfile, being a
    :class:`tempfile.NamedTemporaryFile`, is stored on disk only
    temporarily.

    :param str zip_basename: The basename of the zipfile download.

    :param list submissions: A list of :class:`db.Submission`s to
                             include in the zipfile.
    """
    zf = store.get_bulk_archive(submissions,
                                zip_directory=zip_basename)
    attachment_filename = "{}--{}.zip".format(
        zip_basename, datetime.utcnow().strftime("%Y-%m-%d--%H-%M-%S"))

    # Mark the submissions that have been downloaded as such
    for submission in submissions:
        submission.downloaded = True
    db_session.commit()

    r = send_file(zf.name, mimetype="application/zip",
                     attachment_filename=attachment_filename,
                     as_attachment=True)
    del zf
    import time
    time.sleep(30)
    return r
</pre>
* Download a collection from the journalist interface
* After a file shows up in /var/lib/securedrop/tmp 
* ./manage clean-tmp
* Apache will fail to find the file path created by get_bulk_archive (the path is found in the X-Sendfile header)
* /var/log/apache2/journalist-error.log will show an error looking like this:
<pre>
[Fri Aug 25 11:03:20.596664 2017] [:info] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020] mod_wsgi (pid=3259, process='journalist', application='app-staging:8080|'): Loading WSGI script '/var/www/journalist.wsgi'.
[Fri Aug 25 11:03:51.168580 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020] [2017-08-25 11:03:51,154] ERROR in app: Exception on /col/process [POST]
[Fri Aug 25 11:03:51.168672 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020] Traceback (most recent call last):
[Fri Aug 25 11:03:51.168693 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
[Fri Aug 25 11:03:51.168711 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]     response = self.full_dispatch_request()
[Fri Aug 25 11:03:51.168729 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request
[Fri Aug 25 11:03:51.168746 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]     rv = self.handle_user_exception(e)
[Fri Aug 25 11:03:51.168762 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in handle_user_exception
[Fri Aug 25 11:03:51.168778 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]     reraise(exc_type, exc_value, tb)
[Fri Aug 25 11:03:51.168793 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
[Fri Aug 25 11:03:51.168810 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]     rv = self.dispatch_request()
[Fri Aug 25 11:03:51.168854 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
[Fri Aug 25 11:03:51.168871 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]     return self.view_functions[rule.endpoint](**req.view_args)
[Fri Aug 25 11:03:51.168886 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]   File "/var/www/securedrop/journalist.py", line 89, in wrapper
[Fri Aug 25 11:03:51.168901 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]     return func(*args, **kwargs)
[Fri Aug 25 11:03:51.168915 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]   File "/var/www/securedrop/journalist.py", line 548, in col_process
[Fri Aug 25 11:03:51.168929 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]     return method(cols_selected)
[Fri Aug 25 11:03:51.168943 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]   File "/var/www/securedrop/journalist.py", line 572, in col_download_all
[Fri Aug 25 11:03:51.168958 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]     return download("all", submissions)
[Fri Aug 25 11:03:51.168972 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]   File "/var/www/securedrop/journalist.py", line 792, in download
[Fri Aug 25 11:03:51.168988 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]     as_attachment=True)
[Fri Aug 25 11:03:51.169002 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]   File "/usr/local/lib/python2.7/dist-packages/flask/helpers.py", line 544, in send_file
[Fri Aug 25 11:03:51.169016 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]     fsize = os.path.getsize(filename)
[Fri Aug 25 11:03:51.169031 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]   File "/usr/lib/python2.7/genericpath.py", line 49, in getsize
[Fri Aug 25 11:03:51.169049 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020]     return os.stat(filename).st_size
[Fri Aug 25 11:03:51.169069 2017] [:error] [pid 3259:tid 3087374333696] [remote 127.0.0.1:45020] OSError: [Errno 2] No such file or directory: '/var/lib/securedrop/tmp/tmp_securedrop_bulk_dl_QLYC7z'
[Fri Aug 25 11:03:58.116025 2017] [:info] [pid 3258:tid 3087374976768] mod_wsgi (pid=3258): Create interpreter 'app-staging:8080|'.
[Fri Aug 25 11:03:58.117313 2017] [:info] [pid 3258:tid 3087374976768] mod_wsgi (pid=3258): Adding '/var/www/securedrop' to path.
[Fri Aug 25 11:03:58.117892 2017] [:info] [pid 3258:tid 3087374976768] [remote 127.0.0.1:45020] mod_wsgi (pid=3258, process='journalist', application='app-staging:8080|'): Loading WSGI script '/var/www/journalist.wsgi'.
</pre>

## Expected Behavior

./manage clean-tmp never races against get_bulk_archive

## Actual Behavior

There is a window of opportunity for **./manage clean-tmp** to remove a file and fail a download

## Comments

This is unlikely but possible. The current safeguard only addresses part of the race. It should remove files older than X days instead.



Discussion:

Resolving patch:
diff --git a/securedrop/manage.py b/securedrop/manage.py
--- a/securedrop/manage.py
+++ b/securedrop/manage.py
@@ -8,10 +8,10 @@
 import signal
 import subprocess
 import sys
+import time
 import traceback
 import version
 
-import psutil
 import qrcode
 from sqlalchemy.orm.exc import NoResultFound
 
@@ -231,39 +231,22 @@ def delete_user(args):  # pragma: no cover
 
 
 def clean_tmp(args):  # pragma: no cover
-    """Cleanup the SecureDrop temp directory. This is intended to be run
-    as an automated cron job. We skip files that are currently in use to
-    avoid deleting files that are currently being downloaded."""
-    # Inspired by http://stackoverflow.com/a/11115521/1093000
-    def file_in_use(fname):
-        for proc in psutil.process_iter():
-            try:
-                open_files = proc.open_files()
-                in_use = False or any([open_file.path == fname
-                                       for open_file in open_files])
-                # Early return for perf
-                if in_use:
-                    break
-            except psutil.NoSuchProcess:
-                # This catches a race condition where a process ends before we
-                # can examine its files. Ignore this - if the process ended, it
-                # can't be using fname, so this won't cause an error.
-                pass
-
-        return in_use
+    """Cleanup the SecureDrop temp directory. """
+    if not os.path.exists(args.directory):
+        log.debug('{} does not exist, do nothing'.format(args.directory))
+        return 0
 
     def listdir_fullpath(d):
-        # Thanks to http://stackoverflow.com/a/120948/1093000
         return [os.path.join(d, f) for f in os.listdir(d)]
 
-    try:
-        os.stat(config.TEMP_DIR)
-    except OSError:
-        pass
-    else:
-        for path in listdir_fullpath(config.TEMP_DIR):
-            if not file_in_use(path):
-                os.remove(path)
+    too_old = args.days * 24 * 60 * 60
+    for path in listdir_fullpath(args.directory):
+        if time.time() - os.stat(path).st_mtime > too_old:
+            os.remove(path)
+            log.debug('{} removed'.format(path))
+        else:
+            log.debug('{} modified less than {} days ago'.format(
+                path, args.days))
 
     return 0
 
@@ -347,11 +330,8 @@ def get_args():
                                   "SecureDrop application's state.")
     reset_subp.set_defaults(func=reset)
     # Cleanup the SD temp dir
-    clean_tmp_subp = subps.add_parser('clean-tmp', help='Cleanup the '
-                                      'SecureDrop temp directory.')
-    clean_tmp_subp.set_defaults(func=clean_tmp)
-    clean_tmp_subp_a = subps.add_parser('clean_tmp', help='^')
-    clean_tmp_subp_a.set_defaults(func=clean_tmp)
+    set_clean_tmp_parser(subps, 'clean-tmp')
+    set_clean_tmp_parser(subps, 'clean_tmp')
 
     set_translate_parser(subps)
 
@@ -396,6 +376,24 @@ def set_translate_parser(subps):
     parser.set_defaults(func=translate)
 
 
+def set_clean_tmp_parser(subps, name):
+    parser = subps.add_parser(name, help='Cleanup the '
+                              'SecureDrop temp directory.')
+    default_days = 7
+    parser.add_argument(
+        '--days',
+        default=default_days,
+        type=int,
+        help=('remove files not modified in a given number of DAYS '
+              '(default {} days)'.format(default_days)))
+    parser.add_argument(
+        '--directory',
+        default=config.TEMP_DIR,
+        help=('remove old files from DIRECTORY '
+              '(default {})'.format(config.TEMP_DIR)))
+    parser.set_defaults(func=clean_tmp)
+
+
 def setup_verbosity(args):
     if args.verbose:
         logging.getLogger(__name__).setLevel(logging.DEBUG)


Test patch:
diff --git a/securedrop/tests/test_manage.py b/securedrop/tests/test_manage.py
--- a/securedrop/tests/test_manage.py
+++ b/securedrop/tests/test_manage.py
@@ -197,6 +197,35 @@ def test_translate_compile_arg(self):
         assert 'code hello i18n' in mo
         assert 'template hello i18n' not in mo
 
+    def test_clean_tmp_do_nothing(self, caplog):
+        args = argparse.Namespace(days=0,
+                                  directory=' UNLIKELY ',
+                                  verbose=logging.DEBUG)
+        manage.setup_verbosity(args)
+        manage.clean_tmp(args)
+        assert 'does not exist, do nothing' in caplog.text()
+
+    def test_clean_tmp_too_young(self, caplog):
+        args = argparse.Namespace(days=24*60*60,
+                                  directory=config.TEMP_DIR,
+                                  verbose=logging.DEBUG)
+        open(os.path.join(config.TEMP_DIR, 'FILE'), 'a').close()
+        manage.setup_verbosity(args)
+        manage.clean_tmp(args)
+        assert 'modified less than' in caplog.text()
+
+    def test_clean_tmp_removed(self, caplog):
+        args = argparse.Namespace(days=0,
+                                  directory=config.TEMP_DIR,
+                                  verbose=logging.DEBUG)
+        fname = os.path.join(config.TEMP_DIR, 'FILE')
+        with open(fname, 'a'):
+            old = time.time() - 24*60*60
+            os.utime(fname, (old, old))
+        manage.setup_verbosity(args)
+        manage.clean_tmp(args)
+        assert 'FILE removed' in caplog.text()
+
 
 class TestSh(object):