Repository: freedomofpress/securedrop

Issue:
Allow for removal of "pending" source accounts
## Description

When a source account is created but before a first submission, by default a "pending" flag is set. Pending accounts are not displayed in the journalist interface.

If a user then exits their session and never returns to provide a submission, the account will remain in a pending state indefinitely, and cannot be removed via the Journalist Interface. On long-running high-volume instances this increases the source user count significantly, requiring admin intervention to fix.

One approach to allow for the removal of unused source accounts would be to change the `pending` boolean flag to a datetime value, and purge unused accounts via a cronjob or similar after a set time (say, a month). This has the disadvantage of increasing metadata (source account creation time) about prospective sources, though the field could be set to `null` on submission, meaning that said metadata would not be stored for sources that were actually ever active. This could be mitigated somewhat by giving the datetime a resolution on the order of days or weeks.

Alternatively, the fact that active source sessions without submissions would be nuked when the purge ran could just be accepted. 

Note that this would be moot if/when inverted flow changes land, as accounts would then only be created on first submission. So it might not be worth the effort if it was just a temporary measure.

## To do
1. [x] Add `manage.py` command
2. [ ] Add tests, especially for `--keep-most-recent`'s offset logic
3. [ ] Extract standalone script for testing purposes

## User Research Evidence

long-term observations of instance behavior...

## User Stories



Discussion:
Today @nathandyer and I mapped out the following approach for implementing this lever:

1. Add a `securedrop.management.sources` command that:
    1. Retrieves `Source`s with [`pending=True`](https://github.com/freedomofpress/securedrop/blob/d01e743d2853d42c47459163dfae05e4540344f1/securedrop/models.py#L66), except for the (parameterized) *n* most recent—that is, except those with the *n* highest `id`s.
        * This constraint adds a margin for recent pending sources without introducing the timestamp suggested above.
    3. For each `Source` in (1), calls [`EncryptionManager.delete_source_key_pair()`](https://github.com/freedomofpress/securedrop/blob/d01e743d2853d42c47459163dfae05e4540344f1/securedrop/encryption.py#L150).
    4. Deletes all `Source`s in (1).
        * To my eye, `Source` should have a deletion method or hook that automatically calls `EncryptionManager.delete_source_key_pair()` to avoid orphaned keys.  But that's out of scope of this ticket, and a bulk operation will be more efficient anyway.

2. Once we're happy with the implementation of this command via `manage.py`, we can easily derive a standalone version for testing purposes (even just in a gist, to avoid cluttering up the branch).  @nathandyer and I agreed that we'd much rather work backwards from the intended `manage.py` structure than start with a standalone script and then formalize it in `manage.py`.

Resolving patch:
diff --git a/securedrop/loaddata.py b/securedrop/loaddata.py
--- a/securedrop/loaddata.py
+++ b/securedrop/loaddata.py
@@ -250,7 +250,6 @@ def add_source() -> Tuple[Source, str]:
         source_app_storage=Storage.get_default(),
     )
     source = source_user.get_db_record()
-    source.pending = False
     db.session.commit()
 
     # Generate source key
diff --git a/securedrop/manage.py b/securedrop/manage.py
--- a/securedrop/manage.py
+++ b/securedrop/manage.py
@@ -29,6 +29,7 @@
 from db import db  # noqa: E402
 from management import SecureDropConfig, app_context  # noqa: E402
 from management.run import run  # noqa: E402
+from management.sources import remove_pending_sources  # noqa: E402
 from management.submissions import (  # noqa: E402
     add_check_db_disconnect_parser,
     add_check_fs_disconnect_parser,
@@ -355,6 +356,17 @@ def get_args() -> argparse.ArgumentParser:
     delete_user_subp_a = subps.add_parser("delete_user", help="^")
     delete_user_subp_a.set_defaults(func=delete_user)
 
+    remove_pending_sources_subp = subps.add_parser(
+        "remove-pending-sources", help="Remove pending sources from the server."
+    )
+    remove_pending_sources_subp.add_argument(
+        "--keep-most-recent",
+        default=100,
+        type=int,
+        help="how many of the most recent pending sources to keep",
+    )
+    remove_pending_sources_subp.set_defaults(func=remove_pending_sources)
+
     add_check_db_disconnect_parser(subps)
     add_check_fs_disconnect_parser(subps)
     add_delete_db_disconnect_parser(subps)
diff --git a/securedrop/management/sources.py b/securedrop/management/sources.py
new file mode 100644
--- /dev/null
+++ b/securedrop/management/sources.py
@@ -0,0 +1,58 @@
+import argparse
+from typing import List
+
+from db import db
+from encryption import EncryptionManager, GpgKeyNotFoundError
+from management import app_context
+from models import Source
+
+
+def remove_pending_sources(args: argparse.Namespace) -> int:
+    """
+    Removes pending source accounts, with the option of keeping
+    the n newest source accounts.
+    """
+    n = args.keep_most_recent
+    sources = find_pending_sources(n)
+    print(f"Found {len(sources)} pending sources")
+
+    deleted = []
+    for source in sources:
+        try:
+            EncryptionManager.get_default().delete_source_key_pair(source.filesystem_id)
+        except GpgKeyNotFoundError:
+            pass
+        delete_pending_source(source)
+        deleted.append(source)
+
+    print(f"Deleted {len(sources)} pending sources")
+    return 0
+
+
+def find_pending_sources(keep_most_recent: int) -> List[Source]:
+    """
+    Finds all sources that are marked as pending
+    """
+    with app_context():
+        pending_sources = (
+            Source.query.filter_by(pending=True)
+            .order_by(Source.id.desc())
+            .offset(keep_most_recent)
+            .all()
+        )
+
+    return pending_sources
+
+
+def delete_pending_source(source: Source) -> None:
+    """
+    Delete a pending source from the database
+    """
+    if source.pending:
+        with app_context():
+            try:
+                db.session.delete(source)
+                db.session.commit()
+            except Exception as exc:
+                db.session.rollback()
+                print(f"ERROR: Could not remove pending source: {exc}.")


Test patch:
diff --git a/securedrop/tests/test_remove_pending_sources.py b/securedrop/tests/test_remove_pending_sources.py
new file mode 100644
--- /dev/null
+++ b/securedrop/tests/test_remove_pending_sources.py
@@ -0,0 +1,89 @@
+import argparse
+
+import manage
+import pytest
+from models import Source, db
+from passphrases import PassphraseGenerator
+from source_user import create_source_user
+
+
+@pytest.mark.parametrize("n,m", [(10, 5), (7, 0)])
+def test_remove_pending_sources_none_pending(n, m, source_app, config, app_storage):
+    """remove_pending_sources() is a no-op on active sources."""
+
+    # Override the configuration to point at the per-test database.
+    data_root = config.SECUREDROP_DATA_ROOT
+
+    with source_app.app_context():
+        sources = []
+        for i in range(0, n):
+            source_user = create_source_user(
+                db_session=db.session,
+                source_passphrase=PassphraseGenerator.get_default().generate_passphrase(),
+                source_app_storage=app_storage,
+            )
+            source = source_user.get_db_record()
+            source.pending = False
+            sources.append(source.id)
+        db.session.commit()
+
+        # Make sure we have n sources.
+        assert db.session.query(Source).count() == n
+
+        # If we're keeping the n most-recent sources, then remove_pending_sources()
+        # shouldn't remove any.
+        args = argparse.Namespace(data_root=data_root, verbose=True, keep_most_recent=n)
+        manage.setup_verbosity(args)
+        manage.remove_pending_sources(args)
+        assert db.session.query(Source).count() == n
+
+        # If we're keeping the m most-recent sources, then remove_pending_sources()
+        # still shouldn't do anything, because none are pending.
+        args = argparse.Namespace(data_root=data_root, verbose=True, keep_most_recent=m)
+        manage.setup_verbosity(args)
+        manage.remove_pending_sources(args)
+        assert db.session.query(Source).count() == n
+
+
+@pytest.mark.parametrize("n,m", [(10, 5), (7, 0)])
+def test_remove_pending_sources_all_pending(n, m, source_app, config, app_storage):
+    """remove_pending_sources() removes all but the most-recent m of n pending sources."""
+    # Override the configuration to point at the per-test database.
+    data_root = config.SECUREDROP_DATA_ROOT
+
+    with source_app.app_context():
+        sources = []
+        for i in range(0, n):
+            source_user = create_source_user(
+                db_session=db.session,
+                source_passphrase=PassphraseGenerator.get_default().generate_passphrase(),
+                source_app_storage=app_storage,
+            )
+            source = source_user.get_db_record()
+            sources.append(source.id)
+        db.session.commit()
+
+        # Make sure we have n sources.
+        assert db.session.query(Source).count() == n
+
+        # If we're keeping the n most-recent sources, then remove_pending_sources()
+        # shouldn't remove any.
+        args = argparse.Namespace(data_root=data_root, verbose=True, keep_most_recent=n)
+        manage.setup_verbosity(args)
+        manage.remove_pending_sources(args)
+        assert db.session.query(Source).count() == n
+
+        # If we're keeping the m most-recent sources, then remove_pending_sources()
+        # should remove n - m.
+        args = argparse.Namespace(data_root=data_root, verbose=True, keep_most_recent=m)
+        manage.setup_verbosity(args)
+        manage.remove_pending_sources(args)
+        assert db.session.query(Source).count() == m
+
+        # Specifically, the first n - m sources should be gone...
+        for source in sources[0 : n - m]:
+            assert db.session.query(Source).get(source) is None
+
+        # ...and only the last m should remain.
+        for source in sources[n - m : n]:
+            assert db.session.query(Source).get(source) is not None