Repository: freedomofpress/securedrop

Issue:
0.9.0 data migration is excessively slow with large databases
## Description
After using the `qa_loader.py` script to populate the securedrop app db, database migrations from 0.8.0 schema to 0.9.0 take an excessive amount of time (2hrs+) on a 2014 Mac Mini.

## Steps to Reproduce
 
- Install 0.8.0
- ssh to app server and run

```sudo su
   cd /var/www/securedrop
   ./qa_loader.py
```
- Upgrade to 0.9.0~rc1 by switching servers to use apt-test repo using playbook [here](https://gist.github.com/conorsch/e7556624df59b2a0f8b81f7c0c4f9b7d)
- ssh to app server and run:
```
sudo cron-apt -i -s
```

## Expected Behavior
Database migration completes within a short time.


## Actual Behavior
Database migration takes over 2 hours.

## Comments




Discussion:
As background, this migration is slow because we are moving and re-populating every row in the `replies`, `journalists`, `sources` and `submissions` tables as part of the 0.9.0 release. 

Currently the [qa_loader.py](https://github.com/freedomofpress/securedrop/blob/develop/securedrop/qa_loader.py) used to populate the database prior to upgrade has a `multiplier` command line argument: this argument determines how much data to put into the database. When ran with a multiplier of 1, it will put only 10 journalists and 50 sources into the database. However, in its default configuration it is running with a multiplier of 100, which inserts 1000 journalists and 5000 sources. In order to scope how much time we should put into optimizing this migration, we should test how long the migration will take for smaller databases (multiplier=1 and 10 is sufficient).

Unfortunately the table that is (likely) the largest, and thus the slowest, is the `submissions` table, and there is not much we can do to speed that migration up. However, the following potential changes will result in a speed up, the first much more than the second as there are likely a fair number of replies:

1. squash revisions [6db892e17271](https://github.com/freedomofpress/securedrop/blob/develop/securedrop/alembic/versions/6db892e17271_add_reply_uuid.py) and [e0a525cbab83](https://github.com/freedomofpress/securedrop/blob/develop/securedrop/alembic/versions/e0a525cbab83_add_column_to_track_source_deletion_of_.py) together
2. squash revisions [2d0ce3ee5bdc](https://github.com/freedomofpress/securedrop/blob/develop/securedrop/alembic/versions/2d0ce3ee5bdc_added_passphrase_hash_column_to_.py) and [f2833ac34bb6](https://github.com/freedomofpress/securedrop/blob/develop/securedrop/alembic/versions/f2833ac34bb6_add_uuid_column_for_users_table.py) together

(note that this also means modifying the corresponding migration tests)
I just did this upgrade with a multiplier of 10 (i.e. 100 journalists, 500 sources) and it was very fast (not more than a ~minute or so)

Resolving patch:
diff --git a/securedrop/qa_loader.py b/securedrop/qa_loader.py
--- a/securedrop/qa_loader.py
+++ b/securedrop/qa_loader.py
@@ -225,9 +225,9 @@ def arg_parser():
     parser = ArgumentParser(
         path.basename(__file__),
         description='Loads data into the database for testing upgrades')
-    parser.add_argument('-m', '--multiplier', type=positive_int, default=100,
+    parser.add_argument('-m', '--multiplier', type=positive_int, default=25,
                         help=('Factor to multiply the loaded data by '
-                              '(default 100)'))
+                              '(default 25)'))
     return parser
 
 


Test patch: