Repository: freedomofpress/securedrop

Issue:
Failures on a single host should halt Ansible execution
# Bug

## Description

Right now, when running `./securedrop-admin install`, if an error is encountered on a single host, e.g. the Monitor Server, that host will drop off the play, but the playbook will continue to execute against the Application Server. That's very confusing, and can make reporting problems difficult: lots of scrollback is necessary even to determine that a host did indeed fail.

By way of example, the OSSEC registration bug documented in #2478 will lead to the only-one-host-failed scenario described above.

## Steps to Reproduce

Quite difficult to reproduce reliably. As mentioned above, #2748 is a fine candidate for observing the poor failure mode. 

## Expected Behavior
When encountering an error, Ansible stops executing, leaving the error text prominently displayed. The Admin can then use that error message to debug the underlying issue, or report the problem to the SecureDrop developers for additional support.

## Actual Behavior
If only one host fails, Ansible continues to execute against the remaining host, obscuring the fact that the error occurred at all, and complicating the procedure for sharing critical feedback required for support, such as verbatim error messages.

## Comments

We have a recourse here. There's an Ansible option called [max_fail_percentage](http://docs.ansible.com/ansible/latest/playbooks_delegation.html#maximum-failure-percentage): 

> By default, Ansible will continue executing actions as long as there are hosts in the group that have not yet failed.

Setting `max_fail_percentage` to `0` should enforce fail-fast behavior from Ansible with SecureDrop. While it would be ideal to set the option in `ansible.cfg`, it appears it's only supported at the play level—which means all playbooks should have it set _on each play_. 



Discussion:
Here's an example of this happening during QA of #2748:

```
PLAY [Add FPF apt repository and install base packages.] *******************************************************

TASK [Gathering Facts] *****************************************************************************************
ok: [app]
fatal: [mon]: FAILED! => {"failed": true, "msg": "Timeout (62s) waiting for privilege escalation prompt: "}
META: ran handlers
META: ran handlers
META: ran handlers
META: ran handlers
META: ran handlers

PLAY [Configure OSSEC.] ****************************************************************************************

TASK [Gathering Facts] *****************************************************************************************
ok: [app]
META: ran handlers

TASK [ossec : Install securedrop-ossec-agent package.] *********************************************************
task path: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/ossec/tasks/configure_client.yml:2
ok: [app] => {"cache_update_time": 1516732901, "cache_updated": false, "changed": false}

TASK [ossec : Install OSSEC manager package.] ******************************************************************
task path: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/ossec/tasks/configure_server.yml:2

TASK [ossec : Copy the OSSEC GPG public key for sending encrypted alerts.] *************************************
task path: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/ossec/tasks/configure_server.yml:12

TASK [ossec : Add the OSSEC GPG public key to the OSSEC manager keyring.] **************************************
task path: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/ossec/tasks/configure_server.yml:19

TASK [ossec : Copy script for sending GPG-encrypted OSSEC alerts.] *********************************************
task path: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/ossec/tasks/configure_server.yml:32

TASK [ossec : Create OSSEC manager SSL key.] *******************************************************************
task path: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/ossec/tasks/configure_server.yml:43

TASK [ossec : Create OSSEC manager SSL certificate.] ***********************************************************
task path: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/ossec/tasks/configure_server.yml:50

TASK [ossec : Check whether Application Server is registered as OSSEC agent.] **********************************
task path: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/ossec/tasks/register.yml:2

TASK [ossec : Set host fact for OSSEC registration state.] *****************************************************
task path: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/ossec/tasks/register.yml:14
fatal: [app]: FAILED! => {"failed": true, "msg": "the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: 'dict object' has no attribute 'ossec_list_agents_result'\n\nThe error appears to have been in '/home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/ossec/tasks/register.yml': line 14, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n# copy the result to a more conveniently named fact on both hosts.\n- name: Set host fact for OSSEC registration state.\n  ^ here\n"}

PLAY [Configure mailing utilities.] ****************************************************************************

PLAY [Configure SecureDrop Application Server.] ****************************************************************

PLAY [Lock down firewall configuration for Application and Monitor Servers.] ***********************************

PLAY [Reboot Application and Monitor Servers.] *****************************************************************
 [WARNING]: Could not create retry file '/home/amnesia/Persistent/securedrop/install_files/ansible-base
/securedrop-prod.retry'.         [Errno 1] Operation not permitted:
u'/home/amnesia/Persistent/securedrop/install_files/ansible-base/securedrop-prod.retry'


PLAY RECAP *****************************************************************************************************
app                        : ok=3    changed=0    unreachable=0    failed=1   
localhost                  : ok=2    changed=0    unreachable=0    failed=0   
mon                        : ok=0    changed=0    unreachable=0    failed=1 
```

Note that I used a `--tags ossec` invocation, which drastically abbreviates the error output—otherwise, tracing back the error would have involved lots of scrollback.

Resolving patch:
diff --git a/install_files/ansible-base/roles/upgrade/files/0.3pre_upgrade.py b/install_files/ansible-base/roles/upgrade/files/0.3pre_upgrade.py
deleted file mode 100755
--- a/install_files/ansible-base/roles/upgrade/files/0.3pre_upgrade.py
+++ /dev/null
@@ -1,197 +0,0 @@
-#!/usr/bin/python2.7
-
-from datetime import datetime
-import os
-import shutil
-import sqlite3
-import subprocess
-import sys
-import tarfile
-import traceback
-
-
-def backup_app():
-    tar_fn = 'backup-app-{}.tar.bz2'.format(datetime.now().strftime("%Y-%m-%d--%H-%M-%S"))
-    with tarfile.open(tar_fn, 'w:bz2') as t:
-        t.add('/var/lib/securedrop/')
-        t.add('/var/lib/tor/services/')
-        t.add('/var/www/securedrop/config.py')
-        try:
-            t.add('/var/www/securedrop/static/i/logo.png')
-        except OSError:
-            print "[!] Expected but non-essential file ('logo.png') not found. Continuing..."
-    print "**  Backed up system to {} before migrating.".format(tar_fn)
-
-
-def backup_mon():
-    # The only thing we have to back up for the monitor server is the SSH ATHS cert.
-    # All other required values are available in prod-specific.yml from the installation.
-    tar_fn = 'backup-mon-{}.tar.bz2'.format(datetime.now().strftime("%Y-%m-%d--%H-%M-%S"))
-    with tarfile.open(tar_fn, 'w:bz2') as t:
-        t.add('/var/lib/tor/services/')
-    print "**  Backed up system to {} before migrating.".format(tar_fn)
-
-
-def secure_unlink(path):
-    subprocess.check_call(['srm', '-r', path])
-
-
-def cleanup_deleted_sources(store_dir, c):
-    """
-    In 0.3pre and 0.3, there were two bugs that could potentially lead
-    to the source directory failing to be deleted when a source was
-    deleted from the Journalist Interface. We clean up these leftover
-    directories as part of the migration.
-
-    These sources can be identified because they have a source_dir in
-    the store_dir, but no corresponding Source entry in the database.
-
-    See https://github.com/freedomofpress/securedrop/pull/944 for context.
-    """
-    for source_dir in os.listdir(store_dir):
-        try:
-            source = c.execute("SELECT * FROM sources WHERE filesystem_id=?",
-                (source_dir,)).fetchone()
-            if not source:
-                print "Deleting source with no db entry ('{}')...".format(source_dir)
-                secure_unlink(os.path.join(store_dir, source_dir))
-        except Exception as e:
-            print "\n!!  Error occurred cleaning up deleted sources for source {}".format(source_dir)
-            print "Source had {} submissions".format(len(os.listdir(os.path.join(store_dir, source_dir))))
-            print traceback.format_exc()
-
-
-def get_db_connection():
-    db_path = "/var/lib/securedrop/db.sqlite"
-    assert os.path.isfile(db_path)
-    conn = sqlite3.connect(db_path)
-    return conn, conn.cursor()
-
-
-def migrate_app_db():
-    store_dir = "/var/lib/securedrop/store"
-    conn, c = get_db_connection()
-
-    # Before modifying the database, clean up any source directories that were
-    # left on the filesystem after the sources were deleted.
-    cleanup_deleted_sources(store_dir, c)
-
-    # To get CREATE TABLE from SQLAlchemy:
-    # >>> import db
-    # >>> from sqlalchemy.schema import CreateTable
-    # >>> print CreateTable(db.Journalist.__table__).compile(db.engine)
-    # Or, add `echo=True` to the engine constructor.
-    # CREATE TABLE replies
-    c.execute("""
-CREATE TABLE replies (
-	id INTEGER NOT NULL,
-	journalist_id INTEGER,
-	source_id INTEGER,
-	filename VARCHAR(255) NOT NULL,
-	size INTEGER NOT NULL,
-	PRIMARY KEY (id),
-	FOREIGN KEY(journalist_id) REFERENCES journalists (id),
-	FOREIGN KEY(source_id) REFERENCES sources (id)
-)""")
-
-    # Fill in replies from the replies in STORE_DIR at the time of the migration
-    #
-    # Caveats:
-    #
-    # 1. Before we added the `replies` table, we did not keep track of which
-    # journalist wrote the reply. There is no way for us to reverse-engineer
-    # that information, so the migration will default to saying they were all
-    # created by the first journalist (arbitrarily). Since we do not surface
-    # this in the UI yet anyway, it should not be a big deal.
-    #
-    # 2. We do not try to get the order of the (autoincrementing primary key)
-    # reply_id to match the order in which the replies were created (which could
-    # be inferred from the file timestamps, since we only normalize submission
-    # timestamps and not reply timestamps) since this order is not used anywhere
-    # in the code.
-
-    # Copy from db.py to compute filesystem-safe journalist filenames
-    def journalist_filename(s):
-        valid_chars = 'abcdefghijklmnopqrstuvwxyz1234567890-_'
-        return ''.join([c for c in s.lower().replace(' ', '_') if c in valid_chars])
-
-    reply_id = 1
-    for source_dir in os.listdir(store_dir):
-        try:
-            source_id, journalist_designation = c.execute(
-                "SELECT id, journalist_designation FROM sources WHERE filesystem_id=?",
-                (source_dir,)).fetchone()
-        except sqlite3.Error as e:
-            print "!!\tError occurred migrating replies for source {}".format(source_dir)
-            print traceback.format_exc()
-            continue
-
-        for filename in os.listdir(os.path.join(store_dir, source_dir)):
-            if "-reply.gpg" not in filename:
-                continue
-
-            # Rename the reply file from 0.3pre convention to 0.3 convention
-            interaction_count = filename.split('-')[0]
-            new_filename = "{}-{}-reply.gpg".format(interaction_count,
-                journalist_filename(journalist_designation))
-            os.rename(os.path.join(store_dir, source_dir, filename),
-                      os.path.join(store_dir, source_dir, new_filename))
-
-            # need id, journalist_id, source_id, filename, size
-            journalist_id = 1 # *shrug*
-            full_path = os.path.join(store_dir, source_dir, new_filename)
-            size = os.stat(full_path).st_size
-            c.execute("INSERT INTO replies VALUES (?,?,?,?,?)",
-                      (reply_id, journalist_id, source_id, new_filename, size))
-            reply_id += 1 # autoincrement for next reply
-
-    # CREATE TABLE journalist_login_attempts
-    c.execute("""
-CREATE TABLE journalist_login_attempt (
-	id INTEGER NOT NULL,
-	timestamp DATETIME,
-	journalist_id INTEGER,
-	PRIMARY KEY (id),
-	FOREIGN KEY(journalist_id) REFERENCES journalists (id)
-)""")
-
-    # ALTER TABLE journalists, add last_token column
-    c.execute("""ALTER TABLE journalists ADD COLUMN last_token VARCHAR(6)""")
-
-    # Save changes and close connection
-    conn.commit()
-    conn.close()
-
-
-def app_db_migrated():
-    """To make the upgrade role idempotent, we need to skip migrating the
-    database if it has already been modified. The best way to do this
-    is to check whether the last sql command in `migrate_app_db`
-    (ALTER TABLE to add the last_token column to the journalists
-    table) succeeded. If so, we can assume the database app migration
-    succeeded and can safely skip doing it again.
-
-    """
-    conn, c = get_db_connection()
-    journalist_tables = c.execute('PRAGMA table_info(journalists)').fetchall()
-    table_names = set([table[1] for table in journalist_tables])
-    return 'last_token' in table_names
-
-
-def main():
-    if len(sys.argv) <= 1:
-        print "Usage: 0.3pre_upgrade.py app|mon"
-        sys.exit(1)
-
-    server_role = sys.argv[1]
-    assert server_role in ("app", "mon")
-
-    if server_role == "app":
-        backup_app()
-        if not app_db_migrated():
-            migrate_app_db()
-    else:
-        backup_mon()
-
-if __name__ == "__main__":
-    main()


Test patch:
diff --git a/molecule/ansible-config/tests/test_max_fail_percentage.py b/molecule/ansible-config/tests/test_max_fail_percentage.py
new file mode 100644
--- /dev/null
+++ b/molecule/ansible-config/tests/test_max_fail_percentage.py
@@ -0,0 +1,80 @@
+import os
+
+import pytest
+import yaml
+
+
+# Lots of parent directories to dig out of the Molecule test dir.
+# Could also inspect the Molecule env vars and go from there.
+REPO_ROOT = os.path.abspath(os.path.join(__file__,
+                                         os.path.pardir,
+                                         os.path.pardir,
+                                         os.path.pardir,
+                                         os.path.pardir,
+                                         ))
+ANSIBLE_BASE = os.path.join(REPO_ROOT, 'install_files', 'ansible-base')
+
+
+def find_ansible_playbooks():
+    """
+    Test helper to generate list of filepaths for SecureDrop
+    Ansible playbooks. All files will be validated to contain the
+    max_fail option.
+    """
+    playbooks = []
+    # Not using os.walk since all SecureDrop playbooks are in top-level
+    # of the "ansible-base" directory, and *many* YAML files that are
+    # not playbooks reside in subdirectories.
+    for f in os.listdir(ANSIBLE_BASE):
+        # Assume all YAML files in directory are playbooks.
+        if f.endswith(".yml"):
+            # Ignore deprecated production vars file.
+            if f != "prod-specific.yml":
+                playbooks.append(os.path.join(ANSIBLE_BASE, f))
+    # Sanity checking to make sure list of playbooks is not empty.
+    assert len(playbooks) > 0
+    return playbooks
+
+
+@pytest.mark.parametrize('playbook', find_ansible_playbooks())
+def test_max_fail_percentage(host, playbook):
+    """
+    All SecureDrop playbooks should set `max_fail_percentage` to "0"
+    on each and every play. Doing so ensures that an error on a single
+    host constitutes a play failure.
+
+    In conjunction with the `any_errors_fatal` option, tested separately,
+    this will achieve a "fail fast" behavior from Ansible.
+
+    There's no ansible.cfg option to set for max_fail_percentage, which would
+    allow for a single DRY update that would apply automatically to all
+    invocations of `ansible-playbook`. Therefore this test, which will
+    search for the line present in all playbooks.
+
+    Technically it's only necessary that plays targeting multiple hosts use
+    the parameter, but we'll play it safe and require it everywhere,
+    to avoid mistakes down the road.
+    """
+    with open(playbook, 'r') as f:
+        playbook_yaml = yaml.safe_load(f)
+        # Descend into playbook list structure to validate play attributes.
+        for play in playbook_yaml:
+            assert 'max_fail_percentage' in play
+            assert play['max_fail_percentage'] == 0
+
+
+@pytest.mark.parametrize('playbook', find_ansible_playbooks())
+def test_max_fail_percentage(host, playbook):
+    """
+    All SecureDrop playbooks should set `any_errors_fatal` to "yes"
+    on each and every play. In conjunction with `max_fail_percentage` set
+    to "0", doing so ensures that any errors will cause an immediate failure
+    on the playbook.
+    """
+    with open(playbook, 'r') as f:
+        playbook_yaml = yaml.safe_load(f)
+        # Descend into playbook list structure to validate play attributes.
+        for play in playbook_yaml:
+            assert 'any_errors_fatal' in play
+            # Ansible coerces booleans, so bare assert is sufficient
+            assert play['any_errors_fatal']