Repository: freedomofpress/securedrop

Issue:
Suppress OSSEC alerts asking SecureDrop administrators to upgrade to Xenial
Per the original report in #1530, administrators may under some circumstances receive OSSEC alerts asking them to upgrade their SecureDrop servers to Ubuntu Xenial. Doing so would be *a very bad idea* as Ubuntu Xenial is not in fact supported yet (see #3204). We should therefore investigate whether administrators are still receiving such alerts under any circumstances.

Tasks:
- [x] Verify whether these alerts are still sent
- [x] If appropriate, suppress them


Discussion:
I can confirm that these alerts are still being sent to at least a couple admins. One such admin received this within the past 24 hours:

```
Subject: Cron root@mon test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
/etc/cron.weekly/update-notifier-common:
New release '16.04.4 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
```
I confirm receipt of this alert as well
We've agreed to tackle this as a stretch goal for the 0.8.0 release (i.e. ensuring a robust release is the top priority, but if early QA results are positive, we may attempt a fix). If it doesn't make it, we'll include a note in the release announcement warning admins to ignore this alert.
In prod/staging VMs one won't get this alert by default. Note that I did see it on hardware - every week. To enable it on VMs in `/etc/update-manager/release-upgrades` change `Prompt=never` to `Prompt=lts` for parity with hardware (thanks to @conorsch for pointing me to this file). Now `/etc/cron.weekly/update-notifier-common` should report to the user that `do-release-upgrade` is needed (if you still don't get the alert, you should remove the stamp file stored at `/var/lib/ubuntu-release-upgrader/release-upgrade-available`).

For testing one can just temporarily run the (previously weekly) cronjob more often to get the alert every e.g. 5 minutes:

```
*/5 * * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly)
```

But since the message in question here isn't being parsed by OSSEC, I don't think adding a local rule will suppress it - instead, the simplest resolution is probably just to take the same approach as in our vagrant VMs: in `/etc/update-manager/release-upgrades` replace `Prompt=lts` with `Prompt=never`. We could do this in `postinst` of the SecureDrop OSSEC packages. 

Resolving patch:
diff --git a/testinfra/combine-junit.py b/devops/scripts/combine-junit.py
similarity index 81%
rename from testinfra/combine-junit.py
rename to devops/scripts/combine-junit.py
--- a/testinfra/combine-junit.py
+++ b/devops/scripts/combine-junit.py
@@ -36,7 +36,11 @@ def merge_results(xml_files):
     cases = []
 
     for file_name in xml_files:
-        tree = ET.parse(file_name)
+        # We disable bandit checking to permit B314, which recommends use
+        # of defusedxml to protect against malicious XML files. This code
+        # path only runs in CI, not on developer workstations, and the XML
+        # output is generated by testinfra on staging machines.
+        tree = ET.parse(file_name)  # nosec
         test_suite = tree.getroot()
         failures += int(test_suite.attrib['failures'])
         tests += int(test_suite.attrib['tests'])
diff --git a/molecule/testinfra/staging/conftest.py b/molecule/testinfra/staging/conftest.py
new file mode 100644
--- /dev/null
+++ b/molecule/testinfra/staging/conftest.py
@@ -0,0 +1,91 @@
+"""
+Configuration for TestInfra test suite for SecureDrop.
+Handles importing host-specific test vars, so test functions
+can be reused across multiple hosts, with varied targets.
+
+Vars should be placed in `testinfra/vars/<hostname>.yml`.
+"""
+
+import io
+import os
+import yaml
+
+
+# The config tests target staging by default. It's possible to override
+# for e.g. prod, but the associated vars files are not yet ported.
+target_host = os.environ.get('SECUREDROP_TESTINFRA_TARGET_HOST', 'staging')
+
+
+def securedrop_import_testinfra_vars(hostname, with_header=False):
+    """
+    Import vars from a YAML file to populate tests with host-specific
+    values used in checks. For instance, the SecureDrop docroot will
+    be under /vagrant in development, but /var/www/securedrop in staging.
+
+    Vars must be stored in `testinfra/vars/<hostname>.yml`.
+    """
+    filepath = os.path.join(os.path.dirname(__file__), "vars", hostname+".yml")
+    with io.open(filepath, 'r') as f:
+        hostvars = yaml.safe_load(f)
+
+    if with_header:
+        hostvars = dict(securedrop_test_vars=hostvars)
+
+    if os.environ.get("FPF_CI", False):
+        export_ci_var_overrides()
+
+    return hostvars
+
+
+def export_ci_var_overrides():
+    """
+    In CI environments, the hardcoded local IP addresses aren't valid
+    (since we're testing against remote AWS hosts). Detect the CI env,
+    and look up the proper IPs for use in the testinfra tests.
+    Expose those IPs as environment variables, so the tests can use them.
+    """
+    molecule_info = lookup_molecule_info()
+    app_ip = lookup_aws_private_address(molecule_info, 'app-staging')
+    mon_ip = lookup_aws_private_address(molecule_info, 'mon-staging')
+
+    os.environ['APP_IP'] = app_ip
+    os.environ['MON_IP'] = mon_ip
+
+    # Make SSH calls more resilient, as we're operating against remote hosts,
+    # and running from CI. We've observed flakey connections in CI at times.
+    os.environ['ANSIBLE_SSH_RETRIES'] = '5'
+    ssh_args = [
+        "-o ConnectTimeout=60s",
+        "-o ControlMaster=auto",
+        "-o ControlPersist=180s",
+        "-o StrictHostKeyChecking=no",
+    ]
+    os.environ['ANSIBLE_SSH_ARGS'] = " ".join(ssh_args)
+
+
+def lookup_aws_private_address(molecule_info, hostname):
+    """
+    Inspect Molecule instance config dict (imported from YAML file),
+    and return the attribute for the requested hostname.
+    """
+
+    host_info = list(filter(lambda x: x['instance'] == hostname,
+                            molecule_info))[0]
+    host_ip = host_info['priv_address']
+    return host_ip
+
+
+def lookup_molecule_info():
+    """
+    Molecule automatically writes YAML files documenting dynamic host info
+    such as remote IPs. Read that file and pass back the config dict.
+    """
+    molecule_instance_config_path = os.path.abspath(
+            os.environ['MOLECULE_INSTANCE_CONFIG'])
+    with open(molecule_instance_config_path, 'r') as f:
+        molecule_instance_config = yaml.safe_load(f)
+    return molecule_instance_config
+
+
+def pytest_namespace():
+    return securedrop_import_testinfra_vars(target_host, with_header=True)
diff --git a/testinfra/conftest.py b/testinfra/conftest.py
deleted file mode 100644
--- a/testinfra/conftest.py
+++ /dev/null
@@ -1,36 +0,0 @@
-"""
-Configuration for TestInfra test suite for SecureDrop.
-Handles importing host-specific test vars, so test functions
-can be reused across multiple hosts, with varied targets.
-
-Vars should be placed in `testinfra/vars/<hostname>.yml`.
-"""
-
-import io
-import os
-import yaml
-
-
-target_host = os.environ['SECUREDROP_TESTINFRA_TARGET_HOST']
-assert target_host != ""
-
-
-def securedrop_import_testinfra_vars(hostname, with_header=False):
-    """
-    Import vars from a YAML file to populate tests with host-specific
-    values used in checks. For instance, the SecureDrop docroot will
-    be under /vagrant in development, but /var/www/securedrop in staging.
-
-    Vars must be stored in `testinfra/vars/<hostname>.yml`.
-    """
-    filepath = os.path.join(os.path.dirname(__file__), "vars", hostname+".yml")
-    with io.open(filepath, 'r') as f:
-        hostvars = yaml.safe_load(f)
-
-    if with_header:
-        hostvars = dict(securedrop_test_vars=hostvars)
-    return hostvars
-
-
-def pytest_namespace():
-    return securedrop_import_testinfra_vars(target_host, with_header=True)


Test patch:
diff --git a/molecule/aws/tests/test_tor_interfaces.py b/molecule/aws/tests/test_tor_interfaces.py
--- a/molecule/aws/tests/test_tor_interfaces.py
+++ b/molecule/aws/tests/test_tor_interfaces.py
@@ -21,7 +21,7 @@ def test_www(host, site):
         os.path.dirname(__file__),
         "../../../install_files/ansible-base/{}".format(site['file'])
     )
-    onion_url_raw = io.open(onion_url_filepath, 'ro').read()
+    onion_url_raw = io.open(onion_url_filepath, 'r').read()
     onion_url = re.search("\w+\.onion", onion_url_raw).group()
 
     # Fetch Onion URL via curl to confirm interface is rendered correctly.
diff --git a/testinfra/app-code/test_haveged.py b/molecule/testinfra/staging/app-code/test_haveged.py
similarity index 96%
rename from testinfra/app-code/test_haveged.py
rename to molecule/testinfra/staging/app-code/test_haveged.py
--- a/testinfra/app-code/test_haveged.py
+++ b/molecule/testinfra/staging/app-code/test_haveged.py
@@ -1,3 +1,6 @@
+testinfra_hosts = ["app-staging"]
+
+
 def test_haveged_config(File):
     """
     Ensure haveged's low entrop watermark is sufficiently high.
diff --git a/testinfra/app-code/test_redis_worker.py b/molecule/testinfra/staging/app-code/test_redis_worker.py
similarity index 97%
rename from testinfra/app-code/test_redis_worker.py
rename to molecule/testinfra/staging/app-code/test_redis_worker.py
--- a/testinfra/app-code/test_redis_worker.py
+++ b/molecule/testinfra/staging/app-code/test_redis_worker.py
@@ -2,6 +2,7 @@
 import re
 
 
+testinfra_hosts = ["app-staging"]
 securedrop_test_vars = pytest.securedrop_test_vars
 
 
diff --git a/testinfra/app-code/test_securedrop_app_code.py b/molecule/testinfra/staging/app-code/test_securedrop_app_code.py
similarity index 97%
rename from testinfra/app-code/test_securedrop_app_code.py
rename to molecule/testinfra/staging/app-code/test_securedrop_app_code.py
--- a/testinfra/app-code/test_securedrop_app_code.py
+++ b/molecule/testinfra/staging/app-code/test_securedrop_app_code.py
@@ -1,9 +1,8 @@
-import os
 import pytest
 
 
+testinfra_hosts = ["app-staging"]
 securedrop_test_vars = pytest.securedrop_test_vars
-hostenv = os.environ['SECUREDROP_TESTINFRA_TARGET_HOST']
 
 
 def test_apache_default_docroot_is_absent(File):
diff --git a/testinfra/app-code/test_xvfb.py b/molecule/testinfra/staging/app-code/test_xvfb.py
similarity index 99%
rename from testinfra/app-code/test_xvfb.py
rename to molecule/testinfra/staging/app-code/test_xvfb.py
--- a/testinfra/app-code/test_xvfb.py
+++ b/molecule/testinfra/staging/app-code/test_xvfb.py
@@ -1,3 +1,6 @@
+testinfra_hosts = ["app-staging"]
+
+
 def test_xvfb_is_installed(Package):
     """
     Ensure apt requirements for Xvfb are present.
diff --git a/testinfra/app/apache/test_apache_journalist_interface.py b/molecule/testinfra/staging/app/apache/test_apache_journalist_interface.py
similarity index 99%
rename from testinfra/app/apache/test_apache_journalist_interface.py
rename to molecule/testinfra/staging/app/apache/test_apache_journalist_interface.py
--- a/testinfra/app/apache/test_apache_journalist_interface.py
+++ b/molecule/testinfra/staging/app/apache/test_apache_journalist_interface.py
@@ -2,6 +2,7 @@
 import re
 
 
+testinfra_hosts = ["app-staging"]
 securedrop_test_vars = pytest.securedrop_test_vars
 
 # Setting once so it can be reused in multiple tests.
diff --git a/testinfra/app/apache/test_apache_service.py b/molecule/testinfra/staging/app/apache/test_apache_service.py
similarity index 98%
rename from testinfra/app/apache/test_apache_service.py
rename to molecule/testinfra/staging/app/apache/test_apache_service.py
--- a/testinfra/app/apache/test_apache_service.py
+++ b/molecule/testinfra/staging/app/apache/test_apache_service.py
@@ -1,6 +1,7 @@
 import pytest
 
 
+testinfra_hosts = ["app-staging"]
 securedrop_test_vars = pytest.securedrop_test_vars
 
 
diff --git a/testinfra/app/apache/test_apache_source_interface.py b/molecule/testinfra/staging/app/apache/test_apache_source_interface.py
similarity index 98%
rename from testinfra/app/apache/test_apache_source_interface.py
rename to molecule/testinfra/staging/app/apache/test_apache_source_interface.py
--- a/testinfra/app/apache/test_apache_source_interface.py
+++ b/molecule/testinfra/staging/app/apache/test_apache_source_interface.py
@@ -1,7 +1,7 @@
 import pytest
 import re
 
-
+testinfra_hosts = ["app-staging"]
 securedrop_test_vars = pytest.securedrop_test_vars
 
 
diff --git a/testinfra/app/apache/test_apache_system_config.py b/molecule/testinfra/staging/app/apache/test_apache_system_config.py
similarity index 99%
rename from testinfra/app/apache/test_apache_system_config.py
rename to molecule/testinfra/staging/app/apache/test_apache_system_config.py
--- a/testinfra/app/apache/test_apache_system_config.py
+++ b/molecule/testinfra/staging/app/apache/test_apache_system_config.py
@@ -1,7 +1,7 @@
 import pytest
 import re
 
-
+testinfra_hosts = ["app-staging"]
 securedrop_test_vars = pytest.securedrop_test_vars
 
 
diff --git a/testinfra/app/test_network.py b/molecule/testinfra/staging/app/test_app_network.py
similarity index 98%
rename from testinfra/app/test_network.py
rename to molecule/testinfra/staging/app/test_app_network.py
--- a/testinfra/app/test_network.py
+++ b/molecule/testinfra/staging/app/test_app_network.py
@@ -5,6 +5,7 @@
 from jinja2 import Template
 
 
+testinfra_hosts = ["app-staging"]
 securedrop_test_vars = pytest.securedrop_test_vars
 
 
diff --git a/testinfra/app/test_apparmor.py b/molecule/testinfra/staging/app/test_apparmor.py
similarity index 99%
rename from testinfra/app/test_apparmor.py
rename to molecule/testinfra/staging/app/test_apparmor.py
--- a/testinfra/app/test_apparmor.py
+++ b/molecule/testinfra/staging/app/test_apparmor.py
@@ -1,6 +1,7 @@
 import pytest
 
 
+testinfra_hosts = ["app-staging"]
 sdvars = pytest.securedrop_test_vars
 
 
diff --git a/testinfra/app/test_appenv.py b/molecule/testinfra/staging/app/test_appenv.py
similarity index 98%
rename from testinfra/app/test_appenv.py
rename to molecule/testinfra/staging/app/test_appenv.py
--- a/testinfra/app/test_appenv.py
+++ b/molecule/testinfra/staging/app/test_appenv.py
@@ -1,5 +1,6 @@
 import pytest
 
+testinfra_hosts = ["app-staging"]
 sdvars = pytest.securedrop_test_vars
 
 
diff --git a/testinfra/app/test_ossec.py b/molecule/testinfra/staging/app/test_ossec_agent.py
similarity index 100%
rename from testinfra/app/test_ossec.py
rename to molecule/testinfra/staging/app/test_ossec_agent.py
diff --git a/testinfra/app/test_tor_config.py b/molecule/testinfra/staging/app/test_tor_config.py
similarity index 98%
rename from testinfra/app/test_tor_config.py
rename to molecule/testinfra/staging/app/test_tor_config.py
--- a/testinfra/app/test_tor_config.py
+++ b/molecule/testinfra/staging/app/test_tor_config.py
@@ -1,5 +1,6 @@
 import pytest
 
+testinfra_hosts = ["app-staging"]
 sdvars = pytest.securedrop_test_vars
 
 
diff --git a/testinfra/app/test_tor_hidden_services.py b/molecule/testinfra/staging/app/test_tor_hidden_services.py
similarity index 98%
rename from testinfra/app/test_tor_hidden_services.py
rename to molecule/testinfra/staging/app/test_tor_hidden_services.py
--- a/testinfra/app/test_tor_hidden_services.py
+++ b/molecule/testinfra/staging/app/test_tor_hidden_services.py
@@ -2,6 +2,7 @@
 import re
 
 
+testinfra_hosts = ["app-staging"]
 sdvars = pytest.securedrop_test_vars
 
 
diff --git a/testinfra/common/test_cron_apt.py b/molecule/testinfra/staging/common/test_cron_apt.py
similarity index 100%
rename from testinfra/common/test_cron_apt.py
rename to molecule/testinfra/staging/common/test_cron_apt.py
diff --git a/testinfra/common/test_fpf_apt_repo.py b/molecule/testinfra/staging/common/test_fpf_apt_repo.py
similarity index 100%
rename from testinfra/common/test_fpf_apt_repo.py
rename to molecule/testinfra/staging/common/test_fpf_apt_repo.py
diff --git a/testinfra/common/test_grsecurity.py b/molecule/testinfra/staging/common/test_grsecurity.py
similarity index 100%
rename from testinfra/common/test_grsecurity.py
rename to molecule/testinfra/staging/common/test_grsecurity.py
diff --git a/testinfra/common/test_ip6tables.py b/molecule/testinfra/staging/common/test_ip6tables.py
similarity index 100%
rename from testinfra/common/test_ip6tables.py
rename to molecule/testinfra/staging/common/test_ip6tables.py
diff --git a/testinfra/common/test_platform.py b/molecule/testinfra/staging/common/test_platform.py
similarity index 100%
rename from testinfra/common/test_platform.py
rename to molecule/testinfra/staging/common/test_platform.py
diff --git a/testinfra/common/test_system_hardening.py b/molecule/testinfra/staging/common/test_system_hardening.py
similarity index 83%
rename from testinfra/common/test_system_hardening.py
rename to molecule/testinfra/staging/common/test_system_hardening.py
--- a/testinfra/common/test_system_hardening.py
+++ b/molecule/testinfra/staging/common/test_system_hardening.py
@@ -1,9 +1,6 @@
-import os
 import pytest
 import re
 
-hostenv = os.environ['SECUREDROP_TESTINFRA_TARGET_HOST']
-
 
 @pytest.mark.parametrize('sysctl_opt', [
   ('net.ipv4.conf.all.accept_redirects', 0),
@@ -63,19 +60,21 @@ def test_blacklisted_kernel_modules(Command, File, Sudo, kernel_module):
     assert f.contains("^blacklist {}$".format(kernel_module))
 
 
-@pytest.mark.skipif(hostenv.startswith('mon'),
-                    reason="Monitor Server does not have swap disabled yet.")
 def test_swap_disabled(Command):
     """
     Ensure swap space is disabled. Prohibit writing memory to swapfiles
     to reduce the threat of forensic analysis leaking any sensitive info.
     """
-    c = Command.check_output('swapon --summary')
-    # A leading slash will indicate full path to a swapfile.
-    assert not re.search("^/", c, re.M)
-    # Expect that ONLY the headers will be present in the output.
-    rgx = re.compile("Filename\s*Type\s*Size\s*Used\s*Priority")
-    assert re.search(rgx, c)
+    hostname = Command.check_output('hostname')
+
+    # Mon doesn't have swap disabled yet
+    if not hostname.startswith('mon'):
+        c = Command.check_output('swapon --summary')
+        # A leading slash will indicate full path to a swapfile.
+        assert not re.search("^/", c, re.M)
+        # Expect that ONLY the headers will be present in the output.
+        rgx = re.compile("Filename\s*Type\s*Size\s*Used\s*Priority")
+        assert re.search(rgx, c)
 
 
 def test_twofactor_disabled_on_tty(host):
diff --git a/testinfra/common/test_tor_mirror.py b/molecule/testinfra/staging/common/test_tor_mirror.py
similarity index 100%
rename from testinfra/common/test_tor_mirror.py
rename to molecule/testinfra/staging/common/test_tor_mirror.py
diff --git a/testinfra/common/test_user_config.py b/molecule/testinfra/staging/common/test_user_config.py
similarity index 98%
rename from testinfra/common/test_user_config.py
rename to molecule/testinfra/staging/common/test_user_config.py
--- a/testinfra/common/test_user_config.py
+++ b/molecule/testinfra/staging/common/test_user_config.py
@@ -1,8 +1,6 @@
 import os
 import re
 
-hostenv = os.environ['SECUREDROP_TESTINFRA_TARGET_HOST']
-
 
 def test_sudoers_config(File, Sudo):
     """
diff --git a/testinfra/mon/test_network.py b/molecule/testinfra/staging/mon/test_mon_network.py
similarity index 98%
rename from testinfra/mon/test_network.py
rename to molecule/testinfra/staging/mon/test_mon_network.py
--- a/testinfra/mon/test_network.py
+++ b/molecule/testinfra/staging/mon/test_mon_network.py
@@ -5,6 +5,7 @@
 from jinja2 import Template
 
 
+testinfra_hosts = ["mon-staging"]
 securedrop_test_vars = pytest.securedrop_test_vars
 
 
diff --git a/testinfra/mon/test_ossec_ruleset.py b/molecule/testinfra/staging/mon/test_ossec_ruleset.py
similarity index 96%
rename from testinfra/mon/test_ossec_ruleset.py
rename to molecule/testinfra/staging/mon/test_ossec_ruleset.py
--- a/testinfra/mon/test_ossec_ruleset.py
+++ b/molecule/testinfra/staging/mon/test_ossec_ruleset.py
@@ -1,7 +1,7 @@
 import pytest
 import re
 
-
+testinfra_hosts = ["mon-staging"]
 alert_level_regex = re.compile(r"Level: '(\d+)'")
 rule_id_regex = re.compile(r"Rule id: '(\d+)'")
 sdvars = pytest.securedrop_test_vars
diff --git a/testinfra/mon/test_ossec.py b/molecule/testinfra/staging/mon/test_ossec_server.py
similarity index 98%
rename from testinfra/mon/test_ossec.py
rename to molecule/testinfra/staging/mon/test_ossec_server.py
--- a/testinfra/mon/test_ossec.py
+++ b/molecule/testinfra/staging/mon/test_ossec_server.py
@@ -2,6 +2,7 @@
 import pytest
 
 
+testinfra_hosts = ["mon-staging"]
 securedrop_test_vars = pytest.securedrop_test_vars
 
 
diff --git a/testinfra/mon/test_postfix.py b/molecule/testinfra/staging/mon/test_postfix.py
similarity index 98%
rename from testinfra/mon/test_postfix.py
rename to molecule/testinfra/staging/mon/test_postfix.py
--- a/testinfra/mon/test_postfix.py
+++ b/molecule/testinfra/staging/mon/test_postfix.py
@@ -2,6 +2,7 @@
 import pytest
 
 
+testinfra_hosts = ["mon-staging"]
 securedrop_test_vars = pytest.securedrop_test_vars
 
 
diff --git a/testinfra/ossec/test_journalist_mail.py b/molecule/testinfra/staging/ossec/test_journalist_mail.py
similarity index 91%
rename from testinfra/ossec/test_journalist_mail.py
rename to molecule/testinfra/staging/ossec/test_journalist_mail.py
--- a/testinfra/ossec/test_journalist_mail.py
+++ b/molecule/testinfra/staging/ossec/test_journalist_mail.py
@@ -1,8 +1,14 @@
 import pytest
+import os
 import testinfra
 import time
 
 
+# DRY declaration of why we're skipping all these tests.
+# For details, see https://github.com/freedomofpress/securedrop/issues/3689
+SKIP_REASON = "unimplemented, see GH#3689"
+
+
 class TestBase(object):
 
     @pytest.fixture(autouse=True)
@@ -62,14 +68,18 @@ def service_stopped(self, host, name):
 
 class TestJournalistMail(TestBase):
 
+    @pytest.mark.skip(reason=SKIP_REASON)
     def test_procmail(self, host):
         self.service_started(host, "postfix")
         for (destination, f) in (
                 ('journalist', 'alert-journalist-one.txt'),
                 ('journalist', 'alert-journalist-two.txt'),
                 ('ossec', 'alert-ossec.txt')):
+            # Look up CWD, in case tests move in the future
+            current_dir = os.path.dirname(os.path.abspath(__file__))
             self.ansible(host, "copy",
-                         "dest=/tmp/{f} src=testinfra/ossec/{f}".format(f=f))
+                         "dest=/tmp/{f} src={d}/{f}".format(f=f,
+                                                            d=current_dir))
             assert self.run(host,
                             "/var/ossec/process_submissions_today.sh forget")
             assert self.run(host, "postsuper -d ALL")
@@ -82,6 +92,7 @@ def test_procmail(self, host):
                     destination=destination))
         self.service_stopped(host, "postfix")
 
+    @pytest.mark.skip(reason=SKIP_REASON)
     def test_process_submissions_today(self, host):
         assert self.run(host,
                         "/var/ossec/process_submissions_today.sh "
@@ -90,9 +101,11 @@ def test_process_submissions_today(self, host):
                         "/var/ossec/process_submissions_today.sh "
                         "test_modified_in_the_past_24h")
 
+    @pytest.mark.skip(reason=SKIP_REASON)
     def test_send_encrypted_alert(self, host):
         self.service_started(host, "postfix")
-        src = "install_files/ansible-base/roles/ossec/files/test_admin_key.sec"
+        src = ("../../install_files/ansible-base/roles/ossec/files/"
+               "test_admin_key.sec")
         self.ansible(host, "copy",
                      "dest=/tmp/test_admin_key.sec src={src}".format(src=src))
 
@@ -144,6 +157,7 @@ def trigger(who, payload):
             assert self.run(host, "mv /usr/bin/gpg.save /usr/bin/gpg")
         self.service_stopped(host, "postfix")
 
+    @pytest.mark.skip(reason=SKIP_REASON)
     def test_missing_journalist_alert(self, host):
         #
         # missing journalist mail does nothing
@@ -158,6 +172,7 @@ def test_missing_journalist_alert(self, host):
             """)
 
     # https://ossec-docs.readthedocs.io/en/latest/manual/rules-decoders/testing.html
+    @pytest.mark.skip(reason=SKIP_REASON)
     def test_ossec_rule_journalist(self, host):
         assert self.run(host, """
         set -ex
@@ -166,6 +181,7 @@ def test_ossec_rule_journalist(self, host):
         echo "$l" | /var/ossec/bin/ossec-logtest -U '400600:1:ossec'
         """)
 
+    @pytest.mark.skip(reason=SKIP_REASON)
     def test_journalist_mail_notification(self, host):
         mon = host
         app = testinfra.host.Host.get_host(
diff --git a/testinfra/ansible/test_validate_users.py b/testinfra/ansible/test_validate_users.py
deleted file mode 100644
--- a/testinfra/ansible/test_validate_users.py
+++ /dev/null
@@ -1,19 +0,0 @@
-import pytest
-import os
-
-
-@pytest.mark.skip(reason="Validation not fully implemented yet")
-@pytest.mark.parametrize('username', [
-    'root',
-    'amnesia',
-])
-def test_validate_users(LocalCommand, username):
-    """
-    Check that Ansible halts execution of the playbook if the Admin
-    username is set to any disallowed value.
-    """
-    var_override = "--tags validate --extra-vars ssh_users={}".format(username)
-    os.environ['ANSIBLE_ARGS'] = var_override
-    c = LocalCommand("vagrant provision /staging/")
-
-    assert c.rc != 0
diff --git a/testinfra/test.py b/testinfra/test.py
deleted file mode 100755
--- a/testinfra/test.py
+++ /dev/null
@@ -1,151 +0,0 @@
-#!/usr/bin/env python
-"""
-Wrapper script for running Testinfra against SecureDrop VMs.
-Accepts a single argument: the hostname to run the tests against.
-Script will handle building the list of tests to run, based on hostname.
-"""
-import os
-import subprocess
-import sys
-import tempfile
-
-target_host = sys.argv[1]
-
-# Set env var so that `testinfra/conftest.py` can read in a YAML vars file
-# specific to the host being tested.
-os.environ['SECUREDROP_TESTINFRA_TARGET_HOST'] = target_host
-
-
-def get_target_roles(target_host):
-    """
-    Assemble list of role tests to run. Hard-coded per host.
-    """
-    target_roles = {"app-staging": ['testinfra/app',
-                                    'testinfra/app-code',
-                                    'testinfra/common'],
-                    "staging": ['testinfra/ossec'],
-                    "mon-staging": ['testinfra/mon',
-                                    'testinfra/common'],
-                    "mon-prod":    ['testinfra/mon']}
-
-    try:
-        return target_roles[target_host]
-    except KeyError:
-        print("Unknown host '{}'! Exiting.".format(target_host))
-        sys.exit(1)
-
-
-def run_testinfra(target_host, verbose=True):
-    """
-    Handler for executing testinfra against `target_host`.
-    Queries list of roles via helper def `get_target_roles`.
-    """
-    conn_type = "ssh"
-    target_roles = get_target_roles(target_host)
-    if verbose:
-        # Print informative output prior to test run.
-        print("Running Testinfra suite against '{}'...".format(target_host))
-        print("Target roles:")
-        for role in target_roles:
-            print("    - {}".format(role))
-
-    # Prod hosts host have SSH access over Tor. Let's use the SSH backend
-    # for Testinfra, rather than Ansible. When we write a dynamic inventory
-    # script for Ansible SSH-over-Tor, we can use the Ansible backend
-    # everywhere.
-    if target_host.endswith("-prod"):
-        os.environ['SECUREDROP_SSH_OVER_TOR'] = '1'
-        # Dump SSH config to tempfile so it can be passed as arg to testinfra.
-        ssh_config_output = subprocess.check_output(["vagrant", "ssh-config",
-                                                     target_host])
-        # Create temporary file to store ssh-config. Not deleting it
-        # automatically because there's no sensitive info (HidServAuth is
-        # required to connect), and we'll need it outside of the
-        # context-manager block that writes to it.
-        ssh_config_tmpfile = tempfile.NamedTemporaryFile(delete=False)
-        with ssh_config_tmpfile.file as f:
-            f.write(ssh_config_output)
-        ssh_config_path = ssh_config_tmpfile.name
-        testinfra_command_template = """
-testinfra \
-    -vv \
-    -n auto \
-    --connection ssh \
-    --ssh-config \
-    {ssh_config_path}\
-    --hosts {target_host} \
-    {target_roles}
-""".lstrip().rstrip()
-
-    elif os.environ.get("FPF_CI", 'false') == 'true':
-        ssh_config_path = os.environ["CI_SSH_CONFIG"]
-        testinfra_command_template = """
-testinfra \
-    -vv \
-    -n 8 \
-    --connection {connection_type} \
-    --ssh-config \
-    {ssh_config_path}\
-    --junit-xml=./{target_host}-results.xml\
-    --junit-prefix={target_host}\
-    --hosts {target_host} \
-    {target_roles}
-""".lstrip().rstrip()
-
-    elif target_host == 'staging':
-        if "CI_SSH_CONFIG" in os.environ:
-            ssh_config_path = '--ssh-config ' + os.environ["CI_SSH_CONFIG"]
-            inventory = ""
-            junit = """
-            --junit-xml=./{target_host}-results.xml \
-            --junit-prefix={target_host} \
-            """
-        else:
-            ssh_config_path = ""
-            inventory = """
-            --ansible-inventory \
-            .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory \
-            """
-            junit = ""
-        testinfra_command_template = """
-testinfra \
-    -vv \
-    --connection ansible \
-    {testinfra_args} \
-    {ssh_config_path} \
-    {inventory} \
-    {junit} \
-    --hosts app-staging,mon-staging \
-    {target_roles}
-""".format(ssh_config_path=ssh_config_path,
-           inventory=inventory,
-           target_roles=" ".join(target_roles),
-           junit=junit,
-           testinfra_args=os.environ.get('TESTINFRA_ARGS', ''))
-
-    else:
-        ssh_config_path = ""
-        testinfra_command_template = """
-testinfra \
-    -vv \
-    -n auto \
-    --connection ansible \
-    --ansible-inventory \
-    .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory \
-    --hosts {target_host} \
-    {target_roles}
-""".lstrip().rstrip()
-
-    testinfra_command = testinfra_command_template.format(
-            target_host=target_host,
-            ssh_config_path=ssh_config_path,
-            connection_type=conn_type,
-            target_roles=" ".join(target_roles),
-            ).split()
-
-    # Execute config tests.
-    subprocess.check_call(testinfra_command)
-
-
-if __name__ == "__main__":
-    run_testinfra(target_host)