Repository: freedomofpress/securedrop

Issue:
Testinfra tests are skipped in Travis
# Bug

## Description

All of the config tests in the Travis CI run are marked as "skipped." There are a few config tests that are not relevant on the Travis host, since that host is a combination of an Ansible controller and the development VM, given the localhost provisioning strategy, but we don't want to skip _everything_.

While working on #1736 and running the config tests locally, I discovered that #1770 broke the development VM config tests by upgrading Python dependencies without touching the config tests. Travis CI should have caught that, but didn't. Note that the pip checks may be one of the config tests we need to skip in Travis, due to the insufficient separation of Python library environments. Still, worth a shot.

## Steps to Reproduce

1. Commit a breaking change in a feature branch.
2. Confirm that `./testinfra/test.py development` fails locally as a result of the breaking change.
3. Push feature branch to remote and see the Travis checks fail.

## Expected Behavior

Config tests run in Travis. If not all of them, then most of them.

## Actual Behavior

No config tests run in Travis. All are marked as "SKIPPED."

## Comments

The `testinfra/test.py` wrapper script has gotten ugly. The solution is either to add a "travis" target to that script, or fix the handling of the "development" target to execute against localhost when running on Travis. It's also possible to execute testinfra directly, bypassing the wrapper, since the default behavior of testinfra is to execute against localhost.



Discussion:

Resolving patch:
diff --git a/testinfra/conftest.py b/testinfra/conftest.py
--- a/testinfra/conftest.py
+++ b/testinfra/conftest.py
@@ -10,7 +10,6 @@
 import sys
 import yaml
 import pytest
-from jinja2 import Template
 
 
 target_host = os.environ['SECUREDROP_TESTINFRA_TARGET_HOST']
@@ -26,19 +25,18 @@ def securedrop_import_testinfra_vars(hostname, with_header=False):
     Vars must be stored in `testinfra/vars/<hostname>.yml`.
     """
     filepath = os.path.join(os.path.dirname(__file__), "vars", hostname+".yml")
-    file_load = open(filepath, 'r')
+    with open(filepath, 'r') as f:
+        hostvars = yaml.safe_load(f)
+    # The directory Travis runs builds in varies by PR, so we cannot hardcode
+    # it in the YAML testvars. Read it from env var and concatenate.
     if hostname.lower() == 'travis':
         build_env = os.environ["TRAVIS_BUILD_DIR"]
-        file_load = Template(file_load.read()).render(BUILD_DIR=build_env)
-    hostvars = yaml.load(file_load)
-    
+        hostvars['securedrop_code'] = build_env+"/securedrop"
+
     if with_header:
         hostvars = dict(securedrop_test_vars=hostvars)
     return hostvars
 
 
-
-
-
 def pytest_namespace():
     return securedrop_import_testinfra_vars(target_host, with_header=True)


Test patch:
diff --git a/testinfra/development/test_development_environment.py b/testinfra/development/test_development_environment.py
--- a/testinfra/development/test_development_environment.py
+++ b/testinfra/development/test_development_environment.py
@@ -18,23 +18,23 @@ def test_development_app_dependencies(Package):
 
 
 @pytest.mark.parametrize('pip_package,version', [
-    ('Flask-Testing', '0.6.1'),
-    ('Flask', '0.11.1'),
-    ('Jinja2', '2.8'),
-    ('MarkupSafe', '0.23'),
-    ('Werkzeug', '0.11.11'),
-    ('beautifulsoup4', '4.5.1'),
-    ('click', '6.6'),
-    ('coverage', '4.2'),
+    ('Flask-Testing', '0.6.2'),
+    ('Flask', '0.12.2'),
+    ('Jinja2', '2.9.6'),
+    ('MarkupSafe', '1.0'),
+    ('Werkzeug', '0.12.2'),
+    ('beautifulsoup4', '4.6.0'),
+    ('click', '6.7'),
+    ('coverage', '4.4.1'),
     ('first', '2.0.1'),
     ('funcsigs', '1.0.2'),
     ('itsdangerous', '0.24'),
     ('mock', '2.0.0'),
-    ('pbr', '1.10.0'),
-    ('pip-tools', '1.7.0'),
-    ('py', '1.4.31'),
-    ('pytest-cov', '2.4.0'),
-    ('pytest', '3.0.3'),
+    ('pbr', '3.0.1'),
+    ('pip-tools', '1.9.0'),
+    ('py', '1.4.34'),
+    ('pytest-cov', '2.5.1'),
+    ('pytest', '3.1.1'),
     ('selenium', '2.53.6'),
     ('six', '1.10.0'),
 ])