Repository: freedomofpress/securedrop

Issue:
Switch CircleCI staging environment to GCE + libvirt VMs
## Description

Our staging environment is currently directly provisioned to Xen-based VMs running on AWS, due to the lack of support for nested virtualization. That means we can't use the `grsecurity` kernel, and the environment differs from our locally run staging VMs in other ways. Moreover, we can't add support for new tests such as upgrade testing (#1689). 

We've previously considered workarounds (see #2892), but we now have established that we can use GCE's nested-virtualization capabilities (see #3452). The next logical step will be to switch the staging environment spun up by CircleCI to GCE running our libvirt-based staging VMs.

## User Stories

As a developer, I want CI results to reflect the preferred staging environment as much as possible, so that bugs and errors are caught early in development.


Discussion:
In the current sprint (which ends BOD PDT 10/17), @msheiny will add the GCE configuration as an infrastructure prerequisite for this, which is tracked in our infra repository. For the next sprint, we will likely aim to transition the existing tests to staging VMs run in GCE via nested virtualization (still managed via CircleCI), giving us the immediate benefit of grsecurity kernels. For a future sprint, we'll then move on to automated upgrade testing (#1689).
For this current sprint (10/17-10/31), the top level description describes the scope what we're targeting; @msheiny will drive with @conorsch likely being the primary reviewer.

Resolving patch:
diff --git a/molecule/testinfra/staging/conftest.py b/molecule/testinfra/staging/conftest.py
--- a/molecule/testinfra/staging/conftest.py
+++ b/molecule/testinfra/staging/conftest.py
@@ -31,50 +31,9 @@ def securedrop_import_testinfra_vars(hostname, with_header=False):
     if with_header:
         hostvars = dict(securedrop_test_vars=hostvars)
 
-    if os.environ.get("FPF_CI", False):
-        export_ci_var_overrides()
-
     return hostvars
 
 
-def export_ci_var_overrides():
-    """
-    In CI environments, the hardcoded local IP addresses aren't valid
-    (since we're testing against remote AWS hosts). Detect the CI env,
-    and look up the proper IPs for use in the testinfra tests.
-    Expose those IPs as environment variables, so the tests can use them.
-    """
-    molecule_info = lookup_molecule_info()
-    app_ip = lookup_aws_private_address(molecule_info, 'app-staging')
-    mon_ip = lookup_aws_private_address(molecule_info, 'mon-staging')
-
-    os.environ['APP_IP'] = app_ip
-    os.environ['MON_IP'] = mon_ip
-
-    # Make SSH calls more resilient, as we're operating against remote hosts,
-    # and running from CI. We've observed flakey connections in CI at times.
-    os.environ['ANSIBLE_SSH_RETRIES'] = '5'
-    ssh_args = [
-        "-o ConnectTimeout=60s",
-        "-o ControlMaster=auto",
-        "-o ControlPersist=180s",
-        "-o StrictHostKeyChecking=no",
-    ]
-    os.environ['ANSIBLE_SSH_ARGS'] = " ".join(ssh_args)
-
-
-def lookup_aws_private_address(molecule_info, hostname):
-    """
-    Inspect Molecule instance config dict (imported from YAML file),
-    and return the attribute for the requested hostname.
-    """
-
-    host_info = list(filter(lambda x: x['instance'] == hostname,
-                            molecule_info))[0]
-    host_ip = host_info['priv_address']
-    return host_ip
-
-
 def lookup_molecule_info():
     """
     Molecule automatically writes YAML files documenting dynamic host info


Test patch:
diff --git a/docs/development/testing_continuous_integration.rst b/docs/development/testing_continuous_integration.rst
--- a/docs/development/testing_continuous_integration.rst
+++ b/docs/development/testing_continuous_integration.rst
@@ -14,9 +14,9 @@ the scripts in ``devops/``. You may want to consult the
 to interpret the configuration file. Review the ``workflows`` section of the
 configuration file to understand which jobs are run by CircleCI.
 
-The files under ``devops/`` are used to create a minimized staging environment
-on AWS EC2. The CircleCI host is used as the Ansible controller to provision the
-machines and run the :ref:`tests <config_tests>` against them.
+The files under ``devops/`` are used to create a libvirt-compatible environment on GCE.
+The GCE host is used as the Ansible controller, mimicking a developer's laptop,
+to provision the machines and run the :ref:`tests <config_tests>` against them.
 
 .. note:: We skip unnecessary jobs, such as the staging run, for pull requests that only
   affect the documentation; to do so, we check whether the branch name begins with
@@ -33,17 +33,6 @@ machines and run the :ref:`tests <config_tests>` against them.
   branch. Once your branch is in a PR, you can rebuild, push an additional
   commit, or manually rebase your branch to update the CI results.
 
-Limitations of the CI Staging Environment
------------------------------------------
-Our CI staging environment is currently directly provisioned to Xen-based
-virtual machines running on AWS, due to the lack of support for nested
-virtualization. This means that we cannot use the ``grsecurity`` kernel, and the
-environment differs from our locally run staging VMs in other ways. We may be
-able to overcome these limitations by
-`transitioning <https://github.com/freedomofpress/securedrop/issues/3702>`__ to
-running the CI staging environment on compute infrastructure that supports
-nested virtualization.
-
 Running the CI Staging Environment
 ----------------------------------
 
@@ -53,12 +42,12 @@ of the ``freedomofpress`` GitHub organization). The tests also perform
 basic linting and validation, like checking for formatting errors in the
 Sphinx documentation.
 
-.. tip:: You will need an Amazon Web Services EC2 account to proceed.
-         See the `AWS Getting Started Guide`_ for detailed instructions.
+.. tip:: You will need a Google Cloud Platform account to proceed.
+         See the `Google Cloud Platform Getting Started Guide`_ for detailed instructions.
 
-.. _AWS Getting Started Guide: https://aws.amazon.com/ec2/getting-started/
+.. _Google Cloud Platform Getting Started Guide: https://cloud.google.com/getting-started/
 
-In addition to an EC2 account, you will need a working `Docker installation`_ in
+In addition to a GCP account, you will need a working `Docker installation`_ in
 order to run the container that builds the deb packages.
 
 You can verify that your Docker installation is working by running
@@ -82,11 +71,12 @@ Source the setup script using the following command:
 
 .. code:: sh
 
-    source ./devops/scripts/local-setup.sh
+    source ./devops/gce-nested/ci-env.sh
 
 You will be prompted for the values of the required environment variables. There
-are some defaults set that you may want to change. You will need to determine
-the value of your VPC ID to use which is outside the scope of this guide.
+are some defaults set that you may want to change. You will need to export
+``GOOGLE_CREDENTIALS`` with authentication details for your GCP account, 
+which is outside the scope of this guide.
 
 Use Makefile to Provision Hosts
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -98,28 +88,18 @@ Run ``make help`` to see the full list of CI commands in the Makefile:
     $ make help
     Makefile for developing and testing SecureDrop.
     Subcommands:
-        docs: Build project documentation in live reload for editing.
-        docs-lint: Check documentation for common syntax errors.
-        ci-spinup: Creates AWS EC2 hosts for testing staging environment.
-        ci-teardown: Destroy AWS EC2 hosts for testing staging environment.
-        ci-run: Provisions AWS EC2 hosts for testing staging environment.
-        ci-test: Tests AWS EC2 hosts for testing staging environment.
-        ci-go: Creates, provisions, tests, and destroys AWS EC2 hosts
-               for testing staging environment.
-        ci-debug: Prevents automatic destruction of AWS EC2 hosts on error.
+        ci-go                      Creates, provisions, tests, and destroys GCE host for testing staging environment.
+        ci-lint                    Runs linting in linting container.
+        ci-run                     Provisions GCE host for testing staging environment.
+        ci-spinup                  Creates GCE host for testing staging environment.
+        ci-teardown                Destroys GCE host for testing staging environment.
 
 To run the tests locally:
 
 .. code:: sh
 
-    make ci-debug # hosts will not be destroyed automatically
     make ci-go
 
 You can use ``make ci-run`` to provision the remote hosts while making changes,
 including rebuilding the Debian packages used in the Staging environment.
 See :doc:`virtual_environments` for more information.
-
-Note that if you typed ``make ci-debug`` above, you will have to manually remove
-a blank file in ``${HOME}/.FPF_CI_DEBUG`` and then run ``make ci-teardown`` to
-bring down the CI environment. Otherwise, specifically for AWS, you will be
-charged hourly charges until those machines are terminated.
diff --git a/molecule/aws/securedrop_test.pub b/molecule/aws/securedrop_test.pub
deleted file mode 100644
--- a/molecule/aws/securedrop_test.pub
+++ /dev/null
@@ -1,30 +0,0 @@
------BEGIN PGP PUBLIC KEY BLOCK-----
-
-mQENBFhPGZsBCACzn00s3+i5HdGIldDGYXxY2HKL9Qhk0DhiRrNPaQemhNijuFlC
-geCeKN/smDAUyM5mfEoxmWy3V7n8SEQUpqI4dIS2AohReLkyKEKiIpTuXW7F9kO3
-vcXHgrTka+8B4ZQxDuTHNFJLmBwJnP24LrL6BzkDIUNeQFwM0EFTDOJlW1QV6qkm
-9WGizo2sR0VBJJabfRWrTWd8llYOVcc+LptErVNADPaX6iqb+QnZVJ/nYmCTgABj
-lD3aZ4EPZ+ioVOcOxbgBkAX76COObUUw/XahBGwj4fJ5kyzvDSBCHHlRzN39LKpM
-Y+HfSc1scAOWN+Dd0N/joIa0j0U4SGHo1NdzABEBAAG0MVNlY3VyZURyb3AgVEVT
-VElORyBrZXkgPHNlY3VyZWRyb3BAZnJlZWRvbS5wcmVzcz6JAU4EEwEIADgWIQRO
-15zDNi19EoNwRgJKO+SpIhGwPAUCWE8ZmwIbAwULCQgHAgYVCAkKCwIEFgIDAQIe
-AQIXgAAKCRBKO+SpIhGwPCb9B/9SuVoxbe3nLlU0bHDQtoq5P7adyTZK+5gKIiAo
-mtAkc/EuiF6jYIDLo+DBB1GBJVjyD5igTt14XR3JpMe6nLtztD5zgGk47gYQk3y5
-6f5ydd7zRo9OxulRYDvU1mXMUc0EmqfzuSxY55HJy5KQvjeKIU0fTvwbPYXdhFCC
-42iyBIkp4e4/C5oO4lNrNY2DJEZ+a8H5LHasJ4g9A78f/D5q0HWO1HutzfDeiMvq
-WFwlGMD2OzTEQA2MGlVRIYvLHAG1aV9fXY8kjCFT8ri5hxlQeTkKISfbW3pFSq6s
-Ow4r975zWLTPJNm+WTbBpfIOFBVAW34EHkcb/QmntlvqkNM+uQENBFhPGZsBCAC4
-VEtCQEuZ3WzCNL/0yQFih1EjT/AsS3j3++xvSOYWF+c7AjR9X0MkJFTnUZBHs6MX
-PM33bbkWbBBE2ILdDCEF72Uc5HyyC2lW2DvPY9ZLVSGcMCUsKARv5rbeNdgiLVP5
-8AMkmG48q0Pxrr6UVX14M34Jm5G91c/dj9zHtVwkLg4RG/rcumQdlpQhNmMycB2X
-lat48atmEkutfLEQizXIlgiCdNEpgfUBy/jZZcCOjwr8PUPmSUWjKOVMv6CSLx8K
-z2cP4We7tyq4qhc0cWjJOWOmJpu5tbmi6XEEWGaIJyN+POhHEcb0tI1rTJ88nrMb
-DI/NF/35kuWIIkADOb2vABEBAAGJATYEGAEIACAWIQRO15zDNi19EoNwRgJKO+Sp
-IhGwPAUCWE8ZmwIbDAAKCRBKO+SpIhGwPC3fB/0TfuScS718FiEcVRI3F2wBbzTQ
-VARhGzEvPSU5Z3Cur/EB8ihpWvwi39tUMeg5HTheDl/8A7f1QCjIFSVEr1slGNLh
-YFF07XGWhy837z6kiihK2z6/w6Q9QJqjE+QVZCKr97aIPejvEoHoslZTU5pJ52qF
-J7KQd1hEvVs00DxY6VlyK0FzXqByKYq6Arl2tzlCZ6RPEHKXV2xSP06jLEagzgYe
-DylVo9Xahenj4n/Mtq7Am6tGgU9Vy9cGbWNBdUND/mFQEEZSh9RJabPeluH12sir
-5/tfsDr4DGHSz7ws+5M6Zbk6oNJEwQZ4cR+81qCfXE5X5LW1KlAL8wDl7dfS
-=fYUi
------END PGP PUBLIC KEY BLOCK-----
\ No newline at end of file
diff --git a/molecule/aws/tests/test_tor_interfaces.py b/molecule/aws/tests/test_tor_interfaces.py
deleted file mode 100644
--- a/molecule/aws/tests/test_tor_interfaces.py
+++ /dev/null
@@ -1,35 +0,0 @@
-import io
-import os
-import re
-import pytest
-
-TOR_URL_FILES = [{'file': 'app-source-ths',
-                  'check_string': 'SUBMIT DOCUMENTS',
-                  'error_string': "ERROR"}]
-
-testinfra_hosts = ["docker://apptestclient"]
-
-
-@pytest.mark.parametrize('site', TOR_URL_FILES)
-def test_www(host, site):
-    """
-    Ensure tor interface is reachable and returns expected content.
-    """
-
-    # Extract Onion URL from saved onion file, fetched back from app-staging.
-    onion_url_filepath = os.path.join(
-        os.path.dirname(__file__),
-        "../../../install_files/ansible-base/{}".format(site['file'])
-    )
-    onion_url_raw = io.open(onion_url_filepath, 'r').read()
-    onion_url = re.search("\w+\.onion", onion_url_raw).group()
-
-    # Fetch Onion URL via curl to confirm interface is rendered correctly.
-    curl_tor = 'curl -s --socks5-hostname "${{TOR_PROXY}}":9050 {}'.format(
-        onion_url)
-    curl_tor_status = '{} -o /dev/null -w "%{{http_code}}"'.format(curl_tor)
-
-    site_scrape = host.check_output(curl_tor)
-    assert host.check_output(curl_tor_status) == "200"
-    assert site['check_string'] in site_scrape
-    assert site['error_string'] not in site_scrape
diff --git a/molecule/aws/tor_apt_test.yml b/molecule/aws/tor_apt_test.yml
deleted file mode 100644
--- a/molecule/aws/tor_apt_test.yml
+++ /dev/null
@@ -1,26 +0,0 @@
----
-- name: Add apt SD test public key
-  apt_key:
-    data: "{{ lookup('file','securedrop_test.pub') }}"
-    state: present
-
-- name: Force possible tor update
-  meta: flush_handlers
-
-- name: Squash testinfra failure for packages needing update
-  apt:
-    upgrade: safe
-
-- name: Extract latest tor version
-  shell: |
-    apt-cache policy tor | sed -e 's/^\s*Installed:\ \(\S*\)/\1/g;tx;d;:x'
-  changed_when: false
-  register: extract_tor_version
-
-- name: Dump Tor version to file (for reporting)
-  copy:
-    dest: "{{ playbook_dir }}/../../.tor_version"
-    content: "{{ extract_tor_version.stdout }}"
-  delegate_to: localhost
-  run_once: true
-  become: "no"
diff --git a/molecule/builder/tests/test_security_updates.py b/molecule/builder/tests/test_security_updates.py
--- a/molecule/builder/tests/test_security_updates.py
+++ b/molecule/builder/tests/test_security_updates.py
@@ -1,11 +1,6 @@
-import os
-import pytest
-
 testinfra_hosts = ['docker://trusty-sd-sec-update']
 
 
-@pytest.mark.skipif(os.environ.get("FPF_CI", "false") == "true",
-                    reason="Skip in CI, only fail this test locally")
 def test_ensure_no_updates_avail(host):
     """
         Test to make sure that there are no security-updates in the
diff --git a/molecule/testinfra/staging/common/test_grsecurity.py b/molecule/testinfra/staging/common/test_grsecurity.py
--- a/molecule/testinfra/staging/common/test_grsecurity.py
+++ b/molecule/testinfra/staging/common/test_grsecurity.py
@@ -1,5 +1,4 @@
 import pytest
-import os
 import re
 
 
@@ -16,8 +15,6 @@ def test_ssh_motd_disabled(File):
     assert not f.contains("pam\.motd")
 
 
-@pytest.mark.skipif(os.environ.get('FPF_GRSEC', 'true') == "false",
-                    reason="Need to skip in environment w/o grsec")
 @pytest.mark.parametrize("package", [
     'intel-microcode',
     'linux-firmware-image-{}-grsec'.format(KERNEL_VERSION),
@@ -34,8 +31,6 @@ def test_grsecurity_apt_packages(Package, package):
     assert Package(package).is_installed
 
 
-@pytest.mark.skipif(os.environ.get('FPF_GRSEC', 'true') == "false",
-                    reason="Need to skip in environment w/o grsec")
 @pytest.mark.parametrize("package", [
     'linux-signed-image-generic-lts-utopic',
     'linux-signed-image-generic',
@@ -61,8 +56,6 @@ def test_generic_kernels_absent(Command, package):
     assert c.stderr == error_text
 
 
-@pytest.mark.skipif(os.environ.get('FPF_GRSEC', 'true') == "false",
-                    reason="Need to skip in environment w/o grsec")
 def test_grsecurity_lock_file(File):
     """
     Ensure system is rerunning a grsecurity kernel by testing for the
@@ -74,8 +67,6 @@ def test_grsecurity_lock_file(File):
     assert f.size == 0
 
 
-@pytest.mark.skipif(os.environ.get('FPF_GRSEC', 'true') == "false",
-                    reason="Need to skip in environment w/o grsec")
 def test_grsecurity_kernel_is_running(Command):
     """
     Make sure the currently running kernel is specific grsec kernel.
@@ -85,8 +76,6 @@ def test_grsecurity_kernel_is_running(Command):
     assert c.stdout == '{}-grsec'.format(KERNEL_VERSION)
 
 
-@pytest.mark.skipif(os.environ.get('FPF_GRSEC', 'true') == "false",
-                    reason="Need to skip in environment w/o grsec")
 @pytest.mark.parametrize('sysctl_opt', [
   ('kernel.grsecurity.grsec_lock', 1),
   ('kernel.grsecurity.rwxmap_logging', 0),
@@ -101,8 +90,6 @@ def test_grsecurity_sysctl_options(Sysctl, Sudo, sysctl_opt):
         assert Sysctl(sysctl_opt[0]) == sysctl_opt[1]
 
 
-@pytest.mark.skipif(os.environ.get('FPF_GRSEC', 'true') == "false",
-                    reason="Need to skip in environment w/o grsec")
 @pytest.mark.parametrize('paxtest_check', [
   "Executable anonymous mapping",
   "Executable bss",
@@ -137,8 +124,6 @@ def test_grsecurity_paxtest(Command, Sudo, paxtest_check):
             assert re.search(regex, c.stdout)
 
 
-@pytest.mark.skipif(os.environ.get('FPF_CI', 'false') == "true",
-                    reason="Not needed in CI environment")
 def test_grub_pc_marked_manual(Command):
     """
     Ensure the `grub-pc` packaged is marked as manually installed.
@@ -149,8 +134,6 @@ def test_grub_pc_marked_manual(Command):
     assert c.stdout == "grub-pc"
 
 
-@pytest.mark.skipif(os.environ.get('FPF_GRSEC', 'true') == "false",
-                    reason="Need to skip in environment w/o grsec")
 def test_apt_autoremove(Command):
     """
     Ensure old packages have been autoremoved.
@@ -160,8 +143,8 @@ def test_apt_autoremove(Command):
     assert "The following packages will be REMOVED" not in c.stdout
 
 
-@pytest.mark.skipif(os.environ.get('FPF_GRSEC', 'true') == "false",
-                    reason="Need to skip in environment w/o grsec")
+@pytest.mark.xfail(strict=True,
+                   reason="PaX flags unset at install time, see issue #3916")
 @pytest.mark.parametrize("binary", [
     "/usr/sbin/grub-probe",
     "/usr/sbin/grub-mkdevicemap",
diff --git a/molecule/testinfra/staging/common/test_user_config.py b/molecule/testinfra/staging/common/test_user_config.py
--- a/molecule/testinfra/staging/common/test_user_config.py
+++ b/molecule/testinfra/staging/common/test_user_config.py
@@ -1,4 +1,3 @@
-import os
 import re
 
 
@@ -72,8 +71,6 @@ def test_sudoers_tmux_env_deprecated(File):
     """
 
     admin_user = "vagrant"
-    if os.environ.get("FPF_CI", None):
-        admin_user = "sdrop"
 
     f = File("/home/{}/.bashrc".format(admin_user))
     assert not f.contains("^. \/etc\/bashrc\.securedrop_additions$")