Repository: freedomofpress/securedrop

Issue:
securedrop-admin (setup) fails in Tails 4.0-rc1
## Description

Running `./securedrop-admin setup` in tails 4.0-rc1 (upgraded from 3.16) returns the following error:

```
INFO: Virtualenv already exists, not creating
INFO: Checking Python dependencies for securedrop-admin
ERROR: Failed to install pip dependencies. Check network connection and try again.
```
This was done in VMs, will try to reproduce on hardware.

## Steps to Reproduce

1. Upgrade Tails device from 3.16 to 4.0 (Boot into 4.0-rc1 drive and clone to upgrade 3.16 drive)
2. Boot into newly-upgraded tails drive
3. Verify that the internet is working (tor is bootstrapped, you can reach an external website over tor)
4. check out `1.1.0~rc2` tag
5. Run `./securedrop-admin setup` in ~/Persistent/securedrop
6. Observe error

## Expected Behavior

Securedrop-admin should run and the dependencies should be installed.

## Actual Behavior

Securedrop-admin fails and returns an error, the dependencies are not installed

## Comments

Per https://github.com/freedomofpress/securedrop/pull/4852/files#diff-b5e536cc161fcc0d62e661b4d6eae381R70-R73

When running the commands locally, I get
* `lsb_release --id --short` returns `Debian`
* `uname -a` returns `Linux amnesia 5.3.0-trunk-amd64 #1 SMF Debian 5.3.2-1~exp1 (2019-10-02) x86_64 GNU/Linux`

When i run ./securedrop-admin with no parameter, I get:
```
amnesia@amnesia:~/Persistent/securedrop$ ./securedrop-admin help
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'

Current thread 0x00007cf687450740 (most recent call first):
Aborted
```
securedrop-admin (setup) fails in Tails 4.0-rc1
## Description

Running `./securedrop-admin setup` in tails 4.0-rc1 (upgraded from 3.16) returns the following error:

```
INFO: Virtualenv already exists, not creating
INFO: Checking Python dependencies for securedrop-admin
ERROR: Failed to install pip dependencies. Check network connection and try again.
```
This was done in VMs, will try to reproduce on hardware.

## Steps to Reproduce

1. Upgrade Tails device from 3.16 to 4.0 (Boot into 4.0-rc1 drive and clone to upgrade 3.16 drive)
2. Boot into newly-upgraded tails drive
3. Verify that the internet is working (tor is bootstrapped, you can reach an external website over tor)
4. check out `1.1.0~rc2` tag
5. Run `./securedrop-admin setup` in ~/Persistent/securedrop
6. Observe error

## Expected Behavior

Securedrop-admin should run and the dependencies should be installed.

## Actual Behavior

Securedrop-admin fails and returns an error, the dependencies are not installed

## Comments

Per https://github.com/freedomofpress/securedrop/pull/4852/files#diff-b5e536cc161fcc0d62e661b4d6eae381R70-R73

When running the commands locally, I get
* `lsb_release --id --short` returns `Debian`
* `uname -a` returns `Linux amnesia 5.3.0-trunk-amd64 #1 SMF Debian 5.3.2-1~exp1 (2019-10-02) x86_64 GNU/Linux`

When i run ./securedrop-admin with no parameter, I get:
```
amnesia@amnesia:~/Persistent/securedrop$ ./securedrop-admin help
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'

Current thread 0x00007cf687450740 (most recent call first):
Aborted
```


Discussion:
Removing the existing admin virtualenv in `~/Persistent/securedrop/admin/.venv3` resolves the issue for me locally. 

It seems like Buster/tails4-specific dependencies are required to build a buster-compatible venvs. If the virtualenv created in 3.x (stretch) it doesn't seem to work in 4.x (buster).

EDIT: Not quite, after running `./securedrop-admin setup`, invoking any other `securedrop-admin` command returns:
```
amnesia@amnesia:~/Persistent/securedrop$ ./securedrop-admin -v
Traceback (most recent call last):
 File "/home/amnesia/Persistent/securedrop/admin/.venv3/bin/securedrop-admin", line 7, in <module>
   exec(compile(f.read(), __file__, 'exec'))
 File "/home/amnesia/Persistent/securedrop/admin/bin/securedrop-admin", line 22, in <module>
   main(sys.argv[1:])
 File "/home/amnesia/Persistent/securedrop/admin/securedrop_admin/__init__.py", line 1000, in main
   return_code = args.func(args)
AttributeError: 'Namespace' object has no attribute 'func'
```
>It seems like Buster/tails4-specific dependencies are required to build a buster-compatible venvs. If the virtualenv created in 3.x (stretch) it doesn't seem to work in 4.x (buster).

Yes, this is due to different Python versions, 3.5 in stretch and 3.7 in Buster.
Removing the existing admin virtualenv in `~/Persistent/securedrop/admin/.venv3` resolves the issue for me locally. 

It seems like Buster/tails4-specific dependencies are required to build a buster-compatible venvs. If the virtualenv created in 3.x (stretch) it doesn't seem to work in 4.x (buster).

EDIT: Not quite, after running `./securedrop-admin setup`, invoking any other `securedrop-admin` command returns:
```
amnesia@amnesia:~/Persistent/securedrop$ ./securedrop-admin -v
Traceback (most recent call last):
 File "/home/amnesia/Persistent/securedrop/admin/.venv3/bin/securedrop-admin", line 7, in <module>
   exec(compile(f.read(), __file__, 'exec'))
 File "/home/amnesia/Persistent/securedrop/admin/bin/securedrop-admin", line 22, in <module>
   main(sys.argv[1:])
 File "/home/amnesia/Persistent/securedrop/admin/securedrop_admin/__init__.py", line 1000, in main
   return_code = args.func(args)
AttributeError: 'Namespace' object has no attribute 'func'
```
>It seems like Buster/tails4-specific dependencies are required to build a buster-compatible venvs. If the virtualenv created in 3.x (stretch) it doesn't seem to work in 4.x (buster).

Yes, this is due to different Python versions, 3.5 in stretch and 3.7 in Buster.

Resolving patch:
diff --git a/admin/bootstrap.py b/admin/bootstrap.py
--- a/admin/bootstrap.py
+++ b/admin/bootstrap.py
@@ -20,6 +20,7 @@
 import argparse
 import logging
 import os
+import shutil
 import subprocess
 import sys
 
@@ -68,13 +69,48 @@ def is_tails():
         id = None
 
     # dirty hack to unreliably detect Tails 4.0~beta2
-    if id == 'Debian':
+    if id == b'Debian':
         if os.uname()[1] == 'amnesia':
             id = 'Tails'
 
     return id == 'Tails'
 
 
+def clean_up_tails3_venv(virtualenv_dir=VENV_DIR):
+    """
+    Tails 3.x, based on debian stretch uses libpython3.5, whereas Tails 4.x is
+    based on Debian Buster and uses libpython3.7. This means that the Tails 3.x
+    virtualenv will not work under Tails 4.x, and will need to be destroyed and
+    rebuilt. We can detect if the version of libpython is 3.5 in the
+    admin/.venv3/ folder, and delete it if that's the case. This will ensure a
+    smooth upgrade from Tails 3.x to Tails 4.x.
+    """
+    if is_tails():
+        try:
+            dist = subprocess.check_output('lsb_release --codename --short',
+                                           shell=True).strip()
+        except subprocess.CalledProcessError:
+            dist = None
+
+        # tails4 is based on buster
+        if dist == b'buster':
+            python_lib_path = os.path.join(virtualenv_dir, "lib/python3.5")
+            if os.path.exists(os.path.join(python_lib_path)):
+                sdlog.info(
+                    "Tails 3 Python 3 virtualenv detected. "
+                    "Removing it."
+                )
+                shutil.rmtree(virtualenv_dir)
+                sdlog.info("Tails 3 Python 3 virtualenv deleted.")
+
+
+def checkenv(args):
+    clean_up_tails3_venv(VENV_DIR)
+    if not os.path.exists(os.path.join(VENV_DIR, "bin/activate")):
+        sdlog.error('Please run "securedrop-admin setup".')
+        sys.exit(1)
+
+
 def maybe_torify():
     if is_tails():
         return ['torify']
@@ -130,6 +166,9 @@ def envsetup(args):
     Ansible is available to the Admin on subsequent boots without requiring
     installation of packages again.
     """
+    # clean up tails 3.x venv when migrating to tails 4.x
+    clean_up_tails3_venv(VENV_DIR)
+
     # virtualenv doesnt exist? Install dependencies and create
     if not os.path.exists(VENV_DIR):
 
@@ -209,18 +248,30 @@ def parse_argv(argv):
                         help="Increase verbosity on output")
     parser.set_defaults(func=envsetup)
 
+    subparsers = parser.add_subparsers()
+
+    envsetup_parser = subparsers.add_parser(
+        'envsetup',
+        help='Set up the admin virtualenv.'
+    )
+    envsetup_parser.set_defaults(func=envsetup)
+
+    checkenv_parser = subparsers.add_parser(
+        'checkenv',
+        help='Check that the admin virtualenv is properly set up.'
+    )
+    checkenv_parser.set_defaults(func=checkenv)
+
     return parser.parse_args(argv)
 
 
 if __name__ == "__main__":
     args = parse_argv(sys.argv[1:])
     setup_logger(args.v)
-    if args.v:
+
+    try:
         args.func(args)
+    except Exception:
+        sys.exit(1)
     else:
-        try:
-            args.func(args)
-        except Exception:
-            sys.exit(1)
-        else:
-            sys.exit(0)
+        sys.exit(0)


Test patch:
diff --git a/admin/tests/test_securedrop-admin-setup.py b/admin/tests/test_securedrop-admin-setup.py
--- a/admin/tests/test_securedrop-admin-setup.py
+++ b/admin/tests/test_securedrop-admin-setup.py
@@ -18,6 +18,8 @@
 #
 
 import argparse
+import mock
+import os
 import pytest
 import subprocess
 
@@ -74,3 +76,46 @@ def test_install_pip_dependencies_fail(self, caplog):
         assert 'Failed to install' in caplog.text
         assert 'in stdout' in caplog.text
         assert 'in stderr' in caplog.text
+
+    def test_python3_stretch_venv_deleted_in_buster(self, tmpdir, caplog):
+        venv_path = str(tmpdir)
+        python_lib_path = os.path.join(str(tmpdir), 'lib/python3.5')
+        os.makedirs(python_lib_path)
+        with mock.patch('bootstrap.is_tails', return_value=True):
+            with mock.patch('subprocess.check_output', return_value=b"buster"):
+                bootstrap.clean_up_tails3_venv(venv_path)
+                assert 'Tails 3 Python 3 virtualenv detected.' in caplog.text
+                assert 'Tails 3 Python 3 virtualenv deleted.' in caplog.text
+                assert not os.path.exists(venv_path)
+
+    def test_python3_buster_venv_not_deleted_in_buster(self, tmpdir, caplog):
+        venv_path = str(tmpdir)
+        python_lib_path = os.path.join(venv_path, 'lib/python3.7')
+        os.makedirs(python_lib_path)
+        with mock.patch('bootstrap.is_tails', return_value=True):
+            with mock.patch('subprocess.check_output', return_value="buster"):
+                bootstrap.clean_up_tails3_venv(venv_path)
+                assert (
+                    'Tails 3 Python 3 virtualenv detected' not in caplog.text
+                )
+                assert os.path.exists(venv_path)
+
+    def test_python3_stretch_venv_not_deleted_in_stretch(self, tmpdir, caplog):
+        venv_path = str(tmpdir)
+        python_lib_path = os.path.join(venv_path, 'lib/python3.5')
+        os.makedirs(python_lib_path)
+        with mock.patch('bootstrap.is_tails', return_value=True):
+            with mock.patch('subprocess.check_output', return_value="stretch"):
+                bootstrap.clean_up_tails3_venv(venv_path)
+                assert os.path.exists(venv_path)
+
+    def test_venv_cleanup_subprocess_exception(self, tmpdir, caplog):
+        venv_path = str(tmpdir)
+        python_lib_path = os.path.join(venv_path, 'lib/python3.5')
+        os.makedirs(python_lib_path)
+        with mock.patch('bootstrap.is_tails', return_value=True):
+            with mock.patch('subprocess.check_output',
+                            side_effect=subprocess.CalledProcessError(1,
+                                                                      ':o')):
+                bootstrap.clean_up_tails3_venv(venv_path)
+                assert os.path.exists(venv_path)