Repository: StackStorm/st2

Issue:
st2 python3 interpreter is falling back to importing a python2 lib that is incompatible with python3
##### SUMMARY

 One of my libraries is trying to import `cassandra.cluster`.  That library is trying to import `from concurrent.futures import ThreadPoolExecutor, FIRST_COMPLETED, wait as wait_futures` - which, as I understand, is built into python3 (and a separate library for compatibility with python3)

##### ISSUE TYPE
 - Bug Report

##### STACKSTORM VERSION
```
# st2 --version
st2 3.0.0, on Python 2.7.6
```
##### OS / ENVIRONMENT / INSTALL METHOD
Docker container running Ubuntu 14.04

##### STEPS TO REPRODUCE
Create a python runner in your python3-specific pack.  Inside the runner import cassandra libs and just create an object.
```
from cassandra.cluster import Cluster
        cluster = Cluster()
```

##### EXPECTED RESULTS
I expect the library to import and the object to initialize

##### ACTUAL RESULTS
st2 python3 falls back to python2 to import the lib and it throws an exception similar to
```
  File \"/opt/stackstorm/packs/ostk_common/actions/lib/ostkdbs.py\", line 2, in <module>
    from cassandra.cluster import Cluster
  File \"/opt/stackstorm/virtualenvs/ostk_common/lib/python3.5/site-packages/cassandra/cluster.py\", line 23, in <module>
    from concurrent.futures import ThreadPoolExecutor, FIRST_COMPLETED, wait as wait_futures
  File \"/opt/stackstorm/st2/lib/python2.7/site-packages/concurrent/futures/__init__.py\", line 8, in <module>
    from concurrent.futures._base import (FIRST_COMPLETED,
  File \"/opt/stackstorm/st2/lib/python2.7/site-packages/concurrent/futures/_base.py\", line 414
    raise exception_type, self._exception, self._traceback
                        ^
SyntaxError: invalid syntax
```


Discussion:
Thanks for reporting this.

It looks like you installed a pack using ``--python3`` flag? 

I belive its related to #4654 (as per my comments there, there are a lot of edge cases in mixed Python 2 and 3 environments and that's why we plan to deprecate ``--python3`` flag in the future when Ubuntu Bionic packages using Python 3 go GA).
I will look into this issue this week, so I would also appreciate if you can share the offending pack with us.

At the very least, if you can't share the whole pack, please share at least ``requirements.txt`` file and action in question.
I managed to reproduce and track down the root cause for this.

It's related to the ``PYTHONPATH`` we set for Python 3 actions. We include python3.x lib and site-packages directory from pack virtual environment, but we don't include global one (e.g. ``/usr/lib/python3.5``) so it tries to import ``concurrent`` package from Python 2.7 site-packages instead of from global Python 3.x lib directory.

I'm working on a fix and end to end test cases which will catch such issues in the future.

Resolving patch:
diff --git a/st2common/st2common/util/sandboxing.py b/st2common/st2common/util/sandboxing.py
--- a/st2common/st2common/util/sandboxing.py
+++ b/st2common/st2common/util/sandboxing.py
@@ -160,8 +160,30 @@ def get_sandbox_python_path_for_python_action(pack, inherit_from_parent=True,
         python3_site_packages_directory = os.path.join(pack_virtualenv_lib_path,
                                                        virtualenv_directories[0],
                                                        'site-packages')
-        sandbox_python_path = (python3_lib_directory + ':' + python3_site_packages_directory + ':' +
-                               pack_actions_lib_paths + ':' + sandbox_python_path)
+
+        # Work around to make sure we also add system lib dir to PYTHONPATH and not just virtualenv
+        # one
+        # NOTE: abc.py is always available in base lib directory which is symlinked to virtualenv
+        # lib directory
+        abc_module_path = os.path.join(python3_lib_directory, 'abc.py')
+        link_path = os.path.realpath(abc_module_path)
+        python3_system_lib_directory = os.path.dirname(link_path)
+
+        if not os.path.exists(python3_system_lib_directory):
+            python3_system_lib_directory = None
+
+        full_sandbox_python_path = []
+
+        # NOTE: Order here is very important for imports to function correctly
+        if python3_lib_directory:
+            full_sandbox_python_path.append(python3_system_lib_directory)
+
+        full_sandbox_python_path.append(python3_lib_directory)
+        full_sandbox_python_path.append(python3_site_packages_directory)
+        full_sandbox_python_path.append(pack_actions_lib_paths)
+        full_sandbox_python_path.append(sandbox_python_path)
+
+        sandbox_python_path = ':'.join(full_sandbox_python_path)
 
     return sandbox_python_path
 


Test patch:
diff --git a/st2common/tests/unit/test_util_sandboxing.py b/st2common/tests/unit/test_util_sandboxing.py
--- a/st2common/tests/unit/test_util_sandboxing.py
+++ b/st2common/tests/unit/test_util_sandboxing.py
@@ -146,6 +146,7 @@ def test_get_sandbox_python_path_for_python_action_python2_used_for_venv(self,
     @mock.patch('os.path.exists', mock.Mock(return_value=True))
     @mock.patch('os.path.isdir', mock.Mock(return_value=True))
     @mock.patch('os.listdir', mock.Mock(return_value=['python3.6']))
+    @mock.patch('os.path.realpath', mock.Mock(return_value='/usr/lib/python3.6/abc.py'))
     @mock.patch('st2common.util.sandboxing.get_python_lib')
     @mock.patch('st2common.util.sandboxing.get_pack_base_path',
                 mock.Mock(return_value='/tmp/packs/dummy_pack'))
@@ -161,16 +162,17 @@ def test_get_sandbox_python_path_for_python_action_python3_used_for_venv(self,
                                                                 inherit_parent_virtualenv=False)
 
         split = python_path.strip(':').split(':')
-        self.assertEqual(len(split), 3)
+        self.assertEqual(len(split), 4)
 
-        # First entry should be lib/python3 dir from venv
-        self.assertTrue('virtualenvs/dummy_pack/lib/python3.6' in split[0])
+        # First entry should be system lib/python3 dir
+        # Second entry should be lib/python3 dir from venv
+        self.assertTrue('virtualenvs/dummy_pack/lib/python3.6' in split[1])
 
-        # Second entry should be python3 site-packages dir from venv
-        self.assertTrue('virtualenvs/dummy_pack/lib/python3.6/site-packages' in split[1])
+        # Third entry should be python3 site-packages dir from venv
+        self.assertTrue('virtualenvs/dummy_pack/lib/python3.6/site-packages' in split[2])
 
-        # Third entry should be actions/lib dir from pack root directory
-        self.assertTrue('packs/dummy_pack/actions/lib/' in split[2])
+        # Fourth entry should be actions/lib dir from pack root directory
+        self.assertTrue('packs/dummy_pack/actions/lib/' in split[3])
 
         # Inherit python path from current process
         # Mock the current process python path
@@ -179,7 +181,8 @@ def test_get_sandbox_python_path_for_python_action_python3_used_for_venv(self,
         python_path = get_sandbox_python_path_for_python_action(pack='dummy_pack',
                                                                 inherit_from_parent=True,
                                                                 inherit_parent_virtualenv=False)
-        expected = ('/tmp/virtualenvs/dummy_pack/lib/python3.6:'
+        expected = ('/usr/lib/python3.6:'
+                    '/tmp/virtualenvs/dummy_pack/lib/python3.6:'
                     '/tmp/virtualenvs/dummy_pack/lib/python3.6/site-packages:'
                     '/tmp/packs/dummy_pack/actions/lib/::/data/test1:/data/test2')
         self.assertEqual(python_path, expected)
@@ -198,7 +201,8 @@ def test_get_sandbox_python_path_for_python_action_python3_used_for_venv(self,
                                                                 inherit_from_parent=True,
                                                                 inherit_parent_virtualenv=True)
 
-        expected = ('/tmp/virtualenvs/dummy_pack/lib/python3.6:'
+        expected = ('/usr/lib/python3.6:'
+                    '/tmp/virtualenvs/dummy_pack/lib/python3.6:'
                     '/tmp/virtualenvs/dummy_pack/lib/python3.6/site-packages:'
                     '/tmp/packs/dummy_pack/actions/lib/::/data/test1:/data/test2:'
                     '%s/virtualenvtest' % (sys.prefix))