Repository: python/mypy
Pull request: Skip tests in testpep561.py
Added logic to temporarily skip tests in testpep561.py that fail on Mac due to nesting of ```virtualenv ``` inside a ```venv```. This PR could be updated to only skip tests on Mac but I've left that out for now. Once this bug is fixed these tests should be added back in. 

Fixes [#6798](https://github.com/python/mypy/issues/6798)

Problem statement:
testpep561.py fails locally when run from a venv
There's one test that always fails for me when I run pytest locally. Other people also see this, and it is apparently due to the nesting of virtual environments. In particular, I run pytest in a venv, and then the test creates a _nested_ virtual environment. It's been speculated that it would pass if I created my own virtual environment using `virtualenv` rather than `venv`. (The testpep561.py file uses `virtualenv`, and it has a comment explaining why -- it needs to create a Python 2 virtual environment.)

If this can't be fixed, maybe the offending tests can detect that they're being run inside of a `venv` and then be skipped? Note that Travis-CI and AppVeyor are unaffected.

Hints from issue discussion:
Could you give a snippet of output? Also I presume you mean that it would work if `venv` was used instead of `virtualenv`?
I don't know for sure what combination would work. I chatted a bit with @msullivan and he seems to think that `venv` doesn't nest properly but `virtualenv` does. I haven't tested this theory yet.

Output:
```
(v37) ~/src/mypy$ pytest -n0 mypy/test/testpep561.py 
==================================================================== test session starts ====================================================================
platform darwin -- Python 3.7.2, pytest-4.4.1, py-1.5.3, pluggy-0.9.0
rootdir: /Users/guido/src/mypy, inifile: pytest.ini
plugins: xdist-1.22.2, forked-0.2, cov-2.5.1
collected 12 items                                                                                                                                          

mypy/test/testpep561.py ......F....F                                                                                                                  [100%]

========================================================================= FAILURES ==========================================================================
_____________________________________________________________ TestPEP561.test_typedpkg_editable _____________________________________________________________

self = <mypy.test.testpep561.TestPEP561 testMethod=test_typedpkg_editable>

    def test_typedpkg_editable(self) -> None:
        self.simple_prog.create()
        with self.virtualenv() as venv:
            venv_dir, python_executable = venv
            self.install_package('typedpkg', python_executable, editable=True)
            self.simple_prog.check_mypy_run(
                python_executable,
                [SimpleMsg.msg_tuple],
>               venv_dir=venv_dir,
            )

mypy/test/testpep561.py:269: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <mypy.test.testpep561.ExampleProg object at 0x111a48320>
python_executable = '/var/folders/63/czkyq6090dd0t157zhx54xvhrdlybt/T/tmp64k0yqc7/bin/python'
expected_out = [<SimpleMsg.msg_tuple: "{tempfile}:5: error: Revealed type is 'builtins.tuple[builtins.str]'">], expected_err = '', expected_returncode = 1
venv_dir = '/var/folders/63/czkyq6090dd0t157zhx54xvhrdlybt/T/tmp64k0yqc7'

    def check_mypy_run(self,
                       python_executable: str,
                       expected_out: List[Enum],
                       expected_err: str = '',
                       expected_returncode: int = 1,
                       venv_dir: Optional[str] = None) -> None:
        """Helper to run mypy and check the output."""
        cmd_line = [self._full_fname]
        if venv_dir is not None:
            old_dir = os.getcwd()
            os.chdir(venv_dir)
        try:
            if python_executable != sys.executable:
                cmd_line.append('--python-executable={}'.format(python_executable))
            out, err, returncode = mypy.api.run(cmd_line)
>           assert out == self.build_msg(*expected_out), err
E           AssertionError: 
E           assert "/var/folders...pe is 'Any'\n" == "/var/folders/...ltins.str]'\n"
E             Skipping 67 identical leading characters in diff, use -v to show
E             + rogram.py:5: error: Revealed type is 'builtins.tuple[builtins.str]'
E             - rogram.py:2: error: Cannot find module named 'typedpkg.sample'
E             - /var/folders/63/czkyq6090dd0t157zhx54xvhrdlybt/T/tmpdt5w3203/test_program.py:2: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
E             - /var/folders/63/czkyq6090dd0t157zhx54xvhrdlybt/T/tmpdt5w3203/test_program.py:3: error: Cannot find module named 'typedpkg'
E             - /var/folders/63/czkyq6090dd0t157zhx54xvhrdlybt/T/tmpdt5w3203/test_program.py:5: error: Re...
E             
E             ...Full output truncated (1 line hidden), use '-vv' to show

mypy/test/testpep561.py:119: AssertionError
__________________________________________________________ TestPEP561.test_typedpkg_stubs_python2 ___________________________________________________________

self = <mypy.test.testpep561.TestPEP561 testMethod=test_typedpkg_stubs_python2>

    def test_typedpkg_stubs_python2(self) -> None:
        self.simple_prog.create()
        python2 = try_find_python2_interpreter()
        if python2:
>           with self.virtualenv(python2) as venv:

mypy/test/testpep561.py:228: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:112: in __enter__
    return next(self.gen)
mypy/test/testpep561.py:146: in virtualenv
    self.fail("Failed to create venv. Do you have virtualenv installed?\n" + err)
E   AssertionError: Failed to create venv. Do you have virtualenv installed?
E   New python executable in /var/folders/63/czkyq6090dd0t157zhx54xvhrdlybt/T/tmp98jv_qbv/bin/python
E   Running virtualenv with interpreter /usr/local/bin/python2
E   Traceback (most recent call last):
E     File "/Users/guido/v37/lib/python3.7/site-packages/virtualenv.py", line 2328, in <module>
E       main()
E     File "/Users/guido/v37/lib/python3.7/site-packages/virtualenv.py", line 713, in main
E       symlink=options.symlink)
E     File "/Users/guido/v37/lib/python3.7/site-packages/virtualenv.py", line 925, in create_environment
E       site_packages=site_packages, clear=clear, symlink=symlink))
E     File "/Users/guido/v37/lib/python3.7/site-packages/virtualenv.py", line 1387, in install_python
E       raise e
E   OSError: [Errno 1] Operation not permitted
=========================================================== 2 failed, 10 passed in 170.75 seconds ===========================================================
```
I can confirm that if I use `virtualenv` instead of `venv` to create my own (i.e., the outer) virtual environment, the test passes.
I have very similar symptoms to those Guido has. Also I see this only on Mac, on Linux `venv` works perfectly. So my guess it is something related to `__PYVENV_LAUNCHER__`.
I agree it seems a Mac only bug, as my experience with running the test suite on Windows and Linux has been mostly flawless. I'm afraid I don't have a machine to debug this on then, at least for now.
These tests should be skipped (at least when run from a venv) until a fix has been implemented. Anybody interested on implementing the skip logic?
Hi everyone! I've had a go at this one, would really appreciate any feedback!

Commit message: Fixed line length, added mac check and included test_typedpkg_stubs_python2

Patch to review:
diff --git a/mypy/test/testpep561.py b/mypy/test/testpep561.py
index 4b988503df79..5eaacd731342 100644
--- a/mypy/test/testpep561.py
+++ b/mypy/test/testpep561.py
@@ -1,6 +1,7 @@
 from contextlib import contextmanager
 from enum import Enum
 import os
+import pytest  # type: ignore
 import subprocess
 from subprocess import PIPE
 import sys
@@ -187,6 +188,9 @@ def test_get_pkg_dirs(self) -> None:
         dirs = get_site_packages_dirs(sys.executable)
         assert dirs
 
+    @pytest.mark.skipif(sys.platform == 'darwin' and hasattr(sys, 'base_prefix') and \
+    sys.base_prefix != sys.prefix, \
+    reason="Temporarily skip to avoid having a virtualenv within a venv.")
     def test_typedpkg_stub_package(self) -> None:
         self.simple_prog.create()
         with self.virtualenv() as venv:
@@ -198,6 +202,9 @@ def test_typedpkg_stub_package(self) -> None:
                 venv_dir=venv_dir,
             )
 
+    @pytest.mark.skipif(sys.platform == 'darwin' and hasattr(sys, 'base_prefix') and \
+    sys.base_prefix != sys.prefix, \
+    reason="Temporarily skip to avoid having a virtualenv within a venv.")
     def test_typedpkg(self) -> None:
         self.simple_prog.create()
         with self.virtualenv() as venv:
@@ -221,6 +228,9 @@ def test_stub_and_typed_pkg(self) -> None:
                 venv_dir=venv_dir,
             )
 
+    @pytest.mark.skipif(sys.platform == 'darwin' and hasattr(sys, 'base_prefix') and \
+    sys.base_prefix != sys.prefix, \
+    reason="Temporarily skip to avoid having a virtualenv within a venv.")
     def test_typedpkg_stubs_python2(self) -> None:
         self.simple_prog.create()
         python2 = try_find_python2_interpreter()
@@ -247,6 +257,9 @@ def test_typedpkg_python2(self) -> None:
                     venv_dir=venv_dir,
                 )
 
+    @pytest.mark.skipif(sys.platform == 'darwin' and hasattr(sys, 'base_prefix') and \
+    sys.base_prefix != sys.prefix, \
+    reason="Temporarily skip to avoid having a virtualenv within a venv.")
     def test_typedpkg_egg(self) -> None:
         self.simple_prog.create()
         with self.virtualenv() as venv:
@@ -258,6 +271,9 @@ def test_typedpkg_egg(self) -> None:
                 venv_dir=venv_dir,
             )
 
+    @pytest.mark.skipif(sys.platform == 'darwin' and hasattr(sys, 'base_prefix') and \
+    sys.base_prefix != sys.prefix, \
+    reason="Temporarily skip to avoid having a virtualenv within a venv.")
     def test_typedpkg_editable(self) -> None:
         self.simple_prog.create()
         with self.virtualenv() as venv:
@@ -269,6 +285,9 @@ def test_typedpkg_editable(self) -> None:
                 venv_dir=venv_dir,
             )
 
+    @pytest.mark.skipif(sys.platform == 'darwin' and hasattr(sys, 'base_prefix') and \
+    sys.base_prefix != sys.prefix, \
+    reason="Temporarily skip to avoid having a virtualenv within a venv.")
     def test_typedpkg_egg_editable(self) -> None:
         self.simple_prog.create()
         with self.virtualenv() as venv:


Code review comments:
diff_hunk: @@ -188,7 +188,9 @@ def test_get_pkg_dirs(self) -> None:
         dirs = get_site_packages_dirs(sys.executable)
         assert dirs
 
-    @pytest.mark.skipif(hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix, reason="Temporarily skip to avoid having a virtualenv within a venv.")
+    @pytest.mark.skipif(sys.platform == 'darwin' and hasattr(sys, 'base_prefix') and \
+    sys.base_prefix != sys.prefix, \
+    reason="Temporarily skip to avoid having a virtualenv within a venv.")
path: mypy/test/testpep561.py
text: @user1:
You don't need the backslashes, because it's inside `(...)`. Also, please reindent the continuation lines to line up (i.e. `sys` and `reason` in the 2nd and 3rd lines should start in the same column as `sys` on the previous line).

@author:
Fixed, cheers!