Repository: pypa/pipenv

Issue:
Properly handle pre-existing .venv files
Fixes #2680 as discussed in the issue with bonus support for custom paths in the .venv file. Not sure what to categorise this under for the news fragment so I will wait for someone's advice on that before writing up the news fragment.

Would like reviewers to especially look at the test cases written. Could the repeated code be moved elsewhere? Are path comparisons properly handled? - concerns are accounting for symlinks and case-insensitivity on Windows.
Pipenv breaks with pre-existing usage of .venv files
#### Issue description
tl;dr `.venv` files already exist in some dev setups but when present, `pipenv` attempts to use that file as a directory and install a virtualenv into it.

Virtualenvwrapper has suggested for awhile to use `.venv` files in your project directory to hold the name of your virtualenv so that you could automatically activate virtualenvs.  Spacemacs (an Emacs editor setup) also does this behavior by default.

Two problems:

1. Pipenv will silently fail to create and activate a virtualenv when a `.venv` file is present.
2. If a `.venv` file is added to an existing pipenv project, pipenv breaks because it can't activate the virtualenv at `<project_dir>/.venv` ignoring the place where it had already created a virtualenv.

Prior Art Links:
 - Virtualenvwrapper Tips & Tricks: http://virtualenvwrapper.readthedocs.io/en/latest/tips.html
 - Spacemacs Python Layer:  https://github.com/syl20bnr/spacemacs/tree/develop/layers/%2Blang/python#automatic-activation-of-local-pyenv-version
 - Virtualenv in git directories: https://hmarr.com/2010/jan/19/making-virtualenv-play-nice-with-git/
 - Sample .bash_profile using `.venv` files: https://github.com/justinabrahms/jlilly-bashy-dotfiles/commit/04899f005397499e89da6d562b062545e70d7975#commitcomment-1526375


#### Expected result

At the least, Pipenv should check if `.venv` is a file or a directory before attempting to automatically use it as a directory and install a virtualenv into it.  If it's a file, then behave as if no `.venv` was found at all.

#### Actual result

If a `.venv` file exists in the project directory, Pipenv will attempt to create and use a virtualenv at `<project_dir>/.venv` and yell at the user a bit because it's not a directory but (a) it won't actually fail and (b) it won't actually save a virtualenv _anywhere_.

Output:
```
Creating a virtualenv for this project...
Pipfile: <project_path>\Pipfile
Using <python_path> to create virtualenv...
Running virtualenv with interpreter <python_path>
ERROR: File already exists and is not a directory.
Please provide a different path or delete the file.

Virtualenv location: <project_path>\.venv
```

#### Steps to replicate

`touch .venv`
`pipenv install`

Also see output of `pipenv --venv`
-------------------------------------------------------------------------------
<details><summary>$ pipenv --support</summary>

*REDACTED*

Pipenv version: `'2018.7.1'`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.17134',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'win32'}
```

System environment variables:

  - `ADSK_CLM_WPAD_PROXY_CHECK`
  - `ALIASES`
  - `ALLUSERSPROFILE`
  - `ANSICON`
  - `ANSICON_DEF`
  - `APPDATA`
  - `ARCHITECTURE`
  - `CHOCOLATEYINSTALL`
  - `CHOCOLATEYLASTPATHUPDATE`
  - `CMDER_ROOT`
  - `COMMONPROGRAMFILES`
  - `COMMONPROGRAMFILES(X86)`
  - `COMMONPROGRAMW6432`
  - `COMPUTERNAME`
  - `COMSPEC`
  - `CONEMUANSI`
  - `CONEMUANSILOG`
  - `CONEMUARGS`
  - `CONEMUARGS2`
  - `CONEMUBACKHWND`
  - `CONEMUBASEDIR`
  - `CONEMUBASEDIRSHORT`
  - `CONEMUBUILD`
  - `CONEMUCFGDIR`
  - `CONEMUCONFIG`
  - `CONEMUDIR`
  - `CONEMUDRAWHWND`
  - `CONEMUDRIVE`
  - `CONEMUHOOKS`
  - `CONEMUHWND`
  - `CONEMUPALETTE`
  - `CONEMUPID`
  - `CONEMUSERVERPID`
  - `CONEMUTASK`
  - `CONEMUWORKDIR`
  - `CONEMUWORKDRIVE`
  - `DRIVERDATA`
  - `FPS_BROWSER_APP_PROFILE_STRING`
  - `FPS_BROWSER_USER_PROFILE_STRING`
  - `FSHARPINSTALLDIR`
  - `GIT_INSTALL_ROOT`
  - `GOPATH`
  - `GOROOT`
  - `GTK_BASEPATH`
  - `HOME`
  - `HOMEDRIVE`
  - `HOMEPATH`
  - `LOCALAPPDATA`
  - `LOGONSERVER`
  - `NUMBER_OF_PROCESSORS`
  - `ONEDRIVE`
  - `OS`
  - `PATH`
  - `PATHEXT`
  - `PLINK_PROTOCOL`
  - `PROCESSOR_ARCHITECTURE`
  - `PROCESSOR_IDENTIFIER`
  - `PROCESSOR_LEVEL`
  - `PROCESSOR_REVISION`
  - `PROGRAMDATA`
  - `PROGRAMFILES`
  - `PROGRAMFILES(X86)`
  - `PROGRAMW6432`
  - `PROMPT`
  - `PSMODULEPATH`
  - `PUBLIC`
  - `RUBYOPT`
  - `SESSIONNAME`
  - `SVN_SSH`
  - `SYSTEMDRIVE`
  - `SYSTEMROOT`
  - `TEMP`
  - `TERM`
  - `TMP`
  - `USER-ALIASES`
  - `USERDOMAIN`
  - `USERDOMAIN_ROAMINGPROFILE`
  - `USERNAME`
  - `USERPROFILE`
  - `VERBOSE-OUTPUT`
  - `WINDIR`
  - `WORKON_HOME`
  - `PYTHONDONTWRITEBYTECODE`
  - `PIP_PYTHON_PATH`

Pipenvûspecific environment variables:

---------------------------

Contents of `Pipfile`:

```toml
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[scripts]
server = "python manage.py runserver"
migrate = "python manage.py migrate"
test = "python manage.py test"

[packages]
"boto3" = "==1.7"
dj-database-url = "==0.5.0"
json-logging-py = "*"
"psycopg2" = "==2.7"
pytz = "==2018.4"
requests = "==2.18.4"
Django = "==2.0.6"
django_csp = "==3.4"
"Jinja2" = "==2.10"
djangorestframework = "*"

[dev-packages]
ipython = "*"
pylint = "*"
pylint-django = "*"
pylint-plugin-utils = "*"
autoflake = "*"
importmagic = "*"
epc = "*"

[requires]
python_version = "3.6"

```
</details>



Discussion:

I guess one simple way out is to change the `exists` check to `isdir`. We can depend on users of this style to *not* also set `PIPENV_VENV_IN_PROJECT` since it simply does not make sense for them to work together.

@tgroshon Can you verify that in this configuration, a `.venv` file would contain *the name of the environment you want to activate*? I.e. `$WORKON_HOME/$(cat .venv)` would point me to the virtual environment I want to activate. I’m thinking we might be able to leverage this and make things work even better.
@uranusjr Yes I agree with you!

-  Attempt to fix by checking `isdir` and expect users to not also be setting that env var.
-  If a `.venv` _directory_ is present, expect to use it for containing the virtualenv.
-  If a `.venv` _file_ is present, expect for it to contain _the name of an environment_.

In the future, it may make sense to also allow a `.venv` file to also contain a _path to an environment_.

I agree with the allow path part :D It should also be relatively simple to do—just detect if the content contains a `/` or `\` (it shouldn’t if it’s an environment name), and treat it as path if it does.

Call out for contributors: This would be a very good feature to work on, since it would also solve a common request about knowing the environment location ahead-of-time, or create the environment in a user-defined location, e.g. #589 #746.

Resolving patch:
diff --git a/pipenv/project.py b/pipenv/project.py
--- a/pipenv/project.py
+++ b/pipenv/project.py
@@ -34,6 +34,7 @@
     is_star,
     get_workon_home,
     is_virtual_environment,
+    looks_like_dir
 )
 from .environments import (
     PIPENV_MAX_DEPTH,
@@ -267,7 +268,17 @@ def virtualenv_exists(self):
     def get_location_for_virtualenv(self):
         if self.is_venv_in_project():
             return os.path.join(self.project_directory, ".venv")
-        return str(get_workon_home().joinpath(self.virtualenv_name))
+
+        name = self.virtualenv_name
+        if self.project_directory:
+            venv_path = os.path.join(self.project_directory, ".venv")
+            if os.path.exists(venv_path) and not os.path.isdir(".venv"):
+                with io.open(venv_path, "r") as f:
+                    name = f.read().strip()
+                # Assume file's contents is a path if it contains slashes.
+                if looks_like_dir(name):
+                    return Path(name).absolute().as_posix()
+        return str(get_workon_home().joinpath(name))
 
     def get_installed_packages(self):
         from . import PIPENV_ROOT, PIPENV_VENDOR, PIPENV_PATCHED
diff --git a/pipenv/utils.py b/pipenv/utils.py
--- a/pipenv/utils.py
+++ b/pipenv/utils.py
@@ -1376,3 +1376,8 @@ def chdir(path):
         yield
     finally:
         os.chdir(prev_cwd)
+
+
+def looks_like_dir(path):
+    seps = (sep for sep in (os.path.sep, os.path.altsep) if sep is not None)
+    return any(sep in path for sep in seps)


Test patch:
diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py
--- a/tests/integration/conftest.py
+++ b/tests/integration/conftest.py
@@ -16,6 +16,9 @@ class ResourceWarning(Warning):
         pass
 
 
+HAS_WARNED_GITHUB = False
+
+
 def check_internet():
     try:
         # Kenneth represents the Internet LGTM.
@@ -40,13 +43,15 @@ def check_github_ssh():
         res = True if c.return_code == 1 else False
     except Exception:
         pass
-    if not res:
+    global HAS_WARNED_GITHUB
+    if not res and not HAS_WARNED_GITHUB:
         warnings.warn(
             'Cannot connect to GitHub via SSH', ResourceWarning
         )
         warnings.warn(
             'Will skip tests requiring SSH access to GitHub', ResourceWarning
         )
+        HAS_WARNED_GITHUB = True
     return res
 
 
diff --git a/tests/integration/test_dot_venv.py b/tests/integration/test_dot_venv.py
--- a/tests/integration/test_dot_venv.py
+++ b/tests/integration/test_dot_venv.py
@@ -41,15 +41,17 @@ def test_reuse_previous_venv(PipenvInstance, pypi):
         assert c.return_code == 0
         assert normalize_drive(p.path) in p.pipenv('--venv').out
 
+
 @pytest.mark.dotvenv
-def test_venv_file_exists(PipenvInstance, pypi):
-    """Tests virtualenv creation & package installation when a .venv file exists
-    at the project root.
+@pytest.mark.parametrize('venv_name', ('test-venv', os.path.join('foo', 'test-venv')))
+def test_venv_file(venv_name, PipenvInstance, pypi):
+    """Tests virtualenv creation when a .venv file exists at the project root
+    and contains a venv name.
     """
     with PipenvInstance(pypi=pypi, chdir=True) as p:
         file_path = os.path.join(p.path, '.venv')
         with open(file_path, 'w') as f:
-            f.write('')
+            f.write(venv_name)
 
         with temp_environ(), TemporaryDirectory(
             prefix='pipenv-', suffix='temp_workon_home'
@@ -58,12 +60,43 @@ def test_venv_file_exists(PipenvInstance, pypi):
             if 'PIPENV_VENV_IN_PROJECT' in os.environ:
                 del os.environ['PIPENV_VENV_IN_PROJECT']
 
-            c = p.pipenv('install requests')
+            c = p.pipenv('install')
+            assert c.return_code == 0
+
+            c = p.pipenv('--venv')
+            assert c.return_code == 0
+            venv_loc = Path(c.out.strip()).absolute()
+            assert venv_loc.exists()
+            assert venv_loc.joinpath('.project').exists()
+            venv_path = venv_loc.as_posix()
+            if os.path.sep in venv_name:
+                venv_expected_path = Path(p.path).joinpath(venv_name).absolute().as_posix()
+            else:
+                venv_expected_path = Path(workon_home.name).joinpath(venv_name).absolute().as_posix()
+            assert venv_path == venv_expected_path
+
+
+@pytest.mark.dotvenv
+def test_venv_file_with_path(PipenvInstance, pypi):
+    """Tests virtualenv creation when a .venv file exists at the project root
+    and contains an absolute path.
+    """
+    with temp_environ(), PipenvInstance(chdir=True, pypi=pypi) as p:
+        with TemporaryDirectory(
+            prefix='pipenv-', suffix='-test_venv'
+        ) as venv_path:
+            if 'PIPENV_VENV_IN_PROJECT' in os.environ:
+                del os.environ['PIPENV_VENV_IN_PROJECT']
+
+            file_path = os.path.join(p.path, '.venv')
+            with open(file_path, 'w') as f:
+                f.write(venv_path.name)
+
+            c = p.pipenv('install')
+            assert c.return_code == 0
+            c = p.pipenv('--venv')
             assert c.return_code == 0
+            venv_loc = Path(c.out.strip())
 
-            venv_loc = None
-            for line in c.err.splitlines():
-                if line.startswith('Virtualenv location:'):
-                    venv_loc = Path(line.split(':', 1)[-1].strip())
-            assert venv_loc is not None
             assert venv_loc.joinpath('.project').exists()
+            assert venv_loc == Path(venv_path.name)
diff --git a/tests/integration/test_install_basic.py b/tests/integration/test_install_basic.py
--- a/tests/integration/test_install_basic.py
+++ b/tests/integration/test_install_basic.py
@@ -44,7 +44,7 @@ def test_basic_install(PipenvInstance, pypi):
 @pytest.mark.install
 @flaky
 def test_mirror_install(PipenvInstance, pypi):
-    with temp_environ(), PipenvInstance(chdir=True) as p:
+    with temp_environ(), PipenvInstance(chdir=True, pypi=pypi) as p:
         mirror_url = os.environ.pop(
             "PIPENV_TEST_INDEX", "https://pypi.python.org/simple"
         )
diff --git a/tests/pytest-pypi/pytest_pypi/certs.py b/tests/pytest-pypi/pytest_pypi/certs.py
--- a/tests/pytest-pypi/pytest_pypi/certs.py
+++ b/tests/pytest-pypi/pytest_pypi/certs.py
@@ -15,7 +15,7 @@
 def where():
     """Return the preferred certificate bundle."""
     # vendored bundle inside Requests
-    return os.path.join(os.path.dirname(__file__), 'certs', 'cacert.pem')
+    return os.path.join(os.path.abspath(os.path.dirname(__file__)), 'certs', 'cacert.pem')
 
 if __name__ == '__main__':
     print(where())