Repository: pypa/pip

Issue:
`pip freeze` fails to properly return `hg` URL if virtualenv is under Git controlled project
- Pip version: 8.1.2
- Python version: 2.7.12
- Operating System: Ubuntu 16.04.1
### Description:

I've got my virtulenv directory (`venv`) under my project folder

```
acme-server/
├── bin
├── acme
├── manage.py
├── Procfile
├── requirements
├── requirements.txt
├── venv
└── web
```

One of my requirements is installed via `-e hg+https://`, which ends up getting cloned to `venv/src`.

I'd like to automatically update my `requirements.txt` via `pip freeze`, but `pip` keeps returning my `hg` repo as a git URL instead.

I'd expect

```
-e hg+https://bitbucket.org/somedev/some-project@<some_projects_hash>#egg=some_project
```

but I keep getting this instead

```
-e git+git@github.com:Acme/acme-server.git@<my_projects_hash>#egg=some_project&subdirectory=venv/src/some_project
```

which isn't very helpful for my `requirements.txt`.
### What I've run:

I dug into `pip`'s source, and narrowed it down to `controls_location()` in [`pip/vcs/git.py`](https://github.com/pypa/pip/blob/218fe03db5076025bb8e0c017c559d9e68e354c5/pip/vcs/git.py#L266).

```

   @classmethod
   def controls_location(cls, location):
       if super(Git, cls).controls_location(location):
           return True
       try:
           r = cls().run_command(['rev-parse'],
                                 cwd=location,
                                 show_stdout=False,
                                 on_returncode='ignore')
           return not r
       except BadCommand:
           logger.debug("could not determine if %s is under git control "
                        "because git is not available", location)
           return False
```

Turns out, running `git rev-parse` in the `hg`-cloned copy of `some_project`  does not raise an error, because `git` traverses the tree upwards, eventually reaching the top level of my own project and finding the `.git` there instead. That would seem why `<my_projects_hash>` appears in the `pip freeze` output.

In [`pip/vcs::VcsSupport.get_backend_name()`](https://github.com/pypa/pip/blob/218fe03db5076025bb8e0c017c559d9e68e354c5/pip/vcs/__init__.py#L69), it would appear the the git module is running first, and because `git rev-parse` is not raising an error from within the `hg`-cloned directory under `venv/src`, is claiming the `hg`-cloned project as a git repo instead (even before the `hg` module has a chance to run).



Discussion:
Using the `--show-cdup` option to `git rev-parse` in [`controls_location()`](https://github.com/pypa/pip/blob/218fe03db5076025bb8e0c017c559d9e68e354c5/pip/vcs/git.py#L266) seems to avoid my particular problem.

```
   @classmethod
   def controls_location(cls, location):
       if super(Git, cls).controls_location(location):
           return True
       try:
-          r = cls().run_command(['rev-parse'],
+          r = cls().run_command(['rev-parse', '--show-cdup'],
                                 cwd=location,
                                 show_stdout=False,
                                 on_returncode='ignore')
           return not r
       except BadCommand:
           logger.debug("could not determine if %s is under git control "
                        "because git is not available", location)
           return False
```

In the `hg`-cloned project, with git identifying the top-level project `.git/`, 

```
$ pwd
/home/user/acme-server/venv/src/some-project
$ git rev-parse --show-cdup
../../..
```

In a `git`-cloned project under `venv/src/`

```
$ pwd
/home/user/acme-server/venv/src/some-git-project
$ git rev-parse --show-cdup

$
```

Unfortunately, I don't see why `--show-cdup` would be of any help: `git rev-parse --show-cdup` still returns a valid path so pip will believe it found a git repo.

Maybe pip should try all its supported VCS and if it found several repository, select the one with the smallest subdirectory path ?
But even this heuristic would not always provide the _right_ answer so...
We are clearly on an edge case, I'm not certain we want to complicate pip freeze code to deal with this...
cc @dstufft @pfmoore 

Agreed, this seems like it would be pretty complex code for a rare situation. I'm inclined to just say "don't put your virtualenv directory under your project folder", tbh.

> Unfortunately, I don't see why --show-cdup would be of any help: git rev-parse --show-cdup still returns a valid path so pip will believe it found a git repo.
> 
> Maybe pip should try all its supported VCS and if it found several repository, select the one with the smallest subdirectory path ?

This is what worked for me though. `rev-parse --show-cdup` returned a non zero-length path in the `hg`-cloned repo (vs a zero-length path in another `git`-cloned repo) under `venv/src`). 

The `return not r` line after calling `rev-parse --show-cdup` then returned `False` for the git module's `controls_location()` in that scenario, which was correct.

Since the command is being run in the cloned project's directory, `cwd=location`, wouldn't a non zero-length return indicate that this is _not_ a `git` repo (while potentially _being in_ a `git` repo)?

> We are clearly on an edge case, I'm not certain we want to complicate pip freeze code to deal with this...

Given that pip officially supports doing a `pip install -e hg+https://...`, shouldn't sanely returning the repo during `pip freeze` be a non-edge case?

> Agreed, this seems like it would be pretty complex code for a rare situation. I'm inclined to just say "don't put your virtualenv directory under your project folder", tbh.

It might be that the "edge case" here is the location of the `venv`. There are other sites that do show examples of having the virtuelenv directory under the project folder though
- http://docs.python-guide.org/en/latest/dev/virtualenvs/#virtualenv
- http://flask.pocoo.org/docs/0.11/installation/#virtualenv
- https://tutorial.djangogirls.org/en/django_installation/#virtual-environment

> This is what worked for me though. rev-parse --show-cdup returned a non zero-length path in the hg-cloned repo (vs a zero-length path in another git-cloned repo) under venv/src).
> 
> The return not r line after calling rev-parse --show-cdup then returned False for the git module's controls_location() in that scenario, which was correct.
> 
> Since the command is being run in the cloned project's directory, cwd=location, wouldn't a non zero-length return indicate that this is not a git repo (while potentially being in a git repo)?

My bad, I scanned the code too quickly.
`git rev-parse` should return nothing in a git repo, not the path to root dir.
Adding `--show-cdup` would break `pip freeze` for project installed via subdirectory of a git repo (as explained in https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support).

> It might be that the "edge case" here is the location of the venv. There are other sites that do show examples of having the virtuelenv directory under the project folder though

I agree that having a venv in your project folder (and hence in a vcs repository) is a common usecase.

So maybe changing `get_backend_name` to test all VCS and return the one with the smallest subdirectory would be appropriate ? PR welcome :)

> So maybe changing get_backend_name to test all VCS and return the one with the smallest subdirectory would be appropriate ?

I could take a stab at it when I have some free time from my primary project =) Would this be the preferred way about it?

> Would this be the preferred way about it?

I don't think of any other way (as `--show-cdup` would break the `subdirectory` use case), so yes.


Resolving patch:
diff --git a/src/pip/_internal/vcs/git.py b/src/pip/_internal/vcs/git.py
--- a/src/pip/_internal/vcs/git.py
+++ b/src/pip/_internal/vcs/git.py
@@ -11,7 +11,7 @@
 from pip._vendor.six.moves.urllib import parse as urllib_parse
 from pip._vendor.six.moves.urllib import request as urllib_request
 
-from pip._internal.exceptions import BadCommand
+from pip._internal.exceptions import BadCommand, InstallationError
 from pip._internal.utils.misc import display_path, hide_url
 from pip._internal.utils.subprocess import make_command
 from pip._internal.utils.temp_dir import TempDirectory
@@ -370,20 +370,25 @@ def update_submodules(cls, location):
         )
 
     @classmethod
-    def controls_location(cls, location):
-        if super(Git, cls).controls_location(location):
-            return True
+    def get_repository_root(cls, location):
+        loc = super(Git, cls).get_repository_root(location)
+        if loc:
+            return loc
         try:
-            r = cls.run_command(['rev-parse'],
-                                cwd=location,
-                                show_stdout=False,
-                                on_returncode='ignore',
-                                log_failed_cmd=False)
-            return not r
+            r = cls.run_command(
+                ['rev-parse', '--show-toplevel'],
+                cwd=location,
+                show_stdout=False,
+                on_returncode='raise',
+                log_failed_cmd=False,
+            )
         except BadCommand:
             logger.debug("could not determine if %s is under git control "
                          "because git is not available", location)
-            return False
+            return None
+        except InstallationError:
+            return None
+        return os.path.normpath(r.rstrip('\r\n'))
 
 
 vcs.register(Git)
diff --git a/src/pip/_internal/vcs/mercurial.py b/src/pip/_internal/vcs/mercurial.py
--- a/src/pip/_internal/vcs/mercurial.py
+++ b/src/pip/_internal/vcs/mercurial.py
@@ -137,19 +137,25 @@ def get_subdirectory(cls, location):
         return find_path_to_setup_from_repo_root(location, repo_root)
 
     @classmethod
-    def controls_location(cls, location):
-        if super(Mercurial, cls).controls_location(location):
-            return True
+    def get_repository_root(cls, location):
+        loc = super(Mercurial, cls).get_repository_root(location)
+        if loc:
+            return loc
         try:
-            cls.run_command(
-                ['identify'],
+            r = cls.run_command(
+                ['root'],
                 cwd=location,
                 show_stdout=False,
                 on_returncode='raise',
-                log_failed_cmd=False)
-            return True
-        except (BadCommand, InstallationError):
-            return False
+                log_failed_cmd=False,
+            )
+        except BadCommand:
+            logger.debug("could not determine if %s is under hg control "
+                         "because hg is not available", location)
+            return None
+        except InstallationError:
+            return None
+        return os.path.normpath(r.rstrip('\r\n'))
 
 
 vcs.register(Mercurial)
diff --git a/src/pip/_internal/vcs/versioncontrol.py b/src/pip/_internal/vcs/versioncontrol.py
--- a/src/pip/_internal/vcs/versioncontrol.py
+++ b/src/pip/_internal/vcs/versioncontrol.py
@@ -232,12 +232,24 @@ def get_backend_for_dir(self, location):
         Return a VersionControl object if a repository of that type is found
         at the given directory.
         """
+        vcs_backends = {}
         for vcs_backend in self._registry.values():
-            if vcs_backend.controls_location(location):
-                logger.debug('Determine that %s uses VCS: %s',
-                             location, vcs_backend.name)
-                return vcs_backend
-        return None
+            repo_path = vcs_backend.get_repository_root(location)
+            if not repo_path:
+                continue
+            logger.debug('Determine that %s uses VCS: %s',
+                         location, vcs_backend.name)
+            vcs_backends[repo_path] = vcs_backend
+
+        if not vcs_backends:
+            return None
+
+        # Choose the VCS in the inner-most directory. Since all repository
+        # roots found here would be either `location` or one of its
+        # parents, the longest path should have the most path components,
+        # i.e. the backend representing the inner-most repository.
+        inner_most_repo_path = max(vcs_backends, key=len)
+        return vcs_backends[inner_most_repo_path]
 
     def get_backend_for_scheme(self, scheme):
         # type: (str) -> Optional[VersionControl]
@@ -687,14 +699,18 @@ def is_repository_directory(cls, path):
         return os.path.exists(os.path.join(path, cls.dirname))
 
     @classmethod
-    def controls_location(cls, location):
-        # type: (str) -> bool
+    def get_repository_root(cls, location):
+        # type: (str) -> Optional[str]
         """
-        Check if a location is controlled by the vcs.
+        Return the "root" (top-level) directory controlled by the vcs,
+        or `None` if the directory is not in any.
+
         It is meant to be overridden to implement smarter detection
         mechanisms for specific vcs.
 
-        This can do more than is_repository_directory() alone.  For example,
-        the Git override checks that Git is actually available.
+        This can do more than is_repository_directory() alone. For
+        example, the Git override checks that Git is actually available.
         """
-        return cls.is_repository_directory(location)
+        if cls.is_repository_directory(location):
+            return location
+        return None


Test patch:
diff --git a/tests/functional/test_freeze.py b/tests/functional/test_freeze.py
--- a/tests/functional/test_freeze.py
+++ b/tests/functional/test_freeze.py
@@ -10,6 +10,7 @@
     _create_test_package,
     _create_test_package_with_srcdir,
     _git_commit,
+    _vcs_add,
     need_bzr,
     need_mercurial,
     need_svn,
@@ -495,6 +496,39 @@ def test_freeze_bazaar_clone(script, tmpdir):
     _check_output(result.stdout, expected)
 
 
+@need_mercurial
+@pytest.mark.git
+@pytest.mark.parametrize(
+    "outer_vcs, inner_vcs",
+    [("hg", "git"), ("git", "hg")],
+)
+def test_freeze_nested_vcs(script, outer_vcs, inner_vcs):
+    """Test VCS can be correctly freezed when resides inside another VCS repo.
+    """
+    # Create Python package.
+    pkg_path = _create_test_package(script, vcs=inner_vcs)
+
+    # Create outer repo to clone into.
+    root_path = script.scratch_path.joinpath("test_freeze_nested_vcs")
+    root_path.mkdir()
+    root_path.joinpath(".hgignore").write_text("src")
+    root_path.joinpath(".gitignore").write_text("src")
+    _vcs_add(script, root_path, outer_vcs)
+
+    # Clone Python package into inner directory and install it.
+    src_path = root_path.joinpath("src")
+    src_path.mkdir()
+    script.run(inner_vcs, "clone", pkg_path, src_path, expect_stderr=True)
+    script.pip("install", "-e", src_path, expect_stderr=True)
+
+    # Check the freeze output recognizes the inner VCS.
+    result = script.pip("freeze", expect_stderr=True)
+    _check_output(
+        result.stdout,
+        "...-e {}+...#egg=version_pkg\n...".format(inner_vcs),
+    )
+
+
 # used by the test_freeze_with_requirement_* tests below
 _freeze_req_opts = textwrap.dedent("""\
     # Unchanged requirements below this line
diff --git a/tests/functional/test_vcs_git.py b/tests/functional/test_vcs_git.py
--- a/tests/functional/test_vcs_git.py
+++ b/tests/functional/test_vcs_git.py
@@ -238,3 +238,15 @@ def test_is_immutable_rev_checkout(script):
     assert not Git().is_immutable_rev_checkout(
         "git+https://g.c/o/r@master", version_pkg_path
     )
+
+
+def test_get_repository_root(script):
+    version_pkg_path = _create_test_package(script)
+    tests_path = version_pkg_path.joinpath("tests")
+    tests_path.mkdir()
+
+    root1 = Git.get_repository_root(version_pkg_path)
+    assert os.path.normcase(root1) == os.path.normcase(version_pkg_path)
+
+    root2 = Git.get_repository_root(version_pkg_path.joinpath("tests"))
+    assert os.path.normcase(root2) == os.path.normcase(version_pkg_path)
diff --git a/tests/functional/test_vcs_mercurial.py b/tests/functional/test_vcs_mercurial.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/test_vcs_mercurial.py
@@ -0,0 +1,17 @@
+import os
+
+from pip._internal.vcs.mercurial import Mercurial
+from tests.lib import _create_test_package, need_mercurial
+
+
+@need_mercurial
+def test_get_repository_root(script):
+    version_pkg_path = _create_test_package(script, vcs="hg")
+    tests_path = version_pkg_path.joinpath("tests")
+    tests_path.mkdir()
+
+    root1 = Mercurial.get_repository_root(version_pkg_path)
+    assert os.path.normcase(root1) == os.path.normcase(version_pkg_path)
+
+    root2 = Mercurial.get_repository_root(version_pkg_path.joinpath("tests"))
+    assert os.path.normcase(root2) == os.path.normcase(version_pkg_path)