Repository: pypa/pip

Issue:
pip freeze messes up egg version with git branches, raising ValueError: 'Expected version spec in'
I have a requirements.txt containing this line:

```
-e git://github.com/mfogel/django-timezone-field#egg=django-timezone-field
```

When dumping pip freeze, it writes this:

```
-e git://github.com/mfogel/django-timezone-field@82a9c16eb7955e7d4f037ececf867d10fd5565ae#egg=django_timezone_field-origin/develop
```

This cannot be installed with pip, it raises:

```
Exception:
Traceback (most recent call last):
  File "lib/python2.7/site-packages/pip/basecommand.py", line 134, in main
    status = self.run(options, args)
  File "lib/python2.7/site-packages/pip/commands/install.py", line 220, in run
    for req in parse_requirements(filename, finder=finder, options=options):
  File "lib/python2.7/site-packages/pip/req.py", line 1454, in parse_requirements
    line, comes_from=comes_from, default_vcs=options.default_vcs if options else None)
  File "lib/python2.7/site-packages/pip/req.py", line 85, in from_editable
    res = cls(name, comes_from, source_dir=source_dir, editable=True, url=url, prereleases=True)
  File "lib/python2.7/site-packages/pip/req.py", line 43, in __init__
    req = pkg_resources.Requirement.parse(req)
  File "lib/python2.7/site-packages/pkg_resources.py", line 2914, in parse
    reqs = list(parse_requirements(s))
  File "lib/python2.7/site-packages/pkg_resources.py", line 2839, in parse_requirements
    line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec")
  File "lib/python2.7/site-packages/pkg_resources.py", line 2807, in scan_list
    raise ValueError("Expected "+item_name+" in",line,"at",line[p:])
ValueError: ('Expected version spec in', 'django_timezone_field-origin/develop', 'at', '/develop')
```

```
$ pip --version
pip 1.4 from ...virtualenv/lib/python2.7/site-packages (python 2.7)
$ easy_install --version
setuptools 0.9.8
```



Discussion:
+1 am getting this issue as well.. exact same versions

Me too. Is it a bug or a new backwards incompatible feature? Pip used to dump these with a "-dev" suffix. Now it seems to use "remote/branch" (or sometimes, just "branch").

+1

+1

+1

+1, same versions. It could be fixed in setuptools by changing the regex `DISTRO   = re.compile(r"\s*((\w|[-.])+)").match` to `DISTRO   = re.compile(r"\s*((\w|[-./])+)").match` in `pkg_resources.py` to allow for slashes. But I guess it would be better if pip didn't put slashes in egg names in the first place.

A quick fix is of course to remove the slashes from your `pip freeze` output.

I also hit this in pip 1.5.

It looks like [this](https://github.com/pypa/pip/blob/6bd791df1d921d66e67b7a9127dc01928526f470/pip/__init__.py#L251) is where pip adds the egg name:

```
req = '%s@%s#egg=%s' % (svn_location, rev, cls.egg_name(dist))
```

The `egg_name` is originally defined [here](https://bitbucket.org/pypa/setuptools/src/6ed8f0298e64223e5ebf179d30783d34524f918d/pkg_resources.py?at=default#cl-2257) in setuptools's pkg_resources module:

```
def egg_name(self):
    """Return what this distribution's standard .egg filename should be"""
    filename = "%s-%s-py%s" % (
        to_filename(self.project_name), to_filename(self.version),
        self.py_version or PY_MAJOR
    )
```

And [here](https://github.com/pypa/pip/blob/6bd791df1d921d66e67b7a9127dc01928526f470/pip/__init__.py#L255) it looks like pip is already stripping off the last `-py...` part of that:

```
def egg_name(dist):
    name = dist.egg_name()
    match = re.search(r'-py\d\.\d$', name)
    if match:
        name = name[:match.start()]
    return name
```

So maybe the fix is as easy as having pip strip off a bit more from the end.

I've just closed #1125 as it's a dupe of this issue.

> Is it a bug or a new backwards incompatible feature? Pip used to dump these with a "-dev" suffix. Now it seems to use "remote/branch" (or sometimes, just "branch").

It looks like this change may have been introduced back in May 2013 in commit fa81a41d0b1577788e24861d57c5ad149234e9c5.  For example, you can see one of the test expectations [changing](https://github.com/pypa/pip/commit/fa81a41d0b1577788e24861d57c5ad149234e9c5#diff-ddd05dd04117fc1972e60567655ec76cL110) in that commit from `-dev` to `-0.1.1`.  And see [this line](https://github.com/pypa/pip/commit/fa81a41d0b1577788e24861d57c5ad149234e9c5#diff-4cb54fa0d0d77f2cf5e44310056fec4aR165):

```
+            full_egg_name = '%s-%s' % (egg_project_name, names_by_commit[current_rev])
```

The commit message sounds like it was meant to be only a refactoring ("Combine Git get_tag_revs and get_branch_revs into single get_refs method."), so I wonder whether this change in functionality was indeed deliberate.

The change in functionality was deliberate in that commit (clearly, since a test was changed), although it's true that the commit message didn't reflect that as it should have. In the course of a refactoring I came across functionality that had been apparently intended by broken/dead code paths in the existing code, and restored it. However, I failed to check the case of a branch name including remote name, and apparently failed to notice that the previous dead code path had tried to handle that case with an removal of an "origin/" prefix (which I think is overly simplistic, since it assumes the only possible remote name is "origin").

I think the right solution here is, when constructing `full_egg_name` from `egg_project_name` and `names_by_commit[current_rev]`, to split `names_by_commit[current_rev]` on slashes and use only the final portion.

Thanks for replying so quickly, @carljm.

> I think the right solution here is, when constructing full_egg_name from egg_project_name and names_by_commit[current_rev], to split names_by_commit[current_rev] on slashes and use only the final portion.

It seems somewhat common to have branch names with a slash in it.  To name one example, [Django](https://github.com/django/django) has a series of branches of the form `stable/1.6.x`.  Can you suggest a solution that accommodates cases like that?

@cjerdonek Short of a change to the regex in setuptools as suggested above (which may have other consequences I'm not aware of, and I'm not sure would be accepted by the setuptools maintainers), it's not possible to maintain such branch names unmodified. This egg version is not functionally important; using anything other than "dev" here is simply an attempt to give a better human-readable clue as to which version of the package is in use. In the case of Django you mention, the resulting egg name would be "Django-1.4.x" if the branch was "stable/1.4.x", which still seems reasonable and more useful than "Django-dev".

Another option would be to translate slashes to some other allowed character (underscore?); it's not clear to me that "Django-origin_stable_1.4.x" is really better than "Django-1.4.x", though. Perhaps there are other cases where it would be preferable, but my guess is that in general a) the remote name is not very useful there, and b) in most cases where branch names with slashes are in use, the final portion is probably adequate as a human-readable hint.

Okay, good to know that the name isn't functionally important.  For readability/usability, my instinct would be to lean towards stripping the remote name off the front and then convert slashes to underscores.  So in the example, it would be ""Django-stable_1.4.x".  Locating the corresponding branch name is more immediate if it at least begins with the same characters (otherwise you have to scan every name for the final segment).

Sure, that seems fine to me. According to @mrmachine above, the remote name is only sometimes included, so in order to implement this approach it would be necessary to figure out when it is or isn't, so the remote name alone can be reliably stripped if present.

By the way, the pip documentation can probably use some clarification in this area.  For example, the [VCS Support](http://www.pip-installer.org/en/latest/logic.html#vcs-support) section says:

> The url suffix `"egg=<project name>"` is used by pip in it’s dependency logic to identify the project prior to pip downloading and analyzing the metadata.

which doesn't leave open the possibility of suffixes of the form `"egg=<project_name>-<project_description>"` or say what the "suffix of the suffix" is for.  [This section](http://www.pip-installer.org/en/latest/usage.html#examples) has a similar issue because all of the examples are of the form:

```
$ pip install -e git+https://git.repo/some_pkg.git#egg=SomePackage          # from git
```

+1

Hi all, is there any fix available for this issue?

On Python > 3.2, it's even worse as the generated egg name is non-deterministic and can vary from run to run - see https://github.com/pypa/pip/issues/1867

+1

+1

I'll start working on this and see if I can make some progress.


Resolving patch:
diff --git a/pip/vcs/__init__.py b/pip/vcs/__init__.py
--- a/pip/vcs/__init__.py
+++ b/pip/vcs/__init__.py
@@ -119,6 +119,12 @@ def cmd(self):
         self._cmd = command
         return command
 
+    # See issue #1083 for why this method was introduced:
+    # https://github.com/pypa/pip/issues/1083
+    def translate_egg_surname(self, surname):
+        # For example, Django has branches of the form "stable/1.7.x".
+        return surname.replace('/', '_')
+
     def get_url_rev(self):
         """
         Returns the correct repository URL and revision by parsing the given
diff --git a/pip/vcs/git.py b/pip/vcs/git.py
--- a/pip/vcs/git.py
+++ b/pip/vcs/git.py
@@ -176,9 +176,10 @@ def get_src_requirement(self, dist, location, find_tags):
                 names_by_commit[commit] = ref
 
         if current_rev in names_by_commit:
-            # It's a tag
+            # It's a tag or branch.
+            name = names_by_commit[current_rev]
             full_egg_name = (
-                '%s-%s' % (egg_project_name, names_by_commit[current_rev])
+                '%s-%s' % (egg_project_name, self.translate_egg_surname(name))
             )
         else:
             full_egg_name = '%s-dev' % egg_project_name


Test patch:
diff --git a/tests/functional/test_freeze.py b/tests/functional/test_freeze.py
--- a/tests/functional/test_freeze.py
+++ b/tests/functional/test_freeze.py
@@ -113,16 +113,17 @@ def test_freeze_git_clone(script, tmpdir):
         'pip-test-package',
         expect_stderr=True,
     )
+    repo_dir = script.scratch_path / 'pip-test-package'
     result = script.run(
         'git',
         'checkout',
         '7d654e66c8fa7149c165ddeffa5b56bc06619458',
-        cwd=script.scratch_path / 'pip-test-package',
+        cwd=repo_dir,
         expect_stderr=True,
     )
     result = script.run(
         'python', 'setup.py', 'develop',
-        cwd=script.scratch_path / 'pip-test-package'
+        cwd=repo_dir
     )
     result = script.pip('freeze', expect_stderr=True)
     expected = textwrap.dedent(
@@ -165,6 +166,31 @@ def test_freeze_git_clone(script, tmpdir):
     ).strip()
     _check_output(result, expected)
 
+    # Check that slashes in branch or tag names are translated.
+    # See also issue #1083: https://github.com/pypa/pip/issues/1083
+    script.run(
+        'git', 'checkout', '-b', 'branch/name/with/slash',
+        cwd=repo_dir,
+        expect_stderr=True,
+    )
+    # Create a new commit to ensure that the commit has only one branch
+    # or tag name associated to it (to avoid the non-determinism reported
+    # in issue #1867).
+    script.run(
+        'git', 'revert', '--no-edit', 'HEAD',
+        cwd=repo_dir,
+    )
+    result = script.pip('freeze', expect_stderr=True)
+    expected = textwrap.dedent(
+        """
+            Script result: ...pip freeze
+            -- stdout: --------------------
+            ...-e ...@...#egg=pip_test_package-branch_name_with_slash...
+            ...
+        """
+    ).strip()
+    _check_output(result, expected)
+
 
 def test_freeze_mercurial_clone(script, tmpdir):
     """
diff --git a/tests/unit/test_vcs.py b/tests/unit/test_vcs.py
--- a/tests/unit/test_vcs.py
+++ b/tests/unit/test_vcs.py
@@ -1,4 +1,5 @@
 from tests.lib import pyversion
+from pip.vcs import VersionControl
 from pip.vcs.bazaar import Bazaar
 from pip.vcs.git import Git
 from mock import Mock
@@ -38,6 +39,13 @@ def test_git_get_src_requirements():
     ])
 
 
+def test_translate_egg_surname():
+    vc = VersionControl()
+    assert vc.translate_egg_surname("foo") == "foo"
+    assert vc.translate_egg_surname("foo/bar") == "foo_bar"
+    assert vc.translate_egg_surname("foo/1.2.3") == "foo_1.2.3"
+
+
 def test_bazaar_simple_urls():
     """
     Test bzr url support.