Repository: pypa/pip

Issue:
split_leading_dir falls over installing the latest zipball of Django
```
> pip --version
pip 6.1.1
> pip install https://github.com/django/django/archive/master.zip
Collecting https://github.com/django/django/archive/master.zip
[... Download completes OK ...]
Exception:
  Traceback (most recent call last):
    File "lib/python2.7/site-packages/pip/basecommand.py", line 246, in main
      status = self.run(options, args)
    File "lib/python2.7/site-packages/pip/commands/install.py", line 342, in run
      requirement_set.prepare_files(finder)
    File "lib/python2.7/site-packages/pip/req/req_set.py", line 345, in prepare_files
      functools.partial(self._prepare_file, finder))
    File "lib/python2.7/site-packages/pip/req/req_set.py", line 290, in _walk_req_to_install
      more_reqs = handler(req_to_install)
    File "lib/python2.7/site-packages/pip/req/req_set.py", line 487, in _prepare_file
      download_dir, do_download, session=self.session,
    File "lib/python2.7/site-packages/pip/download.py", line 827, in unpack_url
      session,
    File "lib/python2.7/site-packages/pip/download.py", line 677, in unpack_http_url
      unpack_file(from_path, location, content_type, link)
    File "lib/python2.7/site-packages/pip/utils/__init__.py", line 630, in unpack_file
      flatten=not filename.endswith('.whl')
    File "lib/python2.7/site-packages/pip/utils/__init__.py", line 510, in unzip_file
      leading = has_leading_dir(zip.namelist()) and flatten
    File "lib/python2.7/site-packages/pip/utils/__init__.py", line 224, in has_leading_dir
      prefix, rest = split_leading_dir(path)
    File "lib/python2.7/site-packages/pip/utils/__init__.py", line 208, in split_leading_dir
      path = str(path)
  UnicodeEncodeError: 'ascii' codec can't encode character u'\u2297' in position 60: ordinal not in range(128)
```

the path it ultimately falls down on is:

```
path = u'django-master/tests/staticfiles_tests/apps/test/static/test/\u2297.txt'
prefix, rest = split_leading_dir(path)
```



Discussion:
Similar issue: https://github.com/pypa/pip/issues/2388

This seems to be the file in question:
testing: django-master/tests/staticfiles_tests/apps/test/static/test/#U2297.txt

However I'm worried that this is going to be a no-right-way situation: http://stackoverflow.com/questions/19997312/german-letters-and-encoding-in-c-sharp/19997517#19997517 - there is no global specification for unicode in zip, only various heuristics that depend entirely on the compressor in use. There's no reason to think that whats happening here - the cast back to unicode - is standard. For instance, 'unzip master.zip' gives me a file on disk called `#2297`, not a unicode filename. The test in Django using this does:
    def test_nonascii_filenames(self):
        """
        Can find a file with non-ASCII character in an app static/ directory.
        """
        self.assertFileContains('test/<E2><8A><97>.txt', '<E2><8A><97> in the app dir')

(It uses from **future** import unicode_literals).

I suggest that this should be considered a Django bug: such files are problematic for users too because of different fs encoding on different platforms. The test would be more portable and safer if it made the file on demand.

Just a data point here, working on https://github.com/django-compressor/django-appconf/pull/23, switching from the `zip` to `tar.gz` archive formats works around this issue. 


Resolving patch:
diff --git a/pip/utils/__init__.py b/pip/utils/__init__.py
--- a/pip/utils/__init__.py
+++ b/pip/utils/__init__.py
@@ -221,7 +221,6 @@ def file_contents(filename):
 
 
 def split_leading_dir(path):
-    path = str(path)
     path = path.lstrip('/').lstrip('\\')
     if '/' in path and (('\\' in path and path.find('/') < path.find('\\')) or
                         '\\' not in path):


Test patch: