pypa/pip — pull request #1473 by daniel-kristjansson (2014-01-14 19:06:44+00:00)
Title: Handle UTF-8 characters in filenames coming from tar.

When location comes into untar_file() as a unicode string and member.name is not in the
ASCII subset of UTF-8 then os.path.join() raises a UnicodeDecodeError in python 2.7.3

The problem is that the python 2.x default string encoding is 'ascii' but the string
coming from tarfile is actually utf-8
(per http://at4j.sourceforge.net/releases/current/pg/ch08.xhtml POSIX 2001 it is
possible for it to be encoded with some other encoding, but this would be unwise
in a distributed tar file.)

The most proper place to do the decoding appears to be as soon as we first assign the
string to fn, this means we need to remove the path = str(path) in split_leading_dir().
This line appears to serve no purpose, but it looks like something one might write for
None safety, in which case it could be replaced with another construct such as
if type(path) == type(None):
path = ''"

Note: The problem can be reproduced by installing a package that requires another
package containing a file with non-ascii characters. In our case the problem package
was pyramid 1.3.3 which contains a unicode test file. Installing pyramid 1.3.3 directly
worked, but installing a package that required pyramid did not.