matplotlib/matplotlib — pull request #2433 by mdboom (2013-09-18 14:21:24+00:00)
Title: Handle Unicode font filenames correctly

This is an alternative to #2431.

First, when handling the output of fontconfig (fc-match and fc-list), the bulk of the output is in ascii, so it is maintained, parsed and handled as bytes. Only once the filename part of the output is extracted do we decode it using sys.getfilesystemencoding().

Behind that bug, if you actually try to use a font file with non-ascii characters in the path:

1) Every time we passed the path to FT2Font, we were calling str() on it. This was required when FT2Font did not handle Unicode paths, prior to f4adec. However, then and now, it just serves to raise an exception when provided with a non-ascii path.

2) Within FT2Font, that unicode string was encoded for the purpose of displaying error messages. This could fail both on encoding and displaying it. Encoding it using "unicode_escape" ensures that we'll have something printable in any environment.

3) ttconv does not handle Unicode filenames. ttconv is in the process of being eliminated for MEP14, so I didn't want to invest a lot of effort fixing that. Instead, since it is only called from two places, the path is encoded back to filesystemencoding on the Python side before calling into it. This doesn't change the API to ttconv.

This will require a manual backport to 1.3.x if we want that -- things are different enough there that this won't really work or apply.

This approach is not necessarily "best practice" for POSIX platforms, which is to keep paths as byte strings everywhere. I tried that first, but there are too many assumptions throughout the code that file paths are unicode (particularly post f4adec with the move to six and `from __future__ import unicode_literals`). The downside of that is pretty minor -- if a path contains a sequence that does not decode using the filesystemencoding, such as an invalid utf-8 surrogate pair, it will not be openable by matplotlib. I'd consider that a pathological, even if technically legal, case, however, so I don't think we should worry too much about it. Keeping the paths as unicode objects does make the processing of them much easier, particularly with `six` and Python 3.