swig/swig — pull request #165 by hfalcic (2014-04-22 23:38:20+00:00)
Title: Python 3 byte string output: use errors="surrogateescape"

... if available on the version of Python that's in use. The default
UTF-8 decoding is still performed, but now any bytes that can't be
decoded will be returned as high surrogate characters in the range
U+DC80 - U+DCFF. See PEP 383 for details.

This allows obtaining the original byte string (and potentially trying a
fallback encoding) if the bytes can't be decoded as UTF-8. Previously,
a UnicodeDecodeError would be raised with no way to treat the data as
bytes or try another codec.

This can be simulated purely in Python code with a byte string that
can't be decoded as UTF-8:

>>> bs = b's\xf3me byt\xebs'
>>> bs.decode('utf-8')
Traceback (most recent call last):
File "", line 1, in
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf3 in position 1:
invalid continuation byte
>>> s = bs.decode('utf-8', errors='surrogateescape')
>>> s
's\udcf3me byt\udcebs'

'bs', of course, is internal to the C/C++ module and is not accessible
to Python code. 's' is the text string that would be returned. The
original byte string can be recovered easily:

>>> b = s.encode('utf-8', errors='surrogateescape')
>>> b
b's\xf3me byt\xebs'

The user can then try a fallback encoding if they'd like:

>>> b.decode('latin-1')
'sóme bytës'

Text strings with high surrogate characters will be rejected by most
codecs unless errors is 'surrogateescape' or 'surrogatepass', but this
is still preferable to having no way to obtain the byte string in
Python.