piranha/opster — pull request #38 by oscarbenjamin (2012-05-22 19:19:38+00:00)
Title: Use 'surrogateescape' to decode input arguments

Ok, so I've found the fix for input argument decoding. The problem is described in pep-383 that also provides the solution:
http://www.python.org/dev/peps/pep-0383/

This patch fixes the tests under python 2.6 and 3.1.

In python3 `sys.argv` needs to be a list of unicode strings. When the python interpreter starts up, python takes the raw bytes arguments provided to the interpreter and decodes them using a system-dependent encoding. If I understand correctly, then the encoding used is always available as `sys.getfilesystemencoding()`. I think the reason for using the filesystem encoding is that the most common problem caused by incorrect argument encoding is incorrect representation of file paths. The reason the tests failed for me was because on windows the filesystem encoding is not utf-8.

The input arguments, given as raw bytes in c, may contain characters that don't exist in the encoding that python uses to decode them. So python uses the `surrogate escape` feature described in pep-383 to decode the arguments in a way that can at least be inverted. The raw argument as a bytes object can be obtained using:

raw_arg = arg.encode(sys.getfilesystemencoding(), 'surrogateescape')

We wanted to decode the argument as `desired_encoding`, so then we do

arg = arg.decode(desired_encoding)

In python 2.x only the second step is necessary since the interpreter will give us the raw bytes as a `str`.

This is pretty confusing and it would be good if opster can save its users from ever having to know about it, by doing it automatically: opster could just do this to all arguments all the time. However, I don't think that would work for arguments that had never been decoded in this way. For example under python 3.x, if someone generates a list of arguments in python and passes them to ```main.command``` opster cannot know if that list of arguments was generated from the entries in `sys.argv` or was generated programmatically:

If someone calls main.command or d.dispatch with no arguments opster knows that it is getting the arguments from sys.argv and can do the surrogateescape fix. However opster doesn't know what encoding the script writer really wanted to use (if not the filesystem encoding):

main.command()
d.dispatch()

If someone passes a list of arguments in, opster cannot presume to be able to get anything sensible by encoding the arguments and then decoding them as a different encoding. Probably opster should just leave them

main.command(['my', 'list', 'of', 'args'])

But then how can opster spot this:

main.command(sys.argv[1:])

If opster will automatically decode the arguments then there should be a documented way to tell opster what encoding you want to decode them as

opster.ENCODING = 'cp850'

Perhaps the simplest thing is for opster to just provide an API function similar to `decodearg` in this patch. Then it can be used like in `hello.py` in this patch.