pexpect/pexpect — pull request #96 by dcoshea (2014-07-24 12:37:40+00:00)
Title: Issue #84: Unicode support in screen and ANSI.

Please see the commit message for details.

I'm not sure if I'm on the right track at all, so I didn't spend a huge amount of time trying to tidy the code, get the comments exactly right, etc. I'm happy to fix that stuff up if I'm on the right track. Apologies if I'm on the completely wrong track!

The screen and ANSI unit tests pass on Python 2.6, 2.7 and 3.3.

Some issues I'm aware of which I think are fairly minor:

It's not 100% clear to me which methods are meant to be public and
which are only meant for internal use, so I've tried to perform
decoding in all methods that accept input.

There are some cases (e.g. screen.fill(), fill_region(), put(),
insert_abs(), insert()) where decoding is performed by a method that
then invokes another method that also potentially performs decoding.
I thought that this was safer as it meant that future refactoring
wouldn't inadvertently result in decoding not being performed in some
code path.

Use of the incremental decoder from ANSI.write_ch() and
screen.put_abs() technically doesn't make sense, since those methods
expect to receive exactly one character, and if the caller actually
needs an incremental decoder to be used because they don't know where
the character boundaries are in the byte stream, they shouldn't be
calling those methods. However, I've just used the same incremental
decoder in those methods as in all the other ones. I assume those
methods probably wouldn't typically be called directly by outside
code, and if they were, the user would have to have more of an idea of
what they were doing with regards to multi-byte encodings.

With an incremental decoder, we're meant to call it once at the end
with final=True so that any incomplete byte sequence at the end of the
stream can be detected. I don't know of a sensible way to do that -
perhaps screen() should have a close() method added to it that does
this?