praekelt/vumi — pull request #689 by justinvdm (2014-01-02 08:12:21+00:00)
Title: Handle bad characters better in MTN NG transport

Many of the packets we receive for the MTN Nigeria transport currently contain characters that are [not allowed in xml](http://www.w3.org/TR/2006/REC-xml-20060816/#charsets) (usually as a result of control characters generated by crappy phones). We end up logging the resulting xml parse error and losing the message.

Two solutions I can think of:
- Add `lxml` as a dependency and make use of the `recover=True` option it offers for parsers. `lxml` then seems to try its best to parse the xml, however broken it may be, instead storing errors encountered (`parser.error_log`). Problem is, it doesn't look like there will be a proper way of distinguishing between the errors we don't want to log (things like `\x0b`) and the parse errors we want to log. This means we can either not log any parse errors at all, or continue getting all the sentry spam we currently get.
- Do our own escaping for these invalid characters, parse the xml, unescape the characters

Thoughts?