4teamwork/ftw.mail — pull request #22 by lukasgraf (2014-08-19 14:36:02+00:00)
Title: SearchableTextExtender: Properly decode payloads in multipart messages  

Background:

When composing a mail containing non-ASCII characters, first the content needs to be encoded using a charset like `utf-8` or `latin-1`. This encoding is declared as a parameter for the `Content-Type` header (either in the message headers, or in the part headers of a multipart message:

`Content-Type: text/plain; charset="iso8859-1"`

This will make sure all unicode characters are correctly encoded to an 8-bit bytestream. However, SMTP only allows 7-bit ASCII, so before sending the message, the encoded bytes need to be encoded once again to comply with the 7-bit restriction: This encoding layer is the `Content-Transfer-Encoding`, and it is either quoted printable or base64.

In the change introduced in PR #14 only the `Content-Transfer-Encoding` layer gets decoded, but the actual payload contents are still encoded in whatever character sets the sender used. So that charset happens to be `utf-8` it works, but otherwise the `SearchableText` will be mangled if a non-UTF8 string is passed to the catalog for indexing.

So this pull-request reads out the `content_charset`, and, if present, decodes the payload content using that charset. If no `content_charset` is given, that part of a multipart message usually only contains headers, no body. In that case, we'll assume `utf-8` and decode using `errors='ignore'`.

Also see [this talk](https://www.youtube.com/watch?v=JENdgiAPD6c) for the two different encoding mechanisms.