Add rake tasks for checking the parsing of a random sample of emails

For the upgrade from Rails 2 to Rails 3, we want to check that
existing emails are parsed correctly.  One of these tasks
(temp:random_attachments_hexdigests) is for dumping a CVS file
of details about each attachment, and its hexdigest, and a list
of the randomly selected raw emails.  (It's intended that you
run this on an old Rails 2 install of Alaveteli.  The other task
(temp:recompute_attachments_hexdigests) is intended to be run
on a Rails 3 install of Alaveteli, with the previous files as
input, to check that the same results are obtained.
Add tests for TNEF attachments that should be handled

These two cases were ignored previously, and we need to make sure
that they still are under the switch from TMail to Mail.

One TNEF attachment is a heavily truncated one from a real example
from Alaveteli that has no personal data in it.  The other is
an example from the tests in the distribution of the tnef
package for Ubuntu 1.4.9-1 - it's an HTML version of the US
constitution.
Remove an unnecessary use of 'which' in backticks

The use of backticks in `which tnef` means that which(1) is
invoked under /bin/sh, and returns the first executable
matching 'tnef' in the default PATH for /bin/sh.  In this
usage, however, Popen also uses /bin/sh to execute the
command, which would run that same executable if just a
bare 'tnef' were used.  In summary, I can't see any reason
for the convolution of:

   IO.popen("#{`which tnef`.chomp} -K -C #{dir}", "wb") do |f|

... over just:

   IO.popen("tnef -K -C #{dir}", "wb") do |f|

... so switch to the latter.
Add a test to check that anything in the MIME epilogue is ignored

There is currently a difference in behaviour in the parsing
of nested MIME multipart attachments between the Mail and
TMail based backends.  This commit adds a test that will
pass if the behaviour is the same as the the old (TMail-based)
version, which I believe is correct according to RFC 1521.
The example email has a PNG attachment after the final MIME
boundary, and the RFC says that anything after the final
boundary ("the epilogue") should be ignored.
Add a test for a missing final MIME boundary

If there is a missing final MIME boundary, the behaviour
of Alaveteli with the TMail backend was to still parse
the attachment, but with the new code it currently throws
an exception.  This commit adds a test that asserts that
the attachment should be parsed despite the email being
malformed in this way.
Add a test for parsing a malformed email

This example email indicates the wrong charset and includes
a top bit set character despite Content-Transfer-Encoding: 7bit
- nonetheless, we should be able to convert it to UTF-8 and
interpret the character correctly.
Ignore common TNEF attachment parsing errors

This also introduces a custom error class so that we don't
accidentally catch other problems.

Fixes #920
Fix a syntax error
Switch to just loading fixtures in tests as binary strings

The data loaded with load_file_fixture is generally used
in the tests as if it were binary data received from the
MTA, so just load it as binary data - in Ruby 1.9 this means
that it will end up with the encoding ASCII-8BIT.

The same is the case when reading data from a fixture file
in receive_incoming_mail.
Add a new example for unconvertible-to-UTF-8 text part data

The example file that was used for checking for text
attachment data which could not be converted to UTF-8 is
one that we *can* actually deal with by guessing the
character set, since it's valid GB18030.  So, this commit
changes that test to check for the first few Chinese
characters in that email, and introduces a replacement
test with text from /dev/random, which should not be
interpretable in any sensible way.
Add functions for converting from arbitrary text data to UTF-8

Throughout the codebase it is simplest and most consistent
if we could assume that all text/* attachments are represented
by UTF-8 strings, and this was largely true with the TMail
backend which ensured that all returned text parts were in
UTF-8.  We have to change the replacement Mail-backed to
similarly attempt to convert text parts to UTF-8.  This commit
introduces two functions which are useful for this.

The normalize_string_to_utf8 function will try various
encodings, either suggested or guessed (with charlock_holmes)
to convert the passed string to UTF-8, and if it can't find a
suitable encoding will throw an exception.

Unfortunately, the current behaviour of the site is that
uninterpretable text/* attachments are still passed around and
mangled to UTF-8 just before display.  To mimic this it's also
useful to have the convert_string_to_utf8_or_binary function,
which tries to convert the string to UTF-8 with
normalize_string_to_utf8, but if that's not possible just
returns the original string.  (In Ruby 1.9, encoding will be
set to UTF-8 or ASCII-8BIT appropriately.)
Make efforts to ensure that we're usually dealing with UTF-8 strings

One of these changes is to make sure that the Mail backend,
like the TMail backend it replaces, will return text parts
encoded in UTF-8 if possible.

The other change is to ensure that when text attachments are
reloaded from disk, we attempt to convert them to UTF-8.
Add a helper function for dumping text to disk

This function is useful for investigating problems with
handling of emails, attachments and the related character
encoding issues.  It can safely be removed later, but is
currently useful to have for debugging purposes.
Add another mail parsing test

At one point in development this email was misparsed, so I've
added this as test to check for regressions.
Mark as "pending" two tests relating to odd MIME boundary cases

These cases are rare, and probably need to be resolved by
reporting issues against the Mail gem (although it's debatable
what the more correct or pragmatic behaviour should be in both
cases).
Add failing test for case where a mail part has a content-type header with no charset field.
Only set original_charset when a charset has been defined for the mail part. Fixes #942.
Merge branch 'tmail-to-mail-nil-part' into tmail-to-mail-tests
Move "require 'alaveteli_file_types'" into the initializer
Avoid an FoiAttachment validation failure under Rails 3

Under Rails 3, the uudecoded FoiAttachment in this test
fails validation at the self.save! in
IncomingMessage.parse_raw_email, although the FoiAttachment
has been correctly created and saved to the database in
_uudecode_and_save_attachments.  Forcing a reload=true on
self.foi_attachments fixes this.

Thanks to Louise Crow for finding the fix for this problem.