Re-indent _EncodeLOB and _DecodeLOB
Respect the database Content-Type header in decoding textual parts

The definition of "texual" data has changed over time.  Specifically,
7365c08 caused text/html messages to begin being stored as utf-8 in the
database; prior to that, the claimed "charset" and bytes in the body
were left unmolested during insertion.

text/html attachments inserted into the database prior to 7365c08,
however, are now expected to be utf-8 when being extracted from the
database.  This causes PERLQQ'd garbage to be displayed for the non-UTF8
content stored in the database.  This type of error is likely to also
re-occur in the future whenever the definition of "textual" data
(i.e. data we transcode on insertion) changes.

Respect the Content-Type header when decoding data from the database, or
guess its value from the body; this mirrors the logic in
RT::I18N::SetMIMEEntityToEncoding, which is what is done for
currently-detected-as-textual parts on insert.  In cases like text/html
prior to 7365c08, the Content-Type header was not altered during
database insertion -- and at worst, the claimed character set is
incorrect and decoding will result in PERLQQ'd garbage.  This is no
worse than said message were detected, received, converted, and stored
in the database as text.
Stop needlessly frobbing utf8 internals

bf87c784 swapped in this explicit _utf8_off for what had been a
_utf8_on.  As the values are fetched with decode_utf8 => 0, $content
contains bytes, not characters.  Explicitly turning of fthe utf8 bit is
unnecessary, as they are already bytes.
Decoding content, and returning characters, is incorrect

The whole purpose of this function is to return bytes, not characters.
Decoding the bytes to characters as utf8 defeats the purpose, and only
works because RT's internal representation of characters happens to be
utf8, which gets used by MIME::Entity to later encode the characters.

Switch to simply returning the bytes, as-is.
Stop assuming the data in the database is utf8

As noted in 802dc8d, not all content we currently call "textual" was
always treated as such.  When re-encoding, do not assume that the
encoding in the database is UTF-8 -- rather, read the Content-Type
header, and examine the charset stated there.  Convert from that to the
original encoding.
Modernize and condense t/mail/sendmail.t

t/data/emails/text-html-in-russian was removed because the original
purpose of the test was removed in 46fd04d9, after 90f9c190 stopped
attaching text/html incoming mail to autoreplies.
Always log bytes, not characters

Ensure that we always send UTF-8 encoded bytes to loggers, and not wide
characters.  This is correctly done via an explicit call to
Encode::encode, and not via checks of utf8::is_utf8 (which may be false
for character strings with codepoints > 127 but < 256), and not via
_utf8_off (which would fail similarly for such characters).
The alluded-to deficiency is not a concern in perl ≥ 5.8.3

The comment and code were added in RT 2.1.38, which only required perl
5.6.1; the perl version was increased to 5.8.3 to cover a large number
of encoding bugs, such as the one this comment was likely alluding to.
Ensure all MIME::Entity bodies are UTF-8 encoded bytes

Placing wide characters into MIME::Entity objects can lead to
double-encoding.  Always treat them as byte stores, encoding as UTF-8
and noting their character set.

In the case of Approvals/index.html, there was no need for an explicit
MIME::Entity object; ->Correspond creates one as needed from a "Content"
argument.
Ensure all MIME::Entity headers are UTF-8 encoded bytes

Placing wide characters into MIME::Entity objects can lead to
double-encoding, as discovered most recently in d469cacc.  Explicitly
decode all headers as UTF-8 when retrieving them with ->get(), and
encode them as UTF-8 before updating them with ->set() or ->replace().
This also applies to headers passed to ->build().  The only exceptions
to this are fixed strings in the source (which, in the absence of "use
utf8", are always bytes).

While the majority of these headers will never have wide characters in
them, always decoding and encoding ensures the proper disipline to
guarantee that strings with the "UTF8" flag do not get placed in a
header, which can cause double-encoding.
Make RT::Action::SendEmail->SetHeader take characters, not bytes

This helper method is used in a number of places in
RT::Action::SendEmail, often without remembering that it should be
passed bytes, not characters.  Change it to always take characters, and
modify the two callsite which (correctly) passed it bytes to no longer
do so.
Add a utility method to check that an input is bytes

Note that it is impossible to verify that an input is characters; here,
we can only validate if it _could_ be bytes.

First, any string with the "UTF8" flag off cannot contain codepoints
above 255, and as such is safe.  Additionally, if the "UTF8" flag is on,
having no codepoints above 127 means the bytes are unambigious.  Having
codepoints above 255 is guaranteedly a sign that the input is not a byte
string.

This leaves only the case of a string with the "UTF8" flag on, and
codepoints above 127 but below 255.  The "UTF8" flag is a sign that they
were _likely_ touched by character data at some point.  In such cases we
warn, suggesting that the bytes have the "UTF8" flag disabled by means
of utf8::downgrade, if they are indeed bytes.
Verify that MIME::Entity bodies are bytes, and remove _utf8_off call

Use the newly-added RT::Util::assert_bytes function to verify that the
body is indeed bytes, and not characters.

We also remove the _utf8_off call -- because, contrary to what the
comment implies, the presence or absence of the "UTF8" flag does _not_
determine if a string is "encoded as octets and not as characters"; it
merely states that the string is capable of holding codepoints > 255.
If it happens to not contain any, the _utf8_off does nothing.  If it
does, it effectively encodes all codepoints > 127 in UTF-8.

Given the premise that the string contains bytes in some (probably
non-UTF-8) encoding, re-encoding some bytes of it as UTF-8 cannot
possibly produce valid output.  The flaw in this situation cannot be
fixed by a simple _utf8_off, but instead must be fixed by ensuring that
the body always contains bytes, not wide characters -- as it now does,
thanks to the prior commits.  The call to RT::Util::assert_bytes serves
as an additional safeguard against backsliding on that assumption.
Verify that MIME::Entity headers are bytes, and remove _utf8_off call

See the prior commit for reasoning, which applies just as much to the
header as the body.
Standardize on the stricter Encode::encode("UTF-8", ...) everywhere

This is not only for code consistency, but also for consistency of
output.  Encode::encode_utf8(...) is equivalent to
Encode::encode("utf8",...) which is the non-"strict" form of UTF-8.
Strict UTF-8 encoding differs in that (from `perldoc Encode`):

    ...its range is much narrower (0 ..  0x10_FFFF to cover only 21 bits
    instead of 32 or 64 bits) and some sequences are not allowed, like
    those used in surrogate pairs, the 31 non-character code points
    0xFDD0 .. 0xFDEF, the last two code points in any plane (0xXX_FFFE
    and 0xXX_FFFF), all non-shortest encodings, etc.

RT deals with interchange with databases, email, and other systems.  In
dealing with encodings, it should ensure that it does not produce byte
sequences that are invalid according to official Unicode standards.
Remove "use utf8" from RT::I18N::fr, making NBSP explicit

"use utf8" causes the sourcecode (including all strings) to be
interpreted by perl as characters encoded in UTF-8, not bytes.  In
lib/RT/I18N/fr.pm, this was being used to substitude codepoint 160
(NO-BREAK SPACE, U+00A0) for commas.  The fact that the space character
was not 0x20, but rather 0xA0, was mostly hidden by use of "use utf8".

Remove the "use utf8" and make the replacement character clear.
Remove remaining cases of "use utf8"

All remaining cases of "use utf8" lie in the testsuite.  As "use utf8"
changes the semantics of dealing with Unicode strings, remove it to
allow programmers to always assume that literals are interpreted as
bytestrings, not characters.  To do otherwise means that one must always
ask if "use utf8" is in scope before performing operations on any
literals; instead, simply make the encodings and decodings explicit.

Note that wide characters may appear in editors, and that the encoding
of the characters on _disk_ will always be UTF-8.  The removal of "use
utf8" merely means that perl will generate a two-byte string from "é",
and not a one-character string.
Dashboard: decode bytes in query parameters into characters

The dashboard mailer code replicates many of the original steps of Mason
parameter parsing -- but omitted the important step of decoding the
bytes once they had been un-URI-encoded.
Tests: WWW::Mechanize correctly returns characters now

While there ay have been bugs surrounding encodings in LWP or
WWW::Mechanize previously, the ->content method correctly returns
characters and not bytes for all modern versions.  Remove the explicit
adjustment.
_utf8_on in EncodeToMIME is needless and incorrect; remove it

66930fd8 switched from an explicit _utf8_off to an explicit _utf8_on, in
an attempt to switch from splitting on bytes to splitting on characters.
However, the "UTF8" flag does not magically determine if a string is
bytes or characters.  Instead, only consistency in calling convention
can do so.  All callsites of RT::Interface::Email::EncodeToMIME and
RT::Action::SendEmail::MIMEEncodeString now pass character strings; all
that _utf8_on can do is incorrectly "decode" those strings as UTF-8 if
they happen to not have the "UTF8" flag set.