Repository: thunderbird/thunderbird-android (language: kotlin)

Issue:
`◆?` charcter appears in the subject when it has multiple MIME lines
**Describe the bug**

Graphical character (`◆` + `?`)  appears in the subject of some mails written in Japanese language.

**To Reproduce**

Post a mail with subject like:
```
Subject: =?ISO-2022-JP?B?GyRCRnxLXDhsJEhGfEtcOGwkSEZ8S1w4bCROJUElJyVDGyhC?=
 =?ISO-2022-JP?B?GyRCJS8bKEI=?=
```

Current K-9 shows `日本語と日本語と日本語のチェッ◆ク` on both tray list mode and mail content mode.

**Expected behavior**
Though the example subject has a newline, this should be parsed as one-line
'日本語と日本語と日本語のチェック' 
by MIME decoding rule for subject. It looks current K-9 simply trys to show (unexpected) newline char.

**Screenshots**
Tray mode
![Screenshot_20211030-123812](https://user-images.githubusercontent.com/183523/139519549-1d4877f9-0e52-44a2-a251-514e646b25b3.png)

Content mode has same problem also
![Screenshot_20211030-123822](https://user-images.githubusercontent.com/183523/139519552-ceeaa59d-a253-4c0f-a836-4781d0e62d59.png)

**Environment (please complete the following information):**
 - K-9 Mail version: 5.904
 - Android version: 12
 - Device: Google Pixel 6
 - Account type: IMAP

**Additional context**
It seems same as #3622.
I'm not sure what was done by 'Fixed in master'. (Because I hadn't used K-9 mail since 2019, I didn't know it was fixed at that time.)



Discussion hints:
Works for me when copying the text from your post. It's possible the actual email contains additional invisible characters. Can you attach the unmodified source of such a message?
Sure, atached.
[issue5760-mail.zip](https://github.com/k9mail/k-9/files/7447799/issue5760-mail.zip)

Apologies. Decoding the subject worked fine when running the code on my computer. However, when running on an Android device I can reproduce the issue you're seeing.

The problem seems to be the change we made to support improperly encoded subjects (PR #2725). With the change we strip the Q- or B-encoding from the segments and then perform the character set decoding on the concatenated bytes. 
ISO-2022-JP uses escape sequences to switch character sets. At the end of an encoded word (segment) the encoder has to insert an escape sequence to switch back to ASCII. So, when concatenated, the data reads like this:

[switch to JIS X 0208:1983] [some characters] [switch to ASCII] [switch to JIS X 0208:1983] [some characters] [switch to ASCII]

The switch to ASCII and then back to JIS X 0208:1983 is unnecessary. The charset decoder on the JVM doesn't care and decodes the data as expected. However, the decoder on Android does mind and inserts a replacement character �.

The proper way of decoding the data is to completely decode the individual segments and then concatenate the decoded text. However, many email clients (including old versions of K-9 Mail) mess up the encoding and require the decoding we added with PR #2725.

We'll have to figure out a way to pick which decoding method to select. Maybe assume the text is properly encoded and use the "one segment at a time" approach. And when that leads to a replacement character being present in the output, try the "combine segments, then decode" method.

Side note: If you have any control over the creation of such a message, please use UTF-8 instead of ISO-2022-JP.
Thank you for your speedy observation!
I hope it will be fixed in next release.

> Side note: If you have any control over the creation of such a message, please use UTF-8 instead of ISO-2022-JP.

Well,ISO-2022-JP is still common encoding for Japanese e-mail.
I myself can change the encoding of subject to UTF-8 by modifying my MUA, but can't change sender's MUA or system. :cry: 

Pull request: Properly decode multiple encoded-words using ISO-2022-JP
Add special support for ISO-2022-JP when decoding encoded-words.

Fixes #5760

Test command: ./gradlew :mail:common:testDebugUnitTest --tests "com.fsck.k9.mail.internet.DecoderUtilTest"

Fix patch:
diff --git a/mail/common/build.gradle b/mail/common/build.gradle
index 5711d6feb01..0e4c627e45e 100644
--- a/mail/common/build.gradle
+++ b/mail/common/build.gradle
@@ -19,6 +19,7 @@ dependencies {
     testImplementation "com.google.truth:truth:${versions.truth}"
     testImplementation "org.mockito:mockito-core:${versions.mockito}"
     testImplementation "org.mockito.kotlin:mockito-kotlin:${versions.mockitoKotlin}"
+    testImplementation "com.ibm.icu:icu4j-charset:70.1"
 }
 
 android {
diff --git a/mail/common/src/main/java/com/fsck/k9/mail/internet/DecoderUtil.java b/mail/common/src/main/java/com/fsck/k9/mail/internet/DecoderUtil.java
deleted file mode 100644
index 361752a6165..00000000000
--- a/mail/common/src/main/java/com/fsck/k9/mail/internet/DecoderUtil.java
+++ /dev/null
@@ -1,217 +0,0 @@
-
-package com.fsck.k9.mail.internet;
-
-
-import java.io.ByteArrayInputStream;
-import java.io.IOException;
-import java.io.InputStream;
-import java.nio.charset.Charset;
-
-import com.fsck.k9.mail.Message;
-import com.fsck.k9.mail.MessagingException;
-import okio.Buffer;
-import okio.ByteString;
-import okio.Okio;
-import org.apache.james.mime4j.codec.QuotedPrintableInputStream;
-import org.apache.james.mime4j.util.CharsetUtil;
-import timber.log.Timber;
-
-
-/**
- * Static methods for decoding strings, byte arrays and encoded words.
- *
- * This class is copied from the org.apache.james.mime4j.decoder.DecoderUtil class.  It's modified here in order to
- * decode emoji characters in the Subject headers.  The method to decode emoji depends on the MimeMessage class because
- * it has to be determined with the sender address, the mailer and so on.
- */
-class DecoderUtil {
-    /**
-     * Decodes a string containing encoded words as defined by RFC 2047.
-     * Encoded words in have the form
-     * =?charset?enc?Encoded word?= where enc is either 'Q' or 'q' for
-     * quoted-printable and 'B' or 'b' for Base64.
-     *
-     * ANDROID:  COPIED FROM A NEWER VERSION OF MIME4J
-     *
-     * @param body the string to decode.
-     * @param message the message which has the string.
-     * @return the decoded string.
-     */
-    public static String decodeEncodedWords(String body, Message message) {
-
-        // ANDROID:  Most strings will not include "=?" so a quick test can prevent unneeded
-        // object creation.  This could also be handled via lazy creation of the StringBuilder.
-        if (!body.contains("=?")) {
-            return body;
-        }
-
-        EncodedWord previousWord = null;
-        int previousEnd = 0;
-
-        StringBuilder sb = new StringBuilder();
-
-        while (true) {
-            int begin = body.indexOf("=?", previousEnd);
-            if (begin == -1) {
-                decodePreviousAndAppendSuffix(sb, previousWord, body, previousEnd);
-                return sb.toString();
-            }
-
-            // ANDROID:  The mime4j original version has an error here.  It gets confused if
-            // the encoded string begins with an '=' (just after "?Q?").  This patch seeks forward
-            // to find the two '?' in the "header", before looking for the final "?=".
-            int qm1 = body.indexOf('?', begin + 2);
-            if (qm1 == -1) {
-                decodePreviousAndAppendSuffix(sb, previousWord, body, previousEnd);
-                return sb.toString();
-            }
-
-            int qm2 = body.indexOf('?', qm1 + 1);
-            if (qm2 == -1) {
-                decodePreviousAndAppendSuffix(sb, previousWord, body, previousEnd);
-                return sb.toString();
-            }
-
-            int end = body.indexOf("?=", qm2 + 1);
-            if (end == -1) {
-                decodePreviousAndAppendSuffix(sb, previousWord, body, previousEnd);
-                return sb.toString();
-            }
-            end += 2;
-
-            String sep = body.substring(previousEnd, begin);
-
-            EncodedWord word = extractEncodedWord(body, begin, end, message);
-
-            if (previousWord == null) {
-                sb.append(sep);
-                if (word == null) {
-                    sb.append(body, begin, end);
-                }
-            } else {
-                if (word == null) {
-                    sb.append(charsetDecode(previousWord));
-                    sb.append(sep);
-                    sb.append(body, begin, end);
-                } else {
-                    if (!CharsetUtil.isWhitespace(sep)) {
-                        sb.append(charsetDecode(previousWord));
-                        sb.append(sep);
-                    } else if (previousWord.encoding.equals(word.encoding) &&
-                            previousWord.charset.equals(word.charset)) {
-                        word.data = concat(previousWord.data, word.data);
-                    } else {
-                        sb.append(charsetDecode(previousWord));
-                    }
-                }
-            }
-
-            previousWord = word;
-            previousEnd = end;
-        }
-    }
-
-    private static void decodePreviousAndAppendSuffix(StringBuilder sb, EncodedWord previousWord, String body,
-            int previousEnd) {
-
-        if (previousWord != null) {
-            sb.append(charsetDecode(previousWord));
-        }
-
-        sb.append(body.substring(previousEnd));
-    }
-
-    private static String charsetDecode(EncodedWord word) {
-        try {
-            InputStream inputStream = new Buffer().write(word.data).inputStream();
-            return CharsetSupport.readToString(inputStream, word.charset);
-        } catch (IOException e) {
-            return null;
-        }
-    }
-
-    private static EncodedWord extractEncodedWord(String body, int begin, int end, Message message) {
-        int qm1 = body.indexOf('?', begin + 2);
-        if (qm1 == end - 2)
-            return null;
-
-        int qm2 = body.indexOf('?', qm1 + 1);
-        if (qm2 == end - 2)
-            return null;
-
-        // Extract charset, skipping language information if present (example: =?utf-8*en?Q?Text?=)
-        String charsetPart = body.substring(begin + 2, qm1);
-        int languageSuffixStart = charsetPart.indexOf('*');
-        boolean languageSuffixFound = languageSuffixStart != -1;
-        String mimeCharset = languageSuffixFound ? charsetPart.substring(0, languageSuffixStart) : charsetPart;
-
-        String encoding = body.substring(qm1 + 1, qm2);
-        String encodedText = body.substring(qm2 + 1, end - 2);
-
-        String charset;
-        try {
-            charset = CharsetSupport.fixupCharset(mimeCharset, message);
-        } catch (MessagingException e) {
-            return null;
-        }
-
-        if (encodedText.isEmpty()) {
-            Timber.w("Missing encoded text in encoded word: '%s'", body.substring(begin, end));
-            return null;
-        }
-
-        EncodedWord encodedWord = new EncodedWord();
-        encodedWord.charset = charset;
-        if (encoding.equalsIgnoreCase("Q")) {
-            encodedWord.encoding = "Q";
-            encodedWord.data = decodeQ(encodedText);
-        } else if (encoding.equalsIgnoreCase("B")) {
-            encodedWord.encoding = "B";
-            encodedWord.data = decodeB(encodedText);
-        } else {
-            Timber.w("Warning: Unknown encoding in encoded word '%s'", body.substring(begin, end));
-            return null;
-        }
-        return encodedWord;
-    }
-
-    private static ByteString decodeQ(String encodedWord) {
-        /*
-         * Replace _ with =20
-         */
-        StringBuilder sb = new StringBuilder();
-        for (int i = 0; i < encodedWord.length(); i++) {
-            char c = encodedWord.charAt(i);
-            if (c == '_') {
-                sb.append("=20");
-            } else {
-                sb.append(c);
-            }
-        }
-
-        byte[] bytes = sb.toString().getBytes(Charset.forName("US-ASCII"));
-
-        QuotedPrintableInputStream is = new QuotedPrintableInputStream(new ByteArrayInputStream(bytes));
-        try {
-            return Okio.buffer(Okio.source(is)).readByteString();
-        } catch (IOException e) {
-            return null;
-        }
-    }
-
-    private static ByteString decodeB(String encodedText) {
-        ByteString decoded = ByteString.decodeBase64(encodedText);
-        return decoded == null ? ByteString.EMPTY : decoded;
-    }
-
-    private static ByteString concat(ByteString first, ByteString second) {
-        return new Buffer().write(first).write(second).readByteString();
-    }
-
-
-    private static class EncodedWord {
-        private String charset;
-        private String encoding;
-        private ByteString data;
-    }
-}
diff --git a/mail/common/src/main/java/com/fsck/k9/mail/internet/DecoderUtil.kt b/mail/common/src/main/java/com/fsck/k9/mail/internet/DecoderUtil.kt
new file mode 100644
index 00000000000..ee8053f1e2b
--- /dev/null
+++ b/mail/common/src/main/java/com/fsck/k9/mail/internet/DecoderUtil.kt
@@ -0,0 +1,201 @@
+package com.fsck.k9.mail.internet
+
+import com.fsck.k9.mail.Message
+import com.fsck.k9.mail.MessagingException
+import java.io.ByteArrayInputStream
+import java.io.IOException
+import okio.Buffer
+import okio.ByteString
+import okio.ByteString.Companion.decodeBase64
+import okio.buffer
+import okio.source
+import org.apache.james.mime4j.codec.QuotedPrintableInputStream
+import org.apache.james.mime4j.util.CharsetUtil
+import timber.log.Timber
+
+/**
+ * Decoder for encoded words (RFC 2047).
+ *
+ * This class is based on `org.apache.james.mime4j.decoder.DecoderUtil`. It was modified in order to support early
+ * non-Unicode emoji variants.
+ */
+internal object DecoderUtil {
+    /**
+     * Decodes a string containing encoded words as defined by RFC 2047.
+     *
+     * Encoded words have the form `=?charset?enc?Encoded word?=` where `enc` is either 'Q' or 'q' for
+     * quoted-printable and 'B' or 'b' for Base64.
+     *
+     * @param body The string to decode.
+     * @param message The message containing the string. It will be used to figure out which JIS variant to use for
+     *     charset decoding. May be `null`.
+     * @return The decoded string.
+     */
+    @JvmStatic
+    fun decodeEncodedWords(body: String, message: Message?): String {
+        // Most strings will not include "=?". So a quick test can prevent unneeded work.
+        if (!body.contains("=?")) return body
+
+        var previousWord: EncodedWord? = null
+        var previousEnd = 0
+        val output = StringBuilder()
+
+        while (true) {
+            val begin = body.indexOf("=?", previousEnd)
+            if (begin == -1) {
+                decodePreviousAndAppendSuffix(output, previousWord, body, previousEnd)
+                return output.toString()
+            }
+
+            val qm1 = body.indexOf('?', begin + 2)
+            if (qm1 == -1) {
+                decodePreviousAndAppendSuffix(output, previousWord, body, previousEnd)
+                return output.toString()
+            }
+
+            val qm2 = body.indexOf('?', qm1 + 1)
+            if (qm2 == -1) {
+                decodePreviousAndAppendSuffix(output, previousWord, body, previousEnd)
+                return output.toString()
+            }
+
+            var end = body.indexOf("?=", qm2 + 1)
+            if (end == -1) {
+                decodePreviousAndAppendSuffix(output, previousWord, body, previousEnd)
+                return output.toString()
+            }
+            end += 2
+
+            val sep = body.substring(previousEnd, begin)
+            val word = extractEncodedWord(body, begin, end, message)
+
+            if (previousWord == null) {
+                output.append(sep)
+                if (word == null) {
+                    output.append(body, begin, end)
+                }
+            } else if (word == null) {
+                output.append(charsetDecode(previousWord))
+                output.append(sep)
+                output.append(body, begin, end)
+            } else if (!CharsetUtil.isWhitespace(sep)) {
+                output.append(charsetDecode(previousWord))
+                output.append(sep)
+            } else if (previousWord.canBeCombinedWith(word)) {
+                word.data = previousWord.data + word.data
+            } else {
+                output.append(charsetDecode(previousWord))
+            }
+
+            previousWord = word
+            previousEnd = end
+        }
+    }
+
+    private fun decodePreviousAndAppendSuffix(
+        output: StringBuilder,
+        previousWord: EncodedWord?,
+        body: String,
+        previousEnd: Int
+    ) {
+        if (previousWord != null) {
+            output.append(charsetDecode(previousWord))
+        }
+        output.append(body, previousEnd, body.length)
+    }
+
+    private fun charsetDecode(word: EncodedWord): String? {
+        return try {
+            val inputStream = Buffer().write(word.data).inputStream()
+            CharsetSupport.readToString(inputStream, word.charset)
+        } catch (e: IOException) {
+            null
+        }
+    }
+
+    private fun extractEncodedWord(body: String, begin: Int, end: Int, message: Message?): EncodedWord? {
+        val qm1 = body.indexOf('?', begin + 2)
+        if (qm1 == end - 2) return null
+
+        val qm2 = body.indexOf('?', qm1 + 1)
+        if (qm2 == end - 2) return null
+
+        // Extract charset, skipping language information if present (example: =?utf-8*en?Q?Text?=)
+        val charsetPart = body.substring(begin + 2, qm1)
+        val languageSuffixStart = charsetPart.indexOf('*')
+        val languageSuffixFound = languageSuffixStart != -1
+        val mimeCharset = if (languageSuffixFound) charsetPart.substring(0, languageSuffixStart) else charsetPart
+
+        val encoding = body.substring(qm1 + 1, qm2)
+        val encodedText = body.substring(qm2 + 1, end - 2)
+
+        val charset = try {
+            CharsetSupport.fixupCharset(mimeCharset, message)
+        } catch (e: MessagingException) {
+            return null
+        }
+
+        if (encodedText.isEmpty()) {
+            Timber.w("Missing encoded text in encoded word: '%s'", body.substring(begin, end))
+            return null
+        }
+
+        return if (encoding.equals("Q", ignoreCase = true)) {
+            EncodedWord(charset, Encoding.Q, decodeQ(encodedText))
+        } else if (encoding.equals("B", ignoreCase = true)) {
+            EncodedWord(charset, Encoding.B, decodeB(encodedText))
+        } else {
+            Timber.w("Warning: Unknown encoding in encoded word '%s'", body.substring(begin, end))
+            null
+        }
+    }
+
+    private fun decodeQ(encodedWord: String): ByteString {
+        // Replace _ with =20
+        val bytes = buildString {
+            for (character in encodedWord) {
+                if (character == '_') {
+                    append("=20")
+                } else {
+                    append(character)
+                }
+            }
+        }.toByteArray(Charsets.US_ASCII)
+
+        return QuotedPrintableInputStream(ByteArrayInputStream(bytes)).use { inputStream ->
+            try {
+                inputStream.source().buffer().readByteString()
+            } catch (e: IOException) {
+                ByteString.EMPTY
+            }
+        }
+    }
+
+    private fun decodeB(encodedText: String): ByteString {
+        return encodedText.decodeBase64() ?: ByteString.EMPTY
+    }
+
+    private operator fun ByteString.plus(second: ByteString): ByteString {
+        return Buffer().write(this).write(second).readByteString()
+    }
+
+    private val ASCII_ESCAPE_SEQUENCE = byteArrayOf(0x1B, 0x28, 0x42)
+
+    private class EncodedWord(
+        val charset: String,
+        val encoding: Encoding,
+        var data: ByteString
+    ) {
+        fun canBeCombinedWith(other: EncodedWord): Boolean {
+            return encoding == other.encoding && charset == other.charset && !isAsciiEscapeSequence()
+        }
+
+        private fun isAsciiEscapeSequence(): Boolean {
+            return charset.startsWith("ISO-2022-JP", ignoreCase = true) && data.endsWith(ASCII_ESCAPE_SEQUENCE)
+        }
+    }
+
+    private enum class Encoding {
+        Q, B
+    }
+}


Test patch:
diff --git a/mail/common/src/test/java/com/fsck/k9/mail/internet/DecoderUtilTest.java b/mail/common/src/test/java/com/fsck/k9/mail/internet/DecoderUtilTest.java
index b1f077573ba..e5c027b71fb 100644
--- a/mail/common/src/test/java/com/fsck/k9/mail/internet/DecoderUtilTest.java
+++ b/mail/common/src/test/java/com/fsck/k9/mail/internet/DecoderUtilTest.java
@@ -220,6 +220,16 @@ public void decodeEncodedWords_withLanguageInformation() {
         assertInputDecodesToExpected("=?utf-8*de?b?R3LDvMOfZQ==?=", "Grüße");
     }
 
+    @Test
+    public void decodeEncodedWords_withMultipleIso2022JpEncodedWordsProperlyEndingWithSwitchingToAscii() {
+        // If we try to combine the base64-decoded data of both encoded words and only then perform the charset
+        // decoding, we end up with an escape sequence switching to ASCII (end of first encoded word) followed by an
+        // escape sequence switching to JIS X 0208:1983 (start of second encoded word). The decoder on Android reports
+        // an error for this case, leading to a replacement character being inserted.
+        // We use the ISO-2022-JP-TEST charset to get Android's behavior on the JVM. See TestCharsetProvider.
+        assertInputDecodesToExpected("=?ISO-2022-JP-TEST?B?GyRCRnxLXDhsJEhGfEtcOGwkSEZ8S1w4bCROJUElJyVDGyhC?=\r\n" +
+                " =?ISO-2022-JP-TEST?B?GyRCJS8bKEI=?=", "日本語と日本語と日本語のチェック");
+    }
 
     private void assertInputDecodesToExpected(String input, String expected) {
         String decodedText = DecoderUtil.decodeEncodedWords(input, null);
diff --git a/mail/common/src/test/java/com/fsck/k9/mail/internet/TestCharsetProvider.kt b/mail/common/src/test/java/com/fsck/k9/mail/internet/TestCharsetProvider.kt
new file mode 100644
index 00000000000..0223c88ae57
--- /dev/null
+++ b/mail/common/src/test/java/com/fsck/k9/mail/internet/TestCharsetProvider.kt
@@ -0,0 +1,28 @@
+package com.fsck.k9.mail.internet
+
+import com.ibm.icu.charset.CharsetProviderICU
+import java.nio.charset.Charset
+import java.nio.charset.spi.CharsetProvider
+
+/**
+ * CharsetProvider that adds the "ISO-2022-JP-TEST" charset.
+ *
+ * The "ISO-2022-JP" decoder on the JVM is more lenient than the ICU4J decoder that is used on Android. For tests we
+ * use the ICU4J implementation that is also used on Android.
+ */
+class TestCharsetProvider : CharsetProvider() {
+    private val icuCharsetProvider = CharsetProviderICU()
+    private val charset = icuCharsetProvider.charsetForName("ISO-2022-JP")
+
+    override fun charsets(): Iterator<Charset> {
+        return listOf(charset).iterator()
+    }
+
+    override fun charsetForName(charsetName: String?): Charset? {
+        return if (charsetName?.equals("ISO-2022-JP-TEST", ignoreCase = true) == true) {
+            charset
+        } else {
+            null
+        }
+    }
+}
diff --git a/mail/common/src/test/resources/META-INF/services/java.nio.charset.spi.CharsetProvider b/mail/common/src/test/resources/META-INF/services/java.nio.charset.spi.CharsetProvider
new file mode 100644
index 00000000000..91a9fdbe203
--- /dev/null
+++ b/mail/common/src/test/resources/META-INF/services/java.nio.charset.spi.CharsetProvider
@@ -0,0 +1,1 @@
+com.fsck.k9.mail.internet.TestCharsetProvider