Repository: thunderbird/thunderbird-android (language: kotlin)

Issue:
Incorrect encoding of the email text / Torrent tracker SoftPilot.win
### Checklist

- [X] I have used the search function to see if someone else has already submitted the same bug report.
- [X] I will describe the problem with as much detail as possible.

### App version

5.912

### Where did you get the app from?

F-Droid

### Android version

12 | KOSP 2.3 Unofficial

### Device model

Realme X2 CN (RMX1991)

### Steps to reproduce

1. Wait for an email from this tracker.
2. Open the email and see that the encoding of the text of the email is incorrect.
![Screenshot_20220226-175547_Nova Launcher](https://user-images.githubusercontent.com/47178587/157188524-d7328604-0aa4-4feb-b25b-a0b2f1ffa5d4.png)
![Screenshot_20220226-175559_Почта K-9](https://user-images.githubusercontent.com/47178587/157188568-430168f7-50c6-4acd-8315-1fc7997045b9.png)


### Expected behavior

Emails received from SoftPilot.win have the correct text encoding.

### Actual behavior

Emails received from SoftPilot.win have the incorrect text encoding.

### Logs

_No response_


Discussion hints:
Can you attach the message source? 

Chances are the email is not properly declaring the character set that is used. That would be a bug in the software creating the message, not one in K-9 Mail.
Many email apps attempt to guess the character set in order to display the message as intended. K-9 Mail doesn't do that. So the message being displayed "correctly" in other apps is not an indication that the message isn't broken.
This ZIP only contains an HTML document. I was looking for the whole message source, including email headers. 
[SoftPilot.win_Broken_Email.zip](https://github.com/k9mail/k-9/files/8224535/SoftPilot.win_Broken_Email.zip)

The problem is the following header line:

```text
Content-Type: text/html; charset=UTF-8; charset=utf-8
```

The relevant specifications only state that the ordering of parameters is not significant. It is not specified how duplicate parameter names should be interpreted. To avoid interoperability problems where one client prefers the first parameter value and another one the last parameter value, K-9 Mail ignores both `charset` parameter values (and the default charset US-ASCII is used).
This instance seems to be a special case where we could relax this rule. Because the charset is not case sensitive, both parameters are equivalent.

I'll update K-9 Mail to be able to handle this case. But you should still contact the authors of the newsletter and tell them to create emails with a proper `Content-Type` header.
Thanks.
Yes, I will inform the authors of the site about it.

Pull request: Add support for duplicate "charset" parameters with matching values
- For extracting the `charset` parameter value, we only need basic (RFC 2045) parameter decoding. 
- We typically ignore duplicate parameters (parameter name used more than once) because it's not specified which one to pick. We relax this rule to support the case where all `charset` values are equivalent, e.g. `Content-Type: text/html; charset=UTF-8; charset=utf-8`.

Fixes #5951

Test command: ./gradlew :mail:common:testDebugUnitTest --tests "com.fsck.k9.mail.internet.PartExtensionsTest"

Fix patch:
diff --git a/mail/common/src/main/java/com/fsck/k9/mail/internet/MessageExtractor.java b/mail/common/src/main/java/com/fsck/k9/mail/internet/MessageExtractor.java
index e45aa18e34a..c54f469be8a 100644
--- a/mail/common/src/main/java/com/fsck/k9/mail/internet/MessageExtractor.java
+++ b/mail/common/src/main/java/com/fsck/k9/mail/internet/MessageExtractor.java
@@ -23,7 +23,6 @@
 import timber.log.Timber;
 
 import static com.fsck.k9.mail.internet.CharsetSupport.fixupCharset;
-import static com.fsck.k9.mail.internet.MimeUtility.getHeaderParameter;
 import static com.fsck.k9.mail.internet.MimeUtility.isSameMimeType;
 import static com.fsck.k9.mail.internet.Viewable.Alternative;
 import static com.fsck.k9.mail.internet.Viewable.Html;
@@ -77,7 +76,7 @@ private static String getTextFromTextPart(Part part, Body body, String mimeType,
         /*
          * We've got a text part, so let's see if it needs to be processed further.
          */
-        String charset = getHeaderParameter(part.getContentType(), "charset");
+        String charset = PartExtensions.getCharset(part);
         /*
          * determine the charset from HTML message.
          */
diff --git a/mail/common/src/main/java/com/fsck/k9/mail/internet/MimeParameterDecoder.kt b/mail/common/src/main/java/com/fsck/k9/mail/internet/MimeParameterDecoder.kt
index 7dcf29a6d6a..5f4fc4d4468 100644
--- a/mail/common/src/main/java/com/fsck/k9/mail/internet/MimeParameterDecoder.kt
+++ b/mail/common/src/main/java/com/fsck/k9/mail/internet/MimeParameterDecoder.kt
@@ -54,6 +54,26 @@ object MimeParameterDecoder {
         )
     }
 
+    fun decodeBasic(headerBody: String): MimeValue {
+        val parser = MimeHeaderParser(headerBody)
+
+        val value = parser.readHeaderValue()
+        parser.skipCFWS()
+        if (parser.endReached()) {
+            return MimeValue(value)
+        }
+
+        val (basicParameters, duplicateParameters, parserErrorIndex) = readBasicParameters(parser)
+        val parameters = basicParameters.mapValues { (_, parameterValue) -> parameterValue.value }
+
+        return MimeValue(
+            value = value,
+            parameters = parameters,
+            ignoredParameters = duplicateParameters,
+            parserErrorIndex = parserErrorIndex
+        )
+    }
+
     @JvmStatic
     fun extractHeaderValue(headerBody: String): String {
         val parser = MimeHeaderParser(headerBody)
diff --git a/mail/common/src/main/java/com/fsck/k9/mail/internet/PartExtensions.kt b/mail/common/src/main/java/com/fsck/k9/mail/internet/PartExtensions.kt
new file mode 100644
index 00000000000..a91d7db833c
--- /dev/null
+++ b/mail/common/src/main/java/com/fsck/k9/mail/internet/PartExtensions.kt
@@ -0,0 +1,24 @@
+@file:JvmName("PartExtensions")
+package com.fsck.k9.mail.internet
+
+import com.fsck.k9.mail.Part
+
+/**
+ * Return the `charset` parameter value of this [Part]'s `Content-Type` header.
+ */
+val Part.charset: String?
+    get() {
+        val contentTypeHeader = this.contentType ?: return null
+        val (_, parameters, duplicateParameters) = MimeParameterDecoder.decodeBasic(contentTypeHeader)
+        return parameters["charset"] ?: extractNonConflictingCharsetValue(duplicateParameters)
+    }
+
+// If there are multiple "charset" parameters, but they all agree on the value, we use that value.
+private fun extractNonConflictingCharsetValue(duplicateParameters: List<Pair<String, String>>): String? {
+    val charsets = duplicateParameters.asSequence()
+        .filter { (parameterName, _) -> parameterName == "charset" }
+        .map { (_, charset) -> charset.lowercase() }
+        .toSet()
+
+    return if (charsets.size == 1) charsets.first() else null
+}


Test patch:
diff --git a/mail/common/src/test/java/com/fsck/k9/mail/internet/PartExtensionsTest.kt b/mail/common/src/test/java/com/fsck/k9/mail/internet/PartExtensionsTest.kt
new file mode 100644
index 00000000000..f5afbdec568
--- /dev/null
+++ b/mail/common/src/test/java/com/fsck/k9/mail/internet/PartExtensionsTest.kt
@@ -0,0 +1,54 @@
+package com.fsck.k9.mail.internet
+
+import com.google.common.truth.Truth.assertThat
+import org.junit.Test
+
+class PartExtensionsTest {
+    @Test
+    fun `get charset without charset parameter`() {
+        assertGetCharset(headerValue = "text/plain", expectedCharset = null)
+    }
+
+    @Test
+    fun `get charset with single charset parameter`() {
+        assertGetCharset(headerValue = "text/plain; charset=UTF-8", expectedCharset = "utf-8")
+    }
+
+    @Test
+    fun `get charset with single quoted charset parameter`() {
+        assertGetCharset(headerValue = "text/plain; charset=\"iso-8859-1\"", expectedCharset = "ISO-8859-1")
+    }
+
+    @Test
+    fun `get charset with two charset parameters where values match exactly`() {
+        assertGetCharset(headerValue = "text/plain; charset=utf-8; charset=utf-8", expectedCharset = "utf-8")
+    }
+
+    @Test
+    fun `get charset with two charset parameters where values differ in case`() {
+        assertGetCharset(headerValue = "text/plain; charset=utf-8; charset=UTF-8", expectedCharset = "utf-8")
+    }
+
+    @Test
+    fun `get charset with two charset parameters where values differ in quoting`() {
+        assertGetCharset(headerValue = "text/plain; charset=utf-8; charset=\"utf-8\"", expectedCharset = "utf-8")
+    }
+
+    @Test
+    fun `get charset with two charset parameters with conflicting values`() {
+        assertGetCharset(headerValue = "text/plain; charset=utf-8; charset=iso-8859-1", expectedCharset = null)
+    }
+
+    @Test
+    fun `get charset with extended parameter syntax`() {
+        assertGetCharset(headerValue = "text/plain; charset*=us-ascii'en-us'utf-8", expectedCharset = null)
+    }
+
+    private fun assertGetCharset(headerValue: String, expectedCharset: String?) {
+        val part = MimeBodyPart.create(null, headerValue)
+
+        val charset = part.charset
+
+        assertThat(charset).ignoringCase().isEqualTo(expectedCharset)
+    }
+}