Repository: thunderbird/thunderbird-android (language: kotlin)

Issue:
EHLO keyword contains invalid character
### Checklist

- [X] I have used the search function to see if someone else has already submitted the same bug report.
- [X] I will describe the problem with as much detail as possible.

### App version

d13352a4cd7f5ea540eba738155ff377ee48da62 - 2205202769770bc37feb98414bb8a1ae084b9c98

### Where did you get the app from?

Other

### Android version

11 (LineageOS for microG)

### Device model

Fairphone 3

### Steps to reproduce

1. Set up an email account
2. Go to outgoing server settings
3. Fill in everything and click next

SMTP server: `<REDACTED>`
Security: SSL/TLS
Port: 465
Require sign-in: Yes
Username: `<REDACTED>`
Authentication: Normal password
Password: `<REDACTED>`
Client certificate: No client certificate

Note: This happens on my own self-hosted mailserver, not on a public provider. I obviously don't want to make authentication info for it public, but if it helps to debug please send me an email and I'll get you some login credentials.

### Expected behavior

Succesfully set up SMTP, like it did on K-9 Mail v6.000

### Actual behavior

An error message shows:

# Setup could not finish
Cannot connect to server.
(EHLO keyword contains invalid character)
[ Continue ] [ Edit Details ]

![image](https://user-images.githubusercontent.com/1885159/167604225-40801927-8e97-44b4-b177-0732d62d22df.png)


### Logs

Using git bisect, I was able to pinpoint it to the following commit:

```
d13352a4cd7f5ea540eba738155ff377ee48da62 is the first bad commit
commit d13352a4cd7f5ea540eba738155ff377ee48da62
Author: cketti <cketti@gmail.com>
Date:   Wed Mar 23 04:07:37 2022 +0100

    Use new `SmtpResponseParser` in `SmtpTransport`

 .../smtp/EnhancedNegativeSmtpReplyException.java   |  12 +-
 .../fsck/k9/mail/transport/smtp/SmtpResponse.kt    |   3 +
 .../fsck/k9/mail/transport/smtp/SmtpTransport.java | 268 +++++++--------------
 .../k9/mail/transport/smtp/StatusCodeClass.java    |  12 -
 .../k9/mail/transport/smtp/StatusCodeDetail.java   |  80 ------
 .../k9/mail/transport/smtp/StatusCodeSubject.java  |  31 ---
 .../k9/mail/transport/smtp/SmtpTransportTest.java  |   8 +-
 7 files changed, 100 insertions(+), 314 deletions(-)
 delete mode 100644 mail/protocols/smtp/src/main/java/com/fsck/k9/mail/transport/smtp/StatusCodeDetail.java
 delete mode 100644 mail/protocols/smtp/src/main/java/com/fsck/k9/mail/transport/smtp/StatusCodeSubject.java
```

Please note: In the attached log file, I replaced the mailserver identification with `mail.example.com` and the Hello's hostname and IP with `<REDACTED>`.

[k9-log-redacted.txt](https://github.com/k9mail/k-9/files/8660865/k9-log-redacted.txt)


Discussion hints:
The server's response to the `EHLO` command starts with:

```text
250-mail.example.com Hello <REDACTED> [<REDACTED>]
250-SIZE 52428800
250-8BITMIME
250-PIPELINING
250-PIPE_CONNECT
```

`PIPE_CONNECT` contains an underscore, which is not allowed. See https://datatracker.ietf.org/doc/html/rfc5321#section-4.1.1.1

It looks like this is a known issue that hasn't received much attention: https://www.mail-archive.com/exim-users@exim.org/msg56497.html

I'd prefer for this to be fixed in exim rather than working around the issue in K-9 Mail. When I rewrote the SMTP response parser, I made it throw an error rather than ignore protocol violations to uncover cases like this – so that the code can be fixed.

Unfortunately, it will be a while before I have time to work on trying to get the issue fixed in exim. Given that this involves changing the keyword `PIPE_CONNECT` to something else, maybe you could get the ball rolling by starting a discussion with the exim developers. If not a lot of people are using the extension, a name change could be relatively simple. If the extension has a significant number of users, a migration strategy might be required.
Thanks for the detailed response. I agree that standard-compliance is important and that this is technically not a K-9 Mail issue but an Exim one. I do however worry that K-9 enforcing this compliance the next release will break the use case of sending emails for many users.

**How common is this misconfiguration?**

In my case, I run Exim4 as configured by MyVestaCP and looking through my Exim4 config, the related config options named in https://www.exim.org/exim-html-current/doc/html/spec_html/ch-main_configuration.html (`pipelining_connect_advertise_hosts` and `hosts_pipe_connect`) are not configured anywhere.

Grabbing [the exim4-config (4.94.2-7) package from Debian bullseye](https://packages.debian.org/bullseye/exim4-config) and grepping through it, I don't see any related options defined anywhere either, so I would assume this is the default configuration for at very least Debian Stable as well.

**Are there any other issues on this configuration?**

Just to try, I have edited the `checkAndNormalizeEhloKeyword` function in K-9 Mail to allowlist the `PIPE_CONNECT` keyword as follows:
```
    private fun checkAndNormalizeEhloKeyword(text: String): String {
        val keyword = text.uppercase()
        if (!keyword[0].isCapitalAlphaDigit() || keyword.any { !it.isCapitalAlphaDigit() && it != DASH }) {
            if (keyword != "PIPE_CONNECT") {
                parserError("EHLO keyword contains invalid character")
            }
        }

        return keyword
    }
```

Doing this allows K-9 Mail to connect, so it seems to me like `PIPE_CONNECT` is the only compliance issue Exim currently has (as far as K-9 can detect).

**Difficulty for me reporting this issue upstream**

Frankly, I don't know that much about email so even though this is apparently enabled on my server I have no idea if I am actively using this or not. As the server is in use by more people than just me, I also can't simply turn it off and see if anything breaks. Given my limited knowledge, I doubt I can have any productive discussion with the Exim developers about this, as I wouldn't be able to answer many "basic" questions.

Currently, my understanding of this subject pretty much boils down to:
1. Exim4's default configuration sends a non-compliant keyword
2. K-9 Mail was likely not using this extensions, as previous versions simply ignored invalid options and thus K-9 probably never understood this extension in the first place

As such, all I could really do on the Exim side is create an issue on https://bugs.exim.org/buglist.cgi?quicksearch=PIPE_CONNECT stating the above.

Given this, I believe it would be more sensible for someone with more knowledge about email to report this issue.

**My suggestion**

Given the scale of this misconfiguration and the otherwise seeming compliance, would it be reasonable for K-9 Mail to ignore only `PIPE_CONNECT` until this gets resolved upstream? That way other issues should still become visible without breaking K-9 Mail for users.
You can fix your server easily, I had the same issue on my self hosted mail server, if you are running postfix, open /etc/postfix/main.cf and look for "broken_sasl_auth_clients = yes" then change it to  "broken_sasl_auth_clients = no" restart the postfix service and dovecot and your issue will be gone.

check  https://www.postfix.org/postconf.5.html for more information.
In my case I am using Exim4. Are you sure the Postfix issue you're running into is the exact same issue? If you check logs, do you also see `250-PIPE_CONNECT`?
I got the same  (Cannot connect to server.
(EHLO keyword contains invalid character) sorry  I did not check the logcat. 
With that option enabled Postfix seems to add a syntactically invalid keyword (starting with `AUTH=`):

https://github.com/vdukhovni/postfix/blob/abf156923c91c8c4f2bbd1a591114eb45bb77e81/postfix/src/smtpd/smtpd.c#L1952-L1953



Pull request: Ignore invalid SMTP EHLO response lines
Fixes #6062

Test command: ./gradlew :mail:protocols:smtp:test --tests "com.fsck.k9.mail.transport.smtp.SmtpResponseParserTest"

Fix patch:
diff --git a/mail/protocols/smtp/src/main/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParser.kt b/mail/protocols/smtp/src/main/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParser.kt
index e67febebd16..a7ceefaa101 100644
--- a/mail/protocols/smtp/src/main/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParser.kt
+++ b/mail/protocols/smtp/src/main/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParser.kt
@@ -115,20 +115,25 @@ internal class SmtpResponseParser(
 
     private fun parseEhloLine(ehloLine: String, keywords: MutableMap<String, List<String>>) {
         val parts = ehloLine.split(" ")
-        val keyword = checkAndNormalizeEhloKeyword(parts[0])
-        val parameters = checkEhloParameters(parts)
 
-        if (keywords.containsKey(keyword)) {
-            parserError("Same EHLO keyword present in more than one response line")
-        }
+        try {
+            val keyword = checkAndNormalizeEhloKeyword(parts[0])
+            val parameters = checkEhloParameters(parts)
+
+            if (keywords.containsKey(keyword)) {
+                parserError("Same EHLO keyword present in more than one response line", logging = false)
+            }
 
-        keywords[keyword] = parameters
+            keywords[keyword] = parameters
+        } catch (e: SmtpResponseParserException) {
+            logger.log(e, "Ignoring EHLO keyword line: $ehloLine")
+        }
     }
 
     private fun checkAndNormalizeEhloKeyword(text: String): String {
         val keyword = text.uppercase()
         if (!keyword[0].isCapitalAlphaDigit() || keyword.any { !it.isCapitalAlphaDigit() && it != DASH }) {
-            parserError("EHLO keyword contains invalid character")
+            parserError("EHLO keyword contains invalid character", logging = false)
         }
 
         return keyword
@@ -138,9 +143,9 @@ internal class SmtpResponseParser(
         for (i in 1..parts.lastIndex) {
             val parameter = parts[i]
             if (parameter.isEmpty()) {
-                parserError("EHLO parameter must not be empty")
+                parserError("EHLO parameter must not be empty", logging = false)
             } else if (parameter.any { it.code !in 33..126 }) {
-                parserError("EHLO parameter contains invalid character")
+                parserError("EHLO parameter contains invalid character", logging = false)
             }
         }
 


Test patch:
diff --git a/mail/protocols/smtp/src/test/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParserTest.kt b/mail/protocols/smtp/src/test/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParserTest.kt
index e3f349201c6..9b813c0ae5c 100644
--- a/mail/protocols/smtp/src/test/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParserTest.kt
+++ b/mail/protocols/smtp/src/test/java/com/fsck/k9/mail/transport/smtp/SmtpResponseParserTest.kt
@@ -145,26 +145,36 @@ class SmtpResponseParserTest {
     }
 
     @Test
-    fun `read EHLO response with invalid keyword`() {
+    fun `read EHLO response with invalid keywords`() {
         val input = """
             250-smtp.domain.example
-            250 KEY:WORD
+            250-SIZE 52428800
+            250-8BITMIME
+            250-PIPELINING
+            250-PIPE_CONNECT
+            250-AUTH=PLAIN
+            250 HELP
             """.toPeekableInputStream()
         val parser = SmtpResponseParser(logger, input)
 
-        assertFailsWithMessage("EHLO keyword contains invalid character") {
-            parser.readHelloResponse()
-        }
+        val response = parser.readHelloResponse()
 
-        assertThat(logger.logEntries).containsExactly(
-            LogEntry(
-                throwable = null,
-                message = """
-                    SMTP response data on parser error:
-                    250-smtp.domain.example
-                    250 KEY:WORD
-                """.trimIndent()
+        assertType<SmtpHelloResponse.Hello>(response) { hello ->
+            assertThat(hello.keywords.keys).containsExactly(
+                "SIZE",
+                "8BITMIME",
+                "PIPELINING",
+                "HELP"
             )
+        }
+
+        assertThat(logger.logEntries.map { it.message }).containsExactly(
+            "Ignoring EHLO keyword line: PIPE_CONNECT",
+            "Ignoring EHLO keyword line: AUTH=PLAIN"
+        )
+        assertThat(logger.logEntries.map { it.throwable?.message }).containsExactly(
+            "EHLO keyword contains invalid character",
+            "EHLO keyword contains invalid character"
         )
     }
 
@@ -176,44 +186,37 @@ class SmtpResponseParserTest {
             """.toPeekableInputStream()
         val parser = SmtpResponseParser(logger, input)
 
-        assertFailsWithMessage("EHLO parameter must not be empty") {
-            parser.readHelloResponse()
+        val response = parser.readHelloResponse()
+
+        assertType<SmtpHelloResponse.Hello>(response) { hello ->
+            assertThat(hello.keywords.keys).isEmpty()
         }
 
-        assertThat(logger.logEntries).containsExactly(
-            LogEntry(
-                throwable = null,
-                message = """
-                    SMTP response data on parser error:
-                    250-smtp.domain.example
-                    250 KEYWORD${" "}
-                """.trimIndent()
-            )
-        )
+        assertThat(logger.logEntries).hasSize(1)
+        assertThat(logger.logEntries.first().throwable).hasMessageThat().isEqualTo("EHLO parameter must not be empty")
+        assertThat(logger.logEntries.first().message).isEqualTo("Ignoring EHLO keyword line: KEYWORD ")
     }
 
     @Test
     fun `read EHLO response with invalid parameter`() {
         val input = """
             250-smtp.domain.example
+            250-8BITMIME
             250 KEYWORD para${"\t"}meter
             """.toPeekableInputStream()
         val parser = SmtpResponseParser(logger, input)
 
-        assertFailsWithMessage("EHLO parameter contains invalid character") {
-            parser.readHelloResponse()
+        val response = parser.readHelloResponse()
+
+        assertType<SmtpHelloResponse.Hello>(response) { hello ->
+            assertThat(hello.keywords.keys).containsExactly("8BITMIME")
         }
 
-        assertThat(logger.logEntries).containsExactly(
-            LogEntry(
-                throwable = null,
-                message = """
-                    SMTP response data on parser error:
-                    250-smtp.domain.example
-                    250 KEYWORD para${"\t"}meter
-                """.trimIndent()
-            )
-        )
+        assertThat(logger.logEntries).hasSize(1)
+        assertThat(logger.logEntries.first().throwable)
+            .hasMessageThat().isEqualTo("EHLO parameter contains invalid character")
+        assertThat(logger.logEntries.first().message)
+            .isEqualTo("Ignoring EHLO keyword line: KEYWORD para${"\t"}meter")
     }
 
     @Test