Repository: thunderbird/thunderbird-android (language: kotlin)

Issue:
"Strip signatures on reply" does not work
### Checklist

- [X] I have used the search function to see if someone else has already submitted the same bug report.
- [X] I will describe the problem with as much detail as possible.

### App version

6.202

### Where did you get the app from?

Google Play

### Android version

12

### Device model

Samsung Galaxy S22

### Steps to reproduce

1. Add outlook account 
2. Edit identity
3. Check "Use Signature"
4. Write a signature
5. Check the default setting of "Strip signatures on reply". It is checked.
6. Write an email to myself
7. Reply to myself
8. Reply again

### Expected behavior

Signatures will be removed from quoted messages

### Actual behavior

```text
Hi,

This is the second reply.

On September 7, 2022 10:41:26 AM GMT+02:00, "User" <user@outlook.com> wrote:
Hi,

This is the first reply.

On September 7, 2022 10:40:11 AM GMT+02:00, "User" <user@outlook.com> wrote:
Hi,

This is it.
--
Regards,
User
--
Regards,
User
--
Regards,
User
```

### Logs

![image](https://user-images.githubusercontent.com/95375431/188836344-282793bf-a3b1-4630-ab51-ccea738d191a.png)



Discussion hints:
K-9 Mail only detects the standard signature delimiter. That's two dashes followed by a space. Such signatures will also be displayed slightly differently when viewing a message.

![image](https://user-images.githubusercontent.com/218061/188838300-7ce56654-9c33-4b86-a81d-1048ffa7f957.png)

Hi,

I repeated the test with the standard signature delimiter, and it is slightly different, however still not removed.

![image](https://user-images.githubusercontent.com/95375431/188840604-f74c54ff-61e9-4f6b-b124-8ad13cba90ba.png)



That's unexpected. The signature is detected when displaying the message, but not stripped when replying to the message. It seems to work fine for `text/plain` parts, but not for `text/html` parts.

Anyway, I can reproduce this. That should be enough to find and fix the bug. Thanks for reporting it :+1:
Thanks for the fix! 👍🏻 

Pull request: Fix `HtmlSignatureRemover`
Detecting an email signature in HTML is quite hard, even when the default delimiter `-- ` (dash, dash, space) is used. With these changes we should support more signature formats, including the one currently used by K-9 Mail :see_no_evil: 

Fixes #6279

Test command: ./gradlew :app:core:testDebugUnitTest --tests "com.fsck.k9.message.signature.HtmlSignatureRemoverTest" --tests "com.fsck.k9.message.signature.HtmlSignatureRemoverTest"

Fix patch:
diff --git a/app/core/src/main/java/com/fsck/k9/helper/jsoup/AdvancedNodeTraversor.java b/app/core/src/main/java/com/fsck/k9/helper/jsoup/AdvancedNodeTraversor.java
index cfcab37a4cc..41cd5e245ff 100644
--- a/app/core/src/main/java/com/fsck/k9/helper/jsoup/AdvancedNodeTraversor.java
+++ b/app/core/src/main/java/com/fsck/k9/helper/jsoup/AdvancedNodeTraversor.java
@@ -125,7 +125,7 @@ public FilterResult filter(Node root) {
             Node prev = node;
             node = node.nextSibling();
 
-            if (headResult == HeadFilterDecision.REMOVE) {
+            if (headResult == HeadFilterDecision.REMOVE || tailResult == TailFilterDecision.REMOVE) {
                 prev.remove();
             }
 
diff --git a/app/core/src/main/java/com/fsck/k9/message/signature/HtmlSignatureRemover.java b/app/core/src/main/java/com/fsck/k9/message/signature/HtmlSignatureRemover.java
deleted file mode 100644
index 175623b05d2..00000000000
--- a/app/core/src/main/java/com/fsck/k9/message/signature/HtmlSignatureRemover.java
+++ /dev/null
@@ -1,106 +0,0 @@
-package com.fsck.k9.message.signature;
-
-
-import java.util.regex.Pattern;
-
-import androidx.annotation.NonNull;
-import com.fsck.k9.helper.jsoup.AdvancedNodeTraversor;
-import com.fsck.k9.helper.jsoup.NodeFilter;
-import org.jsoup.Jsoup;
-import org.jsoup.nodes.Document;
-import org.jsoup.nodes.Element;
-import org.jsoup.nodes.Node;
-import org.jsoup.nodes.TextNode;
-import org.jsoup.parser.Tag;
-
-
-public class HtmlSignatureRemover {
-    public static String stripSignature(String content) {
-        return new HtmlSignatureRemover().stripSignatureInternal(content);
-    }
-
-    private String stripSignatureInternal(String content) {
-        Document document = Jsoup.parse(content);
-
-        AdvancedNodeTraversor nodeTraversor = new AdvancedNodeTraversor(new StripSignatureFilter());
-        nodeTraversor.filter(document.body());
-
-        return toCompactString(document);
-    }
-
-    private String toCompactString(Document document) {
-        document.outputSettings()
-                .prettyPrint(false)
-                .indentAmount(0);
-
-        return document.html();
-    }
-
-
-    static class StripSignatureFilter implements NodeFilter {
-        private static final Pattern DASH_SIGNATURE_HTML = Pattern.compile("\\s*-- \\s*", Pattern.CASE_INSENSITIVE);
-        private static final Tag BLOCKQUOTE = Tag.valueOf("blockquote");
-        private static final Tag BR = Tag.valueOf("br");
-        private static final Tag P = Tag.valueOf("p");
-
-
-        private boolean signatureFound = false;
-        private boolean lastElementCausedLineBreak = false;
-        private Element brElementPrecedingDashes;
-
-
-        @NonNull
-        @Override
-        public HeadFilterDecision head(Node node, int depth) {
-            if (signatureFound) {
-                return HeadFilterDecision.REMOVE;
-            }
-
-            if (node instanceof Element) {
-                lastElementCausedLineBreak = false;
-
-                Element element = (Element) node;
-                if (element.tag().equals(BLOCKQUOTE)) {
-                    return HeadFilterDecision.SKIP_ENTIRELY;
-                }
-            } else if (node instanceof TextNode) {
-                TextNode textNode = (TextNode) node;
-                if (lastElementCausedLineBreak && DASH_SIGNATURE_HTML.matcher(textNode.getWholeText()).matches()) {
-                    Node nextNode = node.nextSibling();
-                    if (nextNode instanceof Element && ((Element) nextNode).tag().equals(BR)) {
-                        signatureFound = true;
-                        if (brElementPrecedingDashes != null) {
-                            brElementPrecedingDashes.remove();
-                            brElementPrecedingDashes = null;
-                        }
-
-                        return HeadFilterDecision.REMOVE;
-                    }
-                }
-            }
-
-            return HeadFilterDecision.CONTINUE;
-        }
-
-        @NonNull
-        @Override
-        public TailFilterDecision tail(Node node, int depth) {
-            if (signatureFound) {
-                return TailFilterDecision.CONTINUE;
-            }
-
-            if (node instanceof Element) {
-                Element element = (Element) node;
-                boolean elementIsBr = element.tag().equals(BR);
-                if (elementIsBr || element.tag().equals(P)) {
-                    lastElementCausedLineBreak = true;
-                    brElementPrecedingDashes = elementIsBr ? element : null;
-                    return TailFilterDecision.CONTINUE;
-                }
-            }
-
-            lastElementCausedLineBreak = false;
-            return TailFilterDecision.CONTINUE;
-        }
-    }
-}
diff --git a/app/core/src/main/java/com/fsck/k9/message/signature/HtmlSignatureRemover.kt b/app/core/src/main/java/com/fsck/k9/message/signature/HtmlSignatureRemover.kt
new file mode 100644
index 00000000000..07746c20dcb
--- /dev/null
+++ b/app/core/src/main/java/com/fsck/k9/message/signature/HtmlSignatureRemover.kt
@@ -0,0 +1,142 @@
+package com.fsck.k9.message.signature
+
+import com.fsck.k9.helper.jsoup.AdvancedNodeTraversor
+import com.fsck.k9.helper.jsoup.NodeFilter
+import com.fsck.k9.helper.jsoup.NodeFilter.HeadFilterDecision
+import com.fsck.k9.helper.jsoup.NodeFilter.TailFilterDecision
+import java.util.Stack
+import java.util.regex.Pattern
+import org.jsoup.Jsoup
+import org.jsoup.nodes.Document
+import org.jsoup.nodes.Element
+import org.jsoup.nodes.Node
+import org.jsoup.nodes.TextNode
+import org.jsoup.parser.Tag
+
+class HtmlSignatureRemover {
+    private fun stripSignatureInternal(content: String): String {
+        val document = Jsoup.parse(content)
+
+        val nodeTraversor = AdvancedNodeTraversor(StripSignatureFilter())
+        nodeTraversor.filter(document.body())
+
+        return toCompactString(document)
+    }
+
+    private fun toCompactString(document: Document): String {
+        document.outputSettings()
+            .prettyPrint(false)
+            .indentAmount(0)
+
+        return document.html()
+    }
+
+    private class StripSignatureFilter : NodeFilter {
+        private var signatureFound = false
+        private var signatureParentNode: Node? = null
+
+        override fun head(node: Node, depth: Int): HeadFilterDecision {
+            if (signatureFound) return HeadFilterDecision.REMOVE
+
+            if (node.isBlockquote()) {
+                return HeadFilterDecision.SKIP_ENTIRELY
+            } else if (node.isSignatureDelimiter()) {
+                val precedingLineBreak = node.findPrecedingLineBreak()
+                if (precedingLineBreak != null && node.isFollowedByLineBreak()) {
+                    signatureFound = true
+                    signatureParentNode = node.parent()
+                    precedingLineBreak.takeIf { it.isBR() }?.remove()
+
+                    return HeadFilterDecision.REMOVE
+                }
+            }
+
+            return HeadFilterDecision.CONTINUE
+        }
+
+        override fun tail(node: Node, depth: Int): TailFilterDecision {
+            if (signatureFound) {
+                val signatureParentNode = this.signatureParentNode
+                if (node == signatureParentNode) {
+                    return if (signatureParentNode.isEmpty()) {
+                        this.signatureParentNode = signatureParentNode.parent()
+                        TailFilterDecision.REMOVE
+                    } else {
+                        TailFilterDecision.STOP
+                    }
+                }
+            }
+
+            return TailFilterDecision.CONTINUE
+        }
+
+        private fun Node.isBlockquote(): Boolean {
+            return this is Element && tag() == BLOCKQUOTE
+        }
+
+        private fun Node.isSignatureDelimiter(): Boolean {
+            return this is TextNode && DASH_SIGNATURE_HTML.matcher(wholeText).matches()
+        }
+
+        private fun Node.findPrecedingLineBreak(): Node? {
+            val stack = Stack<Node>()
+            stack.push(this)
+
+            while (stack.isNotEmpty()) {
+                val node = stack.pop()
+                val previousSibling = node.previousSibling()
+                if (previousSibling == null) {
+                    val parent = node.parent()
+                    if (parent is Element && parent.isBlock) {
+                        return parent
+                    } else {
+                        stack.push(parent)
+                    }
+                } else if (previousSibling.isLineBreak()) {
+                    return previousSibling
+                }
+            }
+
+            return null
+        }
+
+        private fun Node.isFollowedByLineBreak(): Boolean {
+            val stack = Stack<Node>()
+            stack.push(this)
+
+            while (stack.isNotEmpty()) {
+                val node = stack.pop()
+                val nextSibling = node.nextSibling()
+                if (nextSibling == null) {
+                    val parent = node.parent()
+                    if (parent is Element && parent.isBlock) {
+                        return true
+                    } else {
+                        stack.push(parent)
+                    }
+                } else if (nextSibling.isLineBreak()) {
+                    return true
+                }
+            }
+
+            return false
+        }
+
+        private fun Node?.isBR() = this is Element && tag() == BR
+
+        private fun Node?.isLineBreak() = isBR() || (this is Element && this.isBlock)
+
+        private fun Node.isEmpty(): Boolean = childNodeSize() == 0
+    }
+
+    companion object {
+        private val DASH_SIGNATURE_HTML = Pattern.compile("\\s*-- \\s*", Pattern.CASE_INSENSITIVE)
+        private val BLOCKQUOTE = Tag.valueOf("blockquote")
+        private val BR = Tag.valueOf("br")
+
+        @JvmStatic
+        fun stripSignature(content: String): String {
+            return HtmlSignatureRemover().stripSignatureInternal(content)
+        }
+    }
+}


Test patch:
diff --git a/app/core/src/test/java/com/fsck/k9/message/signature/HtmlSignatureRemoverTest.java b/app/core/src/test/java/com/fsck/k9/message/signature/HtmlSignatureRemoverTest.java
deleted file mode 100644
index bf5f6068609..00000000000
--- a/app/core/src/test/java/com/fsck/k9/message/signature/HtmlSignatureRemoverTest.java
+++ /dev/null
@@ -1,141 +0,0 @@
-package com.fsck.k9.message.signature;
-
-
-import org.junit.Test;
-
-import static com.fsck.k9.message.html.HtmlHelper.extractText;
-import static org.junit.Assert.assertEquals;
-
-
-public class HtmlSignatureRemoverTest {
-    @Test
-    public void shouldStripSignatureFromK9StyleHtml() throws Exception {
-        String html = "This is the body text" +
-                "<br>" +
-                "-- <br>" +
-                "Sent from my Android device with K-9 Mail. Please excuse my brevity.";
-
-        String withoutSignature = HtmlSignatureRemover.stripSignature(html);
-
-        assertEquals("This is the body text", extractText(withoutSignature));
-    }
-
-    @Test
-    public void shouldStripSignatureFromThunderbirdStyleHtml() throws Exception {
-        String html = "<html>\r\n" +
-                "  <head>\r\n" +
-                "    <meta http-equiv=\"content-type\" content=\"text/html; charset=utf-8\">\r\n" +
-                "  </head>\r\n" +
-                "  <body bgcolor=\"#FFFFFF\" text=\"#000000\">\r\n" +
-                "    <p>This is the body text<br>\r\n" +
-                "    </p>\r\n" +
-                "    -- <br>\r\n" +
-                "    <div class=\"moz-signature\">Sent from my Android device with K-9 Mail." +
-                " Please excuse my brevity.</div>\r\n" +
-                "  </body>\r\n" +
-                "</html>";
-
-        String withoutSignature = HtmlSignatureRemover.stripSignature(html);
-
-        assertEquals("This is the body text", extractText(withoutSignature));
-    }
-
-    @Test
-    public void shouldStripSignatureBeforeBlockquoteTag() throws Exception {
-        String html = "<html><head></head><body>" +
-                "<div>" +
-                "This is the body text" +
-                "<br>" +
-                "-- <br>" +
-                "<blockquote>" +
-                "Sent from my Android device with K-9 Mail. Please excuse my brevity." +
-                "</blockquote>" +
-                "</div>" +
-                "</body></html>";
-
-        String withoutSignature = HtmlSignatureRemover.stripSignature(html);
-
-        assertEquals("<html><head></head><body>" +
-                        "<div>This is the body text</div>" +
-                        "</body></html>",
-                withoutSignature);
-    }
-
-    @Test
-    public void shouldNotStripSignatureInsideBlockquoteTags() throws Exception {
-        String html = "<html><head></head><body>" +
-                "<blockquote>" +
-                "This is some quoted text" +
-                "<br>" +
-                "-- <br>" +
-                "Inner signature" +
-                "</blockquote>" +
-                "<div>" +
-                "This is the body text" +
-                "</div>" +
-                "</body></html>";
-
-        String withoutSignature = HtmlSignatureRemover.stripSignature(html);
-
-        assertEquals("<html><head></head><body>" +
-                        "<blockquote>" +
-                        "This is some quoted text" +
-                        "<br>" +
-                        "-- <br>" +
-                        "Inner signature" +
-                        "</blockquote>" +
-                        "<div>This is the body text</div>" +
-                        "</body></html>",
-                withoutSignature);
-    }
-
-    @Test
-    public void shouldStripSignatureBetweenBlockquoteTags() throws Exception {
-        String html = "<html><head></head><body>" +
-                "<blockquote>" +
-                "Some quote" +
-                "</blockquote>" +
-                "<div>" +
-                "This is the body text" +
-                "<br>" +
-                "-- <br>" +
-                "<blockquote>" +
-                "Sent from my Android device with K-9 Mail. Please excuse my brevity." +
-                "</blockquote>" +
-                "<br>" +
-                "-- <br>" +
-                "Signature inside signature" +
-                "</div>" +
-                "</body></html>";
-
-        String withoutSignature = HtmlSignatureRemover.stripSignature(html);
-
-        assertEquals("<html><head></head><body>" +
-                        "<blockquote>Some quote</blockquote>" +
-                        "<div>This is the body text</div>" +
-                        "</body></html>",
-                withoutSignature);
-    }
-
-    @Test
-    public void shouldStripSignatureAfterLastBlockquoteTags() throws Exception {
-        String html = "<html><head></head><body>" +
-                "This is the body text" +
-                "<br>" +
-                "<blockquote>" +
-                "Some quote" +
-                "</blockquote>" +
-                "<br>" +
-                "-- <br>" +
-                "Sent from my Android device with K-9 Mail. Please excuse my brevity." +
-                "</body></html>";
-
-        String withoutSignature = HtmlSignatureRemover.stripSignature(html);
-
-        assertEquals("<html><head></head><body>" +
-                        "This is the body text<br>" +
-                        "<blockquote>Some quote</blockquote>" +
-                        "</body></html>",
-                withoutSignature);
-    }
-}
diff --git a/app/core/src/test/java/com/fsck/k9/message/signature/HtmlSignatureRemoverTest.kt b/app/core/src/test/java/com/fsck/k9/message/signature/HtmlSignatureRemoverTest.kt
new file mode 100644
index 00000000000..f9ca1af7d4d
--- /dev/null
+++ b/app/core/src/test/java/com/fsck/k9/message/signature/HtmlSignatureRemoverTest.kt
@@ -0,0 +1,184 @@
+package com.fsck.k9.message.signature
+
+import com.fsck.k9.message.html.HtmlHelper.extractText
+import com.fsck.k9.message.signature.HtmlSignatureRemover.Companion.stripSignature
+import com.fsck.k9.testing.removeNewlines
+import com.google.common.truth.Truth.assertThat
+import org.junit.Test
+
+class HtmlSignatureRemoverTest {
+    @Test
+    fun `old K-9 Mail signature format`() {
+        val html =
+            """This is the body text<br>-- <br>Sent from my Android device with K-9 Mail. Please excuse my brevity."""
+
+        val withoutSignature = stripSignature(html)
+
+        assertThat(extractText(withoutSignature)).isEqualTo("This is the body text")
+    }
+
+    @Test
+    fun `old Thunderbird signature format`() {
+        val html =
+            """
+            <html>
+              <head>
+                <meta http-equiv="content-type" content="text/html; charset=utf-8">
+              </head>
+              <body bgcolor="#FFFFFF" text="#000000">
+                <p>This is the body text<br>
+                </p>
+                -- <br>
+                <div class="moz-signature">Sent from my Android device with K-9 Mail. Please excuse my brevity.</div>
+              </body>
+            </html>
+            """.trimIndent()
+
+        val withoutSignature = stripSignature(html)
+
+        assertThat(extractText(withoutSignature)).isEqualTo("This is the body text")
+    }
+
+    @Test
+    fun `signature before blockquote tag`() {
+        val html =
+            """
+            <html>
+            <head></head>
+            <body>
+            <div>
+            This is the body text<br>
+            -- <br>
+            <blockquote>Sent from my Android device with K-9 Mail. Please excuse my brevity.</blockquote>
+            </div>
+            </body>
+            </html>
+            """.trimIndent().removeNewlines()
+
+        val withoutSignature = stripSignature(html)
+
+        assertThat(withoutSignature).isEqualTo(
+            """<html><head></head><body><div>This is the body text</div></body></html>"""
+        )
+    }
+
+    @Test
+    fun `should not strip signature inside blockquote tag`() {
+        val html =
+            """
+            <html>
+            <head></head>
+            <body>
+            <blockquote>
+            This is some quoted text<br>
+            -- <br>
+            Inner signature
+            </blockquote>
+            <div>
+            This is the body text
+            </div>
+            </body>
+            </html>
+            """.trimIndent().removeNewlines()
+
+        val withoutSignature = stripSignature(html)
+
+        assertThat(withoutSignature).isEqualTo(html)
+    }
+
+    @Test
+    fun `signature between blockquote tags`() {
+        val html =
+            """
+            <html>
+            <head></head>
+            <body>
+            <blockquote>Some quote</blockquote>
+            <div>This is the body text<br>
+            -- <br>
+            <blockquote>Sent from my Android device with K-9 Mail. Please excuse my brevity.</blockquote>
+            <br>-- <br>Signature inside signature
+            </div>
+            </body>
+            </html>
+            """.trimIndent().removeNewlines()
+
+        val withoutSignature = stripSignature(html)
+
+        assertThat(withoutSignature).isEqualTo(
+            """
+            <html>
+            <head></head>
+            <body>
+            <blockquote>Some quote</blockquote>
+            <div>This is the body text</div>
+            </body>
+            </html>
+            """.trimIndent().removeNewlines()
+        )
+    }
+
+    @Test
+    fun `signature after last blockquote tag`() {
+        val html =
+            """
+            <html>
+            <head></head>
+            <body>
+            This is the body text<br>
+            <blockquote>Some quote</blockquote>
+            <br>
+            -- <br>
+            Sent from my Android device with K-9 Mail. Please excuse my brevity.
+            </body>
+            </html>
+            """.trimIndent().removeNewlines()
+
+        val withoutSignature = stripSignature(html)
+
+        assertThat(withoutSignature).isEqualTo(
+            """
+            <html>
+            <head></head>
+            <body>
+            This is the body text<br>
+            <blockquote>Some quote</blockquote>
+            </body>
+            </html>
+            """.trimIndent().removeNewlines()
+        )
+    }
+
+    @Test
+    fun `K-9 Mail signature format`() {
+        val html =
+            """
+            <!DOCTYPE html>
+            <html>
+            <body>
+            This is the body text.<br>
+            <br>
+            <div class='k9mail-signature'>
+            -- <br>
+            And this is the signature text.
+            </div>
+            </body>
+            </html>
+            """.trimIndent().removeNewlines()
+
+        val withoutSignature = stripSignature(html)
+
+        assertThat(withoutSignature).isEqualTo(
+            """
+            <!doctype html>
+            <html>
+            <head></head>
+            <body>
+            This is the body text.<br>
+            <br>
+            </body>
+            </html>
+            """.trimIndent().removeNewlines()
+        )
+    }
+}
diff --git a/app/testing/src/main/java/com/fsck/k9/testing/StringExtensions.kt b/app/testing/src/main/java/com/fsck/k9/testing/StringExtensions.kt
new file mode 100644
index 00000000000..2e5a461c19c
--- /dev/null
+++ b/app/testing/src/main/java/com/fsck/k9/testing/StringExtensions.kt
@@ -0,0 +1,3 @@
+package com.fsck.k9.testing
+
+fun String.removeNewlines(): String = replace("([\\r\\n])".toRegex(), "")