Repository: mailgun/talon

Issue:
Quotes extraction failed with Polymail replies
Replies initiated from [polymail](https://polymail.io/) do not extract the quoted text.

Steps to reproduce:
1. Send a message to yourself using polymail
2. Read that message in gmail and reply to it
3. Read the message in polymail and reply to it
4. Get the original message and test on [talon testing site](http://talon.mailgun.net/).

Example:
```
Return-Path: <xxxx@gmail.com>
Received: from localhost (ec2-54-158-139-194.compute-1.amazonaws.com. [54.158.139.194])
        by smtp.gmail.com with ESMTPSA id q32sm10037057qtd.5.2017.04.11.19.08.54
        for <xxxx@gmail.com>
        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
        Tue, 11 Apr 2017 19:08:55 -0700 (PDT)
Mime-Version: 1.0
X-Mailer: Polymail
Message-ID: <58ed8c0700bfbb00001339d3@polymail.io>
Subject: Re: Test
From: Ethan Setnik <xxxxgmail.com>
To: Ethan Setnik <xxxx@gmail.com>
Date: Tue, 11 Apr 2017 19:08:45 -0700
In-Reply-To: <CAKzr07V771zMRYAE9m9yU1NqnGGtkbmUSOvWWz84J2O+i5-ubg@mail.gmail.com>
References: <CAKzr07V771zMRYAE9m9yU1NqnGGtkbmUSOvWWz84J2O+i5-ubg@mail.gmail.com>
Content-Type: multipart/alternative; boundary=379c710cb6a99096f577366db69dccf23b4f785368751bb5acb86a2e162a

--379c710cb6a99096f577366db69dccf23b4f785368751bb5acb86a2e162a
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8

This is the reply from Polymail

On Tue, Apr 11, 2017 at 10:07 PM Ethan Setnik

<
mailto:Ethan Setnik <xxxx@gmail.com>
> wrote:

a, pre, code, a:link, body { word-wrap: break-word !important; }

This is the reply

On Tue, Apr 11, 2017 at 10:07 PM Ethan Setnik <
mailto:xxxx@gmail.com
> wrote:

This is the initial message
--379c710cb6a99096f577366db69dccf23b4f785368751bb5acb86a2e162a
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8

<img style=3D"border: none; background:none; width: 0; height: 0;" src=3D"h=
ttps://share.polymail.io/v2/z/a/NThlZDhjMDcwMGJm/y6ry9vAsZIPxUbBfihWq6SToKm=
8ok2BlbhzkP296GDPgMIHiWWU4BGEw66uif81ucgL4a8UOrQQwwC_UjBeGCbM9In5alO4MMVE8t=
vBv8AZYGvniUINmkwConAMIHC8HJqpDarTqW6RRpNOc.png" alt=3D"" width=3D"0px" hei=
ght=3D"0px" border=3D"0" /><div></div>
<div>
This is the reply from Polymail<br><br>
</div>
<div id=3D"psignature"><br></div>
<div class=3D"gmail_extra"><br><div class=3D"gmail_quote"><div dir=3D"ltr">=
On Tue, Apr 11, 2017 at 10:07 PM Ethan Setnik <xxxxx@gmail.com> &lt;<a hr=
ef=3D"mailto:Ethan Setnik <xxxxx@gmail.com>">Ethan Setnik <xxxxx@gmail.=
com></a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"mar=
gin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><!DOCTYPE html>=
<html><head><meta name=3D"viewport" content=3D"width=3Ddevice-width"><style=
>a, pre, code, a:link, body { word-wrap: break-word !important; }</style></=
head><body>
<div dir=3D"ltr">This is the reply</div>
<br><div class=3D"gmail_quote"></div>
</body></html><div dir=3D"ltr">On Tue, Apr 11, 2017 at 10:07 PM Ethan Setni=
k &lt;<a href=3D"mailto:xxxxx@gmail.com">xxxxx@gmail.com</a>&gt; wrote:=
<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bord=
er-left:1px #ccc solid;padding-left:1ex"><img style=3D"border:none;backgrou=
nd:none;width:0;height:0" src=3D"https://ci4.googleusercontent.com/proxy/JX=
XTuFue-6g3aDYP_klLyNmTSWdgpANzRviNIzs2xpEC3ZMdoU8vXxg9DwFJpUnjRTFU6s4roA3f7=
IW_t3ZahYdgJo_3GAl4qZoNqEse6VsyfI6rmoLF5dB2iRmlV1GUNQgdinzLxOFTi_ycHjBzJQ-r=
vKcMqnQAw3tD9GyeqniSbhysEgK2xMhP4OakJkFptqHAzVbC3TClDGj_UN617anYUW0043qgREP=
JDoT0wBMyTqNruExX68G_alPEZjE-VN3Xr_X3sb4EwxlEoIEU30xD9KHS7tnGwkB9BN0=3Ds0-d=
-e1-ft#https://share.polymail.io/v2/z/a/NThlZDhiYjVjZDU1/OoigMTRZhepebW8cok=
gLWKlfeOn7lYi8eweMZ5IXUoacBlsPJJOLaoXSc0u1N-c2QTywVrJgUzzUmzFmBx1UwiHWkaj_W=
gHTH6qG4m-Eo_t1ETpIvzwZKvwaUKsm_3VFDUQj4i5HZZojujdK.png" alt=3D"" width=3D"=
0px" height=3D"0px" border=3D"0" class=3D"gmail_msg">This is the initial me=
ssage</blockquote></div></div></blockquote></div><br></div>
--379c710cb6a99096f577366db69dccf23b4f785368751bb5acb86a2e162a--
```

Discussion:

Patch:
diff --git a/talon/quotations.py b/talon/quotations.py
index 6016310..9286209 100644
--- a/talon/quotations.py
+++ b/talon/quotations.py
@@ -146,6 +146,14 @@ RE_ANDROID_WROTE = re.compile(u'[\s]*[-]+.*({})[ ]*[-]+'.format(
         'wrote'
     ))), re.I)
 
+# Support polymail.io reply format
+# On Tue, Apr 11, 2017 at 10:07 PM John Smith
+#
+# <
+# mailto:John Smith <johnsmith@gmail.com>
+# > wrote:
+RE_POLYMAIL = re.compile('On.*\s{2}<\smailto:.*\s> wrote:', re.I)
+
 SPLITTER_PATTERNS = [
     RE_ORIGINAL_MESSAGE,
     RE_ON_DATE_SMB_WROTE,
@@ -162,7 +170,8 @@ SPLITTER_PATTERNS = [
                '( \S+){3,6}@\S+:'),
     # Sent from Samsung MobileName <address@example.com> wrote:
     re.compile('Sent from Samsung .*@.*> wrote'),
-    RE_ANDROID_WROTE
+    RE_ANDROID_WROTE,
+    RE_POLYMAIL
     ]
 
 RE_LINK = re.compile('<(http://[^>]*)>')
@@ -170,7 +179,7 @@ RE_NORMALIZED_LINK = re.compile('@@(http://[^>@]*)@@')
 
 RE_PARENTHESIS_LINK = re.compile("\(https?://")
 
-SPLITTER_MAX_LINES = 4
+SPLITTER_MAX_LINES = 6
 MAX_LINES_COUNT = 1000
 # an extensive research shows that exceeding this limit
 # leads to excessive processing time


Test patch:
diff --git a/tests/text_quotations_test.py b/tests/text_quotations_test.py
index 7a81c99..c02c375 100644
--- a/tests/text_quotations_test.py
+++ b/tests/text_quotations_test.py
@@ -35,6 +35,19 @@ On 11-Apr-2011, at 6:54 PM, Roman Tkachenko <romant@example.com> wrote:
 
     eq_("Test reply", quotations.extract_from_plain(msg_body))
 
+def test_pattern_on_date_polymail():
+    msg_body = """Test reply
+
+On Tue, Apr 11, 2017 at 10:07 PM John Smith
+
+<
+mailto:John Smith <johnsmith@gmail.com>
+> wrote:
+Test quoted data
+"""
+
+    eq_("Test reply", quotations.extract_from_plain(msg_body))
+
 
 def test_pattern_sent_from_samsung_smb_wrote():
     msg_body = """Test reply
@@ -54,7 +67,7 @@ def test_pattern_on_date_wrote_somebody():
     """Lorem
 
 Op 13-02-2014 3:18 schreef Julius Caesar <pantheon@rome.com>:
-    
+
 Veniam laborum mlkshk kale chips authentic. Normcore mumblecore laboris, fanny pack readymade eu blog chia pop-up freegan enim master cleanse.
 """))
 
@@ -256,7 +269,7 @@ def test_with_indent():
 
 ------On 12/29/1987 17:32 PM, Julius Caesar wrote-----
 
-Brunch mumblecore pug Marfa tofu, irure taxidermy hoodie readymade pariatur. 
+Brunch mumblecore pug Marfa tofu, irure taxidermy hoodie readymade pariatur.
     """
     eq_("YOLO salvia cillum kogi typewriter mumblecore cardigan skateboard Austin.", quotations.extract_from_plain(msg_body))
 
@@ -381,11 +394,11 @@ Veniam laborum mlkshk kale chips authentic. Normcore mumblecore laboris, fanny p
 
 def test_dutch_from_block():
     eq_('Gluten-free culpa lo-fi et nesciunt nostrud.', quotations.extract_from_plain(
-    """Gluten-free culpa lo-fi et nesciunt nostrud. 
+    """Gluten-free culpa lo-fi et nesciunt nostrud.
 
 Op 17-feb.-2015, om 13:18 heeft Julius Caesar <pantheon@rome.com> het volgende geschreven:
-    
-Small batch beard laboris tempor, non listicle hella Tumblr heirloom. 
+
+Small batch beard laboris tempor, non listicle hella Tumblr heirloom.
 """))