Repository: willdurand/EmailReplyParser (PHP)

Issue:
EmailReplyParser::parseReply issue
Hello,

I just got a bug on `EmailReplyParser::parseReply`. In some specific case, the reply is truncated in the middle.

I tried to reduce at maximum a case, here it is : 

```
Bonjour ​Test​,

​Text paragraph 1​


Le paragraphe qui pose problème.

​Text paragraph 2​


- Le 2021-04-09 23:25, ​Test a écrit :
``` 


The correct result should be : 

```
Bonjour ​Test​,

​Text paragraph 1​


Le paragraphe qui pose problème.

​Text paragraph 2​
```

The current result is : 
```
Bonjour ​Test​,

        ​Text paragraph 1​
```

Context : 
* This is french. If I remove the first "Le" : problem solved. If I remove "écrit" : problem solved.
* `Le paragraphe qui pose problème` means `The paragraph which provoke issue`. 
* `Le 2021-04-09 23:25, ​Test a écrit :` means `On 2021-04-09 23:25, ​Test wrote :`

So the issue is here : 
```
'/^\s*(Le(?:(?!^>*\s*Le\b|\bécrit:).){0,1000}écrit(\s|\xc2\xa0):)$/ms', // Le DATE, NAME <EMAIL> a écrit :
```

If I change it to have the same format as the english work : problem half solved (not truncated).
```
'/^\s*(Le(?:(?!^>*\s*Le\b|\bécrit:).){0,1000}écrit:)$/ms', // Le DATE, NAME <EMAIL> a écrit :
```

But I think that `(\s|\xc2\xa0)` was here for some reason ? @Spone ?

see also : 
https://github.com/willdurand/EmailReplyParser/commit/e888d795f277916ec62c2e812b9a10a9614682e8#diff-5e5f4da6cc02b271ae11752a0b5e45ecb842d913728f7b7d9756d88d9219a0f8
https://github.com/willdurand/EmailReplyParser/commit/b71d9983c07b0455cb5b9727b90642a32bc11ae0#diff-5e5f4da6cc02b271ae11752a0b5e45ecb842d913728f7b7d9756d88d9219a0f8


Discussion:
Notes & Questions :

1. The space after `écrit` should be optionnal ?
2. That space should be replicated to the subpattern ?
3. Those space changes from @Spone should be applied to other languages ?
4. This example should be added to tests maybe ?

But I still can't find a correct regexp for my example
Hi @Nuranto!

The problem in your case it probably the `-` at the beginning of this line:

```
- Le 2021-04-09 23:25, ​Test a écrit :
```

Do you know why it's there?

Here is a fix that would make the regex match your specific case:

```
/^-+?\s*(Le(?:(?!^>*\s*Le\b|\bécrit:).){0,1000}écrit(\s|\xc2\xa0)?:)$/ms
```

Try in on Regexr: https://regexr.com/5qha2
Thank you for your answer @Spone 

After some investigation, it seems that it is the way of doing of https://zimbra.free.fr/ which is a french webmail...

Your solution seems to work for that particular case, but it should be generic. And most of all, the regex should not failed like this whatever the source string is. 
And I think it is a global issue, not only the FR regexp.

What about something like this : 
```
/^.{0,5}(Le\b(?:(?!^>*\s*Le\b|\bécrit(\s|\xc2\xa0)?:).){0,1000}écrit(\s|\xc2\xa0)?:)$/m
```
?
> What about something like this :
> 
> ```
> /^.{0,5}(Le\b(?:(?!^>*\s*Le\b|\bécrit(\s|\xc2\xa0)?:).){0,1000}écrit(\s|\xc2\xa0)?:)$/m
> ```
> 
> ?

It seems to work. Have you tried running the tests with this change?

I'm not sure why the current regexes use _both_ the `m` and `s` flags. Maybe @willdurand can shed some light?
Yes test passed. For that regex at least.
For EN regex it seems more complicated (see https://github.com/willdurand/EmailReplyParser/pull/79).

I wonder if it wouldn't be better to have only one regexep template for that, and an array for locales (`[ ['Le', 'écrit'], ['On', 'wrote'] ]`), so the regexp and the tests can be standardized.



Patch:
diff --git a/src/EmailReplyParser/Parser/EmailParser.php b/src/EmailReplyParser/Parser/EmailParser.php
index dc1122d..61233f8 100644
--- a/src/EmailReplyParser/Parser/EmailParser.php
+++ b/src/EmailReplyParser/Parser/EmailParser.php
@@ -31,11 +31,11 @@ class EmailParser
      * @var string[]
      */
     private $quoteHeadersRegex = array(
-        '/^\s*(On(?:(?!^>*\s*On\b|\bwrote:).){0,1000}wrote:)$/ms', // On DATE, NAME <EMAIL> wrote:
-        '/^\s*(Le(?:(?!^>*\s*Le\b|\bécrit:).){0,1000}écrit(\s|\xc2\xa0):)$/ms', // Le DATE, NAME <EMAIL> a écrit :
-        '/^\s*(El(?:(?!^>*\s*El\b|\bescribió:).){0,1000}escribió:)$/ms', // El DATE, NAME <EMAIL> escribió:
-        '/^\s*(El(?:(?!^>*\s*El\b|\bha escrit:).){0,1000}ha escrit:)$/ms', // El DATE, NAME <EMAIL> ha escrit:
-        '/^\s*(Il(?:(?!^>*\s*Il\b|\bscritto:).){0,1000}scritto:)$/ms', // Il DATE, NAME <EMAIL> ha scritto:
+        '/^.{0,5}(On(?:(?!\bOn\b|\bwrote(\s|\xc2\xa0)?:).){0,1000}wrote(\s|\xc2\xa0)?:)$/ms', // On DATE, NAME <EMAIL> wrote:
+        '/^.{0,5}(Le\b(?:(?!\bLe\b|\bécrit(\s|\xc2\xa0)?:).){0,1000}écrit(\s|\xc2\xa0)?:)$/ms', // Le DATE, NAME <EMAIL> a écrit :
+        '/^.{0,5}(El(?:(?!\bEl\b|\bescribió\s?:).){0,1000}escribió\s?:)$/ms', // El DATE, NAME <EMAIL> escribió:
+        '/^.{0,5}(El(?:(?!\bEl\b|\bha escrit\s?:).){0,1000}ha escrit\s?:)$/ms', // El DATE, NAME <EMAIL> ha escrit:
+        '/^.{0,5}(Il(?:(?!\bIl\b|\bscritto(\s|\xc2\xa0)?:).){0,1000}scritto(\s|\xc2\xa0)?:)$/ms', // Il DATE, NAME <EMAIL> ha scritto:
         '/^[\S\s]+ (написа(л|ла|в)+)+:$/msu', // Everything before написал: not ending on wrote:
         '/^\s*(Op\s.+?(schreef|geschreven).+:)$/ms', // Op DATE schreef NAME <EMAIL>:, Op DATE heeft NAME <EMAIL> het volgende geschreven:
         '/^\s*((W\sdniu|Dnia)\s.+?(pisze|napisał(\(a\))?):)$/msu', // W dniu DATE, NAME <EMAIL> pisze|napisał:


Test patch:
diff --git a/tests/EmailReplyParser/Tests/Parser/EmailParserTest.php b/tests/EmailReplyParser/Tests/Parser/EmailParserTest.php
index 4cc2765..1d277d6 100644
--- a/tests/EmailReplyParser/Tests/Parser/EmailParserTest.php
+++ b/tests/EmailReplyParser/Tests/Parser/EmailParserTest.php
@@ -273,6 +273,22 @@ public function testEmailGmailNo()
         $this->assertEquals(static::COMMON_FIRST_FRAGMENT, trim($fragments[0]));
     }
 
+    public function testEmailFreeZimbraFr()
+    {
+        $email     = $this->parser->parse($this->getFixtures('email_zimbra_free_fr.txt'));
+        $fragments = $email->getFragments();
+        $this->assertStringContainsString('Michael Scott', $fragments[0]);
+        $this->assertStringNotContainsString('Toby Flenderson', $fragments[0]);
+    }
+
+    public function testEmailFreeZimbraEn()
+    {
+        $email     = $this->parser->parse($this->getFixtures('email_zimbra_free_en.txt'));
+        $fragments = $email->getFragments();
+        $this->assertStringContainsString('Michael Scott', $fragments[0]);
+        $this->assertStringNotContainsString('Toby Flenderson', $fragments[0]);
+    }
+
     public function testReadsEmailWithCorrectSignature()
     {
         $email     = $this->parser->parse($this->getFixtures('correct_sig.txt'));
diff --git a/tests/Fixtures/email_zimbra_free_en.txt b/tests/Fixtures/email_zimbra_free_en.txt
new file mode 100644
index 0000000..a5bef91
--- /dev/null
+++ b/tests/Fixtures/email_zimbra_free_en.txt
@@ -0,0 +1,16 @@
+Bonjour Toby,
+
+​Text paragraph 1​
+
+On this line is the issue
+
+​Text paragraph 2​
+
+Michael Scott
+
+- On 2021-04-09 23:25, Toby Flenderson a wrote:
+> [1]
+>
+>  BONJOUR Mike,
+>
+>  Vous venez de recevoir un message. Le visiteur vous l'envoie depuis:
\ No newline at end of file
diff --git a/tests/Fixtures/email_zimbra_free_fr.txt b/tests/Fixtures/email_zimbra_free_fr.txt
new file mode 100644
index 0000000..38ff581
--- /dev/null
+++ b/tests/Fixtures/email_zimbra_free_fr.txt
@@ -0,0 +1,17 @@
+Bonjour ​Test​,
+
+​Text paragraph 1​
+
+
+Le paragraphe qui pose problème.
+
+​Text paragraph 2​
+
+Michael Scott
+
+- Le 2021-04-09 23:25, Toby Flenderson a écrit :
+> [1]
+>
+>  BONJOUR TEST,
+>
+>  Vous venez de recevoir un message. Le visiteur vous l'envoie depuis:
\ No newline at end of file