Repository: langchain-ai/langchain (Python)
Date: 2023-05-05 17:05:02+00:00
Task type: Bug Fix

Issue:
WhatsAppChatLoader doesn't work on chats exported from WhatsApp
### System Info

langchain               0.0.158
Mac OS M1
Python 3.11

### Who can help?

@ey

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Use 'Export Chat' feature on WhatsApp.
2. Observe this format for the txt file
```
[11/8/21, 9:41:32 AM] User name: Message text
```

The regular expression used by WhatsAppChatLoader doesn't parse this format successfully

### Expected behavior

Parsing fails


Issue discussion:
it also doesn't work on Ukrainian date format, e.g.

```
[05.05.23, 15:45:46] User: text
```

---

I used the following input formats:
```
[05.05.23, 15:48:11] James: Hi here
[11/8/21, 9:41:32 AM] User name: Message 123
1/23/23, 3:19 AM - User 2: Bye!
1/23/23, 3:22_AM - User 1: And let me know if anything changes
```

New regex that seems to work with all three:
```python

message_line_regex = r"""
    \[?                      # Optional opening square bracket
    (                        # Start of group 1
        \d{1,2}              # Match 1-2 digits for the day
        [\/.]                # Match a forward slash or period as the date separator
        \d{1,2}              # Match 1-2 digits for the month
        [\/.]                # Match a forward slash or period as the date separator
        \d{2,4}              # Match 2-4 digits for the year
        ,\s                  # Match a comma and a space
        \d{1,2}              # Match 1-2 digits for the hour
        :\d{2}               # Match 2 digits for the minutes
        (?:                  # Optional group for seconds
            :\d{2}           # Match 2 digits for the seconds
        )?                   # Make seconds group optional
        (?:[ _](?:AM|PM))?   # Optional space or underscore and AM/PM suffix for 12-hour format
    )                        # End of group 1
    \]?                      # Optional closing square bracket
    [\s-]*                   # Match any number of spaces or hyphens
    ([\w\s]+)                # Match and capture one or more word characters or spaces as group 2 (the sender)
    [:]+                     # Match one or more colons
    \s                       # Match a single space
    (.+)                     # Match and capture one or more of any character as group 3 (the message content)
"""
```

I can make a PR, but should I test any other formats before?

Solution patch:
diff --git a/langchain/document_loaders/whatsapp_chat.py b/langchain/document_loaders/whatsapp_chat.py
--- a/langchain/document_loaders/whatsapp_chat.py
+++ b/langchain/document_loaders/whatsapp_chat.py
@@ -26,16 +26,31 @@ def load(self) -> List[Document]:
         with open(p, encoding="utf8") as f:
             lines = f.readlines()
 
-        message_line_regex = (
-            r"(\d{1,2}/\d{1,2}/\d{2,4}, "
-            r"\d{1,2}:\d{1,2}[ _]?(?:AM|PM)?) - "
-            r"(.*?): (.*)"
-        )
-        for line in lines:
-            result = re.match(
-                message_line_regex,
-                line.strip(),
+        message_line_regex = r"""
+            \[?
+            (
+                \d{1,2}
+                [\/.]
+                \d{1,2}
+                [\/.]
+                \d{2,4}
+                ,\s
+                \d{1,2}
+                :\d{2}
+                (?:
+                    :\d{2}
+                )?
+                (?:[ _](?:AM|PM))?
             )
+            \]?
+            [\s-]*
+            ([\w\s]+)
+            [:]+
+            \s
+            (.+)
+        """
+        for line in lines:
+            result = re.match(message_line_regex, line.strip(), flags=re.VERBOSE)
             if result:
                 date, sender, text = result.groups()
                 text_content += concatenate_rows(date, sender, text)


Test patch:
diff --git a/tests/integration_tests/document_loaders/test_whatsapp_chat.py b/tests/integration_tests/document_loaders/test_whatsapp_chat.py
new file mode 100644
--- /dev/null
+++ b/tests/integration_tests/document_loaders/test_whatsapp_chat.py
@@ -0,0 +1,19 @@
+from pathlib import Path
+
+from langchain.document_loaders import WhatsAppChatLoader
+
+
+def test_whatsapp_chat_loader() -> None:
+    """Test WhatsAppChatLoader."""
+    file_path = Path(__file__).parent.parent / "examples" / "whatsapp_chat.txt"
+    loader = WhatsAppChatLoader(str(file_path))
+    docs = loader.load()
+
+    assert len(docs) == 1
+    assert docs[0].metadata["source"] == str(file_path)
+    assert docs[0].page_content == (
+        "James on 05.05.23, 15:48:11: Hi here\n\n"
+        "User name on 11/8/21, 9:41:32 AM: Message 123\n\n"
+        "User 2 on 1/23/23, 3:19 AM: Bye!\n\n"
+        "User 1 on 1/23/23, 3:22_AM: And let me know if anything changes\n\n"
+    )
diff --git a/tests/integration_tests/examples/whatsapp_chat.txt b/tests/integration_tests/examples/whatsapp_chat.txt
new file mode 100644
--- /dev/null
+++ b/tests/integration_tests/examples/whatsapp_chat.txt
@@ -0,0 +1,4 @@
+[05.05.23, 15:48:11] James: Hi here
+[11/8/21, 9:41:32 AM] User name: Message 123
+1/23/23, 3:19 AM - User 2: Bye!
+1/23/23, 3:22_AM - User 1: And let me know if anything changes
\ No newline at end of file