Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Migration from ng TypeError: argument of type 'EmailAddress' is not iterable
Hi.


Just migrated from ng by replacing the image and see the following errors in the log:

```
paperless-ng | [2022-03-15 15:48:40,811] [ERROR] [paperless_mail] Rule Alex Gmail.Alex mailbox: Error while processing mail 137569: argument of type 'EmailAddress' is not iterable
paperless-ng | Traceback (most recent call last):
paperless-ng |   File "/usr/src/paperless/src/paperless_mail/mail.py", line 222, in handle_mail_rule
paperless-ng |     processed_files = self.handle_message(message, rule)
paperless-ng |   File "/usr/src/paperless/src/paperless_mail/mail.py", line 266, in handle_message
paperless-ng |     correspondent = self.get_correspondent(message, rule)
paperless-ng |   File "/usr/src/paperless/src/paperless_mail/mail.py", line 138, in get_correspondent
paperless-ng |     and "name" in message.from_values
paperless-ng | TypeError: argument of type 'EmailAddress' is not iterable
```

docker-compose file:
```
version: "3.4"
services:
  broker:
    image: redis:6.0
    restart: unless-stopped
    container_name: paperless-redis
    volumes:
      - ./redis_data:/data
    command: redis-server --appendonly yes
  webserver:
    image: ghcr.io/paperless-ngx/paperless-ngx:latest
    container_name: paperless-ng
    restart: unless-stopped
    depends_on:
      - broker
    ports:
      - 8008:8000
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000"]
      interval: 30s
      timeout: 10s
      retries: 5
    volumes:
      - ./data:/usr/src/paperless/data
      - ./media:/usr/src/paperless/media
      - ./export:/usr/src/paperless/export
      - ./consume:/usr/src/paperless/consume
    env_file: docker-compose.env
    environment:
      PAPERLESS_REDIS: redis://broker:6379
```

docker-compose.env
```
USERMAP_UID=1000
USERMAP_GID=1000

PAPERLESS_OCR_LANGUAGES=eng heb

PAPERLESS_SECRET_KEY=edited

PAPERLESS_TIME_ZONE=Asia/Jerusalem

PAPERLESS_OCR_LANGUAGE=heb
PAPERLESS_FILENAME_FORMAT={document_type}/{created_year}/{created_month}/{title}
```
[BUG] Email rule with correspondent assignment from sender doesn't work
Docker on Synology NAS, migrated from linuxserver/paperless-ng

All is working so far (after adjusting environment variables) except mail rule:
- Create mail rule with "Korrespondent zuweisen von:" (probably "Assign correspondent from") as third option "Absendername benutzen (oder E-Mail-Adressen, wenn nicht verfügbar)" (probably "Use sender name (or mail addresses if not available)")
- After running rule check mail log:
___
`[2022-03-12 16:55:58,489] [ERROR] [paperless_mail] Rule xx.xxx@gmail.com.Gmail 30d PDF only: Error while processing mail 40169: argument of type 'EmailAddress' is not iterable

Traceback (most recent call last):

  File "/usr/src/paperless/src/paperless_mail/mail.py", line 222, in handle_mail_rule

    processed_files = self.handle_message(message, rule)

  File "/usr/src/paperless/src/paperless_mail/mail.py", line 266, in handle_message

    correspondent = self.get_correspondent(message, rule)

  File "/usr/src/paperless/src/paperless_mail/mail.py", line 138, in get_correspondent

    and "name" in message.from_values

TypeError: argument of type 'EmailAddress' is not iterable`
___
After changing this setting to "Keinen Korrespondenten zuweisen" (perhaps "Don't assign correspondent") the rule works and produces content.

Best,
Norbert


Discussion:
Thank you for the report.  This was a breaking change introduced in `imap_tools` v0.48.0.  I'll work on a fix.
@stumpylog might this be related to https://github.com/paperless-ngx/paperless-ngx/issues/293?
Yes, the traceback is the same, so I would imagine so.
I have the same issue running the Stack on my Synology NAS in Docker.

```
[2022-03-13 11:45:18,265] [ERROR] [paperless_mail] Rule <xx>.<xx>: Error while processing mail 1: argument of type '
EmailAddress' is not iterable                                                                                                                   
Traceback (most recent call last):                                                                                                              
  File "/usr/src/paperless/src/paperless_mail/mail.py", line 222, in handle_mail_rule                                                           
    processed_files = self.handle_message(message, rule)                                                                                        
  File "/usr/src/paperless/src/paperless_mail/mail.py", line 266, in handle_message                                                             
    correspondent = self.get_correspondent(message, rule)                                                                                       
  File "/usr/src/paperless/src/paperless_mail/mail.py", line 138, in get_correspondent                                                          
    and "name" in message.from_values                                                                                                           
TypeError: argument of type 'EmailAddress' is not iterable
```

I used this setting (which leads to the error above):
Assign correspondent from:
-> Use correspondent selected below
-> Assign this correspondent: (My user)

It's working with:
-> Do not assign a correspondent
Attached you can see the Docker Images i use if this helps. I run the Stack without docker-compose.

<img width="250" alt="Screenshot 2022-03-13 at 17 23 27" src="https://user-images.githubusercontent.com/17730136/158069139-60cec69e-bfc1-4447-869f-2c0799c17e60.png">
I had no luck reproducing this yet. See below my steps:

```bash
$ docker-compose -f docker-compose.sqlite.yml up -d
$ docker-compose -f docker-compose.sqlite.yml run --rm webserver createsuperuser
[...]
```

Login and create mail account, correspondent and mail rule as follows:

![image](https://user-images.githubusercontent.com/5375334/158256011-d498c20d-b3df-43dd-a617-aedecc7c1556.png)
![image](https://user-images.githubusercontent.com/5375334/158255270-483a5884-ac9e-4654-a7bc-37a60792d50f.png)

I created an unread mail in my inbox with a PDF `bug.pdf` attached containing the word `bug`. I then triggered the mail consumer:

```bash
$ docker exec -it 218 mail_fetcher
[2022-03-14 20:21:23,246] [INFO] [paperless_mail] Rule Test Mail.bug_rule: Consuming attachment bug.pdf from mail bug from mail@XXXXX
20:21:23 [Q] INFO Enqueued 1
```

Log looks fine:

```bash
webserver_1  | [2022-03-14 20:21:23,411] [INFO] [paperless.consumer] Consuming bug.pdf
webserver_1  | [2022-03-14 20:21:26,814] [INFO] [paperless.consumer] Document 2022-03-14 Mail user bug consumption finished
webserver_1  | 20:21:26 [Q] INFO Process-1:1 stopped doing work
webserver_1  | 20:21:26 [Q] INFO Processed [bug.pdf]
webserver_1  | 20:21:27 [Q] INFO recycled worker Process-1:1
webserver_1  | 20:21:27 [Q] INFO Process-1:5 ready for work at 131
```

Did I get something wrong in my test setup? Please assist. Willing to fix the issue once I can reproduce it.

Resolving patch:
diff --git a/src/paperless_mail/mail.py b/src/paperless_mail/mail.py
--- a/src/paperless_mail/mail.py
+++ b/src/paperless_mail/mail.py
@@ -16,6 +16,7 @@
 from imap_tools import MailBox
 from imap_tools import MailboxFolderSelectError
 from imap_tools import MailBoxUnencrypted
+from imap_tools import MailMessage
 from imap_tools import MailMessageFlags
 from paperless_mail.models import MailAccount
 from paperless_mail.models import MailRule
@@ -122,7 +123,7 @@ def get_title(self, message, att, rule):
                 "Unknown title selector.",
             )  # pragma: nocover
 
-    def get_correspondent(self, message, rule):
+    def get_correspondent(self, message: MailMessage, rule):
         c_from = rule.assign_correspondent_from
 
         if c_from == MailRule.CORRESPONDENT_FROM_NOTHING:
@@ -132,12 +133,9 @@ def get_correspondent(self, message, rule):
             return self._correspondent_from_name(message.from_)
 
         elif c_from == MailRule.CORRESPONDENT_FROM_NAME:
-            if (
-                message.from_values
-                and "name" in message.from_values
-                and message.from_values["name"]
-            ):
-                return self._correspondent_from_name(message.from_values["name"])
+            from_values = message.from_values
+            if from_values is not None and len(from_values.name) > 0:
+                return self._correspondent_from_name(from_values.name)
             else:
                 return self._correspondent_from_name(message.from_)
 


Test patch:
diff --git a/src/paperless_mail/tests/test_mail.py b/src/paperless_mail/tests/test_mail.py
--- a/src/paperless_mail/tests/test_mail.py
+++ b/src/paperless_mail/tests/test_mail.py
@@ -1,7 +1,12 @@
+import dataclasses
+import email.contentmanager
 import os
+import random
 import uuid
 from collections import namedtuple
 from typing import ContextManager
+from typing import List
+from typing import Union
 from unittest import mock
 
 from django.core.management import call_command
@@ -9,7 +14,9 @@
 from django.test import TestCase
 from documents.models import Correspondent
 from documents.tests.utils import DirectoriesMixin
+from imap_tools import EmailAddress
 from imap_tools import MailboxFolderSelectError
+from imap_tools import MailMessage
 from imap_tools import MailMessageFlags
 from paperless_mail import tasks
 from paperless_mail.mail import MailAccountHandler
@@ -18,8 +25,16 @@
 from paperless_mail.models import MailRule
 
 
-class BogusFolderManager:
+@dataclasses.dataclass
+class _AttachmentDef(object):
+    filename: str = "a_file.pdf"
+    maintype: str = "application/pdf"
+    subtype: str = "pdf"
+    disposition: str = "attachment"
+    content: bytes = b"a PDF document"
+
 
+class BogusFolderManager:
     current_folder = "INBOX"
 
     def set(self, new_folder):
@@ -36,8 +51,8 @@ def __exit__(self, exc_type, exc_val, exc_tb):
         pass
 
     def __init__(self):
-        self.messages = []
-        self.messages_spam = []
+        self.messages: List[MailMessage] = []
+        self.messages_spam: List[MailMessage] = []
 
     def login(self, username, password):
         if not (username == "admin" and password == "secret"):
@@ -59,7 +74,7 @@ def fetch(self, criteria, mark_seen, charset=""):
 
         if "BODY" in criteria:
             body = criteria[criteria.index("BODY") + 1].strip('"')
-            msg = filter(lambda m: body in m.body, msg)
+            msg = filter(lambda m: body in m.text, msg)
 
         if "FROM" in criteria:
             from_ = criteria[criteria.index("FROM") + 1].strip('"')
@@ -84,7 +99,7 @@ def flag(self, uid_list, flag_set, value):
 
     def move(self, uid_list, folder):
         if folder == "spam":
-            self.messages_spam.append(
+            self.messages_spam += list(
                 filter(lambda m: m.uid in uid_list, self.messages),
             )
             self.messages = list(filter(lambda m: m.uid not in uid_list, self.messages))
@@ -92,44 +107,66 @@ def move(self, uid_list, folder):
             raise Exception()
 
 
+_used_uids = set()
+
+
 def create_message(
-    num_attachments=1,
-    body="",
-    subject="the suject",
-    from_="noone@mail.com",
-    seen=False,
-    flagged=False,
-):
-    message = namedtuple("MailMessage", [])
-
-    message.uid = uuid.uuid4()
-    message.subject = subject
-    message.attachments = []
-    message.from_ = from_
-    message.body = body
-    for i in range(num_attachments):
-        message.attachments.append(create_attachment(filename=f"file_{i}.pdf"))
-
-    message.seen = seen
-    message.flagged = flagged
-
-    return message
-
-
-def create_attachment(
-    filename="the_file.pdf",
-    content_disposition="attachment",
-    payload=b"a PDF document",
-):
-    attachment = namedtuple("Attachment", [])
-    attachment.filename = filename
-    attachment.content_disposition = content_disposition
-    attachment.payload = payload
-    return attachment
+    attachments: Union[int, List[_AttachmentDef]] = 1,
+    body: str = "",
+    subject: str = "the suject",
+    from_: str = "noone@mail.com",
+    seen: bool = False,
+    flagged: bool = False,
+) -> MailMessage:
+    email_msg = email.message.EmailMessage()
+    # TODO: This does NOT set the UID
+    email_msg["Message-ID"] = str(uuid.uuid4())
+    email_msg["Subject"] = subject
+    email_msg["From"] = from_
+    email_msg.set_content(body)
+
+    # Either add some default number of attachments
+    # or the provided attachments
+    if isinstance(attachments, int):
+        for i in range(attachments):
+            attachment = _AttachmentDef(filename=f"file_{i}.pdf")
+            email_msg.add_attachment(
+                attachment.content,
+                maintype=attachment.maintype,
+                subtype=attachment.subtype,
+                disposition=attachment.disposition,
+                filename=attachment.filename,
+            )
+    else:
+        for attachment in attachments:
+            email_msg.add_attachment(
+                attachment.content,
+                maintype=attachment.maintype,
+                subtype=attachment.subtype,
+                disposition=attachment.disposition,
+                filename=attachment.filename,
+            )
 
+    # Convert the EmailMessage to an imap_tools MailMessage
+    imap_msg = MailMessage.from_bytes(email_msg.as_bytes())
 
-def fake_magic_from_buffer(buffer, mime=False):
+    # TODO: Unsure how to add a uid to the actual EmailMessage. This hacks it in,
+    #  based on how imap_tools uses regex to extract it.
+    #  This should be a large enough pool
+    uid = random.randint(1, 10000)
+    while uid in _used_uids:
+        uid = random.randint(1, 10000)
+    _used_uids.add(uid)
+
+    imap_msg._raw_uid_data = f"UID {uid}".encode()
+
+    imap_msg.seen = seen
+    imap_msg.flagged = flagged
 
+    return imap_msg
+
+
+def fake_magic_from_buffer(buffer, mime=False):
     if mime:
         if "PDF" in str(buffer):
             return "application/pdf"
@@ -188,11 +225,19 @@ def reset_bogus_mailbox(self):
     def test_get_correspondent(self):
         message = namedtuple("MailMessage", [])
         message.from_ = "someone@somewhere.com"
-        message.from_values = {"name": "Someone!", "email": "someone@somewhere.com"}
+        message.from_values = EmailAddress(
+            "Someone!",
+            "someone@somewhere.com",
+            "Someone! <someone@somewhere.com>",
+        )
 
         message2 = namedtuple("MailMessage", [])
         message2.from_ = "me@localhost.com"
-        message2.from_values = {"name": "", "email": "fake@localhost.com"}
+        message2.from_values = EmailAddress(
+            "",
+            "fake@localhost.com",
+            "",
+        )
 
         me_localhost = Correspondent.objects.create(name=message2.from_)
         someone_else = Correspondent.objects.create(name="someone else")
@@ -253,7 +298,7 @@ def test_handle_message(self):
         message = create_message(
             subject="the message title",
             from_="Myself",
-            num_attachments=2,
+            attachments=2,
         )
 
         account = MailAccount()
@@ -290,14 +335,15 @@ def test_handle_empty_message(self):
         self.assertEqual(result, 0)
 
     def test_handle_unknown_mime_type(self):
-        message = create_message()
-        message.attachments = [
-            create_attachment(filename="f1.pdf"),
-            create_attachment(
-                filename="f2.json",
-                payload=b"{'much': 'payload.', 'so': 'json', 'wow': true}",
-            ),
-        ]
+        message = create_message(
+            attachments=[
+                _AttachmentDef(filename="f1.pdf"),
+                _AttachmentDef(
+                    filename="f2.json",
+                    content=b"{'much': 'payload.', 'so': 'json', 'wow': true}",
+                ),
+            ],
+        )
 
         account = MailAccount()
         rule = MailRule(assign_title_from=MailRule.TITLE_FROM_FILENAME, account=account)
@@ -312,11 +358,15 @@ def test_handle_unknown_mime_type(self):
         self.assertEqual(kwargs["override_filename"], "f1.pdf")
 
     def test_handle_disposition(self):
-        message = create_message()
-        message.attachments = [
-            create_attachment(filename="f1.pdf", content_disposition="inline"),
-            create_attachment(filename="f2.pdf", content_disposition="attachment"),
-        ]
+        message = create_message(
+            attachments=[
+                _AttachmentDef(
+                    filename="f1.pdf",
+                    disposition="inline",
+                ),
+                _AttachmentDef(filename="f2.pdf"),
+            ],
+        )
 
         account = MailAccount()
         rule = MailRule(assign_title_from=MailRule.TITLE_FROM_FILENAME, account=account)
@@ -330,11 +380,15 @@ def test_handle_disposition(self):
         self.assertEqual(kwargs["override_filename"], "f2.pdf")
 
     def test_handle_inline_files(self):
-        message = create_message()
-        message.attachments = [
-            create_attachment(filename="f1.pdf", content_disposition="inline"),
-            create_attachment(filename="f2.pdf", content_disposition="attachment"),
-        ]
+        message = create_message(
+            attachments=[
+                _AttachmentDef(
+                    filename="f1.pdf",
+                    disposition="inline",
+                ),
+                _AttachmentDef(filename="f2.pdf"),
+            ],
+        )
 
         account = MailAccount()
         rule = MailRule(
@@ -349,13 +403,14 @@ def test_handle_inline_files(self):
         self.assertEqual(self.async_task.call_count, 2)
 
     def test_filename_filter(self):
-        message = create_message()
-        message.attachments = [
-            create_attachment(filename="f1.pdf"),
-            create_attachment(filename="f2.pdf"),
-            create_attachment(filename="f3.pdf"),
-            create_attachment(filename="f2.png"),
-        ]
+        message = create_message(
+            attachments=[
+                _AttachmentDef(filename="f1.pdf"),
+                _AttachmentDef(filename="f2.pdf"),
+                _AttachmentDef(filename="f3.pdf"),
+                _AttachmentDef(filename="f2.png"),
+            ],
+        )
 
         tests = [
             ("*.pdf", ["f1.pdf", "f2.pdf", "f3.pdf"]),
@@ -391,7 +446,7 @@ def test_handle_mail_account_mark_read(self):
             password="secret",
         )
 
-        rule = MailRule.objects.create(
+        _ = MailRule.objects.create(
             name="testrule",
             account=account,
             action=MailRule.ACTION_MARK_READ,
@@ -414,7 +469,7 @@ def test_handle_mail_account_delete(self):
             password="secret",
         )
 
-        rule = MailRule.objects.create(
+        _ = MailRule.objects.create(
             name="testrule",
             account=account,
             action=MailRule.ACTION_DELETE,
@@ -435,7 +490,7 @@ def test_handle_mail_account_flag(self):
             password="secret",
         )
 
-        rule = MailRule.objects.create(
+        _ = MailRule.objects.create(
             name="testrule",
             account=account,
             action=MailRule.ACTION_FLAG,
@@ -458,7 +513,7 @@ def test_handle_mail_account_move(self):
             password="secret",
         )
 
-        rule = MailRule.objects.create(
+        _ = MailRule.objects.create(
             name="testrule",
             account=account,
             action=MailRule.ACTION_MOVE,
@@ -469,7 +524,9 @@ def test_handle_mail_account_move(self):
         self.assertEqual(self.async_task.call_count, 0)
         self.assertEqual(len(self.bogus_mailbox.messages), 3)
         self.assertEqual(len(self.bogus_mailbox.messages_spam), 0)
+
         self.mail_account_handler.handle_mail_account(account)
+
         self.assertEqual(self.async_task.call_count, 1)
         self.assertEqual(len(self.bogus_mailbox.messages), 2)
         self.assertEqual(len(self.bogus_mailbox.messages_spam), 1)
@@ -482,15 +539,14 @@ def test_error_login(self):
             password="wrong",
         )
 
-        try:
+        with self.assertRaises(MailError) as context:
             self.mail_account_handler.handle_mail_account(account)
-        except MailError as e:
-            self.assertTrue(str(e).startswith("Error while authenticating account"))
-        else:
-            self.fail("Should raise exception")
+            self.assertTrue(
+                str(context).startswith("Error while authenticating account"),
+            )
 
     def test_error_skip_account(self):
-        account_faulty = MailAccount.objects.create(
+        _ = MailAccount.objects.create(
             name="test",
             imap_server="",
             username="admin",
@@ -503,7 +559,7 @@ def test_error_skip_account(self):
             username="admin",
             password="secret",
         )
-        rule = MailRule.objects.create(
+        _ = MailRule.objects.create(
             name="testrule",
             account=account,
             action=MailRule.ACTION_MOVE,
@@ -524,7 +580,7 @@ def test_error_skip_rule(self):
             username="admin",
             password="secret",
         )
-        rule = MailRule.objects.create(
+        _ = MailRule.objects.create(
             name="testrule",
             account=account,
             action=MailRule.ACTION_MOVE,
@@ -533,7 +589,7 @@ def test_error_skip_rule(self):
             order=1,
             folder="uuuhhhh",
         )
-        rule2 = MailRule.objects.create(
+        _ = MailRule.objects.create(
             name="testrule2",
             account=account,
             action=MailRule.ACTION_MOVE,
@@ -563,7 +619,7 @@ def get_correspondent_fake(message, rule):
             username="admin",
             password="secret",
         )
-        rule = MailRule.objects.create(
+        _ = MailRule.objects.create(
             name="testrule",
             account=account,
             action=MailRule.ACTION_MOVE,
@@ -587,7 +643,7 @@ def test_error_create_correspondent(self):
             username="admin",
             password="secret",
         )
-        rule = MailRule.objects.create(
+        _ = MailRule.objects.create(
             name="testrule",
             filter_from="amazon@amazon.de",
             account=account,
@@ -676,7 +732,6 @@ class TestManagementCommand(TestCase):
         "paperless_mail.management.commands.mail_fetcher.tasks.process_mail_accounts",
     )
     def test_mail_fetcher(self, m):
-
         call_command("mail_fetcher")
 
         m.assert_called_once()
@@ -711,7 +766,6 @@ def test_all_accounts(self, m):
 
     @mock.patch("paperless_mail.tasks.MailAccountHandler.handle_mail_account")
     def test_single_accounts(self, m):
-
         MailAccount.objects.create(
             name="A",
             imap_server="A",