Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Unknown argument X-GM-LABELS error while fetching mails via IMAP
### Description

Currently when my paperless-ngx instance tries to fetch mails over IMAP with the option to use tagging enabled, it fails with an error "Unknown argument X-GM-LABELS".
The IMAP mailserver queried is my dovecot (within mailcow-dockerized), which does not know/support "X-GM-LABELS" and therefore returns the error. I also confirmed this by running the query locally on the dovecot - it returned the same error.

This seems to be related to #1433 as it adds the support for gmail accounts and use the "gmail_label" which imap-tools translates to "X-GM-LABELS".

I tried a quick and dirty workaround and removed "gmail_label" for one run from "criterias" and it worked just fine and as expected.

As an idea: maybe it is possible to check the mailbox host ("M._host") for gmail/googlemail as it is done in the post consume (post_consume) and only then use "gmail_label"/"X-GM-LABELS"?

### Steps to reproduce

1. Configure a mail account with a non-gmail mailserver (f.e. dovecot)
2. Create a rule with action "Tag" (5)
3. Let paperless try to fetch mails

### Webserver logs

```bash
[2022-10-06 23:14:56,558] [DEBUG] [paperless_mail] Processing mail account <redacted>
[2022-10-06 23:14:56,673] [DEBUG] [paperless_mail] Account <redacted>: Processing 1 rule(s)
[2022-10-06 23:14:56,676] [DEBUG] [paperless_mail] Rule <redacted>.Catch all: Selecting folder INBOX
[2022-10-06 23:14:56,691] [DEBUG] [paperless_mail] Rule <redacted>.Catch all: Searching folder with criteria (NOT (X-GM-LABELS "Paperless") SINCE 6-Sep-2022 UNKEYWORD Paperless)
[2022-10-06 23:14:56,707] [ERROR] [paperless_mail] Rule <redacted>.Catch all: Error while processing rule: SEARCH command error: BAD [b'Error in IMAP command SEARCH: Unknown argument X-GM-LABELS (0.001 + 0.000 secs).']
Traceback (most recent call last):
  File "/usr/src/paperless/src/paperless_mail/mail.py", line 218, in handle_mail_account
    total_processed_files += self.handle_mail_rule(M, rule)
  File "/usr/src/paperless/src/paperless_mail/mail.py", line 291, in handle_mail_rule
    for message in messages:
  File "/usr/local/lib/python3.9/site-packages/imap_tools/mailbox.py", line 170, in fetch
    nums = tuple((reversed if reverse else iter)(self.numbers(criteria, charset)))[limit_range]
  File "/usr/local/lib/python3.9/site-packages/imap_tools/mailbox.py", line 108, in numbers
    search_result = self.client.search(charset, encoded_criteria)
  File "/usr/local/lib/python3.9/imaplib.py", line 732, in search
    typ, dat = self._simple_command(name, 'CHARSET', charset, *criteria)
  File "/usr/local/lib/python3.9/imaplib.py", line 1230, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/local/lib/python3.9/imaplib.py", line 1055, in _command_complete
    raise self.error('%s command error: %s %s' % (name, typ, data))
imaplib.IMAP4.error: SEARCH command error: BAD [b'Error in IMAP command SEARCH: Unknown argument X-GM-LABELS (0.001 + 0.000 secs).']
```


### Paperless-ngx version

1.9.2

### Host OS

UnraidOS

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Quick addition: this was added to imap-tools within https://github.com/ikvk/imap_tools/pull/43 they also discussed, that this is non-standard and gmail specific.
The best would be to check for the imap capability X-GM-EXT-1 [Google dev docu on IMAP extensions](https://developers.google.com/gmail/imap/imap-extensions) and only then use X-GM-LABELS but I do not think imap-tools provides the feature to list the capabilities.

Resolving patch:
diff --git a/src/paperless_mail/mail.py b/src/paperless_mail/mail.py
--- a/src/paperless_mail/mail.py
+++ b/src/paperless_mail/mail.py
@@ -4,6 +4,7 @@
 from datetime import date
 from datetime import timedelta
 from fnmatch import fnmatch
+from typing import Dict
 
 import magic
 import pathvalidate
@@ -30,7 +31,7 @@ class MailError(Exception):
 
 
 class BaseMailAction:
-    def get_criteria(self):
+    def get_criteria(self) -> Dict:
         return {}
 
     def post_consume(self, M, message_uids, parameter):
@@ -78,7 +79,7 @@ def post_consume(self, M: MailBox, message_uids, parameter):
             M.flag(message_uids, [self.keyword], True)
 
 
-def get_rule_action(rule):
+def get_rule_action(rule) -> BaseMailAction:
     if rule.action == MailRule.MailAction.FLAG:
         return FlagMailAction()
     elif rule.action == MailRule.MailAction.DELETE:
@@ -108,7 +109,7 @@ def make_criterias(rule):
     return {**criterias, **get_rule_action(rule).get_criteria()}
 
 
-def get_mailbox(server, port, security):
+def get_mailbox(server, port, security) -> MailBox:
     if security == MailAccount.ImapSecurity.NONE:
         mailbox = MailBoxUnencrypted(server, port)
     elif security == MailAccount.ImapSecurity.STARTTLS:
@@ -167,7 +168,7 @@ def get_correspondent(self, message: MailMessage, rule):
                 "Unknown correspondent selector",
             )  # pragma: nocover
 
-    def handle_mail_account(self, account):
+    def handle_mail_account(self, account: MailAccount):
 
         self.renew_logging_group()
 
@@ -181,7 +182,14 @@ def handle_mail_account(self, account):
                 account.imap_security,
             ) as M:
 
+                supports_gmail_labels = "X-GM-EXT-1" in M.client.capabilities
+                supports_auth_plain = "AUTH=PLAIN" in M.client.capabilities
+
+                self.log("debug", f"GMAIL Label Support: {supports_gmail_labels}")
+                self.log("debug", f"AUTH=PLAIN Support: {supports_auth_plain}")
+
                 try:
+
                     M.login(account.username, account.password)
 
                 except UnicodeEncodeError:
@@ -215,7 +223,11 @@ def handle_mail_account(self, account):
 
                 for rule in account.rules.order_by("order"):
                     try:
-                        total_processed_files += self.handle_mail_rule(M, rule)
+                        total_processed_files += self.handle_mail_rule(
+                            M,
+                            rule,
+                            supports_gmail_labels,
+                        )
                     except Exception as e:
                         self.log(
                             "error",
@@ -233,7 +245,12 @@ def handle_mail_account(self, account):
 
         return total_processed_files
 
-    def handle_mail_rule(self, M: MailBox, rule):
+    def handle_mail_rule(
+        self,
+        M: MailBox,
+        rule: MailRule,
+        supports_gmail_labels: bool = False,
+    ):
 
         self.log("debug", f"Rule {rule}: Selecting folder {rule.folder}")
 
@@ -261,11 +278,19 @@ def handle_mail_rule(self, M: MailBox, rule):
             ) from err
 
         criterias = make_criterias(rule)
-        criterias_imap = AND(**criterias)
+
+        # Deal with the Gmail label extension
         if "gmail_label" in criterias:
+
             gmail_label = criterias["gmail_label"]
             del criterias["gmail_label"]
-            criterias_imap = AND(NOT(gmail_label=gmail_label), **criterias)
+
+            if not supports_gmail_labels:
+                criterias_imap = AND(**criterias)
+            else:
+                criterias_imap = AND(NOT(gmail_label=gmail_label), **criterias)
+        else:
+            criterias_imap = AND(**criterias)
 
         self.log(
             "debug",


Test patch:
diff --git a/src/paperless_mail/tests/test_live_mail.py b/src/paperless_mail/tests/test_live_mail.py
new file mode 100644
--- /dev/null
+++ b/src/paperless_mail/tests/test_live_mail.py
@@ -0,0 +1,70 @@
+import os
+
+import pytest
+from django.test import TestCase
+from paperless_mail.mail import MailAccountHandler
+from paperless_mail.mail import MailError
+from paperless_mail.models import MailAccount
+from paperless_mail.models import MailRule
+
+# Only run if the environment is setup
+# And the environment is not empty (forks, I think)
+@pytest.mark.skipif(
+    "PAPERLESS_MAIL_TEST_HOST" not in os.environ
+    or not len(os.environ["PAPERLESS_MAIL_TEST_HOST"]),
+    reason="Live server testing not enabled",
+)
+class TestMailLiveServer(TestCase):
+    def setUp(self) -> None:
+
+        self.mail_account_handler = MailAccountHandler()
+        self.account = MailAccount.objects.create(
+            name="test",
+            imap_server=os.environ["PAPERLESS_MAIL_TEST_HOST"],
+            username=os.environ["PAPERLESS_MAIL_TEST_USER"],
+            password=os.environ["PAPERLESS_MAIL_TEST_PASSWD"],
+            imap_port=993,
+        )
+
+        return super().setUp()
+
+    def tearDown(self) -> None:
+        self.account.delete()
+        return super().tearDown()
+
+    def test_process_non_gmail_server_flag(self):
+
+        try:
+            rule1 = MailRule.objects.create(
+                name="testrule",
+                account=self.account,
+                action=MailRule.MailAction.FLAG,
+            )
+
+            self.mail_account_handler.handle_mail_account(self.account)
+
+            rule1.delete()
+
+        except MailError as e:
+            self.fail(f"Failure: {e}")
+        except Exception as e:
+            pass
+
+    def test_process_non_gmail_server_tag(self):
+
+        try:
+
+            rule2 = MailRule.objects.create(
+                name="testrule",
+                account=self.account,
+                action=MailRule.MailAction.TAG,
+            )
+
+            self.mail_account_handler.handle_mail_account(self.account)
+
+            rule2.delete()
+
+        except MailError as e:
+            self.fail(f"Failure: {e}")
+        except Exception as e:
+            pass
diff --git a/src/paperless_mail/tests/test_mail.py b/src/paperless_mail/tests/test_mail.py
--- a/src/paperless_mail/tests/test_mail.py
+++ b/src/paperless_mail/tests/test_mail.py
@@ -47,15 +47,16 @@ def set(self, new_folder):
 
 
 class BogusClient:
+    def __init__(self, messages):
+        self.messages: List[MailMessage] = messages
+        self.capabilities: List[str] = []
+
     def __enter__(self):
         return self
 
     def __exit__(self, exc_type, exc_val, exc_tb):
         pass
 
-    def __init__(self, messages):
-        self.messages: List[MailMessage] = messages
-
     def authenticate(self, mechanism, authobject):
         # authobject must be a callable object
         auth_bytes = authobject(None)
@@ -80,12 +81,6 @@ class BogusMailBox(ContextManager):
     # Note the non-ascii characters here
     UTF_PASSWORD: str = "w57äöüw4b6huwb6nhu"
 
-    def __enter__(self):
-        return self
-
-    def __exit__(self, exc_type, exc_val, exc_tb):
-        pass
-
     def __init__(self):
         self.messages: List[MailMessage] = []
         self.messages_spam: List[MailMessage] = []
@@ -93,6 +88,12 @@ def __init__(self):
         self.client = BogusClient(self.messages)
         self._host = ""
 
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        pass
+
     def updateClient(self):
         self.client = BogusClient(self.messages)
 
@@ -648,6 +649,7 @@ def test_handle_mail_account_tag(self):
 
     def test_handle_mail_account_tag_gmail(self):
         self.bogus_mailbox._host = "imap.gmail.com"
+        self.bogus_mailbox.client.capabilities = ["X-GM-EXT-1"]
 
         account = MailAccount.objects.create(
             name="test",