Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Email processing fails because of non-ascii character in email account password
### Description

The email processing fails if the email account password contains a non-ascii character.


### Expected behavior

Non-ascii characters should be acceptable in passwords anywhere.

### Steps to reproduce

1. Set up an email account with a password containing the letter ž
2. Connect the email account to paperless-ngx
3. Review the logs.

### Webserver logs

```bash
`[2022-04-07 09:02:15,780] [ERROR] [paperless.mail.tasks] Error while processing mail account XXXXXX

Traceback (most recent call last):

  File "/opt/paperless/src/paperless_mail/mail.py", line 166, in handle_mail_account

    M.login(account.username, account.password)

  File "/opt/paperless/.local/lib/python3.9/site-packages/imap_tools/mailbox.py", line 60, in login

    login_result = self.box._simple_command('LOGIN', username, self.box._quote(password))  # noqa

  File "/usr/lib/python3.9/imaplib.py", line 1230, in _simple_command

    return self._command_complete(name, self._command(name, *args))

  File "/usr/lib/python3.9/imaplib.py", line 987, in _command

    arg = bytes(arg, self._encoding)

UnicodeEncodeError: 'ascii' codec can't encode character '\u017e' in position 16: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/opt/paperless/src/paperless_mail/tasks.py", line 14, in process_mail_accounts

    total_new_documents += MailAccountHandler().handle_mail_account(account)

  File "/opt/paperless/src/paperless_mail/mail.py", line 168, in handle_mail_account

    raise MailError(f"Error while authenticating account {account}")

paperless_mail.mail.MailError: Error while authenticating account XXXXXX`
```


### Screenshots

_No response_

### Paperless-ngx version

1.6.0

### Host OS

Linux

### Installation method

Bare metal

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
![image](https://user-images.githubusercontent.com/103181380/162260708-ac746dbc-bb25-49e8-ac10-f228e2d856b9.png)
This is the character set for the email account.
In my testing, this looks to be an issue in imap_tools.
```
>>> from imap_tools import MailBox
>>> m_box = MailBox("imap.gmx.com", 993)
>>> m_box.login("dev-testing-paperless@gmx.com", "réception")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tank/users/trenton/projects/paperless/paperless-ngx/.venv/lib/python3.8/site-packages/imap_tools/mailbox.py", line 63, in login
    login_result = self.client._simple_command('LOGIN', username, self.client._quote(password))  # noqa
  File "/usr/lib/python3.8/imaplib.py", line 1205, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python3.8/imaplib.py", line 962, in _command
    arg = bytes(arg, self._encoding)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 2: ordinal not in range(128)
```

(obviously spoofed values in the log here)
Well, it's actually more down in IMAP I think.  I don't think the basic LOGIN command only supports non US-ASCII charsets and doesn't have an optional CHARSET which can be provided.  In fact, per [the RFC](https://datatracker.ietf.org/doc/html/rfc3501#section-6.2.3) we shouldn't be using login except as a last resort.  The better command I guess is [authenticate](https://datatracker.ietf.org/doc/html/rfc3501#section-6.2.2).

I wonder if the solution is to update mail fetching to support a user name and token (I think that's OAuth2?)
Unfortunately, as suspected, this isn't a bug with imap_tools or the Python IMAP client.  The IMAP specification only supports ASCII for a login with username and password.  The encoding being set for an account is for *messages* (which I agree is not clear).

Does your email server provide application specific passwords?  Does it support some form of OAUTH 2 or other authentication method?  If it does, how long do those tokens last?
Thanks for the comment, I am afraid it goes beyond my limited knowledge: I run Postfix 3.4.14  and Dovecot 2.3.4.1 and use non-ascii letters in passwords without problems in desktop, mobile and web mail clients. I looked around and found this [RFC](https://bugs.python.org/issue21800) and this [Python bug report](https://bugs.python.org/issue21800) related to it and they confirm the need for using **authenticate**.
Yeah, that's about what I found as well.  That RFC [specifically tells us](https://datatracker.ietf.org/doc/html/rfc6855#section-5) to use `authenticate`.

So the solution is to use that, but I'm afraid it's beyond my understanding of IMAP currently.

Resolving patch:
diff --git a/src/paperless_mail/mail.py b/src/paperless_mail/mail.py
--- a/src/paperless_mail/mail.py
+++ b/src/paperless_mail/mail.py
@@ -3,6 +3,7 @@
 from datetime import date
 from datetime import timedelta
 from fnmatch import fnmatch
+from imaplib import IMAP4
 
 import magic
 import pathvalidate
@@ -145,7 +146,7 @@ def get_correspondent(self, message: MailMessage, rule):
 
         else:
             raise NotImplementedError(
-                "Unknwown correspondent selector",
+                "Unknown correspondent selector",
             )  # pragma: nocover
 
     def handle_mail_account(self, account):
@@ -164,7 +165,33 @@ def handle_mail_account(self, account):
 
             try:
                 M.login(account.username, account.password)
+
+            except UnicodeEncodeError:
+                self.log("debug", "Falling back to AUTH=PLAIN")
+                try:
+                    # rfc2595 section 6 - PLAIN SASL mechanism
+                    client: IMAP4 = M.client
+                    encoded = (
+                        b"\0"
+                        + account.username.encode("utf8")
+                        + b"\0"
+                        + account.password.encode("utf8")
+                    )
+                    # Assumption is the server supports AUTH=PLAIN capability
+                    # Could check the list with client.capability(), but then what?
+                    # We're failing anyway then
+                    client.authenticate("PLAIN", lambda x: encoded)
+
+                    # Need to transition out of AUTH state to SELECTED
+                    M.folder.set("INBOX")
+                except Exception:
+                    self.log(
+                        "error",
+                        "Unable to authenticate with mail server using AUTH=PLAIN",
+                    )
+                    raise MailError(f"Error while authenticating account {account}")
             except Exception:
+                self.log("error", "Unable to authenticate with mail server")
                 raise MailError(f"Error while authenticating account {account}")
 
             self.log(
@@ -184,7 +211,7 @@ def handle_mail_account(self, account):
 
         return total_processed_files
 
-    def handle_mail_rule(self, M, rule):
+    def handle_mail_rule(self, M: MailBox, rule):
 
         self.log("debug", f"Rule {rule}: Selecting folder {rule.folder}")
 


Test patch:
diff --git a/src/paperless_mail/tests/test_mail.py b/src/paperless_mail/tests/test_mail.py
--- a/src/paperless_mail/tests/test_mail.py
+++ b/src/paperless_mail/tests/test_mail.py
@@ -17,6 +17,7 @@
 from imap_tools import EmailAddress
 from imap_tools import FolderInfo
 from imap_tools import MailboxFolderSelectError
+from imap_tools import MailboxLoginError
 from imap_tools import MailMessage
 from imap_tools import MailMessageFlags
 from paperless_mail import tasks
@@ -44,6 +45,14 @@ def set(self, new_folder):
         self.current_folder = new_folder
 
 
+class BogusClient(object):
+    def authenticate(self, mechanism, authobject):
+        # authobject must be a callable object
+        auth_bytes = authobject(None)
+        if auth_bytes != b"\x00admin\x00w57\xc3\xa4\xc3\xb6\xc3\xbcw4b6huwb6nhu":
+            raise MailboxLoginError("BAD", "OK")
+
+
 class BogusMailBox(ContextManager):
     def __enter__(self):
         return self
@@ -55,10 +64,14 @@ def __init__(self):
         self.messages: List[MailMessage] = []
         self.messages_spam: List[MailMessage] = []
         self.folder = BogusFolderManager()
+        self.client = BogusClient()
 
     def login(self, username, password):
-        if not (username == "admin" and password == "secret"):
-            raise Exception()
+        # This will raise a UnicodeEncodeError if the password is not ASCII only
+        password.encode("ascii")
+        # Otherwise, check for correct values
+        if username != "admin" or password not in {"secret"}:
+            raise MailboxLoginError("BAD", "OK")
 
     def fetch(self, criteria, mark_seen, charset=""):
         msg = self.messages
@@ -821,6 +834,66 @@ def test_filters(self):
         self.assertEqual(len(self.bogus_mailbox.messages), 2)
         self.assertEqual(self.async_task.call_count, 5)
 
+    def test_auth_plain_fallback(self):
+        """
+        GIVEN:
+            - Mail account with password containing non-ASCII characters
+        THEN:
+            - Should still authenticate to the mail account
+        """
+        account = MailAccount.objects.create(
+            name="test",
+            imap_server="",
+            username="admin",
+            # Note the non-ascii characters here
+            password="w57äöüw4b6huwb6nhu",
+        )
+
+        _ = MailRule.objects.create(
+            name="testrule",
+            account=account,
+            action=MailRule.MailAction.MARK_READ,
+        )
+
+        self.assertEqual(len(self.bogus_mailbox.messages), 3)
+        self.assertEqual(self.async_task.call_count, 0)
+        self.assertEqual(len(self.bogus_mailbox.fetch("UNSEEN", False)), 2)
+
+        self.mail_account_handler.handle_mail_account(account)
+
+        self.assertEqual(self.async_task.call_count, 2)
+        self.assertEqual(len(self.bogus_mailbox.fetch("UNSEEN", False)), 0)
+        self.assertEqual(len(self.bogus_mailbox.messages), 3)
+
+    def test_auth_plain_fallback_fails_still(self):
+        """
+        GIVEN:
+            - Mail account with password containing non-ASCII characters
+            - Incorrect password value
+        THEN:
+            - Should raise a MailError for the account
+        """
+        account = MailAccount.objects.create(
+            name="test",
+            imap_server="",
+            username="admin",
+            # Note the non-ascii characters here
+            # Passes the check in login, not in authenticate
+            password="réception",
+        )
+
+        _ = MailRule.objects.create(
+            name="testrule",
+            account=account,
+            action=MailRule.MailAction.MARK_READ,
+        )
+
+        self.assertRaises(
+            MailError,
+            self.mail_account_handler.handle_mail_account,
+            account,
+        )
+
 
 class TestManagementCommand(TestCase):
     @mock.patch(